Miracle-dz/UniFAR-FTUs huggingface dataset:
Quantization for Distributed Optimization

Massive amounts of data have led to the training of large-scale machine learning models on a single worker ineﬃcient. Distributed machine learning methods such as Parallel-SGD have received signiﬁcant interest as a solution to tackle this problem. However, the performance of distributed systems does not scale linearly with the number of workers due to the high network communication cost for synchronizing gradients and parameters. Researchers have proposed techniques such as quantization and sparsiﬁcation to alleviate this problem by compressing the gradients. Most of the compression schemes result in compressed gradients that cannot be directly aggregated with eﬃcient protocols such as all-reduce. In this paper, we present a set of all-reduce compatible gradient compression schemes which signiﬁcantly reduce the communication overhead while maintaining the performance of vanilla SGD. We present the results of our experiments with the CIFAR10 dataset and observations derived during the process. Our compression methods perform better than the in-built methods currently oﬀered by the deep learning frameworks. Code is available at the repository: https://github.com/vineeths96/Gradient-Compression .
Does referent predictability affect the choice of referential form? A computational approach using masked coreference resolution

It is often posited that more predictable parts of a speaker’s meaning tend to be made less explicit, for instance using shorter, less informative words. Studying these dynamics in the domain of referring expressions has proven difficult, with existing studies, both psycholinguistic and corpus-based, providing contradictory results. We test the hypothesis that speakers produce less informative referring expressions (e.g., pronouns vs. full noun phrases) when the context is more informative about the referent, using novel computational estimates of referent predictability. We obtain these estimates training an existing coreference resolution system for English on a new task, masked coreference resolution, giving us a probability distribution over referents that is conditioned on the context but not the referring expression. The resulting system retains standard coreference resolution performance while yielding a better estimate of human-derived referent predictability than previous attempts. A statistical analysis of the relationship between model output and mention form supports the hypothesis that predictability affects the form of a mention, both its morphosyntactic type and its length.
Opacus: User-Friendly Differential Privacy Library in PyTorch

We introduce Opacus, a free, open-source PyTorch library for training deep learning models with differential privacy (hosted at opacus.ai). Opacus is designed for simplicity, flexibility, and speed. It provides a simple and user-friendly API, and enables machine learning practitioners to make a training pipeline private by adding as little as two lines to their code. It supports a wide variety of layers, including multi-head attention, convolution, LSTM, GRU (and generic RNN), and embedding, right out of the box and provides the means for supporting other user-defined layers. Opacus computes batched per-sample gradients, providing higher efficiency compared to the traditional"micro batch"approach. In this paper we present Opacus, detail the principles that drove its implementation and unique features, and benchmark it against other frameworks for training models with differential privacy as well as standard PyTorch.
Logical Credal Networks

This paper introduces Logical Credal Networks, an expressive probabilistic logic that generalizes many prior models that combine logic and probability. Given imprecise information represented by probability bounds and conditional probability bounds of logic formulas, this logic specifies a set of probability distributions over all interpretations. On the one hand, our approach allows propositional and first-order logic formulas with few restrictions, e.g., without requiring acyclicity. On the other hand, it has a Markov condition similar to Bayesian networks and Markov random fields that is critical in real-world applications. Having both these properties makes this logic unique, and we investigate its performance on maximum a posteriori inference tasks, including solving Mastermind games with uncertainty and detecting credit card fraud. The results show that the proposed method outperforms existing approaches, and its advantage lies in aggregating multiple sources of imprecise information.
Every time I fire a conversational designer, the performance of the dialogue system goes down

Incorporating handwritten domain scripts into neural-based task-oriented dialogue systems may be an effective way to reduce the need for large sets of annotated dialogues. In this paper, we investigate how the use of domain scripts written by conversational designers affects the performance of neural-based dialogue systems. To support this investigation, we propose the Conversational-Logic-Injection-in-Neural-Network system (CLINN) where domain scripts are coded in semi-logical rules. By using CLINN, we evaluated semi-logical rules produced by a team of differently-skilled conversational designers. We experimented with the Restaurant domain of the MultiWOZ dataset. Results show that external knowledge is extremely important for reducing the need for annotated examples for conversational systems. In fact, rules from conversational designers used in CLINN significantly outperform a state-of-the-art neural-based dialogue system when trained with smaller sets of annotated dialogues.
Recall and Learn: A Memory-augmented Solver for Math Word Problems

In this article, we tackle the math word problem, namely, automatically answering a mathematical problem according to its textual description. Although recent methods have demonstrated their promising results, most of these methods are based on template-based generation scheme which results in limited generalization capability. To this end, we propose a novel human-like analogical learning method in a recall and learn manner. Our proposed framework is composed of modules of memory, representation, analogy, and reasoning, which are designed to make a new exercise by referring to the exercises learned in the past. Specifically, given a math word problem, the model first retrieves similar questions by a memory module and then encodes the unsolved problem and each retrieved question using a representation module. Moreover, to solve the problem in a way of analogy, an analogy module and a reasoning module with a copy mechanism are proposed to model the interrelationship between the problem and each retrieved question. Extensive experiments on two well-known datasets show the superiority of our proposed algorithm as compared to other state-of-the-art competitors from both overall performance comparison and micro-scope studies.
Accelerated steady-state electrostatic particle-in-cell simulation of Langmuir probes

Accelerated Steady-State Electrostatic Particle-in-Cell Simulation of Langmuir Probes Gregory R. Werner,1 Scott Robertson,1 Thomas G. Jenkins,2 Andrew M. Chap,2, 3 and John R. Cary1, 2 1)Center for Integrated Plasma Studies, University of Colorado, Boulder, Colorado 80309, USA 2)Tech-X Corporation, 5621 Arapahoe Avenue Suite A, Boulder, Colorado 80303, USA 3)Currently at AST & Science, 5825 University Research Ct. #2300, College Park, MD 20740, USA
Use of the Deep Learning Approach to Measure Alveolar Bone Level

AIM The goal was to use a Deep Convolutional Neural Network to measure the radiographic alveolar bone level to aid periodontal diagnosis. MATERIAL AND METHODS A Deep Learning (DL) model was developed by integrating three segmentation networks (bone area, tooth, cementoenamel junction) and image analysis to measure the radiographic bone level and assign radiographic bone loss (RBL) stages. The percentage of RBL was calculated to determine the stage of RBL for each tooth. A provisional periodontal diagnosis was assigned using the 2018 periodontitis classification. RBL percentage, staging, and presumptive diagnosis were compared to the measurements and diagnoses made by the independent examiners. RESULTS The average Dice Similarity Coefficient (DSC) for segmentation was over 0.91. There was no significant difference in RBL percentage measurements determined by DL and examiners (p = 0.65). The Area Under the Receiver Operating Characteristics Curve of RBL stage assignment for stage I, II and III was 0.89, 0.90 and 0.90, respectively. The accuracy of the case diagnosis was 0.85. CONCLUSION The proposed DL model provides reliable RBL measurements and image-based periodontal diagnosis using periapical radiographic images. However, this model has to be further optimized and validated by a larger number of images to facilitate its application.
Multi-source Few-shot Domain Adaptation

Multi-source Domain Adaptation (MDA) aims to transfer predictive models from multiple, fully-labeled source domains to an unlabeled target domain. However, in many applications, relevant labeled source datasets may not be available, and collecting source labels can be as expensive as labeling the target data itself. In this paper, we investigate Multi-source Few-shot Domain Adaptation (MFDA): a new domain adaptation scenario with limited multi-source labels and unlabeled target data. As we show, existing methods often fail to learn discriminative features for both source and target domains in the MFDA setting. Therefore, we propose a novel framework, termed Multi-Source Few-shot Adaptation Network (MSFAN), which can be trained end-to-end in a non-adversarial manner. MSFAN operates by first using a type of prototypical, multi-domain, self-supervised learning to learn features that are not only domain-invariant but also class-discriminative. Second, MSFAN uses a small, labeled support set to enforce feature consistency and domain invariance across domains. Finally, prototypes from multiple sources are leveraged to learn better classifiers. Compared with state-of-the-art MDA methods, MSFAN improves the mean classification accuracy over different domain pairs on MFDA by 20.2%, 9.4%, and 16.2% on Office, Office-Home, and DomainNet, respectively.
Accelerated primal-dual methods for linearly constrained convex optimization problems

This work proposes an accelerated primal-dual dynamical system for affine constrained convex optimization and presents a class of primal-dual methods with nonergodic convergence rates. In continuous level, exponential decay of a novel Lyapunov function is established and in discrete level, implicit, semi-implicit and explicit numerical discretizations for the continuous model are considered sequentially and lead to new accelerated primal-dual methods for solving linearly constrained optimization problems. Special structures of the subproblems in those schemes are utilized to develop efficient inner solvers. In addition, nonergodic convergence rates in terms of primal-dual gap, primal objective residual and feasibility violation are proved via a tailored discrete Lyapunov function. Moreover, our method has also been applied to decentralized distributed optimization for fast and efficient solution.
Philosophy-Guided Mathematical Formalism for Complex Systems Modelling

We recently presented the so-called allagmatic method, which includes a system metamodel providing a framework for describing, modelling, simulating, and interpreting complex systems. Its development and programming was guided by philosophy, especially by Gilbert Simondon’s philosophy of individuation, Alfred North Whitehead’s philosophy of organism, and concepts from cybernetics. Here, a mathematical formalism is presented to better describe and define the system metamodel of the allagmatic method, thereby further generalising it and extending its reach to a more formal treatment and allowing more theoretical studies. By using the formalism, an example for such a further study is provided with mathematical definitions and proofs for model creation and equivalence of cellular automata and artificial neural networks.
Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?

While additional training data improves the robustness of deep neural networks against adversarial examples, it presents the challenge of curating a large number of specific real-world samples. We circumvent this challenge by using additional data from proxy distributions learned by advanced generative models. We first seek to formally understand the transfer of robustness from classifiers trained on proxy distributions to the real data distribution. We prove that the difference between the robustness of a classifier on the two distributions is upper bounded by the conditional Wasserstein distance between them. Next we use proxy distributions to significantly improve the performance of adversarial training on five different datasets. For example, we improve robust accuracy by up to 7.5% and 6.7% in $\ell_{\infty}$ and $\ell_2$ threat model over baselines that are not using proxy distributions on the CIFAR-10 dataset. We also improve certified robust accuracy by 7.6% on the CIFAR-10 dataset. We further demonstrate that different generative models bring a disparate improvement in the performance in robust training. We propose a robust discrimination approach to characterize the impact of individual generative models and further provide a deeper understanding of why current state-of-the-art in diffusion-based generative models are a better choice for proxy distribution than generative adversarial networks.
Evidence for a high-energy tail in the gamma-ray spectra of globular clusters

Millisecond pulsars are very likely the main source of gamma-ray emission from globular clusters. However, the relative contributions of two separate emission processes – curvature radiation from millisecond pulsar magnetospheres versus inverse Compton emission from relativistic pairs launched into the globular cluster environment by millisecond pulsars – have long been unclear. To address this, we search for evidence of inverse Compton emission in 8-yr Fermi–LAT data from the directions of 157 Milky Way globular clusters. We find a mildly statistically significant (3.8σ) correlation between the measured globular cluster gamma-ray luminosities and their photon field energy densities. However, this may also be explained by a hidden correlation between the photon field densities and the stellar encounter rates of globular clusters. Analysed in toto, we demonstrate that the gamma-ray emission of globular clusters can be resolved spectrally into two components: (i) an exponentially cut-off power law and (ii) a pure power law. The latter component – which we uncover at a significance of 8. 2σ – has a power index of 2.79 ± 0.25. It is most naturally interpreted as inverse Compton emission by cosmic-ray electrons and positrons injected by millisecond pulsars. We find the luminosity of this power-law component is comparable to, or slightly smaller than, the luminosity of the curved component, suggesting the fraction of millisecond pulsar spin-down luminosity into relativistic leptons is similar to the fraction of the spin-down luminosity into prompt magnetospheric radiation.
Natural gradient updates for Cholesky factor in Gaussian and structured variational inference

Stochastic gradient methods have enabled variational inference for high-dimensional models and large data sets. However, the direction of steepest ascent in the parameter space of a statistical model is given not by the commonly used Euclidean gradient, but the natural gradient which premultiplies the Euclidean gradient by the inverse of the Fisher information matrix. Use of natural gradients in optimization can improve convergence signiﬁcantly, but inverting the Fisher information matrix is daunting in high-dimensions. The contribution of this article is twofold. First, we derive the natural gradient updates of a Gaussian variational approximation in terms of the mean and Cholesky factor of the covariance matrix, and show that these updates depend only on the ﬁrst derivative of the variational objective function. Second, we provide detailed derivation of the natural gradient updates for structured variational approximations with a minimal conditional exponential family representation, which include highly ﬂexible mixture of exponential family distributions that can ﬁt skewed or multimodal posteriors. Further experiments will be carried out to evaluate the performance of proposed methods.
The Finite-Skip Method for Multiserver Analysis

Multiserver queueing systems are found at the core of a wide variety of practical systems. Unfortunately, existing tools for analyzing multiserver models have major limitations: Techniques for exact analysis often struggle with high-dimensional models, while techniques for deriving bounds are often too specialized to handle realistic system features, such as variable service rates of jobs. New techniques are needed to handle these complex, important, high-dimensional models. In this paper we introduce the work-conserving finite-skip class of models. This class includes many important models, such as the heterogeneous M/G/k, the limited processor sharing policy for the M/G/1, the threshold parallelism model, and the multiserver-job model under a simple scheduling policy. We prove upper and lower bounds on mean response time for any model in the work-conserving finite-skip class. Our bounds are separated by an additive constant, giving a strong characterization of mean response time at all loads. When specialized to each of the models above, these bounds represent the first bounds on mean response time known for each setting.
Dynamic Sequential Graph Learning for Click-Through Rate Prediction

Click-through rate prediction plays an important role in the field of recommender system and many other applications. Existing methods mainly extract user interests from user historical behaviors. However, behavioral sequences only contain users' directly interacted items, which are limited by the system's exposure, thus they are often not rich enough to reflect all the potential interests. In this paper, we propose a novel method, named Dynamic Sequential Graph Learning (DSGL), to enhance users or items' representations by utilizing collaborative information from the local sub-graphs associated with users or items. Specifically, we design the Dynamic Sequential Graph (DSG), i.e., a lightweight ego subgraph with timestamps induced from historical interactions. At every scoring moment, we construct DSGs for the target user and the candidate item respectively. Based on the DSGs, we perform graph convolutional operations iteratively in a bottom-up manner to obtain the final representations of the target user and the candidate item. As for the graph convolution, we design a Time-aware Sequential Encoding Layer that leverages the interaction time information as well as temporal dependencies to learn evolutionary user and item dynamics. Besides, we propose a Target-Preference Dual Attention Layer, composed of a preference-aware attention module and a target-aware attention module, to automatically search for parts of behaviors that are relevant to the target and alleviate the noise from unreliable neighbors. Results on real-world CTR prediction benchmarks demonstrate the improvements brought by DSGL.
X-ray signature of antistars in the Galaxy

The existence of macroscopic objects from antimatter (antistars) is envisaged in some models of baryogenesis. Searches for antistars have been usually carried out in gamma-rays originated from hadronic annihilation of matter. In astrophysically plausible cases of the interaction of neutral atmospheres or winds from antistars with ionized interstellar gas, the formation of excited pp̅ and Hep̅ atoms precedes the hadronic annihilation. These atoms rapidly cascade down to low levels before annihilation giving rise to a series of narrow lines which can be associated with the hadronic annihilation gamma-ray emission. The most significant are L (3p-2p) 1.73 keV line (yield more than 90%) from pp̅ atoms, and M (4–3) 4.86 keV (yield ∼ 60%) and L (3–2) 11.13 keV (yield about 25%) lines from 4Hep̅ atoms. These lines can be searched for in dedicated observations by the forthcoming sensitive X-ray spectroscopic missions XRISM, Athena and Lynx and in wide-field X-ray surveys like SRG/eROSITA all-sky survey.
A Multi-Stencil Fast Marching Method with Path Correction for Efficient Reservoir Simulation and Automated History Matching

The efficiency of reservoir simulation is important for automated history matching (AHM) and production optimization, etc. The fast marching marching method (FMM) has been used for efficient reservoir simulation. FMM can be regarded as a generalised streamline method but without the need to construct streamlines. In FMM reservoir simulation, the Eikonal equation for the diffusive time-of-flight (DTOF) is solved by FMM and then the governing equations are computed on the 1D DTOF coordinate. Standard FMM solves the Eikonal equation using a 4-stencil algorithm on 2D Cartesian grids, ignoring the diagonal neighbouring cells. In the current study, we build a 8-stencil algorithm considering all neighbouring cells, and use local analytical propagation speeds. In addition, a local path-correction coefficient is introduced to further increase the accuracy of DTOF solution. Next, a discretisation scheme is built on the 1D DTOF coordinate for efficient reservoir simulation. The algorithm is validated on homogeneous and heterogeneous test cases, and its potential for efficient forward simulation in AHM is demonstrated by ∗Corresponding author Email addresses: zhaozhang@sdu.edu. cn (Zhao Zhang), guojianchun@vip.163. com (Jianchun Guo) Preprint submitted to Journal September 28, 2021 ar X iv :2 10 9. 12 60 1v 1 [ ph ys ic s. fl udy n] 2 6 Se p 20 21 two examples of dimensions 2 and 6.
Anti-Collision Technologies for Unmanned Aerial Vehicles: Recent Advances and Future Trends

Unmanned aerial vehicles (UAVs) are widely applied in civil applications, such as disaster relief, agriculture and cargo transportation, and so on. With the massive number of UAV flight activities, the anti-collision technologies aiming to avoid the collisions between UAVs and other objects have attracted much attention. The anti-collision technologies are of vital importance to guarantee the survivability and safety of UAVs. In this article, a comprehensive survey on UAV anti-collision technologies is presented. We firstly introduce laws and regulations on UAV safety which prevent a collision at the policy level. Then, the process of anti-collision technologies is reviewed from three aspects, i.e., obstacle sensing, collision prediction, and collision avoidance. We provide a detailed survey and comparison of the methods of each aspect and analyze their pros and cons. Besides, the future trends on UAV anti-collision technologies are presented from the perspective of fast obstacle sensing and fast wireless networking. Finally, we summarize this article.