Miracle-dz/UniFAR-FTUs huggingface dataset:
Computer simulations of supercooled polymer melts in the bulk and in confined geometry

We survey results of computer simulations for the structure and dynamics of supercooled polymer melts and films. Our survey is mainly concerned with features of a coarse grained polymer model—a bead–spring model—in the temperature regime above the critical glass temperature Tc of the ideal mode-coupling theory (MCT). We divide our discussion into two parts: a part devoted to bulk properties and a part dealing with thin films. The discussion of the bulk properties focuses on two aspects: a comparison of the simulation results with MCT and an analysis of dynamic heterogeneities. We explain in detail how the analyses are performed and what results may be obtained, and we critically assess their strengths and weaknesses. In discussing the application of MCT we also present first results of a quantitative comparison which does not rely on fits, but exploits static input from the simulation to predict the relaxation dynamics. The second part of this review is devoted to extensions of the simulations from the bulk to thin films. We explore in detail the influence of the boundary condition, imposed by smooth or rough walls, on the structure and dynamics of the polymer melt. Geometric confinement is found to shift the glass transition temperature Tg (or Tc in our case) relative to the bulk. We compare our and other simulation results for the Tg shift with experimental data, briefly survey some theoretical ideas for explaining these shifts and discuss related simulation work on the glass transition of confined liquids. Finally, we also present some technical details of how to perform fits to MCT and give a brief introduction to another approach to the glass transition based on the potential energy landscape of a liquid.
Does peer-reviewed theory help predict the cross-section of stock returns?

We compare four groups of cross-sectional return predictors: (1) published with a risk-based explanation, (2) published with a mispricing explanation, (3) published with uncertain origins, and (4) naively data-mined from accounting variables. For all groups, predictability decays by 50% post-sample, showing theory does not help predict returns above naive backtesting. Data-mined predictors display features of published predictors including the rise in returns as in-sample periods end, the speed of post-sample decay, and themes from the literature like investment, issuance, and accruals. Our results imply peer-review systematically mislabels mispricing as risk, though only 18% of predictors are attributed to risk.
RangeAugment: Efficient Online Augmentation with Range Learning

State-of-the-art automatic augmentation methods (e.g., AutoAugment and RandAugment) for visual recognition tasks diversify training data using a large set of augmentation operations. The range of magnitudes of many augmentation operations (e.g., brightness and contrast) is continuous. Therefore, to make search computationally tractable, these methods use fixed and manually-defined magnitude ranges for each operation, which may lead to sub-optimal policies. To answer the open question on the importance of magnitude ranges for each augmentation operation, we introduce RangeAugment that allows us to efficiently learn the range of magnitudes for individual as well as composite augmentation operations. RangeAugment uses an auxiliary loss based on image similarity as a measure to control the range of magnitudes of augmentation operations. As a result, RangeAugment has a single scalar parameter for search, image similarity, which we simply optimize via linear search. RangeAugment integrates seamlessly with any model and learns model- and task-specific augmentation policies. With extensive experiments on the ImageNet dataset across different networks, we show that RangeAugment achieves competitive performance to state-of-the-art automatic augmentation methods with 4-5 times fewer augmentation operations. Experimental results on semantic segmentation, object detection, foundation models, and knowledge distillation further shows RangeAugment's effectiveness.
Domain Generalization with Correlated Style Uncertainty

Domain generalization (DG) approaches intend to extract domain invariant features that can lead to a more robust deep learning model. In this regard, style augmentation is a strong DG method taking advantage of instance-specific feature statistics containing informative style characteristics to synthetic novel domains. While it is one of the state-of-the-art methods, prior works on style augmentation have either disregarded the interdependence amongst distinct feature channels or have solely constrained style augmentation to linear interpolation. To address these research gaps, in this work, we introduce a novel augmentation approach, named Correlated Style Uncertainty (CSU), surpassing the limitations of linear interpolation in style statistic space and simultaneously preserving vital correlation information. Our method's efficacy is established through extensive experimentation on diverse cross-domain computer vision and medical imaging classification tasks: PACS, Office-Home, and Camelyon17 datasets, and the Duke-Market1501 instance retrieval task. The results showcase a remarkable improvement margin over existing state-of-the-art techniques. The source code is available https://github.com/freshman97/CSU.
Fine-Grained Distillation for Long Document Retrieval

Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the \textit{scope hypothesis} that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.
HINT: Hypernetwork Instruction Tuning for Efficient Zero- and Few-Shot Generalisation

Recent NLP models have shown the remarkable ability to effectively generalise ‘zero-shot’ to new tasks using only natural language instructions as guidance. However, many of these approaches suffer from high computational costs due to their reliance on concatenating lengthy instructions with every input example, resulting in costly reprocessing of the instruction. To avoid this, we introduce Hypernetworks for INstruction Tuning (HINT), which convert task instructions and examples into parameter-efficient modules inserted into an underlying model using a pretrained text encoder, eliminating the need to include instructions in the model input. The hypernetwork in HINT also produces an encoded instruction, which we concatenate with encoded inputs during decoding to further improve performance. HINT models outperform strong state-of-the-art baselines by over 10% when controlling for compute (measured in FLOPs). By converting instructions into modules, HINT models can effectively disregard the length of instructions and few-shot example inputs in terms of compute usage. As a result, HINT can enhance its performance by up to 25% by incorporating additional few-shot data, while utilizing only up to 5% more compute. This combines the strengths of parameter-efficient fine-tuning and in-context learning.
Data Augmentation on Graphs: A Survey

In recent years, graph representation learning has achieved remarkable success while suffering from low-quality data problems. As a mature technology to improve data quality in computer vision, data augmentation has also attracted increasing attention in graph domain. For promoting the development of this emerging research direction, in this survey, we comprehensively review and summarize the existing graph data augmentation (GDAug) techniques. Specifically, we first summarize a variety of feasible taxonomies, and then classify existing GDAug studies based on fine-grained graph elements. Furthermore, for each type of GDAug technique, we formalize the general definition, discuss the technical details, and give schematic illustration. In addition, we also summarize common performance metrics and specific design metrics for constructing a GDAug evaluation system. Finally, we summarize the applications of GDAug from both data and model levels, as well as future directions.
Listening for dark photon radio signals from the Galactic Center

Dark photon dark matter that has a kinetic mixing with the Standard Model photon can resonantly convert in environments where its mass $m_{A'}$ coincides with the plasma frequency. We show that such conversion in neutron stars or accreting white dwarfs in the galactic centre can lead to detectable radio signals. Depending on the dark matter spatial distribution, future radio telescopes could be sensitive to values of the kinetic mixing parameter that exceed current constraints by orders of magnitude for $m_{A'} \in \left(6\times 10^{-6},7\times 10^{-4}\right)$ eV.
Training Trajectories of Language Models Across Scales

Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. How do language models of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors? In this paper, we analyze the intermediate training checkpoints of differently sized OPT models (Zhang et al., 2022)—from 125M to 175B parameters—on next-token prediction, sequence-level generation and downstream tasks. We find that 1) at a given perplexity and independent of model sizes, a similar subset of training tokens see the most significant reduction in loss, with the rest stagnating or showing double-descent behavior (Nakkiran et al., 2020); 2) early in training, all models learn to reduce the perplexity of grammatical sequences that contain hallucinations, with small models halting at this suboptimal distribution and larger ones eventually learning to assign these sequences lower probabilities; and 3) perplexity is a strong predictor of in-context learning performance on 74 multiple-choice tasks from BIG-Bench, and this holds independent of the model size. Together, these results show that perplexity is more predictive of model behaviors than model size or training computation.
Redistribution of Weights and Activations for AdderNet Quantization

Adder Neural Network (AdderNet) provides a new way for developing energy-efficient neural networks by replacing the expensive multiplications in convolution with cheaper additions (i.e.l1-norm). To achieve higher hardware efficiency, it is necessary to further study the low-bit quantization of AdderNet. Due to the limitation that the commutative law in multiplication does not hold in l1-norm, the well-established quantization methods on convolutional networks cannot be applied on AdderNets. Thus, the existing AdderNet quantization techniques propose to use only one shared scale to quantize both the weights and activations simultaneously. Admittedly, such an approach can keep the commutative law in the l1-norm quantization process, while the accuracy drop after low-bit quantization cannot be ignored. To this end, we first thoroughly analyze the difference on distributions of weights and activations in AdderNet and then propose a new quantization algorithm by redistributing the weights and the activations. Specifically, the pre-trained full-precision weights in different kernels are clustered into different groups, then the intra-group sharing and inter-group independent scales can be adopted. To further compensate the accuracy drop caused by the distribution difference, we then develop a lossless range clamp scheme for weights and a simple yet effective outliers clamp strategy for activations. Thus, the functionality of full-precision weights and the representation ability of full-precision activations can be fully preserved. The effectiveness of the proposed quantization method for AdderNet is well verified on several benchmarks, e.g., our 4-bit post-training quantized adder ResNet-18 achieves an 66.5% top-1 accuracy on the ImageNet with comparable energy efficiency, which is about 8.5% higher than that of the previous AdderNet quantization methods.
Temperature analysis of X-ray Thomson scattering data

The accurate interpretation of experiments with matter at extreme densities and pressures is a notoriously difficult challenge. In a recent work [T. ~Dornheim et al., Nature Comm. (in print), arXiv:2206.12805], we have introduced a formally exact methodology that allows extracting the temperature of arbitrarily complex materials without any model assumptions or simulations. Here, we provide a more detailed introduction to this approach and analyze the impact of experimental noise on the extracted temperatures. In particular, we extensively apply our method both to synthetic scattering data and to previous experimental measurements over a broad range of temperatures and wave numbers. We expect that our approach will be of high interest to a gamut of applications, including inertial confinement fusion, laboratory astrophysics, and the compilation of highly accurate equation-of-state databases.
Sanctions and Imports of Essential Goods: A Closer Look at the Equipo Anova (2021) Results

We revisit the results of a recent paper by Equipo Anova, who claim to find evidence of an improvement in Venezuelan imports of food and medicines associated with the adoption of U.S. financial sanctions towards Venezuela in 2017. We show that their results are consequence of data coding errors and questionable methodological choices, including the use an unreasonable functional form that implies a counterfactual of negative imports in the absence of sanctions, the omission of data accounting for four-fifths of the country's food imports at the time of sanctions and incorrect application of regression discontinuity methods. Once these errors are corrected, the evidence of a significant improvement in the level and rate of change in imports of essentials disappears.
Next Period Recommendation Reality Check

Over the past decade, tremendous progress has been made in Recommender Systems (RecSys) for well-known tasks such as next-item and next-basket prediction. On the other hand, the recently proposed next-period recommendation (NPR) task is not covered as much. Current works about NPR are mostly based around distinct problem formulations, methods, and proprietary datasets, making solutions difficult to reproduce. In this article, we aim to fill the gap in RecSys methods evaluation on the NPR task using publicly available datasets and (1) introduce the TTRS, a large-scale financial transactions dataset suitable for RecSys methods evaluation; (2) benchmark popular RecSys approaches on several datasets for the NPR task. When performing our analysis, we found a strong repetitive consumption pattern in several real-world datasets. With this setup, our results suggest that the repetitive nature of data is still hard to generalize for the evaluated RecSys methods, and novel item prediction performance is still questionable.
Quantum Bounds on the Generalized Lyapunov Exponents

We discuss the generalized quantum Lyapunov exponents Lq, defined from the growth rate of the powers of the square commutator. They may be related to an appropriately defined thermodynamic limit of the spectrum of the commutator, which plays the role of a large deviation function, obtained from the exponents Lq via a Legendre transform. We show that such exponents obey a generalized bound to chaos due to the fluctuation–dissipation theorem, as already discussed in the literature. The bounds for larger q are actually stronger, placing a limit on the large deviations of chaotic properties. Our findings at infinite temperature are exemplified by a numerical study of the kicked top, a paradigmatic model of quantum chaos.
Learning to Play General-Sum Games against Multiple Boundedly Rational Agents

We study the problem of training a principal in a multi-agent general-sum game using reinforcement learning (RL). Learning a robust principal policy requires anticipating the worst possible strategic responses of other agents, which is generally NP-hard. However, we show that no-regret dynamics can identify these worst-case responses in poly-time in smooth games. We propose a framework that uses this policy evaluation method for efficiently learning a robust principal policy using RL. This framework can be extended to provide robustness to boundedly rational agents too. Our motivating application is automated mechanism design: we empirically demonstrate our framework learns robust mechanisms in both matrix games and complex spatiotemporal games. In particular, we learn a dynamic tax policy that improves the welfare of a simulated trade-and-barter economy by 15%, even when facing previously unseen boundedly rational RL taxpayers.
Seafloor-Invariant Caustics Removal From Underwater Imagery

Mapping the seafloor with underwater imaging cameras is of significant importance for various applications including marine engineering, geology, geomorphology, archaeology, and biology. For shallow waters, among the underwater imaging challenges, caustics, i.e., the complex physical phenomena resulting from the projection of light rays being refracted by the wavy surface, is likely the most crucial one. Caustics is the main factor during underwater imaging campaigns that massively degrades image quality and affects severely any 2-D mosaicking or 3-D reconstruction of the seabed. In this article, we propose a novel method for correcting the radiometric effects of caustics on shallow underwater imagery. Contrary to the state-of-the-art, the developed method can handle the seabed and riverbed of any anaglyph, correcting the images using real pixel information, thus, improving image matching and 3-D reconstruction processes. In particular, the developed method employs deep learning architectures to classify image pixels to “noncaustics” and “caustics. ” Then, it exploits the 3-D geometry of the scene to achieve a pixelwise correction, by transferring appropriate color values between the overlapping underwater images. Moreover, to fill the current gap, we have collected, annotated, and structured a real-world caustic data set, namely, R-CAUSTIC, which is openly available. Overall, based on the experimental results and validation, the developed methodology is quite promising in both detecting caustics and reconstructing their intensity.
Pretraining Without Attention

Transformers have been essential to pretraining success in NLP. While other architectures have been used, downstream accuracy is either significantly worse, or requires attention layers to match standard benchmarks such as GLUE. This work explores pretraining without attention by using recent advances in sequence routing based on state-space models (SSMs). Our proposed model, Bidirectional Gated SSM (BiGS), combines SSM layers with a multiplicative gating architecture that has been effective in simplified sequence modeling architectures. The model learns static layers that do not consider pair-wise interactions. Even so, BiGS is able to match BERT pretraining accuracy on GLUE and can be extended to long-form pretraining of 4096 tokens without approximation. Analysis shows that while the models have similar average accuracy, the approach has different inductive biases than BERT in terms of interactions and syntactic representations. All models from this work are available at https://github.com/jxiw/BiGS.
Conditioned Generative Transformers for Histopathology Image Synthetic Augmentation

Deep learning networks have demonstrated state-of-the-art performance on medical image analysis tasks. However, the majority of the works rely heavily on abundantly labeled data, which necessi-tates extensive involvement of domain experts. Vision transformer (ViT) based generative adversarial networks (GANs) recently demonstrated superior potential in general image synthesis, yet are less explored for histopathology images. In this paper, we address these challenges by proposing a pure ViT-based conditional GAN model for histopathology image synthetic augmentation. To alleviate training instability and improve generation robustness, we ﬁrst introduce a conditioned class projection method to facilitate class separation. We then implement a multi-loss weighing function to dynamically balance the losses between classiﬁcation tasks. We further propose a selective augmentation mechanism to actively choose the appropriate generated images and bring additional performance improvements. Extensive experiments on the histopathology datasets show that leveraging our synthetic augmentation framework results in signiﬁcant and consistent improvements in classiﬁcation performance.
Asynchronous Distributed Bilevel Optimization

Bilevel optimization plays an essential role in many machine learning tasks, ranging from hyperparameter optimization to meta-learning. Existing studies on bilevel optimization, however, focus on either centralized or synchronous distributed setting. The centralized bilevel optimization approaches require collecting massive amount of data to a single server, which inevitably incur significant communication expenses and may give rise to data privacy risks. Synchronous distributed bilevel optimization algorithms, on the other hand, often face the straggler problem and will immediately stop working if a few workers fail to respond. As a remedy, we propose Asynchronous Distributed Bilevel Optimization (ADBO) algorithm. The proposed ADBO can tackle bilevel optimization problems with both nonconvex upper-level and lower-level objective functions, and its convergence is theoretically guaranteed. Furthermore, it is revealed through theoretic analysis that the iteration complexity of ADBO to obtain the $\epsilon$-stationary point is upper bounded by $\mathcal{O}(\frac{1}{{{\epsilon ^2}}})$. Thorough empirical studies on public datasets have been conducted to elucidate the effectiveness and efficiency of the proposed ADBO.
In and Out-of-Domain Text Adversarial Robustness via Label Smoothing

Recently it has been shown that state-of-the-art NLP models are vulnerable to adversarial attacks, where the predictions of a model can be drastically altered by slight modifications to the input (such as synonym substitutions). While several defense techniques have been proposed, and adapted, to the discrete nature of text adversarial attacks, the benefits of general-purpose regularization methods such as label smoothing for language models, have not been studied. In this paper, we study the adversarial robustness provided by label smoothing strategies in foundational models for diverse NLP tasks in both in-domain and out-of-domain settings. Our experiments show that label smoothing significantly improves adversarial robustness in pre-trained models like BERT, against various popular attacks. We also analyze the relationship between prediction confidence and robustness, showing that label smoothing reduces over-confident errors on adversarial examples.