Miracle-dz/UniFAR-FTUs huggingface dataset:
Extending LLMs' Context Window with 100 Samples

Large Language Models (LLMs) are known to have limited extrapolation ability beyond their pre-trained context window, constraining their application in downstream tasks with lengthy inputs. Recent studies have sought to extend LLMs' context window by modifying rotary position embedding (RoPE), a popular position encoding method adopted by well-known LLMs such as LLaMA, PaLM, and GPT-NeoX. However, prior works like Position Interpolation (PI) and YaRN are resource-intensive and lack comparative experiments to assess their applicability. In this work, we identify the inherent need for LLMs' attention entropy (i.e. the information entropy of attention scores) to maintain stability and introduce a novel extension to RoPE which combines adjusting RoPE's base frequency and scaling the attention logits to help LLMs efficiently adapt to a larger context window. We validate the superiority of our method in both fine-tuning performance and robustness across different context window sizes on various context-demanding tasks. Notably, our method extends the context window of LLaMA-2-7B-Chat to 16,384 with only 100 samples and 6 training steps, showcasing extraordinary efficiency. Finally, we also explore how data compositions and training curricula affect context window extension for specific downstream tasks, suggesting fine-tuning LLMs with lengthy conversations as a good starting point. We release our code and SFT data at https://github.com/GAIR-NLP/Entropy-ABF.
Accurate Loss Prediction of Realistic Hollow-Core Anti-Resonant Fibers Using Machine Learning

Hollow-core anti-resonant fibers (HC-ARFs) have proven to be an indispensable platform for various emerging applications due to their unique and extraordinary optical properties. However, accurately estimating the propagation loss of nested HC-ARFs remains a challenging task due to their complex structure and the lack of precise analytical and theoretical models. To address this challenge, we propose a supervised machine-learning framework that presents an effective solution to accurately predict the propagation loss of a 5-tube nested HC-ARF. Multiple supervised learning models, including random forest, logistic regression, quadratic discriminant analysis, tree-based methods, extreme gradient boosting, and K-nearest neighbors are implemented and compared using a simulated dataset. Among these methods, the random forest algorithm is identified as the most effective, delivering accurate predictions. Notably, this study considers the impact of random structural perturbations on fiber geometry, encompassing random variations in tube wall thicknesses and tube gap separations. In particular, these perturbations involve randomly varying outer and nested tube wall thicknesses, tube angle offsets, and randomly distributed non-circular, anisotropic shapes within the cladding structure. It is worth noting that these specific perturbations have not been previously investigated. Each tube exhibits its unique set of random values, leading to longer simulation times for combinations of these values compared to regular random variables in HC-ARFs with similar tube characteristics. The comprehensive consideration of these factors allows for precise predictions, significantly contributing to the advancement of HC-ARFs for many emerging applications.
Low-Rank Gradient Compression with Error Feedback for MIMO Wireless Federated Learning

This paper presents a novel approach to enhance the communication efficiency of federated learning (FL) in multiple input and multiple output (MIMO) wireless systems. The proposed method centers on a low-rank matrix factorization strategy for local gradient compression based on alternating least squares, along with over-the-air computation and error feedback. The proposed protocol, termed over-the-air low-rank compression (Ota-LC), is demonstrated to have lower computation cost and lower communication overhead as compared to existing benchmarks while guaranteeing the same inference performance. As an example, when targeting a test accuracy of 80% on the Cifar-10 dataset, Ota-LC achieves a reduction in total communication costs of at least 30% when contrasted with benchmark schemes, while also reducing the computational complexity order by a factor equal to the sum of the dimension of the gradients.
Stability analysis of a departure time choice problem with atomic vehicle models

In this study, we analyse the global stability of the equilibrium in a departure time choice problem using a game-theoretic approach that deals with atomic users. We first formulate the departure time choice problem as a strategic game in which atomic users select departure times to minimise their trip cost; we call this game the 'departure time choice game'. The concept of the epsilon-Nash equilibrium is introduced to ensure the existence of pure-strategy equilibrium corresponding to the departure time choice equilibrium in conventional fluid models. Then, we prove that the departure time choice game is a weakly acyclic game. By analysing the convergent better responses, we clarify the mechanisms of global convergence to equilibrium. This means that the epsilon-Nash equilibrium is achieved by sequential better responses of users, which are departure time changes to improve their own utility, in an appropriate order. Specifically, the following behavioural rules are important to ensure global convergence: (i) the adjustment of the departure time of the first user departing from the origin to the corresponding equilibrium departure time and (ii) the fixation of users to their equilibrium departure times in order (starting with the earliest). Using convergence mechanisms, we construct evolutionary dynamics under which global stability is guaranteed. We also investigate the stable and unstable dynamics studied in the literature based on convergence mechanisms, and gain insight into the factors influencing the different stability results. Finally, numerical experiments are conducted to demonstrate the theoretical results.
Effective Dielectric Response of Dense Ferroelectric Nanocomposites

We measured the temperature dependences of the capacity and dielectric losses of the tape-casted dense ferroelectric composites consisting of 28 vol.% BaTiO3 nanoparticles (average size 24 nm) in the polyvinyl butyral polymer and 35 vol.% BaTiO3 microparticles (0.5-0.7 {\mu}m) dispersed in the ethyl-cellulose, as well as the ceramics made from the nanoparticles using the annealing at 1250oC. The composite films were covered with Ag electrodes and the measurements were performed in a wide frequency range. Using the Lichtenecker effective media approximation, we analysed the effective dielectric response of the dense ferroelectric composites. To establish the phase state of the nanoparticles, we analysed the static 137Ba NMR spectra of BaTiO3 nanoparticles in a wide temperature range from 200 to 400 K. Obtained results revealed that the ferroelectric properties of the nanoparticles contribute to the effective dielectric response of the composites in a significantly different way than predicted by the effective media model. This counterintuitive conclusion indicates on the strong crosstalk effects in the dense ferroelectric nanocomposites, which can be very promising for their applications in nanoelectronics and energy storage.
Multifractal-spectral features enhance classification of anomalous diffusion

Anomalous diffusion processes pose a unique challenge in classification and characterization. Previously (Mangalam et al., 2023, Physical Review Research 5, 023144), we established a framework for understanding anomalous diffusion using multifractal formalism. The present study delves into the potential of multifractal spectral features for effectively distinguishing anomalous diffusion trajectories from five widely used models: fractional Brownian motion, scaled Brownian motion, continuous time random walk, annealed transient time motion, and L\'evy walk. To accomplish this, we generate extensive datasets comprising $10^6$ trajectories from these five anomalous diffusion models and extract multiple multifractal spectra from each trajectory. Our investigation entails a thorough analysis of neural network performance, encompassing features derived from varying numbers of spectra. Furthermore, we explore the integration of multifractal spectra into traditional feature datasets, enabling us to assess their impact comprehensively. To ensure a statistically meaningful comparison, we categorize features into concept groups and train neural networks using features from each designated group. Notably, several feature groups demonstrate similar levels of accuracy, with the highest performance observed in groups utilizing moving-window characteristics and $p$-variation features. Multifractal spectral features, particularly those derived from three spectra involving different timescales and cutoffs, closely follow, highlighting their robust discriminatory potential. Remarkably, a neural network exclusively trained on features from a single multifractal spectrum exhibits commendable performance, surpassing other feature groups. Our findings underscore the diverse and potent efficacy of multifractal spectral features in enhancing classification of anomalous diffusion.
On Inter-dataset Code Duplication and Data Leakage in Large Language Models

Motivation. Large language models (LLMs) have exhibited remarkable proficiency in diverse software engineering (SE) tasks. Handling such tasks typically involves acquiring foundational coding knowledge on large, general-purpose datasets during a pre-training phase, and subsequently refining on smaller, task-specific datasets as part of a fine-tuning phase. Problem statement. Data leakage is a well-known issue in training of machine learning models. A manifestation of this issue is the intersection of the training and testing splits. While intra-dataset code duplication examines this intersection within a given dataset and has been addressed in prior research, inter-dataset code duplication, which gauges the overlap between different datasets, remains largely unexplored. If this phenomenon exists, it could compromise the integrity of LLM evaluations because of the inclusion of fine-tuning test samples that were already encountered during pre-training, resulting in inflated performance metrics. Contribution. This paper explores the phenomenon of inter-dataset code duplication and its impact on evaluating LLMs across diverse SE tasks. Study design. We conduct an empirical study using the CSN dataset, a widely adopted pre-training dataset, and five fine-tuning datasets used for various SE tasks. We first identify the intersection between the pre-training and fine-tuning datasets using a deduplication process. Then, we fine-tune four models pre-trained on CSN to evaluate their performance on samples encountered during pre-training and those unseen during that phase. Results. Our findings reveal a potential threat to the evaluation of various LLMs across multiple SE tasks, stemming from the inter-dataset code duplication phenomenon. Moreover, we demonstrate that this threat is accentuated by factors like the LLM's size and the chosen fine-tuning technique.
Inference for high-dimensional linear expectile regression with de-biased method

In this paper, we address the inference problem in high-dimensional linear expectile regression. We transform the expectile loss into a weighted-least-squares form and apply a de-biased strategy to establish Wald-type tests for multiple constraints within a regularized framework. Simultaneously, we construct an estimator for the pseudo-inverse of the generalized Hessian matrix in high dimension with general amenable regularizers including Lasso and SCAD, and demonstrate its consistency through a new proof technique. We conduct simulation studies and real data applications to demonstrate the efficacy of our proposed test statistic in both homoscedastic and heteroscedastic scenarios.
When Does Feature Learning Happen? Perspective from an Analytically Solvable Model

We identify and solve a hidden-layer model that is analytically tractable at any finite width and whose limits exhibit both the kernel phase and the feature learning phase. We analyze the phase diagram of this model in all possible limits of common hyperparameters including width, layer-wise learning rates, scale of output, and scale of initialization. We apply our result to analyze how and when feature learning happens in both infinite and finite-width models. Three prototype mechanisms of feature learning are identified: (1) learning by alignment, (2) learning by disalignment, and (3) learning by rescaling. In sharp contrast, neither of these mechanisms is present when the model is in the kernel regime. This discovery explains why large initialization often leads to worse performance. Lastly, we empirically demonstrate that discoveries we made for this analytical model also appear in nonlinear networks in real tasks.
Energy-level inversion for vortex states in spin-orbit-coupled Bose-Einstein condensates

We investigate vortex states in Bose-Einstein condensates under the combined action of the spin-orbit coupling (SOC), gradient magnetic field, and harmonic-oscillator trapping potential. The linear version of the system is solved exactly. Through the linear-spectrum analysis, we find that, varying the SOC strength and magnetic-field gradient, one can perform energy-level inversion. With suitable parameters, initial higher-order vortex states can be made the ground state (GS). The nonlinear system is solved numerically, revealing that the results are consistent with the linear predictions in the case of repulsive inter-component interactions. On the other hand, inter-component attraction creates the GS in the form of mixed-mode states in a vicinity of the GS phase-transition points. The spin texture of both vortex- and mixed-mode GSs reveals that they feature the structure of 2D (baby) skyrmions.
Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy

Recently, sharpness-aware minimization (SAM) has attracted a lot of attention because of its surprising effectiveness in improving generalization performance.However, training neural networks with SAM can be highly unstable since the loss does not decrease along the direction of the exact gradient at the current point, but instead follows the direction of a surrogate gradient evaluated at another point nearby. To address this issue, we propose a simple renormalization strategy, dubbed StableSAM, so that the norm of the surrogate gradient maintains the same as that of the exact gradient. Our strategy is easy to implement and flexible enough to integrate with SAM and its variants, almost at no computational cost. With elementary tools from convex optimization and learning theory, we also conduct a theoretical analysis of sharpness-aware training, revealing that compared to stochastic gradient descent (SGD), the effectiveness of SAM is only assured in a limited regime of learning rate. In contrast, we show how StableSAM extends this regime of learning rate and when it can consistently perform better than SAM with minor modification. Finally, we demonstrate the improved performance of StableSAM on several representative data sets and tasks.
CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities

Recent large language models (LLMs) have shown indications of mathematical reasoning ability. However it has not been clear how they would fare on more challenging competition-level problems. And while self-generated verbalizations of intermediate reasoning steps (i.e., chain-of-thought prompting) have been shown to be helpful, whether LLMs can make use of helpful side information such as problem-specific hints has not been investigated before. In this paper, we propose a challenging benchmark dataset for enabling such analyses. The Concept and Hint-Annotated Math Problems (CHAMP) consists of high school math competition problems, annotated with concepts, or general math facts, and hints, or problem-specific tricks. These annotations allow us to explore the effects of additional information, such as relevant hints, misleading concepts, or related problems. This benchmark is difficult, with the best model only scoring 58.1% in standard settings. With concepts and hints, performance sometimes improves, indicating that some models can make use of such side information. We further annotate model-generated solutions for their correctness. Using this corpus, we find that models often arrive at the correct final answer through wrong reasoning steps. In addition, we test whether models are able to verify these solutions, and find that most models struggle. The dataset and code are available on the project website.
The MacWilliams Identity for Krawtchouk Association Schemes

The MacWilliams Identity is a well established theorem relating the weight enumerator of a code to the weight enumerator of its dual. The ability to use a known weight enumerator to generate the weight enumerator of another through a simple transform proved highly effective and efficient. An equivalent relation was also developed by Delsarte which linked the eigenvalues of any association scheme to the eigenvalues of it's dual association scheme but this was less practical to use in reality. A functional transform was developed for some specific association schemes including those based on the rank metric, the skew rank metric and Hermitian matrices. In this paper those results are unified into a single consistent theory applied to these"Krawtchouk association schemes"using a $b$-algebra. The derivatives formed using the $b$-algebra have also been applied to derive the moments of the weight distribution for any code within these association schemes.
Fermi-Level Pinning and Barrier Height Control at epitaxially grown ferromagnet/ZnO/metal Schottky Interfaces for opto-spintronics applications

Schottky contacts (SC) at the ferromagnet/ZnO interface are good candidates for the realization and control of several semiconductor emerging magnetic phenomena such spin injection and spin-controlled photonics. In this work, we demonstrate the epitaxial growth of single-phase and wurtzite-ZnO thin films on fcc Pt/Co$_{0.30}$Pt$_{0.70}$ (111) electrodes by MBE technique. While the magnetic properties of the Pt/Co$_{0.30}$Pt$_{0.70}$ buffer remain unchanged after the ZnO growth, the electric measurements of back-to-back Schottky diodes revealed a Schottky barrier height at the metal/ZnO interfaces in the range of 590-690 meV using Cu, Pt and Co$_{0.30}$Pt$_{0.70}$ contacts. A pinning factor S and a charge neutrality level (CNL) ${\Phi}_{CNL}$ of 0.08 and 4.94 eV, respectively, were obtained indicating a strong Fermi-level pining with a CNL level that lies 0.64 eV below the conductance band of ZnO semiconductor. These experimental findings indicate that Co$_{0.30}$Pt$_{0.70}$/ZnO interface follows the metal-induced gap states model and can open a pathway for the realization of opto-spintronics applications such spin-LEDs.
Exploring pNIPAM Lyogels for SMART Reactors: Experimental Study on Swelling Equilibria in Various Organic Solvents and Mixtures, Supported by COSMO-RS Analysis

Stimuli-responsive lyogels are considered to be smart materials due to their capability of undergoing significant macroscopic changes in response to external triggers. Due to their versatile and unique properties, smart lyogels exhibit great potential in various applications such as drug delivery or actuation processes. While Poly-N-isopropylacrylamide (pNIPAM) is widely known as a thermo-responsive material, it also shows significant solvent-responsive swelling behavior. As polar solvents induce strong swelling due to hydrogen bonding with the amide group, nonpolar solvents lead to significant shrinkage of the lyogels. The aim of this study is to investigate and to model this behavior for future application in chemical or biochemical reactors. As a current area of research, incorporating smart hydrogel technology into (bio-)chemical reactors facilitates the development of smart reactor systems. Applying thermodynamic modelling with the gE model COSMO-RS on the monomer or oligomers of pNIPAM, the correlation between solvent-polymer interactions and the degree of swelling can be observed. pNIPAM derivatives exhibit low infinitive dilution activity coefficients (IDACs) in polar solvents with large degrees of swelling, while displaying an increase of IDACs in nonpolar solvents. Hydrogen bonds dominate the swelling behavior of lyogels not only in pure solvents but also in mixtures of solvents with varying polarity. Even in mixtures containing high amounts of nonpolar solvents, large degrees of swelling were observed due to the uptake of the polar solvent in the lyogel matrix. This effect can be observed in binary solvent mixtures but also in representative mixtures along an esterification reaction with varying carboxylic chain length of alcohol and carboxylic acids.
On global solutions of quasilinear second-order elliptic inequalities

We consider the inequality $$ - \operatorname{div} A (x, \nabla u) \ge f (u) \quad \mbox{in } {\mathbb R}^n, $$ where $$ C_1 |\zeta|^p \le \zeta A (x, \zeta) \quad \mbox{and} \quad |A (x, \zeta)| \le C_2 |\zeta|^{p-1} $$ with some constants $C_1>0$, $C_2>0$, and $p>1$ for almost all $x \in {\mathbb R}^n$ and for all $\zeta \in {\mathbb R}^n$, $n \ge 2$. Our aim is to find exact conditions on the function $f$ guaranteeing that any non-negative solution of this inequality is identically zero.
Causative Insights into Open Source Software Security using Large Language Code Embeddings and Semantic Vulnerability Graph

Open Source Software (OSS) security and resilience are worldwide phenomena hampering economic and technological innovation. OSS vulnerabilities can cause unauthorized access, data breaches, network disruptions, and privacy violations, rendering any benefits worthless. While recent deep-learning techniques have shown great promise in identifying and localizing vulnerabilities in source code, it is unclear how effective these research techniques are from a usability perspective due to a lack of proper methodological analysis. Usually, these methods offload a developer's task of classifying and localizing vulnerable code; still, a reasonable study to measure the actual effectiveness of these systems to the end user has yet to be conducted. To address the challenge of proper developer training from the prior methods, we propose a system to link vulnerabilities to their root cause, thereby intuitively educating the developers to code more securely. Furthermore, we provide a comprehensive usability study to test the effectiveness of our system in fixing vulnerabilities and its capability to assist developers in writing more secure code. We demonstrate the effectiveness of our system by showing its efficacy in helping developers fix source code with vulnerabilities. Our study shows a 24% improvement in code repair capabilities compared to previous methods. We also show that, when trained by our system, on average, approximately 9% of the developers naturally tend to write more secure code with fewer vulnerabilities.