Miracle-dz/UniFAR-FTUs huggingface dataset:
Pushing Boundaries: Mixup's Influence on Neural Collapse

Mixup is a data augmentation strategy that employs convex combinations of training instances and their respective labels to augment the robustness and calibration of deep neural networks. Despite its widespread adoption, the nuanced mechanisms that underpin its success are not entirely understood. The observed phenomenon of Neural Collapse, where the last-layer activations and classifier of deep networks converge to a simplex equiangular tight frame (ETF), provides a compelling motivation to explore whether mixup induces alternative geometric configurations and whether those could explain its success. In this study, we delve into the last-layer activations of training data for deep networks subjected to mixup, aiming to uncover insights into its operational efficacy. Our investigation, spanning various architectures and dataset pairs, reveals that mixup's last-layer activations predominantly converge to a distinctive configuration different than one might expect. In this configuration, activations from mixed-up examples of identical classes align with the classifier, while those from different classes delineate channels along the decision boundary. Moreover, activations in earlier layers exhibit patterns, as if trained with manifold mixup. These findings are unexpected, as mixed-up features are not simple convex combinations of feature class means (as one might get, for example, by training mixup with the mean squared error loss). By analyzing this distinctive geometric configuration, we elucidate the mechanisms by which mixup enhances model calibration. To further validate our empirical observations, we conduct a theoretical analysis under the assumption of an unconstrained features model, utilizing the mixup loss. Through this, we characterize and derive the optimal last-layer features under the assumption that the classifier forms a simplex ETF.
The Deep Equilibrium Algorithmic Reasoner

Recent work on neural algorithmic reasoning has demonstrated that graph neural networks (GNNs) could learn to execute classical algorithms. Doing so, however, has always used a recurrent architecture, where each iteration of the GNN aligns with an algorithm's iteration. Since an algorithm's solution is often an equilibrium, we conjecture and empirically validate that one can train a network to solve algorithmic problems by directly finding the equilibrium. Note that this does not require matching each GNN iteration with a step of the algorithm.
Improving 2D-3D Dense Correspondences with Diffusion Models for 6D Object Pose Estimation

Estimating 2D-3D correspondences between RGB images and 3D space is a fundamental problem in 6D object pose estimation. Recent pose estimators use dense correspondence maps and Point-to-Point algorithms to estimate object poses. The accuracy of pose estimation depends heavily on the quality of the dense correspondence maps and their ability to withstand occlusion, clutter, and challenging material properties. Currently, dense correspondence maps are estimated using image-to-image translation models based on GANs, Autoencoders, or direct regression models. However, recent advancements in image-to-image translation have led to diffusion models being the superior choice when evaluated on benchmarking datasets. In this study, we compare image-to-image translation networks based on GANs and diffusion models for the downstream task of 6D object pose estimation. Our results demonstrate that the diffusion-based image-to-image translation model outperforms the GAN, revealing potential for further improvements in 6D object pose estimation models.
Recurrence relations of Exceptional Laurent biorthogonal polynomials

Exceptional extensions of a class of Laurent biorthogonal polynomials (the so-called Hendriksen-van Rossum polynomials) have been presented by the authors recently. This is achieved through Darboux transformations of generalized eigenvalue problems. In this paper, we discuss the recurrence relations satisfied by these exceptional Laurent biorthogonal polynomials and provide a type of recurrence relations with $3l_0+4$ terms explicitly, where the parameter $l_0$ corresponds to the degree of the polynomial part in the seed function used in the Darboux transformation. In the proof of these recurrence relations, the backward operator which maps an exceptional polynomial into a classical one plays a significant role.
Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings

Transformers generalize to novel compositions of structures and entities after being trained on a complex dataset, but easily overfit on datasets of insufficient complexity. We observe that when the training set is sufficiently complex, the model encodes sentences that have a common syntactic structure using a systematic attention pattern. Inspired by this observation, we propose SQ-Transformer (Structurally Quantized) that explicitly encourages systematicity in the embeddings and attention layers, even with a training set of low complexity. At the embedding level, we introduce Structure-oriented Vector Quantization (SoVQ) to cluster word embeddings into several classes of structurally equivalent entities. At the attention level, we devise the Systematic Attention Layer (SAL) and an alternative, Systematically Regularized Layer (SRL) that operate on the quantized word embeddings so that sentences of the same structure are encoded with invariant or similar attention patterns. Empirically, we show that SQ-Transformer achieves stronger compositional generalization than the vanilla Transformer on multiple low-complexity semantic parsing and machine translation datasets. In our analysis, we show that SoVQ indeed learns a syntactically clustered embedding space and SAL/SRL induces generalizable attention patterns, which lead to improved systematicity.
Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study

Compared to Full-Model Fine-Tuning (FMFT), Parameter Efficient Fine-Tuning (PEFT) has demonstrated superior performance and lower computational overhead in several code understanding tasks, such as code summarization and code search. This advantage can be attributed to PEFT's ability to alleviate the catastrophic forgetting issue of Pre-trained Language Models (PLMs) by updating only a small number of parameters. As a result, PEFT effectively harnesses the pre-trained general-purpose knowledge for downstream tasks. However, existing studies primarily involve static code comprehension, aligning with the pre-training paradigm of recent PLMs and facilitating knowledge transfer, but they do not account for dynamic code changes. Thus, it remains unclear whether PEFT outperforms FMFT in task-specific adaptation for code-change-related tasks. To address this question, we examine two prevalent PEFT methods, namely Adapter Tuning (AT) and Low-Rank Adaptation (LoRA), and compare their performance with FMFT on five popular PLMs. Specifically, we evaluate their performance on two widely-studied code-change-related tasks: Just-In-Time Defect Prediction (JIT-DP) and Commit Message Generation (CMG). The results demonstrate that both AT and LoRA achieve state-of-the-art (SOTA) results in JIT-DP and exhibit comparable performances in CMG when compared to FMFT and other SOTA approaches. Furthermore, AT and LoRA exhibit superiority in cross-lingual and low-resource scenarios. We also conduct three probing tasks to explain the efficacy of PEFT techniques on JIT-DP and CMG tasks from both static and dynamic perspectives. The study indicates that PEFT, particularly through the use of AT and LoRA, offers promising advantages in code-change-related tasks, surpassing FMFT in certain aspects.
Fundamental limits of metrology at thermal equilibrium

We consider the estimation of an unknown parameter $\theta$ through a quantum probe at thermal equilibrium. The probe is assumed to be in a Gibbs state according to its Hamiltonian $H_\theta$, which is divided in a parameter-encoding term $H^P_\theta$ and an additional, parameter-independent, control $H^C$. Given a fixed encoding, we find the maximal Quantum Fisher Information attainable via arbitrary $H^C$, which provides a fundamental bound on the measurement precision. Our bounds show that: (i) assuming full control of $H^C$, quantum non-commutativity does not offer any fundamental advantage in the estimation of $\theta$; (ii) an exponential quantum advantage arises at low temperatures if $H^C$ is constrained to have a spectral gap; (iii) in the case of locally-encoded parameters, the optimal sensitivity presents a Heisenberg-like $N^2$-scaling in terms of the number of particles of the probe, which can be reached with local measurements. We apply our results to paradigmatic spin chain models, showing that these fundamental limits can be approached using local two-body interactions. Our results set the fundamental limits and optimal control for metrology with thermal and ground state probes, including probes at the verge of criticality.
A Survey on Transformer Compression

Large models based on the Transformer architecture play increasingly vital roles in artificial intelligence, particularly within the realms of natural language processing (NLP) and computer vision (CV). Model compression methods reduce their memory and computational cost, which is a necessary step to implement the transformer models on practical devices. Given the unique architecture of transformer, featuring alternative attention and Feedforward Neural Network (FFN) modules, specific compression techniques are required. The efficiency of these compression methods is also paramount, as it is usually impractical to retrain large models on the entire training dataset.This survey provides a comprehensive review of recent compression methods, with a specific focus on their application to transformer models. The compression methods are primarily categorized into pruning, quantization, knowledge distillation, and efficient architecture design. In each category, we discuss compression methods for both CV and NLP tasks, highlighting common underlying principles. At last, we delve into the relation between various compression methods, and discuss the further directions in this domain.
How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers

Background. A main theoretical puzzle is why over-parameterized Neural Networks (NNs) generalize well when trained to zero loss (i.e., so they interpolate the data). Usually, the NN is trained with Stochastic Gradient Descent (SGD) or one of its variants. However, recent empirical work examined the generalization of a random NN that interpolates the data: the NN was sampled from a seemingly uniform prior over the parameters, conditioned on that the NN perfectly classifying the training set. Interestingly, such a NN sample typically generalized as well as SGD-trained NNs. Contributions. We prove that such a random NN interpolator typically generalizes well if there exists an underlying narrow ``teacher NN"that agrees with the labels. Specifically, we show that such a `flat' prior over the NN parametrization induces a rich prior over the NN functions, due to the redundancy in the NN structure. In particular, this creates a bias towards simpler functions, which require less relevant parameters to represent -- enabling learning with a sample complexity approximately proportional to the complexity of the teacher (roughly, the number of non-redundant parameters), rather than the student's.
Coherence-based Input Design for Sparse System Identification

The maximum absolute correlation between regressors, which is called mutual coherence, plays an essential role in sparse estimation. A regressor matrix whose columns are highly correlated may result from optimal input design, since there is no constraint on the mutual coherence, so when this regressor is used to estimate sparse parameter vectors of a system, it may yield a large estimation error. This paper aims to tackle this issue for fixed denominator models, which include Laguerre, Kautz, and generalized orthonormal basis function expansion models, for example. The paper proposes an optimal input design method where the achieved Fisher information matrix is fitted to the desired Fisher matrix, together with a coordinate transformation designed to make the regressors in the transformed coordinates have low mutual coherence. The method can be used together with any sparse estimation method and in a numerical study we show its potential for alleviating the problem of model order selection when used in conjunction with, for example, classical methods such as AIC and BIC.
Age of Geminids Derived from the Statistics of Meteoroid Orbits

Statistical analysis of samples of the orbits of celestial bodies is complicated by the fact that the Keplerian orbit is a multidimensional object, the coordinate representation of which nonlinearly depends on the choice of orbital elements. In this work, using the construction of the Fréchet mean, concepts of mean orbit and dispersion of the orbit family are introduced, consistent with the distance function on the orbit set. The introduced statistical characteristics serve as analogs of sample mean and variance of a one-dimensional random variable. Exact formulas for calculating the elements of mean orbits and dispersion quantities with respect to two metrics on the orbit space are derived. For a large sample of meteoroid orbits from the Geminid stream, numerical simulations of orbit evolution over 20 000 years in the past are conducted. By analyzing the dependency of statistical characteristics on time, estimates for the age of the stream and the gas outflow velocity are obtained under the assumption of the birth of the Geminids due to the rapid destruction of the cometary nucleus. The estimate of the age of the stream lies in the interval from 1 200 to 2 000 years, and the speed of gas outflow at perihelion should have been more than 1.2 km/s.
Sub-parsec-scale jet-driven water maser with possible gravitational acceleration in the radio galaxy NGC 1052

We report sub-parsec-scale observations of the 321 GHz H2O emission line in the radio galaxy NGC 1052. The H2O line emitter size is constrained in <0.6 mas distributed on the continuum core component. The brightness temperature exceeding 106 K and the intensity variation indicate certain evidence for maser emission. The maser spectrum consists of redshifted and blueshifted velocity components spanning ∼400 km s−1, separated by a local minimum around the systemic velocity of the galaxy. The spatial distribution of maser components shows a velocity gradient along the jet direction, implying that the population-inverted gas is driven by the jets interacting with the molecular torus. We identified a significant change of the maser spectra between two sessions separated by 14 days. The maser profile showed a radial velocity drift of 127 ± 13 km s−1 yr−1 implying inward gravitational acceleration at 5000 Schwarzschild radii. The results demonstrate the feasibility of future very long baseline interferometry observations to resolve the jet–torus interacting region.
Lipschitz bounds for nonuniformly elliptic integral functionals in the plane

We study local regularity properties of local minimizer of scalar integral functionals with controlled $(p,q)$-growth in the two-dimensional plane. We establish Lipschitz continuity for local minimizer under the condition $1<p\leq q<\infty$ with $q<3p$ which improve upon the classical results valid in the regime $q<2p$. Along the way, we establish an $L^\infty$-$L^2$-estimate for solutions of linear uniformly elliptic equations in the plane which is optimal with respect to the ellipticity contrast of the coefficients.
Collider sensitivity to SMEFT heavy-quark operators at one-loop in top-quark processes

We study the effects of four-heavy-quark operators in the production of top quarks in the framework of the Standard Model Effective Field Theory (SMEFT) at the LHC. In particular, we compute for the first time the total contribution of the four-top-quark operator which enters only at the one-loop level in the top-quark pair production process. Analytical results at one-loop are presented for the gluon- and quark-initiated sub-processes, which allowed a first complete validation of the SMEFT@NLO in Madgraph5_aMC@NLO. The 95% CL bounds on four-heavy-quark operators from the available top-quark pair and four-top-quark production data are provided, which are complementary to other bounds found in the literature. We focus on the comparison of the sensitivities of the top-quark pair and the four-top-quark production processes, where in the latter case the four-top-quark operator contributes at tree-level. We conclude that the sensitivities of the two processes to four-heavy-quark operators are comparable. The projected sensitivities of both processes at HL-LHC are also presented.
A glance at evolvability: a theoretical analysis of its role in the evolutionary dynamics of cell populations

Evolvability is defined as the ability of a population to generate heritable variation to facilitate its adaptation to new environments or selection pressures. In this article, we consider evolvability as a phenotypic trait subject to evolution and discuss its implications in the adaptation of cell populations. We explore the evolutionary dynamics of an actively proliferating population of cells subject to changes in their proliferative potential and their evolvability using a stochastic individual-based model and its deterministic continuum counterpart through numerical simulations of these models. We find robust adaptive trajectories that rely on cells with high evolvability rapidly exploring the phenotypic landscape and reaching the proliferative potential with the highest fitness. The strength of selection on the proliferative potential, and the cost associated with evolvability, can alter these trajectories such that, if both are sufficiently constraining, highly evolvable populations can become extinct in our individual-based model simulations. We explore the impact of this interaction at various scales, discussing its effects in undisturbed environments and also in disrupted contexts, such as cancer.
Large Language Models for Captioning and Retrieving Remote Sensing Images

Image captioning and cross-modal retrieval are examples of tasks that involve the joint analysis of visual and linguistic information. In connection to remote sensing imagery, these tasks can help non-expert users in extracting relevant Earth observation information for a variety of applications. Still, despite some previous efforts, the development and application of vision and language models to the remote sensing domain have been hindered by the relatively small size of the available datasets and models used in previous studies. In this work, we propose RS-CapRet, a Vision and Language method for remote sensing tasks, in particular image captioning and text-image retrieval. We specifically propose to use a highly capable large decoder language model together with image encoders adapted to remote sensing imagery through contrastive language-image pre-training. To bridge together the image encoder and language decoder, we propose training simple linear layers with examples from combining different remote sensing image captioning datasets, keeping the other parameters frozen. RS-CapRet can then generate descriptions for remote sensing images and retrieve images from textual descriptions, achieving SOTA or competitive performance with existing methods. Qualitative results illustrate that RS-CapRet can effectively leverage the pre-trained large language model to describe remote sensing images, retrieve them based on different types of queries, and also show the ability to process interleaved sequences of images and text in a dialogue manner.
Function Aligned Regression: A Method Explicitly Learns Functional Derivatives from Data

Regression is a fundamental task in machine learning that has garnered extensive attention over the past decades. The conventional approach for regression involves employing loss functions that primarily concentrate on aligning model prediction with the ground truth for each individual data sample, which, as we show, can result in sub-optimal prediction of the relationships between the different samples. Recent research endeavors have introduced novel perspectives by incorporating label similarity information to regression. However, a notable gap persists in these approaches when it comes to fully capturing the intricacies of the underlying ground truth function. In this work, we propose FAR (Function Aligned Regression) as a arguably better and more efficient solution to fit the underlying function of ground truth by capturing functional derivatives. We demonstrate the effectiveness of the proposed method practically on 2 synthetic datasets and on 8 extensive real-world tasks from 6 benchmark datasets with other 8 competitive baselines. The code is open-sourced at \url{https://github.com/DixianZhu/FAR}.
CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models

Collaborative search supports multiple users working together to accomplish a specific search task. Research has found that designing lightweight collaborative search plugins within instant messaging platforms aligns better with users' collaborative habits. However, due to the complexity of multi-user interaction scenarios, it is challenging to implement a fully functioning lightweight collaborative search system. Therefore, previous studies on lightweight collaborative search had to rely on the Wizard of Oz paradigm. In recent years, large language models (LLMs) have been demonstrated to interact naturally with users and achieve complex information-seeking tasks through LLM-based agents. Hence, to better support the research in collaborative search, in this demo, we propose CoSearchAgent, a lightweight collaborative search agent powered by LLMs. CoSearchAgent is designed as a Slack plugin that can support collaborative search during multi-party conversations on this platform. Equipped with the capacity to understand the queries and context in multi-user conversations and the ability to search the Web for relevant information via APIs, CoSearchAgent can respond to user queries with answers grounded on the relevant search results. It can also ask clarifying questions when the information needs are unclear. The proposed CoSearchAgent is highly flexible and would be useful for supporting further research on collaborative search. The code and demo video are accessible.
On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference

Despite the recent success associated with Large Language Models (LLMs), they are notably cost-prohibitive to deploy in resource-constrained environments due to their excessive memory and computational demands. In addition to model parameters, the key-value cache is also stored in GPU memory, growing linearly with batch size and sequence length. As a remedy, recent works have proposed various eviction policies for maintaining the overhead of key-value cache under a given budget. This paper embarks on the efficacy of existing eviction policies in terms of importance score calculation and eviction scope construction. We identify the deficiency of prior policies in these two aspects and introduce RoCo, a robust cache omission policy based on temporal attention scores and robustness measures. Extensive experimentation spanning prefilling and auto-regressive decoding stages validates the superiority of RoCo. Finally, we release EasyKV, a versatile software package dedicated to user-friendly key-value constrained generative inference. Code available at https://github.com/DRSY/EasyKV.