Miracle-dz/UniFAR-FTUs huggingface dataset:
Efficient Knowledge Distillation from Model Checkpoints

Knowledge distillation is an effective approach to learn compact models (students) with the supervision of large and strong models (teachers). As empirically there exists a strong correlation between the performance of teacher and student models, it is commonly believed that a high performing teacher is preferred. Consequently, practitioners tend to use a well trained network or an ensemble of them as the teacher. In this paper, we make an intriguing observation that an intermediate model, i.e., a checkpoint in the middle of the training procedure, often serves as a better teacher compared to the fully converged model, although the former has much lower accuracy. More surprisingly, a weak snapshot ensemble of several intermediate models from a same training trajectory can outperform a strong ensemble of independently trained and fully converged models, when they are used as teachers. We show that this phenomenon can be partially explained by the information bottleneck principle: the feature representations of intermediate models can have higher mutual information regarding the input, and thus contain more"dark knowledge"for effective distillation. We further propose an optimal intermediate teacher selection algorithm based on maximizing the total task-related mutual information. Experiments verify its effectiveness and applicability.
Multi-messenger constraints on Abelian-Higgs cosmic string networks

Nielsen-Olesen vortices in the Abelian-Higgs (AH) model are the simplest realisations of cosmic strings in a gauge field theory. Large-scale numerical solutions show that the dominant decay channel of a network of AH strings produced from random initial conditions is classical field radiation. However, they also show that with special initial conditions, loops of string can be created for which classical field radiation is suppressed, and which behave like Nambu-Goto (NG) strings with a dominant decay channel into gravitational radiation. This indicates that cosmic strings are generically sources of both high-energy particles and gravitational waves. Here we adopt a simple parametrisation of the AH string network allowing for both particle and gravitational wave production. With a reference to a specific model for NG-like loop distribution, this sets the basis for a “multi-messenger” investigation of this model. We find that, in order to explain the NANOGrav detection of a possible gravitational wave background, while satisfying the constraint on NG-like loop production from simulations and bounds from the cosmic microwave background, the tension of the AH string in Planck units Gμ and the fraction of the NG-like loops f NG should satisfy Gμ f NG 2.6 ≳ 3.2 × 10-13 at 95% confidence. On the other hand, for such string tensions, constraints from the diffuse gamma-ray background (DGRB) indicate that more than 97% of the total network energy should be converted to dark matter (DM) or dark radiation. We also consider joint constraints on the annihilation cross-section, the mass, and the relic abundance of DM produced by decays of strings. For example, for a DM mass of 500 GeV, the observed relic abundance can be explained by decaying AH strings that also account for the NANOGrav signal.
Aergia: leveraging heterogeneity in federated learning systems

Federated Learning (FL) is a popular deep learning approach that prevents centralizing large amounts of data, and instead relies on clients that update a global model using their local datasets. Classical FL algorithms use a central federator that, for each training round, waits for all clients to send their model updates before aggregating them. In practical deployments, clients might have different computing powers and network capabilities, which might lead slow clients to become performance bottlenecks. Previous works have suggested to use a deadline for each learning round so that the federator ignores the late updates of slow clients, or so that clients send partially trained models before the deadline. To speed up the training process, we instead propose Aergia, a novel approach where slow clients (i) freeze the part of their model that is the most computationally intensive to train; (ii) train the unfrozen part of their model; and (iii) offload the training of the frozen part of their model to a faster client that trains it using its own dataset. The offloading decisions are orchestrated by the federator based on the training speed that clients report and on the similarities between their datasets, which are privately evaluated thanks to a trusted execution environment. We show through extensive experiments that Aergia maintains high accuracy and significantly reduces the training time under heterogeneous settings by up to 27% and 53% compared to FedAvg and TiFL, respectively.
SegViT: Semantic Segmentation with Plain Vision Transformers

We explore the capability of plain Vision Transformers (ViTs) for semantic segmentation and propose the SegVit. Previous ViT-based segmentation networks usually learn a pixel-level representation from the output of the ViT. Differently, we make use of the fundamental component -- attention mechanism, to generate masks for semantic segmentation. Specifically, we propose the Attention-to-Mask (ATM) module, in which the similarity maps between a set of learnable class tokens and the spatial feature maps are transferred to the segmentation masks. Experiments show that our proposed SegVit using the ATM module outperforms its counterparts using the plain ViT backbone on the ADE20K dataset and achieves new state-of-the-art performance on COCO-Stuff-10K and PASCAL-Context datasets. Furthermore, to reduce the computational cost of the ViT backbone, we propose query-based down-sampling (QD) and query-based up-sampling (QU) to build a Shrunk structure. With the proposed Shrunk structure, the model can save up to $40\%$ computations while maintaining competitive performance.
Differentially Private Bootstrap: New Privacy Analysis and Inference Strategies

Differentially private (DP) mechanisms protect individual-level information by introducing randomness into the statistical analysis procedure. Despite the availability of numerous DP tools, there remains a lack of general techniques for conducting statistical inference under DP. We examine a DP bootstrap procedure that releases multiple private bootstrap estimates to infer the sampling distribution and construct confidence intervals (CIs). Our privacy analysis presents new results on the privacy cost of a single DP bootstrap estimate, applicable to any DP mechanisms, and identifies some misapplications of the bootstrap in the existing literature. Using the Gaussian-DP (GDP) framework (Dong et al.,2022), we show that the release of $B$ DP bootstrap estimates from mechanisms satisfying $(\mu/\sqrt{(2-2/\mathrm{e})B})$-GDP asymptotically satisfies $\mu$-GDP as $B$ goes to infinity. Moreover, we use deconvolution with the DP bootstrap estimates to accurately infer the sampling distribution, which is novel in DP. We derive CIs from our density estimate for tasks such as population mean estimation, logistic regression, and quantile regression, and we compare them to existing methods using simulations and real-world experiments on 2016 Canada Census data. Our private CIs achieve the nominal coverage level and offer the first approach to private inference for quantile regression.
Deep Active Ensemble Sampling For Image Classification

Conventional active learning (AL) frameworks aim to reduce the cost of data annotation by actively requesting the labeling for the most informative data points. However, introducing AL to data hungry deep learning algorithms has been a challenge. Some proposed approaches include uncertainty-based techniques, geometric methods, implicit combination of uncertainty-based and geometric approaches, and more recently, frameworks based on semi/self supervised techniques. In this paper, we address two specific problems in this area. The first is the need for efficient exploitation/exploration trade-off in sample selection in AL. For this, we present an innovative integration of recent progress in both uncertainty-based and geometric frameworks to enable an efficient exploration/exploitation trade-off in sample selection strategy. To this end, we build on a computationally efficient approximate of Thompson sampling with key changes as a posterior estimator for uncertainty representation. Our framework provides two advantages: (1) accurate posterior estimation, and (2) tune-able trade-off between computational overhead and higher accuracy. The second problem is the need for improved training protocols in deep AL. For this, we use ideas from semi/self supervised learning to propose a general approach that is independent of the specific AL technique being used. Taken these together, our framework shows a significant improvement over the state-of-the-art, with results that are comparable to the performance of supervised-learning under the same setting. We show empirical results of our framework, and comparative performance with the state-of-the-art on four datasets, namely, MNIST, CIFAR10, CIFAR100 and ImageNet to establish a new baseline in two different settings.
Back to the Future: On Potential Histories in NLP

Machine learning and NLP require the construction of datasets to train and ﬁne-tune models. In this context, previous work has demon-strated the sensitivity of these data sets. For instance, potential societal biases in this data are likely to be encoded and to be ampliﬁed in the models we deploy. In this work, we draw from developments in the ﬁeld of history and take a novel perspective on these prob-lems: considering datasets and models through the lens of historical ﬁction surfaces their po-litical nature, and affords re-conﬁguring how we view the past, such that marginalized discourses are surfaced. Building on such in-sights, we argue that contemporary methods for machine learning are prejudiced towards dominant and hegemonic histories. Employ-ing the example of neopronouns, we show that by surfacing marginalized histories within contemporary conditions, we can create models that better represent the lived realities of tra-ditionally marginalized and excluded commu-nities.
Are Sample-Efficient NLP Models More Robust?

Recent results in image classification and extractive question answering have observed that pre-trained models trained on less in-distribution data have better out-ofdistribution performance. However, it is unclear how broadly these trends hold. We conduct a large empirical study across three tasks, three broadly-applicable modeling interventions (increasing model size, using a different adaptation method, and pre-training on more data), and 14 diverse datasets to investigate the relationship between sample efficiency (amount of data needed to reach a given ID accuracy) and robustness (how models fare on OOD evaluation). We find that higher sample efficiency is only correlated with better average OOD robustness on some modeling interventions and tasks, but not others. On individual datasets, models with lower sample efficiency can even be more robust. These results suggest that general-purpose methods for improving sample efficiency are unlikely to yield universal OOD robustness improvements, since such improvements are highly dataset- and task-dependent. Even in an era of large, multi-purpose pre-trained models, task-specific decisions may often be necessary for OOD generalization.
Model-based clustering in simple hypergraphs through a stochastic blockmodel

We propose a new model to address the overlooked problem of node clustering in simple hypergraphs. Simple hypergraphs are suitable when a node may not appear multiple times in the same hyperedge, such as in co-authorship datasets. Our model assumes the existence of latent node groups and hyperedges are conditionally independent given these groups. We first establish the generic identifiability of the model parameters. We then develop a variational approximation Expectation-Maximization algorithm for parameter inference and node clustering, and derive a statistical criterion for model selection. To illustrate the performance of our R package HyperSBM, we compare it with other node clustering methods using synthetic data generated from the model, as well as from a line clustering experiment and a co-authorship dataset. As a by-product, our synthetic experiments demonstrate that the detectability thresholds for non-uniform sparse hypergraphs cannot be deduced from the uniform case.
Optimal $L^2$ extension for holomorphic vector bundles with singular hermitian metrics

In the present paper, we study the properties of singular Nakano positivity of singular hermitian metrics on holomorphic vector bundles, and establish an optimal $L^2$ extension theorem for holomorphic vector bundles with singular hermitian metrics on weakly pseudoconvex K\"{a}hler manifolds. As applications, we give a necessary condition for the holding of the equality in optimal $L^2$ extension theorem, and present singular hermitian holomorphic vector bundle versions of some $L^2$ extension theorems with optimal estimate.
The evolution of HCO+ in molecular clouds using a novel chemical post-processing algorithm

Modelling the chemistry of molecular clouds is critical to accurately simulating their evolution. To reduce computational cost, 3D simulations generally restrict their chemistry to species with strong heating and cooling effects. Time-dependent information about the evolution of other species is therefore often neglected. We address this gap by post-processing tracer particles in the SILCC-Zoom molecular cloud simulations. Using a chemical network of 39 species and 301 reactions (including freeze-out of CO and H2O) and a novel algorithm to reconstruct a density grid from sparse tracer particle data, we produce time-dependent density distributions for various species. We focus upon the evolution of HCO+, which is a critical formation reactant of CO but is not typically modelled on the fly. We find that ∼ 90 per cent of the HCO+ content of the cold molecular gas forms in situ around nHCO+ ∼ 103–104 cm−3, over a time-scale of approximately 1 Myr. The remaining ∼ 10 per cent forms at high extinction sites, with minimal turbulent mixing out into the less dense gas. We further show that the dominant HCO+ formation pathway is dependent on the visual extinction, with the reaction H3+ + CO contributing 90 per cent of the total HCO+ production above AV, 3D = 3. We produce the very first maps of the HCO+ column density, N(HCO+), and show that it reaches values as high as 1015 cm−2. We find that 50 per cent of the HCO+ mass is located within AV ∼ 10–30 in a density range of 103.5–104.5 cm−3. Our maps of N(HCO+) are shown to be in good agreement with recent observations of the W49A star-forming region.
Revisiting Neutron Propagation-Based Phase-Contrast Imaging and Tomography: Use of Phase Retrieval to Amplify the Effective Degree of Brilliance

Propagation-based neutron phase-contrast tomography was demonstrated using the ISIS pulsed spallation source. The proof-of-concept tomogram with Paganin-type phase-retrieval filter applied exhibited an effective net boost of $23\pm 1$ in the signal-to-noise ratio as compared to an attenuation-based tomogram, implying a boost in the effective degree of neutron brilliance of over two orders of magnitude. This comparison is for phase retrieval versus conventional absorption with no additional collimation in place. Expressions are provided for the optimal phase-contrast geometry as well as conditions for the validity of the method. The underpinning theory is derived under the assumption of the sample being composed of a single material. The effective boost in brilliance may be employed to give reduced acquisition time, or may instead be used to keep exposure times fixed while improving the measured contrast.
Interpretable AI for relating brain structural and functional connectomes

One of the central problems in neuroscience is understanding how brain structure relates to function. Naively one can relate the direct connections of white matter fiber tracts between brain regions of interest (ROIs) to the increased co-activation in the same pair of ROIs, but the link between structural and functional connectomes (SCs and FCs) has proven to be much more complex. To learn a realistic generative model characterizing population variation in SCs, FCs, and the SC-FC coupling, we develop a graph auto-encoder that we refer to as Staf-GATE. We trained Staf-GATE with data from the Human Connectome Project (HCP) and show state-of-the-art performance in predicting FC and joint generation of SC and FC. In addition, as a crucial component of the proposed approach, we provide a masking-based algorithm to extract interpretable inferences about SC-FC coupling. Our interpretation methods identified important SC subnetworks for FC coupling and relating SC and FC with sex.
A Momentum Accelerated Adaptive Cubic Regularization Method for Nonconvex Optimization

The cubic regularization method (CR) and its adaptive version (ARC) are popular Newton-type methods in solving unconstrained non-convex optimization problems, due to its global convergence to local minima under mild conditions. The main aim of this paper is to develop a momentum-accelerated adaptive cubic regularization method (ARCm) to improve the convergent performance. With the proper choice of momentum step size, we show the global convergence of ARCm and the local convergence can also be guaranteed under the \KL property. Such global and local convergence can also be established when inexact solvers with low computational costs are employed in the iteration procedure. Numerical results for non-convex logistic regression and robust linear regression models are reported to demonstrate that the proposed ARCm significantly outperforms state-of-the-art cubic regularization methods (e.g., CR, momentum-based CR, ARC) and the trust region method. In particular, the number of iterations required by ARCm is less than 10\% to 50\% required by the most competitive method (ARC) in the experiments.
SaiT: Sparse Vision Transformers through Adaptive Token Pruning

While vision transformers have achieved impressive results, effectively and efficiently accelerating these models can further boost performances. In this work, we propose a dense/sparse training framework to obtain a unified model, enabling weight sharing across various token densities. Thus one model offers a range of accuracy and throughput tradeoffs for different applications. Besides, we introduce adaptive token pruning to optimize the patch token sparsity based on the input image. In addition, we investigate knowledge distillation to enhance token selection capability in early transformer modules. Sparse adaptive image Transformer (SaiT) offers varying levels of model acceleration by merely changing the token sparsity on the fly. Specifically, SaiT reduces the computation complexity (FLOPs) by 39% - 43% and increases the throughput by 67% - 91% with less than 0.5% accuracy loss for various vision transformer models. Meanwhile, the same model also provides the zero accuracy drop option by skipping the sparsification step. SaiT achieves better accuracy and computation tradeoffs than state-of-the-art transformer and convolutional models.
Do Question Answering Modeling Improvements Hold Across Benchmarks?

Do question answering (QA) modeling improvements (e.g., choice of architecture and training procedure) hold consistently across the diverse landscape of QA benchmarks? To study this question, we introduce the notion of concurrence—two benchmarks have high concurrence on a set of modeling approaches if they rank the modeling approaches similarly. We measure the concurrence between 32 QA benchmarks on a set of 20 diverse modeling approaches and find that human-constructed benchmarks have high concurrence amongst themselves, even if their passage and question distributions are very different. Surprisingly, even downsampled human-constructed benchmarks (i.e., collecting less data) and programmatically-generated benchmarks (e.g., cloze-formatted examples) have high concurrence with human-constructed benchmarks. These results indicate that, despite years of intense community focus on a small number of benchmarks, the modeling improvements studied hold broadly.
Near-Optimal Multi-Agent Learning for Safe Coverage Control

In multi-agent coverage control problems, agents navigate their environment to reach locations that maximize the coverage of some density. In practice, the density is rarely known $\textit{a priori}$, further complicating the original NP-hard problem. Moreover, in many applications, agents cannot visit arbitrary locations due to $\textit{a priori}$ unknown safety constraints. In this paper, we aim to efficiently learn the density to approximately solve the coverage problem while preserving the agents' safety. We first propose a conditionally linear submodular coverage function that facilitates theoretical analysis. Utilizing this structure, we develop MacOpt, a novel algorithm that efficiently trades off the exploration-exploitation dilemma due to partial observability, and show that it achieves sublinear regret. Next, we extend results on single-agent safe exploration to our multi-agent setting and propose SafeMac for safe coverage and exploration. We analyze SafeMac and give first of its kind results: near optimal coverage in finite time while provably guaranteeing safety. We extensively evaluate our algorithms on synthetic and real problems, including a bio-diversity monitoring task under safety constraints, where SafeMac outperforms competing methods.
SMOOTHING ROTATION CURVES AND MASS PROFILES

We show that spiral activity can erase pronounced features in disk galaxy rotation curves. We present simulations of growing disks, in which the added material has a physically motivated distribution, as well as other examples of physically less realistic accretion. In all cases, attempts to create unrealistic rotation curves were unsuccessful because spiral activity rapidly smoothed away features in the disk mass profile. The added material was redistributed radially by the spiral activity, which was itself provoked by the density feature. In the case of a ridge-like feature in the surface density profile, we show that two unstable spiral modes develop, and the associated angular momentum changes in horseshoe orbits remove particles from the ridge and spread them both inward and outward. This process rapidly erases the density feature from the disk. We also find that the lack of a feature when transitioning from disk to halo dominance in the rotation curves of disk galaxies, the so called “disk–halo conspiracy,” could also be accounted for by this mechanism. We do not create perfectly exponential mass profiles in the disk, but suggest that this mechanism contributes to their creation.