Miracle-dz/UniFAR-FTUs huggingface dataset:
Implicit Bias and Fast Convergence Rates for Self-attention

Self-attention, the core mechanism of transformers, distinguishes them from traditional neural networks and drives their outstanding performance. Towards developing the fundamental optimization principles of self-attention, we investigate the implicit bias of gradient descent (GD) in training a self-attention layer with fixed linear decoder in binary classification. Drawing inspiration from the study of GD in linear logistic regression over separable data, recent work demonstrates that as the number of iterations $t$ approaches infinity, the key-query matrix $W_t$ converges locally (with respect to the initialization direction) to a hard-margin SVM solution $W_{mm}$. Our work enhances this result in four aspects. Firstly, we identify non-trivial data settings for which convergence is provably global, thus shedding light on the optimization landscape. Secondly, we provide the first finite-time convergence rate for $W_t$ to $W_{mm}$, along with quantifying the rate of sparsification in the attention map. Thirdly, through an analysis of normalized GD and Polyak step-size, we demonstrate analytically that adaptive step-size rules can accelerate the convergence of self-attention. Additionally, we remove the restriction of prior work on a fixed linear decoder. Our results reinforce the implicit-bias perspective of self-attention and strengthen its connections to implicit-bias in linear logistic regression, despite the intricate non-convex nature of the former.
An Optimal Control Formulation of Tool Affordance Applied to Impact Tasks

Humans use tools to complete impact-aware tasks, such as hammering a nail or playing tennis. The postures adopted to use these tools can significantly influence the performance of these tasks, where the force or velocity of the hand holding a tool plays a crucial role. The underlying motion planning challenge consists of grabbing the tool in preparation for the use of this tool with an optimal body posture. Directional manipulability describes the dexterity of force and velocity in a joint configuration along a specific direction. In order to take directional manipulability and tool affordances into account, we apply an optimal control method combining iterative linear quadratic regulator with the alternating direction method of multipliers. Our approach considers the notion of tool affordances to solve motion planning problems, by introducing a cost based on directional velocity manipulability. The proposed approach is applied to impact tasks in simulation and on a real 7-axis robot, specifically in a nail-hammering task with the assistance of a pilot hole. Our comparison study demonstrates the importance of maximizing directional manipulability in impact-aware tasks.
Efficient Stagewise Pretraining via Progressive Subnetworks

Recent developments in large language models have sparked interest in efficient pretraining methods. A recent effective paradigm is to perform stage-wise training, where the size of the model is gradually increased over the course of training (e.g. gradual stacking (Reddi et al., 2023)). While the resource and wall-time savings are appealing, it has limitations, particularly the inability to evaluate the full model during earlier stages, and degradation in model quality due to smaller model capacity in the initial stages. In this work, we propose an alternative framework, progressive subnetwork training, that maintains the full model throughout training, but only trains subnetworks within the model in each step. We focus on a simple instantiation of this framework, Random Path Training (RaPTr) that only trains a sub-path of layers in each step, progressively increasing the path lengths in stages. RaPTr achieves better pre-training loss for BERT and UL2 language models while requiring 20-33% fewer FLOPs compared to standard training, and is competitive or better than other efficient training methods. Furthermore, RaPTr shows better downstream performance on UL2, improving QA tasks and SuperGLUE by 1-5% compared to standard training and stacking. Finally, we provide a theoretical basis for RaPTr to justify (a) the increasing complexity of subnetworks in stages, and (b) the stability in loss across stage transitions due to residual connections and layer norm.
I-FENN with Temporal Convolutional Networks: expediting the load-history analysis of non-local gradient damage propagation

In this paper, we demonstrate for the first time how the Integrated Finite Element Neural Network (I-FENN) framework, previously proposed by the authors, can efficiently simulate the entire loading history of non-local gradient damage propagation. To achieve this goal, we first adopt a Temporal Convolutional Network (TCN) as the neural network of choice to capture the history-dependent evolution of the non-local strain in a coarsely meshed domain. The quality of the network predictions governs the computational performance of I-FENN, and therefore we perform an extended investigation aimed at enhancing them. We explore a data-driven vs. physics-informed TCN setup to arrive at an optimum network training, evaluating the network based on a coherent set of relevant performance metrics. We address the crucial issue of training a physics-informed network with input data that span vastly different length scales by proposing a systematic way of input normalization and output un-normalization. We then integrate the trained TCN within the nonlinear iterative FEM solver and apply I-FENN to simulate the damage propagation analysis. I-FENN is always applied in mesh idealizations different from the one used for the TCN training, showcasing the framework's ability to be used at progressively refined mesh resolutions. We illustrate several cases that I-FENN completes the simulation using either a modified or a full Newton-Raphson scheme, and we showcase its computational savings compared to both the classical monolithic and staggered FEM solvers. We underline that we satisfy very strict convergence criteria for every increment across the entire simulation, providing clear evidence of the robustness and accuracy of I-FENN. All the code and data used in this work will be made publicly available upon publication of the article.
The promising path of evolutionary optimization to avoid barren plateaus

Variational quantum algorithms are viewed as promising candidates for demonstrating quantum advantage on near-term devices. These approaches typically involve the training of parameterized quantum circuits through a classical optimization loop. However, they often encounter challenges attributed to the exponentially diminishing gradient components, known as the barren plateau (BP) problem. This work introduces a novel optimization method designed to alleviate the adverse effects of BPs during circuit training. Our approach to select the optimization search direction relies on the distant features of the cost-function landscape. This enables the optimization path to navigate around barren plateaus without the need for external control mechanisms. We have successfully applied our optimization strategy to quantum circuits comprising $16$ qubits and $15000$ entangling gates, demonstrating robust resistance against BPs. Additionally, we have extended our optimization strategy by incorporating an evolutionary selection framework, enhancing its ability to avoid local minima in the landscape. The modified algorithm has been successfully utilized in quantum gate synthesis applications, showcasing a significantly improved efficiency in generating highly compressed quantum circuits compared to traditional gradient-based optimization approaches.
VerAs: Verify then Assess STEM Lab Reports

With an increasing focus in STEM education on critical thinking skills, science writing plays an ever more important role in curricula that stress inquiry skills. A recently published dataset of two sets of college level lab reports from an inquiry-based physics curriculum relies on analytic assessment rubrics that utilize multiple dimensions, specifying subject matter knowledge and general components of good explanations. Each analytic dimension is assessed on a 6-point scale, to provide detailed feedback to students that can help them improve their science writing skills. Manual assessment can be slow, and difficult to calibrate for consistency across all students in large classes. While much work exists on automated assessment of open-ended questions in STEM subjects, there has been far less work on long-form writing such as lab reports. We present an end-to-end neural architecture that has separate verifier and assessment modules, inspired by approaches to Open Domain Question Answering (OpenQA). VerAs first verifies whether a report contains any content relevant to a given rubric dimension, and if so, assesses the relevant sentences. On the lab reports, VerAs outperforms multiple baselines based on OpenQA systems or Automated Essay Scoring (AES). VerAs also performs well on an analytic rubric for middle school physics essays.
Safe Human-UAS Collaboration Abstraction

This paper studies the problem of safe humanuncrewed aerial system (UAS) collaboration in a shared work environment. By considering human and UAS as co-workers, we use Petri Nets to abstractly model evolution of shared tasks assigned to human and UAS co-workers. Particularly, the Petri Nets places represent work stations; therefore, the Petri Nets transitions can formally specify displacements between the work stations. The first objective is to incorporate uncertainty regarding the intentions of human co-workers into motion planning for UAS, when UAS co-workers closely interact with human co-workers. To this end, the proposed Petri Nets model uses conflict constructs to represent situations at which UAS deals with incomplete knowledge about human co-worker intention. The second objective is then to plan the motion of the UAS in a resilient and safe manner, in the presence of non-cooperative human co-workers. In order to achieve this objective, UAS equipped with onboard perception and decision-making capabilities are able to, through real-time processing of in-situ observation, predict human intention, quantify human distraction, and apply a non-stationary Markov Decision model to safely plan UAS motion in the presence of uncertainty.
From atomic to global connectivity in the structure of the SARS-CoV2-Human ACE2 receptor complex

We investigate connectivity properties of the SARS-CoV2 spike protein-human ACE2-receptor complex employing a protein side chain-based network method that allows us to span a range from atomic to global protein scales. We analyze network topology in terms of clusters and cliques obtained from averaging over snapshots of MD simulations (from D.E. Shaw Research). We demonstrate that SARS-CoV2 forms a more dominant, robust connection with the ACE2-receptor as compared to the less virulent SARS-CoV1. Globally, this stronger connectivity is reflected by our percolation analysis where the interface cluster for the SARS-CoV2-ACE2 complex persists when restricted to stronger and stronger bonds, as compared to the SARSCoV1- ACE2 complex. At the atomic level, interface clique structure reflects a stronger connectivity in the former complex. We pinpoint key functional residues in SARSCoV2 that play important roles in establishing this higher connectivity. Thus, our studies provide an objective method to map spatial connectivity of atomic level non-covalent interactions to global connectivity between any two amino acids in the complex. We also analyze specific snapshots of the MD simulations to highlight prominent variations in network topology that explore diverse conformational landscapes. Finally, we demonstrate that a majority of mutations that occur in the SARSCoV2 spike protein in variants of concern/interest (including the currently circulating JN.1) have been observed at the interface with the ACE2 receptor. Our analyses highlight the importance of interface interactions and provide a rationale for designing receptor-like peptides and proteins to combat immunity-escaping variants.
Background independent tensor networks

Conventional holographic tensor networks can be described as toy holographic maps constructed from many small linear maps acting in a spatially local way, all connected together with ``background entanglement'', i.e. links of a fixed state, often the maximally entangled state. However, these constructions fall short of modeling real holographic maps. One reason is that their ``areas'' are trivial, taking the same value for all states, unlike in gravity where the geometry is dynamical. Recently, new constructions have ameliorated this issue by adding degrees of freedom that ``live on the links''. This makes areas non-trivial, equal to the background entanglement piece plus a new positive piece that depends on the state of the link degrees of freedom. Nevertheless, this still has the downside that there is background entanglement, and hence it only models relatively limited code subspaces in which every area has a definite minimum value given by the background entanglement. In this note, we simply point out that a version of these constructions goes one step further: they can be background independent, with no background entanglement in the holographic map. This is advantageous because it allows tensor networks to model holographic maps for larger code subspaces. In addition to pointing this out, we address some subtleties involved in making it work and point out a nice connection it offers to recent discussions of random CFT data.
A Resource Model For Neural Scaling Law

Neural scaling laws characterize how model performance improves as the model size scales up. Inspired by empirical observations, we introduce a resource model of neural scaling. A task is usually composite hence can be decomposed into many subtasks, which compete for resources (measured by the number of neurons allocated to subtasks). On toy problems, we empirically find that: (1) The loss of a subtask is inversely proportional to its allocated neurons. (2) When multiple subtasks are present in a composite task, the resources acquired by each subtask uniformly grow as models get larger, keeping the ratios of acquired resources constants. We hypothesize these findings to be generally true and build a model to predict neural scaling laws for general composite tasks, which successfully replicates the neural scaling law of Chinchilla models reported in arXiv:2203.15556. We believe that the notion of resource used in this paper will be a useful tool for characterizing and diagnosing neural networks.
Succint Interaction-Aware Explanations

SHAP is a popular approach to explain black-box models by revealing the importance of individual features. As it ignores feature interactions, SHAP explanations can be confusing up to misleading. NSHAP, on the other hand, reports the additive importance for all subsets of features. While this does include all interacting sets of features, it also leads to an exponentially sized, difficult to interpret explanation. In this paper, we propose to combine the best of these two worlds, by partitioning the features into parts that significantly interact, and use these parts to compose a succinct, interpretable, additive explanation. We derive a criterion by which to measure the representativeness of such a partition for a models behavior, traded off against the complexity of the resulting explanation. To efficiently find the best partition out of super-exponentially many, we show how to prune sub-optimal solutions using a statistical test, which not only improves runtime but also helps to detect spurious interactions. Experiments on synthetic and real world data show that our explanations are both more accurate resp. more easily interpretable than those of SHAP and NSHAP.
CREMA: Multimodal Compositional Video Reasoning via Efficient Modular Adaptation and Fusion

Despite impressive advancements in multimodal compositional reasoning approaches, they are still limited in their flexibility and efficiency by processing fixed modality inputs while updating a lot of model parameters. This paper tackles these critical challenges and proposes CREMA, an efficient and modular modality-fusion framework for injecting any new modality into video reasoning. We first augment multiple informative modalities (such as optical flow, 3D point cloud, audio) from given videos without extra human annotation by leveraging existing pre-trained models. Next, we introduce a query transformer with multiple parameter-efficient modules associated with each accessible modality. It projects diverse modality features to the LLM token embedding space, allowing the model to integrate different data types for response generation. Furthermore, we propose a fusion module designed to compress multimodal queries, maintaining computational efficiency in the LLM while combining additional modalities. We validate our method on video-3D, video-audio, and video-language reasoning tasks and achieve better/equivalent performance against strong multimodal LLMs, including BLIP-2, 3D-LLM, and SeViLA while using 96% fewer trainable parameters. We provide extensive analyses of CREMA, including the impact of each modality on reasoning domains, the design of the fusion module, and example visualizations.
Scalable Wasserstein Gradient Flow for Generative Modeling through Unbalanced Optimal Transport

Wasserstein Gradient Flow (WGF) describes the gradient dynamics of probability density within the Wasserstein space. WGF provides a promising approach for conducting optimization over the probability distributions. Numerically approximating the continuous WGF requires the time discretization method. The most well-known method for this is the JKO scheme. In this regard, previous WGF models employ the JKO scheme and parametrize transport map for each JKO step. However, this approach results in quadratic training complexity $O(K^2)$ with the number of JKO step $K$. This severely limits the scalability of WGF models. In this paper, we introduce a scalable WGF-based generative model, called Semi-dual JKO (S-JKO). Our model is based on the semi-dual form of the JKO step, derived from the equivalence between the JKO step and the Unbalanced Optimal Transport. Our approach reduces the training complexity to $O(K)$. We demonstrate that our model significantly outperforms existing WGF-based generative models, achieving FID scores of 2.62 on CIFAR-10 and 5.46 on CelebA-HQ-256, which are comparable to state-of-the-art image generative models.
On the detection of alpha emission from a low-voltage DC deuterium discharge with palladium electrodes

The possibility of nuclear reactions within a low voltage DC deuterium discharge with palladium electrodes is addressed. The solid-state nuclear track detector known as CR-39 was chosen to investigate the emission of energetic charged particles from the electrodes. A partially automated imaging platform and feature classification process was developed to scan the CR-39 surface and detect tracks. Typical discharge parameters were 10 Torr deuterium, 5-7 mm electrode gap distance, 20-40 mA/cm$^2$ current density, and -500 $\pm$ 100 V cathode bias. After discharge treatments to varied ion-cathode fluences, tracks formed in CR-39 which consistently corresponded to 138 $\pm$ 21 keV alpha particles emitted from the palladium electrodes. The track densities for deuterium discharges were often $\sim$100 times above controls with hydrogen and helium. Currently, there are no known mechanisms to accelerate ions to these energies within the apparatus. The production of energetic alpha particles with no source of helium or a means to accelerate the ions to such high energies indicate a nuclear origin. From particle trajectory estimates based on track geometries, it was concluded the reactions originated at the Pd electrodes and not from external sources such as atmospheric radon or cosmic rays.
The Effect of Sampling Temperature on Problem Solving in Large Language Models

In this research study, we empirically investigate the effect of sampling temperature on the performance of Large Language Models (LLMs) on various problem-solving tasks. We created a multiple-choice question-and-answer (MCQA) exam by randomly sampling problems from standard LLM benchmarks. Then, we used four popular LLMs with five prompt-engineering techniques to solve the MCQA problems while increasing the sampling temperature from 0.0 to 1.0. Despite anecdotal reports to the contrary, our empirical results indicate that changes in temperature in the range 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks. In addition, these results appear to hold regardless of the LLM, the prompt-engineering technique, or the problem domain. All code, data, and supplemental materials are available on GitHub at: https://github.com/matthewrenze/jhu-llm-temperature.
cecilia: A Machine Learning-Based Pipeline for Measuring Metal Abundances of Helium-rich Polluted White Dwarfs

Over the past several decades, conventional spectral analysis techniques of polluted white dwarfs have become powerful tools to learn about the geology and chemistry of extrasolar bodies. Despite their proven capabilities and extensive legacy of scientific discoveries, these techniques are however still limited by their manual, time-intensive, and iterative nature. As a result, they are susceptible to human errors and are difficult to scale up to population-wide studies of metal pollution. This paper seeks to address this problem by presenting cecilia, the first Machine Learning (ML)-powered spectral modeling code designed to measure the metal abundances of intermediate-temperature (10,000 ≤ Teff≤ 20 000 K), Helium-rich polluted white dwarfs. Trained with more than 22 000 randomly drawn atmosphere models and stellar parameters, our pipeline aims to overcome the limitations of classical methods by replacing the generation of synthetic spectra from computationally expensive codes and uniformly spaced model grids, with a fast, automated, and efficient neural-network-based interpolator. More specifically, cecilia combines state-of-the-art atmosphere models, powerful artificial intelligence tools, and robust statistical techniques to rapidly generate synthetic spectra of polluted white dwarfs in high-dimensional space, and enable accurate (≲0.1 dex) and simultaneous measurements of 14 stellar parameters – including 11 elemental abundances – from real spectroscopic observations. As massively multiplexed astronomical surveys begin scientific operations, cecilia’s performance has the potential to unlock large-scale studies of extrasolar geochemistry and propel the field of white dwarf science into the era of Big Data. In doing so, we aspire to uncover new statistical insights that were previously impractical with traditional white dwarf characterisation techniques.
SpiRit-LM: Interleaved Spoken and Written Language Model

We introduce SPIRIT-LM, a foundation multimodal language model that freely mixes text and speech. Our model is based on a pretrained text language model that we extend to the speech modality by continuously training it on text and speech units. Speech and text sequences are concatenated as a single set of tokens, and trained with a word-level interleaving method using a small automatically-curated speech-text parallel corpus. SPIRIT-LM comes in two versions: a BASE version that uses speech semantic units and an EXPRESSIVE version that models expressivity using pitch and style units in addition to the semantic units. For both versions, the text is encoded with subword BPE tokens. The resulting model displays both the semantic abilities of text models and the expressive abilities of speech models. Additionally, we demonstrate that SPIRIT-LM is able to learn new tasks in a few-shot fashion across modalities (i.e. ASR, TTS, Speech Classification).