Miracle-dz/UniFAR-FTUs huggingface dataset:
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

The success of large language models (LLMs), like GPT-4 and ChatGPT, has led to the development of numerous cost-effective and accessible alternatives that are created by finetuning open-access LLMs with task-specific data (e.g., ChatDoctor) or instruction data (e.g., Alpaca). Among the various fine-tuning methods, adapter-based parameter-efficient fine-tuning (PEFT) is undoubtedly one of the most attractive topics, as it only requires fine-tuning a few external parameters instead of the entire LLMs while achieving comparable or even better performance. To enable further research on PEFT methods of LLMs, this paper presents LLM-Adapters, an easy-to-use framework that integrates various adapters into LLMs and can execute these adapter-based PEFT methods of LLMs for different tasks. The framework includes state-of-the-art open-access LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as Series adapters, Parallel adapter, Prompt-based learning and Reparametrization-based methods. Moreover, we conduct extensive empirical studies on the impact of adapter types, placement locations, and hyper-parameters to the best design for each adapter-based methods. We evaluate the effectiveness of the adapters on fourteen datasets from two different reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning. The results demonstrate that using adapter-based PEFT in smaller-scale LLMs (7B) with few extra trainable parameters yields comparable, and in some cases superior, performance to powerful LLMs (175B) in zero-shot inference on both reasoning tasks.
Early Results from GLASS-JWST. XXII. Rest-frame UV–Optical Spectral Properties of Lyα Emitting Galaxies at 3 < z < 6

Lyα emission is possibly the best indirect diagnostic of Lyman continuum (LyC) escape since the conditions that favor the escape of Lyα photons are often the same that allow for the escape of LyC photons. In this work, we present the rest-frame UV–optical spectral characteristics of 11 Lyα emitting galaxies at 3 < z < 6—the redshift range that optimizes between intergalactic medium attenuation effects and temporal proximity to the epoch of reionization. From a combined analysis of JWST/NIRSpec and MUSE data, we present the Lyα escape fraction and study its correlation with other physical properties of galaxies that might facilitate Lyα escape. We find that our galaxies have low masses (80% of the sample with log10M⋆<9.5M⊙ ), compact sizes (median R e ∼ 0.7 kpc), low dust content, moderate [O iii]/[O ii] flux ratios (mean ∼ 6.8 ± 1.2), and moderate Lyα escape fractions (mean fescLyα∼ 0.11). Our sample shows characteristics that are broadly consistent with low-redshift galaxies with Lyα emission, which are termed as “analogs” of the high-redshift population. We predict the LyC escape fraction in our sample to be low (0.03–0.07), although larger samples in the postreionization epoch are needed to confirm these trends.
Interactions of a collapsing laser-induced cavitation bubble with a hemispherical droplet attached to a rigid boundary

Abstract We investigate experimentally and theoretically the interactions between a cavitation bubble and a hemispherical pendant oil droplet immersed in water. In experiments, the cavitation bubble is generated by a focused laser pulse right below the pendant droplet with well-controlled bubble–wall distances and bubble–droplet size ratios. By high-speed imaging, four typical interactions are observed, namely: oil droplet rupture; water droplet entrapment; oil droplet large deformation; and oil droplet mild deformation. The bubble jetting at the end of collapse and the migration of the bubble centroid are particularly different in each bubble–droplet interaction. We propose theoretical models based on the method of images for calculating the Kelvin impulse and the anisotropy parameter which quantitatively reflects the migration of the bubble centroid at the end of the collapse. Finally, we explain that a combination of the Weber number and the anisotropy parameter determines the regimes of the bubble–droplet interactions.
The Role of Mass and Environment on Satellite Distributions around Milky Way Analogs in the Romulus25 Simulation

We study satellite counts and quenched fractions for satellites of Milky Way analogs in Romulus25, a large-volume cosmological hydrodynamic simulation. Depending on the definition of a Milky Way analog, we have between 66 and 97 Milky Way analogs in Romulus25, a 25 Mpc per-side uniform volume simulation. We use these analogs to quantify the effect of environment and host properties on satellite populations. We find that the number of satellites hosted by a Milky Way analog increases predominantly with host stellar mass, while environment, as measured by the distance to a Milky Way–mass or larger halo, may have a notable impact in high isolation. Similarly, we find that the satellite quenched fraction for our analogs also increases with host stellar mass, and potentially in higher-density environments. These results are robust for analogs within 3 Mpc of another Milky Way–mass or larger halo, the environmental parameter space where the bulk of our sample resides. We place these results in the context of observations through comparisons to the Exploration of Local VolumE Satellites and Satellites Around Galactic Analogs surveys. Our results are robust to changes in Milky Way analog selection criteria, including those that mimic observations. Finally, as our samples naturally include Milky Way–Andromeda pairs, we examine quenched fractions in pairs versus isolated systems. We find potential evidence, though not conclusive, that pairs, defined as being within 1 Mpc of another Milky Way–mass or larger halo, may have higher satellite quenched fractions.
So, I Can Feel Normal: Participatory Design for Accessible Social Media Sites for Individuals with Traumatic Brain Injury

Traumatic brain injury (TBI) can result in chronic sensorimotor, cognitive, psychosocial, and communication challenges that can limit social participation. Social media can be a useful outlet for social participation for individuals with TBI, but there are barriers to access. While research has drawn attention to the nature of access barriers, few studies have investigated technological solutions to address these barriers, particularly considering the perspectives of individuals with TBI. To address this gap in knowledge, we used a participatory approach to engage 10 adults with TBI in conceptualizing tools to address their challenges accessing Facebook. Participants described multifaceted challenges in using social media, including interface overload, social comparisons, and anxiety over self-presentation and communication after injury. They discussed their needs and preferences and generated ideas for design solutions. Our work contributes to designing assistive and accessibility technology to facilitate an equal access to the benefits of social media for individuals with TBI.
End-to-End Models for Chemical–Protein Interaction Extraction: Better Tokenization and Span-Based Pipeline Strategies

End-to-end relation extraction (E2ERE) is an important task in information extraction, more so for biomedicine as scientific literature continues to grow exponentially. E2ERE typically involves identifying entities (or named entity recognition (NER)) and associated relations, while most RE tasks simply assume that the entities are provided upfront and end up performing relation classification. E2ERE is inherently more difficult than RE alone given the potential snowball effect of errors from NER leading to more errors in RE. A complex dataset in biomedical E2ERE is the ChemProt dataset (BioCreative VI, 2017) that identifies relations between chemical compounds and genes/proteins in scientific literature. ChemProt is included in all recent biomedical natural language processing benchmarks including BLUE, BLURB, and BigBio. However, its treatment in these benchmarks and in other separate efforts is typically not end-to-end, with few exceptions. In this effort, we employ a span-based pipeline approach to produce a new state-of-the-art E2ERE performance on the ChemProt dataset, resulting in >4% improvement in F1-score over the prior best effort. Our results indicate that a straightforward fine-grained tokenization scheme helps span-based approaches excel in E2ERE, especially with regards to handling complex named entities. Our error analysis also identifies a few key failure modes in E2ERE for ChemProt.
Metallic glass ingots based on yttrium

We report a family of yttrium metallic alloys that are able to form glassy ingots directly from the liquid, as well as forming bulk-sized amorphous rods with diameters over 2 cm by water cooling of the alloy melt sealed in quartz tubes. It is apparent that, in addition to the strong chemical interaction among the components, the simultaneous occurrence of well-distributed atom sizes and a strongly depressed liquidus temperature in multicomponent metallic alloys is responsible for the formation of glassy ingots.
Sparse Cholesky Factorization for Solving Nonlinear PDEs via Gaussian Processes

In recent years, there has been widespread adoption of machine learning-based approaches to automate the solving of partial differential equations (PDEs). Among these approaches, Gaussian processes (GPs) and kernel methods have garnered considerable interest due to their flexibility, robust theoretical guarantees, and close ties to traditional methods. They can transform the solving of general nonlinear PDEs into solving quadratic optimization problems with nonlinear, PDE-induced constraints. However, the complexity bottleneck lies in computing with dense kernel matrices obtained from pointwise evaluations of the covariance kernel, and its \textit{partial derivatives}, a result of the PDE constraint and for which fast algorithms are scarce. The primary goal of this paper is to provide a near-linear complexity algorithm for working with such kernel matrices. We present a sparse Cholesky factorization algorithm for these matrices based on the near-sparsity of the Cholesky factor under a novel ordering of pointwise and derivative measurements. The near-sparsity is rigorously justified by directly connecting the factor to GP regression and exponential decay of basis functions in numerical homogenization. We then employ the Vecchia approximation of GPs, which is optimal in the Kullback-Leibler divergence, to compute the approximate factor. This enables us to compute $\epsilon$-approximate inverse Cholesky factors of the kernel matrices with complexity $O(N\log^d(N/\epsilon))$ in space and $O(N\log^{2d}(N/\epsilon))$ in time. We integrate sparse Cholesky factorizations into optimization algorithms to obtain fast solvers of the nonlinear PDE. We numerically illustrate our algorithm's near-linear space/time complexity for a broad class of nonlinear PDEs such as the nonlinear elliptic, Burgers, and Monge-Amp\`ere equations.
Primordial black holes from single-field inflation: a fine-tuning audit

All single-field inflationary models invoke varying degrees of tuning in order to account for cosmological observations. Mechanisms that generate primordial black holes (PBHs) from enhancement of primordial power at small scales posit inflationary potentials that transiently break scale invariance and possibly adiabaticity over a range of modes. This requires additional tuning on top of that required to account for observations at scales probed by cosmic microwave background (CMB) anisotropies. In this paper we study the parametric dependence of various single-field models of inflation that enhance power at small scales and quantify the degree to which coefficients in the model construction have to be tuned in order for certain observables to lie within specified ranges. We find significant tuning: changing the parameters of the potentials by between one part in a hundred and one part in 108 (depending on the model) is enough to change the power spectrum peak amplitude by an order one factor. The fine-tuning of the PBH abundance is larger still by 1–2 orders of magnitude. We highlight the challenges imposed by this tuning on any given model construction. Furthermore, polynomial potentials appear to require significant additional fine-tuning to also match the CMB observations.
High-performance Time Series Anomaly Discovery on Graphics Processors

Currently, discovering subsequence anomalies in time series remains one of the most topical research problems. A subsequence anomaly refers to successive points in time that are collectively abnormal, although each point is not necessarily an outlier. Among numerous approaches to discovering subsequence anomalies, the discord concept is considered one of the best. A time series discord is intuitively defined as a subsequence of a given length that is maximally far away from its non-overlapping nearest neighbor. Recently introduced, the MERLIN algorithm discovers time series discords of every possible length in a specified range, thereby eliminating the need to set even that sole parameter to discover discords in a time series. However, MERLIN is serial, and its parallelization could increase the performance of discord discovery. In this article, we introduce a novel parallelization scheme for GPUs called PALMAD, parallel arbitrary length MERLIN-based anomaly discovery. As opposed to its serial predecessor, PALMAD employs recurrent formulas we have derived to avoid redundant calculations, and advanced data structures for the efficient implementation of parallel processing. Experimental evaluation over real-world and synthetic time series shows that our algorithm outperforms parallel analogs. We also apply PALMAD to discover anomalies in a real-world time series, employing our proposed discord heatmap technique to illustrate the results.
PEACH: Pre-Training Sequence-to-Sequence Multilingual Models for Translation with Semi-Supervised Pseudo-Parallel Document Generation

Multilingual pre-training significantly improves many multilingual NLP tasks, including machine translation. Most existing methods are based on some variants of masked language modeling and text-denoising objectives on monolingual data. Multilingual pre-training on monolingual data ignores the availability of parallel data in many language pairs. Also, some other works integrate the available human-generated parallel translation data in their pre-training. This kind of parallel data is definitely helpful, but it is limited even in high-resource language pairs. This paper introduces a novel semi-supervised method, SPDG, that generates high-quality pseudo-parallel data for multilingual pre-training. First, a denoising model is pre-trained on monolingual data to reorder, add, remove, and substitute words, enhancing the pre-training documents’ quality. Then, we generate different pseudo-translations for each pre-training document using dictionaries for word-by-word translation and applying the pre-trained denoising model. The resulting pseudo-parallel data is then used to pre-train our multilingual sequence-to-sequence model, PEACH. Our experiments show that PEACH outperforms existing approaches used in training mT5 and mBART on various translation tasks, including supervised, zero- and few-shot scenarios. Moreover, PEACH’s ability to transfer knowledge between similar languages makes it particularly useful for low-resource languages. Our results demonstrate that with high-quality dictionaries for generating accurate pseudo-parallel, PEACH can be valuable for low-resource languages.
Empirical Design in Reinforcement Learning

Empirical design in reinforcement learning is no small task. Running good experiments requires attention to detail and at times significant computational resources. While compute resources available per dollar have continued to grow rapidly, so have the scale of typical experiments in reinforcement learning. It is now common to benchmark agents with millions of parameters against dozens of tasks, each using the equivalent of 30 days of experience. The scale of these experiments often conflict with the need for proper statistical evidence, especially when comparing algorithms. Recent studies have highlighted how popular algorithms are sensitive to hyper-parameter settings and implementation details, and that common empirical practice leads to weak statistical evidence (Machado et al., 2018; Henderson et al., 2018). Here we take this one step further. This manuscript represents both a call to action, and a comprehensive resource for how to do good experiments in reinforcement learning. In particular, we cover: the statistical assumptions underlying common performance measures, how to properly characterize performance variation and stability, hypothesis testing, special considerations for comparing multiple agents, baseline and illustrative example construction, and how to deal with hyper-parameters and experimenter bias. Throughout we highlight common mistakes found in the literature and the statistical consequences of those in example experiments. The objective of this document is to provide answers on how we can use our unprecedented compute to do good science in reinforcement learning, as well as stay alert to potential pitfalls in our empirical design.
Moment Methods for the 3D Radiative Transfer Equation Based on $\varphi$-Divergences

The method of moments is widely used for the reduction of kinetic equations into fluid models. It consists in extracting the moments of the kinetic equation with respect to a velocity variable, but the resulting system is a priori underdetermined and requires a closure relation. In this paper, we adapt the $\varphi$-divergence based closure, recently developed for rarefied gases i.e. with a velocity variable describing $\mathbb{R}^d$, to the radiative transfer equation where velocity describes the unit sphere $\mathbb{S}^2$. This closure is analyzed and a numerical method to compute it is provided. Eventually, it provides the main desirable properties to the resulting system of moments: Similarily to the entropy minimizing closure ($M_N$), it dissipates an entropy and captures exactly the equilibrium distribution. However, contrarily to $M_N$, it remains computationnally tractable even at high order and it relies on an exact quadrature formula which preserves exactly symmetry properties, i.e. it does not trigger ray effects. The purely anisotropic regimes (beams) are not captured exactly but they can be approached as close as desired and the closures remains again tractable in this limit.
Thermal Convection in a Rotating Annulus of Liquid: Effect of Viscosity on the Transition Between Axisymmetric and Non-Axisymmetric Flow Regimes

Abstract When a vertical rotating annulus of liquid is subject to a horizontal temperature gradient, provided that the coefficient of kinematical viscosity, ν¯, is not too great and the angular velocity of rotation,Ω is sufficiently high, four distinct regimes of hydrodynamical flow are possible, as shown in previous work by Hide. Only one of these regimes is characterized by symmetry about the axis of rotation. The principal properties of the flow depend largely on the dimensionless parameters Π2≡d/(b−a), Π4gdΔρ/ρ¯Ω2(b−a)2, Π5≡4Ω2(b−a)5/ν¯2d and Π6≡ν¯/κ¯, where d is the depth of the fluid,b and a are the radi of curvature of the surfaces of the annulus, g is the acceleration of gravity, ρ¯ is the mean density of the fluid, Δρ is the density contrast associated with the impressed horizontal temperature gradient and κ¯ is the thermal diffusivity of the fluid. In a diagram with log10Π5 as abscissa and log10Π4 as ordinate, axisymmetric flow is found outside an anvil-shaped region whose upper boundary lies be.
Autoregressive Neural TensorNet: Bridging Neural Networks and Tensor Networks for Quantum Many-Body Simulation

Quantum many-body physics simulation has important impacts on understanding fundamental science and has applications to quantum materials design and quantum technology. However, due to the exponentially growing size of the Hilbert space with respect to the particle number, a direct simulation is intractable. While representing quantum states with tensor networks and neural networks are the two state-of-the-art methods for approximate simulations, each has its own limitations in terms of expressivity and optimization. To address these challenges, we develop a novel architecture, Autoregressive Neural Ten-sorNet (ANTN), which bridges tensor networks and autoregressive neural networks. We show that Autoregressive Neural TensorNet parameterizes normalized wavefunctions with exact sampling, generalizes the expressivity of tensor networks and autoregressive neural networks, and inherits a variety of symmetries from autoregressive neural networks. We demonstrate our approach on the 2D J 1 - J 2 Heisenberg model with different systems sizes and coupling parameters, outper-forming both tensor networks and autoregressive neural networks. Our work opens up new opportunities for both scientiﬁc simulations and machine learning applications.
EGC: Image Generation and Classiﬁcation via a Single Energy-Based Model

Learning image classiﬁcation and image generation us-ing the same set of network parameters is a challenging problem. Recent advanced approaches perform well in one task often exhibit poor performance in the other. This work introduces an energy-based classiﬁer and generator, namely EGC, which can achieve superior performance in both tasks using a single neural network. Unlike a conventional classiﬁer that outputs a label given an image ( i.e ., a conditional distribution p ( y | x ) ), the forward pass in EGC is a classiﬁer that outputs a joint distribution p ( x , y ) , enabling an image generator in its backward pass by marginalizing out the label y . This is done by estimating the classiﬁcation probability given a noisy image from the diffusion process in the forward pass, while denoising it using the score function estimated in the backward pass. EGC achieves competitive generation results compared with state-of-the-art approaches on ImageNet-1k, CelebA-HQ and LSUN Church, while achieving superior classiﬁcation accuracy and robustness against adversarial attacks on CIFAR-10. This work represents the ﬁrst successful attempt to simultaneously excel in both tasks using a single set of network parameters. We believe that EGC bridges the gap between discriminative and generative learning. Code will be released at https://github.com/GuoQiushan/EGC.
Matched Machine Learning: A Generalized Framework for Treatment Effect Inference With Learned Metrics

We introduce Matched Machine Learning, a framework that combines the flexibility of machine learning black boxes with the interpretability of matching, a longstanding tool in observational causal inference. Interpretability is paramount in many high-stakes application of causal inference. Current tools for nonparametric estimation of both average and individualized treatment effects are black-boxes that do not allow for human auditing of estimates. Our framework uses machine learning to learn an optimal metric for matching units and estimating outcomes, thus achieving the performance of machine learning black-boxes, while being interpretable. Our general framework encompasses several published works as special cases. We provide asymptotic inference theory for our proposed framework, enabling users to construct approximate confidence intervals around estimates of both individualized and average treatment effects. We show empirically that instances of Matched Machine Learning perform on par with black-box machine learning methods and better than existing matching methods for similar problems. Finally, in our application we show how Matched Machine Learning can be used to perform causal inference even when covariate data are highly complex: we study an image dataset, and produce high quality matches and estimates of treatment effects.