Miracle-dz/UniFAR-FTUs huggingface dataset:
In-context Learning and Induction Heads

“Induction heads” are attention heads that implement a simple algorithm to complete token sequences like [A][B] . [A] → [B]. In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all “incontext learning” in large transformer models (i.e. decreasing loss at increasing token indices). We find that induction heads develop at precisely the same point as a sudden sharp increase in incontext learning ability, visible as a bump in the training loss. We present six complementary lines of evidence, arguing that induction heads may be the mechanistic source of general in-context learning in transformer models of any size. For small attention-only models, we present strong, causal evidence; for larger models with MLPs, we present correlational evidence. We recommend reading this paper as an HTML article. As Transformer generative models continue to scale and gain increasing real world use, addressing their associated safety problems becomes increasingly important. Mechanistic interpretability – attempting to reverse engineer the detailed computations performed by the model – offers one possible avenue for addressing these safety issues. If we can understand the internal structures that cause Transformer models to produce the outputs they do, then we may be able to address current safety problems more systematically, as well as anticipating safety problems in future more powerful models. [1, 2, 3, 4, 5]
Learning to Drop Out: An Adversarial Approach to Training Sequence VAEs

In principle, applying variational autoencoders (VAEs) to sequential data offers a method for controlled sequence generation, manipulation, and structured representation learning. However, training sequence VAEs is challenging: autoregressive decoders can often explain the data without utilizing the latent space, known as posterior collapse. To mitigate this, state-of-the-art models weaken the powerful decoder by applying uniformly random dropout to the decoder input. We show theoretically that this removes pointwise mutual information provided by the decoder input, which is compensated for by utilizing the latent space. We then propose an adversarial training strategy to achieve information-based stochastic dropout. Compared to uniform dropout on standard text benchmark datasets, our targeted approach increases both sequence modeling performance and the information captured in the latent space.
Learned Force Fields Are Ready For Ground State Catalyst Discovery

We present evidence that learned density functional theory (``DFT'') force fields are ready for ground state catalyst discovery. Our key finding is that relaxation using forces from a learned potential yields structures with similar or lower energy to those relaxed using the RPBE functional in over 50\% of evaluated systems, despite the fact that the predicted forces differ significantly from the ground truth. This has the surprising implication that learned potentials may be ready for replacing DFT in challenging catalytic systems such as those found in the Open Catalyst 2020 dataset. Furthermore, we show that a force field trained on a locally harmonic energy surface with the same minima as a target DFT energy is also able to find lower or similar energy structures in over 50\% of cases. This ``Easy Potential'' converges in fewer steps than a standard model trained on true energies and forces, which further accelerates calculations. Its success illustrates a key point: learned potentials can locate energy minima even when the model has high force errors. The main requirement for structure optimisation is simply that the learned potential has the correct minima. Since learned potentials are fast and scale linearly with system size, our results open the possibility of quickly finding ground states for large systems.
A review and reformulation of macroscopic realism: resolving its deficiencies using the framework of generalized probabilistic theories

The notion of macrorealism was introduced by Leggett and Garg in an attempt to capture our intuitive conception of the macroscopic world, which seems difficult to reconcile with our knowledge of quantum physics. By now, numerous experimental witnesses have been proposed as methods of falsifying macrorealism. In this work, I critically review and analyze both the definition of macrorealism and the various proposed tests thereof, identifying a number of problems with these (and revisiting key criticisms raised by other authors). I then show that all these problems can be resolved by reformulating macrorealism within the framework of generalized probabilistic theories. In particular, I argue that a theory should be considered to be macrorealist if and only if it describes every macroscopic system by a strictly classical (i.e., simplicial) generalized probabilistic theory. This approach brings significant clarity and precision to our understanding of macrorealism, and provides us with a host of new tools – both conceptual and technical – for studying macrorealism. I leverage this approach i) to clarify in what sense macrorealism is a notion of classicality, ii) to propose a new test of macrorealism that is maximally informative and theory-independent (unlike all prior tests of macrorealism), and iii) to show that every proof of generalized contextuality on a macroscopic system implies the failure of macrorealism.
Re-contextualizing Fairness in NLP: The Case of India

Recent research has revealed undesirable biases in NLP data and models. However, these efforts focus of social disparities in West, and are not directly portable to other geo-cultural contexts. In this paper, we focus on NLP fairness in the context of India. We start with a brief account of the prominent axes of social disparities in India. We build resources for fairness evaluation in the Indian context and use them to demonstrate prediction biases along some of the axes. We then delve deeper into social stereotypes for Region and Religion, demonstrating its prevalence in corpora and models. Finally, we outline a holistic research agenda to re-contextualize NLP fairness research for the Indian context, accounting for Indian societal context, bridging technological gaps in NLP capabilities and resources, and adapting to Indian cultural values. While we focus on India, this framework can be generalized to other geo-cultural contexts.
Inhomogeneous Khitchine-Groshev type theorems on manifolds over function fields

The goal of this paper is to establish a complete Khintchine-Groshev type theorem in both homogeneous and inhomogeneous setting, on analytic nondegenerate manifolds over a local ﬁeld of positive characteristic. The dual form of Diophantine approximation has been considered here. Our treatise provides the function ﬁeld analogues of the various earlier results of this type, studied in the euclidean and S -adic framework, by Bernik, Kleinbock and Margulis [10], Beresnevich, Bernik, Kleinbock and Margulis [3], Badziahin, Beresnevich and Velani [4], and Golseﬁdy [38, 39] and Datta and Ghosh [13].
Least-Squares Methods for Nonnegative Matrix Factorization Over Rational Functions

Nonnegative matrix factorization (NMF) models are widely used to analyze linearly mixed nonnegative data. When the data is made of samplings of continuous signals, the factors in NMF can be constrained to be samples of nonnegative rational functions. This leads to a fairly general model referred to as NMF using rational functions (R-NMF). We first show that, unlike NMF, R-NMF possesses an essentially unique factorization under mild assumptions, which is crucial in applications where the ground-truth factors need to be recovered, as in blind source separation problems. Then we present different approaches to solve R-NMF: the R-HANLS, R-ANLS and R-NLS methods. In our tests, no method significantly outperforms the others in all cases, and all three methods offer a different trade-off between solution accuracy and computational requirements. Indeed, while R-HANLS is fast and accurate for large problems, R-ANLS is more accurate, but also more resources demanding, both in time and memory and R-NLS is even more accurate but only for small problems. Then, crucially we show that R-NMF models outperforms NMF in various tasks including the recovery of semi-synthetic continuous signals, and a classification problem of real hyperspectral signals.
News Summarization and Evaluation in the Era of GPT-3

The recent success of prompting large language models like GPT-3 has led to a paradigm shift in NLP research. In this paper, we study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries. Finally, we evaluate models on a setting beyond generic summarization, specifically keyword-based summarization, and show how dominant fine-tuning approaches compare to prompting. To support further research, we release: (a) a corpus of 10K generated summaries from fine-tuned and prompt-based models across 4 standard summarization benchmarks, (b) 1K human preference judgments comparing different systems for generic- and keyword-based summarization.
Face Super-Resolution Using Stochastic Differential Equations

Diffusion models have proven effective for various applications such as images, audio and graph generation. Other important applications are image super-resolution and the solution of inverse problems. More recently, some works have used stochastic differential equations (SDEs) to generalize diffusion models to continuous time. In this work, we introduce SDEs to generate super-resolution face images. To the best of our knowledge, this is the first time SDEs have been used for such an application. The proposed method provides an improved peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and consistency than the existing super-resolution methods based on diffusion models. In particular, we also assess the potential application of this method for the face recognition task. A generic facial feature extractor is used to compare the super-resolution images with the ground truth, and superior results were obtained compared with other methods. Our code is publicly available at https://github.com/marcelowds/sr-sde.
First Detection of the Molecular Cloud Population in the Extended Ultraviolet Disk of M83

We report a CO(J = 3−2) detection of 23 molecular clouds in the extended ultraviolet (XUV) disk of the spiral galaxy M83 with the Atacama Large Millimeter/submillimeter Array. The observed 1 kpc2 region is at about 1.24 times the optical radius (R 25) of the disk, where CO(J = 2–1) was previously not detected. The detection and nondetection, as well as the level of star formation (SF) activity in the region, can be explained consistently if the clouds have the mass distribution common among Galactic clouds, such as Orion A—with star-forming dense clumps embedded in thick layers of bulk molecular gas, but in a low-metallicity regime where their outer layers are CO-deficient and CO-dark. The cloud and clump masses, estimated from CO(3−2), range from 8.2 × 102 to 2.3 × 104 M ⊙ and from 2.7 × 102 to 7.5 × 103 M ⊙, respectively. The most massive clouds appear similar to Orion A in star formation activity as well as in mass, as expected if the cloud mass structure is common. The overall low SF activity in the XUV disk could be due to the relative shortage of gas in the molecular phase. The clouds are distributed like chains up to 600 pc (or longer) in length, suggesting that the trigger of cloud formation is on large scales. The common cloud mass structure also justifies the use of high-J CO transitions to trace the total gas mass of clouds, or galaxies, even in the high-z universe. This study is the first demonstration that CO(3−2) is an efficient tracer of molecular clouds even in low-metallicity environments.
Assessing the Most Vulnerable Subgroup to Type II Diabetes Associated with Statin Usage: Evidence from Electronic Health Record Data

ABSTRACT There have been increased concerns that the use of statins, one of the most commonly prescribed drugs for treating coronary artery disease, is potentially associated with the increased risk of new-onset Type II diabetes (T2D). Nevertheless, to date, there is no robust evidence supporting as to whether and what kind of populations are indeed vulnerable for developing T2D after taking statins. In this case study, leveraging the biobank and electronic health record data in the Partner Health System, we introduce a new data analysis pipeline and a novel statistical methodology that address existing limitations by (i) designing a rigorous causal framework that systematically examines the causal effects of statin usage on T2D risk in observational data, (ii) uncovering which patient subgroup is most vulnerable for developing T2D after taking statins, and (iii) assessing the replicability and statistical significance of the most vulnerable subgroup via a bootstrap calibration procedure. Our proposed approach delivers asymptotically sharp confidence intervals and debiased estimate for the treatment effect of the most vulnerable subgroup in the presence of high-dimensional covariates. With our proposed approach, we find that females with high T2D genetic risk are at the highest risk of developing T2D due to statin usage. Supplementary materials for this article are available online.
Geometric Regularization from Overparameterization explains Double Descent and other findings

The volume of the distribution of weight sets associated with a loss value may be the source of implicit regularization from overparameterization due to the phenomenon of contracting volume with increasing dimensions for geometric figures demonstrated by hyperspheres. We introduce the geometric regularization conjecture and extract to an explanation for the double descent phenomenon by considering a similar property resulting from shrinking intrinsic dimensionality of the distribution of potential weight set updates available along training path, where if that distribution retracts across a volume verses dimensionality curve peak when approaching the global minima we could expect geometric regularization to re-emerge. We illustrate how data fidelity representational complexity may influence model capacity double descent interpolation thresholds. The existence of epoch and model capacity double descent curves originating from different geometric forms may imply universality of closed n-manifolds having dimensionally adjusted n-sphere volumetric correspondence.
A new practical framework for the stability analysis of perturbed saddle-point problems and applications

In this paper we prove a new abstract stability result for perturbed saddle-point problems based on a norm fitting technique. We derive the stability condition according to Babuška’s theory from a small inf-sup condition, similar to the famous Ladyzhenskaya-Babuška-Brezzi (LBB) condition, and the other standard assumptions in Brezzi’s theory, in a combined abstract norm. The construction suggests to form the latter from individual fitted norms that are composed from proper seminorms. This abstract framework not only allows for simpler (shorter) proofs of many stability results but also guides the design of parameter-robust norm-equivalent preconditioners. These benefits are demonstrated on mixed variational formulations of generalized Poisson, Stokes, vector Laplace and Biot’s equations.
Convergence Rate of the (1+1)-ES on Locally Strongly Convex and Lipschitz Smooth Functions

Evolution strategy (ES) is one of promising classes of algorithms for black-box continuous optimization. Despite its broad successes in applications, theoretical analysis on the speed of its convergence is limited on convex quadratic functions and their monotonic transformation. In this study, an upper bound and a lower bound of the rate of linear convergence of the (1+1)-ES on locally $L$-strongly convex functions with $U$-Lipschitz continuous gradient are derived as $\exp\left(-\Omega_{d\to\infty}\left(\frac{L}{d\cdot U}\right)\right)$ and $\exp\left(-\frac1d\right)$, respectively. Notably, any prior knowledge on the mathematical properties of the objective function such as Lipschitz constant is not given to the algorithm, whereas the existing analyses of derivative-free optimization algorithms require them.
DISCOVERY OF GeV γ-RAY EMISSION FROM PSR B1259−63/LS 2883

The binary system PSR B1259−63/LS 2883 consists of a 47.8 ms radio pulsar that orbits the companion Be star with a period of 3.4 years in a highly eccentric orbit. The system has been well sampled in radio, X-ray, and TeV γ-ray bands, and shows orbital phase-dependent variability in all observed frequencies. Here we report on the discovery of >100 MeV γ-rays from PSR B1259−63/LS 2883 through the 2010 periastron passage. Using data collected with the Large Area Telescope on board Fermi from 33 days before periastron to 75 days after periastron, PSR B1259−63/LS 2883 was detected at a significance of 13.6 standard deviations. The γ-ray light curve was highly variable over this period, with a changing photon index that correlates with the γ-ray flux. In particular, two major flares that occur after the periastron passage were observed. The onset of γ-ray emission occurs close to, but not at the same orbital phases as, the two disk passages that occur ∼1 month before and ∼1 month after the periastron passage. The fact that the GeV orbital light curve is different from that of the X-ray and TeV light curves strongly suggests that GeV γ-ray emission originates from a different component. We speculate that the observed GeV flares may be resulting from Doppler boosting effects.
Frechet differentiability via partial Frechet differentiability

Let $X_1, \dots, X_n$ be Banach spaces and $f$ a real function on $X=X_1 \times\dots \times X_n$. Let $A_f$ be the set of all points $x \in X$ at which $f$ is partially Fr\' echet differentiable but is not Fr\' echet differentiable. Our results imply that if $X_1, \dots, X_{n-1}$ are Asplund spaces and $f$ is continuous (resp. Lipschitz) on $X$, then $A_f$ is a first category set (resp. a $\sigma$-upper porous set). We also prove that if $X$, $Y$ are separable Banach spaces and $f:X \to Y$ is a Lipschitz mapping, then the set of all points $x \in X$ at which $f$ is G\^ ateaux differentiable, is Fr\' echet differentiable along a closed subspace of finite codimension but is not Fr\' echet differentiable, is $\sigma$-upper porous. A number of related more general results are also proved.
On the sensitivity of extrasolar mass-loss rate ranges: HD 209458b a case study

Fil: Villarreal D'angelo, Carolina Susana. Consejo Nacional de Investigaciones Cientificas y Tecnicas. Centro Cientifico Tecnologico Conicet - Cordoba. Instituto de Astronomia Teorica y Experimental. Universidad Nacional de Cordoba. Observatorio Astronomico de Cordoba. Instituto de Astronomia Teorica y Experimental; Argentina
Flow Forces on a Cylinder Near a Wall or Near Another Cylinder

Abstract : Lift, drag and pressure distributions have been measured on cylinders (of various cross-section shapes) near a wall. It is found that as the cylinder approaches the wall its drag decreases and a lift (away from the wall) develops. A similar development occurs for two cylinders approaching each other (in a plane perpendicular to the flow) but in this case there is a range of gap sizes for which the flow is asymmetrical and the forces on the two cylinders are different. The asymmetry is associated with an instability in the near-wake flow, and it can be flipped (switched) by small disturbances. The results are of interest for the effects of wind on cylindrical structures parallel to the ground, for pairs of structures, like stacks, near each other, etc. (Author)
Localization and defects in axial (In,Ga)N/GaN nanowire heterostructures investigated by spatially resolved luminescence spectroscopy

(In,Ga)N insertions embedded in self-assembled GaN nanowires (NWs) are of current interest for applications in solid-state light emitters. Such structures exhibit a notoriously broad emission band. We use cathodoluminescence spectral imaging in a scanning electron microscope and micro-photoluminescence spectroscopy on single NWs to learn more about the mechanisms underlying this emission. We observe a shift of the emission energy along the stack of six insertions within single NWs that may be explained by compositional pulling. Our results also corroborate reports that the localization of carriers at potential fluctuations within the insertions plays a crucial role for the luminescence of these NW based emitters. Furthermore, we resolve contributions from both structural and point defects in our measurements.
A NUMERICAL TREATMENT OF ANISOTROPIC RADIATION FIELDS COUPLED WITH RELATIVISTIC RESISTIVE MAGNETOFLUIDS

We develop a numerical scheme for solving fully special relativistic, resistive radiation magnetohydrodynamics. Our code guarantees conservation of total mass, momentum, and energy. The radiation energy density and the radiation flux are consistently updated using the M-1 closure method, which can resolve an anisotropic radiation field, in contrast to the Eddington approximation, as well as the flux-limited diffusion approximation. For the resistive part, we adopt a simple form of Ohm's law. The advection terms are explicitly solved with an approximate Riemann solver, mainly the Harten–Lax–van Leer scheme; the HLLC and HLLD schemes are also solved for some tests. The source terms, which describe the gas–radiation interaction and the magnetic energy dissipation, are implicitly integrated, relaxing the Courant–Friedrichs–Lewy condition even in an optically thick regime or a large magnetic Reynolds number regime. Although we need to invert 4 × 4 matrices (for the gas–radiation interaction) and 3 × 3 matrices (for the magnetic energy dissipation) at each grid point for implicit integration, they are obtained analytically without preventing massive parallel computing. We show that our code gives reasonable outcomes in numerical tests for ideal magnetohydrodynamics, propagating radiation, and radiation hydrodynamics. We also applied our resistive code to the relativistic Petschek-type magnetic reconnection, revealing the reduction of the reconnection rate via radiation drag.