Miracle-dz/UniFAR-FTUs huggingface dataset:
Winning Prize Comes from Losing Tickets: Improve Invariant Learning by Exploring Variant Parameters for Out-of-Distribution Generalization

Out-of-Distribution (OOD) Generalization aims to learn robust models that generalize well to various environments without fitting to distribution-specific features. Recent studies based on Lottery Ticket Hypothesis (LTH) address this problem by minimizing the learning target to find some of the parameters that are critical to the task. However, in OOD problems, such solutions are suboptimal as the learning task contains severe distribution noises, which can mislead the optimization process. Therefore, apart from finding the task-related parameters (i.e., invariant parameters), we propose Exploring Variant parameters for Invariant Learning (EVIL) which also leverages the distribution knowledge to find the parameters that are sensitive to distribution shift (i.e., variant parameters). Once the variant parameters are left out of invariant learning, a robust subnetwork that is resistant to distribution shift can be found. Additionally, the parameters that are relatively stable across distributions can be considered invariant ones to improve invariant learning. By fully exploring both variant and invariant parameters, our EVIL can effectively identify a robust subnetwork to improve OOD generalization. In extensive experiments on integrated testbed: DomainBed, EVIL can effectively and efficiently enhance many popular methods, such as ERM, IRM, SAM, etc.
Dwarf Galaxies Show Little ISM Evolution from z ∼ 1 to z ∼ 0: A Spectroscopic Study of Metallicity, Star Formation, and Electron Density

We present gas-phase metallicity measurements for 583 emission line galaxies at 0.3 < z < 0.85, including 388 dwarf galaxies with log(M⋆/M⊙)<9.5 , and explore the dependence of the metallicity on the stellar mass and star formation (SF) properties of the galaxies. Metallicities are determined through the measurement of emission lines in very deep (∼7 hr exposure) Keck/DEIMOS spectra taken primarily from the HALO7D survey. We measure metallicity with three strong-line calibrations (O3Hβ, R23, and O3O2) for the overall sample, as well as with the faint [Ne iii]λ3869 and [O iii]λ4363 emission lines for 112 and 17 galaxies where robust detections were possible. We construct mass–metallicity relations (MZR) for each calibration method, finding MZRs consistent with other strong-line results at comparable redshift, as well as with z ∼ 0 galaxies. We quantify the intrinsic scatter in the MZR as a function of mass, finding it increases with lower stellar mass. We also measure a weak but significant correlation between increased MZR scatter and higher specific star formation rate (SFR). We find a weak influence of SFR in the fundamental metallicity relation as well, with an SFR coefficient of α = 0.21. Finally, we use the flux ratios of the [O ii]λ λ 3727,3729 doublet to calculate gas electron density in ∼1000 galaxies with log(M⋆/M⊙)<10.5 as a function of redshift. We measure low electron densities (n e ∼ 25 cm−3) for z < 1 galaxies, again consistent with z ≈ 0 conditions, but measure higher densities (n e ∼ 100 cm−3) at z > 1. These results all suggest that there is little evolution in star-forming interstellar medium conditions from z ∼ 1 to z = 0, confirmed with a more complete sample of low-mass galaxies than has previously been available in this redshift range.
Real-time 6-DoF Pose Estimation by an Event-based Camera using Active LED Markers

Real-time applications for autonomous operations depend largely on fast and robust vision-based localization systems. Since image processing tasks require processing large amounts of data, the computational resources often limit the performance of other processes. To overcome this limitation, traditional marker-based localization systems are widely used since they are easy to integrate and achieve reliable accuracy. However, classical marker-based localization systems significantly depend on standard cameras with low frame rates, which often lack accuracy due to motion blur. In contrast, event-based cameras provide high temporal resolution and a high dynamic range, which can be utilized for fast localization tasks, even under challenging visual conditions. This paper proposes a simple but effective event-based pose estimation system using active LED markers (ALM) for fast and accurate pose estimation. The proposed algorithm is able to operate in real time with a latency below \SI{0.5}{\milli\second} while maintaining output rates of \SI{3}{\kilo \hertz}. Experimental results in static and dynamic scenarios are presented to demonstrate the performance of the proposed approach in terms of computational speed and absolute accuracy, using the OptiTrack system as the basis for measurement.
Precise Cosmological Constraints from BOSS Galaxy Clustering with a Simulation-Based Emulator of the Wavelet Scattering Transform

We perform a reanalysis of the BOSS CMASS DR12 galaxy dataset using a simulation-based emulator for the Wavelet Scattering Transform (WST) coefficients. Moving beyond our previous works, which laid the foundation for the first galaxy clustering application of this estimator, we construct a neural net-based emulator for the cosmological dependence of the WST coefficients and the 2-point correlation function multipoles, trained from the state-of-the-art suite of \textsc{AbacusSummit} simulations combined with a flexible Halo Occupation Distribution (HOD) galaxy model. In order to confirm the accuracy of our pipeline, we subject it to a series of thorough internal and external mock parameter recovery tests, before applying it to reanalyze the CMASS observations in the redshift range $0.46<z<0.57$. We find that a joint WST + 2-point correlation function likelihood analysis allows us to obtain marginalized 1$\sigma$ errors on the $\Lambda$CDM parameters that are tighter by a factor of $2.5-6$, compared to the 2-point correlation function, and by a factor of $1.4-2.5$ compared to the WST-only results. This corresponds to a competitive $0.9\%$, $2.3\%$ and $1\%$ level of determination for parameters $\omega_c$, $\sigma_8$ $\&$ $n_s$, respectively, and also to a $0.7\%$ $\&$ $2.5 \%$ constraint on derived parameters h and $f(z)\sigma_8(z)$, in agreement with the \textit{Planck} 2018 results. Our results reaffirm the constraining power of the WST and highlight the exciting prospect of employing higher-order statistics in order to fully exploit the power of upcoming Stage-IV spectroscopic observations.
CommonCanvas: An Open Diffusion Model Trained with Creative-Commons Images

We assemble a dataset of Creative-Commons-licensed (CC) images, which we use to train a set of open diffusion models that are qualitatively competitive with Stable Diffusion 2 (SD2). This task presents two challenges: (1) high-resolution CC images lack the captions necessary to train text-to-image generative models; (2) CC images are relatively scarce. In turn, to address these challenges, we use an intuitive transfer learning technique to produce a set of high-quality synthetic captions paired with curated CC images. We then develop a data- and compute-efficient training recipe that requires as little as 3% of the LAION-2B data needed to train existing SD2 models, but obtains comparable quality. These results indicate that we have a sufficient number of CC images (~70 million) for training high-quality models. Our training recipe also implements a variety of optimizations that achieve ~3X training speed-ups, enabling rapid model iteration. We leverage this recipe to train several high-quality text-to-image models, which we dub the CommonCanvas family. Our largest model achieves comparable performance to SD2 on a human evaluation, despite being trained on our CC dataset that is significantly smaller than LAION and using synthetic captions for training. We release our models, data, and code at https://github.com/mosaicml/diffusion/blob/main/assets/common-canvas.md
Data Optimization in Deep Learning: A Survey

Large-scale, high-quality data are considered an essential factor for the successful application of many deep learning techniques. Meanwhile, numerous real-world deep learning tasks still have to contend with the lack of sufficient amounts of high-quality data. Additionally, issues such as model robustness, fairness, and trustworthiness are also closely related to training data. Consequently, a huge number of studies in the existing literature have focused on the data aspect in deep learning tasks. Some typical data optimization techniques include data augmentation, logit perturbation, sample weighting, and data condensation. These techniques usually come from different deep learning divisions and their theoretical inspirations or heuristic motivations may seem unrelated to each other. This study aims to organize a wide range of existing data optimization methodologies for deep learning from the previous literature, and makes the effort to construct a comprehensive taxonomy for them. The constructed taxonomy considers the diversity of split dimensions, and deep sub-taxonomies are constructed for each dimension. On the basis of the taxonomy, connections among the extensive data optimization methods for deep learning are built in terms of four aspects. We probe into rendering several promising and interesting future directions. The constructed taxonomy and the revealed connections will enlighten the better understanding of existing methods and the design of novel data optimization techniques. Furthermore, our aspiration for this survey is to promote data optimization as an independent subdivision of deep learning. A curated, up-to-date list of resources related to data optimization in deep learning is available at \url{https://github.com/YaoRujing/Data-Optimization}.
Notes on limits of accessible categories

Let $\kappa$ be a regular cardinal, $\lambda<\kappa$ be a smaller infinite cardinal, and $\mathsf K$ be a $\kappa$-accessible category where colimits of $\lambda$-indexed chains exist. We show that various category-theoretic constructions applied to $\mathsf K$, such as the inserter and the equifier, produce $\kappa$-accessible categories $\mathsf E$ again, and the most obvious expected description of the full subcategory of $\kappa$-presentable objects in $\mathsf E$ in terms of $\kappa$-presentable objects in $\mathsf K$ holds true. In particular, if $\mathsf C$ is a $\kappa$-small category, then the category of functors $\mathsf C\rightarrow\mathsf K$ is $\kappa$-accessible, and its $\kappa$-presentable objects are precisely all the functors from $\mathsf C$ to the $\kappa$-presentable objects of $\mathsf K$. We proceed to discuss the preservation of $\kappa$-accessibility by conical pseudolimits, lax and oplax limits, and weighted pseudolimits. The results of this paper go back to an unpublished 1977 preprint of Ulmer. Our motivation comes from the theory of flat modules and flat quasi-coherent sheaves.
SkelFMM: A Simplified Fast Multipole Method Based on Recursive Skeletonization

This work introduces the kernel-independent multi-level algorithm"skelFMM"for evaluating all pairwise interactions between $N$ points connected through a kernel such as the fundamental solution of the Laplace or the Helmholtz equations. The method is based on linear algebraic tools such as randomized low rank approximation and"skeleton representations"of far-field interactions. The work is related to previously proposed linear algebraic reformulations of the fast multipole method (FMM), but is distinguished by relying on simpler data structures. In particular, skelFMM does not require an"interaction list", as it relies instead on algebraically-modified kernel interactions between near-neighbors at every level. Like other kernel independent algorithms, it only requires evaluation of the kernel function, allowing the methodology to easily be extended to a range of different kernels in 2D and 3D. The simplicity of the algorithm makes it particularly amenable to parallel implementation on heterogeneous hardware architectures. The performance of the algorithm is demonstrated through numerical experiments conducted on uniform and non-uniform point distributions in 2D and 3D, involving Laplace and (low frequency) Helmholtz kernels. The algorithm relies on a precomputation stage that constructs a tailored representation for a given geometry of points. Once the precomputation has completed, the matrix-vector multiplication attains high speed through GPU acceleration that leverages batched linear algebra.
Mock plectic points

A $p$-arithmetic subgroup of $\mathrm{SL}_2(\mathbb{Q})$ like the Ihara group $\Gamma := \mathrm{SL}_2(\mathbb{Z}[1/p])$ acts by M\"obius transformations on the Poincar\'e upper half plane $\mathcal{H}$ and on Drinfeld's $p$-adic upper half plane $\mathcal{H}_p := \mathbb{P}_1(\mathbb{C}_p)\setminus\mathbb{P}_1(\mathbb{Q}_p)$. The diagonal action of $\Gamma$ on the product is discrete, and the quotient $\Gamma\backslash(\mathcal{H}_p\times \mathcal{H})$ can be envisaged as a"mock Hilbert modular surface". According to a striking prediction of Nekov\'a$\check{\text{r}}$ and Scholl, the CM points on genuine Hilbert modular surfaces should give rise to"plectic Heegner points"that encode non-trivial regulators attached, notably, to elliptic curves of rank two over real quadratic fields. This article develops the analogy between Hilbert modular surfaces and their mock counterparts, with the aim of transposing the plectic philosophy to the mock Hilbert setting, where the analogous plectic invariants are expected to lie in the alternating square of the Mordell-Weil group of certain elliptic curves of rank two over $\mathbb{Q}$.
SuperHF: Supervised Iterative Learning from Human Feedback

While large language models demonstrate remarkable capabilities, they often present challenges in terms of safety, alignment with human values, and stability during training. Here, we focus on two prevalent methods used to align these models, Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). SFT is simple and robust, powering a host of open-source models, while RLHF is a more sophisticated method used in top-tier models like ChatGPT but also suffers from instability and susceptibility to reward hacking. We propose a novel approach, Supervised Iterative Learning from Human Feedback (SuperHF), which seeks to leverage the strengths of both methods. Our hypothesis is two-fold: that the reward model used in RLHF is critical for efficient data use and model generalization and that the use of Proximal Policy Optimization (PPO) in RLHF may not be necessary and could contribute to instability issues. SuperHF replaces PPO with a simple supervised loss and a Kullback-Leibler (KL) divergence prior. It creates its own training data by repeatedly sampling a batch of model outputs and filtering them through the reward model in an online learning regime. We then break down the reward optimization problem into three components: robustly optimizing the training rewards themselves, preventing reward hacking-exploitation of the reward model that degrades model performance-as measured by a novel METEOR similarity metric, and maintaining good performance on downstream evaluations. Our experimental results show SuperHF exceeds PPO-based RLHF on the training objective, easily and favorably trades off high reward with low reward hacking, improves downstream calibration, and performs the same on our GPT-4 based qualitative evaluation scheme all the while being significantly simpler to implement, highlighting SuperHF's potential as a competitive language model alignment technique.
ClearMark: Intuitive and Robust Model Watermarking via Transposed Model Training

Due to costly efforts during data acquisition and model training, Deep Neural Networks (DNNs) belong to the intellectual property of the model creator. Hence, unauthorized use, theft, or modification may lead to legal repercussions. Existing DNN watermarking methods for ownership proof are often non-intuitive, embed human-invisible marks, require trust in algorithmic assessment that lacks human-understandable attributes, and rely on rigid thresholds, making it susceptible to failure in cases of partial watermark erasure. This paper introduces ClearMark, the first DNN watermarking method designed for intuitive human assessment. ClearMark embeds visible watermarks, enabling human decision-making without rigid value thresholds while allowing technology-assisted evaluations. ClearMark defines a transposed model architecture allowing to use of the model in a backward fashion to interwove the watermark with the main task within all model parameters. Compared to existing watermarking methods, ClearMark produces visual watermarks that are easy for humans to understand without requiring complex verification algorithms or strict thresholds. The watermark is embedded within all model parameters and entangled with the main task, exhibiting superior robustness. It shows an 8,544-bit watermark capacity comparable to the strongest existing work. Crucially, ClearMark's effectiveness is model and dataset-agnostic, and resilient against adversarial model manipulations, as demonstrated in a comprehensive study performed with four datasets and seven architectures.
Particle detector models from path integrals of localized quantum fields

Using the Schwinger-Keldysh path integral, we draw a connection between localized quantum field theories and more commonly used models of local probes in Relativistic Quantum Information (RQI). By integrating over and then tracing out the inaccessible modes of the localized field being used as a probe, we show that, at leading order in perturbation theory, the dynamics of any finite number of modes of the probe field is exactly that of a finite number of harmonic-oscillator Unruh-DeWitt (UDW) detectors. The equivalence is valid for a rather general class of input states of the probe-target field system, as well as for any arbitrary number of modes included as detectors. The path integral also provides a closed-form expression which gives us a systematic way of obtaining the corrections to the UDW model at higher orders in perturbation theory due to the existence of the additional modes that have been traced out. This approach vindicates and extends a recently proposed bridge between detector-based and field-theory-based measurement frameworks for quantum field theory [T. R. Perche et al., Particle detectors from localized quantum field theories, Phys. Rev. D 109, 045013 (2024)], and also points to potential connections between particle detector models in RQI and other areas of physics where path integral methods are more commonplace -- in particular, the Wilsonian approach to the renormalization group and effective field theories.
IntenDD: A Unified Contrastive Learning Approach for Intent Detection and Discovery

Identifying intents from dialogue utterances forms an integral component of task-oriented dialogue systems. Intent-related tasks are typically formulated either as a classification task, where the utterances are classified into predefined categories or as a clustering task when new and previously unknown intent categories need to be discovered from these utterances. Further, the intent classification may be modeled in a multiclass (MC) or multilabel (ML) setup. While typically these tasks are modeled as separate tasks, we propose IntenDD, a unified approach leveraging a shared utterance encoding backbone. IntenDD uses an entirely unsupervised contrastive learning strategy for representation learning, where pseudo-labels for the unlabeled utterances are generated based on their lexical features. Additionally, we introduce a two-step post-processing setup for the classification tasks using modified adsorption. Here, first, the residuals in the training data are propagated followed by smoothing the labels both modeled in a transductive setting. Through extensive evaluations on various benchmark datasets, we find that our approach consistently outperforms competitive baselines across all three tasks. On average, IntenDD reports percentage improvements of 2.32%, 1.26%, and 1.52% in their respective metrics for few-shot MC, few-shot ML, and the intent discovery tasks respectively.
Adaptive importance sampling for heavy-tailed distributions via $\alpha$-divergence minimization

Adaptive importance sampling (AIS) algorithms are widely used to approximate expectations with respect to complicated target probability distributions. When the target has heavy tails, existing AIS algorithms can provide inconsistent estimators or exhibit slow convergence, as they often neglect the target's tail behaviour. To avoid this pitfall, we propose an AIS algorithm that approximates the target by Student-t proposal distributions. We adapt location and scale parameters by matching the escort moments - which are defined even for heavy-tailed distributions - of the target and the proposal. These updates minimize the $\alpha$-divergence between the target and the proposal, thereby connecting with variational inference. We then show that the $\alpha$-divergence can be approximated by a generalized notion of effective sample size and leverage this new perspective to adapt the tail parameter with Bayesian optimization. We demonstrate the efficacy of our approach through applications to synthetic targets and a Bayesian Student-t regression task on a real example with clinical trial data.
Performative Prediction: Past and Future

Predictions in the social world generally influence the target of prediction, a phenomenon known as performativity. Self-fulfilling and self-negating predictions are examples of performativity. Of fundamental importance to economics, finance, and the social sciences, the notion has been absent from the development of machine learning. In machine learning applications, performativity often surfaces as distribution shift. A predictive model deployed on a digital platform, for example, influences consumption and thereby changes the data-generating distribution. We survey the recently founded area of performative prediction that provides a definition and conceptual framework to study performativity in machine learning. A consequence of performative prediction is a natural equilibrium notion that gives rise to new optimization challenges. Another consequence is a distinction between learning and steering, two mechanisms at play in performative prediction. The notion of steering is in turn intimately related to questions of power in digital markets. We review the notion of performative power that gives an answer to the question how much a platform can steer participants through its predictions. We end on a discussion of future directions, such as the role that performativity plays in contesting algorithmic systems.
Over-the-air Federated Policy Gradient

In recent years, over-the-air aggregation has been widely considered in large-scale distributed learning, optimization, and sensing. In this paper, we propose the over-the-air federated policy gradient algorithm, where all agents simultaneously broadcast an analog signal carrying local information to a common wireless channel, and a central controller uses the received aggregated waveform to update the policy parameters. We investigate the effect of noise and channel distortion on the convergence of the proposed algorithm, and establish the complexities of communication and sampling for finding an $\epsilon$-approximate stationary point. Finally, we present some simulation results to show the effectiveness of the algorithm.