Miracle-dz/UniFAR-FTUs huggingface dataset:
Geodesic Multi-Modal Mixup for Robust Fine-Tuning

Pre-trained multi-modal models, such as CLIP, provide transferable embeddings and show promising results in diverse applications. However, the analysis of learned multi-modal embeddings is relatively unexplored, and the embedding transferability can be improved. In this work, we observe that CLIP holds separated embedding subspaces for two different modalities, and then we investigate it through the lens of uniformity-alignment to measure the quality of learned representation. Both theoretically and empirically, we show that CLIP retains poor uniformity and alignment even after fine-tuning. Such a lack of alignment and uniformity might restrict the transferability and robustness of embeddings. To this end, we devise a new fine-tuning method for robust representation equipping better alignment and uniformity. First, we propose a Geodesic Multi-Modal Mixup that mixes the embeddings of image and text to generate hard negative samples on the hypersphere. Then, we fine-tune the model on hard negatives as well as original negatives and positives with contrastive loss. Based on the theoretical analysis about hardness guarantee and limiting behavior, we justify the use of our method. Extensive experiments on retrieval, calibration, few- or zero-shot classification (under distribution shift), embedding arithmetic, and image captioning further show that our method provides transferable representations, enabling robust model adaptation on diverse tasks. Code: https://github.com/changdaeoh/multimodal-mixup
SKID: Self-Supervised Learning for Knee Injury Diagnosis from MRI Data

—In medical image analysis, the cost of acquiring high-quality data and their annotation by experts is a barrier in many medical applications. Most of the techniques used are based on supervised learning framework and need a large amount of annotated data to achieve satisfactory performance. As an alternative, in this paper, we propose a self-supervised learning (SSL) approach to learn the spatial anatomical representations from the frames of magnetic resonance (MR) video clips for the diagnosis of knee medical conditions. The pretext model learns meaningful spatial context-invariant representations. The downstream task in our paper is a class imbalanced multi-label classiﬁcation. Different experiments show that the features learnt by the pretext model provide competitive performance in the downstream task. Moreover, the efﬁciency and reliability of the proposed pretext model in learning representations of minority classes without applying any strategy towards imbalance in the dataset can be seen from the results. To the best of our knowledge, this work is the ﬁrst work of its kind in showing the effectiveness and reliability of self-supervised learning algorithms in class imbalanced multi-label classiﬁcation tasks on MR videos. The code for evaluation of the proposed work is available at https://github.com/sadimanna/skid . Impact Statement —Self-Supervised learning has emerged as one of the best tools for representation learning without using human-annotated data. In this paper, we design a self-supervised framework that can effectively classify different knee injuries from MR scans. Apart from handling this multi-label classiﬁ- cation task, we also highlighted the limitations of the existing geometrical transformation prediction-based pretext tasks, which will help us to understand the latent behaviours of the SSL algorithms.
Robust Covariance Matrix Estimation via Matrix Depth

Covariance matrix estimation is one of the most important problems in statistics. To accommodate the complexity of modern datasets, it is desired to have estimation procedures that not only can incorporate the structural assumptions of covariance matrices, but are also robust to outliers from arbitrary sources. In this paper, we define a new concept called matrix depth and we propose a robust covariance matrix estimator by maximizing the empirical depth function. The proposed estimator is shown to achieve minimax optimal rate under Huber's $\epsilon$-contamination model for estimating covariance/scatter matrices with various structures including bandedness and sparsity.
OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mapping

We introduce OpenEarthMap, a benchmark dataset, for global high-resolution land cover mapping. OpenEarth-Map consists of 2.2 million segments of 5000 aerial and satellite images covering 97 regions from 44 countries across 6 continents, with manually annotated 8-class land cover labels at a 0.25–0.5m ground sampling distance. Se-mantic segmentation models trained on the OpenEarth-Map generalize worldwide and can be used as off-the-shelf models in a variety of applications. We evaluate the performance of state-of-the-art methods for unsupervised domain adaptation and present challenging problem settings suitable for further technical development. We also investigate lightweight models using automated neural architecture search for limited computational resources and fast mapping. The dataset is available at https: //open-earth-map.org.
Hierarchical Multi-Interest Co-Network For Coarse-Grained Ranking

In this era of information explosion, a personalized recommendation system is convenient for users to get information they are interested in. To deal with billions of users and items, large-scale online recommendation services usually consist of three stages: candidate generation, coarse-grained ranking, and fine-grained ranking. The success of each stage depends on whether the model accurately captures the interests of users, which are usually hidden in users' behavior data. Previous research shows that users' interests are diverse, and one vector is not sufficient to capture users' different preferences. Therefore, many methods use multiple vectors to encode users' interests. However, there are two unsolved problems: (1) The similarity of different vectors in existing methods is too high, with too much redundant information. Consequently, the interests of users are not fully represented. (2) Existing methods model the long-term and short-term behaviors together, ignoring the differences between them. This paper proposes a Hierarchical Multi-Interest Co-Network (HCN) to capture users' diverse interests in the coarse-grained ranking stage. Specifically, we design a hierarchical multi-interest extraction layer to update users' diverse interest centers iteratively. The multiple embedded vectors obtained in this way contain more information and represent the interests of users better in various aspects. Furthermore, we develop a Co-Interest Network to integrate users' long-term and short-term interests. Experiments on several real-world datasets and one large-scale industrial dataset show that HCN effectively outperforms the state-of-the-art methods. We deploy HCN into a large-scale real world E-commerce system and achieve extra 2.5\% improvements on GMV (Gross Merchandise Value).
Scaling Laws for Reward Model Overoptimization

In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed"gold-standard"reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of-$n$ sampling. We find that this relationship follows a different functional form depending on the method of optimization, and that in both cases its coefficients scale smoothly with the number of reward model parameters. We also study the effect on this relationship of the size of the reward model dataset, the number of reward model and policy parameters, and the coefficient of the KL penalty added to the reward in the reinforcement learning setup. We explore the implications of these empirical results for theoretical considerations in AI alignment.
Enhanced vectors for top-k document retrieval in Question Answering

Modern day applications, especially information retrieval webapps that involve “search” as their use cases are gradually moving towards “answering” modules. Conversational chatbots which have been proved to be more engaging to users, use Question Answering as their core. Since, precise answering is computationally expensive, several approaches have been developed to prefetch the most relevant documents/passages from the database that contain the answer. We propose a different approach that retrieves the evidence documents efficiently and accurately, making sure that the relevant document for a given user query is not missed. We do so by assigning each document (or passage in our case), a unique identifier and using them to create dense vectors which can be efficiently indexed. More precisely, we use the identifier to predict randomly sampled context window words of the relevant question corresponding to the passage along with the words of passage itself. This naturally embeds the passage identifier into the vector space in such a way that the embedding is closer to the question without compromising he information content. This approach enables efficient creation of real-time query vectors in ~4 milliseconds.
1/f noise and anomalous scaling in Lévy noise-driven on–off intermittency

On–off intermittency occurs in nonequilibrium physical systems close to bifurcation points, and is characterised by an aperiodic switching between a large-amplitude ‘on’ state and a small-amplitude ‘off’ state. Lévy on–off intermittency is a recently introduced generalisation of on–off intermittency to multiplicative Lévy noise, which depends on a stability parameter α and a skewness parameter β. Here, we derive two novel results on Lévy on–off intermittency by leveraging known exact results on the first-passage time statistics of Lévy flights. First, we compute anomalous critical exponents explicitly as a function of arbitrary Lévy noise parameters (α,β) for the first time, by a heuristic method, extending previous results. The predictions are verified using numerical solutions of the fractional Fokker–Planck equation. Second, we derive the power spectrum S(f) of Lévy on–off intermittency, and show that it displays a power law S(f)∝fκ at low frequencies f, where κ∈(−1,0) depends on the noise parameters α,β . An explicit expression for κ is obtained in terms of (α,β) . The predictions are verified using long time series realisations of Lévy on–off intermittency. Our findings help shed light on instabilities subject to non-equilibrium, power-law-distributed fluctuations, emphasizing that their properties can differ starkly from the case of Gaussian fluctuations.
Robustness of Demonstration-based Learning Under Limited Data Scenario

Demonstration-based learning has shown great potential in stimulating pretrained language models’ ability under limited data scenario. Simply augmenting the input with some demonstrations can significantly improve performance on few-shot NER. However, why such demonstrations are beneficial for the learning process remains unclear since there is no explicit alignment between the demonstrations and the predictions. In this paper, we design pathological demonstrations by gradually removing intuitively useful information from the standard ones to take a deep dive of the robustness of demonstration-based sequence labeling and show that (1) demonstrations composed of random tokens still make the model a better few-shot learner; (2) the length of random demonstrations and the relevance of random tokens are the main factors affecting the performance; (3) demonstrations increase the confidence of model predictions on captured superficial patterns. We have publicly released our code at https://github.com/SALT-NLP/RobustDemo.
Stability for the Helmholtz equation in deterministic and random periodic structures

Stability results for the Helmholtz equations in both deterministic and random periodic structures are proved in this paper. Under the assumption of excluding resonances, by a variational method and Fourier analysis in the energy space, the stability estimate for the Helmholtz equation in a deterministic periodic structure is established. For the stochastic case, by introducing a variable transform, the variational formulation of the scattering problem in a random domain is reduced to that in a definite domain with random medium. Combining the stability result for the deteministic case with regularity and stochastic regularity of the scattering surface, Pettis measurability theorem and Bochner's Theorem further yield the stability result for the scattering problem by random periodic structures. Both stability estimates are explicit with respect to the wavenumber.
Proximal Learning With Opponent-Learning Awareness

Learning With Opponent-Learning Awareness (LOLA) (Foerster et al. [2018a]) is a multi-agent reinforcement learning algorithm that typically learns reciprocity-based cooperation in partially competitive environments. However, LOLA often fails to learn such behaviour on more complex policy spaces parameterized by neural networks, partly because the update rule is sensitive to the policy parameterization. This problem is especially pronounced in the opponent modeling setting, where the opponent's policy is unknown and must be inferred from observations; in such settings, LOLA is ill-specified because behaviorally equivalent opponent policies can result in non-equivalent updates. To address this shortcoming, we reinterpret LOLA as approximating a proximal operator, and then derive a new algorithm, proximal LOLA (POLA), which uses the proximal formulation directly. Unlike LOLA, the POLA updates are parameterization invariant, in the sense that when the proximal objective has a unique optimum, behaviorally equivalent policies result in behaviorally equivalent updates. We then present practical approximations to the ideal POLA update, which we evaluate in several partially competitive environments with function approximation and opponent modeling. This empirically demonstrates that POLA achieves reciprocity-based cooperation more reliably than LOLA.
On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardness

Generalization in Reinforcement Learning (RL) aims to learn an agent during training that generalizes to the target environment. This paper studies RL generalization from a theoretical aspect: how much can we expect pre-training over training environments to be helpful? When the interaction with the target environment is not allowed, we certify that the best we can obtain is a near-optimal policy in an average sense, and we design an algorithm that achieves this goal. Furthermore, when the agent is allowed to interact with the target environment, we give a surprising result showing that asymptotically, the improvement from pre-training is at most a constant factor. On the other hand, in the non-asymptotic regime, we design an efficient algorithm and prove a distribution-based regret bound in the target environment that is independent of the state-action space.
Pure spinor formulation of the superstring and its applications

The pure spinor formalism for the superstring has the advantage over the more conventional Ramond-Neveu-Schwarz formalism of being manifestly spacetime supersymmetric, which simpliﬁes the computation of multiparticle and multiloop amplitudes and allows the description of Ramond-Ramond backgrounds. In addition to the worldsheet variables of the Green-Schwarz-Siegel action, the pure spinor formalism includes bosonic ghost variables which are constrained spacetime spinors and are needed for covariant quantization using a nilpotent BRST operator. In this review, several applications of the formalism are described including the explicit computation in D=10 superspace of the general disk amplitude with an arbitrary number of external massless states, genus one amplitudes with up to seven external states, genus two amplitudes with up to ﬁve external states, and the low-energy limit of the genus three amplitude with up to four external states. The pure spinor formalism has also been used to covariantly quantize the superstring in an AdS 5 × S 5 background and might be useful for proving the AdS-CFT correspondence in the limit of small AdS radius. This is an overview written for the “Handbook of Quantum Gravity”, eds. C. Bambi, L. Modesto and I. Shapiro.
On Representing Mixed-Integer Linear Programs by Graph Neural Networks

While Mixed-integer linear programming (MILP) is NP-hard in general, practical MILP has received roughly 100--fold speedup in the past twenty years. Still, many classes of MILPs quickly become unsolvable as their sizes increase, motivating researchers to seek new acceleration techniques for MILPs. With deep learning, they have obtained strong empirical results, and many results were obtained by applying graph neural networks (GNNs) to making decisions in various stages of MILP solution processes. This work discovers a fundamental limitation: there exist feasible and infeasible MILPs that all GNNs will, however, treat equally, indicating GNN's lacking power to express general MILPs. Then, we show that, by restricting the MILPs to unfoldable ones or by adding random features, there exist GNNs that can reliably predict MILP feasibility, optimal objective values, and optimal solutions up to prescribed precision. We conducted small-scale numerical experiments to validate our theoretical findings.
Catch-22s of reservoir computing

Reservoir Computing (RC) is a simple and efficient model-free framework for forecasting the behavior of nonlinear dynamical systems from data. Here, we show that there exist commonly-studied systems for which leading RC frameworks struggle to learn the dynamics unless key information about the underlying system is already known. We focus on the important problem of basin prediction -- determining which attractor a system will converge to from its initial conditions. First, we show that the predictions of standard RC models (echo state networks) depend critically on warm-up time, requiring a warm-up trajectory containing almost the entire transient in order to identify the correct attractor. Accordingly, we turn to Next-Generation Reservoir Computing (NGRC), an attractive variant of RC that requires negligible warm-up time. By incorporating the exact nonlinearities in the original equations, we show that NGRC can accurately reconstruct intricate and high-dimensional basins of attraction, even with sparse training data (e.g., a single transient trajectory). Yet, a tiny uncertainty in the exact nonlinearity can render prediction accuracy no better than chance. Our results highlight the challenges faced by data-driven methods in learning the dynamics of multistable systems and suggest potential avenues to make these approaches more robust.
A normal mode treatment of semi-diurnal body tides on an aspherical, rotating and anelastic Earth

SUMMARY Normal mode treatments of the Earth’s body tide response were developed in the 1980s to account for the effects of Earth rotation, ellipticity, anelasticity and resonant excitation within the diurnal band. Recent space-geodetic measurements of the Earth’s crustal displacement in response to luni-solar tidal forcings have revealed geographical variations that are indicative of aspherical deep mantle structure, thus providing a novel data set for constraining deep mantle elastic and density structure. In light of this, we make use of advances in seismic free oscillation literature to develop a new, generalized normal mode theory for the tidal responsewithinthesemi-diurnalandlong-periodtidalband.Ourtheoryinvolvesaperturbation method that permits an efficient calculation of the impact of aspherical structure on the tidal response. In addition, we introduce a normal mode treatment of anelasticity that is distinct from both earlier work in body tides and the approach adopted in free oscillation seismology. We present several simple numerical applications of the new theory. First, we compute the tidal response of a spherically symmetric, non-rotating, elastic and isotropic Earth model and demonstrate that our predictions match those based on standard Love number theory. Second, we compute perturbations to this response associated with mantle anelasticity and demonstrate that the usual set of seismic modes adopted for this purpose must be augmented by a family of relaxation modes to accurately capture the full effect of anelasticity on the body tide response. Finally, we explore aspherical effects including rotation and we benchmark results from several illustrative case studies of aspherical Earth structure against independent finitevolume numerical calculations of the semi-diurnal body tide response. These tests confirm the accuracy of the normal mode methodology to at least the level of numerical error in the finite-volume predictions. They also demonstrate that full coupling of normal modes, rather than group coupling, is necessary for accurate predictions of the body tide response.
Optimized Telecloning Circuits: Theory and Practice of Nine NISQ Clones

Although perfect copying of an unknown quantum state is not possible, approximate cloning is possible in quantum mechanics. Quantum telecloning is a variant of approximate quantum cloning which uses quantum teleportation to allow for the use of classical communication to create physically separate clones of a quantum state. We present results of a of 1 →9 universal, symmetric, optimal quantum telecloning implementation on a cloud accessible quantum computer - the Quantinuum H1-1 device. The H1-1 device allows direct creation of the telecloning protocol due to real time classical if-statements that are conditional on the mid-circuit measurement outcome of a Bell measurement. In this implementation, we also provide an improvement over previous work for the circuit model description of quantum telecloning, which reduces the required gate depth and gate count for an all-to-all connectivity. The demonstration of creating 9 approximate clones on a quantum processor is the largest number of clones that has been generated, telecloning or otherwise.
A Unified View of Masked Image Modeling

Masked image modeling has demonstrated great potential to eliminate the label-hungry problem of training large-scale vision Transformers, achieving impressive performance on various downstream tasks. In this work, we propose a unified view of masked image modeling after revisiting existing methods. Under the unified view, we introduce a simple yet effective method, termed as MaskDistill, which reconstructs normalized semantic features from teacher models at the masked positions, conditioning on corrupted input images. Experimental results on image classification and semantic segmentation show that MaskDistill achieves comparable or superior performance than state-of-the-art methods. When using the huge vision Transformer and pretraining 300 epochs, MaskDistill obtains 88.3% fine-tuning top-1 accuracy on ImageNet-1k (224 size) and 58.8% semantic segmentation mIoU metric on ADE20k (512 size). The code and pretrained models will be available at https://aka.ms/unimim.