Miracle-dz/UniFAR-FTUs huggingface dataset:
Hebbian and Gradient-based Plasticity Enables Robust Memory and Rapid Learning in RNNs

Rapidly learning from ongoing experiences and remembering past events with a flexible memory system are two core capacities of biological intelligence. While the underlying neural mechanisms are not fully understood, various evidence supports that synaptic plasticity plays a critical role in memory formation and fast learning. Inspired by these results, we equip Recurrent Neural Networks (RNNs) with plasticity rules to enable them to adapt their parameters according to ongoing experiences. In addition to the traditional local Hebbian plasticity, we propose a global, gradient-based plasticity rule, which allows the model to evolve towards its self-determined target. Our models show promising results on sequential and associative memory tasks, illustrating their ability to robustly form and retain memories. In the meantime, these models can cope with many challenging few-shot learning problems. Comparing different plasticity rules under the same framework shows that Hebbian plasticity is well-suited for several memory and associative learning tasks; however, it is outperformed by gradient-based plasticity on few-shot regression tasks which require the model to infer the underlying mapping. Code is available at https://github.com/yuvenduan/PlasticRNNs.
Easy Learning from Label Proportions

We consider the problem of Learning from Label Proportions (LLP), a weakly supervised classification setup where instances are grouped into"bags", and only the frequency of class labels at each bag is available. Albeit, the objective of the learner is to achieve low task loss at an individual instance level. Here we propose Easyllp: a flexible and simple-to-implement debiasing approach based on aggregate labels, which operates on arbitrary loss functions. Our technique allows us to accurately estimate the expected loss of an arbitrary model at an individual level. We showcase the flexibility of our approach by applying it to popular learning frameworks, like Empirical Risk Minimization (ERM) and Stochastic Gradient Descent (SGD) with provable guarantees on instance level performance. More concretely, we exhibit a variance reduction technique that makes the quality of LLP learning deteriorate only by a factor of k (k being bag size) in both ERM and SGD setups, as compared to full supervision. Finally, we validate our theoretical results on multiple datasets demonstrating our algorithm performs as well or better than previous LLP approaches in spite of its simplicity.
Delving Deep into Simplicity Bias for Long-Tailed Image Recognition

Simplicity Bias (SB) is a phenomenon that deep neural networks tend to rely favorably on simpler predictive patterns but ignore some complex features when applied to supervised discriminative tasks. In this work, we investigate SB in long-tailed image recognition and find the tail classes suffer more severely from SB, which harms the generalization performance of such underrepresented classes. We empirically report that self-supervised learning (SSL) can mitigate SB and perform in complementary to the supervised counterpart by enriching the features extracted from tail samples and consequently taking better advantage of such rare samples. However, standard SSL methods are designed without explicitly considering the inherent data distribution in terms of classes and may not be optimal for long-tailed distributed data. To address this limitation, we propose a novel SSL method tailored to imbalanced data. It leverages SSL by triple diverse levels, i.e., holistic-, partial-, and augmented-level, to enhance the learning of predictive complex patterns, which provides the potential to overcome the severe SB on tail data. Both quantitative and qualitative experimental results on five long-tailed benchmark datasets show our method can effectively mitigate SB and significantly outperform the competing state-of-the-arts.
Another remark on the global regularity issue of the Hall-magnetohydrodynamics system

We discover cancellations upon $H^{2}(\mathbb{R}^{n})$-estimate of the Hall term for $n \in \{2,3\}$. As its consequence, first, we derive a regularity criterion for the 3-dimensional Hall-magnetohydrodynamics system in terms of only horizontal components of velocity and magnetic fields. Second, we prove the global regularity of the $2\frac{1}{2}$-dimensional electron magnetohydrodynamics system with magnetic diffusion $(-\Delta)^{\frac{3}{2}} (b_{1}, b_{2}, 0) + (-\Delta)^{\alpha} (0, 0, b_{3})$ for $\alpha>\frac{1}{2}$. Lastly, we extend this result to the $2\frac{1}{2}$-dimensional Hall-magnetohydrodynamics system with $-\Delta u$ replaced by $(-\Delta)^{\alpha} (u_{1}, u_{2}, 0) -\Delta (0, 0, u_{3})$ for $\alpha>\frac{1}{2}$. The sum of the derivatives in diffusion that our global regularity result requires is $11+ \epsilon$ for any $\epsilon>0$ while the analogous sum for the classical $2\frac{1}{2}$-dimensional Hall-magnetohydrodynamics system is 12 considering $-\Delta u$ and $-\Delta b$.
Single Cells Are Spatial Tokens: Transformers for Spatial Transcriptomic Data Imputation

Spatially resolved transcriptomics brings exciting breakthroughs to single-cell analysis by providing physical locations along with gene expression. However, as a cost of the extremely high spatial resolution, the cellular level spatial transcriptomic data suffer significantly from missing values. While a standard solution is to perform imputation on the missing values, most existing methods either overlook spatial information or only incorporate localized spatial context without the ability to capture long-range spatial information. Using multi-head self-attention mechanisms and positional encoding, transformer models can readily grasp the relationship between tokens and encode location information. In this paper, by treating single cells as spatial tokens, we study how to leverage transformers to facilitate spatial tanscriptomics imputation. In particular, investigate the following two key questions: (1) $\textit{how to encode spatial information of cells in transformers}$, and (2) $\textit{ how to train a transformer for transcriptomic imputation}$. By answering these two questions, we present a transformer-based imputation framework, SpaFormer, for cellular-level spatial transcriptomic data. Extensive experiments demonstrate that SpaFormer outperforms existing state-of-the-art imputation algorithms on three large-scale datasets while maintaining superior computational efficiency.
Two Losses Are Better Than One: Faster Optimization Using a Cheaper Proxy

We present an algorithm for minimizing an objective with hard-to-compute gradients by using a related, easier-to-access function as a proxy. Our algorithm is based on approximate proximal point iterations on the proxy combined with relatively few stochastic gradients from the objective. When the difference between the objective and the proxy is $\delta$-smooth, our algorithm guarantees convergence at a rate matching stochastic gradient descent on a $\delta$-smooth objective, which can lead to substantially better sample efficiency. Our algorithm has many potential applications in machine learning, and provides a principled means of leveraging synthetic data, physics simulators, mixed public and private data, and more.
Federated Variational Inference Methods for Structured Latent Variable Models

Federated learning methods enable model training across distributed data sources without data leaving their original locations and have gained increasing interest in various fields. However, existing approaches are limited, excluding many structured probabilistic models. We present a general and elegant solution based on structured variational inference, widely used in Bayesian machine learning, adapted for the federated setting. Additionally, we provide a communication-efficient variant analogous to the canonical FedAvg algorithm. The proposed algorithms' effectiveness is demonstrated, and their performance is compared with hierarchical Bayesian neural networks and topic models.
Computing Non-Archimedean Cylinder Counts in Blowups of Toric Surfaces

Counts of holomorphic disks are at the heart of the SYZ approach to mirror symmetry. In the non archimedean framework, these counts are expressed as counts of analytic cylinders. In simple cases, such as cluster varieties, these counts can be extracted from a combinatorial algorithm encoded by a scattering diagram. The relevant scattering diagram encodes information about infinitesimal analytic cylinders. In this paper, we give a formula relating counts of a restricted class of cylinders on affine log Calabi Yau surfaces to counts on a single blowup of a toric surface, using non archimedean Gromov Witten theory and a degeneration procedure parametrized by tropical data. This closed form formula constrasts with the scattering diagram approach.
Evaluating Self-Supervised Learning via Risk Decomposition

Self-supervised learning (SSL) pipelines differ in many design choices such as the architecture, augmentations, or pretraining data. Yet SSL is typically evaluated using a single metric: linear probing on ImageNet. This does not provide much insight into why or when a model is better, now how to improve it. To address this, we propose an SSL risk decomposition, which generalizes the classical supervised approximation-estimation decomposition by considering errors arising from the representation learning step. Our decomposition consists of four error components: approximation, representation usability, probe generalization, and encoder generalization. We provide efficient estimators for each component and use them to analyze the effect of 30 design choices on 169 SSL vision models evaluated on ImageNet. Our analysis gives valuable insights for designing and using SSL models. For example, it highlights the main sources of error and shows how to improve SSL in specific settings (full- vs few-shot) by trading off error components. All results and pretrained models are at https://github.com/YannDubs/SSL-Risk-Decomposition.
Generalization Bounds of Nonconvex-(Strongly)-Concave Stochastic Minimax Optimization

This paper takes an initial step to systematically investigate the generalization bounds of algorithms for solving nonconvex-(strongly)-concave (NC-SC/NC-C) stochastic minimax optimization measured by the stationarity of primal functions. We first establish algorithm-agnostic generalization bounds via uniform convergence between the empirical minimax problem and the population minimax problem. The sample complexities for achieving $\epsilon$-generalization are $\tilde{\mathcal{O}}(d\kappa^2\epsilon^{-2})$ and $\tilde{\mathcal{O}}(d\epsilon^{-4})$ for NC-SC and NC-C settings, respectively, where $d$ is the dimension and $\kappa$ is the condition number. We further study the algorithm-dependent generalization bounds via stability arguments of algorithms. In particular, we introduce a novel stability notion for minimax problems and build a connection between generalization bounds and the stability notion. As a result, we establish algorithm-dependent generalization bounds for stochastic gradient descent ascent (SGDA) algorithm and the more general sampling-determined algorithms.
Continual Learning of Language Models

For each domain i , we compare its importance vector with the importance vector of every other domain, and then average the cosine similarities to get the value for domain i . We get 0.92 for Restaurant, the same 0.91 for ACL, AI, and Phone, 0.89 for PubMed and 0.92 for Camera. We see that different domains give similar importance values, which indirectly shows that our proxy can approximately identify the common general knowledge.
Using t-distributed stochastic neighbor embedding for visualization and segmentation of 3D point clouds of plants

In this work, the use of t-SNE is proposed to embed 3D point clouds of plants into 2D space for plant characterization. It is demonstrated that t-SNE operates as a practical tool to flatten and visualize a complete 3D plant model in 2D space. The perplexity parameter of t-SNE allows 2D rendering of plant structures at various organizational levels. Aside from the promise of serving as a visualization tool for plant scientists, t-SNE also provides a gateway for processing 3D point clouds of plants using their embedded counterparts in 2D. In this paper, simple methods were proposed to perform semantic segmentation and instance segmentation via grouping the embedded 2D points. The evaluation of these methods on a public 3D plant data set conveys the potential of t-SNE for enabling of 2D implementation of various steps involved in automatic 3D phenotyping pipelines.
Planted Bipartite Graph Detection

We consider the task of detecting a hidden bipartite subgraph in a given random graph. This is formulated as a hypothesis testing problem, under the null hypothesis, the graph is a realization of an Erd\H{o}s-R\'{e}nyi random graph over $n$ vertices with edge density $q$. Under the alternative, there exists a planted $k_{\mathsf{R}} \times k_{\mathsf{L}}$ bipartite subgraph with edge density $p>q$. We characterize the statistical and computational barriers for this problem. Specifically, we derive information-theoretic lower bounds, and design and analyze optimal algorithms matching those bounds, in both the dense regime, where $p,q = \Theta\left(1\right)$, and the sparse regime where $p,q = \Theta\left(n^{-\alpha}\right), \alpha \in \left(0,2\right]$. We also consider the problem of testing in polynomial-time. As is customary in similar structured high-dimensional problems, our model undergoes an"easy-hard-impossible"phase transition and computational constraints penalize the statistical performance. To provide an evidence for this statistical computational gap, we prove computational lower bounds based on the low-degree conjecture, and show that the class of low-degree polynomials algorithms fail in the conjecturally hard region.
Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspective

We study the problem of optimizing a recommender system for outcomes that occur over several weeks or months. We begin by drawing on reinforcement learning to formulate a comprehensive model of users' recurring relationships with a recommender system. Measurement, attribution, and coordination challenges complicate algorithm design. We describe careful modeling -- including a new representation of user state and key conditional independence assumptions -- which overcomes these challenges and leads to simple, testable recommender system prototypes. We apply our approach to a podcast recommender system that makes personalized recommendations to hundreds of millions of listeners. A/B tests demonstrate that purposefully optimizing for long-term outcomes leads to large performance gains over conventional approaches that optimize for short-term proxies.
APAM: Adaptive Pre-Training and Adaptive Meta Learning in Language Model for Noisy Labels and Long-Tailed Learning

Practical natural language processing (NLP) tasks often exhibit long-tailed distributions accompanied by noisy labels, posing significant challenges to the generalization and robustness of complex models such as Deep Neural Networks (DNNs). Traditional resampling techniques like oversampling or undersampling, while commonly employed, can easily lead to overfitting. A growing trend involves leveraging small amounts of metadata to learn data weights, alongside the demonstrated benefits of self-supervised pre-training, particularly for under-represented data. In this work, we propose a general framework that addresses both the long-tail and noisy label issues. Our model is adaptively tailored to the problem domain using a contrastive learning approach. The re-weighting module, a feed-forward network, learns explicit weighting functions and adjusts weights based on metadata. Additionally, our framework modifies the weights of terms in the loss function through a combination of polynomial expansion of the cross-entropy loss and focal loss. Extensive experiments consistently demonstrate the superior performance of our proposed framework compared to baseline methods. Finally, our comprehensive sensitivity analysis underscores the efficacy of the proposed framework in handling long-tailed problems and mitigating the adverse effects of noisy labels.
Volume complexity of dS bubbles

In the framework of the static patch approach to de Sitter holography introduced in [arXiv:2109.14104], the growth of holographic complexity has a hyperfast behaviour, which leads to a divergence in a finite time. This is very different from the AdS spacetime, where instead the complexity rate asymptotically reaches a constant value. We study holographic volume complexity in a class of asymptotically AdS geometries which include de Sitter bubbles in their interior. With the exception of the static bubble case, the complexity obtained from the volume of the smooth extremal surfaces which are anchored just to the AdS boundary has a similar behaviour to the AdS case, because it asymptotically grows linearly with time. The static bubble configuration has a zero complexity rate and corresponds to a discontinuous behaviour, which resembles a first order phase transition. If instead we consider extremal surfaces which are anchored at both the AdS boundary and the de Sitter stretched horizon, we find that complexity growth is hyperfast, as in the de Sitter case.
Cache-Aided Communication Schemes via Combinatorial Designs and Their q-Analogs

We consider the standard broadcast setup with a single server broadcasting information to a number of clients, each of which contains local storage (called cache) of some size, which can store some parts of the available files at the server. The centralized coded caching framework, consists of a caching phase and a delivery phase, both of which are carefully designed in order to use the cache and the channel together optimally. In prior literature, various combinatorial structures have been used to construct coded caching schemes. One of the chief drawbacks of many of these existing constructions is the large subpacketization level, which denotes the number of times a file should be split for the schemes to provide coding gain. In this work, using a new binary matrix model, we present several novel constructions for coded caching based on the various types of combinatorial designs and their $q$ -analogs, which are also called subspace designs. While most of the schemes constructed in this work (based on existing designs) have a high cache requirement, they provide a rate that is either constant or decreasing, and moreover require competitively small levels of subpacketization, which is an extremely important feature in practical applications of coded caching. We also apply our constructions to the distributed computing framework of MapReduce, which consists of three phases, the Map phase, the Shuffle phase and the Reduce phase. Using our binary matrix framework, we present a new simple generic coded data shuffling scheme. Employing our designs-based constructions in conjunction with this new shuffling scheme, we obtain new coded computing schemes which have low file complexity, with marginally higher communication load compared to the optimal scheme for equivalent parameters. We show that our schemes can neatly extend to the scenario with full and partial stragglers also.
Data Selection for Language Models via Importance Resampling

Selecting a suitable pretraining dataset is crucial for both general-domain (e.g., GPT-3) and domain-specific (e.g., Codex) language models (LMs). We formalize this problem as selecting a subset of a large raw unlabeled dataset to match a desired target distribution given unlabeled target samples. Due to the scale and dimensionality of the raw text data, existing methods use simple heuristics or require human experts to manually curate data. Instead, we extend the classic importance resampling approach used in low-dimensions for LM data selection. We propose Data Selection with Importance Resampling (DSIR), an efficient and scalable framework that estimates importance weights in a reduced feature space for tractability and selects data with importance resampling according to these weights. We instantiate the DSIR framework with hashed n-gram features for efficiency, enabling the selection of 100M documents from the full Pile dataset in 4.5 hours. To measure whether hashed n-gram features preserve the aspects of the data that are relevant to the target, we define KL reduction, a data metric that measures the proximity between the selected pretraining data and the target on some feature space. Across 8 data selection methods (including expert selection), KL reduction on hashed n-gram features highly correlates with average downstream accuracy (r=0.82). When selecting data for continued pretraining on a specific domain, DSIR performs comparably to expert curation across 8 target distributions. When pretraining general-domain models (target is Wikipedia and books), DSIR improves over random selection and heuristic filtering baselines by 2-2.5% on the GLUE benchmark. Code is available at https://github.com/p-lambda/dsir.
On oscillating sticky Brownian motion

Starting with a Brownian motion, we define and study a novel diffusion process by combining stickiness and oscillation properties. The associated stochastic differential equation, resolvent and semigroup are provided. Also the trivariate density of position, local time and occupation time of this diffusion is obtained explicitly. Furthermore, we give a construction of two Brownian motions with drift and scaling whose difference is an oscillating sticky Brownian motion, up to a multiplicative constant.
An Information Bottleneck Approach to Secrecy-Preserving Gaussian Mixture Classification

The information bottleneck (IB) approach, initially introduced by [1] to assess the “compression–relevance” tradeoff for a remote source coding problem in communication, quickly gains popularity recently in its application to modern machine learning (ML). Unlike the use of most – if not all – IB in the literature, either for the analysis of, say deep neural networks, or as an optimization objective, in this paper, we propose to address the secrecy issue in ML, by considering the fundamental model of Gaussian mixture classiﬁcation. We derive, for the ﬁrst time, closed-form achievable bounds for the IB problem under the above setting, and provide precise characterization of the underlying “performance–secrecy” tradeoff. Experiments on both synthetic and real-world data are performed to conﬁrm the satisfactory performance of the proposed scheme
Learning Translation Quality Evaluation on Low Resource Languages from Large Language Models

Learned metrics such as BLEURT have in recent years become widely employed to evaluate the quality of machine translation systems. Training such metrics requires data which can be expensive and difficult to acquire, particularly for lower-resource languages. We show how knowledge can be distilled from Large Language Models (LLMs) to improve upon such learned metrics without requiring human annotators, by creating synthetic datasets which can be mixed into existing datasets, requiring only a corpus of text in the target language. We show that the performance of a BLEURT-like model on lower resource languages can be improved in this way.