Miracle-dz/UniFAR-FTUs huggingface dataset:
Towards understanding how momentum improves generalization in deep learning

Stochastic gradient descent (SGD) with momentum is widely used for training modern deep learning architectures. While it is well-understood that using momentum can lead to faster convergence rate in various settings, it has also been observed that momentum yields higher generalization. Prior work argue that momentum stabilizes the SGD noise during training and this leads to higher generalization. In this paper, we adopt another perspective and first empirically show that gradient descent with momentum (GD+M) significantly improves generalization compared to gradient descent (GD) in some deep learning problems. From this observation, we formally study how momentum improves generalization. We devise a binary classification setting where a one-hidden layer (over-parameterized) convolutional neural network trained with GD+M provably generalizes better than the same network trained with GD, when both algorithms are similarly initialized. The key insight in our analysis is that momentum is beneficial in datasets where the examples share some feature but differ in their margin. Contrary to GD that memorizes the small margin data, GD+M still learns the feature in these data thanks to its historical gradients. Lastly, we empirically validate our theoretical findings.
Learning Dynamics and Generalization in Deep Reinforcement Learning

Solving a reinforcement learning (RL) problem poses two competing challenges: fitting a potentially discontinuous value function, and generalizing well to new observations. In this paper, we analyze the learning dynamics of temporal difference algorithms to gain novel insight into the tension between these two objectives. We show theoretically that temporal difference learning encourages agents to fit non-smooth components of the value function early in training, and at the same time induces the second-order effect of discouraging generalization. We corroborate these findings in deep RL agents trained on a range of environments, finding that neural networks trained using temporal difference algorithms on dense reward tasks exhibit weaker generalization between states than randomly initialized networks and networks trained with policy gradient methods. Finally, we investigate how post-training policy distillation may avoid this pit-fall, and show that this approach improves generalization to novel environments in the ProcGen suite and improves robustness to input perturbations.
Certified Neural Network Watermarks with Randomized Smoothing

Watermarking is a commonly used strategy to protect creators' rights to digital images, videos and audio. Recently, watermarking methods have been extended to deep learning models -- in principle, the watermark should be preserved when an adversary tries to copy the model. However, in practice, watermarks can often be removed by an intelligent adversary. Several papers have proposed watermarking methods that claim to be empirically resistant to different types of removal attacks, but these new techniques often fail in the face of new or better-tuned adversaries. In this paper, we propose a certifiable watermarking method. Using the randomized smoothing technique proposed in Chiang et al., we show that our watermark is guaranteed to be unremovable unless the model parameters are changed by more than a certain l2 threshold. In addition to being certifiable, our watermark is also empirically more robust compared to previous watermarking methods. Our experiments can be reproduced with code at https://github.com/arpitbansal297/Certified_Watermarks
ASAP.SGD: Instance-based Adaptiveness to Staleness in Asynchronous SGD

Concurrent algorithmic implementations of Stochastic Gradient Descent (SGD) give rise to critical questions for compute-intensive Machine Learning (ML). Asynchrony implies speedup in some contexts, and challenges in others, as stale updates may lead to slower, or non-converging executions. While previous works showed asynchrony-adaptiveness can improve stability and speedup by reducing the step size for stale updates according to static rules, there is no one-size-fits-all adaptation rule, since the optimal strategy depends on several factors. We introduce (i) ASAP . SGD , an analytical framework capturing necessary and desired properties of staleness-adaptive step size functions and (ii) TAIL - τ , a method for utilizing key properties of the execution instance , generating a tailored strategy that not only dampens the impact of stale updates, but also leverages fresh ones. We recover convergence bounds for adaptiveness functions satisfying the ASAP . SGD conditions, for general, convex and non-convex problems, and establish novel bounds for ones satisfying the Polyak-Lojasiewicz property. We evaluate TAIL - τ with representative AsyncSGD concurrent algorithms, for Deep Learning problems, showing TAIL - τ is a vital complement to AsyncSGD , with (i) persistent speedup in wall-clock convergence time in the parallelism spectrum, (ii) considerably lower risk of non-convergence, as well as (iii) precision levels for which original SGD implementations fail.
Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning

In view of its power in extracting feature representation, contrastive self-supervised learning has been successfully integrated into the practice of (deep) reinforcement learning (RL), leading to efficient policy learning in various applications. Despite its tremendous empirical successes, the understanding of contrastive learning for RL remains elusive. To narrow such a gap, we study how RL can be empowered by contrastive learning in a class of Markov decision processes (MDPs) and Markov games (MGs) with low-rank transitions. For both models, we propose to extract the correct feature representations of the low-rank model by minimizing a contrastive loss. Moreover, under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs. We further theoretically prove that our algorithm recovers the true representations and simultaneously achieves sample efficiency in learning the optimal policy and Nash equilibrium in MDPs and MGs. We also provide empirical studies to demonstrate the efficacy of the UCB-based contrastive learning method for RL. To the best of our knowledge, we provide the first provably efficient online RL algorithm that incorporates contrastive learning for representation learning. Our codes are available at https://github.com/Baichenjia/Contrastive-UCB.
Adversarial Robustness against Multiple and Single lp-Threat Models via Quick Fine-Tuning of Robust Classifiers

A major drawback of adversarially robust models, in particular for large scale datasets like ImageNet, is the extremely long training time compared to standard ones. Moreover, models should be robust not only to one $l_p$-threat model but ideally to all of them. In this paper we propose Extreme norm Adversarial Training (E-AT) for multiple-norm robustness which is based on geometric properties of $l_p$-balls. E-AT costs up to three times less than other adversarial training methods for multiple-norm robustness. Using E-AT we show that for ImageNet a single epoch and for CIFAR-10 three epochs are sufficient to turn any $l_p$-robust model into a multiple-norm robust model. In this way we get the first multiple-norm robust model for ImageNet and boost the state-of-the-art for multiple-norm robustness to more than $51\%$ on CIFAR-10. Finally, we study the general transfer via fine-tuning of adversarial robustness between different individual $l_p$-threat models and improve the previous SOTA $l_1$-robustness on both CIFAR-10 and ImageNet. Extensive experiments show that our scheme works across datasets and architectures including vision transformers.
Stochastic Rising Bandits

This paper is in the ﬁeld of stochastic Multi-Armed Bandits (MABs), i.e., those sequential selection techniques able to learn online using only the feedback given by the chosen option (a.k.a. arm). We study a particular case of the rested and restless bandits in which the arms’ expected payoff is monotonically non-decreasing. This characteristic allows designing speciﬁcally crafted algorithms that exploit the regularity of the payoffs to provide tight regret bounds. We design an algorithm for the rested case ( R-ed-UCB ) and one for the restless case ( R-less-UCB ), providing a regret bound depending on the properties of the instance and, under certain circumstances, of r O p T 23 q . We empirically compare our algorithms with state-of-the-art methods for non-stationary MABs over several synthetically generated tasks and an online model selection problem for a real-world dataset. Finally, using synthetic and real-world data, we illustrate the effectiveness of the proposed approaches compared with state-of-the-art algorithms for the non-stationary bandits.
Calibrated and Sharp Uncertainties in Deep Learning via Density Estimation

Accurate probabilistic predictions can be characterized by two properties -- calibration and sharpness. However, standard maximum likelihood training yields models that are poorly calibrated and thus inaccurate -- a 90% confidence interval typically does not contain the true outcome 90% of the time. This paper argues that calibration is important in practice and is easy to maintain by performing low-dimensional density estimation. We introduce a simple training procedure based on recalibration that yields calibrated models without sacrificing overall performance; unlike previous approaches, ours ensures the most general property of distribution calibration and applies to any model, including neural networks. We formally prove the correctness of our procedure assuming that we can estimate densities in low dimensions and we establish uniform convergence bounds. Our results yield empirical performance improvements on linear and deep Bayesian models and suggest that calibration should be increasingly leveraged across machine learning.
Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks

In temporal-difference reinforcement learning algorithms, variance in value estimation can cause instability and overestimation of the maximal target value. Many algorithms have been proposed to reduce overestimation, including several recent ensemble methods, however none have shown success in sample-efficient learning through addressing estimation variance as the root cause of overestimation. In this paper, we propose MeanQ, a simple ensemble method that estimates target values as ensemble means. Despite its simplicity, MeanQ shows remarkable sample efficiency in experiments on the Atari Learning Environment benchmark. Importantly, we find that an ensemble of size 5 sufficiently reduces estimation variance to obviate the lagging target network, eliminating it as a source of bias and further gaining sample efficiency. We justify intuitively and empirically the design choices in MeanQ, including the necessity of independent experience sampling. On a set of 26 benchmark Atari environments, MeanQ outperforms all tested baselines, including the best available baseline, SUNRISE, at 100K interaction steps in 16/26 environments, and by 68% on average. MeanQ also outperforms Rainbow DQN at 500K steps in 21/26 environments, and by 49% on average, and achieves average human-level performance using 200K ($\pm$100K) interaction steps. Our implementation is available at https://github.com/indylab/MeanQ.
State Transition of Dendritic Spines Improves Learning of Sparse Spiking Neural Networks

Spiking Neural Networks (SNNs) are considered a promising alternative to Artiﬁcial Neural Networks (ANNs) for their event-driven computing paradigm when deployed on energy-efﬁcient neuromorphic hardware. Recently, deep SNNs have shown breathtaking performance improvement through cutting-edge training strategy and ﬂexi-ble structure, which also scales up the number of parameters and computational burdens in a single network. Inspired by the state transition of dendritic spines in the ﬁlopodial model of spinogen-esis, we model different states of SNN weights, facilitating weight optimization for pruning. Furthermore, the pruning speed can be regulated by using different functions describing the growing threshold of state transition. We organize these techniques as a dynamic pruning algorithm based on nonlinear reparameterization mapping from spine size to SNN weights. Our approach yields sparse deep networks on the large-scale dataset (SEW ResNet18 on ImageNet) while maintaining state-of-the-art low performance loss ( ∼ 3% at 88.8% sparsity) compared to existing pruning methods on directly trained SNNs. Moreover, we ﬁnd out pruning speed regulation while learning is crucial to avoiding disastrous performance degradation at the ﬁnal stages of training, which may shed light on future work on SNN pruning.
One-Pass Diversified Sampling with Application to Terabyte-Scale Genomic Sequence Streams

A popular approach to reduce the size of a massive dataset is to apply efﬁcient online sampling to the stream of data as it is read or generated. Online sampling routines are currently restricted to variations of reservoir sampling, where each sample is selected uniformly and independently of other samples. This renders them unsuitable for large-scale applications in computational biology, such as metagenomic community proﬁling and protein function annotation, which suffer from severe class imbalance. To maintain a representative and diverse sample, we must identify and preferentially select data that are likely to belong to rare classes. We argue that existing schemes for diversity sampling have prohibitive overhead for large-scale problems and high-throughput streams. We propose an efﬁcient sampling routine that uses an online representation of the data distribution as a preﬁlter to retain elements from rare groups. We apply this method to several genomic data analysis tasks and demonstrate signiﬁcant speedup in downstream analysis without sacriﬁcing the quality of the results. Because our algorithm is 2x faster and uses 1000x less memory than coreset, reservoir and sketch-based alternatives, we anticipate that it will become a useful preprocessing step for applications with large-scale streaming data. Whole-genome shotgun sequencing (WGS) has inspired nu-*
Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers

Despite the ubiquitous use of stochastic optimization algorithms in machine learning, the precise impact of these algorithms and their dynamics on generalization performance in realis-tic non-convex settings is still poorly understood. While recent work has revealed connections between generalization and heavy-tailed behavior in stochastic optimization, this work mainly relied on continuous-time approximations; and a rigorous treatment for the original discrete-time iterations is yet to be performed. To bridge this gap, we present novel bounds linking generalization to the lower tail exponent of the transition kernel associated with the optimizer around a local minimum, in both discrete- and continuous-time settings. To achieve this, we ﬁrst prove a data-and algorithm-dependent generalization bound in terms of the celebrated Fernique–Talagrand functional applied to the trajectory of the optimizer. Then, we specialize this result by exploiting the Markovian structure of stochastic optimizers, and derive bounds in terms of their (data-dependent) transition kernels. We support our theory with empirical results from a variety of neural networks, showing correlations between generalization error and lower tail exponents.
Dating the photosynthetic organelle evolution in Archaeplastida, Paulinella and secondary-plastid bearing lineages

Photosynthetic eukaryotes have shaped the Earth’s biosphere by producing O2 and converting light into organic compounds in specialized organelles called plastids. Plastids evolved from free-living cyanobacteria engulfed by heterotrophic unicellular eukaryotes in processes called cyanobacterial endosymbioses. Two independent such processes have been reported so far. The first gave rise to primary plastids and three Archaeplastida lineages: glaucophytes, red algae and green algae with land plants, whereas the second resulted in chromatophores in the rhizarian amoeba Paulinella. Importantly, archaeplastidans donated their plastids to many protist groups, thereby further spreading photosynthesis across the tree of life. To reveal the complex plastid evolution, we performed comprehensive phylogenetic and multi-clock analyses based on new fossil calibration points and the greatest number yet of plastid-encoded proteins from 108 taxa, representing a large diversity of photosynthetic organisms. Our results indicate that primary plastids evolved prior to 2.1 - 1.8 Bya, i.e. before glaucophytes diverged from the other archaeplastidans. Like the primary plastids before, Paulinella chromatophores evolved in low salinity habitats and possibly before 292 - 266 Mya. Red and green algae were engulfed by cryptophyte and chlorarachniophyte ancestors between 1.7 - 1.4 Bya, and 1.1 - 1.0 Bya, respectively; the former subsequently triggered plastid transfers to other eukaryotes. The diversification rate of the photosynthetic organisms increased with temperature and CO2 but decreased with O2 and volcanic activity. We also studied the impact of various molecular clocks and calibration sets on the age estimation and clearly indicate that the clocks are the source of greater differences. Significance Statement Cyanobacteria and eukaryote endosymbioses created a multitude of photosynthetic organelles called plastids that feed most life on our planet. For decades scientists have been trying to untangle the puzzle of plastid evolution, i.e. when and how plastids were acquired and spread throughout the eukaryotic tree of life. To answer these questions we applied phylogenetic and multi-clock methods combined with new fossil calibration points on large data sets. Our results push back in the Earth’s history most key events concerning plastid evolution compared to previous reports. Additionally, we discovered a significant impact of climatic and atmospheric parameters on the diversification rate of plastid lineages. The estimated divergence times enabled us to reinterpret taxonomic classification of controversial fossils.
DMC1 attenuates RAD51-mediated recombination in Arabidopsis

Ensuring balanced distribution of chromosomes in gametes, meiotic recombination is essential for fertility in most sexually reproducing organisms. The repair of the programmed DNA double strand breaks that initiate meiotic recombination requires two DNA strand-exchange proteins, RAD51 and DMC1, to search for and invade an intact DNA molecule on the homologous chromosome. DMC1 is meiosis-specific, while RAD51 is essential for both mitotic and meiotic homologous recombination. DMC1 is the main catalytically active strand-exchange protein during meiosis, while this activity of RAD51 is downregulated. RAD51 is however an essential cofactor in meiosis, supporting the function of DMC1. This work presents a study of the mechanism(s) involved in this and our results point to DMC1 being, at least, a major actor in the meiotic suppression of the RAD51 strand-exchange activity in plants. Ectopic expression of DMC1 in somatic cells renders plants hypersensitive to DNA damage and specifically impairs RAD51-dependent homologous recombination. DNA damage-induced RAD51 focus formation in somatic cells is not however suppressed by ectopic expression of DMC1. Interestingly, DMC1 also forms damage-induced foci in these cells and we further show that the ability of DMC1 to prevent RAD51-mediated recombination is associated with local assembly of DMC1 at DNA breaks. In support of our hypothesis, expression of a dominant negative DMC1 protein in meiosis impairs RAD51-mediated DSB repair. We propose that DMC1 acts to prevent RAD51-mediated recombination in Arabidopsis and that this down-regulation requires local assembly of DMC1 nucleofilaments. Author Summary Essential for fertility and responsible for a major part of genetic variation in sexually reproducing species, meiotic recombination establishes the physical linkages between homologous chromosomes which ensure their balanced segregation in the production of gametes. These linkages, or chiasmata, result from DNA strand exchange catalyzed by the RAD51 and DMC1 recombinases and their numbers and distribution are tightly regulated. Essential for maintaining chromosomal integrity in mitotic cells, the strand-exchange activity of RAD51 is downregulated in meiosis, where it plays a supporting role to the activity of DMC1. Notwithstanding considerable attention from the genetics community, precisely why this is done and the mechanisms involved are far from being fully understood. We show here in the plant Arabidopsis that DMC1 can downregulate RAD51 strand-exchange activity and propose that this may be a general mechanism for suppression of RAD51-mediated recombination in meiosis.
Weight cycling induces innate immune memory in adipose tissue macrophages

Weight loss improves obesity-associated diabetes risk. However, most individuals regain weight, which worsens the risk of developing diabetes and cardiovascular disease. We previously reported that male mice retain obesity-associated immunological changes even after weight loss, suggesting that immune cells may remember the state of obesity. Therefore, we hypothesized that cycles of weight gain and loss, otherwise known as weight cycling, can induce innate memory in adipose macrophages. We first treated bone marrow derived macrophages in a culture model of innate immune memory. Priming the cells with palmitic acid or adipose tissue conditioned media increased maximal glycolysis, oxidative phosphorylation, and increased LPS-induced TNFα and IL-6 production. While weight loss improved glucose tolerance, adipose macrophages retained elevated LPS-induced cytokine production. In a model of weight cycling, adipose macrophages had elevated metabolism and secreted higher levels of basal TNFα. Together, these data suggest that obesity and subsequent weight loss can prime adipose macrophages for enhanced inflammation upon weight regain. This innate immune memory response may contribute to worsened glucose tolerance following weight cycling.