Miracle-dz/UniFAR-FTUs huggingface dataset:
Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Vision transformers (ViTs) have recently received explosive popularity, but their enormous model sizes and training costs remain daunting. Conventional post-training pruning often incurs higher training budgets. In contrast, this paper aims to trim down both the training memory overhead and the inference complexity, without sacrificing the achievable accuracy. We carry out the first-of-its-kind comprehensive exploration, on taking a unified approach of integrating sparsity in ViTs"from end to end". Specifically, instead of training full ViTs, we dynamically extract and train sparse subnetworks, while sticking to a fixed small parameter budget. Our approach jointly optimizes model parameters and explores connectivity throughout training, ending up with one sparse network as the final output. The approach is seamlessly extended from unstructured to structured sparsity, the latter by considering to guide the prune-and-grow of self-attention heads inside ViTs. We further co-explore data and architecture sparsity for additional efficiency gains by plugging in a novel learnable token selector to adaptively determine the currently most vital patches. Extensive results on ImageNet with diverse ViT backbones validate the effectiveness of our proposals which obtain significantly reduced computational cost and almost unimpaired generalization. Perhaps most surprisingly, we find that the proposed sparse (co-)training can sometimes improve the ViT accuracy rather than compromising it, making sparsity a tantalizing"free lunch". For example, our sparsified DeiT-Small at (5%, 50%) sparsity for (data, architecture), improves 0.28% top-1 accuracy, and meanwhile enjoys 49.32% FLOPs and 4.40% running time savings. Our codes are available at https://github.com/VITA-Group/SViTE.
Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning

High-quality estimates of uncertainty and robustness are crucial for numerous real-world applications, especially for deep learning which underlies many deployed ML systems. The ability to compare techniques for improving these estimates is therefore very important for research and practice alike. Yet, competitive comparisons of methods are often lacking due to a range of reasons, including: compute availability for extensive tuning, incorporation of sufficiently many baselines, and concrete documentation for reproducibility. In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-of-the-art deep learning methods on a variety of tasks. As of this writing, the collection spans 19 methods across 9 tasks, each with at least 5 metrics. Each baseline is a self-contained experiment pipeline with easily reusable and extendable components. Our goal is to provide immediate starting points for experimentation with new methods or applications. Additionally we provide model checkpoints, experiment outputs as Python notebooks, and leaderboards for comparing results. Code available at https://github.com/google/uncertainty-baselines.
Viewing Angle Observations and Effects of Evolution with Redshift, Black Hole Mass, and Eddington Ratio in Quasar-based Cosmology

This study is focused on the observational measurement of the viewing angle of individual quasars by modeling the broadband quasar spectrum ranging from the IR to soft X-ray band. Sources are selected from various published catalogs, and their broadband quasi-simultaneous spectral data points were collected and used for modeling. We started with a COSMOS sample of Type-1 sources that have broadband photometric points. To include more data points, we cross-matched the COSMOS with the Sloan Digital Sky Survey Data Release 14 quasar catalog, and eventually we find 90 sources which have broadband data ranging from IR to soft X-ray. The broadband spectral energy distribution modeling is done in Xspec by using the optxagnf and the SKIRTOR models for the X-ray, UV, optical, and IR regimes for each source. The whole sample is divided into four bins with respect to redshift, black hole (BH) mass, and Eddington ratio with an equal number of sources in each bin. The viewing angle is estimated in each bin, and its evolution with respect to redshift, BH mass, and Eddington ratio is examined. In result, we did not find any significant evolution of viewing angle with those parameters within the 95% confidence interval. We conclude that the use of quasars in cosmology to determine the expansion rate of the universe is therefore justified, and biases are not expected.
Self-supervised and Supervised Joint Training for Resource-rich Machine Translation

Self-supervised pre-training of text representations has been successfully applied to low-resource Neural Machine Translation (NMT). However, it usually fails to achieve notable gains on resource-rich NMT. In this paper, we propose a joint training approach, $F_2$-XEnDec, to combine self-supervised and supervised learning to optimize NMT models. To exploit complementary self-supervised signals for supervised learning, NMT models are trained on examples that are interbred from monolingual and parallel sentences through a new process called crossover encoder-decoder. Experiments on two resource-rich translation benchmarks, WMT'14 English-German and WMT'14 English-French, demonstrate that our approach achieves substantial improvements over several strong baseline methods and obtains a new state of the art of 46.19 BLEU on English-French when incorporating back translation. Results also show that our approach is capable of improving model robustness to input perturbations such as code-switching noise which frequently appears on social media.
Nonsmooth Implicit Differentiation for Machine Learning and Optimization

In view of training increasingly complex learning architectures, we establish a nonsmooth implicit function theorem with an operational calculus. Our result applies to most practical problems (i.e., definable problems) provided that a nonsmooth form of the classical invertibility condition is fulfilled. This approach allows for formal subdifferentiation: for instance, replacing derivatives by Clarke Jacobians in the usual differentiation formulas is fully justified for a wide class of nonsmooth problems. Moreover this calculus is entirely compatible with algorithmic differentiation (e.g., backpropagation). We provide several applications such as training deep equilibrium networks, training neural nets with conic optimization layers, or hyperparameter-tuning for nonsmooth Lasso-type models. To show the sharpness of our assumptions, we present numerical experiments showcasing the extremely pathological gradient dynamics one can encounter when applying implicit algorithmic differentiation without any hypothesis.
Batch Normalization Orthogonalizes Representations in Deep Random Networks

This paper underlines a subtle property of batch-normalization (BN): Successive batch normalizations with random linear transformations make hidden representations increasingly orthogonal across layers of a deep neural network. We establish a non-asymptotic characterization of the interplay between depth, width, and the orthogonality of deep representations. More precisely, under a mild assumption, we prove that the deviation of the representations from orthogonality rapidly decays with depth up to a term inversely proportional to the network width. This result has two main implications: 1) Theoretically, as the depth grows, the distribution of the representation -- after the linear layers -- contracts to a Wasserstein-2 ball around an isotropic Gaussian distribution. Furthermore, the radius of this Wasserstein ball shrinks with the width of the network. 2) In practice, the orthogonality of the representations directly influences the performance of stochastic gradient descent (SGD). When representations are initially aligned, we observe SGD wastes many iterations to orthogonalize representations before the classification. Nevertheless, we experimentally show that starting optimization from orthogonal representations is sufficient to accelerate SGD, with no need for BN.
A Simulation Study of Ultra-relativistic Jets. II. Structures and Dynamics of FR-II Jets

We study the structures of ultra-relativistic jets injected into the intracluster medium and the associated flow dynamics, such as shocks, velocity shear, and turbulence, through three-dimensional relativistic hydrodynamic (RHD) simulations. To that end, we have developed a high-order accurate RHD code, equipped with a weighted essentially non-oscillatory scheme and a realistic equation of state. Using the code, we explore a set of jet models with the parameters relevant to FR-II radio galaxies. We confirm that the overall jet morphology is primarily determined by the jet power, and the jet-to-background density and pressure ratios play secondary roles. Jets with higher powers propagate faster, resulting in more elongated structures, while those with lower powers produce more extended cocoons. Shear interfaces in the jet are dynamically unstable, and hence, chaotic structures with shocks and turbulence develop. We find that the fraction of the jet-injected energy dissipated through shocks and turbulence is greater in less powerful jets, although the actual amount of the dissipated energy is larger in more powerful jets. In lower power jets, the backflow is dominant in the energy dissipation owing to the broad cocoon filled with shocks and turbulence. In higher power jets, by contrast, both the backflow and jet-spine flow are important for the energy dissipation. Our results imply that different mechanisms, such as diffusive shock acceleration, shear acceleration, and stochastic turbulent acceleration, may be involved in the production of ultra-high energy cosmic rays in FR-II radio galaxies.
Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study

Recent research in multilingual language models (LM) has demonstrated their ability to effectively handle multiple languages in a single model. This holds promise for low web-resource languages (LRL) as multilingual models can enable transfer of supervision from high resource languages to LRLs. However, incorporating a new language in an LM still remains a challenge, particularly for languages with limited corpora and in unseen scripts. In this paper we argue that relatedness among languages in a language family may be exploited to overcome some of the corpora limitations of LRLs, and propose RelateLM. We focus on Indian languages, and exploit relatedness along two dimensions: (1) script (since many Indic scripts originated from the Brahmic script), and (2) sentence structure. RelateLM uses transliteration to convert the unseen script of limited LRL text into the script of a Related Prominent Language (RPL) (Hindi in our case). While exploiting similar sentence structures, RelateLM utilizes readily available bilingual dictionaries to pseudo translate RPL text into LRL corpora. Experiments on multiple real-world benchmark datasets provide validation to our hypothesis that using a related language as pivot, along with transliteration and pseudo translation based data augmentation, can be an effective way to adapt LMs for LRLs, rather than direct training or pivoting through English.
Closed-Form Analytical Results for Maximum Entropy Reinforcement Learning

We introduce a mapping between Maximum Entropy Reinforcement Learning (MaxEnt RL) and Markovian processes conditioned on rare events. In the long time limit, this mapping allows us to derive analytical expressions for the optimal policy, dynamics and initial state distributions for the general case of stochastic dynamics in MaxEnt RL. We ﬁnd that soft-Q functions in MaxEnt RL can be obtained from the Perron-Frobenius eigenvalue and the corresponding left eigenvector of a regular, non-negative matrix derived from the underlying Markov Decision Process (MDP). The results derived lead to novel algorithms for model-based and model-free MaxEnt RL, which we validate by numerical simulations. The mapping established in this work opens further avenues for the application of novel analytical and computational approaches to problems in MaxEnt RL. We make our code available at: https://github.com/argearriojas/maxent-rl-mdp-scripts .
Aggregation sheaves for greedy modal decompositions

This article develops a new theoretical basis for decomposing signals that are formed by the linear superposition of a finite number of modes. Each mode depends linearly on the weights within the superposition and nonlinearly upon several other parameters. The particular focus of this article is upon finding both the weights and the parameters when the number of modes is not known in advance. This article introduces a novel mathematical formalism, aggregation sheaves, and shows how they characterize the behavior of greedy algorithms that attempt to solve modal decomposition problems. It is shown that minimizing the local consistency radius within the aggregation sheaf is guaranteed to solve all modal decomposition problems. Since the modes may or may not be well-separated, a greedy algorithm that identifies the most distinct modes first may not work reliably.
Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation

Natural policy gradient (NPG) methods with entropy regularization achieve impressive empirical success in reinforcement learning problems with large state-action spaces. However, their convergence properties and the impact of entropy regularization remain elusive in the function approximation regime. In this paper, we establish finite-time convergence analyses of entropy-regularized NPG with linear function approximation under softmax parameterization. In particular, we prove that entropy-regularized NPG with averaging satisfies the \emph{persistence of excitation} condition, and achieves a fast convergence rate of $\tilde{O}(1/T)$ up to a function approximation error in regularized Markov decision processes. This convergence result does not require any a priori assumptions on the policies. Furthermore, under mild regularity conditions on the concentrability coefficient and basis vectors, we prove that entropy-regularized NPG exhibits \emph{linear convergence} up to a function approximation error.
Harnessing Unrecognizable Faces for Face Recognition

The common implementation of face recognition systems as a cascade of a detection stage and a recognition or veri-ﬁcation stage can cause problems beyond failures of the detector. When the detector succeeds, it can detect faces that cannot be recognized, no matter how capable the recognition system. Recognizability, a latent variable, should therefore be factored into the design and implementation of face recognition systems. We propose a measure of recognizability of a face image that leverages a key empirical observation: An embedding of face images, implemented by a deep neural network trained using mostly recognizable identities, induces a partition of the hypersphere whereby unrecogniz-able identities cluster together . This occurs regardless of the phenomenon that causes a face to be unrecognizable, it be optical or motion blur, partial occlusion, spatial quantization, poor illumination. Therefore, we use the distance from such an “unrecognizable identity” as a measure of recognizability, and incorporate it in the design of the overall sys-tem. We show that accounting for recognizability reduces the error rate of single-image face recognition by 58% at FAR=1e-5 on the IJB-C Covariate Veriﬁcation benchmark, and reduces the veriﬁcation error rate by 24% at FAR=1e-5 in set-based recognition on the IJB-C benchmark.
Stability and Generalization of Bilevel Programming in Hyperparameter Optimization

The (gradient-based) bilevel programming framework is widely used in hyperparameter optimization and has achieved excellent performance empirically. Previous theoretical work mainly focuses on its optimization properties, while leaving the analysis on generalization largely open. This paper attempts to address the issue by presenting an expectation bound w.r. t. the validation set based on uniform stability. Our results can explain some mysterious behaviours of the bilevel programming in practice, for instance, overfitting to the validation set. We also present an expectation bound for the classical cross-validation algorithm. Our results suggest that gradient-based algorithms can be better than cross-validation under certain conditions in a theoretical perspective. Furthermore, we prove that regularization terms in both the outer and inner levels can relieve the overfitting problem in gradient-based algorithms. In experiments on feature learning and data reweighting for noisy labels, we corroborate our theoretical findings.
Sample Complexity of Tree Search Configuration: Cutting Planes and Beyond

Cutting-plane methods have enabled remarkable successes in integer programming over the last few decades. State-of-the-art solvers integrate a myriad of cutting-plane techniques to speed up the underlying tree-search algorithm used to find optimal solutions. In this paper we prove the first guarantees for learning high-performing cut-selection policies tailored to the instance distribution at hand using samples. We first bound the sample complexity of learning cutting planes from the canonical family of Chv\'atal-Gomory cuts. Our bounds handle any number of waves of any number of cuts and are fine tuned to the magnitudes of the constraint coefficients. Next, we prove sample complexity bounds for more sophisticated cut selection policies that use a combination of scoring rules to choose from a family of cuts. Finally, beyond the realm of cutting planes for integer programming, we develop a general abstraction of tree search that captures key components such as node selection and variable selection. For this abstraction, we bound the sample complexity of learning a good policy for building the search tree.
Interpretable agent communication from scratch(with a generic visual processor emerging on the side)

As deep networks begin to be deployed as autonomous agents, the issue of how they can communicate with each other becomes important. Here, we train two deep nets from scratch to perform realistic referent identification through unsupervised emergent communication. We show that the largely interpretable emergent protocol allows the nets to successfully communicate even about object types they did not see at training time. The visual representations induced as a by-product of our training regime, moreover, show comparable quality, when re-used as generic visual features, to a recent self-supervised learning model. Our results provide concrete evidence of the viability of (interpretable) emergent deep net communication in a more realistic scenario than previously considered, as well as establishing an intriguing link between this field and self-supervised visual learning.
Analogue models for Schwarzschild and Reissner-Nordström spacetimes

We present analogue models for the Schwarzschild and Reissner-Nordstr\"om (RN) spacetimes based on unidirectional hydrodynamic flows. We show that, with appropriate coordinate transformations, sound waves in a moving fluid propagate as scalar fields at the equatorial sections of Schwarzschild and RN black holes. The coordinate associated with the direction of the flow plays the role of the radial coordinate in standard Schwarzschild/RN coordinates, while the transversal spatial coordinate is related to the angular coordinate on each equatorial slice. The magnitude of the flow velocity is related to the mass and charge of the analogue black hole. Physical quantities like pressure and density remain finite at the analogue horizon, thereby resulting in sound waves that are well defined almost everywhere, except possibly at the analogue black hole singularity.
The blowdown of ancient noncollapsed mean curvature flows

In this paper, we consider ancient noncollapsed mean curvature flows $M_t=\partial K_t\subset \mathbb{R}^{n+1}$ that do not split off a line. It follows from general theory that the blowdown of any time-slice, $\lim_{\lambda \to 0} \lambda K_{t_0}$, is at most $n-1$ dimensional. Here, we show that the blowdown is in fact at most $n-2$ dimensional. Our proof is based on fine cylindrical analysis, which generalizes the fine neck analysis that played a key role in many recent papers. Moreover, we show that in the uniformly $k$-convex case, the blowdown is at most $k-2$ dimensional. This generalizes recent results from Choi-Haslhofer-Hershkovits to higher dimensions, and also has some applications towards the classification problem for singularities in 3-convex mean curvature flow.
Safe reopening of university campuses is possible with COVID-19 vaccination

We construct an agent-based SEIR model to simulate COVID-19 spread at a 16000-student mostly non-residential urban university during the Fall 2021 Semester. We find that mRNA vaccine coverage at 100% combined with weekly screening testing of 25% of the campus population make it possible to safely reopen to in-person instruction. Our simulations exhibit a right-skew for total infections over the semester that becomes more pronounced with less vaccine coverage, less vaccine effectiveness and no additional preventative measures. This suggests that high levels of infection are not exceedingly rare with campus social connections the main transmission route. Finally, we find that if vaccine coverage is 100% and vaccine effectiveness is above 80%, then a safe reopening is possible even without facemask use. This models possible future scenarios with high coverage of additional “booster” doses of COVID-19 vaccines.
Learning to Detect Multi-Modal Grasps for Dexterous Grasping in Dense Clutter

We propose an approach to multi-modal grasp detection that jointly predicts the probabilities that several types of grasps succeed at a given grasp pose. Given a partial point cloud of a scene, the algorithm proposes a set of feasible grasp candidates, then estimates the probabilities that a grasp of each type would succeed at each candidate pose. Predicting grasp success probabilities directly from point clouds makes our approach agnostic to the number and placement of depth sensors at execution time. We evaluate our system both in simulation and on a real robot with a Robotiq 3-Finger Adaptive Gripper and compare our network against several baselines that perform fewer types of grasps. Our experiments show that a system that explicitly models grasp type achieves an object retrieval rate 8.5% higher in a complex cluttered environment than our highest-performing baseline.
The consistency strength of determinacy when all sets are universally Baire

It is known that the large cardinal strength of the Axiom of Determinacy when enhanced with the hypothesis that all sets of reals are universally Baire is much stronger than the Axiom of Determinacy itself. Sargsyan conjectured it to be as strong as the existence of a cardinal that is both a limit of Woodin cardinals and a limit of strong cardinals. Larson, Sargsyan and Wilson used a generalization of Woodin's derived model construction to show that this conjectured result would be optimal. In this paper we introduce a new translation procedure for hybrid mice extending work of Steel, Zhu and Sargsyan and apply it to prove Sargsyan's conjecture.