Miracle-dz/UniFAR-FTUs huggingface dataset:
A Past Episode of Rapid Tidal Evolution of Enceladus?

Saturn possesses a dynamically rich system containing numerous moons and impressive rings. Whether the rings of Saturn are much younger than the planet itself has been a long-standing open question; more recently a young age has been proposed for some moons. The recent detection of the fast orbital evolution of Rhea and Titan strongly suggests a highly frequency-dependent tidal response of Saturn, possibly through the excitation of inertial waves within the planet’s convective envelope. Here we show that resonance locking to inertial waves cannot explain the dynamical structure of the Saturnian system or the current tidal heating of Enceladus. On the other hand, both the observations and our modeling results indicate that the system is not consistent with evolution under equilibrium tides. We propose that the system’s architecture can best be explained by a relatively high “background” tidal response coupled with discrete resonant modes. In this view, only Titan may be in a true long-term resonance lock with a tidal mode of Saturn. Rhea is most likely currently experiencing a transient period of fast tidal evolution as it passes through a mode, rather than being locked into it. Assuming that Enceladus went through a temporary period of fast tidal evolution, we can reproduce its present resonance with Dione and satisfy other dynamical constraints. Additionally, we conclude that the long-term tidal response of Saturn to Tethys must be weaker than expected from frequency-independent tides, as already found by observations.
Early-growing Supermassive Black Holes Strengthen Bars and Boxy/Peanut Bulges

Using N-body simulations, we explore the effects of growing a supermassive black hole (SMBH) prior to or during the formation of a stellar bar. Keeping the final mass and growth rate of the SMBH fixed, we show that if it is introduced before or while the bar is still growing, the SMBH does not cause a decrease in bar amplitude. Rather, in most cases, it is strengthened. In addition, an early-growing SMBH always either decreases the buckling amplitude, delays buckling, or both. This weakening of buckling is caused by an increase in the disk vertical velocity dispersion at radii well beyond the nominal black hole sphere of influence. While we find considerable stochasticity and sensitivity to initial conditions, the only case where the SMBH causes a decrease in bar amplitude is when it is introduced after the bar has attained a steady state. In this case, we confirm previous findings that the decrease in bar strength is a result of scattering of bar-supporting orbits with small pericenter radii. By heating the inner disk both radially and vertically, an early-growing SMBH increases the fraction of stars that can be captured by the inner Lindblad resonance (ILR) and the vertical ILR, thereby strengthening both the bar and the boxy-peanut-shaped bulge. Using orbital frequency analysis of star particles, we show that when an SMBH is introduced early and the bar forms around it, the bar is populated by different families of regular bar-supporting orbits than when the bar forms without an SMBH.
Learning Unnormalized Statistical Models via Compositional Optimization

Learning unnormalized statistical models (e.g., energy-based models) is computationally challenging due to the complexity of handling the partition function. To eschew this complexity, noise-contrastive estimation~(NCE) has been proposed by formulating the objective as the logistic loss of the real data and the artificial noise. However, as found in previous works, NCE may perform poorly in many tasks due to its flat loss landscape and slow convergence. In this paper, we study it a direct approach for optimizing the negative log-likelihood of unnormalized models from the perspective of compositional optimization. To tackle the partition function, a noise distribution is introduced such that the log partition function can be written as a compositional function whose inner function can be estimated with stochastic samples. Hence, the objective can be optimized by stochastic compositional optimization algorithms. Despite being a simple method, we demonstrate that it is more favorable than NCE by (1) establishing a fast convergence rate and quantifying its dependence on the noise distribution through the variance of stochastic estimators; (2) developing better results for one-dimensional Gaussian mean estimation by showing our objective has a much favorable loss landscape and hence our method enjoys faster convergence; (3) demonstrating better performance on multiple applications, including density estimation, out-of-distribution detection, and real image generation.
Gibbs Dynamics for the Weakly Dispersive Nonlinear Schr\"odinger Equations

We consider the Cauchy problem for the one-dimensional periodic cubic nonlinear fractional Schr\"odinger equation (FNLS) with initial data distributed via Gibbs measure. We construct global strong solutions with the flow property to FNLS on the support of the Gibbs measure in the full dispersive range, thus resolving a question proposed by Sun-Tzvetkov (2021). As a byproduct, we prove the invariance of the Gibbs measure and almost sure global well-posedness for FNLS.
Competitive Channel-Capacity

We consider communication over channels whose statistics are not known in full, but can be parameterized as a finite family of memoryless channels. A typical approach to address channel uncertainty is to design codes for the worst channel in the family, resulting in the well-known compound channel capacity. Although this approach is robust, it may suffer a loss of performance if the capacity-achieving distribution of the worst channel attains low rates over other channels. In this work, we cope with channel uncertainty through the lens of competitive analysis. The idea is to optimize a relative metric that compares the performance of the designed code and a clairvoyant code that has access to the true channel. To allow communication rates that can adapt to the channel at use, we consider rateless codes with a fixed number of information bits and random decoding times. We propose two competitive metrics: the competitive ratio between the decoding times of the two codes, and a regret defined as the difference between the expected rates. Our main results are single-letter expressions for the competitive-ratio and the regret, expressed as a max-min or min-max optimization. Several examples illustrate our results and the benefits of the competitive analysis approach to code design.
Symmetry & Critical Points for Symmetric Tensor Decomposition Problems

We consider the nonconvex optimization problem associated with the decomposition of a real symmetric tensor into a sum of rank one terms. Use is made of the rich symmetry structure to construct infinite families of critical points represented by Puiseux series in the problem dimension, and so obtain precise analytic estimates on the value of the objective function and the Hessian spectrum. The results allow an analytic characterization of various obstructions to using local optimization methods, revealing in particular a complex array of saddles and minima differing by their symmetry, structure and analytic properties. A~desirable phenomenon, occurring for all critical points considered, concerns the number of negative Hessian eigenvalues increasing with the value of the objective function. Our approach makes use of Newton polyhedra as well as results from real algebraic geometry, notably the Curve Selection Lemma, to determine the extremal character of degenerate critical points, establishing in particular the existence of infinite families of third-order saddles which can significantly slow down the optimization process.
BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information

Automated reasoning with unstructured natural text is a key requirement for many potential applications of NLP and for developing robust AI systems. Recently, Language Models (LMs) have demonstrated complex reasoning capacities even without any finetuning. However, existing evaluation for automated reasoning assumes access to a consistent and coherent set of information over which models reason. When reasoning in the real-world, the available information is frequently inconsistent or contradictory, and therefore models need to be equipped with a strategy to resolve such conflicts when they arise. One widely-applicable way of resolving conflicts is to impose preferences over information sources (e.g., based on source credibility or information recency) and adopt the source with higher preference. In this paper, we formulate the problem of reasoning with contradictory information guided by preferences over sources as the classical problem of defeasible reasoning, and develop a dataset called BoardgameQA for measuring the reasoning capacity of LMs in this setting. BoardgameQA also incorporates reasoning with implicit background knowledge, to better reflect reasoning problems in downstream applications. We benchmark various LMs on BoardgameQA and the results reveal a significant gap in the reasoning capacity of state-of-the-art LMs on this problem, showing that reasoning with conflicting information does not surface out-of-the-box in LMs. While performance can be improved with finetuning, it nevertheless remains poor.
Best constants in subelliptic fractional Sobolev and Gagliardo-Nirenberg inequalities and ground states on stratified Lie groups

In this paper, we establish the sharp fractional subelliptic Sobolev inequalities and Gagliardo-Nirenberg inequalities on stratified Lie groups. The best constants are given in terms of a ground state solution of a fractional subelliptic equation involving the fractional $p$-sublaplacian ($1<p<\infty$) on stratified Lie groups. We also prove the existence of ground state (least energy) solutions to nonlinear subelliptic fractional Schr\"odinger equation on stratified Lie groups. Different from the proofs of analogous results in the setting of classical Sobolev spaces on Euclidean spaces given by Weinstein (Comm. Math. Phys. 87(4):576-676 (1982/1983)) using the rearrangement inequality which is not available in stratified Lie groups, we apply a subelliptic version of vanishing lemma due to Lions extended in the setting of stratified Lie groups combining it with the compact embedding theorem for subelliptic fractional Sobolev spaces obtained in our previous paper (Math. Ann. (2023)). We also present subelliptic fractional logarithmic Sobolev inequalities with explicit constants on stratified Lie groups. The main results are new for $p=2$ even in the context of the Heisenberg group.
Identification of Nonlinear Latent Hierarchical Models

Identifying latent variables and causal structures from observational data is essential to many real-world applications involving biological data, medical data, and unstructured data such as images and languages. However, this task can be highly challenging, especially when observed variables are generated by causally related latent variables and the relationships are nonlinear. In this work, we investigate the identification problem for nonlinear latent hierarchical causal models in which observed variables are generated by a set of causally related latent variables, and some latent variables may not have observed children. We show that the identifiability of causal structures and latent variables (up to invertible transformations) can be achieved under mild assumptions: on causal structures, we allow for multiple paths between any pair of variables in the graph, which relaxes latent tree assumptions in prior work; on structural functions, we permit general nonlinearity and multi-dimensional continuous variables, alleviating existing work's parametric assumptions. Specifically, we first develop an identification criterion in the form of novel identifiability guarantees for an elementary latent variable model. Leveraging this criterion, we show that both causal structures and latent variables of the hierarchical model can be identified asymptotically by explicitly constructing an estimation procedure. To the best of our knowledge, our work is the first to establish identifiability guarantees for both causal structures and latent variables in nonlinear latent hierarchical models.
A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation

In recent years, concept-based approaches have emerged as some of the most promising explainability methods to help us interpret the decisions of Artificial Neural Networks (ANNs). These methods seek to discover intelligible visual 'concepts' buried within the complex patterns of ANN activations in two key steps: (1) concept extraction followed by (2) importance estimation. While these two steps are shared across methods, they all differ in their specific implementations. Here, we introduce a unifying theoretical framework that comprehensively defines and clarifies these two steps. This framework offers several advantages as it allows us: (i) to propose new evaluation metrics for comparing different concept extraction approaches; (ii) to leverage modern attribution methods and evaluation metrics to extend and systematically evaluate state-of-the-art concept-based approaches and importance estimation techniques; (iii) to derive theoretical guarantees regarding the optimality of such methods. We further leverage our framework to try to tackle a crucial question in explainability: how to efficiently identify clusters of data points that are classified based on a similar shared strategy. To illustrate these findings and to highlight the main strategies of a model, we introduce a visual representation called the strategic cluster graph. Finally, we present https://serre-lab.github. io/Lens, a dedicated website that offers a complete compilation of these visualizations for all classes of the ImageNet dataset.
Lookaround Optimizer: k steps around, 1 step average

Weight Average (WA) is an active research topic due to its simplicity in ensembling deep networks and the effectiveness in promoting generalization. Existing weight average approaches, however, are often carried out along only one training trajectory in a post-hoc manner (i.e., the weights are averaged after the entire training process is finished), which significantly degrades the diversity between networks and thus impairs the effectiveness. In this paper, inspired by weight average, we propose Lookaround, a straightforward yet effective SGD-based optimizer leading to flatter minima with better generalization. Specifically, Lookaround iterates two steps during the whole training period: the around step and the average step. In each iteration, 1) the around step starts from a common point and trains multiple networks simultaneously, each on transformed data by a different data augmentation, and 2) the average step averages these trained networks to get the averaged network, which serves as the starting point for the next iteration. The around step improves the functionality diversity while the average step guarantees the weight locality of these networks during the whole training, which is essential for WA to work. We theoretically explain the superiority of Lookaround by convergence analysis, and make extensive experiments to evaluate Lookaround on popular benchmarks including CIFAR and ImageNet with both CNNs and ViTs, demonstrating clear superiority over state-of-the-arts. Our code is available at https://github.com/Ardcy/Lookaround.
On Achieving Optimal Adversarial Test Error

We first elucidate various fundamental properties of optimal adversarial predictors: the structure of optimal adversarial convex predictors in terms of optimal adversarial zero-one predictors, bounds relating the adversarial convex loss to the adversarial zero-one loss, and the fact that continuous predictors can get arbitrarily close to the optimal adversarial error for both convex and zero-one losses. Applying these results along with new Rademacher complexity bounds for adversarial training near initialization, we prove that for general data distributions and perturbation sets, adversarial training on shallow networks with early stopping and an idealized optimal adversary is able to achieve optimal adversarial test error. By contrast, prior theoretical work either considered specialized data distributions or only provided training error guarantees.
Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training

The use of self-supervised pre-training has emerged as a promising approach to enhance the performance of visual tasks such as image classification. In this context, recent approaches have employed the Masked Image Modeling paradigm, which pre-trains a backbone by reconstructing visual tokens associated with randomly masked image patches. This masking approach, however, introduces noise into the input data during pre-training, leading to discrepancies that can impair performance during the fine-tuning phase. Furthermore, input masking neglects the dependencies between corrupted patches, increasing the inconsistencies observed in downstream fine-tuning tasks. To overcome these issues, we propose a new self-supervised pre-training approach, named Masked and Permuted Vision Transformer (MaPeT), that employs autoregressive and permuted predictions to capture intra-patch dependencies. In addition, MaPeT employs auxiliary positional information to reduce the disparity between the pre-training and fine-tuning phases. In our experiments, we employ a fair setting to ensure reliable and meaningful comparisons and conduct investigations on multiple visual tokenizers, including our proposed $k$-CLIP which directly employs discretized CLIP features. Our results demonstrate that MaPeT achieves competitive performance on ImageNet, compared to baselines and competitors under the same model setting. Source code and trained models are publicly available at: https://github.com/aimagelab/MaPeT.
Parting with Misconceptions about Learning-based Vehicle Motion Planning

The release of nuPlan marks a new era in vehicle motion planning research, offering the first large-scale real-world dataset and evaluation schemes requiring both precise short-term planning and long-horizon ego-forecasting. Existing systems struggle to simultaneously meet both requirements. Indeed, we find that these tasks are fundamentally misaligned and should be addressed independently. We further assess the current state of closed-loop planning in the field, revealing the limitations of learning-based methods in complex real-world scenarios and the value of simple rule-based priors such as centerline selection through lane graph search algorithms. More surprisingly, for the open-loop sub-task, we observe that the best results are achieved when using only this centerline as scene context (i.e., ignoring all information regarding the map and other agents). Combining these insights, we propose an extremely simple and efficient planner which outperforms an extensive set of competitors, winning the nuPlan planning challenge 2023.
Finite Gaussian Neurons: Defending against adversarial attacks by making neural networks say "I don't know"

Since 2014, artificial neural networks have been known to be vulnerable to adversarial attacks, which can fool the network into producing wrong or nonsensical outputs by making humanly imperceptible alterations to inputs. While defenses against adversarial attacks have been proposed, they usually involve retraining a new neural network from scratch, a costly task. In this work, I introduce the Finite Gaussian Neuron (FGN), a novel neuron architecture for artificial neural networks. My works aims to: - easily convert existing models to Finite Gaussian Neuron architecture, - while preserving the existing model's behavior on real data, - and offering resistance against adversarial attacks. I show that converted and retrained Finite Gaussian Neural Networks (FGNN) always have lower confidence (i.e., are not overconfident) in their predictions over randomized and Fast Gradient Sign Method adversarial images when compared to classical neural networks, while maintaining high accuracy and confidence over real MNIST images. To further validate the capacity of Finite Gaussian Neurons to protect from adversarial attacks, I compare the behavior of FGNs to that of Bayesian Neural Networks against both randomized and adversarial images, and show how the behavior of the two architectures differs. Finally I show some limitations of the FGN models by testing them on the more complex SPEECHCOMMANDS task, against the stronger Carlini-Wagner and Projected Gradient Descent adversarial attacks.
Typical Lipschitz maps on rectifiable metric spaces

This article studies typical 1-Lipschitz images of $n$-rectifiable metric spaces $E$ into $\mathbb{R}^m$ for $m\geq n$. For example, if $E\subset \mathbb{R}^k$, we show that the Jacobian of such a typical 1-Lipschitz map equals 1 $\mathcal{H}^n$-almost everywhere and, if $m>n$, preserves the Hausdorff measure of $E$. In general, we provide sufficient conditions, in terms of the tangent norms of $E$, for when a typical 1-Lipschitz map preserves the Hausdorff measure of $E$, up to some constant multiple. Almost optimal results for strongly $n$-rectifiable metric spaces are obtained. On the other hand, for any norm $|\cdot|$ on $\mathbb{R}^m$, we show that, in the space of 1-Lipschitz functions from $([-1,1]^n,|\cdot|_\infty)$ to $(\mathbb{R}^m,|\cdot|)$, the $\mathcal{H}^n$-measure of a typical image is not bounded below by any $\Delta>0$.
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. Unlike conventional neural networks or updated versions of Recurrent Neural Networks (RNNs) such as Long Short-Term Memory (LSTM), transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. As a result, transformer-based models have attracted substantial interest among researchers in the field of artificial intelligence. This can be attributed to their immense potential and remarkable achievements, not only in Natural Language Processing (NLP) tasks but also in a wide range of domains, including computer vision, audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published highlighting the transformer's contributions in specific fields, architectural differences, or performance evaluations, there is still a significant absence of a comprehensive survey paper encompassing its major applications across various domains. Therefore, we undertook the task of filling this gap by conducting an extensive survey of proposed transformer models from 2017 to 2022. Our survey encompasses the identification of the top five application domains for transformer-based models, namely: NLP, Computer Vision, Multi-Modality, Audio and Speech Processing, and Signal Processing. We analyze the impact of highly influential transformer-based models in these domains and subsequently classify them based on their respective tasks using a proposed taxonomy. Our aim is to shed light on the existing potential and future possibilities of transformers for enthusiastic researchers, thus contributing to the broader understanding of this groundbreaking technology.