Miracle-dz/UniFAR-FTUs huggingface dataset:
MLP Architectures for Vision-and-Language Modeling: An Empirical Study

We initiate the first empirical study on the use of MLP architectures for vision-and-language (VL) fusion. Through extensive experiments on 5 VL tasks and 5 robust VQA benchmarks, we find that: (i) Without pre-training, using MLPs for multimodal fusion has a noticeable performance gap compared to transformers; (ii) However, VL pre-training can help close the performance gap; (iii) Instead of heavy multi-head attention, adding tiny one-head attention to MLPs is sufficient to achieve comparable performance to transformers. Moreover, we also find that the performance gap between MLPs and transformers is not widened when being evaluated on the harder robust VQA benchmarks, suggesting using MLPs for VL fusion can generalize roughly to a similar degree as using transformers. These results hint that MLPs can effectively learn to align vision and text features extracted from lower-level encoders without heavy reliance on self-attention. Based on this, we ask an even bolder question: can we have an all-MLP architecture for VL modeling, where both VL fusion and the vision encoder are replaced with MLPs? Our result shows that an all-MLP VL model is sub-optimal compared to state-of-the-art full-featured VL models when both of them get pre-trained. However, pre-training an all-MLP can surprisingly achieve a better average score than full-featured transformer models without pre-training. This indicates the potential of large-scale pre-training of MLP-like architectures for VL modeling and inspires the future research direction on simplifying well-established VL modeling with less inductive design bias. Our code is publicly available at: https://github.com/easonnie/mlp-vil
On visual self-supervision and its effect on model robustness

Recent self-supervision methods have found success in learning feature representations that could rival ones from full supervision, and have been shown to be beneficial to the model in several ways: for example improving models robustness and out-of-distribution detection. In our paper, we conduct an empirical study to understand more precisely in what way can self-supervised learning - as a pre-training technique or part of adversarial training - affects model robustness to $l_2$ and $l_{\infty}$ adversarial perturbations and natural image corruptions. Self-supervision can indeed improve model robustness, however it turns out the devil is in the details. If one simply adds self-supervision loss in tandem with adversarial training, then one sees improvement in accuracy of the model when evaluated with adversarial perturbations smaller or comparable to the value of $\epsilon_{train}$ that the robust model is trained with. However, if one observes the accuracy for $\epsilon_{test} \ge \epsilon_{train}$, the model accuracy drops. In fact, the larger the weight of the supervision loss, the larger the drop in performance, i.e. harming the robustness of the model. We identify primary ways in which self-supervision can be added to adversarial training, and observe that using a self-supervised loss to optimize both network parameters and find adversarial examples leads to the strongest improvement in model robustness, as this can be viewed as a form of ensemble adversarial training. Although self-supervised pre-training yields benefits in improving adversarial training as compared to random weight initialization, we observe no benefit in model robustness or accuracy if self-supervision is incorporated into adversarial training.
Why risk analysis outperforms probabilistic choice as the effective decision support paradigm for power system planning

This paper demonstrates that a classical stochastic optimization is, in many cases, not convenient for power system planning. Instead, a risk analysis approach is proposed. In a comparison of both planning paradigms, the probabilistic approach is in occasions not adequate, is half blind to compromise solutions and leads, in numerous, cases to riskier decisions. The technical discussion is illustrated with a distribution planning example.
Edge spin transport in the disordered two-dimensional topological insulator WTe$_2$

The spin conductance of two-dimensional topological insulators (2D TIs) is not expected to be quantized in the presence of perturbations that break the spin-rotational symmetry. However, the deviation from the pristine-limit quantization has yet to be studied in detail. In this paper, we define the spin current operator for the helical edge modes of a 2D TI and introduce a four-terminal setup to measure spin conductances. Using the developed formalism, we consider the effects of disorder terms that break spin-rotational symmetry or give rise to edge-to-edge coupling. We identify a key role played by spin torque in an out-of-equilibrium edge. We then utilize a tight-binding model of topological monolayer WTe$_2$ and scattering matrix formalism to numerically study spin transport in a four-terminal 2D TI device. In particular, we calculate the spin conductances and characteristic spin decay length in the presence of magnetic disorder. In addition, we study the effects of inter-edge scattering in a quantum point contact geometry. We find that the spin Hall conductance is surprisingly robust to spin symmetry-breaking perturbations, as long as time-reversal symmetry is preserved and inter-edge scattering is weak.
What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods

A multitude of explainability methods has been described to try to help users better understand how modern AI systems make decisions. However, most performance metrics developed to evaluate these methods have remained largely theoretical - without much consideration for the human end-user. In particular, it is not yet clear (1) how useful current explainability methods are in real-world scenarios; and (2) whether current performance metrics accurately reflect the usefulness of explanation methods for the end user. To fill this gap, we conducted psychophysics experiments at scale (n = 1,150) to evaluate the usefulness of representative attribution methods in three real-world scenarios. Our results demonstrate that the degree to which individual attribution methods help human participants better understand an AI system varies widely across these scenarios. This suggests the need to move beyond quantitative improvements of current attribution methods, towards the development of complementary approaches that provide qualitatively different sources of information to human end-users.
Isoharmonic deformations and constrained Schlesinger systems

We introduce and study the dynamics of Chebyshev polynomials on d > 2 real intervals. We define isoharmonic deformations as a natural generalization of the Chebyshev dynamics. This dynamics is associated with a novel class of constrained isomonodromic deformations for which we derive the constrained Schlesinger equations. We provide explicit solutions to these equations in terms of differentials on an appropriate family of hyperelliptic curves of any genus g = d−1 ≥ 2. The verification of the obtained solutions relies on the combinatorial properties of the Bell polynomials and on the analysis on the Hurwitz spaces. From the point of view of the classical algebraic geometry we formulate and solve the problem of constrained Jacobi inversion for hyperelliptic curves. We discuss applications of the obtained results in integrable systems, e.g. billiards within ellipsoids in R. MSC: 30F30, 31A15, 32G15 (35Q07, 14H40, 14H70)
Relating transformers to models and neural representations of the hippocampal formation

Many deep neural network architectures loosely based on brain networks have recently been shown to replicate neural firing patterns observed in the brain. One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version. This work continues to bind computations of artificial and brain networks, offers a novel understanding of the hippocampal-cortical interaction, and suggests how wider cortical areas may perform complex tasks beyond current neuroscience models such as language comprehension.
Learning Theory Can (Sometimes) Explain Generalisation in Graph Neural Networks

In recent years, several results in the supervised learning setting suggested that classical statistical learning-theoretic measures, such as VC dimension, do not adequately explain the performance of deep learning models which prompted a slew of work in the infinite-width and iteration regimes. However, there is little theoretical explanation for the success of neural networks beyond the supervised setting. In this paper we argue that, under some distributional assumptions, classical learning-theoretic measures can sufficiently explain generalization for graph neural networks in the transductive setting. In particular, we provide a rigorous analysis of the performance of neural networks in the context of transductive inference, specifically by analysing the generalisation properties of graph convolutional networks for the problem of node classification. While VC Dimension does result in trivial generalisation error bounds in this setting as well, we show that transductive Rademacher complexity can explain the generalisation properties of graph convolutional networks for stochastic block models. We further use the generalisation error bounds based on transductive Rademacher complexity to demonstrate the role of graph convolutions and network architectures in achieving smaller generalisation error and provide insights into when the graph structure can help in learning. The findings of this paper could re-new the interest in studying generalisation in neural networks in terms of learning-theoretic measures, albeit in specific problems.
VIRT: Improving Representation-based Text Matching via Virtual Interaction

Text matching is a fundamental research problem in natural language understanding. Interaction-based approaches treat the text pair as a single sequence and encode it through cross encoders, while representation-based models encode the text pair independently with siamese or dual encoders. Interaction-based models require dense computations and thus are impractical in real-world applications. Representation-based models have become the mainstream paradigm for efficient text matching. However, these models suffer from severe performance degradation due to the lack of interactions between the pair of texts. To remedy this, we propose a Virtual InteRacTion mechanism (VIRT) for improving representation-based text matching while maintaining its efficiency. In particular, we introduce an interactive knowledge distillation module that is only applied during training. It enables deep interaction between texts by effectively transferring knowledge from the interaction-based model. A light interaction strategy is designed to fully leverage the learned interactive knowledge. Experimental results on six text matching benchmarks demonstrate the superior performance of our method over several state-of-the-art representation-based models. We further show that VIRT can be integrated into existing methods as plugins to lift their performances.
One-Point Statistics Matter in Extended Cosmologies

The late universe contains a wealth of information about fundamental physics and gravity, wrapped up in non-Gaussian fields. To make use of as much information as possible it is necessary to go beyond two-point statistics. Rather than going to higher order N-point correlation functions, we demonstrate that the probability distribution function (PDF) of spheres in the matter field (a one-point function) already contains a significant amount of this non-Gaussian information. The matter PDF dissects different density environments which are lumped together in two-point statistics, making it particularly useful for probing modifications of gravity or expansion history. Our approach in Cataneo et. al. 2021 extends the success of Large Deviation Theory for predicting the matter PDF in ΛCDM in these “extended” cosmologies. A Fisher forecast demonstrates the information content in the matter PDF via constraints for a Euclid-like survey volume combining the 3D matter PDF with the 3D matter power spectrum. Adding the matter PDF halves the uncertainties on parameters in an evolving dark energy model, relative to the power spectrum alone. Additionally, the matter PDF contains enough non-linear information to substantially increase the detection significance of departures from General Relativity, with improvements up to six times the power spectrum alone. This analysis demonstrates that the matter PDF is a promising non-Gaussian statistic for extracting cosmological information, particularly for beyond ΛCDM models.
Learning Music Audio Representations Via Weak Language Supervision

Audio representations for music information retrieval are typically learned via supervised learning in a task-specific fashion. Although effective at producing state-of-the-art results, this scheme lacks flexibility with respect to the range of applications a model can have and requires extensively annotated datasets. In this work, we pose the question of whether it may be possible to exploit weakly aligned text as the only supervisory signal to learn general-purpose music audio representations. To address this question, we design a multimodal architecture for music and language pre-training (MuLaP) optimised via a set of proxy tasks. Weak supervision is provided in the form of noisy natural language descriptions conveying the overall musical content of the track. After pre-training, we transfer the audio backbone of the model to a set of music audio classification and regression tasks. We demonstrate the usefulness of our approach by comparing the performance of audio representations produced by the same audio backbone with different training strategies and show that our pre-training method consistently achieves comparable or higher scores on all tasks and datasets considered. Our experiments also confirm that MuLaP effectively leverages audio-caption pairs to learn representations that are competitive with audio-only and cross-modal self-supervised methods in the literature.
Improving language models by retrieving from trillions of tokens

We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETRO from scratch, yet can also rapidly RETROfit pre-trained transformers with retrieval and still achieve good performance. Our work opens up new avenues for improving language models through explicit memory at unprecedented scale.
On the Rankin-Selberg $L$-factors for ${\rm SO}_{5}\times{\rm GL}_2$

Let π and τ be a smooth generic representation of SO5 and GL2 respectively over a nonarchimedean local field. Assume that π is irreducible and τ is irreducible or induced of Langlands’ type. We show that the Land ǫ-factors attached to π×τ defined by the Rankin-Selberg integrals and the associated Weil-Deligne representation coincide. Similar compatibility results are also obtained for the local factors defined through the Novodvorsky’s local zeta integrals that attached to generic representations of GSp4×GL2.
JABER: Junior Arabic BERt

Language-speciﬁc pre-trained models have proven to be more accurate than multilingual ones in a monolingual evaluation setting, Arabic is no exception. However, we found that previously released Arabic BERT models were signiﬁcantly under-trained. In this technical report, we present JABER, J unior Arabic BERt, our pretrained language model prototype dedicated for Arabic. We conduct an empirical study to systematically evaluate the performance of models across a diverse set of existing Arabic NLU tasks. Experimental results show that JABER achieves the state-of-the-art performances on ALUE, a new benchmark for Arabic Language Understanding Evaluation, as well as on a well-established NER benchmark.
Partial Okounkov bodies and Duistermaat--Heckman measures of non-Archimedean metrics

Let $X$ be a smooth projective variety. We construct partial Okounkov bodies associated to Hermitian pseudo-effective line bundles $(L,\phi)$ on $X$. We show that partial Okounkov bodies are universal invariants of the singularity of $\phi$. As an application, we generalize the theorem of Boucksom--Chen and construct Duistermaat--Heckman measures associated to finite energy metrics on the Berkovich analytification of an ample line bundle.
Heuristic algorithms for minimising total recovery cost of end-of-life products under quality constraints

Recently, the optimisation of end-of-life (EOL) product recovery processes has been highlighted. At the inspection phase after disassembly, each part can have various recovery options such as reuse, reconditioning, remanufacturing, and disposal. Depending on the selected options of parts, the values of recovered products that are made by reassembling parts will be different. Hence, it is important to decide appropriate recovery options of parts at the treatment of EOL products, in order to maximise the values of recovered products. To this end, this study deals with a decision making problem to select the best recovery options of parts for minimising the total recovery cost of products under quality constraints. This problem is formulated with a mixed integer nonlinear programming model and heuristic search algorithms are proposed to resolve it. A case study for a turbocharger product is introduced with computational experiments of the proposed algorithms.
Progressive Multi-stage Interactive Training in Mobile Network for Fine-grained Recognition

Fine-grained Visual Classification (FGVC) aims to identify objects from subcategories. It is a very challenging task because of the subtle inter-class differences. Existing research applies large-scale convolutional neural networks or visual transformers as the feature extractor, which is extremely computationally expensive. In fact, real-world scenarios of fine-grained recognition often require a more lightweight mobile network that can be utilized offline. However, the fundamental mobile network feature extraction capability is weaker than large-scale models. In this paper, based on the lightweight MobilenetV2, we propose a Progressive Multi-Stage Interactive training method with a Recursive Mosaic Generator (RMG-PMSI). First, we propose a Recursive Mosaic Generator (RMG) that generates images with different granularities in different phases. Then, the features of different stages pass through a Multi-Stage Interaction (MSI) module, which strengthens and complements the corresponding features of different stages. Finally, using the progressive training (P), the features extracted by the model in different stages can be fully utilized and fused with each other. Experiments on three prestigious fine-grained benchmarks show that RMG-PMSI can significantly improve the performance with good robustness and transferability.
Robust parameter estimation of regression model with weaker moment∗

This paper provides some extended results on estimating the parameter matrix of high-dimensional regression model when the covariate or response possess weaker moment condition. We investigate the M -estimator of Fan et al. (Ann Stat 49(3):1239–1266, 2021) for matrix completion model with (1 + ǫ)-th moments. The corresponding phase transition phenomenon is observed. When ǫ ≥ 1, the robust estimator possesses the same convergence rate as previous literature. While 1 > ǫ > 0, the rate will be slower. For high dimensional multiple index coefficient model, we also apply the element-wise truncation method to construct a robust estimator which handle missing and heavy-tailed data with finite fourth moment.
ExoMiner: A Highly Accurate and Explainable Deep Learning Classifier That Validates 301 New Exoplanets

The Kepler and Transiting Exoplanet Survey Satellite (TESS) missions have generated over 100,000 potential transit signals that must be processed in order to create a catalog of planet candidates. During the past few years, there has been a growing interest in using machine learning to analyze these data in search of new exoplanets. Different from the existing machine learning works, ExoMiner, the proposed deep learning classifier in this work, mimics how domain experts examine diagnostic tests to vet a transit signal. ExoMiner is a highly accurate, explainable, and robust classifier that (1) allows us to validate 301 new exoplanets from the MAST Kepler Archive and (2) is general enough to be applied across missions such as the ongoing TESS mission. We perform an extensive experimental study to verify that ExoMiner is more reliable and accurate than the existing transit signal classifiers in terms of different classification and ranking metrics. For example, for a fixed precision value of 99%, ExoMiner retrieves 93.6% of all exoplanets in the test set (i.e., recall = 0.936), while this rate is 76.3% for the best existing classifier. Furthermore, the modular design of ExoMiner favors its explainability. We introduce a simple explainability framework that provides experts with feedback on why ExoMiner classifies a transit signal into a specific class label (e.g., planet candidate or not planet candidate).
Rapid voltage changes - definition and minimum requirements

Rapid voltage change (RVC) is one of the voltage disturbances classified under the term voltage quality and is included in several standards internationally but without a fully unique definition. This paper describes the Norwegian regulator's motive for introducing requirements for RVCs in the public electricity quality of supply regulation. Furthermore, a research project is described with focus on which rapid voltage changes that are visible in lighting. This project gave a sound basis for providing a unique definition on RVCs, and the definition is also presented. Finally, the paper describes the regulator minimum requirements for RVCs.