Miracle-dz/UniFAR-FTUs huggingface dataset:
GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures

Attention based language models have become a critical component in state-of-the-art natural language processing systems. However, these models have significant computational requirements, due to long training times, dense operations and large parameter count. In this work we demonstrate a set of modifications to the structure of a Transformer layer, producing a more efficient architecture. First, we add a convolutional module to complement the self-attention module, decoupling the learning of local and global interactions. Secondly, we rely on grouped transformations to reduce the computational cost of dense feed-forward layers and convolutions, while preserving the expressivity of the model. We apply the resulting architecture to language representation learning and demonstrate its superior performance compared to BERT models of different scales. We further highlight its improved efficiency, both in terms of floating-point operations (FLOPs) and time-to-train.
Detection of Ongoing Mass Loss from HD 63433c, a Young Mini-Neptune

We detect Lyα absorption from the escaping atmosphere of HD 63433c, a R = 2.67R ⊕, P = 20.5 day mini-Neptune orbiting a young (440 Myr) solar analog in the Ursa Major Moving Group. Using Hubble Space Telescope (HST)/Space Telescope Imaging Spectrograph, we measure a transit depth of 11.1 ± 1.5% in the blue wing and 8 ± 3% in the red. This signal is unlikely to be due to stellar variability, but should be confirmed by an upcoming second transit observation with HST. We do not detect Lyα absorption from the inner planet, a smaller R = 2.15R ⊕ mini-Neptune on a 7.1 day orbit. We use Keck/NIRSPEC to place an upper limit of 0.5% on helium absorption for both planets. We measure the host star’s X-ray spectrum and mid-ultraviolet flux with XMM-Newton, and model the outflow from both planets using a 3D hydrodynamic code. This model provides a reasonable match to the light curve in the blue wing of the Lyα line and the helium nondetection for planet c, although it does not explain the tentative red wing absorption or reproduce the excess absorption spectrum in detail. Its predictions of strong Lyα and helium absorption from b are ruled out by the observations. This model predicts a much shorter mass-loss timescale for planet b, suggesting that b and c are fundamentally different: while the latter still retains its hydrogen/helium envelope, the former has likely lost its primordial atmosphere.
Online Learning for Stochastic Shortest Path Model via Posterior Sampling

We consider the problem of online reinforcement learning for the Stochastic Shortest Path (SSP) problem modeled as an unknown MDP with an absorbing state. We propose PSRL-SSP, a simple posterior sampling-based reinforcement learning algorithm for the SSP problem. The algorithm operates in epochs. At the beginning of each epoch, a sample is drawn from the posterior distribution on the unknown model dynamics, and the optimal policy with respect to the drawn sample is followed during that epoch. An epoch completes if either the number of visits to the goal state in the current epoch exceeds that of the previous epoch, or the number of visits to any of the state-action pairs is doubled. We establish a Bayesian regret bound of $O(B_\star S\sqrt{AK})$, where $B_\star$ is an upper bound on the expected cost of the optimal policy, $S$ is the size of the state space, $A$ is the size of the action space, and $K$ is the number of episodes. The algorithm only requires the knowledge of the prior distribution, and has no hyper-parameters to tune. It is the first such posterior sampling algorithm and outperforms numerically previously proposed optimism-based algorithms.
Fair Normalizing Flows

Fair representation learning is an attractive approach that promises fairness of downstream predictors by encoding sensitive data. Unfortunately, recent work has shown that strong adversarial predictors can still exhibit unfairness by recovering sensitive attributes from these representations. In this work, we present Fair Normalizing Flows (FNF), a new approach offering more rigorous fairness guarantees for learned representations. Specifically, we consider a practical setting where we can estimate the probability density for sensitive groups. The key idea is to model the encoder as a normalizing flow trained to minimize the statistical distance between the latent representations of different groups. The main advantage of FNF is that its exact likelihood computation allows us to obtain guarantees on the maximum unfairness of any potentially adversarial downstream predictor. We experimentally demonstrate the effectiveness of FNF in enforcing various group fairness notions, as well as other attractive properties such as interpretability and transfer learning, on a variety of challenging real-world datasets.
Early-stopped neural networks are consistent

This work studies the behavior of shallow ReLU networks trained with the logistic loss via gradient descent on binary classification data where the underlying data distribution is general, and the (optimal) Bayes risk is not necessarily zero. In this setting, it is shown that gradient descent with early stopping achieves population risk arbitrarily close to optimal in terms of not just logistic and misclassification losses, but also in terms of calibration, meaning the sigmoid mapping of its outputs approximates the true underlying conditional distribution arbitrarily finely. Moreover, the necessary iteration, sample, and architectural complexities of this analysis all scale naturally with a certain complexity measure of the true conditional model. Lastly, while it is not shown that early stopping is necessary, it is shown that any univariate classifier satisfying a local interpolation property is inconsistent.
An empirical calibration of the helium abundance in H ii regions based in literature and CALIFA survey data

Helium is the second most common chemical species in the Universe. The study of helium abundance has the potential to unravel the chemical evolution of and within galaxies. In this study, we provide an empirical calibration for the singly ionized helium abundance: 12 + log10 (He+/H+), based on the emission line flux ratio Heλ5876/Hα from Galactic and extragalactic H II regions compiled from the literature. Based on this calibrator, we explore for the first time the helium abundance in a large sample of H II regions located in galaxies representative of the nearby Universe from the CALIFA survey. Furthermore, this calibrator allows us to explore the variations of the helium abundance with respect to the oxygen abundance. The observed trends are in agreement with a change in the chemical enrichment with mass/oxygen abundance similar to the one observed due to the inside-out model in a MW-galaxy (highlighting the connection between resolved and global trends in galaxies). Our calibrator provides an empirical proxy to estimate the helium abundance at kpc scales as well as to constrain chemical evolutionary models.
Explaining Time Series Predictions with Dynamic Masks

How can we explain the predictions of a machine learning model? When the data is structured as a multivariate time series, this question induces additional difficulties such as the necessity for the explanation to embody the time dependency and the large number of inputs. To address these challenges, we propose dynamic masks (Dynamask). This method produces instance-wise importance scores for each feature at each time step by fitting a perturbation mask to the input sequence. In order to incorporate the time dependency of the data, Dynamask studies the effects of dynamic perturbation operators. In order to tackle the large number of inputs, we propose a scheme to make the feature selection parsimonious (to select no more feature than necessary) and legible (a notion that we detail by making a parallel with information theory). With synthetic and real-world data, we demonstrate that the dynamic underpinning of Dynamask, together with its parsimony, offer a neat improvement in the identification of feature importance over time. The modularity of Dynamask makes it ideal as a plug-in to increase the transparency of a wide range of machine learning models in areas such as medicine and finance, where time series are abundant.
Bohmian Zitterbewegung

A new Bohmian quantum-relativistic model, in which from the Klein-Gordon equation a generalization of the standard Zitterbewegung arises, is explored. It is obtained by introducing a new independent time parameter, whose relative motions are not directly observable but cause the quantum uncertainties of the observables. Unlike Bohm's original theory, the quantum potential does not affect the observable motion, as for a normal external potential, but it only determines that one relative to the new time variable, of which the Zitterbewegung of a free particle is an example. The model also involves a relativistic revision of the uncertainty principle.
NetFC: Enabling Accurate Floating-point Arithmetic on Programmable Switches

Programmable switches are recently used for accelerating data-intensive distributed applications. Some computational tasks, traditionally performed on servers in data centers, are offloaded to the network on programmable switches. These tasks may require the support of on-the-fly floatingpoint operations. Unfortunately, the computational capacity of programmable switches is limited to simple integer arithmetic operations. To address this issue, prior approaches either adopt a float-to-integer method or rely on local CPUs of switches, incurring accuracy loss and delayed processing.To this end, we propose NetFC, a table-lookup method to achieve on-the-fly in-network floating-point arithmetic operations nearly without accuracy loss. NetFC adopts a divide-and-conquer mechanism that converts the original huge table into several much smaller tables that are operated by the built-in integer operations. NetFC further leverages a scaling-factor mechanism for improving computational accuracy, and a prefix-based lossless table compression method to reduce memory consumption. We use both synthetic and real-life datasets to evaluate NetFC. The experimental results show that the average accuracy of NetFC is above 99.94% with only 448KB memory consumption. Furthermore, we integrate NetFC into Sonata [12] for detecting Slowloris attack, yielding significant decrease of detection delay.
Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses

How related are the representations learned by neural language models, translation models, and language tagging tasks? We answer this question by adapting an encoder-decoder transfer learning method from computer vision to investigate the structure among 100 different feature spaces extracted from hidden representations of various networks trained on language tasks. This method reveals a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We call this low-dimensional structure a language representation embedding because it encodes the relationships between representations needed to process language for a variety of NLP tasks. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. Additionally, we find that the principal dimension of this structure can be used to create a metric which highlights the brain's natural language processing hierarchy. This suggests that the embedding captures some part of the brain's natural language representation structure.
Distribution-Aware Semantics-Oriented Pseudo-label for Imbalanced Semi-Supervised Learning

The capability of the traditional semi-supervised learning (SSL) methods is far from real-world application since they do not consider (1) class imbalance and (2) class distribution mismatch between labeled and unlabeled data. This paper addresses such a relatively under-explored problem, imbalanced semi-supervised learning , where heavily biased pseudo-labels can harm the model performance. Interestingly, we ﬁnd that the semantic pseudo-labels from a similarity-based classiﬁer in feature space and the traditional pseudo-labels from the linear classiﬁer show the complementary property. To this end, we propose a general pseudo-labeling framework to address the bias motivated by this observation. The key idea is to class-adaptively blend the semantic pseudo-label to the linear one, depending on the current pseudo-label distribution. Thereby, the increased semantic pseudo-label component suppresses the false positives in the majority classes and vice versa. We term the novel pseudo-labeling framework for imbalanced SSL as D istribution-A ware S emantics-O riented (DASO) Pseudo-label. Extensive evaluation on CIFAR10/100-LT and STL10-LT shows that DASO consistently outperforms both recently proposed re-balancing methods for label and pseudo-label. Moreover, we demonstrate that typical SSL algorithms can effectively beneﬁt from unlabeled data with DASO, especially when (1) class imbalance and (2) class distribution mismatch exist and even on recent real-world Semi-Aves benchmark.
AggGen: Ordering and Aggregating while Generating

We present AggGen (pronounced ‘again’) a data-to-text model which re-introduces two explicit sentence planning stages into neural data-to-text systems: input ordering and input aggregation. In contrast to previous work using sentence planning, our model is still end-to-end: AggGen performs sentence planning at the same time as generating text by learning latent alignments (via semantic facts) between input representation and target text. Experiments on the WebNLG and E2E challenge data show that by using fact-based alignments our approach is more interpretable, expressive, robust to noise, and easier to control, while retaining the advantages of end-to-end systems in terms of fluency. Our code is available at https://github.com/XinnuoXu/AggGen.
Solution of Wiener-Hopf and Fredholm integral equations by fast Hilbert and Fourier transforms

We present numerical methods based on the fast Fourier transform (FFT) to solve convolution integral equations on a semi-infinite interval (Wiener-Hopf equation) or on a finite interval (Fredholm equation). We extend and improve a FFT-based method for the Wiener-Hopf equation due to Henery, expressing it in terms of the Hilbert transform, and computing the latter in a more sophisticated way with sinc functions. We then generalise our method to the Fredholm equation reformulating it as two coupled Wiener-Hopf equations and solving them iteratively. We provide numerical tests and open-source code.
Well-posedness and averaging principle of McKean-Vlasov SPDEs driven by cylindrical $\alpha$-stable process

Abstract. In this paper, we first study the well-posedness of a class of McKean-Vlasov stochastic partial differential equations driven by cylindrical α-stable process, where α ∈ (1, 2). Then by the method of the Khasminskii’s time discretization, we prove the averaging principle of a class of multiscale McKean-Vlasov stochastic partial differential equations driven by cylindrical α-stable processes. Meanwhile, we obtain a specific strong convergence rate.
MST: Masked Self-Supervised Transformer for Visual Representation

Transformer has been widely used for self-supervised pre-training in Natural Language Processing (NLP) and achieved great success. However, it has not been fully explored in visual self-supervised learning. Meanwhile, previous methods only consider the high-level feature and learning representation from a global perspective, which may fail to transfer to the downstream dense prediction tasks focusing on local features. In this paper, we present a novel Masked Self-supervised Transformer approach named MST, which can explicitly capture the local context of an image while preserving the global semantic information. Specifically, inspired by the Masked Language Modeling (MLM) in NLP, we propose a masked token strategy based on the multi-head self-attention map, which dynamically masks some tokens of local patches without damaging the crucial structure for self-supervised learning. More importantly, the masked tokens together with the remaining tokens are further recovered by a global image decoder, which preserves the spatial information of the image and is more friendly to the downstream dense prediction tasks. The experiments on multiple datasets demonstrate the effectiveness and generality of the proposed method. For instance, MST achieves Top-1 accuracy of 76.9% with DeiT-S only using 300-epoch pre-training by linear evaluation, which outperforms supervised methods with the same epoch by 0.4% and its comparable variant DINO by 1.0\%. For dense prediction tasks, MST also achieves 42.7% mAP on MS COCO object detection and 74.04% mIoU on Cityscapes segmentation only with 100-epoch pre-training.
Data Clustering-Driven Neural Network for Intra Prediction

As a crucial part of video compression, intra prediction utilizes local information of images to eliminate the redundancy in spatial domain. In both H.265/HEVC and H.266/VVC, multiple directional prediction modes are employed to find the texture trend of each small block and then the prediction is made based on reference samples in the selected direction. Recently, the intra prediction schemes based on neural networks have achieved great success. In these methods, the networks are trained and applied to intra prediction in addition to the directional prediction modes. In this paper, we propose a novel data clustering-driven neural network (dubbed DCDNN) for intra prediction, which can learn deep features of the clustered data. In DCDNN, each network can be split into two networks by adding or subtracting Gaussian random noise. Then a data clustering-driven training is applied to train all the derived networks recursively. In each iteration, the entire training dataset is partitioned according to the recovery qualities of the derived networks. For the experiment, DCDNN is implemented into HEVC reference software HM-16.9. The experimental results demonstrate that DCDNN can reach an average of 4.2% Bjontegaard distortion rate (BDrate) improvement (up to 7.0%) over HEVC with all intra configuration. Compared with existing fully connected networkbased intra prediction methods, the bitrate saving performance is further improved.
Support Recovery of Sparse Signals from a Mixture of Linear Measurements

Recovery of support of a sparse vector from simple measurements is a widely-studied problem, considered under the frameworks of compressed sensing, 1-bit compressed sensing, and more general single index models. We consider generalizations of this problem: mixtures of linear regressions, and mixtures of linear classifiers, where the goal is to recover supports of multiple sparse vectors using only a small number of possibly noisy linear, and 1-bit measurements respectively. The key challenge is that the measurements from different vectors are randomly mixed. Both of these problems have also received attention recently. In mixtures of linear classifiers, the observations correspond to the side of queried hyperplane a random unknown vector lies in, whereas in mixtures of linear regressions we observe the projection of a random unknown vector on the queried hyperplane. The primary step in recovering the unknown vectors from the mixture is to first identify the support of all the individual component vectors. In this work, we study the number of measurements sufficient for recovering the supports of all the component vectors in a mixture in both these models. We provide algorithms that use a number of measurements polynomial in $k, \log n$ and quasi-polynomial in $\ell$, to recover the support of all the $\ell$ unknown vectors in the mixture with high probability when each individual component is a $k$-sparse $n$-dimensional vector.
Hadronization model tuning in genie v3

(GENIE Collaboration) University of Liverpool, Dept. of Physics, Liverpool L69 7ZE, UK Science and Technology Facilities Council, Rutherford Appleton Laboratory, Particle Physics Dept., Oxfordshire OX11 0QX, UK University of Pittsburgh, Dept. of Physics and Astronomy, Pittsburgh PA 15260, USA Tufts University, Dept. of Physics and Astronomy, Medford MA 02155, USA Fermi National Accelerator Laboratory, Batavia, Illinois 60510, USA Massachusetts Institute of Technology, Dept. of Physics, Cambridge, MA 02139, USA Joint Institute for Nuclear Research (JINR), Dubna, Moscow region, 141980, Russia Alikhanov Institute for Theoretical and Experimental Physics (ITEP) of NRC “Kurchatov Institute”, Moscow, 117218, Russia CENBG, Université de Bordeaux, CNRS/IN2P3, 33175 Gradignan, France University of Liverpool, Dept. of Physics, Liverpool L69 7ZE, UK (Dated: June 11, 2021)
Probing transfer learning with a model of synthetic correlated datasets

Transfer learning can significantly improve the sample efficiency of neural networks, by exploiting the relatedness between a data-scarce target task and a data-abundant source task. Despite years of successful applications, transfer learning practice often relies on ad-hoc solutions, while theoretical understanding of these procedures is still limited. In the present work, we re-think a solvable model of synthetic data as a framework for modeling correlation between data-sets. This setup allows for an analytic characterization of the generalization performance obtained when transferring the learned feature map from the source to the target task. Focusing on the problem of training two-layer networks in a binary classification setting, we show that our model can capture a range of salient features of transfer learning with real data. Moreover, by exploiting parametric control over the correlation between the two data-sets, we systematically investigate under which conditions the transfer of features is beneficial for generalization.
Programming Puzzles

We introduce a new type of programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis, and release an open-source dataset of Python Programming Puzzles (P3). Each puzzle is defined by a short Python program $f$, and the goal is to find an input which makes $f$ return True. The puzzles are objective in that each one is specified entirely by the source code of its verifier $f$, so evaluating $f$ is all that is needed to test a candidate solution. They do not require an answer key or input/output examples, nor do they depend on natural language understanding. The dataset is comprehensive in that it spans problems of a range of difficulties and domains, ranging from trivial string manipulation problems, to classic programming puzzles (e.g., Tower of Hanoi), to interview/competitive-programming problems (e.g., dynamic programming), to longstanding open problems in algorithms and mathematics (e.g., factoring). We develop baseline enumerative program synthesis, GPT-3 and Codex solvers that are capable of solving puzzles -- even without access to any reference solutions -- by learning from their own past solutions. Codex performs best, solving up to 18% of 397 test problems with a single try and 80% of the problems with 1,000 tries per problem. In a small user study, we find a positive correlation between puzzle-solving performance and coding experience, and between the puzzle difficulty for humans and AI solvers. Therefore, further improvements on P3 could have a significant impact on many program synthesis areas.