Miracle-dz/UniFAR-FTUs huggingface dataset:
Geometry-aware Instance-reweighted Adversarial Training

In adversarial machine learning, there was a common belief that robustness and accuracy hurt each other. The belief was challenged by recent studies where we can maintain the robustness and improve the accuracy. However, the other direction, whether we can keep the accuracy while improving the robustness, is conceptually and practically more interesting, since robust accuracy should be lower than standard accuracy for any model. In this paper, we show this direction is also promising. Firstly, we find even over-parameterized deep networks may still have insufficient model capacity, because adversarial training has an overwhelming smoothing effect. Secondly, given limited model capacity, we argue adversarial data should have unequal importance: geometrically speaking, a natural data point closer to/farther from the class boundary is less/more robust, and the corresponding adversarial data point should be assigned with larger/smaller weight. Finally, to implement the idea, we propose geometry-aware instance-reweighted adversarial training, where the weights are based on how difficult it is to attack a natural data point. Experiments show that our proposal boosts the robustness of standard adversarial training; combining two directions, we improve both robustness and accuracy of standard adversarial training.
Mastering Atari with Discrete World Models

Intelligent agents need to generalize from past experience to achieve goals in complex environments. World models facilitate such generalization and allow learning behaviors from imagined outcomes to increase sample-efficiency. While learning world models from image inputs has recently become feasible for some tasks, modeling Atari games accurately enough to derive successful behaviors has remained an open challenge for many years. We introduce DreamerV2, a reinforcement learning agent that learns behaviors purely from predictions in the compact latent space of a powerful world model. The world model uses discrete representations and is trained separately from the policy. DreamerV2 constitutes the first agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model. With the same computational budget and wall-clock time, DreamerV2 reaches 200M frames and exceeds the final performance of the top single-GPU agents IQN and Rainbow.
Calibrating the Galactic Cepheid Period–Luminosity Relation from the Maximum-likelihood Technique

We present a realization of the maximum-likelihood technique, which is one of the latest modifications of the Baade–Becker–Wesselink method. Our approach is based on nonlinear calibrations of the effective temperature and bolometric correction, which take into account metallicity and surface gravity. It allows one to estimate the key Cepheid parameters, the distance modulus, and the interstellar reddening, combining photometric and spectroscopic data (including the effective temperature data). This method is applied to a sample of 44 Galactic Cepheids for which multiphase temperature measurements are available. The additional data correction is performed to subtract the impact of the component in binary/multiple systems. We also study the effect of shock waves, whose presence in the stellar atmosphere distorts the observational data and leads to systematic errors in the obtained parameters. We determine the optimal restriction on the input data to eliminate this effect. This restriction provides accurate period–radius and period–luminosity relations that are consistent with the results in previous studies. We found the following relations: log R = (0.68 ± 0.03) · log P + (1.14 ± 0.03), Mv = − (2.67 ± 0.16) · (log P − 1) − (4.14 ± 0.05).
Gravitational spin-orbit and aligned 
spin1−spin2
 couplings through third-subleading post-Newtonian orders

The study of scattering encounters continues to provide new insights into the general relativistic two-body problem. The local-in-time conservative dynamics of an aligned-spin binary, for both unbound and bound orbits, is fully encoded in the gauge-invariant scattering-angle function, which is most naturally expressed in a post-Minkowskian (PM) expansion, and which exhibits a remarkably simple dependence on the masses of the two bodies (in terms of appropriate geometric variables). This dependence links the PM and small-mass-ratio approximations, allowing gravitational self-force results to determine new post-Newtonian (PN) information to all orders in the mass ratio. In this paper, we exploit this interplay between relativistic scattering and self-force theory to obtain the third-subleading (4.5PN) spin-orbit dynamics for generic spins, and the third-subleading (5PN) spin$_1$-spin$_2$ dynamics for aligned spins. We further implement these novel PN results in an effective-one-body framework, and demonstrate the improvement in accuracy by comparing against numerical-relativity simulations.
The act of remembering: a study in partially observable reinforcement learning

Reinforcement Learning (RL) agents typically learn memoryless policies---policies that only consider the last observation when selecting actions. Learning memoryless policies is efficient and optimal in fully observable environments. However, some form of memory is necessary when RL agents are faced with partial observability. In this paper, we study a lightweight approach to tackle partial observability in RL. We provide the agent with an external memory and additional actions to control what, if anything, is written to the memory. At every step, the current memory state is part of the agent's observation, and the agent selects a tuple of actions: one action that modifies the environment and another that modifies the memory. When the external memory is sufficiently expressive, optimal memoryless policies yield globally optimal solutions. Unfortunately, previous attempts to use external memory in the form of binary memory have produced poor results in practice. Here, we investigate alternative forms of memory in support of learning effective memoryless policies. Our novel forms of memory outperform binary and LSTM-based memory in well-established partially observable domains.
Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media

Recent studies on domain-specific BERT models show that effectiveness on downstream tasks can be improved when models are pretrained on in-domain data. Often, the pretraining data used in these models are selected based on their subject matter, e.g., biology or computer science. Given the range of applications using social media text, and its unique language variety, we pretrain two models on tweets and forum text respectively, and empirically demonstrate the effectiveness of these two resources. In addition, we investigate how similarity measures can be used to nominate in-domain pretraining data. We publicly release our pretrained models at https://bit.ly/35RpTf0.
On the Effects of Knowledge-Augmented Data in Word Embeddings

This paper investigates techniques for knowledge injection into word embeddings learned from large corpora of unannotated data. These representations are trained with word cooccurrence statistics and do not commonly exploit syntactic and semantic information from linguistic knowledge bases, which potentially limits their transferability to domains with differing language distributions or usages. We propose a novel approach for linguistic knowledge injection through data augmentation to learn word embeddings that enforce semantic relationships from the data, and systematically evaluate the impact it has on the resulting representations. We show our knowledge augmentation approach improves the intrinsic characteristics of the learned embeddings while not significantly altering their results on a downstream text classification task.
Modified gravity or imperfect dark matter: a model-independent discrimination

We analyze how to parametrize general modifications of the dark matter perturbations equations in a model-independent way. We prove that a general model with an imperfect and non-conserved dark matter fluid with bulk and shear viscosities and heat flux in a modified gravity scenario can be described with five general functions of time and scale. We focus on the sub-Hubble regime within the quasi-static approximation and calculate the observable power spectra of the galaxy distribution, galaxy velocities and weak lensing and find that these observables are only sensitive to three combinations of the initial five functions. Deviations of these three observable functions with respect to ΛCDM give us different characteristic signals which allow us to determine in which cases it is possible to discriminate a modification of gravity from an imperfect or non-conserved dark matter. Finally, we perform a Fisher forecast analysis for these three parameters and show an example for a particular model with shear viscosity.
The X-Ray Fundamental Plane of the Platinum Sample, the Kilonovae, and the SNe Ib/c Associated with GRBs

A large fraction of gamma-ray burst (GRB) lightcurves (LCs) show X-ray plateaus. We analyze all GRBs with known redshifts presenting plateaus observed by The Neil Gehrels Swift Observatory from its launch until 2019 August. The fundamental plane relation between the rest-frame time and X-ray luminosity at the end of the plateau emission and the peak prompt luminosity holds for all the GRB classes when selection biases and cosmological evolutions are applied. We have discovered two important findings: (1) a new class of long GRBs with good data coverage: the platinum sample; and (2) the platinum, the SNe-LGRB, and the KN-SGRB samples, yield the smallest intrinsic scatter with σ platinum,GRB−SNe = 0.22 ± 0.10 and σ KN−SGRB = 0.24 ± 0.12. The SNe-LGRBs are composed of GRBs associated spectroscopically with the SNe Ib,c, the KN-SGRBs are composed by eight GRBs associated with kilonovae or where there could have been such an association. The highest correlation coefficients are yielded for the SN-LGRB-ABC sample, which includes GRBs spectroscopically associated with SNe Ib/c or with a clear optical bump in the LC resembling the SNe Ib/c, ( ), the SN-LGRBs ( ), and the KN-SGRBs ( ) when the redshift evolution is considered. These category planes are reliable candidates to use as cosmological tools. Furthermore, the distance from the gold fundamental plane is a crucial discriminant among classes. In fact, we find that the distributions of the distances of the SNe-LGRB, SNe-LGRB-ABC, KN-SGRB, and SGRB samples from the gold fundamental plane are statistically different from the distribution of the gold GRBs’ distances from the gold fundamental plane with and without considering evolution cases.
First antineutrino energy spectrum from 235U fissions with the STEREO detector at ILL

This article reports the measurement of the $^{235}$U-induced antineutrino spectrum shape by the STEREO experiment. 43'000 antineutrinos have been detected at about 10 m from the highly enriched core of the ILL reactor during 118 full days equivalent at nominal power. The measured inverse beta decay spectrum is unfolded to provide a pure $^{235}$U spectrum in antineutrino energy. A careful study of the unfolding procedure, including a cross-validation by an independent framework, has shown that no major biases are introduced by the method. A significant local distortion is found with respect to predictions around $E_\nu \simeq 5.3$ MeV. A gaussian fit of this local excess leads to an amplitude of $A = 12.1 \pm 3.4\%$ (3.5$\sigma$).
Magnetic field dependent cycloidal rotation in pristine and Ge-doped 
CoCr2O4

We report a soft x-ray resonant magnetic scattering study of the spin configuration in multiferroic thin films of Co$_{0.975}$Ge$_{0.025}$Cr$_2$O$_4$ (Ge-CCO) and CoCr$_2$O$_4$ (CCO), under low- and high-magnetic fields, from 0.2 T up to 6.5 T. A characterization of Ge-CCO at a low magnetic field is performed and the results are compared to those of pure CCO. The ferrimagnetic phase transition temperature $T_C \approx 95$ K and the multiferroic transition temperature $T_S \approx 27$ K in Ge-CCO are comparable to those observed in CCO. In Ge-CCO, the ordering wave vector $\textit{(qq0)}$ observed below $T_S$ is slightly larger compared to that of CCO, and, unlike CCO, the diffraction intensity consists of two contributions that show a dissimilar x-ray polarization dependence. In Ge-CCO, the coercive field observed at low temperatures was larger than the one reported for CCO. In both compounds, an unexpected reversal of the spiral helicity and therefore the electric polarization was observed on simply magnetic field cooling. In addition, we find a change in the helicity as a function of momentum transfer in the magnetic diffraction peak of Ge-CCO, indicative of the presence of multiple magnetic spirals.
High-Resolution Piano Transcription With Pedals by Regressing Onset and Offset Times

Automatic music transcription (AMT) is the task of transcribing audio recordings into symbolic representations. Recently, neural network-based methods have been applied to AMT, and have achieved state-of-the-art results. However, many previous systems only detect the onset and offset of notes frame-wise, so the transcription resolution is limited to the frame hop size. There is a lack of research on using different strategies to encode onset and offset targets for training. In addition, previous AMT systems are sensitive to the misaligned onset and offset labels of audio recordings. Furthermore, there are limited researches on sustain pedal transcription on large-scale datasets. In this article, we propose a high-resolution AMT system trained by regressing precise onset and offset times of piano notes. At inference, we propose an algorithm to analytically calculate the precise onset and offset times of piano notes and pedal events. We show that our AMT system is robust to the misaligned onset and offset labels compared to previous systems. Our proposed system achieves an onset F1 of 96.72% on the MAESTRO dataset, outperforming previous onsets and frames system of 94.80%. Our system achieves a pedal onset F1 score of 91.86%, which is the first benchmark result on the MAESTRO dataset. We have released the source code and checkpoints of our work at https://github.com/bytedance/piano_transcription
On the Gauge Invariance of Scalar Induced Gravitational Waves: Gauge Fixings Considered

The energy density spectrum is an observable of gravitational waves. Divergence has appeared in the energy density spectra of the scalar induced gravitational waves for different gauge fixings. To resolve the discrepancy, we investigate the gauge invariance of the scalar induced gravitational waves. It is shown that the gauge invariant induced gravitational waves can be obtained by subtracting the fictitious tensor perturbations via introducing the counter term composed of the first order scalar perturbations. The kernel function uniquely determines the energy density spectrum of the scalar induced gravitational waves. We explicitly calculate the gauge invariant kernel functions in the Newtonian gauge and the uniform density gauge, respectively. The discrepancy between the energy density spectra upon the Newtonian gauge and the uniform density gauge is shown to be eliminated in the gauge invariant framework. In fact, the gauge invariant approach is also available to other kinds of gauge fixings.
The Grammar of Emergent Languages

In this paper, we consider the syntactic properties of languages emerged in referential games, using unsupervised grammar induction (UGI) techniques originally designed to analyse natural language. We show that the considered UGI techniques are appropriate to analyse emergent languages and we then study if the languages that emerge in a typical referential game setup exhibit syntactic structure, and to what extent this depends on the maximum message length and number of symbols that the agents are allowed to use. Our experiments demonstrate that a certain message length and vocabulary size are required for structure to emerge, but they also illustrate that more sophisticated game scenarios are required to obtain syntactic properties more akin to those observed in human language. We argue that UGI techniques should be part of the standard toolkit for analysing emergent languages and release a comprehensive library to facilitate such analysis for future researchers.
Spectropolarimetry of Primitive Phototrophs as Global Surface Biosignatures

Photosynthesis is an ancient metabolic process that began on early Earth and offers plentiful energy to organisms that can utilize it such that that they achieve global significance. The potential exists for similar processes to operate on habitable exoplanets and result in observable biosignatures. Before the advent of oxygenic photosynthesis, the most primitive phototrophs, anoxygenic phototrophs, dominated surface environments on the planet. Here, we characterize surface polarization biosignatures associated with a diverse sample of anoxygenic phototrophs and cyanobacteria, examining both pure cultures and microbial communities from the natural environment. Polarimetry is a tool that can be used to measure the chiral signature of biomolecules. Chirality is considered a universal, agnostic biosignature that is independent of a planet's biochemistry, receiving considerable interest as a target biosignature for life-detection missions. In contrast to preliminary indications from earlier work, we show that there is a diversity of distinctive circular polarization signatures, including the magnitude of the polarization, associated with the variety of chiral photosynthetic pigments and pigment complexes of anoxygenic and oxygenic phototrophs. We also show that the apparent death and release of pigments from one of the phototrophs is accompanied by an elevation of the reflectance polarization signal by an order of magnitude, which may be significant for remotely detectable environmental signatures. This work and others suggest that circular polarization signals up to ∼1% may occur, significantly stronger than previously anticipated circular polarization levels. We conclude that global surface polarization biosignatures may arise from anoxygenic and oxygenic phototrophs, which have dominated nearly 80% of the history of our rocky, inhabited planet.
Lipschitz Bounded Equilibrium Networks

This paper introduces new parameterizations of equilibrium neural networks, i.e. networks defined by implicit equations. This model class includes standard multilayer and residual networks as special cases. The new parameterization admits a Lipschitz bound during training via unconstrained optimization: no projections or barrier functions are required. Lipschitz bounds are a common proxy for robustness and appear in many generalization bounds. Furthermore, compared to previous works we show well-posedness (existence of solutions) under less restrictive conditions on the network weights and more natural assumptions on the activation functions: that they are monotone and slope restricted. These results are proved by establishing novel connections with convex optimization, operator splitting on non-Euclidean spaces, and contracting neural ODEs. In image classification experiments we show that the Lipschitz bounds are very accurate and improve robustness to adversarial attacks.
Bayesian Fixed-domain Asymptotics: Bernstein-von Mises Theorem for Covariance Parameters in a Gaussian Process Model.

Gaussian process models typically contain finite dimensional parameters in the covariance functions that need to be estimated from the data. We study the Bayesian fixed-domain asymptotic properties of the covariance parameters in a Gaussian process with an isotropic Matern covariance function. Under fixed-domain asymptotics, it is well known that when the dimension of data is less than or equal to three, the microergodic parameter can be consistently estimated with asymptotic normality while the variance parameter and the range (or length-scale) parameter cannot. Motivated by the frequentist theory, we prove a Bernstein-von Mises theorem for the covariance parameters in isotropic Matern covariance functions. We show that under fixed-domain asymptotics, the joint posterior distribution of the microergodic parameter and the range parameter can be factored independently into the product of their marginal posteriors as the sample size goes to infinity. The posterior of the microergodic parameter converges in total variation norm to a normal distribution with shrinking variance, while the posterior of the range parameter does not necessarily converge to any degenerate distribution in general. Our theory allows unbounded prior support for the range parameter. Furthermore, we propose a new property called the posterior asymptotic efficiency in linear prediction, and show that the Bayesian kriging predictor at a new spatial location with covariance parameters randomly drawn from their posterior has the same prediction mean squared error as if the true parameters were known. In the special case of one-dimensional Ornstein-Uhlenbeck process, we derive an explicit form for the limiting posterior distribution of the range parameter and an explicit posterior convergence rate for the posterior asymptotic efficiency. We verify these asymptotic results in numerical examples.
On Losses for Modern Language Models

BERT set many state-of-the-art results over varied NLU benchmarks by pre-training over two tasks: masked language modelling (MLM) and next sentence prediction (NSP), the latter of which has been highly criticized. In this paper, we 1) clarify NSP's effect on BERT pre-training, 2) explore fourteen possible auxiliary pre-training tasks, of which seven are novel to modern language models, and 3) investigate different ways to include multiple tasks into pre-training. We show that NSP is detrimental to training due to its context splitting and shallow semantic signal. We also identify six auxiliary pre-training tasks -- sentence ordering, adjacent sentence prediction, TF prediction, TF-IDF prediction, a FastSent variant, and a Quick Thoughts variant -- that outperform a pure MLM baseline. Finally, we demonstrate that using multiple tasks in a multi-task pre-training framework provides better results than using any single auxiliary task. Using these methods, we outperform BERT Base on the GLUE benchmark using fewer than a quarter of the training tokens.
Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functions

We consider the problem of local planning in fixed-horizon Markov Decision Processes (MDPs) with linear function approximation and a generative model under the assumption that the optimal action-value function lies in the span of a feature map that is available to the planner. Previous work has left open the question of whether there exists sound planners that need only poly(H, d) queries regardless of the MDP, where H is the horizon and d is the dimensionality of the features. We answer this question in the negative: we show that any sound planner must query at least min(exp({\Omega}(d)), {\Omega}(2^H)) samples. We also show that for any {\delta}>0, the least-squares value iteration algorithm with O(H^5d^(H+1)/{\delta}^2) queries can compute a {\delta}-optimal policy. We discuss implications and remaining open questions.