Miracle-dz/UniFAR-FTUs huggingface dataset:
From Hero to Zéroe: A Benchmark of Low-Level Adversarial Attacks

Adversarial attacks are label-preserving modifications to inputs of machine learning classifiers designed to fool machines but not humans. Natural Language Processing (NLP) has mostly focused on high-level attack scenarios such as paraphrasing input texts. We argue that these are less realistic in typical application scenarios such as in social media, and instead focus on low-level attacks on the character-level. Guided by human cognitive abilities and human robustness, we propose the first large-scale catalogue and benchmark of low-level adversarial attacks, which we dub Zéroe, encompassing nine different attack modes including visual and phonetic adversaries. We show that RoBERTa, NLP’s current workhorse, fails on our attacks. Our dataset provides a benchmark for testing robustness of future more human-like NLP models.
SJTU-NICT’s Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task

In this paper, we introduced our joint team SJTU-NICT ‘s participation in the WMT 2020 machine translation shared task. In this shared task, we participated in four translation directions of three language pairs: English-Chinese, English-Polish on supervised machine translation track, German-Upper Sorbian on low-resource and unsupervised machine translation tracks. Based on different conditions of language pairs, we have experimented with diverse neural machine translation (NMT) techniques: document-enhanced NMT, XLM pre-trained language model enhanced NMT, bidirectional translation as a pre-training, reference language based UNMT, data-dependent gaussian prior objective, and BT-BLEU collaborative filtering self-training. We also used the TF-IDF algorithm to filter the training set to obtain a domain more similar set with the test set for finetuning. In our submissions, the primary systems won the first place on English to Chinese, Polish to English, and German to Upper Sorbian translation directions.
Deep Active Learning for Joint Classification & Segmentation with Weak Annotator

CNN visualization and interpretation methods, like class-activation maps (CAMs), are typically used to highlight the image regions linked to class predictions. These models allow to simultaneously classify images and extract class-dependent saliency maps, without the need for costly pixel-level annotations. However, they typically yield segmentations with high false-positive rates and, therefore, coarse visualisations, more so when processing challenging images, as encountered in histology. To mitigate this issue, we propose an active learning (AL) framework, which progressively integrates pixel-level annotations during training. Given training data with global image-level labels, our deep weakly-supervised learning model jointly performs supervised image-level classification and active learning for segmentation, integrating pixel annotations by an oracle. Unlike standard AL methods that focus on sample selection, we also leverage large numbers of unlabeled images via pseudo-segmentations (i.e., self-learning at the pixel level), and integrate them with the oracle-annotated samples during training. We report extensive experiments over two challenging benchmarks – high-resolution medical images (histology GlaS data for colon cancer) and natural images (CUB-200-2011 for bird species). Our results indicate that, by simply using random sample selection, the proposed approach can significantly outperform state-of the-art CAMs and AL methods, with an identical oracle-supervision budget. Our code is publicly available1.
Nonrelativistic giant magnons and $ SU(1,2|3) $ limit of strings in $ AdS_5 \times S^5 $.

We derive nonrelativistic giant magnon spectrum starting from $ SU(1,2|3) $ Spin-Matrix theory (SMT) limit of strings in $ AdS_5 \times S^5 $. The single spin as well as two spin nonrelativistic magnon spectrum are obtained considering rotating string configurations on $ R \times S^p (p=2,3) $. In a parallel computation, we explore rigidly rotating string configurations in various sub-spaces of $ AdS_5 $ and rediscover nonrelativistic magnon like excitations in these models. We further generalise our results by constructing a three spin solution where two of the spins are considered from $ S^5 $ while the remaining one is aligned along $ S^3\subset AdS_5 $. Unlike its relativistic cousins, our analysis reveals the non-existence of spikes in all of these nonrelativistic models.
MultiWOZ 2.3: A multi-domain task-oriented dataset enhanced with annotation corrections and co-reference annotation

Task-oriented dialogue systems have made unprecedented progress with multiple state-of-the-art (SOTA) models underpinned by a number of publicly available MultiWOZ datasets. Dialogue state annotations are error-prone, leading to sub-optimal performance. Various efforts have been put in rectifying the annotation errors presented in the original MultiWOZ dataset. In this paper, we introduce MultiWOZ 2.3, in which we differentiate incorrect annotations in dialogue acts from dialogue states, identifying a lack of co-reference when publishing the updated dataset. To ensure consistency between dialogue acts and dialogue states, we implement co-reference features and unify annotations of dialogue acts and dialogue states. We update the state of the art performance of natural language understanding and dialog state tracking on MultiWOZ 2.3, where the results show significant improvements than on previous versions of MultiWOZ datasets (2.0-2.2).
Nonparametric Bounds for Causal Effects in Imperfect Randomized Experiments

Abstract Nonignorable missingness and noncompliance can occur even in well-designed randomized experiments, making the intervention effect that the experiment was designed to estimate nonidentifiable. Nonparametric causal bounds provide a way to narrow the range of possible values for a nonidentifiable causal effect with minimal assumptions. We derive novel bounds for the causal risk difference for a binary outcome and intervention in randomized experiments with nonignorable missingness that is caused by a variety of mechanisms, with both perfect and imperfect compliance. We show that the so-called worst-case imputation, whereby all missing subjects on the intervention arm are assumed to have events and all missing subjects on the control or placebo arm are assumed to be event-free, can be too pessimistic in blinded studies with perfect compliance, and is not bounding the correct estimand with imperfect compliance. We illustrate the use of the proposed bounds in our motivating data example of peanut consumption on the development of peanut allergies in infants. We find that, even accounting for potentially nonignorable missingness and noncompliance, our derived bounds confirm that regular exposure to peanuts reduces the risk of development of peanut allergies, making the results of this study much more compelling.
What causes the test error? Going beyond bias-variance via ANOVA

Modern machine learning methods are often overparametrized, allowing adaptation to the data at a fine level. This can seem puzzling; in the worst case, such models do not need to generalize. This puzzle inspired a great amount of work, arguing when overparametrization reduces test error, in a phenomenon called "double descent". Recent work aimed to understand in greater depth why overparametrization is helpful for generalization. This leads to discovering the unimodality of variance as a function of the level of parametrization, and to decomposing the variance into that arising from label noise, initialization, and randomness in the training data to understand the sources of the error. In this work we develop a deeper understanding of this area. Specifically, we propose using the analysis of variance (ANOVA) to decompose the variance in the test error in a symmetric way, for studying the generalization performance of certain two-layer linear and non-linear networks. The advantage of the analysis of variance is that it reveals the effects of initialization, label noise, and training data more clearly than prior approaches. Moreover, we also study the monotonicity and unimodality of the variance components. While prior work studied the unimodality of the overall variance, we study the properties of each term in variance decomposition. One key insight is that in typical settings, the interaction between training samples and initialization can dominate the variance; surprisingly being larger than their marginal effect. Also, we characterize "phase transitions" where the variance changes from unimodal to monotone. On a technical level, we leverage advanced deterministic equivalent techniques for Haar random matrices, that---to our knowledge---have not yet been used in the area. We also verify our results in numerical simulations and on empirical data examples.
Discrete Latent Space World Models for Reinforcement Learning

Sample efficiency remains a fundamental issue of reinforcement learning. Model-based algorithms try to make better use of data by simulating the environment with a model. We propose a new neural network architecture for world models based on a vector quantized-variational autoencoder (VQ-VAE) to encode observations and a convolutional LSTM to predict the next embedding indices. A model-free PPO agent is trained purely on simulated experience from the world model. We adopt the setup introduced by Kaiser et al. (2020), which only allows 100K interactions with the real environment, and show that we reach better performance than their SimPLe algorithm in five out of six randomly selected Atari environments, while our model is significantly smaller.
Permuted AdaIN: Enhancing the Representation of Local Cues in Image Classifiers

Recent work has shown that convolutional neural network classifiers overly rely on texture at the expense of shape cues, which adversely affects the classifier's performance in shifted domains. In this work, we make a similar but different distinction between local image cues, including shape and texture, and global image statistics. We provide a method that enhances the representation of local cues in the hidden layers of image classifiers. Our method, called Permuted Adaptive Instance Normalization (pAdaIN), samples a random permutation $\pi$ that rearranges the samples in a given batch. Adaptive Instance Normalization (AdaIN) is then applied between the activations of each (non-permuted) sample $i$ and the corresponding activations of the sample $\pi(i)$, thus swapping statistics between the samples of the batch. Since the global image statistics are distorted, this swapping procedure causes the network to rely on the local image cues. By choosing the random permutation with probability $p$ and the identity permutation otherwise, one can control the strength of this effect. With the correct choice of $p$, selected without considering the test data, our method consistently outperforms baseline methods in image classification, as well as in the setting of domain generalization.
Relation Extraction as Two-way Span-Prediction

The current supervised relation classification (RC) task uses a single embedding to represent the relation between a pair of entities. We argue that a better approach is to treat the RC task as a Question answering (QA) like span prediction problem. We present a span-prediction based system for RC and evaluate its performance compared to the embedding based system. We achieve state-of-the-art results on the TACRED and SemEval task 8 datasets.
Joint Transceiver and Large Intelligent Surface Design for Massive MIMO mmWave Systems

Large intelligent surface (LIS) has recently emerged as a potential low-cost solution to reshape the wireless propagation environment for improving the spectral efficiency. In this article, we consider a downlink millimeter-wave (mmWave) multiple-input-multiple-output (MIMO) system, where an LIS is deployed to assist the downlink data transmission from a base station (BS) to a user equipment (UE). Both the BS and the UE are equipped with a large number of antennas, and a hybrid analog/digital precoding/combining structure is used to reduce the hardware cost and energy consumption. We aim to maximize the spectral efficiency by jointly optimizing the LIS’s reflection coefficients and the hybrid precoder (combiner) at the BS (UE). To tackle this non-convex problem, we reformulate the complex optimization problem into a much more friendly optimization problem by exploiting the inherent structure of the effective (cascade) mmWave channel. A manifold optimization (MO)-based algorithm is then developed. Simulation results show that by carefully devising LIS’s reflection coefficients, our proposed method can help realize a favorable propagation environment with a small channel matrix condition number. Besides, it can achieve a performance comparable to those of state-of-the-art algorithms, while at a much lower computational complexity.
Uniform asymptotic expansions for solutions of the parabolic cylinder and Weber equations

Asymptotic expansions are derived for solutions of the parabolic cylinder and Weber differential equations. In addition the inhomogeneous versions of the equations are considered, for the case of polynomial forcing terms. The expansions involve exponential, Airy and Scorer functions and slowly varying analytic coefficient functions involving simple coefficients. The approximations are uniformly valid for large values of the parameter and unbounded real and complex values of the argument. Explicit and readily computable error bounds are either furnished or available for all approximations.
The effect of electrostatic charges on particle-laden duct flows

Abstract We report on direct numerical simulations of the effect of electrostatic charges on particle-laden duct flows. The corresponding electrostatic forces are known to affect the particle dynamics at small scales and the associated turbophoretic drift. Our simulations, however, predicted that electrostatic forces also dominate the vortical motion of the particles, induced by the secondary flows of Prandtl's second kind of the carrier fluid. Herein, we treated flows at two frictional Reynolds numbers ($Re_{\tau }= 300$ and 600), two particle-to-gas density ratios ($\rho _{p}/\rho =1000$ and 7500) and three Coulombic-to-gravitational force ratios ($F_{el}/F_{g}=0$, 0.004 and 0.026). In flows with a high density ratio at $Re_{\tau }=600$ and $F_{el}/F_{g}=0.004$, the particles tend to accumulate at the walls. On the other hand, at a lower density ratio, respectively a higher $F_{el}/F_{g}$ of 0.026, the charged particles still follow the secondary flow structures that are developed in the duct. However, even in this case, the electrostatic forces counteract the particles’ inward flux from the wall and, as a result, their vortical motion in these secondary structures is significantly attenuated. This change in the flow pattern results in an increase of the particle number density at the bisectors of the walls by a factor of five compared with the corresponding flow with uncharged particles. Finally, at $Re_{\tau }=300$, $\rho _{p}/\rho =1000$ and $F_{el}/F_{g}=0.026$ the electrostatic forces dominate over the aerodynamic forces and gravity and, consequently, the particles no longer follow the streamlines of the carrier gas.
The Infrared Medium-deep Survey. VIII. Quasar Luminosity Function at z ∼ 5

Faint z ∼ 5 quasars with M1450 ∼ −23 mag are known to be potentially important contributors to the ultraviolet ionizing background in the postreionization era. However, their number density has not been well determined, making it difficult to assess their role in the early ionization of the intergalactic medium (IGM). In this work, we present the updated results of our z ∼ 5 quasar survey using the Infrared Medium-deep Survey (IMS), a near-infrared imaging survey covering an area of 85 deg2. From our spectroscopic observations with the Gemini Multi-Object Spectrograph on the Gemini-South 8 m telescope, we discovered eight new quasars at z ∼ 5 with −26.1 ≤ M1450 ≤ −23.3. Combining our IMS faint quasars (M1450 > −27 mag) with the brighter Sloan Digital Sky Survey quasars (M1450 < −27 mag), we derive the z ∼ 5 quasar luminosity function (QLF) without any fixed parameters down to the magnitude limit of M1450 = −23 mag. We find that the faint-end slope of the QLF is very flat ( ), with a characteristic luminosity of mag. The number density of z ∼ 5 quasars from the QLF gives an ionizing emissivity at 912 Å of ϵ912 = (3.7–7.1) × 1023 erg s−1 Hz−1 Mpc−3 and an ionizing photon density of Mpc−3 s−1. These results imply that quasars are responsible for only 10%–20% (up to 50% even in the extreme case) of the photons required to completely ionize the IGM at z ∼ 5, disfavoring the idea that quasars alone could have ionized the IGM at z ∼ 5.
Interpretable Neural Networks for Panel Data Analysis in Economics

The lack of interpretability and transparency are preventing economists from using advanced tools like neural networks in their empirical work. In this paper, we propose a new class of interpretable neural network models that can achieve both high prediction accuracy and interpretability in regression problems with time series cross-sectional data. Our model can essentially be written as a simple function of a limited number of interpretable features. In particular, we incorporate a class of interpretable functions named persistent change filters as part of the neural network. We apply this model to predicting individual's monthly employment status using high-dimensional administrative data in China. We achieve an accuracy of 94.5% on the out-of-sample test set, which is comparable to the most accurate conventional machine learning methods. Furthermore, the interpretability of the model allows us to understand the mechanism that underlies the ability for predicting employment status using administrative data: an individual's employment status is closely related to whether she pays different types of insurances. Our work is a useful step towards overcoming the "black box" problem of neural networks, and provide a promising new tool for economists to study administrative and proprietary big data.
Kernel Methods for Policy Evaluation: Treatment Effects, Mediation Analysis, and Off-Policy Planning

We propose a novel framework for non-parametric policy evaluation in static and dynamic settings. Under the assumption of selection on observables, we consider treatment effects of the population, of sub-populations, and of alternative populations that may have alternative covariate distributions. We further consider the decomposition of a total effect into a direct effect and an indirect effect (as mediated by a particular mechanism). Under the assumption of sequential selection on observables, we consider the effects of sequences of treatments. Across settings, we allow for treatments that may be discrete, continuous, or even text. Across settings, we allow for estimation of not only counterfactual mean outcomes but also counterfactual distributions of outcomes. We unify analyses across settings by showing that all of these causal learning problems reduce to the re-weighting of a prediction, i.e. causal adjustment. We implement the re-weighting as an inner product in a function space called a reproducing kernel Hilbert space (RKHS), with a closed form solution that can be computed in one line of code. We prove uniform consistency and provide finite sample rates of convergence. We evaluate our estimators in simulations devised by other authors. We use our new estimators to evaluate continuous and heterogeneous treatment effects of the US Jobs Corps training program for disadvantaged youth.
Investigating the growing population of massive quiescent galaxies at cosmic noon

We explore the buildup of quiescent galaxies using a sample of 28,469 massive ($M_\star \ge 10^{11}$M$_\odot$) galaxies at redshifts $1.5<z<3.0$, drawn from a 17.5 deg$^2$ area (0.33 Gpc$^3$ comoving volume at these redshifts). This allows for a robust study of the quiescent fraction as a function of mass at $1.5<z<3.0$ with a sample $\sim$40 times larger at log($M_{\star}$/$\rm M_{\odot}$)$\ge11.5$ than previous studies. We derive the quiescent fraction using three methods: specific star-formation rate, distance from the main sequence, and UVJ color-color selection. All three methods give similar values at $1.5<z<2.0$, however the results differ by up to a factor of two at $2.0<z<3.0$. At redshifts $1.5 < z < 3.0$ the quiescent fraction increases as a function of stellar mass. By $z=2$, only 3.3 Gyr after the Big Bang, the universe has quenched $\sim$25% of $M_\star = 10^{11}$M$_\odot$ galaxies and $\sim$45% of $M_\star = 10^{12}$M$_\odot$ galaxies. We discuss physical mechanisms across a range of epochs and environments that could explain our results. We compare our results with predictions from hydrodynamical simulations SIMBA and IllustrisTNG and semi-analytic models (SAMs) SAG, SAGE, and Galacticus. The quiescent fraction from IllustrisTNG is higher than our empirical result by a factor of $2-5$, while those from SIMBA and the three SAMs are lower by a factor of $1.5-10$ at $1.5<z<3.0$.
Quench Dynamics of a Fermi Gas with Strong Nonlocal Interactions

We induce strong nonlocal interactions in a 2D Fermi gas in an optical lattice using Rydberg dressing. The system is approximately described by a t − V model on a square lattice where the fermions experience isotropic nearest-neighbor interactions and are free to hop only along one direction. We measure the interactions using many-body Ramsey interferometry and study the lifetime of the gas in the presence of tunneling, finding that tunneling does not reduce the lifetime. To probe the interplay of nonlocal interactions with tunneling, we investigate the short-time-relaxation dynamics of charge-density waves in the gas. We find that strong nearest-neighbor interactions slow down the relaxation. Our work opens the door for quantum simulations of systems with strong nonlocal interactions such as extended Fermi-Hubbard models.
On the modified fractional Korteweg–de Vries and related equations

We consider in this paper modified fractional Korteweg–de Vries and related equations (modified Burgers–Hilbert and Whitham). They have the advantage with respect to the usual fractional KdV equation to have a defocusing case with a different dynamics. We will distinguish the weakly dispersive case where the phase velocity is unbounded for low frequencies and tends to zero at infinity and the strongly dispersive case where the phase velocity vanishes at the origin and goes to infinity at infinity. In the former case, the nonlinear hyperbolic effects dominate for large data, leading to the possibility of shock formation though the dispersive effects manifest for small initial data where scattering is possible. In the latter case, finite time blow-up is possible in the focusing case but not the shock formation. In the defocusing case global existence and scattering is expected in the energy subcritical case, while finite time blow-up is expected in the energy supercritical case. We establish rigorously the existence of shocks with blow-up time and location being explicitly computed in the weakly dispersive case, while most of the results on the strongly dispersive case are derived via numerical simulations, for large solutions. Moreover, the shock formation result can be extended to the weakly dispersive equation with some generalized nonlinearity. We will also comment briefly on the BBM versions of those equations.