Miracle-dz/UniFAR-FTUs huggingface dataset:
Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer

Adversarial attacks and backdoor attacks are two common security threats that hang over deep learning. Both of them harness task-irrelevant features of data in their implementation. Text style is a feature that is naturally irrelevant to most NLP tasks, and thus suitable for adversarial and backdoor attacks. In this paper, we make the first attempt to conduct adversarial and backdoor attacks based on text style transfer, which is aimed at altering the style of a sentence while preserving its meaning. We design an adversarial attack method and a backdoor attack method, and conduct extensive experiments to evaluate them. Experimental results show that popular NLP models are vulnerable to both adversarial and backdoor attacks based on text style transfer—the attack success rates can exceed 90% without much effort. It reflects the limited ability of NLP models to handle the feature of text style that has not been widely realized. In addition, the style transfer-based adversarial and backdoor attack methods show superiority to baselines in many aspects. All the code and data of this paper can be obtained at https://github.com/thunlp/StyleAttack.
Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling

Given the claims of improved text generation quality across various pre-trained neural models, we consider the coherence evaluation of machine generated text to be one of the principal applications of coherence models that needs to be investigated. Prior work in neural coherence modeling has primarily focused on devising new architectures for solving the permuted document task. We instead use a basic model architecture and show significant improvements over state of the art within the same training regime. We then design a harder self-supervision objective by increasing the ratio of negative samples within a contrastive learning setup, and enhance the model further through automatic hard negative mining coupled with a large global negative queue encoded by a momentum encoder. We show empirically that increasing the density of negative samples improves the basic model, and using a global negative queue further improves and stabilizes the model while training with hard negative samples. We evaluate the coherence model on task-independent test sets that resemble real-world applications and show significant improvements in coherence evaluations of downstream tasks.
Cross-Lingual GenQA: A Language-Agnostic Generative Question Answering Approach for Open-Domain Question Answering

Open-Retrieval Generative Question Answering (G EN QA) is proven to deliver high-quality, natural-sounding answers in English. In this paper, we present the ﬁrst generalization of the G EN QA approach for the multilingual environment. To this end, we present the G EN -T Y D I QA dataset, which extends the Ty-DiQA evaluation data (Clark et al., 2020) with natural-sounding, well-formed answers in Arabic, Bengali, English, Japanese, and Russian. For all these languages, we show that a G EN QA sequence-to-sequence-based model outperforms a state-of-the-art Answer Sentence Selection model. We also show that a multilingually-trained model competes with, and in some cases outperforms, its monolingual counterparts. Finally, we show that our system can even compete with strong base-lines, even when fed with information from a variety of languages. Essentially, our system is able to answer a question in any language of our language set using information from many languages, making it the ﬁrst Language-Agnostic G EN QA system.
Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans

Recursive processing is considered a hallmark of human linguistic abilities. A recent study evaluated recursive processing in recurrent neural language models (RNN-LMs) and showed that such models perform below chance level on embedded dependencies within nested constructions -- a prototypical example of recursion in natural language. Here, we study if state-of-the-art Transformer LMs do any better. We test four different Transformer LMs on two different types of nested constructions, which differ in whether the embedded (inner) dependency is short or long range. We find that Transformers achieve near-perfect performance on short-range embedded dependencies, significantly better than previous results reported for RNN-LMs and humans. However, on long-range embedded dependencies, Transformers' performance sharply drops below chance level. Remarkably, the addition of only three words to the embedded dependency caused Transformers to fall from near-perfect to below-chance performance. Taken together, our results reveal Transformers' shortcoming when it comes to recursive, structure-based, processing.
Using Multitask Gaussian Processes to estimate the effect of a targeted effort to remove firearms∗

Gun violence is a critical public safety concern in the United States. In 2006 California implemented a unique firearm monitoring program, the Armed and Prohibited Persons System (APPS), to address gun violence in the state by identifying those firearm owners who become prohibited from owning a firearm due to federal and state law and confiscating their firearms. Our goal is to assess the effect of APPS on California murder rates using annual, state-level crime data across the US for the years before and after the introduction of the program. To do so, we adapt a non-parametric Bayesian approach, multitask Gaussian Processes (MTGPs), to the panel data setting. MTGPs allow for flexible and parsimonious panel data models that nest many existing approaches and allow for direct control over both dependence across time and dependence across units, as well as natural uncertainty quantification. We extend this approach to incorporate non-Normal outcomes, auxiliary covariates, and multiple outcome series, which are all important in our application. We also show that this approach has attractive Frequentist properties, including a representation as a weighting estimator with separate weights over units and time periods. Applying this approach, we find that the increased monitoring and enforcement from the APPS program substantially decreased gun-related homicides in California. We would like to thank Naoki Egami, Max Gopelrud, Jacob Montgomery, Liyang Sun, and seminar participants at Polmeth 2021 for helpful comments and discussion. We are especially grateful to Eli Sherman for his involvement in an earlier version of this project. This research was supported in part by the Hellman Family Fund at UC Berkeley and by the Institute of Education Sciences, U.S. Department of Education, through Grant R305D200010. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education. ar X iv :2 11 0. 07 00 6v 1 [ st at . M E ] 1 3 O ct 2 02 1
Causally Estimating the Sensitivity of Neural NLP Models to Spurious Features

Recent work finds modern natural language processing (NLP) models relying on spurious features for prediction. Mitigating such effects is thus important. Despite this need, there is no quantitative measure to evaluate or compare the effects of different forms of spurious features in NLP. We address this gap in the literature by quantifying model sensitivity to spurious features with a causal estimand, dubbed CENT, which draws on the concept of average treatment effect from the causality literature. By conducting simulations with four prominent NLP models — TextRNN, BERT, RoBERTa and XLNet — we rank the models against their sensitivity to artificial injections of eight spurious features. We further hypothesize and validate that models that are more sensitive to a spurious feature will be less robust against perturbations with this feature during inference. Conversely, data augmentation with this feature improves robustness to similar perturbations. We find statistically significant inverse correlations between sensitivity and robustness, providing empirical support for our hypothesis.
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking

In various natural language processing tasks, passage retrieval and passage re-ranking are two key procedures in finding and ranking relevant information. Since both the two procedures contribute to the final performance, it is important to jointly optimize them in order to achieve mutual improvement. In this paper, we propose a novel joint training approach for dense passage retrieval and passage reranking. A major contribution is that we introduce the dynamic listwise distillation, where we design a unified listwise training approach for both the retriever and the re-ranker. During the dynamic distillation, the retriever and the re-ranker can be adaptively improved according to each other’s relevance information. We also propose a hybrid data augmentation strategy to construct diverse training instances for listwise training approach. Extensive experiments show the effectiveness of our approach on both MSMARCO and Natural Questions datasets. Our code is available at https://github.com/PaddlePaddle/RocketQA.
Exploratory X-Ray Monitoring of Luminous Radio-quiet Quasars at High Redshift: Extended Time-series Analyses and Stacked Imaging Spectroscopy

We present three new Chandra X-ray epochs along with new ground-based optical–UV observations as the third installment in a time-series analysis of four high-redshift (z ≈ 4.1–4.4) radio-quiet quasars. In total, we present nine epochs for these sources with rest-frame temporal baselines of ∼1300–2000 days. We utilize the X-ray data to determine basic variability properties, as well as produce mean spectra and stacked images based on effective exposure times of ∼40–70 ks per source. We perform time-series analyses in the soft and hard bands, separately, and compare variability properties to those of sources at lower redshifts and luminosities. The magnitude of X-ray variability of our sources remains consistent with or lower than that of similar sources at lower redshifts, in agreement with the variability–luminosity anticorrelation. The mean power-law photon indices in the stacked Chandra spectra of our sources are consistent with the values measured from their archival XMM-Newton spectra separated by about 3 yr in the rest frame. Along with the X-ray observations, we provide near-simultaneous optical monitoring of the sources in the optical–UV regime. The overall variability in the optical-to-X-ray spectral slope is consistent with sources at lower redshifts, and the optical–UV observations display mild variability on monthly timescales.
Clear evidence against superconductivity in hydrides under high pressure

The Meissner effect, magnetic field expulsion, is a hallmark of superconductivity. Associated with it, superconductors exclude applied magnetic fields. Recently Minkov et al. presented experimental results reportedly showing ``definitive evidence of the Meissner effect'' in sulfur hydride and lanthanum hydride under high pressure [1]. Instead, we show here that the evidence presented in that paper does not support the case for superconductivity in these materials. Together with experimental evidence discussed in earlier papers, we argue that this clearly indicates that hydrides under pressure are not high temperature superconductors.
Transferring Semantic Knowledge Into Language Encoders

We introduce semantic form mid-tuning, an approach for transferring semantic knowledge from semantic meaning representations into transformer-based language encoders. In mid-tuning, we learn to align the text of general sentences -- not tied to any particular inference task -- and structured semantic representations of those sentences. Our approach does not require gold annotated semantic representations. Instead, it makes use of automatically generated semantic representations, such as from off-the-shelf PropBank and FrameNet semantic parsers. We show that this alignment can be learned implicitly via classification or directly via triplet loss. Our method yields language encoders that demonstrate improved predictive performance across inference, reading comprehension, textual similarity, and other semantic tasks drawn from the GLUE, SuperGLUE, and SentEval benchmarks. We evaluate our approach on three popular baseline models, where our experimental results and analysis concludes that current pre-trained language models can further benefit from structured semantic frames with the proposed mid-tuning method, as they inject additional task-agnostic knowledge to the encoder, improving the generated embeddings as well as the linguistic properties of the given model, as evident from improvements on a popular sentence embedding toolkit and a variety of probing tasks.
Gauge effects in bound-bound Rydberg transition matrix elements

Accurate data on electric-dipole transition matrix elements (EDTMs) for bound-bound Rydbergatom transitions become increasingly important in science and technology. Here we compute radial EDTMs of rubidium using the length, velocity and acceleration gauges for electric-dipole-allowed transitions between states with principal and angular-momentum quantum numbers ranging from 15 to 100. Wave-functions are computed based upon model potentials from Marinescu et al., Phys. Rev. A 49, 982 (1994). Length-gauge EDTMs, often used for low-l transitions, are found to deviate from the fundamentally more accurate velocity-gauge EDTMs by relative amounts of up to ∼ 10. We discuss the physical reasons for the observed gauge differences, explain the conditions for applicability of the velocity and length gauges for different transition series, and present a decision tree of how to choose EDTMs. Implications for contemporary Rydberg-atom applications are discussed.
Advances in Scaling Community Discovery Methods for Large Signed Graph Networks

Community detection is a common task in social network analysis (SNA) with applications in a variety of fields including medicine, criminology, and business. Despite the popularity of community detection, there is no clear consensus on the most effective methodology for signed networks. In this paper, we summarize the development of community detection in signed networks and evaluate current state-of-the-art techniques on several real-world data sets. First, we give a comprehensive background of community detection in signed graphs. Next, we compare various adaptations of the Laplacian matrix in recovering ground-truth community labels via spectral clustering in small signed graph data sets. Then, we evaluate the scalability of leading algorithms on small, large, dense, and sparse real-world signed graph networks. We conclude with a discussion of our novel findings and recommendations for extensions and improvements in state-of-the-art techniques for signed graph community discovery in real-world signed graphs. } {Sign Graph Clustering, Community Discovery, Sparse Networks
Relation-aware Heterogeneous Graph for User Profiling

User profiling has long been an important problem that investigates user interests in many real applications. Some recent works regard users and their interacted objects as entities of a graph and turn the problem into a node classification task. However, they neglect the difference of distinct interaction types, e.g. user clicks an item v.s. user purchases an item, and thus cannot incorporate such information well. To solve these issues, we propose to leverage the relation-aware heterogeneous graph method for user profiling, which also allows capturing significant meta relations. We adopt the query, key, and value mechanism in a transformer fashion for heterogeneous message passing so that entities can effectively interact with each other. Via such interactions on different relation types, our model can generate representations with rich information for the user profile prediction. We conduct experiments on two real-world e-commerce datasets and observe a significant performance boost of our approach.
Spectral Convergence of Symmetrized Graph Laplacian on manifolds with boundary

We study the spectral convergence of a symmetrized Graph Laplacian matrix induced by a Gaussian kernel evaluated on pairs of embedded data, sampled from a manifold with boundary, a sub-manifold of $\mathbb{R}^m$. Specifically, we deduce the convergence rates for eigenpairs of the discrete Graph-Laplacian matrix to the eigensolutions of the Laplace-Beltrami operator that are well-defined on manifolds with boundary, including the homogeneous Neumann and Dirichlet boundary conditions. For the Dirichlet problem, we deduce the convergence of the \emph{truncated Graph Laplacian}, which is recently numerically observed in applications, and provide a detailed numerical investigation on simple manifolds. Our method of proof relies on the min-max argument over a compact and symmetric integral operator, leveraging the RKHS theory for spectral convergence of integral operator and a recent pointwise asymptotic result of a Gaussian kernel integral operator on manifolds with boundary.
A two-orbital quantum gas with tunable interactions

Im letzten Jahrzehnt haben sich Quantengasexperimente als gut kontrollierbare Modellsysteme zur Untersuchung komplexer Fragestellungen aus diversen Bereichen der Physik etabliert. Ultrakalte Quantengase zeichnen sich insbesondere dadurch aus, dass sie einen direkten und experimentell einfach realisierbaren Zugang zu ihrerWechselwirkung bieten. Das gezielte Einstellen der Wechselwirkungsstarke und die Erforschung der daraus resultierenden Aggregatzustande erlaubt es ein tiefes Verstandnis der kondensierten Materie zu gewinnen. Insbesondere erdalkaliahnliche Atome wie Ytterbium bieten die Moglichkeit Phanomene der Festkorperphysik zu untersuchen, die durch die Wechselwirkung von Elektronen in verschiedenen Orbitalen oder durch eine grosere Rotationssymmetrie des Spins als in gewohnlichen Spin-1/2 Systemen hervorgerufen werden. Diese Doktorarbeit prasentiert die experimentelle Charakterisierung der Wechselwirkung ultrakalter, fermionischer Ytterbium-Atome (173Yb) in verschiedenen elektronischen Orbitalen. Dabei wird nachgewiesen, dass sich die Wechselwirkungsstarke mit Hilfe eines externen Magnetfeldes, analog zu einer Feshbach-Resonanz bei Alkali-Atomen, einstellen lasst. Bei Ytterbium wird diese Resonanz durch eine starke Spinaustauschwechselwirkung zwischen den verschiedenen Orbitalen hervorgerufen. Der Nachweis der einstellbaren Wechselwirkung erfolgt uber Thermalisierungsexperimente in einer harmonischen Falle und mit Hilfe von hochauflosender Spektroskopie in einem dreidimensionalen Gitter. Des Weiteren wird mit Hilfe der neu entdeckten Resonanz zum ersten Mal experimentell ein stark wechselwirkendes Fermigas in verschiedenen Orbitalen erzeugt und spektroskopisch untersucht. Die Moglichkeit, die interorbitale Wechselwirkung direkt zu manipulieren und somit stark wechselwirkende Quantengase zu erzeugen, ebnet den Weg fur die Realisierung und Untersuchung neuartiger Aggregatzustande der kondensierten Materie.
DeepOrder: Deep Learning for Test Case Prioritization in Continuous Integration Testing

Continuous integration testing is an important step in the modern software engineering life cycle. Test prioritization is a method that can improve the efficiency of continuous integration testing by selecting test cases that can detect faults in the early stage of each cycle. As continuous integration testing produces voluminous test execution data, test history is a commonly used artifact in test prioritization. However, existing test prioritization techniques for continuous integration either cannot handle large test history or are optimized for using a limited number of historical test cycles. We show that such a limitation can decrease fault detection effectiveness of prioritized test suites. This work introduces DeepOrder, a deep learning-based model that works on the basis of regression machine learning. DeepOrder ranks test cases based on the historical record of test executions from any number of previous test cycles. DeepOrder learns failed test cases based on multiple factors including the duration and execution status of test cases. We experimentally show that deep neural networks, as a simple regression model, can be efficiently used for test case prioritization in continuous integration testing. DeepOrder is evaluated with respect to time-effectiveness and fault detection effectiveness in comparison with an industry practice and the state of the art approaches. The results show that DeepOrder outperforms the industry practice and state-of-the-art test prioritization approaches in terms of these two metrics.
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning

Recent parameter-efficient language model tuning (PELT) methods manage to match the performance of fine-tuning with much fewer trainable parameters and perform especially well when training data is limited. However, different PELT methods may perform rather differently on the same task, making it nontrivial to select the most appropriate method for a specific task, especially considering the fast-growing number of new PELT methods and tasks. In light of model diversity and the difficulty of model selection, we propose a unified framework, UniPELT, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup via gating mechanism. On the GLUE benchmark, UniPELT consistently achieves 1 4% gains compared to the best individual PELT method that it incorporates and even outperforms fine-tuning under different setups. Moreover, UniPELT generally surpasses the upper bound that takes the best performance of all its submodules used individually on each task, indicating that a mixture of multiple PELT methods may be inherently more effective than single methods.
On the formation of hydrogen peroxide in water microdroplets

Recent reports on the formation of hydrogen peroxide (H2O2) in water microdroplets produced via pneumatic spraying or capillary condensation have garnered significant attention. How covalent bonds in water could break under such mild conditions challenges our textbook understanding of physical chemistry and water. While there is no definitive answer, it has been speculated that ultrahigh electric fields at the air–water interface are responsible for this chemical transformation. Here, we report on our comprehensive experimental investigation of H2O2 formation in (i) water microdroplets sprayed over a range of liquid flow-rates, (shearing) air flow rates, and air composition, and (ii) water microdroplets condensed on hydrophobic substrates formed via hot water or humidifier under controlled air composition. Specifically, we assessed the contributions of the evaporative concentration and shock waves in sprays and the effects of trace O3(g) on the H2O2 formation. Glovebox experiments revealed that the H2O2 formation in water microdroplets was most sensitive to the air–borne ozone (O3) concentration. In the absence of O3(g), we could not detect H2O2(aq) in sprays or condensates (detection limit ≥250 nM). In contrast, microdroplets exposed to atmospherically relevant O3(g) concentration (10–100 ppb) formed 2–30 µM H2O2(aq), increasing with the gas–liquid surface area, mixing, and contact duration. Thus, the water surface area facilitates the O3(g) mass transfer, which is followed by the chemical transformation of O3(aq) into H2O2(aq). These findings should also help us understand the implications of this chemistry in natural and applied contexts.