nils-herrmann/clean_scirepeval huggingface dataset:
Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space

A notable property of word embeddings is that word relationships can exist as linear substructures in the embedding space. For example, ‘gender’ corresponds to v_woman - v_man and v_queen - v_king. This, in turn, allows word analogies to be solved arithmetically: v_king - v_man + v_woman = v_queen. This property is notable because it suggests that models trained on word embeddings can easily learn such relationships as geometric translations. However, there is no evidence that models exclusively represent relationships in this manner. We document an alternative way in which downstream models might learn these relationships: orthogonal and linear transformations. For example, given a translation vector for ‘gender’, we can find an orthogonal matrix R, representing a rotation and reflection, such that R(v_king) = v_queen and R(v_man) = v_woman. Analogical reasoning using orthogonal transformations is almost as accurate as using vector arithmetic; using linear transformations is more accurate than both. Our findings suggest that these transformations can be as good a representation of word relationships as translation vectors.
Commonsense Knowledge Mining from Pretrained Models

Inferring commonsense knowledge is a key challenge in machine learning. Due to the sparsity of training data, previous work has shown that supervised methods for commonsense knowledge mining underperform when evaluated on novel data. In this work, we develop a method for generating commonsense knowledge using a large, pre-trained bidirectional language model. By transforming relational triples into masked sentences, we can use this model to rank a triple’s validity by the estimated pointwise mutual information between the two entities. Since we do not update the weights of the bidirectional model, our approach is not biased by the coverage of any one commonsense knowledge base. Though we do worse on a held-out test set than models explicitly trained on a corresponding training set, our approach outperforms these methods when mining commonsense knowledge from new sources, suggesting that our unsupervised technique generalizes better than current supervised approaches.
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings

Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks. However, just how contextual are the contextualized representations produced by models such as ELMo and BERT? Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations? For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers. This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations. In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word’s contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.
Causal Discovery by Kernel Intrinsic Invariance Measure

Reasoning based on causality, instead of association has been considered as a key ingredient towards real machine intelligence. However, it is a challenging task to infer causal relationship/structure among variables. In recent years, an Independent Mechanism (IM) principle was proposed, stating that the mechanism generating the cause and the one mapping the cause to the effect are independent. As the conjecture, it is argued that in the causal direction, the conditional distributions instantiated at different value of the conditioning variable have less variation than the anti-causal direction. Existing state-of-the-arts simply compare the variance of the RKHS mean embedding norms of these conditional distributions. In this paper, we prove that this norm-based approach sacrifices important information of the original conditional distributions. We propose a Kernel Intrinsic Invariance Measure (KIIM) to capture higher order statistics corresponding to the shapes of the density functions. We show our algorithm can be reduced to an eigen-decomposition task on a kernel matrix measuring intrinsic deviance/invariance. Causal directions can then be inferred by comparing the KIIM scores of two hypothetic directions. Experiments on synthetic and real data are conducted to show the advantages of our methods over existing solutions.
On the Knowledge Graph Completion Using Translation Based Embedding: The Loss Is as Important as the Score

Knowledge graphs (KGs) represent world's facts in structured forms. KG completion exploits the existing facts in a KG to discover new ones. Translation-based embedding model (TransE) is a prominent formulation to do KG completion. Despite the efficiency of TransE in memory and time, it suffers from several limitations in encoding relation patterns such as many-to-many relation patterns, symmetric, reflexive etc. To tackle this problem, most of the attempts have circled around the revision of the score function of TransE i.e., proposing a more complicated score function such as Trans(A, D, G, H, R, etc) to mitigate the limitations. In this paper, we tackle this problem from a different perspective. We pose theoretical investigations of the main limitations of TransE in the light of loss function rather than the score function. To the best of our knowledge, this has not been investigated so far comprehensively. We show that by a proper selection of the loss function for training the TransE model, the main limitations of the model are mitigated. This is explained by setting upper-bound for the scores of positive samples, showing the region of truth (i.e., the region that a triple is considered positive by the model). Our theoretical proofs with experimental results fill the gap between the capability of translation-based class of embedding models and the loss function. The theories emphasize the importance of the selection of the loss functions for training the models. Our experimental evaluations on different loss functions used for training the models justify our theoretical proofs and confirm the importance of the loss functions on the performance.
Modeling Long-Range Context for Concurrent Dialogue Acts Recognition

In dialogues, an utterance is a chain of consecutive sentences produced by one speaker which ranges from a short sentence to a thousand-word post. When studying dialogues at the utterance level, it is not uncommon that an utterance would serve multiple functions. For instance, "Thank you. It works great." expresses both gratitude and positive feedback in the same utterance. Multiple dialogue acts (DA) for one utterance breeds complex dependencies across dialogue turns. Therefore, DA recognition challenges a model's predictive power over long utterances and complex DA context. We term this problem Concurrent Dialogue Acts (CDA) recognition. Previous work on DA recognition either assumes one DA per utterance or fails to realize the sequential nature of dialogues. In this paper, we present an adapted Convolutional Recurrent Neural Network (CRNN) which models the interactions between utterances of long-range context. Our model significantly outperforms existing work on CDA recognition on a tech forum dataset.
On Privacy of Socially Contagious Attributes

A common approach to protect user's privacy in data collection is to perform random perturbations on user's sensitive data before collection in a way that aggregated statistics can still be inferred without endangering individual secrets. In this paper, we take a closer look at the validity of Differential Privacy guarantees, when sensitive attributes are subject to social contagion. We first show that in the absence of any knowledge about the contagion network, an adversary that tries to predict the real values from perturbed ones, cannot train a classifier that achieves an area under the ROC curve (AUC) above 1-(1-δ)/(1+e^ε), if the dataset is perturbed using an (ε,δ)-differentially private mechanism. Then, we show that with the knowledge of the contagion network and model, one can do substantially better. We demonstrate that our method passes the performance limit imposed by differential privacy. Our experiments also reveal that nodes with high influence on others are at more risk of revealing their secrets than others. Our method's superior performance is demonstrated through extensive experiments on synthetic and real-world networks.
A Relation between Short-Term and Long-Term Arbitrage

In this work a relation between a measure of short-term arbitrage in the market and the excess growth of portfolios as a notion of long-term arbitrage is established. The former originates from "Geometric Arbitrage Theory" and the latter from "Stochastic Portfolio Theory". Both aim to describe non-equilibrium effects in financial markets. Thereby, a connection between two different theoretical frameworks of arbitrage is drawn.
SUM-QE: a BERT-based Summary Quality Estimation Model

We propose SUM-QE, a novel Quality Estimation model for summarization based on BERT. The model addresses linguistic quality aspects that are only indirectly captured by content-based approaches to summary evaluation, without involving comparison with human references. SUM-QE achieves very high correlations with human ratings, outperforming simpler models addressing these linguistic aspects. Predictions of the SUM-QE model can be used for system development, and to inform users of the quality of automatically produced summaries and other types of generated text.
Subword Language Model for Query Auto-Completion

Current neural query auto-completion (QAC) systems rely on character-level language models, but they slow down when queries are long. We present how to utilize subword language models for the fast and accurate generation of query completion candidates. Representing queries with subwords shorten a decoding length significantly. To deal with issues coming from introducing subword language model, we develop a retrace algorithm and a reranking method by approximate marginalization. As a result, our model achieves up to 2.5 times faster while maintaining a similar quality of generated results compared to the character-level baseline. Also, we propose a new evaluation metric, mean recoverable length (MRL), measuring how many upcoming characters the model could complete correctly. It provides more explicit meaning and eliminates the need for prefix length sampling for existing rank-based metrics. Moreover, we performed a comprehensive analysis with ablation study to figure out the importance of each component.
Photometric redshifts for X-ray-selected active galactic nuclei in the eROSITA era

With the launch of eROSITA (extended Roentgen Survey with an Imaging Telescope Array), successfully occurred on 2019 July 13, we are facing the challenge of computing reliable photometric redshifts for 3 million of active galactic nuclei (AGNs) over the entire sky, having available only patchy and inhomogeneous ancillary data. While we have a good understanding of the photo-z quality obtainable for AGN using spectral energy distribution (SED)-fitting technique, we tested the capability of machine learning (ML), usually reliable in computing photo-z for QSO in wide and shallow areas with rich spectroscopic samples. Using MLPQNA as example of ML, we computed photo-z for the X-ray-selected sources in Stripe 82X, using the publicly available photometric and spectroscopic catalogues. Stripe 82X is at least as deep as eROSITA will be and wide enough to include also rare and bright AGNs. In addition, the availability of ancillary data mimics what can be available in the whole sky. We found that when optical, and near- and mid-infrared data are available, ML and SED fitting perform comparably well in terms of overall accuracy, realistic redshift probability density functions, and fraction of outliers, although they are not the same for the two methods. The results could further improve if the photometry available is accurate and including morphological information. Assuming that we can gather sufficient spectroscopy to build a representative training sample, with the current photometry coverage we can obtain reliable photo-z for a large fraction of sources in the Southern hemisphere well before the spectroscopic follow-up, thus timely enabling the eROSITA science return. The photo-z catalogue is released here.
Interstellar dust along the line of sight of GX 3+1

Studying absorption and scattering of X-ray radiation by interstellar dust grains allows us to access the physical and chemical properties of cosmic grains even in the densest regions of the Galaxy.We aim at characterising the dust silicate population which presents clear absorption features in the energy band covered by theChandraX-ray Observatory. Through these absorption features, in principle, it is possible to infer the size distribution, composition, and structure of silicate in the interstellar medium. In particular, in this work we investigate magnesium and silicon K-edges.We built X-ray extinction models for 15 dust candidates using newly acquired synchrotron measurements. These models were adapted for astrophysical analysis and implemented in the SPEXspectral fitting program. We used the models to reproduce the dust absorption features observed in the spectrum of the bright low mass X-ray binary GX 3+1, which is used as a background source.With the simultaneous analysis of the two edges we test two different size distributions of dust: one corresponding to the standard Mathis-Rumpl-Nordsieck model and one considering larger grains (n(a) ∝ai−3.5with 0.005μm <a1< 0.25μm and 0.05μm <a2< 0.5μm, respectively, withathe grain size). These distributions may be representative of the complex Galactic region towards this source. We find that up to 70% of dust is constituted by amorphous olivine. We discuss the crystallinity of the cosmic dust found along this line of sight. Both magnesium and silicon are highly depleted into dust (δZ= 0.89 and 0.94, respectively), while their total abundance does not depart from solar values.
Dark matter signals on laser interferometer

WIMPs are promising dark matter candidates. When WIMPs collide with a mirror equipped with interferometric gravitational wave detectors, we expect that characteristic motions of the pendulum and mirror are excited, and those signals could be extracted by highly sophisticated sensors developed for gravitational wave detection. We analyze the motions of the pendulum and mirror, and estimate the detectability of these motions. For the “Thin-ET” detector, the signal to noise ratio may be 1.7(mDM100GeV) , where m DM is the mass of a WIMP. We may set a more strict upper limit on the cross section between a WIMP and a nucleon than the limits obtained by other experiments so far when m DM is approximately lower than 0.2 GeV. We find an order of magnitude improvement in the upper limit around m DM = 0.2 GeV.
Potential automorphy of $\widehat{G}$-local systems.

Vincent Lafforgue has recently made a spectacular breakthrough in the setting of the global Langlands correspondence for global fields of positive characteristic, by constructing the `automorphic--to--Galois' direction of the correspondence for an arbitrary reductive group $G$. We discuss a result that starts with Lafforgue's work and proceeds in the opposite (`Galois--to--automorphic') direction.
PrTransH: Embedding Probabilistic Medical Knowledge from Real World EMR Data

This paper proposes an algorithm named as PrTransH to learn embedding vectors from real world EMR data based medical knowledge. The unique challenge in embedding medical knowledge graph from real world EMR data is that the uncertainty of knowledge triplets blurs the border between "correct triplet" and "wrong triplet", changing the fundamental assumption of many existing algorithms. To address the challenge, some enhancements are made to existing TransH algorithm, including: 1) involve probability of medical knowledge triplet into training objective; 2) replace the margin-based ranking loss with unified loss calculation considering both valid and corrupted triplets; 3) augment training data set with medical background knowledge. Verifications on real world EMR data based medical knowledge graph prove that PrTransH outperforms TransH in link prediction task. To the best of our survey, this paper is the first one to learn and verify knowledge embedding on probabilistic knowledge graphs.
Optimizing exoplanet atmosphere retrieval using unsupervised machine-learning classification

One of the principal bottlenecks to atmosphere characterization in the era of all-sky surveys is the availability of fast, autonomous, and robust atmospheric retrieval methods. We present a new approach using unsupervised machine learning to generate informed priors for retrieval of exoplanetary atmosphere parameters from transmission spectra. We use principal component analysis (PCA) to efficiently compress the information content of a library of transmission spectra forward models generated using the platon package. We then apply a k-means clustering algorithm in PCA space to segregate the library into discrete classes. We show that our classifier is almost always able to instantaneously place a previously unseen spectrum into the correct class, for low-to-moderate spectral resolutions, R, in the range R = 30−300 and noise levels up to 10 per cent of the peak-to-trough spectrum amplitude. The distribution of physical parameters for all members of the class therefore provides an informed prior for standard retrieval methods such as nested sampling. We benchmark our informed-prior approach against a standard uniform-prior nested sampler, finding that our approach is up to a factor of 2 faster, with negligible reduction in accuracy. We demonstrate the application of this method to existing and near-future observatories, and show that it is suitable for real-world application. Our general approach is not specific to transmission spectroscopy and should be more widely applicable to cases that involve the repetitive fitting of trusted high-dimensional models to large data catalogues, including beyond exoplanetary science.
Pathologies of Factorised Gaussian and MC Dropout Posteriors in Bayesian Neural Networks

Neural networks provide state-of-the-art performance on a variety of tasks. However, they are often overconfident when making predictions. This inability to properly account for uncertainty limits their application to high-risk decision making, active learning and Bayesian optimisation. To address this, Bayesian inference has been proposed as a framework for improving uncertainty estimates. In practice, Bayesian neural networks rely on poorly understood approximations for computational tractability. We prove that two commonly used approximation methods, the factorised Gaussian assumption and Monte Carlo dropout, lead to pathological estimates of the predictive uncertainty in single hidden layer ReLU networks. This indicates that more flexible approximations are needed to obtain reliable uncertainty estimates.
War pact model of shrinking networks

Many real systems can be described by a set of interacting entities forming a complex network. To some surprise, these have been shown to share a number of structural properties regardless of their type or origin. It is thus of vital importance to design simple and intuitive models that can explain their intrinsic structure and dynamics. These can, for instance, be used to study networks analytically or to construct networks not observed in real life. Most models proposed in the literature are of two types. A model can be either static, where edges are added between a fixed set of nodes according to some predefined rule, or evolving, where the number of nodes or edges increases over time. However, some real networks do not grow but rather shrink, meaning that the number of nodes or edges decreases over time. We here propose a simple model of shrinking networks called the war pact model. We show that networks generated in such a way exhibit common structural properties of real networks. Furthermore, compared to classical models, these resemble international trade, correlates of war, Bitcoin transactions and other networks more closely. Network shrinking may therefore represent a reasonable explanation of the evolution of some networks and greater emphasis should be put on such models in the future.
Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generation

Existing approaches to dialogue state tracking rely on pre-defined ontologies consisting of a set of all possible slot types and values. Though such approaches exhibit promising performance on single-domain benchmarks, they suffer from computational complexity that increases proportionally to the number of pre-defined slots that need tracking. This issue becomes more severe when it comes to multi-domain dialogues which include larger numbers of slots. In this paper, we investigate how to approach DST using a generation framework without the pre-defined ontology list. Given each turn of user utterance and system response, we directly generate a sequence of belief states by applying a hierarchical encoder-decoder structure. In this way, the computational complexity of our model will be a constant regardless of the number of pre-defined slots. Experiments on both the multi-domain and the single domain dialogue state tracking dataset show that our model not only scales easily with the increasing number of pre-defined domains and slots but also reaches the state-of-the-art performance.
Targeted Example Generation for Compilation Errors

We present TEGCER, an automated feedback tool for novice programmers. TEGCER uses supervised classification to match compilation errors in new code submissions with relevant pre-existing errors, submitted by other students before. The dense neural network used to perform this classification task is trained on 15000+ error-repair code examples. The proposed model yields a test set classification Pred@3 accuracy of 97.7% across 212 error category labels. Using this model as its base, TEGCER presents students with the closest relevant examples of solutions for their specific error on demand. A large scale (N>230) usability study shows that students who use TEGCER are able to resolve errors more than 25% faster on average than students being assisted by human tutors.
Conditional Vehicle Trajectories Prediction in CARLA Urban Environment

Imitation learning is becoming more and more successful for autonomous driving. End-to-end (raw signal to command) performs well on relatively simple tasks (lane keeping and navigation). Mid-to-mid (environment abstraction to mid-level trajectory representation) or direct perception (raw signal to performance) approaches strive to handle more complex, real life environment and tasks (e.g. complex intersection). In this work, we show that complex urban situations can be handled with raw signal input and mid-level representation. We build a hybrid end-to-mid approach predicting trajectories for neighbor vehicles and for the ego vehicle with a conditional navigation goal. We propose an original architecture inspired from social pooling LSTM taking low and mid level data as input and producing trajectories as polynomials of time. We introduce a label augmentation mechanism to get the level of generalization that is required to control a vehicle. The performance is evaluated on CARLA 0.8 benchmark, showing significant improvements over previously published state of the art.