Reviewerly/exHarmony huggingface dataset:
Testing Pre-trained Language Models’ Understanding of Distributivity via Causal Mediation Analysis

To what extent do pre-trained language models grasp semantic knowledge regarding the phenomenon of distributivity?In this paper, we introduce DistNLI, a new diagnostic dataset for natural language inference that targets the semantic difference arising from distributivity, and employ the causal mediation analysis framework to quantify the model behavior and explore the underlying mechanism in this semantically-related task.We find that the extent of models' understanding is associated with model size and vocabulary size.We also provide insights into how models encode such high-level semantic knowledge.Our dataset and code are available on GitHub.
Exploring the Influence of Dialog Input Format for Unsupervised Clinical Questionnaire Filling

In the medical field, we have seen the emergence of health-bots that interact with patients to gather data and track their state.One of the downstream application is automatic questionnaire filling, where the content of the dialog is used to automatically fill a pre-defined medical questionnaire.Previous work has shown that answering questions from the dialog context can successfully be cast as a Natural Language Inference (NLI) task and therefore benefit from current pre-trained NLI models.However, NLI models have mostly been trained on text rather than dialogs, which may have an influence on their performance.In this paper, we study the influence of content transformation and content selection on the questionnaire filling task.Our results demonstrate that dialog pre-processing can significantly improve the performance of zero-shot questionnaire filling models which take health-bots dialogs as input.
An Empirical Study on Finding Spans

We present an empirical study on methods for span finding, the selection of consecutive tokens in text for some downstream tasks.We focus on approaches that can be employed in training end-to-end information extraction systems, and find there is no definitive solution without considering task properties, and provide our observations to help with future design choices: 1) a tagging approach often yields higher precision while span enumeration and boundary prediction provide higher recall; 2) span type information can benefit a boundary prediction approach; 3) additional contextualization does not help span finding in most cases.
Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification

To obtain a large amount of training labels inexpensively, researchers have recently adopted the weak supervision (WS) paradigm, which leverages labeling rules to synthesize training labels rather than using individual annotations to achieve competitive results for natural language processing (NLP) tasks.However, data imbalance is often overlooked in applying the WS paradigm, despite being a common issue in a variety of NLP tasks.To address this challenge, we propose Adaptive Ranking-based Sample Selection (ARS2), a model-agnostic framework to alleviate the data imbalance issue in the WS paradigm.Specifically, it calculates a probabilistic margin score based on the output of the current model to measure and rank the cleanliness of each data point.Then, the ranked data are sampled based on both class-wise and rule-aware ranking.In particular, the two sample strategies corresponds to our motivations: (1) to train the model with balanced data batches to reduce the data imbalance issue and (2) to exploit the expertise of each labeling rule for collecting clean samples.Experiments on four text classification datasets with four different imbalance ratios show that ARS2 outperformed the state-of-the-art imbalanced learning and WS methods, leading to a 2%-57.8%improvement on their F1-score.Our implementation can be found in https:
Coordinated Topic Modeling

We propose a new problem called coordinated topic modeling that imitates human behavior while describing a text corpus.It considers a set of well-defined topics like the axes of a semantic space with a reference representation.It then uses the axes to model a corpus for easily understandable representation.This new task helps represent a corpus more interpretably by reusing existing knowledge and benefits the corpora comparison task.We design ECTM, an embedding-based coordinated topic model that effectively uses the reference representation to capture the target corpus-specific aspects while maintaining each topic's global semantics.In ECTM, we introduce the topic-and documentlevel supervision with a self-training mechanism to solve the problem.Finally, extensive experiments on multiple domains show the superiority of our model over other baselines. 1
Semantic Role Labeling Meets Definition Modeling: Using Natural Language to Describe Predicate-Argument Structures

One of the common traits of past and present approaches for Semantic Role Labeling (SRL) is that they rely upon discrete labels drawn from a predefined linguistic inventory to classify predicate senses and their arguments.However, we argue this need not be the case.In this paper, we present an approach that leverages Definition Modeling to introduce a generalized formulation of SRL as the task of describing predicate-argument structures using natural language definitions instead of discrete labels.Our novel formulation takes a first step towards placing interpretability and flexibility foremost, and yet our experiments and analyses on PropBank-style and FrameNetstyle, dependency-based and span-based SRL also demonstrate that a flexible model with an interpretable output does not necessarily come at the expense of performance.We release our software for research purposes at https://github.com/SapienzaNLP/dsrl.
On Breadth Alone: Improving the Precision of Terminology Extraction Systems on Patent Corpora

Automatic Terminology Extraction (ATE) methods are a class of linguistic, statistical, machine learning or hybrid techniques for identifying terminology in a set of documents.Most modern ATE methods use a statistical measure of how important or characteristic a potential term is to a foreground corpus by using a second background corpus as a baseline.While many variables with ATE methods have been carefully evaluated and tuned in the literature, the effects of choosing a particular background corpus over another are not obvious.In this paper, we propose a methodology that allows us to adjust the relative breadth of the foreground and background corpora in patent documents by taking advantage of the Cooperative Patent Classification (CPC) scheme.Our results show that for every foreground corpus, the broadest background corpus gave the worst performance, in the worst case that difference is 17%.Similarly, the least broad background corpus gave suboptimal performance in all three experiments.We also demonstrate qualitative differences between background corpora -narrower background corpora tend towards more technical output.We expect our results to generalize to terminology extraction for other legal and technical documents and, generally, to the foreground/background approach to ATE.
Hard Gate Knowledge Distillation - Leverage Calibration for Robust and Reliable Language Model

In knowledge distillation, a student model is trained with supervisions from both knowledge from a teacher and observations drawn from a training data distribution.Knowledge of a teacher is considered a subject that holds interclass relations which send a meaningful supervision to a student; hence, much effort has been put to find such knowledge to be distilled.In this paper, we explore a question that has been given little attention: "when to distill such knowledge."The question is answered in our work with the concept of model calibration; we view a teacher model not only as a source of knowledge but also as a gauge to detect miscalibration of a student.This simple and yet novel view leads to a hard gate knowledge distillation scheme that switches between learning from a teacher model and training data.We verify the gating mechanism in the context of natural language generation at both the token-level and the sentence-level.Empirical comparisons with strong baselines show that hard gate knowledge distillation not only improves model generalization, but also significantly lowers model calibration error.
Recovering Gold from Black Sand: Multilingual Dense Passage Retrieval with Hard and False Negative Samples

Negative samples have not been efficiently explored in multilingual dense passage retrieval.In this paper, we propose a novel multilingual dense passage retrieval framework, mHFN, to recover and utilize hard and false negative samples.mHFN consists of three key components: 1) a multilingual hard negative sample augmentation module that allows knowledge of indistinguishable passages to be shared across multiple languages and synthesizes new hard negative samples by interpolating representations of queries and existing hard negative samples, 2) a multilingual negative sample cache queue that stores negative samples from previous batches in each language to increase the number of multilingual negative samples used in training beyond the batch size limit, and 3) a lightweight adaptive false negative sample filter that uses generated pseudo labels to separate unlabeled false negative samples and converts them into positive passages in training.We evaluate mHFN on Mr. TyDi, a high-quality multilingual dense passage retrieval dataset covering eleven typologically diverse languages, and experimental results show that mHFN outperforms strong sparse, dense and hybrid baselines and achieves new stateof-the-art performance on all languages.
Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE

Figurative language (e.g., "he flew like the wind") is challenging to understand, as it is hard to tell what implicit information is being conveyed from the surface form alone.We hypothesize that to perform this task well, the reader needs to mentally elaborate the scene being described to identify a sensible meaning of the language.We present DREAM-FLUTE, a figurative language understanding system that does this, first forming a "mental model" of situations described in a premise and hypothesis before making an entailment/contradiction decision and generating an explanation.DREAM-FLUTE uses an existing scene elaboration model, DREAM, for constructing its "mental model."In the FigLang2022 Shared Task evaluation, DREAM-FLUTE achieved (joint) first place (Acc@60=63.3%), and can perform even better with ensemble techniques, demonstrating the effectiveness of this approach. 1More generally, this work suggests that adding a reflective component to pretrained language models can improve their performance beyond standard fine-tuning (3.3% improvement in Acc@60).
DiscoSense: Commonsense Reasoning with Discourse Connectives

We present DISCOSENSE, a benchmark for commonsense reasoning via understanding a wide variety of discourse connectives.We generate compelling distractors in DISCOSENSE using Conditional Adversarial Filtering, an extension of Adversarial Filtering that employs conditional generation.We show that state-ofthe-art pre-trained language models struggle to perform well on DISCOSENSE, which makes this dataset ideal for evaluating next-generation commonsense reasoning systems.
Guiding Abstractive Dialogue Summarization with Content Planning

Abstractive dialogue summarization has recently been receiving more attention.We propose a coarse-to-fine model for generating abstractive dialogue summaries, and introduce a fact-aware reinforcement learning (RL) objective that improves the fact consistency between the dialogue and the generated summary.Initially, the model generates the predicateargument spans of the dialogue, and then generates the final summary through a fact-aware RL objective.Extensive experiments and analysis on two benchmark datasets demonstrate that our proposed method effectively improves the quality of the generated summary, especially in coherence and consistency.
SciFact-Open: Towards open-domain scientific claim verification

While research on scientific claim verification has led to the development of powerful systems that appear to approach human performance, these approaches have yet to be tested in a realistic setting against large corpora of scientific literature.Moving to this open-domain evaluation setting, however, poses unique challenges; in particular, it is infeasible to exhaustively annotate all evidence documents.In this work, we present SCIFACT-OPEN, a new test collection designed to evaluate the performance of scientific claim verification systems on a corpus of 500K research abstracts.Drawing upon pooling techniques from information retrieval, we collect evidence for scientific claims by pooling and annotating the top predictions of four state-of-the-art scientific claim verification models.We find that systems developed on smaller corpora struggle to generalize to SCIFACT-OPEN, exhibiting performance drops of at least 15 F1.In addition, analysis of the evidence in SCIFACT-OPEN reveals interesting phenomena likely to appear when claim verification systems are deployed in practice, e.g., cases where the evidence supports only a special case of the claim.Our dataset is available at https://github.com/dwadden/scifact-open.
GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation

The phenomenon of zero pronoun (ZP) has attracted increasing interest in the machine translation (MT) community due to its importance and difficulty.However, previous studies generally evaluate the quality of translating ZPs with BLEU scores on MT testsets, which are not expressive or sensitive enough for accurate assessment.To bridge the data and evaluation gaps, we propose a benchmark testset for target evaluation on Chinese-English ZP translation.The human-annotated testset covers five challenging genres, which reveal different characteristics of ZPs for comprehensive evaluation.We systematically revisit eight advanced models on ZP translation and identify current challenges for future exploration.We release data, code, models and annotation guidelines, which we hope can significantly promote research in this field.
Play música alegre: A Large-Scale Empirical Analysis of Cross-Lingual Phenomena in Voice Assistant Interactions

Cross-lingual phenomena are quite common in informal contexts like social media, where users are likely to mix their native language with English or other languages.However, few studies have focused so far on analyzing cross-lingual interactions in voice-assistant data, which present peculiar features in terms of sentence length, named entities, and use of spoken language.Also, little attention has been posed to European countries, where English is frequently used as a second language.In this paper, we present a large-scale empirical analysis of cross-lingual phenomena (code-mixing, linguistic borrowing, foreign named entities) in the interactions with Alexa in European countries.To do this, we first introduce a general, highly-scalable technique to generate synthetic mixed training data annotated with token-level language labels and we train two neural network models to predict them.We evaluate the models both on the synthetic dataset and on a real dataset of code-switched utterances, showing that the best performance is obtained by a character convolution based model.The results of the analysis highlight different behaviors between countries, having Italy with the highest ratio of cross-lingual utterances and Spain with a marked preference in keeping Spanish words.Our research, paired to the increase of the cross-lingual phenomena in time, motivates further research in developing multilingual Natural Language Understanding (NLU) models, which can naturally deal with cross-lingual interactions.
IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models

We introduce a new open information extraction (OIE) benchmark for pre-trained language models (LM).Recent studies have demonstrated that pre-trained LMs, such as BERT and GPT, may store linguistic and relational knowledge.In particular, LMs are able to answer "fill-in-the-blank" questions when given a pre-defined relation category.Instead of focusing on pre-defined relations, we create an OIE benchmark aiming to fully examine the open relational information present in the pretrained LMs.We accomplish this by turning pre-trained LMs into zero-shot OIE systems.Surprisingly, pre-trained LMs are able to obtain competitive performance on both standard OIE datasets (CaRB and Re-OIE2016) and two new large-scale factual OIE datasets (TAC KBP-OIE and Wikidata-OIE) that we establish via distant supervision.For instance, the zeroshot pre-trained LMs outperform the F1 score of the state-of-the-art supervised OIE methods on our factual OIE datasets without needing to use any training sets. 1
Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering

Fighting online hate speech is a challenge that is usually addressed using Natural Language Processing via automatic detection and removal of hate content. Besides this approach, counter narratives have emerged as an effective tool employed by NGOs to respond to online hate on social media platforms. For this reason, Natural Language Generation is currently being studied as a way to automatize counter narrative writing. However, the existing resources necessary to train NLG models are limited to 2-turn interactions (a hate speech and a counter narrative as response), while in real life, interactions can consist of multiple turns. In this paper, we present a hybrid approach for dialogical data collection, which combines the intervention of human expert annotators over machine generated dialogues obtained using 19 different configurations. The result of this work is DIALOCONAN, the first dataset comprising over 3000 fictitious multi-turn dialogues between a hater and an NGO operator, covering 6 targets of hate.
Lexi: Self-Supervised Learning of the UI Language

Humans can learn to operate the user interface (UI) of an application by reading an instruction manual or how-to guide.Along with text, these resources include visual content such as UI screenshots and images of application icons referenced in the text.We explore how to leverage this data to learn generic visio-linguistic representations of UI screens and their components.These representations are useful in many real applications, such as accessibility, voice navigation, and task automation.Prior UI representation models rely on UI metadata (UI trees and accessibility labels), which is often missing, incompletely defined, or not accessible.We avoid such a dependency, and propose Lexi, a pre-trained vision and language model designed to handle the unique features of UI screens, including their text richness and context sensitivity.To train Lexi we curate the UICaption dataset consisting of 114k UI images paired with descriptions of their functionality.We evaluate Lexi on four tasks: UI action entailment, instruction-based UI image retrieval, grounding referring expressions, and UI entity recognition.
Calibrating Student Models for Emotion-related Tasks

Knowledge Distillation (KD) is an effective method to transfer knowledge from one network (a.k.a.teacher) to another (a.k.a.student).In this paper, we study KD on the emotionrelated tasks from a new perspective: calibration.We further explore the impact of the mixup data augmentation technique on the distillation objective and propose to use a simple yet effective mixup method informed by training dynamics for calibrating the student models.Underpinned by the regularization impact of the mixup process by providing better training signals to the student models using training dynamics, our proposed mixup strategy gradually enhances the student model's calibration while effectively improving its performance.We evaluate the calibration of pretrained language models through knowledge distillation over three tasks of emotion detection, sentiment analysis, and empathy detection.By conducting extensive experiments on different datasets, with both in-domain and out-ofdomain test sets, we demonstrate that student models distilled from teacher models trained using our proposed mixup method obtained the lowest Expected Calibration Errors (ECEs) and best performance on both in-domain and outof-domain test sets.
Outlier Dimensions that Disrupt Transformers are Driven by Frequency

While Transformer-based language models are generally very robust to pruning, there is the recently discovered outlier phenomenon: disabling only 48 out of 110M parameters in BERT-base drops its performance by nearly 30% on MNLI. We replicate the original evidence for the outlier phenomenon and we link it to the geometry of the embedding space. We find that in both BERT and RoBERTa the magnitude of hidden state coefficients corresponding to outlier dimensions correlate with the frequencies of encoded tokens in pre-training data, and they also contribute to the “vertical” self-attention pattern enabling the model to focus on the special tokens. This explains the drop in performance from disabling the outliers, and it suggests that to decrease anisotopicity in future models we need pre-training schemas that would better take into account the skewed token distributions.
Syntax-guided Localized Self-attention by Constituency Syntactic Distance

Recent works have revealed that Transformers are implicitly learning the syntactic information in its lower layers from data, albeit is highly dependent on the quality and scale of the training data.However, learning syntactic information from data is not necessary if we can leverage an external syntactic parser, which provides better parsing quality with welldefined syntactic structures.This could potentially improve Transformer's performance and sample efficiency.In this work, we propose a syntax-guided localized self-attention for Transformer that allows directly incorporating grammar structures from an external constituency parser.It prohibits the attention mechanism to overweight the grammatically distant tokens over close ones.Experimental results show that our model could consistently improve translation performance on a variety of machine translation datasets, ranging from small to large dataset sizes, and with different source languages.
Mind Your Bias: A Critical Review of Bias Detection Methods for Contextual Language Models

The awareness and mitigation of biases are of fundamental importance for the fair and transparent use of contextual language models, yet they crucially depend on the accurate detection of biases as a precursor.Consequently, numerous bias detection methods have been proposed, which vary in their approach, the considered type of bias, and the data used for evaluation.However, while most detection methods are derived from the word embedding association test for static word embeddings, the reported results are heterogeneous, inconsistent, and ultimately inconclusive.To address this issue, we conduct a rigorous analysis and comparison of bias detection methods for contextual language models.Our results show that minor design and implementation decisions (or errors) have a substantial and often significant impact on the derived bias scores.Overall, we find the state of the field to be both worse than previously acknowledged due to systematic and propagated errors in implementations, yet better than anticipated since divergent results in the literature homogenize after accounting for implementation errors.Based on our findings, we conclude with a discussion of paths towards more robust and consistent bias detection methods.