Reviewerly/exHarmony huggingface dataset:
To Prefer or to Choose? Generating Agency and Power Counterfactuals Jointly for Gender Bias Mitigation

Gender bias may emerge from an unequal representation of agency and power, for example, by portraying women frequently as passive and powerless ("She accepted her future") and men as proactive and powerful ("He chose his future").When language models learn from respective texts, they may reproduce or even amplify the bias.An effective way to mitigate bias is to generate counterfactual sentences with opposite agency and power to the training.Recent work targeted agency-specific verbs from a lexicon to this end.We argue that this is insufficient, due to the interaction of agency and power and their dependence on context.In this paper, we thus develop a new rewriting model that identifies verbs with the desired agency and power in the context of the given sentence.The verbs' probability is then boosted to encourage the model to rewrite both connotations jointly.According to automatic metrics, our model effectively controls for power while being competitive in agency to the state of the art.In our evaluation, human annotators favored its counterfactuals in terms of both connotations, also deeming its meaning preservation better.
Syntactic and Semantic Uniformity for Semantic Parsing and Task-Oriented Dialogue Systems

This paper proposes a data representation framework for semantic parsing and taskoriented dialogue systems, aiming to achieve a uniform representation for syntactically and semantically diverse machine-readable formats.Current NLP systems heavily rely on adapting pre-trained language models to specific tasks, and this approach has been proven effective for modeling natural language texts.However, little attention has been paid to the representation of machine-readable formats, such as database queries and dialogue states.We present a method for converting original machine-readable formats of semantic parsing and task-oriented dialogue datasets into a syntactically and semantically uniform representation.We define a meta grammar for syntactically uniform representations and translate semantically equivalent functions into a uniform vocabulary.Empirical experiments on 13 datasets show that accuracy consistently improves over original formats, revealing the advantage of the proposed representation.Additionally, we show that the proposed representation allows for transfer learning across datasets.
CHIA: CHoosing Instances to Annotate for Machine Translation

Neural machine translation (MT) systems have been shown to perform poorly on low-resource language pairs, for which large-scale parallel data is unavailable.Making the data annotation process faster and cheaper is therefore important to ensure equitable access to MT systems.To make optimal use of a limited annotation budget, we present CHIA (choosing instances to annotate), a method for selecting instances to annotate for machine translation.Using an existing multi-way parallel dataset of high-resource languages, we first identify instances, based on model training dynamics, that are most informative for training MT models for high-resource languages.We find that there are cross-lingual commonalities in instances that are useful for MT model training, which we use to identify instances that will be useful to train models on a new target language.Evaluating on 20 languages from two corpora, we show that training on instances selected using our method provides an average performance improvement of 1.59 BLEU over training on randomly selected instances of the same size.
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization

Combining the visual modality with pretrained language models has been surprisingly effective for simple descriptive tasks such as image captioning.More general text generation however remains elusive.We take a step back and ask: How do these models work for more complex generative tasks, i.e. conditioning on both text and images?Are multimodal models simply visually adapted language models, or do they combine they reason jointly over modalities?We investigate these questions in the context of self-rationalization (jointly generating task labels/answers and free-text explanations) of three tasks: (i) visual question answering in VQA-X, (ii) visual commonsense reasoning in VCR, and (iii) visual-textual entailment in E-SNLI-VE.We show that recent unimodal advances, CLIP image representations and scaling of language models, do not consistently improve self-rationalization in multimodal tasks.We find that no single model type works universally best across tasks, datasets, and finetuning data sizes.Our findings motivate the need for novel general backbones approach that move text generation from images and text beyond image captioning.
How to Stop an Avalanche? JoDeM: Joint Decision Making through Compare and Contrast for Dialog State Tracking

Dialog state tracking (DST) is a core component in task-oriented dialog systems.Existing state-of-the-art DST model incorporates insight and intuition from the human experience into design of supplementary labels, which greatly assisted the training process of turn-by-turn DST model.Though the turn-by-turn scheme and supplementary labels enabled satisfactory performance on the task, most of the DST models of this fashion label or process the raw dialogue data on the premise that the last turn dialogue state is always correct, which is usually not the case.In this paper, we address the negative impact resulted from the premise above as the avalanche phenomenon.After that, we propose JoDeM, a state-of-the-art DST model which can tackle the Avalanche phenomenon with two mechanisms.First mechanism is a jointly decision making method to extract key information from the dialogue.Second mechanism is a compare and contrast dialogue update technique to prevent error accumulation.Example study and graph analysis are presented to support our claim about the harmfulness of avalanche phenomenon.We also conduct quantitative and qualitative experiments on the high quality MultiWOZ2.3corpus dataset to demonstrate that the proposed model not only outperforms the existing state-of-the-art methods, but also proves the validity of solving avalanche degradation problem.
OpenStance: Real-world Zero-shot Stance Detection

Prior studies of zero-shot stance detection identify the attitude of texts towards unseen topics occurring in the same document corpus.Such task formulation has three limitations: (i) Single domain/dataset.A system is optimized on a particular dataset from a single domain; therefore, the resulting system cannot work well on other datasets; (ii) the model is evaluated on a limited number of unseen topics; (iii) it is assumed that part of the topics has rich annotations, which might be impossible in real-world applications.These drawbacks will lead to an impractical stance detection system that fails to generalize to open domains and open-form topics. This work definesOpenStance: opendomain zero-shot stance detection, aiming to handle stance detection in an open world with neither domain constraints nor topicspecific annotations.The key challenge of OpenStance lies in the open-domain generalization: learning a system with fully unspecific supervision but capable of generalizing to any dataset.To solve OpenStance, we propose to combine indirect supervision, from textual entailment datasets, and weak supervision, from data generated automatically by pretrained Language Models.Our single system, without any topic-specific supervision, outperforms the supervised method on three popular datasets.To our knowledge, this is the first work that studies stance detection under the open-domain zero-shot setting.All data and code are publicly released. 1
Simple Questions Generate Named Entity Recognition Datasets

Recent named entity recognition (NER) models often rely on human-annotated datasets, requiring the significant engagement of professional knowledge on the target domain and entities.This research introduces an ask-to-generate approach that automatically generates NER datasets by asking questions in simple natural language to an open-domain question answering system (e.g., "Which disease?").Despite using fewer in-domain resources, our models, solely trained on the generated datasets, largely outperform strong low-resource models by an average F1 score of 19.4 for six popular NER benchmarks.Furthermore, our models provide competitive performance with rich-resource models that additionally leverage in-domain dictionaries provided by domain experts.In few-shot NER, we outperform the previous best model by an F1 score of 5.2 on three benchmarks and achieve new state-of-the-art performance.The code and datasets are available at
Coherent Long Text Generation by Contrastive Soft Prompt

Improving the coherence of long text generation is an important but challenging task.Existing models still struggle to generate a logical and coherent sentence sequence.It is difficult for a model to plan long text generation and avoid generating incoherent texts from a high-level semantic perspective.We conjecture that this is due to two factors: (1) current training methods mainly rely on maximum likelihood estimation computed from token-level probability prediction; (2) the role of incoherent texts has been largely under-explored, thus the noised generated texts with errors are outof-distribution for the model.To address these issues, in this paper, we propose a Contrastive Soft Prompt (CSP) model for improving the coherence of long text generation.It learns text representations in the hidden space for better planning long text generation.To this end, it jointly learns to generate a text representation close to representations of coherent texts and away from incoherent ones, and then generates long text taking this representation as the soft prompt.We conduct experiments on two public story generation datasets, and experimental results show that our method can generate more coherent stories than the state-of-the-art model.
Instance Regularization for Discriminative Language Model Pre-training

Discriminative pre-trained language models (PrLMs) can be generalized as denoising auto-encoders that work with two procedures, ennoising and denoising.First, an ennoising process corrupts texts with arbitrary noising functions to construct training instances.Then, a denoising language model is trained to restore the corrupted tokens.Existing studies have made progress by optimizing independent strategies of either ennoising or denosing.They treat training instances equally throughout the training process, with little attention on the individual contribution of those instances.To model explicit signals of instance contribution, this work proposes to estimate the complexity of restoring the original sentences from corrupted ones in language model pretraining.The estimations involve the corruption degree in the ennoising data construction process and the prediction confidence in the denoising counterpart.Experimental results on natural language understanding and reading comprehension benchmarks show that our approach improves pre-training efficiency, effectiveness, and robustness.Code is publicly
Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models

End-to-end spoken language understanding (SLU) systems are gaining popularity over cascaded approaches due to their simplicity and ability to avoid error propagation.However, these systems model sequence labeling as a sequence prediction task causing a divergence from its well-established token-level tagging formulation.We build compositional end-toend SLU systems that explicitly separate the added complexity of recognizing spoken mentions in SLU from the NLU task of sequence labeling.By relying on intermediate decoders trained for ASR, our end-to-end systems transform the input modality from speech to tokenlevel representations that can be used in the traditional sequence labeling framework.This composition of ASR and NLU formulations in our end-to-end SLU system offers direct compatibility with pre-trained ASR and NLU systems, allows performance monitoring of individual components and enables the use of globally normalized losses like CRF, making them attractive in practical scenarios.Our models outperform both cascaded and direct end-toend models on a labeling task of named entity recognition across SLU benchmarks. 1
Learning to Explain Selectively: A Case Study on Question Answering

Explanations promise to bridge the gap between humans and AI, yet it remains difficult to achieve consistent improvement in AIaugmented human decision making.The usefulness of AI explanations depends on many factors, and always showing the same type of explanation in all cases is suboptimal-so is relying on heuristics to adapt explanations for each scenario.We propose learning to explain "selectively": for each decision that the user makes, we use a model to choose the best explanation from a set of candidates and update this model with feedback to optimize human performance.We experiment on a question answering task, Quizbowl, and show that selective explanations improve human performance for both experts and crowdworkers.
Fine-Tuning Pre-trained Transformers into Decaying Fast Weights

Autoregressive Transformers are strong language models but incur O(T) complexity during per-token generation due to the self-attention mechanism. Recent work proposes kernel-based methods to approximate causal self-attention by replacing it with recurrent formulations with various update rules and feature maps to achieve O(1) time and memory complexity. We explore these approaches and find that they are unnecessarily complex, and propose a simple alternative - decaying fast weights - that runs fast on GPU, outperforms prior methods, and retains 99% of attention’s performance for GPT-2. We also show competitive performance on WikiText-103 against more complex attention substitutes.
Fixing Model Bugs with Natural Language Patches

Current approaches for fixing systematic problems in NLP models (e.g., regex patches, finetuning on more data) are either brittle, or labor-intensive and liable to shortcuts.In contrast, humans often provide corrections to each other through natural language.Taking inspiration from this, we explore natural language patches-declarative statements that allow developers to provide corrective feedback at the right level of abstraction, either overriding the model ("if a review gives 2 stars, the sentiment is negative") or providing additional information the model may lack ("if something is described as the bomb, then it is good").We model the task of determining if a patch applies separately from the task of integrating patch information, and show that with a small amount of synthetic data, we can teach models to effectively use real patches on real data-1 to 7 patches improve accuracy by ~1-4 accuracy points on different slices of a sentiment analysis dataset, and F1 by 7 points on a relation extraction dataset.Finally, we show that finetuning on as many as 100 labeled examples may be needed to match the performance of a small set of language patches.
COMET-QE and Active Learning for Low-Resource Machine Translation

Active learning aims to deliver maximum benefit when resources are scarce.We use COMET-QE, a reference-free evaluation metric, to select sentences for low-resource neural machine translation.Using Swahili, Kinyarwanda and Spanish for our experiments, we show that COMET-QE significantly outperforms two variants of Round Trip Translation Likelihood (RTTL) and random sentence selection by up to 5 BLEU points for 20k sentences selected by Active Learning on a 30k baseline.This suggests that COMET-QE is a powerful tool for sentence selection in the very low-resource limit.
CDConv: A Benchmark for Contradiction Detection in Chinese Conversations

Dialogue contradiction is a critical issue in open-domain dialogue systems.The contextualization nature of conversations makes dialogue contradiction detection rather challenging.In this work, we propose a benchmark for Contradiction Detection in Chinese Conversations, namely CDCONV.It contains 12K multi-turn conversations annotated with three typical contradiction categories: Intrasentence Contradiction, Role Confusion, and History Contradiction.To efficiently construct the CDCONV conversations, we devise a series of methods for automatic conversation generation, which simulate common user behaviors that trigger chatbots to make contradictions.We conduct careful manual quality screening of the constructed conversations and show that state-of-the-art Chinese chatbots can be easily goaded into making contradictions.Experiments on CDCONV show that properly modeling contextual information is critical for dialogue contradiction detection, but there are still unresolved challenges that require future research.
JoeyS2T: Minimalistic Speech-to-Text Modeling with JoeyNMT

JoeyS2T is a JoeyNMT (Kreutzer et al., 2019) extension for speech-to-text tasks such as automatic speech recognition and end-to-end speech translation.It inherits the core philosophy of JoeyNMT, a minimalist NMT toolkit built on PyTorch, seeking simplicity and accessibility.JoeyS2T's workflow is self-contained, starting from data pre-processing, over model training and prediction to evaluation, and is seamlessly integrated into JoeyNMT's compact and simple code base.On top of JoeyNMT's state-of-the-art Transformer-based encoderdecoder architecture, JoeyS2T provides speechoriented components such as convolutional layers, SpecAugment, CTC-loss, and WER evaluation.Despite its simplicity compared to prior implementations, JoeyS2T performs competitively on English speech recognition and English-to-German speech translation benchmarks.The implementation is accompanied by a walk-through tutorial and available on https://github.com/may-/joeys2t.
Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Games

Language models pre-trained on large selfsupervised corpora, followed by task-specific fine-tuning has become the dominant paradigm in NLP.These pre-training datasets often have a one-to-many structure-e.g. in dialogue there are many valid responses for a given context.However, only some of these responses will be desirable in our downstream task.This raises the question of how we should train the model such that it can emulate the desirable behaviours, but not the undesirable ones.Current approaches train in a one-to-one setup-only a single target response is given for a single dialogue context-leading to models only learning to predict the average response, while ignoring the full range of possible responses.Using textbased games as a testbed, our approach, PASA, uses discrete latent variables to capture the range of different behaviours represented in our larger pre-training dataset.We then use knowledge distillation to distil the posterior probability distribution into a student model.This probability distribution is far richer than learning from only the hard targets of the dataset, and thus allows the student model to benefit from the richer range of actions the teacher model has learned.Results show up to 49% empirical improvement over the previous state-of-the-art model on the Jericho Walkthroughs dataset.
Robust Task-Oriented Dialogue Generation with Contrastive Pre-training and Adversarial Filtering

Data artifacts incentivize machine learning models to learn non-transferable generalizations by taking advantage of shortcuts in the data, and there is growing evidence that data artifacts play a role for the strong results that deep learning models achieve in recent natural language processing benchmarks.In this paper, we focus on task-oriented dialogue and investigate whether popular datasets such as Mul-tiWOZ contain such data artifacts.We found that by only keeping frequent phrases in the training examples, state-of-the-art models perform similarly compared to the variant trained with full data, suggesting they exploit these spurious correlations to solve the task.Motivated by this, we propose a contrastive learning based framework to encourage the model to ignore these cues and focus on learning generalisable patterns.We also experiment with adversarial filtering to remove "easy" training instances so that the model would focus on learning from the "harder" instances.We conduct a number of generalization experiments -e.g., cross-domain/dataset and adversarial tests -to assess the robustness of our approach and found that it works exceptionally well.Source code of our experiments is available at: https://github.com/shiquanyang/Robust-Dialogue.
CLSE: Corpus of Linguistically Significant Entities

One of the biggest challenges of natural language generation (NLG) is the proper handling of named entities. Named entities are a common source of grammar mistakes such as wrong prepositions, wrong article handling, or incorrect entity inflection. Without factoring linguistic representation, such errors are often underrepresented when evaluating on a small set of arbitrarily picked argument values, or when translating a dataset from a linguistically simpler language, like English, to a linguistically complex language, like Russian. However, for some applications, broadly precise grammatical correctness is critical – native speakers may find entity-related grammar errors silly, jarring, or even offensive. To enable the creation of more linguistically diverse NLG datasets, we release a Corpus of Linguistically Significant Entities (CLSE) annotated by linguist experts. The corpus includes 34 languages and covers 74 different semantic types to support various applications from airline ticketing to video games. To demonstrate one possible use of CLSE, we produce an augmented version of the Schema-Guided Dialog Dataset, SGD-CLSE. Using the CLSE’s entities and a small number of human translations, we create a linguistically representative NLG evaluation benchmark in three languages: French (high-resource), Marathi (low-resource), and Russian (highly inflected language). We establish quality baselines for neural, template-based, and hybrid NLG systems and discuss the strengths and weaknesses of each approach.
Probing for Understanding of English Verb Classes and Alternations in Large Pre-trained Language Models

We investigate the extent to which verb alternation classes, as described by Levin (1993), are encoded in the embeddings of Large Pretrained Language Models (PLMs) such as BERT, RoBERTa, ELECTRA, and DeBERTa using selectively constructed diagnostic classifiers for word and sentence-level prediction tasks.We follow and expand upon the experiments of Kann et al. ( 2019), which aim to probe whether static embeddings encode frameselectional properties of verbs.At both the word and sentence level, we find that contextual embeddings from PLMs not only outperform non-contextual embeddings, but achieve astonishingly high accuracies on tasks across most alternation classes.Additionally, we find evidence that the middle-to-upper layers of PLMs achieve better performance on average than the lower layers across all probing tasks.
Adversarial Text-to-Speech for low-resource languages

In this paper we propose a new method for training adversarial text-to-speech (TTS) models for low-resource languages using auxiliary data.Specifically, we modify the MelGAN (Kumar et al., 2019) architecture to achieve better performance in Arabic speech generation, exploring multiple additional datasets and architectural choices, which involved extra discriminators designed to exploit high-frequency similarities between languages.In our evaluation, we used subjective human evaluation, MOS -Mean Opinion Score, and a novel quantitative metric, the Fréchet Wav2Vec Distance, which we found to be well correlated with MOS.Both subjectively and quantitatively, our method outperformed the standard MelGAN model.