Reviewerly/exHarmony huggingface dataset:
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation

While often assumed a gold standard, effective human evaluation of text generation remains an important, open area for research.We revisit this problem with a focus on producing consistent evaluations that are reproducible-over time and across different populations.We study this goal in different stages of the human evaluation pipeline.In particular, we consider design choices for the annotation interface used to elicit human judgments and their impact on reproducibility.Furthermore, we develop an automated mechanism for maintaining annotator quality via a probabilistic model that detects and excludes noisy annotators.Putting these lessons together, we introduce GENIE: a system for running standardized human evaluations across different generation tasks.We instantiate GENIE with datasets representing four core challenges in text generation: machine translation, summarization, commonsense reasoning, and machine comprehension.For each task, GENIE offers a leaderboard that automatically crowdsources annotations for submissions, evaluating them along axes such as correctness, conciseness, and fluency.We have made the GE-NIE leaderboards publicly available, and have already ranked 50 submissions from 10 different research groups. 1 We hope GENIE encourages further progress toward effective, standardized evaluations for text generation.
Improving the Generalizability of Text-Based Emotion Detection by Leveraging Transformers with Psycholinguistic Features

In recent years, there has been increased interest in building predictive models that harness natural language processing and machine learning techniques to detect emotions from various text sources, including social media posts, micro-blogs or news articles.Yet, deployment of such models in real-world sentiment and emotion applications faces challenges, in particular poor out-of-domain generalizability.This is likely due to domainspecific differences (e.g., topics, communicative goals, and annotation schemes) that make transfer between different models of emotion recognition difficult.In this work we propose approaches for text-based emotion detection that leverage transformer models (BERT and RoBERTa) in combination with Bidirectional Long Short-Term Memory (BiLSTM) networks trained on a comprehensive set of psycholinguistic features.First, we evaluate the performance of our models within-domain on two benchmark datasets: GoEmotion (Demszky et al., 2020) and ISEAR (Scherer and Wallbott, 1994).Second, we conduct transfer learning experiments on six datasets from the Unified Emotion Dataset (Bostan and Klinger, 2018) to evaluate their out-of-domain robustness.We find that the proposed hybrid models improve the ability to generalize to out-of-distribution data compared to a standard transformer-based approach.Moreover, we observe that these models perform competitively on in-domain data.
Goal-oriented Vision-and-Dialog Navigation via Reinforcement Learning

Vision-and-dialog navigation is a recent benchmark for evaluating the AI capabilities of perception, interaction, and decision making.While existing methods developed for this benchmark have demonstrated great successes, they mostly rely on large datasets, where data collection can be a challenge, and the learned policies are not adaptive to domain changes.In this paper, we focus on a new problem, referred to as goal-oriented vision-and-dialog navigation (GVDN), where an agent uses reinforcement learning techniques to compute dialog-navigation policies from trial and error.A robot conducts visual navigation to locate target objects, and can talk to a remote human operator as needed.Our remote human is able to provide guidance on navigation only if the robot correctly conveys its location through dialog.Experiments have been conducted using photo-realistic simulation environments.Results suggest that, our agent outperforms competitive baselines in success rate.
Dialogue Meaning Representation for Task-Oriented Dialogue Systems

Dialogue meaning representation formulates natural language utterance semantics in their conversational context in an explicit and machine-readable form. Previous work typically follows the intent-slot framework, which is easy for annotation yet limited in scalability for complex linguistic expressions. A line of works alleviates the representation issue by introducing hierarchical structures but challenging to express complex compositional semantics, such as negation and coreference. We propose Dialogue Meaning Representation (DMR), a pliable and easily extendable representation for task-oriented dialogue. Our representation contains a set of nodes and edges to represent rich compositional semantics. Moreover, we propose an inheritance hierarchy mechanism focusing on domain extensibility. Additionally, we annotated DMR-FastFood, a multi-turn dialogue dataset with more than 70k utterances, with DMR. We propose two evaluation tasks to evaluate different dialogue models and a novel coreference resolution model GNNCoref for the graph-based coreference resolution task. Experiments show that DMR can be parsed well with pre-trained Seq2Seq models, and GNNCoref outperforms the baseline models by a large margin.The dataset and code are available at https://github.com/amazon-research/dialogue-meaning-representation
Semantic Similarity as a Window into Vector- and Graph-Based Metrics

In this work, we use sentence similarity as a lens through which to investigate the representation of meaning in graphs vs. vectors.On semantic textual similarity data, we examine how similarity metrics based on vectors alone (SENTENCE-BERT and BERTSCORE) fare compared to metrics based on AMR graphs (SMATCH and S 2 MATCH).Quantitative and qualitative analyses show that the AMR-based metrics can better capture meanings dependent on sentence structures, but can also be distracted by structural differences-whereas the BERT-based metrics represent finer-grained meanings of individual words, but often fail to capture the ordering effect of words within sentences and suffer from interpretability problems.These findings contribute to our understanding of each approach to semantic representation and motivate distinct use cases for graph and vector-based representations.
iCompass Working Notes for the Nuanced Arabic Dialect Identification Shared task

We describe our submitted system to the Nuanced Arabic Dialect Identification (NADI) shared task.We tackled only the first subtask (Subtask 1).We used state-of-the-art Deep Learning models and pre-trained contextualized text representation models that we finetuned according to the downstream task in hand.As a first approach, we used BERT Arabic variants: MARBERT with its two versions MARBERT v1 and MARBERT v2, then we combined MARBERT embeddings with a CNN classifier, and finally, we tested the Quasi-Recurrent Neural Networks (QRNN) model.The results found show that version 2 of MAR-BERT outperforms all of the previously mentioned models on Subtask 1.
FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue

Task transfer, transferring knowledge contained in related tasks, holds the promise of reducing the quantity of labeled data required to finetune language models.Dialogue understanding encompasses many diverse tasks, yet task transfer has not been thoroughly studied in conversational AI.This work explores conversational task transfer by introducing FETA: a benchmark for FEw-sample TAsk transfer in open-domain dialogue.FETA contains two underlying sets of conversations upon which there are 10 and 7 tasks annotated, enabling the study of intra-dataset task transfer; task transfer without domain adaptation.We utilize three popular language models and three learning algorithms to analyze the transferability between 132 source-target task pairs and create a baseline for future work.We run experiments in the single-and multi-source settings and report valuable findings, e.g., most performance trends are model-specific, and span extraction and multiple-choice tasks benefit the most from task transfer.In addition to task transfer, FETA can be a valuable resource for future research into the efficiency and generalizability of pre-training datasets and model architectures, as well as for learning settings such as continual and multitask learning.
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization

Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowledge from a larger model (a teacher).Although KD methods achieve state-of-the-art performance in numerous settings, they suffer from several problems limiting their performance.It is shown in the literature that the capacity gap between the teacher and the student networks can make KD ineffective.Additionally, existing KD techniques do not mitigate the noise in the teacher's output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features.We propose a new KD method that addresses these problems and facilitates the training compared to previous techniques.Inspired by continuation optimization, we design a training procedure that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds.Our method (Continuation-KD) achieves stateof-the-art performance across various compact architectures on NLU (GLUE benchmark) and computer vision tasks (CIFAR-10 and CIFAR-100).
Tutoring Helps Students Learn Better: Improving Knowledge Distillation for BERT with Tutor Network

Pre-trained language models have achieved remarkable successes in natural language processing tasks, coming at the cost of increasing model size.To address this issue, knowledge distillation (KD) has been widely applied to compress language models.However, typical KD approaches for language models have overlooked the difficulty of training examples, suffering from incorrect teacher prediction transfer and sub-efficient training.In this paper, we propose a novel KD framework, Tutor-KD, which improves the distillation effectiveness by controlling the difficulty of training examples during pre-training.We introduce a tutor network that generates samples that are easy for the teacher but difficult for the student, with training on a carefully designed policy gradient method.Experimental results show that Tutor-KD significantly and consistently outperforms the state-of-the-art KD methods with variously sized student models on the GLUE benchmark, demonstrating that the tutor can effectively generate training examples for the student 1 .
Using Developer Discussions to Guide Fixing Bugs in Software

Automatically fixing software bugs is a challenging task.While recent work showed that natural language context is useful in guiding bug-fixing models, the approach required prompting developers to provide this context, which was simulated through commit messages written after the bug-fixing code changes were made.We instead propose using bug report discussions, which are available before the task is performed and are also naturally occurring, avoiding the need for any additional information from developers.For this, we augment standard bug-fixing datasets with bug report discussions.Using these newly compiled datasets, we demonstrate that various forms of natural language context derived from such discussions can aid bug-fixing, even leading to improved performance over using commit messages corresponding to the oracle bug-fixing commits.
Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation

In NMT we search for the mode of the model distribution to form predictions.The mode and other high-probability translations found by beam search have been shown to often be inadequate in a number of ways.This prevents improving translation quality through better search, as these idiosyncratic translations end up selected by the decoding algorithm, a problem known as the beam search curse.Recently, an approximation to minimum Bayes risk (MBR) decoding has been proposed as an alternative decision rule that would likely not suffer from the same problems.We analyse this approximation and establish that it has no equivalent to the beam search curse.We then design approximations that decouple the cost of exploration from the cost of robust estimation of expected utility.This allows for much larger hypothesis spaces, which we show to be beneficial.We also show that mode-seeking strategies can aid in constructing compact sets of promising hypotheses and that MBR is effective in identifying good translations in them.We conduct experiments on three language pairs varying in amounts of resources available: English into and from German, Romanian, and Nepali. 1
Prompt-based Distribution Alignment for Domain Generalization in Text Classification

Prompt-based learning (a.k.a.prompting) achieves high performance by bridging the gap between the objectives of language modeling and downstream tasks.Domain generalization ability can be improved by prompting since classification across different domains can be unified into the prediction of the same set of label words.The remaining challenge for domain generalization by prompting comes from discrepancies between the data distribution of different domains.To improve domain generalization with prompting, we learn distributional invariance across source domains via two alignment regularization loss functions.The first is vocabulary distribution alignment, which uses a Kullback-Leibler divergence regularization on source-domain vocabulary distributions.The second is feature distribution alignment, which uses a novel adversarial training strategy to learn domain invariant representation across source domains.Experiments on sentiment analysis and natural language inference show the effectiveness of our method and achieve state-of-the-art results on six datasets.
RLET: A Reinforcement Learning Based Approach for Explainable QA with Entailment Trees

Interpreting the reasoning process from questions to answers poses a challenge in approaching explainable QA.A recently proposed structured reasoning format, entailment tree, manages to offer explicit logical deductions with entailment steps in a tree structure.To generate entailment trees, prior single pass sequence-tosequence models lack visible internal decision probability, while stepwise approaches are supervised with extracted single step data and cannot model the tree as a whole.In this work, we propose RLET, a Reinforcement Learning based Entailment Tree generation framework, which is trained utilising the cumulative signals across the whole tree.RLET iteratively performs single step reasoning with sentence selection and deduction generation modules, from which the training signal is accumulated across the tree with elaborately designed aligned reward function that is consistent with the evaluation.To the best of our knowledge, we are the first to introduce RL into the entailment tree generation task.Experiments on three settings of the EntailmentBank dataset demonstrate the strength of using RL framework.
Time-aware Prompting for Text Generation

In this paper, we study the effects of incorporating timestamps, such as document creation dates, into generation systems.Two types of time-aware prompts are investigated:(1) textual prompts that encode document timestamps in natural language sentences; and(2) linear prompts that convert timestamps into continuous vectors.To explore extrapolation to future data points, we further introduce a new data-to-text generation dataset, TEMPWIKIBIO, containing more than 4 millions of chronologically ordered revisions of biographical articles from English Wikipedia, each paired with structured personal profiles.Through data-to-text generation on TEMPWIK-IBIO, text-to-text generation on the content transfer dataset, and summarization on XSum, we show that linear prompts on encoder and textual prompts improve the generation quality on all datasets.Despite having less performance drop when testing on data drawn from a later time, linear prompts focus more on nontemporal information and are less sensitive to the given timestamps, according to human evaluations and sensitivity analyses.Meanwhile, textual prompts establish the association between the given timestamps and the output dates, yielding more factual temporal information in the output.
Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation

In this paper, we propose a variational autoencoder with disentanglement priors, VAE-DPRIOR, for task-specific natural language generation with none or a handful of taskspecific labeled examples.In order to tackle compositional generalization across tasks, our model performs disentangled representation learning by introducing a conditional prior for the latent content space and another conditional prior for the latent label space.Both types of priors satisfy a novel property called ϵ-disentangled.We show both empirically and theoretically that the novel priors can disentangle representations even without specific regularizations as in the prior work.The content prior enables directly sampling diverse content representations from the content space learned from the seen tasks, and fuse them with the representations of novel tasks for generating semantically diverse texts in the low-resource settings.Our extensive experiments demonstrate the superior performance of our model over competitive baselines in terms of i) data augmentation in continuous zero/few-shot learning, and ii) text style transfer in the few-shot setting.
Where’s the Learning in Representation Learning for Compositional Semantics and the Case of Thematic Fit

Observing that for certain NLP tasks, such as semantic role prediction or thematic fit estimation, random embeddings perform as well as pretrained embeddings, we explore what settings allow for this and examine where most of the learning is encoded: the word embeddings, the semantic role embeddings, or "the network".We find nuanced answers, depending on the task and its relation to the training objective.We examine these representation learning aspects in multi-task learning, where role prediction and role-filling are supervised tasks, while several thematic fit tasks are outside the models' direct supervision.We observe a non-monotonous relation between some tasks' quality score and the training data size.In order to better understand this observation, we analyze these results using easier, per-verb versions of these tasks.
Multimodal Conversation Modelling for Topic Derailment Detection

Conversations on social media tend to go offtopic and turn into different and sometimes toxic exchanges.Previous work focuses on analysing textual dialogues that have derailed into toxic content, but the range of derailment types is much broader, including spam or bot content, tangential comments, etc.In addition, existing work disregards conversations that involve visual information (i.e.images or videos), which are prevalent on most platforms.In this paper, we take a broader view of conversation derailment and propose a new challenge: detecting derailment based on the "change of conversation topic", where the topic is defined by an initial post containing both a text and an image.For that, we (i) create the first Multimodal Conversation Derailment (MCD) dataset, 1 and (ii) introduce a new multimodal conversational architecture (MMConv) that utilises visual and conversational contexts to classify comments for derailment.Experiments show that MM-Conv substantially outperforms previous textbased approaches to detect conversation derailment, as well as general multimodal classifiers.MMConv is also more robust to textual noise, since it relies on richer contextual information.1 https://github.com/Nickeilf/Multimodal-Conversation-Derailment I love our pediatrician's shirt today If I was a pediatrician I would purposely tweak anti-vaxxers.Whatever I could do to get them to come into the light so I could fire them as patients.This is exactly one of the reasons why she wears it.I complimented the shirt, asked if I could take a photo, and then she told me about how many people take issue with the shirt.This office immediately fires patients if they find out they're not up to date on vaccines.Is that actually the correct term?"Fire patients"?I know it but I just want to know if that's actually what you would say.
Argument Mining for Review Helpfulness Prediction

The importance of reliably determining the helpfulness of product reviews is rising as both helpful and unhelpful reviews continue to accumulate on e-commerce websites.And argumentational features-such as the structure of arguments and the types of underlying elementary units-have shown to be promising indicators of product review helpfulness.However, their adoption has been limited due to the lack of sufficient resources and large-scale experiments investigating their utility.To this end, we present the AMazon Argument Mining (AM 2 ) corpus-a corpus of 878 Amazon reviews on headphones annotated according to a theoretical argumentation model designed to evaluate argument quality.Experiments show that employing argumentational features leads to statistically significant improvements over the state-of-the-art review helpfulness predictors under both text-only and text-and-image settings. 1
You Truly Understand What I Need : Intellectual and Friendly Dialog Agents grounding Persona and Knowledge

To build a conversational agent that interacts fluently with humans, previous studies blend knowledge or personal profile into the pre-trained language model.However, the model that considers knowledge and persona at the same time is still limited, leading to hallucination and a passive way of using personas.We propose an effective dialogue agent that grounds external knowledge and persona simultaneously.The agent selects the proper knowledge and persona to use for generating the answers with our candidate scoring implemented with a poly-encoder.Then, our model generates the utterance with lesser hallucination and more engagingness utilizing retrieval augmented generation with knowledge-persona enhanced query.We conduct experiments on the personaknowledge chat and achieve state-of-the-art performance in grounding and generation tasks on the automatic metrics.Moreover, we validate the answers from the models regarding hallucination and engagingness through human evaluation and qualitative results.We show our retriever's effectiveness in extracting relevant documents compared to the other previous retrievers, along with the comparison of multiple candidate scoring methods.
Do Language Models Understand Measurements?

Recent success of pre-trained language models (PLMs) has stimulated interest in their ability to understand and work with numbers. Yet, the numerical reasoning over measurements has not been formally studied despite their importance. In this study, we show that PLMs lack the capability required for reasoning over measurements. Furthermore, we find that a language model trained on a measurement-rich corpus shows better performance on understanding measurements. We propose a simple embedding strategy to better distinguish between numbers and units, which leads to a significant improvement in the probing tasks.