Reviewerly/exHarmony huggingface dataset:
PreQuEL: Quality Estimation of Machine Translation Outputs in Advance

We present the task of PreQuEL, Pre-(Quality-Estimation) Learning.A PreQuEL system predicts how well a given sentence will be translated, without recourse to the actual translation, thus eschewing unnecessary resource allocation when translation quality is bound to be low.PreQuEL can be defined relative to a given MT system (e.g., some industry service) or generally relative to the state-of-theart.From a theoretical perspective, PreQuEL places the focus on the source text, tracing properties, possibly linguistic features, that make a sentence harder to machine translate.We develop a baseline model for the task and analyze its performance.We also develop a data augmentation method (from parallel corpora), that improves results substantially.We show that this augmentation method can improve the performance of the Quality-Estimation task as well. 1 We investigate the properties of the input text that our model is sensitive to, by testing it on challenge sets and different languages.We conclude that it is aware of syntactic and semantic distinctions, and correlates and even over-emphasizes the importance of standard NLP features.
Unsupervised Learning of Hierarchical Conversation Structure

Human conversations can evolve in many different ways, creating challenges for automatic understanding and summarization.Goaloriented conversations often have meaningful sub-dialogue structure, but it can be highly domain-dependent.This work introduces an unsupervised approach to learning hierarchical conversation structure, including turn and sub-dialogue segment labels, corresponding roughly to dialogue acts and sub-tasks, respectively.The decoded structure is shown to be useful in enhancing neural models of language for three conversation-level understanding tasks.Further, the learned finite-state subdialogue network is made interpretable through automatic summarization.
Towards In-Context Non-Expert Evaluation of Reflection Generation for Counselling Conversations

Reflection is an essential counselling strategy, where the therapist listens actively and responds with their own interpretation of the client's words.Recent work leveraged pretrained language models (PLMs) to approach reflection generation as a promising tool to aid counsellor training.However, those studies used limited dialogue context for modelling and simplistic error analysis for human evaluation.In this work, we take the first step towards addressing those limitations.First, we fine-tune PLMs on longer dialogue contexts for reflection generation.Then, we collect free-text error descriptions from non-experts about generated reflections, identify common patterns among them, and accordingly establish discrete error categories using thematic analysis.Based on this scheme, we plan for future work a mass non-expert error annotation phase for generated reflections followed by an expert-based validation phase, namely "whether a coherent and consistent response is a good reflection".
Zero-Shot Cross-Lingual Sequence Tagging as Seq2Seq Generation for Joint Intent Classification and Slot Filling

The joint intent classification and slot filling task seeks to detect the intent of an utterance and extract its semantic concepts. In the zero-shot cross-lingual setting, a model is trained on a source language and then transferred to other target languages through multi-lingual representations without additional training data. While prior studies show that pre-trained multilingual sequence-to-sequence (Seq2Seq) models can facilitate zero-shot transfer, there is little understanding on how to design the output template for the joint prediction tasks. In this paper, we examine three aspects of the output template – (1) label mapping, (2) task dependency, and (3) word order. Experiments on the MASSIVE dataset consisting of 51 languages show that our output template significantly improves the performance of pre-trained cross-lingual language models.
Hyperdecoders: Instance-specific decoders for multi-task NLP

We investigate input-conditioned hypernetworks for multi-tasking in NLP, generating parameter-efficient adaptations for a decoder using a hypernetwork conditioned on the output of an encoder.This approach produces a unique decoder adaptation for every input instance, allowing the network a larger degree of flexibility than prior work that only produces one decoder adaptation per task.We apply our method to sequence classification tasks, extractive QA, and summarisation and find that it surpasses previous parameter efficient finetuning methods and often outperforms fully finetuning the underlying model.An analysis of the embeddings used by our hypernetwork shows that they are sensitive to output label and type, suggesting that our approach better maps from encoder representations to output labels.
Improving Faithfulness by Augmenting Negative Summaries from Fake Documents

Current abstractive summarization systems tend to hallucinate content that is unfaithful to the source document, posing a risk of misinformation.To mitigate hallucination, we must teach the model to distinguish hallucinated summaries from faithful ones.However, the commonly used maximum likelihood training does not disentangle factual errors from other model errors.To address this issue, we propose a back-translation-style approach to augment negative samples that mimic factual errors made by the model.Specifically, we train an elaboration model that generates hallucinated documents given the reference summaries, and then generates negative summaries from the fake documents.We incorporate the negative samples into training through a controlled generator, which produces faithful/unfaithful summaries conditioned on the control codes.Additionally, we find that adding textual entailment data through multitasking further boosts the performance.Experiments on three datasets (XSum, GigaWord, and WikiHow) show that our method consistently improves faithfulness without sacrificing informativeness according to both human and automatic evaluation. 1
Robustifying Sentiment Classification by Maximally Exploiting Few Counterfactuals

For text classification tasks, finetuned language models perform remarkably well.Yet, they tend to rely on spurious patterns in training data, thus limiting their performance on outof-distribution (OOD) test data.Among recent models aiming to avoid this spurious pattern problem, adding extra counterfactual samples to the training data has proven to be very effective.Yet, counterfactual data generation is costly since it relies on human annotation.Thus, we propose a novel solution that only requires annotation of a small fraction (e.g., 1%) of the original training data, and uses automatic generation of extra counterfactuals in an encoding vector space.We demonstrate the effectiveness of our approach in sentiment classification, using IMDb data for training and other sets for OOD tests (i.e., Amazon, SemEval and Yelp).We achieve noticeable accuracy improvements by adding only 1% manual counterfactuals: +3% compared to adding +100% in-distribution training samples, +1.3% compared to alternate counterfactual approaches. Original Sample (x)Counterfactually Revised Sample (x CAD ) NEGATIVE → POSITIVE one of the worst ever scenes in a sports movie.3 stars out of 10. one of the wildest ever scenes in a sports movie.8 stars out of 10.POSITIVE → NEGATIVE The world of Atlantis, hidden beneath the earth's core, is fantastic.The world of Atlantis, hidden beneath the earth's core is supposed to be fantastic.
AdapterShare: Task Correlation Modeling with Adapter Differentiation

Thanks to the development of pre-trained language models, multitask learning (MTL) methods have achieved great success in natural language understanding.However, current MTL methods pay more attention to task selection or model design to fuse as much knowledge as possible, while the intrinsic task correlation is often neglected.It is important to learn sharing strategies among multiple tasks rather than sharing everything.In this paper, we propose AdapterShare, an adapter differentiation method to explicitly model task correlation among multiple tasks.AdapterShare is automatically learned based on the gradients on tiny held-out validation data.Compared to single-task learning and fully shared MTL methods, our proposed method obtains obvious performance improvements.Compared to the existing MTL method AdapterFusion, AdapterShare achieves an absolute average improvement of 1.90 points on five dialogue understanding tasks and 2.33 points on NLU tasks.Our implementation is available at https:// github.com/microsoft/ContextualSP.
Towards Robust NLG Bias Evaluation with Syntactically-diverse Prompts

We present a robust methodology for evaluating biases in natural language generation(NLG) systems.Previous works use fixed hand-crafted prefix templates with mentions of various demographic groups to prompt models to generate continuations for bias analysis.These fixed prefix templates could themselves be specific in terms of styles or linguistic structures, which may lead to unreliable fairness conclusions that are not representative of the general trends from tone varying prompts.To study this problem, we paraphrase the prompts with different syntactic structures and use these to evaluate demographic bias in NLG systems.Our results suggest similar overall bias trends but some syntactic structures lead to contradictory conclusions compared to past works.We show that our methodology is more robust and that some syntactic structures prompt more toxic content while others could prompt less biased generation.This suggests the importance of not relying on a fixed syntactic structure and using tone-invariant prompts.Introducing syntactically-diverse prompts can achieve more robust NLG (bias) evaluation.
Should We Ban English NLP for a Year?

Around two thirds of NLP research at top venues is devoted exclusively to developing technology for speakers of English, most speech data comes from young urban speakers, and most texts used to train language models come from male writers.These biases feed into consumer technologies to widen existing inequality gaps, not only within, but also across, societies.Many have argued that it is almost impossible to mitigate inequality amplification.I argue that, on the contrary, it is quite simple to do so, and that counter-measures would have little-to-no negative impact, except for, perhaps, in the very short term.1 See Bender ( 2009), Ruder et al. ( 2022), as well as https: //sjmielke.com/acl-language-diversity.htm 2 What explains the dominance of English in NLP research?Prestige seems to be an important factor.What is considered state of the art, is what achieves best performance on English.Consider, for example, the ACL Wiki's list of 'state-of-the-art' part-of-speech taggers
Learning to Infer from Unlabeled Data: A Semi-supervised Learning Approach for Robust Natural Language Inference

Natural Language Inference (NLI) or Recognizing Textual Entailment (RTE) aims at predicting the relation between a pair of sentences (premise and hypothesis) as entailment, contradiction or semantic independence.Although deep learning models have shown promising performance for NLI in recent years, they rely on large scale expensive human-annotated datasets.Semi-supervised learning (SSL) is a popular technique for reducing the reliance on human annotation by leveraging unlabeled data for training.However, despite its substantial success on single sentence classification tasks where the challenge in making use of unlabeled data is to assign "good enough" pseudo-labels, for NLI tasks, the nature of unlabeled data is more complex: one of the sentences in the pair (usually the hypothesis) along with the class label are missing from the data and require human annotations, which makes SSL for NLI more challenging.In this paper, we propose a novel way to incorporate unlabeled data in SSL for NLI where we use a conditional language model, BART to generate the hypotheses for the unlabeled sentences (used as premises).Our experiments show that our SSL framework successfully exploits unlabeled data and substantially improves the performance of four NLI datasets in low-resource settings.We release our code at:
Model and Data Transfer for Cross-Lingual Sequence Labelling in Zero-Resource Settings

Zero-resource cross-lingual transfer approaches aim to apply supervised models from a source language to unlabelled target languages.In this paper we perform an in-depth study of the two main techniques employed so far for cross-lingual zero-resource sequence labelling, based either on data or model transfer.Although previous research has proposed translation and annotation projection (data-based cross-lingual transfer) as an effective technique for cross-lingual sequence labelling, in this paper we experimentally demonstrate that high capacity multilingual language models applied in a zero-shot (model-based cross-lingual transfer) setting consistently outperform data-based cross-lingual transfer approaches.A detailed analysis of our results suggests that this might be due to important differences in language use.More specifically, machine translation often generates a textual signal which is different to what the models are exposed to when using gold standard data, which affects both the fine-tuning and evaluation processes.Our results also indicate that data-based cross-lingual transfer approaches remain a competitive option when high-capacity multilingual language models are not available.
SBU Figures It Out: Models Explain Figurative Language

Figurative language is ubiquitous in human communication.However, current NLP models are unable to demonstrate a significant understanding of instances of this phenomena.FigLang shared task on figurative language understanding posed the problem of predicting and explaining the relation between a premise and a hypothesis containing an instance of the use of figurative language.We experiment with different variations of using T5-large for this task and build a model that significantly outperforms the task baseline.Treating it as a new task for T5 and simply finetuning on the data achieves the best score on the defined evaluation.Furthermore, we find that hypothesis-only models are able to achieve most of the performance.
Audience-Centric Natural Language Generation via Style Infusion

Adopting contextually appropriate, audiencetailored linguistic styles is critical to the success of user-centric language generation systems (e.g., chatbots, computer-aided writing, dialog systems).While existing approaches demonstrate textual style transfer with large volumes of parallel or non-parallel data, we argue that grounding style on audience-independent external factors is innately limiting for two reasons.First, it is difficult to collect large volumes of audience-specific stylistic data.Second, some stylistic objectives (e.g., persuasiveness, memorability, empathy) are hard to define without audience feedback.In this paper, we propose the novel task of style infusion -infusing the stylistic preferences of audiences in pretrained language generation models.Since humans are better at pairwise comparisons than direct scoring -i.e., is Sample-A more persuasive/polite/empathic than Sample-B -we leverage limited pairwise human judgments to bootstrap a style analysis model and augment our seed set of judgments.We then infuse the learned textual style in a GPT-2 based text generator while balancing fluency and style adoption.With quantitative and qualitative assessments, we show that our infusion approach can generate compelling stylized examples with generic text prompts.The code and data are accessible at https://github.com/CrowdDynamicsLab/StyleInfusion.
Text Editing as Imitation Game

Text editing, such as grammatical error correction, arises naturally from imperfect textual data.Recent works frame text editing as a multi-round sequence tagging task, where operations -such as insertion and substitutionare represented as a sequence of tags.While achieving good results, this encoding is limited in flexibility as all actions are bound to token-level tags.In this work, we reformulate text editing as an imitation game using behavioral cloning.Specifically, we convert conventional sequence-to-sequence data into state-toaction demonstrations, where the action space can be as flexible as needed.Instead of generating the actions one at a time, we introduce a dual decoders structure to parallel the decoding while retaining the dependencies between action tokens, coupled with trajectory augmentation to alleviate the distribution shift that imitation learning often suffers.In experiments on a suite of Arithmetic Equation benchmarks, our model consistently outperforms the autoregressive baselines in terms of performance, efficiency, and robustness.We hope our findings will shed light on future studies in reinforcement learning applying sequence-level action generation to natural language processing.
PLM-based World Models for Text-based Games

World models have improved the ability of reinforcement learning agents to operate in a sample efficient manner, by being trained to predict plausible changes in the underlying environment.As the core tasks of world models are future prediction and commonsense understanding, our claim is that pre-trained language models (PLMs) already provide a strong base upon which to build world models.Worldformer is a recently proposed world model for text-based game environments, based only partially on PLM and transformers.Our distinction is to fully leverage PLMs as actionable world models in text-based game environments, by reformulating generation as constrained decoding which decomposes actions into verb templates and objects.We show that our model improves future valid action prediction and graph change prediction. 1 Additionally, we show that our model better reflects commonsense than standard PLM.
Boosting Natural Language Generation from Instructions with Meta-Learning

Recent work has shown that language models (LMs) trained with multi-task instructional learning (MTIL) can solve diverse NLP tasks in zero-and few-shot settings with improved performance compared to prompt tuning.MTIL illustrates that LMs can extract and use information about the task from instructions beyond the surface patterns of the inputs and outputs.This suggests that meta-learning may further enhance the utilization of instructions for effective task transfer.In this paper we investigate whether meta-learning applied to MTIL can further improve generalization to unseen tasks in a zero-shot setting.Specifically, we propose to adapt meta-learning to MTIL in three directions: 1) Model Agnostic Meta Learning (MAML), 2) Hyper-Network (HNet) based adaptation to generate task specific parameters conditioned on instructions, and 3) an approach combining HNet and MAML.Through extensive experiments on the large scale Natural Instructions V2 dataset, we show that our proposed approaches significantly improve over strong baselines in zero-shot settings.In particular, meta-learning improves the effectiveness of instructions and is most impactful when the test tasks are strictly zero-shot (i.e.no similar tasks in the training set) and are "hard" for LMs, illustrating the potential of meta-learning for MTIL for out-of-distribution tasks.
HSE at TempoWiC: Detecting Meaning Shift in Social Media with Diachronic Language Models

This paper describes our methods for temporal meaning shift detection, implemented during the TempoWiC shared task.We present two systems: with and without time span data usage.Our approach is based on masked language models continuously pre-trained with Twitter data.Both systems outperformed all the competition's baselines except TimeLMs-SIM.Our best submission achieved the macro-F1 score of 70.09% and took the 7th place.This result was achieved by using diachronic language models from the TimeLMs project.
Improved grammatical error correction by ranking elementary edits

We offer a two-stage reranking method for grammatical error correction: the first model serves as edit generator, while the second classifies the proposed edits as correct or false.We show how to use both encoder-decoder and sequence labeling models for the first step of our pipeline.We achieve state-of-the-art quality on BEA 2019 English dataset even using weak BERT-GEC edit generator.Combining our roberta-base scorer with state-of-the-art GEC-ToR edit generator, we surpass GECToR by 2 -3%.With a larger model we establish a new SOTA on BEA development and test sets.Our model also sets a new SOTA on Russian, despite using smaller models and less data than the previous approaches.
Few-Shot Cross-Lingual Learning for Event Detection

Cross-Lingual Event Detection (CLED) models are capable of performing the Event Detection (ED) task in multiple languages.Such models are trained using data from a source language and then evaluated on data from a distinct target language.Training is usually performed in the standard supervised setting with labeled data available in the source language.The Few-Shot Learning (FSL) paradigm is yet to be explored for CLED despite its inherent advantage of allowing models to better generalize to unseen event types.As such, in this work we study the CLED task under an FSL setting.Our contribution is threefold: first, we introduce a novel FSL classification method based on Optimal Transport (OT); second, we present a novel regularization term to incorporate the global distance between the support and query sets; and third, we adapt our approach to the cross-lingual setting by exploiting the alignment between source and target data.Our experiments on three, syntactically-different, target languages show the applicability of our approach and its effectiveness at improving the cross-lingual performance of few-shot models for event detection.
HeLo: Learning-Free Lookahead Decoding for Conversation Infilling

We propose Heuristic Guided Lookahead Decoding (HeLo), a novel decoding strategy for conversation infilling.Conversation infilling aims to generate a seamless bridge of utterances connecting a given pair of source and target utterances.HeLo does not require fine-tuning or extra models -only the generating model itself.Instead, HeLo leverages a greedy lookahead phase before committing to any token.The HeLo framework is simple and can augment conventional decoding strategies paired with any autoregressive language model.Smooth transitions between utterances are encouraged with an annealing schedule.Our experiments show HeLo outperforms several baselines when evaluated with both automatic and human evaluation metrics, which, we argue, are appropriate for the task. 1
ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcripts

Despite tremendous progress in automatic summarization, state-of-the-art methods are predominantly trained to excel in summarizing short newswire articles, or documents with strong layout biases such as scientific articles or government reports.Efficient techniques to summarize financial documents, discussing facts and figures, have largely been unexplored, majorly due to the unavailability of suitable datasets.In this work, we present ECTSum, a new dataset with transcripts of earnings calls (ECTs), hosted by publicly traded companies, as documents, and experts-written short telegram-style bullet point summaries derived from corresponding Reuters articles.ECTs are long unstructured documents without any prescribed length limit or format.We benchmark our dataset with state-of-the-art summarization methods across various metrics evaluating the content quality and factual consistency of the generated summaries.Finally, we present a simple yet effective approach, ECT-BPS, to generate a set of bullet points that precisely capture the important facts discussed in the calls.