Reviewerly/exHarmony huggingface dataset:
Human Ratings Do Not Reflect Downstream Utility: A Study of Free-Text Explanations for Model Predictions

Models able to generate free-text rationales that explain their output have been proposed as an important step towards interpretable NLP for "reasoning" tasks such as natural language inference and commonsense question answering.However, the relative merits of different architectures and types of rationales are not well understood and hard to measure.In this paper, we contribute two insights to this line of research: First, we find that models trained on gold explanations learn to rely on these but, in the case of the more challenging question answering data set we use, fail when given generated explanations at test time.However, additional fine-tuning on generated explanations teaches the model to distinguish between reliable and unreliable information in explanations.Second, we compare explanations by a generation-only model to those generated by a self-rationalizing model and find that, while the former score higher in terms of validity, factual correctness, and similarity to gold explanations, they are not more useful for downstream classification.We observe that the self-rationalizing model is prone to hallucination, which is punished by most metrics but may add useful context for the classification step.
Multitask Instruction-based Prompting for Fallacy Recognition

Fallacies are used as seemingly valid arguments to support a position and persuade the audience about its validity.Recognizing fallacies is an intrinsically difficult task both for humans and machines.Moreover, a big challenge for computational models lies in the fact that fallacies are formulated differently across the datasets with differences in the input format (e.g., question-answer pair, sentence with fallacy fragment), genre (e.g., social media, dialogue, news), as well as types and number of fallacies (from 5 to 18 types per dataset).To move towards solving the fallacy recognition task, we approach these differences across datasets as multiple tasks and show how instruction-based prompting in a multitask setup based on the T5 model improves the results against approaches built for a specific dataset such as T5, BERT or GPT-3.We show the ability of this multitask prompting approach to recognize 28 unique fallacies across domains and genres and study the effect of model size and prompt choice by analyzing the per-class (i.e., fallacy type) results.Finally, we analyze the effect of annotation quality on model performance, and the feasibility of complementing this approach with external knowledge.
TEDB System Description to a Shared Task on Euphemism Detection 2022

In this report, we describe our Transformers for euphemism detection baseline (TEDB) submissions to a shared task on euphemism detection 2022. We cast the task of predicting euphemism as text classification. We considered Transformer-based models which are the current state-of-the-art methods for text classification. We explored different training schemes, pretrained models, and model architectures. Our best result of 0.816 F1-score (0.818 precision and 0.814 recall) consists of a euphemism-detection-finetuned TweetEval/TimeLMs-pretrained RoBERTa model as a feature extractor frontend with a KimCNN classifier backend trained end-to-end using a cosine annealing scheduler. We observed pretrained models on sentiment analysis and offensiveness detection to correlate with more F1-score while pretraining on other tasks, such as sarcasm detection, produces less F1-scores. Also, putting more word vector channels does not improve the performance in our experiments.
Code Vulnerability Detection via Nearest Neighbor Mechanism

Code vulnerability detection is a fundamental and challenging task in the software security field.Existing research works aim to learn semantic information from the source code by utilizing NLP technologies.However, in vulnerability detection tasks, some vulnerable samples are very similar to non-vulnerable samples, which are difficult to identify.To address this issue and improve detection performance, we introduce the k-nearest neighbor mechanism which retrieves multiple neighbor samples and utilizes label information of retrieved neighbor samples to provide help for model predictions.Besides, we use supervised contrastive learning to make the model learn the discriminative representation and ensure that label information of retrieved neighbor samples is as consistent as possible with the label information of testing samples.Extensive experiments show that our method can achieve obvious performance improvements compared to baseline models.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling

Abstractive summarization models often generate inconsistent summaries containing factual errors or hallucinated content.Recent works focus on correcting factual errors in generated summaries via post-editing.Such correction models are trained using adversarial nonfactual summaries constructed using heuristic rules for injecting errors.However, generating non-factual summaries using heuristics often does not generalize well to actual model errors.In this work, we propose to generate hard, representative synthetic examples of nonfactual summaries through infilling language models.With this data, we train a more robust fact-correction model to post-edit the summaries to improve factual consistency.Through quantitative and qualitative experiments on two popular summarization datasets-CNN/DM and XSum-we show that our approach vastly outperforms prior methods in correcting erroneous summaries.Our model-FACTEDITimproves factuality scores by over ∼11 points on CNN/DM and over ∼31 points on XSum on average across multiple summarization models, producing more factual summaries while maintaining competitive summarization quality. 1The first vaccine for Ebola was approved by the FDA in 2019 in the US, five years after the initial outbreak in 2014.To produce the vaccine, scientists had to sequence the DNA of Ebola, then identify possible vaccines, and finally show successful clinical trials.Scientists say a vaccine for COVID-19 is unlikely to be ready this year, although clinical trials have already started. Scientists believe a vaccine for Covid-19might not be ready this year.The first vaccine for Ebola took 5 years to be approved by the FDA.Scientists believe a vaccine for Ebola might not be ready this year.The first vaccine for Ebola took 5 years to be produced by the CBP.
Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters

Natural Language Inference (NLI) has been extensively studied by the NLP community as a framework for estimating the semantic relation between sentence pairs.While early work identified certain biases in NLI models, recent advancements in modeling and datasets demonstrated promising performance.In this work, we further explore the direct zero-shot applicability of NLI models to real applications, beyond the sentence-pair setting they were trained on.First, we analyze the robustness of these models to longer and outof-domain inputs.Then, we develop new aggregation methods to allow operating over full documents, reaching state-of-the-art performance on the ContractNLI dataset.Interestingly, we find NLI scores to provide strong retrieval signals, leading to more relevant evidence extractions compared to common similarity-based methods.Finally, we go further and investigate whole document clusters to identify both discrepancies and consensus among sources.In a test case, we find real inconsistencies between Wikipedia pages in different languages about the same topic. 1
ParaTag: A Dataset of Paraphrase Tagging for Fine-Grained Labels, NLG Evaluation, and Data Augmentation

Paraphrase identification has been formulated as a binary classification task to decide whether two sentences hold a paraphrase relationship.Existing paraphrase datasets only annotate a binary label for each sentence pair.However, after a systematical analysis of existing paraphrase datasets, we found that the degree of paraphrase cannot be well characterized by a single binary label.And the criteria of paraphrase are not even consistent within the same dataset.We hypothesize that such issues would limit the effectiveness of paraphrase models trained on these data.To this end, we propose a novel fine-grained paraphrase annotation schema that labels the minimum spans of tokens in a sentence that don't have the corresponding paraphrases in the other sentence.Under this setting, we frame paraphrasing as a sequence tagging task.We collect 30k sentence pairs in English with the new annotation schema, resulting in the ParaTag dataset.In addition to reporting baseline results on ParaTag using state-of-art language models, we show that ParaTag is especially useful for training an automatic scorer for language generation evaluation.Finally, we train a paraphrase generation model from ParaTag and achieve better data augmentation performance on the GLUE benchmark than other public paraphrasing datasets. 1* Equal contribution. 1 Data and code: https://github.com/microsoft/ParaTagM-1 However, commercial use of the 2.6.0 kernel is still months off for most customers.M-2 Commercial releases of the 2.6 kernel by major Linux distributors still remain months away.
Mitigating Spurious Correlation in Natural Language Understanding with Counterfactual Inference

Despite their promising results on standard benchmarks, NLU models are still prone to make predictions based on shortcuts caused by unintended bias in the dataset.For example, an NLI model may use lexical overlap as a shortcut to make entailment predictions due to repetitive data generation patterns from annotators, also called annotation artifacts.In this paper, we propose a causal analysis framework to help debias NLU models.We show that (1) by defining causal relationships, we can introspect how much annotation artifacts affect the outcomes.(2) We can utilize counterfactual inference to mitigate bias with this knowledge.We found that viewing a model as a treatment can mitigate bias more effectively than viewing annotation artifacts as treatment.(3) In addition to bias mitigation, we can interpret how much each debiasing strategy is affected by annotation artifacts.Our experimental results show that using counterfactual inference can improve out-of-distribution performance in all settings while maintaining high in-distribution performance. 1
Knowledge Transfer from Answer Ranking to Answer Generation

Recent studies show that Question Answering (QA) based on Answer Sentence Selection (AS2) can be improved by generating an improved answer from the top-k ranked answer sentences (termed GenQA).This allows for synthesizing the information from multiple candidates into a concise, natural-sounding answer.However, creating large-scale supervised training data for GenQA models is very challenging.In this paper, we propose to train a GenQA model by transferring knowledge from a trained AS2 model, to overcome the aforementioned issue.First, we use an AS2 model to produce a ranking over answer candidates for a set of questions.Then, we use the top ranked candidate as the generation target, and the next k top ranked candidates as context for training a GenQA model.We also propose to use the AS2 model prediction scores for loss weighting and score-conditioned input/output shaping, to aid the knowledge transfer.Our evaluation on three public and one large industrial datasets demonstrates the superiority of our approach over the AS2 baseline, and GenQA trained using supervised data.
Can Contextualizing User Embeddings Improve Sarcasm and Hate Speech Detection?

While implicit embeddings so far have been mostly concerned with creating an overall representation of the user, we evaluate a different approach. By only considering content directed at a specific topic, we create sub-user embeddings, and measure their usefulness on the tasks of sarcasm and hate speech detection. In doing so, we show that task-related topics can have a noticeable effect on model performance, especially when dealing with intended expressions like sarcasm, but less so for hate speech, which is usually labelled as such on the receiving end.
Learning with Rejection for Abstractive Text Summarization

State-of-the-art abstractive summarization systems frequently hallucinate content that is not supported by the source document, mainly due to noise in the training dataset.Existing methods opt to drop the noisy samples or tokens from the training set entirely, reducing the effective training set size and creating an artificial propensity to copy words from the source.In this work, we propose a training objective for abstractive summarization based on rejection learning, in which the model learns whether or not to reject potentially noisy tokens.We further propose a regularized decoding objective that penalizes non-factual candidate summaries during inference by using the rejection probability learned during training.We show that our method considerably improves the factuality of generated summaries in automatic and human evaluations when compared to five baseline models, and that it does so while increasing the abstractiveness of the generated summaries.1 Source: (...) Chris Cox, the university's director of development, said the research centre would translate discoveries made in the laboratory into new treatments.He said it would house 150 additional researchers who will be developing new ideas and treatments.Research will focus on radiation therapy, lung cancer, women's cancers
RAPO: An Adaptive Ranking Paradigm for Bilingual Lexicon Induction

Zhoujin Tian, Chaozhuo Li, Shuo Ren, Zhiqiang Zuo, Zengxuan Wen, Xinyue Hu, Xiao Han, Haizhen Huang, Denvy Deng, Qi Zhang, Xing Xie. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.
Semantic Simplification for Sentiment Classification

Recent work on document-level sentiment classification has shown that the sentiment in the original text is often hard to capture, since the sentiment is usually either expressed implicitly or shifted due to the occurrences of negation and rhetorical words.To this end, we enhance the original text with a sentiment-driven simplified clause to intensify its sentiment.The simplified clause shares the same opinion with the original text but expresses the opinion much more simply.Meanwhile, we employ Abstract Meaning Representation (AMR) for generating simplified clauses, since AMR explicitly provides core semantic knowledge, and potentially offers core concepts and explicit structures of original texts.Empirical studies show the effectiveness of our proposed model over several strong baselines.The results also indicate the importance of simplified clauses for sentiment classification.
Subword-Delimited Downsampling for Better Character-Level Translation

Subword-level models have been the dominant paradigm in NLP.However, character-level models have the benefit of seeing each character individually, providing the model with more detailed information that ultimately could lead to better models.Recent works have shown character-level models to be competitive with subword models, but costly in terms of time and computation.Character-level models with a downsampling component alleviate this, but at the cost of quality, particularly for machine translation.This work analyzes the problems of previous downsampling methods and introduces a novel downsampling method which is informed by subwords.This new downsampling method not only outperforms existing downsampling methods, showing that downsampling characters can be done without sacrificing quality, but also leads to promising performance compared to subword models for translation.
Don’t Say What You Don’t Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search

Abstractive summarization systems today produce fluent and relevant output, but often "hallucinate" statements not supported by the source text.We analyze the connection between hallucinations and training data, and find evidence that models hallucinate because they train on target summaries that are unsupported by the source.Based on our findings, we present PINOCCHIO, a new decoding method that improves the consistency of a transformerbased abstractive summarizer by constraining beam search to avoid hallucinations.Given the model states and outputs at a given step, PINOC-CHIO detects likely model hallucinations based on various measures of attribution to the source text.PINOCCHIO backtracks to find more consistent output, and can opt to produce no summary at all when no consistent generation can be found.In experiments, we find that PINOC-CHIO improves the consistency of generation by an average of 68% on two abstractive summarization datasets, without hurting recall.
He Said, She Said: Style Transfer for Shifting the Perspective of Dialogues

In this work, we define a new style transfer task: perspective shift, which reframes a dialogue from informal first person to a formal third person rephrasing of the text.This task requires challenging coreference resolution, emotion attribution, and interpretation of informal text.We explore several baseline approaches and discuss further directions on this task when applied to short dialogues.As a sample application, we demonstrate that applying perspective shifting to a dialogue summarization dataset (SAMSum) substantially improves the zero-shot performance of extractive news summarization models on this data.Additionally, supervised extractive models perform better when trained on perspective shifted data than on the original dialogues.We release our code publicly.
Improving Complex Knowledge Base Question Answering via Question-to-Action and Question-to-Question Alignment

Complex knowledge base question answering can be achieved by converting questions into sequences of predefined actions.However, there is a significant semantic and structural gap between natural language and action sequences, which makes this conversion difficult.In this paper, we introduce an alignment-enhanced complex question answering framework, called ALCQA, which mitigates this gap through question-to-action alignment and question-toquestion alignment.We train a question rewriting model to align the question and each action, and utilize a pretrained language model to implicitly align the question and KG artifacts.Moreover, considering that similar questions correspond to similar action sequences, we retrieve top-k similar question-answer pairs at the inference stage through question-to-question alignment and propose a novel reward-guided action sequence selection strategy to select from candidate action sequences.We conduct experiments on CQA and WQSP datasets, and the results show that our approach outperforms state-of-the-art methods and obtains a 9.88% improvements in the F1 metric on CQA dataset.
Learning to Model Multimodal Semantic Alignment for Story Visualization

Story visualization aims to generate a sequence of images to narrate each sentence in a multisentence story, where the images should be realistic and keep global consistency across dynamic scenes and characters.Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities.To address this problem, we explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model.More specifically, we introduce dynamic interactions according to learning to dynamically explore various semantic depths and fuse the different-modal information at a matched semantic level, which thus relieves the text-image semantic misalignment problem.Extensive experiments on different datasets demonstrate the improvements of our approach, neither using segmentation masks nor auxiliary captioning networks, on image quality and story consistency, compared with state-of-the-art methods.
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling

Recent large-scale video-language pre-trained models have shown appealing performance on various downstream tasks.However, the pretraining process is computationally expensive due to the requirement of millions of videotext pairs and the redundant data structure of each video.To mitigate these problems, we propose LiteVL, which adapts a pre-trained image-language model BLIP into a video-text model directly on downstream tasks, without heavy pre-training.To enhance the temporal modeling lacking in the image-language model, we propose to add temporal attention modules in the image encoder of BLIP with dynamic temporal scaling.Besides the model-wise adaptation, we also propose a non-parametric pooling mechanism to adaptively reweight the fine-grained video embedding conditioned on the text.Experimental results on text-video retrieval and video question answering show that the proposed LiteVL even outperforms previous video-language pre-trained models by a clear margin, though without any videolanguage pre-training.
Improving Large-scale Paraphrase Acquisition and Generation

This paper addresses the quality issues in existing Twitter-based paraphrase datasets, and discusses the necessity of using two separate definitions of paraphrase for identification and generation tasks.We present a new Multi-Topic Paraphrase in Twitter (MULTIPIT) corpus that consists of a total of 130k sentence pairs with crowdsoursing (MULTIPIT CROWD ) and expert (MULTIPIT EXPERT ) annotations using two different paraphrase definitions for paraphrase identification, in addition to a multi-reference test set (MULTIPIT NMR ) and a large automatically constructed training set (MULTIPIT AUTO ) for paraphrase generation.With improved data annotation quality and task-specific paraphrase definition, the best pre-trained language model fine-tuned on our dataset achieves the stateof-the-art performance of 84.2 F 1 for automatic paraphrase identification.Furthermore, our empirical results also demonstrate that the paraphrase generation models trained on MUL-TIPIT AUTO generate more diverse and highquality paraphrases compared to their counterparts fine-tuned on other corpora such as Quora, MSCOCO, and ParaNMT.
SALTED: A Framework for SAlient Long-tail Translation Error Detection

Traditional machine translation (MT) metrics provide an average measure of translation quality that is insensitive to the long tail of behavioral problems.Examples include translation of numbers, physical units, dropped content and hallucinations.These errors, which occur rarely and unpredictably in Neural Machine Translation (NMT), greatly undermine the reliability of state-of-the-art MT systems.Consequently, it is important to have visibility into these problems during model development.Towards this end, we introduce SALTED, a specifications-based framework for behavioral testing of NMT models.At the core of our approach is the use of high-precision detectors that flag errors (or alternatively, verify output correctness) between a source sentence and a system output.These detectors provide fine-grained measurements of long-tail errors, providing a trustworthy view of problems that were previously invisible.We demonstrate that such detectors could be used not just to identify salient long-tail errors in MT systems, but also for higher-recall filtering of the training data, fixing targeted errors with model fine-tuning in NMT and generating novel data for metamorphic testing to elicit further bugs in models.