Reviewerly/exHarmony huggingface dataset:
Recurrence Boosts Diversity! Revisiting Recurrent Latent Variable in Transformer-Based Variational AutoEncoder for Diverse Text Generation

Variational Auto-Encoder (VAE) has been widely adopted in text generation.Among many variants, recurrent VAE learns tokenwise latent variables with each conditioned on the preceding ones, which captures sequential variability better in the era of RNN.However, it is unclear how to incorporate such recurrent dynamics into the recently dominant Transformer due to its parallelism.In this work, we propose TRACE, a Transformer-based recurrent VAE structure.TRACE imposes recurrence on segment-wise latent variables with arbitrarily separated text segments and constructs the posterior distribution with residual parameterization.Besides, we design an acceleration method by approximating idempotent matrices, which allows parallelism while maintaining the conditional dependence of latent variables.We demonstrate that TRACE could enhance the entanglement of each segment and preceding latent variables and deduce a non-zero lower bound of the KL term, providing a theoretical guarantee of generation diversity.Experiments on two unconditional and one conditional generation tasks show that TRACE achieves significantly improved diversity while maintaining satisfactory generation quality.
Multi-Label Intent Detection via Contrastive Task Specialization of Sentence Encoders

Deploying task-oriented dialog (TOD) systems for new domains and tasks requires natural language understanding models that are 1) resource-efficient and work under low-data regimes; 2) adaptable, efficient, and quickto-train; 3) expressive and can handle complex TOD scenarios with multiple user intents in a single utterance.Motivated by these requirements, we introduce a novel framework for multi-label intent detection (mID): MULTI-CONVFIT (Multi-Label Intent Detection via Contrastive Conversational Fine-Tuning).While previous work on efficient single-label intent detection learns a classifier on top of a fixed sentence encoder (SE), we propose to 1) transform general-purpose SEs into task-specialized SEs via contrastive fine-tuning on annotated multi-label data, 2) where task specialization knowledge can be stored into lightweight adapter modules without updating the original parameters of the input SE, and then 3) we build improved mID classifiers stacked on top of fixed specialized SEs.Our main results indicate that MULTI-CONVFIT yields effective mID models, with large gains over non-specialized SEs reported across a spectrum of different mID datasets, both in low-data and high-data regimes.
Evaluating and Improving Factuality in Multimodal Abstractive Summarization

Current metrics for evaluating factuality for abstractive document summarization have achieved high correlations with human judgment, but they do not account for the vision modality and thus are not adequate for visionand-language summarization.We propose CLIPBERTSCORE, a simple weighted combination of CLIPScore (Hessel et al., 2021) and BERTScore (Zhang* et al., 2020) to leverage the robustness and strong factuality detection performance between image-summary and document-summary, respectively.Next, due to the lack of meta-evaluation benchmarks to evaluate the quality of multimodal factuality metrics, we collect human judgments of factuality with respect to documents and images.We show that this simple combination of two metrics in the zero-shot setting achieves higher correlations than existing factuality metrics for document summarization, outperforms an existing multimodal summarization metric, and performs competitively with strong multimodal factuality metrics specifically fine-tuned for the task.Our thorough analysis demonstrates the robustness and high correlation of CLIP-BERTSCORE and its components on four factuality metric-evaluation benchmarks.Finally, we demonstrate two practical downstream applications of our CLIPBERTSCORE metric: for selecting important images to focus on during training, and as a reward for reinforcement learning to improve factuality of multimodal summary generation w.r.t automatic and human evaluation. 1
POQue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events

Knowledge about outcomes is critical for complex event understanding but is hard to acquire.We show that by pre-identifying a participant in a complex event, crowdworkers are able to (1) infer the collective impact of salient events that make up the situation, (2) annotate the volitional engagement of participants in causing the situation, and (3) ground the outcome of the situation in state changes of the participants.By creating a multi-step interface and a careful quality control strategy, we collect a high quality annotated dataset of 8K short newswire narratives and ROCStories with high inter-annotator agreement (0.74-0.96 weighted Fleiss Kappa).Our dataset, POQue (Participant Outcome Questions), enables the exploration and development of models that address multiple aspects of semantic understanding.Experimentally, we show that current language models lag behind human performance in subtle ways through our task formulations that target abstract and specific comprehension of a complex event, its outcome, and a participant's influence over the event culmination.
Predicting Prerequisite Relations for Unseen Concepts

Concept prerequisite learning (CPL) plays a key role in developing technologies that assist people to learn a new complex topic or concept.Previous work commonly assumes that all concepts are given at training time and solely focuses on predicting the unseen prerequisite relationships between them.However, many real-world scenarios deal with concepts that are left undiscovered at training time, which is relatively unexplored.This paper studies this problem and proposes a novel alternating knowledge distillation approach to take advantage of both content-and graph-based models for this task.Extensive experiments on three public benchmarks demonstrate up to 10% improvements in terms of F1 score.
A Framework for Automatic Generation of Spoken Question-Answering Data

This paper describes a framework to automatically generate a spoken question answering (QA) dataset.The framework consists of a question generation (QG) module to generate questions automatically from given text documents, a text-to-speech (TTS) module to convert the text documents into spoken form and an automatic speech recognition (ASR) module to transcribe the spoken content.The final dataset contains question-answer pairs for both the reference text and ASR transcriptions as well as the audio files corresponding to each reference text.For QG and ASR systems we used pre-trained multilingual encoder-decoder transformer models and fine-tuned these models using a limited amount of manually generated QA data and TTS-based speech data, respectively.As a proof of concept, we investigated the proposed framework for Turkish and generated the Turkish Question Answering (TurQuAse) dataset using Wikipedia articles.Manual evaluation of the automatically generated questionanswer pairs and QA performance evaluation with state-of-the-art models on TurQuAse show that the proposed framework is efficient for automatically generating spoken QA datasets.To the best of our knowledge, TurQuAse is the first publicly available spoken question answering dataset for Turkish.The proposed framework can be easily extended to other languages where a limited amount of QA data is available.
PaCo: Preconditions Attributed to Commonsense Knowledge

Humans can seamlessly reason with circumstantial preconditions of commonsense knowledge.We understand that a glass is used for drinking water, unless the glass is broken or the water is toxic.Despite state-of-the-art (SOTA) language models' (LMs) impressive performance on inferring commonsense knowledge, it is unclear whether they understand the circumstantial preconditions.To address this gap, we propose a novel challenge of reasoning with circumstantial preconditions.We collect a dataset, called PaCo, consisting of 12.4 thousand preconditions of commonsense statements expressed in natural language.Based on this dataset, we create three canonical evaluation tasks and use them to examine the capability of existing LMs to understand situational preconditions.Our results reveal a 10-30% gap between machine and human performance on our tasks, which shows that reasoning with preconditions is an open challenge. 1
Controlled Language Generation for Language Learning Items

This work aims to employ natural language generation (NLG) to rapidly generate items for English language learning applications: this requires both language models capable of generating fluent, high-quality English, and to control the output of the generation to match the requirements of the relevant items.We experiment with deep pretrained models for this task, developing novel methods for controlling items for factors relevant in language learning: diverse sentences for different proficiency levels and argument structure to test grammar.Human evaluation demonstrates high grammatically scores for all models (3.4 and above out of 4), and higher length (24%) and complexity (9%) over the baseline for the advanced proficiency model.Our results show that we can achieve strong performance while adding additional control to ensure diverse, tailored content for individual users.
MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling

Static subword tokenization algorithms have been an essential component of recent works on language modeling.However, their static nature results in important flaws that degrade the models' downstream performance and robustness.In this work, we propose MANTa, a Module for Adaptive Neural TokenizAtion.MANTa is a differentiable tokenizer trained end-to-end with the language model.The resulting system offers a trade-off between the expressiveness of byte-level models and the speed of models trained using subword tokenization.In addition, our tokenizer is highly explainable since it produces an explicit segmentation of sequences into blocks.We evaluate our pretrained model on several English datasets from different domains as well as on synthetic noise.We find that MANTa improves robustness to character perturbations and out-of-domain data.We then show that MANTa performs comparably to other models on the general-domain GLUE benchmark.Finally, we show that it is considerably faster than strictly byte-level models.
Quality Scoring of Source Words in Neural Translation Models

Word-level quality scores on input source sentences can provide useful feedback to an enduser when translating into an unfamiliar target language.Recent approaches either require training custom models on synthetic data or repeatedly invoking the translation model.We propose a simple approach based on comparing probabilities from two language models.The basic premise of our method is to reason how well each source word is explained by the generated translation as against the preceding source language words.Our approach provides between 2.2 and 27.1 higher F1 score and is significantly faster than state of the art methods on three language pairs.Also, our method does not require training any new model.We release a public dataset on word omissions and mistranslations on a new language pair. 1
A GlobalPointer based Robust Approach for Information Extraction from Dialog Transcripts

With the widespread popularisation of intelligent technology, task-based dialogue systems (TOD) are increasingly being applied to a wide variety of practical scenarios.As the key tasks in dialogue systems, named entity recognition and slot filling play a crucial role in the completeness and accuracy of information extraction.This paper is an evaluation paper for Sere-TOD 2022 Workshop challenge (Track 1: Information extraction from dialog transcripts).We proposed a multi-model fusion approach based on GlobalPointer, combined with some optimisation tricks, finally achieved an entity F1 of 60.73, an entity-slot-value triple F1 of 56, and an average F1 of 58.37, and got the highest score in SereTOD 2022 Workshop challenge 1 .
Using Transformer-based Models for Taxonomy Enrichment and Sentence Classification

In this paper, we present a system that addresses the taxonomy enrichment problem for "Environment, Social and Governance" issues in the financial domain, as well as classifying sentences as sustainable or unsustainable, for FinSim4-ESG, a shared task for the FinNLP workshop at IJCAI-2022.We first created a derived dataset for taxonomy enrichment by using a sentence-BERT-based paraphrase detector (Reimers and Gurevych, 2019) (on the train set) to create positive and negative termconcept pairs.We then model the problem by fine-tuning the sentence-BERT-based paraphrase detector on this derived dataset, and use it as the encoder, and use a Logistic Regression classifier as the decoder, resulting in test Accuracy: 0.6 and Avg.Rank: 1.97.In case of the sentence classification task, the best performing classifier (Accuracy: 0.92) consists of a pre-trained RoBERTa model (Liu et al., 2019a) as the encoder and a Feed Forward Neural Network classifier as the decoder.
What do Large Language Models Learn beyond Language?

Large language models (LMs) have rapidly become a mainstay in Natural Language Processing.These models are known to acquire rich linguistic knowledge from training on large amounts of text.In this paper, we investigate if pre-training on text also confers these models with helpful 'inductive biases' for nonlinguistic reasoning.On a set of 19 diverse non-linguistic tasks involving quantitative computations, recognizing regular expressions and reasoning over strings.We find that pretrained models significantly outperform comparable non-pretrained neural models.This remains true also in experiments with training nonpretrained models with fewer parameters to account for model regularization effects.We further explore the effect of text domain on LMs by pretraining models from text from different domains and provenances.Our experiments surprisingly reveal that the positive effects of pre-training persist even when pretraining on multi-lingual text or computer code, and even for text generated from synthetic languages.Our findings suggest a hithertho unexplored deep connection between pre-training and inductive learning abilities of language models 1 .
Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models

Model compression by way of parameter pruning, quantization, or distillation has recently gained popularity as an approach for reducing the computational requirements of modern deep neural network models for NLP.Inspired by prior works suggesting a connection between simpler, more generalizable models and those that lie within wider loss basins, we hypothesize that optimizing for flat minima should lead to simpler parameterizations and thus more compressible models.We propose to combine sharpness-aware minimization (SAM) with various task-specific model compression methods, including iterative magnitude pruning (IMP), structured pruning with a distillation objective, and post-training dynamic quantization.Empirically, we show that optimizing for flatter minima consistently leads to greater compressibility of parameters compared to vanilla Adam when fine-tuning BERT models, with little to no loss in accuracy on the GLUE text classification and SQuAD question answering benchmarks.Moreover, SAM finds superior winning tickets during IMP that 1) are amenable to vanilla Adam optimization, and 2) transfer more effectively across tasks. 1
Faithful to the Document or to the World? Mitigating Hallucinations via Entity-Linked Knowledge in Abstractive Summarization

Existing abstractive summarization systems are hampered by content hallucinations in which models generate text that is not directly inferable from the source alone.Annotations from prior work have shown that some of these hallucinations, while being 'unfaithful' to the source, are nonetheless factual.Our analysis in this paper suggests that these factual hallucinations occur as a result of the prevalence of factual yet unfaithful entities in summarization datasets.We find that these entities are not aberrations, but instead examples of additional world knowledge being readily used to latently connect entities and concepts -in this case connecting entities in the source document to those in the target summary.In our analysis and experiments, we demonstrate that connecting entities to an external knowledge base can lend provenance to many of these unfaithful yet factual entities, and further, this knowledge can be used to improve the factuality of summaries without simply making them more extractive.
CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation

Named entity recognition (NER) suffers from the scarcity of annotated training data, especially for low-resource languages without labeled data.Cross-lingual NER has been proposed to alleviate this issue by transferring knowledge from high-resource languages to low-resource languages via aligned crosslingual representations or machine translation results.However, the performance of crosslingual NER methods is severely affected by the unsatisfactory quality of translation or label projection.To address these problems, we propose a Cross-lingual Entity Projection framework (CROP) to enable zero-shot crosslingual NER with the help of a multilingual labeled sequence translation model.Specifically, the target sequence is first translated into the source language and then tagged by a source NER model.We further adopt a labeled sequence translation model to project the tagged sequence back to the target language and label the target raw sentence.Ultimately, the whole pipeline is integrated into an end-to-end model by the way of self-training.Experimental results on two benchmarks demonstrate that our method substantially outperforms the previous strong baseline by a large margin of +3∼7 F1 scores and achieves state-of-the-art performance.
From Spelling to Grammar: A New Framework for Chinese Grammatical Error Correction

Chinese Grammatical Error Correction (CGEC) aims to generate a correct sentence from an erroneous sequence, where different kinds of errors are mixed.This paper divides the CGEC task into two steps, namely spelling error correction and grammatical error correction.Specially, we propose a novel zero-shot approach for spelling error correction, which is simple but effective, obtaining a high precision to avoid error accumulation of the pipeline structure.To handle grammatical error correction, we design part-of-speech (POS) features and semantic class features to enhance the neural network model, and propose an auxiliary task to predict the POS sequence of the target sentence.Our proposed framework achieves a 42.11 F 0 .5 score on CGEC dataset without using any synthetic data or data augmentation methods, which outperforms the previous state-of-the-art by a wide margin of 1.30 points.Moreover, our model produces meaningful POS representations that capture different POS words and convey reasonable POS transition rules.
ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Models

Data-to-text generation is challenging due to the great variety of the input data in terms of domains (e.g., finance vs sports) or schemata (e.g., diverse predicates).Recent end-to-end neural methods thus require substantial training examples to learn to disambiguate and describe the data.Yet, real-world data-to-text problems often suffer from various data-scarce issues: one may have access to only a handful of or no training examples, and/or have to rely on examples in a different domain or schema.To fill this gap, we propose Any-Shot Data-to-Text (ASDOT), a new approach flexibly applicable to diverse settings by making efficient use of any given (or no) examples.ASDOT consists of two steps, data disambiguation and sentence fusion, both of which are amenable to be solved with off-the-shelf pretrained language models (LMs) with optional finetuning.In the data disambiguation stage, we employ the prompted GPT-3 model to understand possibly ambiguous triples from the input data and convert each into a short sentence with reduced ambiguity.The sentence fusion stage then uses an LM like T5 to fuse all the resulting sentences into a coherent paragraph as the final description.We evaluate extensively on various datasets in different scenarios, including the zero-/few-/fullshot settings, and generalization to unseen predicates and out-of-domain data.Experimental results show that ASDOT consistently achieves significant improvement over baselines, e.g., a 30.81BLEU gain on the DART dataset under the zero-shot setting. 1
Model Criticism for Long-Form Text Generation

Language models have demonstrated the ability to generate highly fluent text; however, it remains unclear whether their output retains coherent high-level structure (e.g., story progression).Here, we propose to apply a statistical tool, model criticism in latent space, to evaluate the high-level structure of the generated text.Model criticism compares the distributions between real and generated data in a latent space obtained according to an assumptive generative process.Different generative processes identify specific failure modes of the underlying model.We perform experiments on three representative aspects of highlevel discourse-coherence, coreference, and topicality-and find that transformer-based language models are able to capture topical structures but have a harder time maintaining structural coherence or modeling coreference.
Graph-Induced Transformers for Efficient Multi-Hop Question Answering

A graph is a suitable data structure to represent the structural information of text.Recently, multi-hop question answering (MHQA) tasks, which require inter-paragraph/sentence linkages, have come to exploit such properties of a graph.Previous approaches to MHQA relied on leveraging the graph information along with the pre-trained language model (PLM) encoders.However, this trend exhibits the following drawbacks: (i) sample inefficiency while training in a low-resource setting; (ii) lack of reusability due to changes in the model structure or input.Our work proposes the Graph-Induced Transformer (GIT) that applies graphderived attention patterns directly into a PLM, without the need to employ external graph modules.GIT can leverage the useful inductive bias of graphs while retaining the unperturbed Transformer structure and parameters.Our experiments on HotpotQA successfully demonstrate both the sample efficient characteristic of GIT and its capacity to replace the graph modules while preserving model performance.
Overview of the FinNLP-2022 ERAI Task: Evaluating the Rationales of Amateur Investors

This paper provides an overview of the shared task, Evaluating the Rationales of Amateur Investors (ERAI), in FinNLP-2022 at EMNLP-2022.This shared task aims to sort out investment opinions that would lead to higher profit from social platforms.We obtained 19 registered teams; 9 teams submitted their results for final evaluation, and 8 teams submitted papers to share their methods.The discussed directions are various: prompting, fine-tuning, translation system comparison, and tailor-made neural network architectures.We provide details of the task settings, data statistics, participants' results, and fine-grained analysis.