Reviewerly/exHarmony huggingface dataset:
KeywordScape: Visual Document Exploration using Contextualized Keyword Embeddings

Although contextualized word embeddings have led to great improvements in automatic language understanding, their potential for practical applications in document exploration and visualization has been little explored.Common visualization techniques used for, e.g., model analysis usually provide simple scatter plots of token-level embeddings that do not provide insight into their contextual use.In this work, we propose KeywordScape, a visual exploration tool that allows users to overview, summarize, and explore the semantic content of documents based on their keywords.While existing keyword-based exploration tools assume that keywords have static meanings, our tool represents keywords in terms of their contextualized embeddings.Our application visualizes these embeddings in a semantic landscape that represents keywords as islands on a spherical map.This keeps keywords with similar context close to each other, allowing for a more precise search and comparison of documents.
SepLL: Separating Latent Class Labels from Weak Supervision Noise

In the weakly supervised learning paradigm, labeling functions automatically assign heuristic, often noisy, labels to data samples.In this work, we provide a method for learning from weak labels by separating two types of complementary information associated with the labeling functions: information related to the target label and information specific to one labeling function only.Both types of information are reflected to different degrees by all labeled instances.In contrast to previous works that aimed at correcting or removing wrongly labeled instances, we learn a branched deep model that uses all data as-is, but splits the labeling function information in the latent space.Specifically, we propose the end-to-end model SepLL which extends a transformer classifier by introducing a latent space for labeling function specific and task-specific information.The learning signal is only given by the labeling functions matches, no pre-processing or label model is required for our method.Notably, the task prediction is made from the latent layer without any direct task signal.Experiments on Wrench text classification tasks show that our model is competitive with the state-of-the-art, and yields a new best average performance.
Distilling Multilingual Transformers into CNNs for Scalable Intent Classification

We describe an application of Knowledge Distillation used to distill and deploy multilingual Transformer models for voice assistants, enabling text classification for customers globally.Transformers have set new state-of-theart results for tasks like intent classification, and multilingual models exploit cross-lingual transfer to allow serving requests across 100+ languages.However, their prohibitive inference time makes them impractical to deploy in real-world scenarios with low latency requirements, such as is the case of voice assistants.We address the problem of cross-architecture distillation of multilingual Transformers to simpler models, while maintaining multilinguality without performance degradation.Training multilingual student models has received little attention, and is our main focus.We show that a teacher-student framework, where the teacher's unscaled activations (logits) on unlabelled data are used to supervise student model training, enables distillation of Transformers into efficient multilingual CNN models.Our student model achieves equivalent performance as the teacher, and outperforms a similar model trained on the labelled data used to train the teacher model.This approach has enabled us to accurately serve global customer requests at speed (18x improvement), scale, and low cost.
Towards Compositional Generalization in Code Search

We study compositional generalization, which aims to generalize on unseen combinations of seen structural elements, for code search.Unlike existing approaches of partially pursuing this goal, we study how to extract structural elements, which we name a template that directly targets compositional generalization.Thus we propose CTBERT, or Code Template BERT, representing codes using automatically extracted templates as building blocks.We empirically validate CTBERT on two public code search benchmarks, AdvTest and CSN.Further, we show that templates are complementary to data flow graphs in GraphCodeBERT, by enhancing structural context around variables.
Analyzing Multi-Task Learning for Abstractive Text Summarization

Despite the recent success of multi-task learning and pre-finetuning for natural language understanding, few works have studied the effects of task families on abstractive text summarization.Task families are a form of task grouping during the pre-finetuning stage to learn common skills, such as reading comprehension.To close this gap, we analyze the influence of multi-task learning strategies using task families for the English abstractive text summarization task.We group tasks into one of three strategies, i.e., sequential, simultaneous, and continual multi-task learning, and evaluate trained models through two downstream tasks.We find that certain combinations of task families (e.g., advanced reading comprehension and natural language inference) positively impact downstream performance.Further, we find that choice and combinations of task families influence downstream performance more than the training scheme, supporting the use of task families for abstractive text summarization.Our code is publicly available 1 .
When More Data Hurts: A Troubling Quirk in Developing Broad-Coverage Natural Language Understanding Systems

In natural language understanding (NLU) production systems, users' evolving needs necessitate the addition of new features over time, indexed by new symbols added to the meaning representation space.This requires additional training data and results in ever-growing datasets.We present the first systematic investigation of this incremental symbol learning scenario.Our analysis reveals a troubling quirk in building broad-coverage NLU systems: as the training dataset grows, performance on the new symbol often decreases if we do not accordingly increase its training data.This suggests that it becomes more difficult to learn new symbols with a larger training dataset.We show that this trend holds for multiple mainstream models on two common NLU tasks: intent recognition and semantic parsing.Rejecting class imbalance as the sole culprit, we reveal that the trend is closely associated with an effect we call source signal dilution, where strong lexical cues for the new symbol become diluted as the training dataset grows.Selectively dropping training examples to prevent dilution often reverses the trend, showing the over-reliance of mainstream neural NLU models on simple lexical cues. 1 * Work done as an intern at Microsoft Semantic Machines. 1 Code, models, and data are available at https://aka.ms/nlu-incremental-symbol-learning.
Successive Prompting for Decomposing Complex Questions

Answering complex questions that require making latent decisions is a challenging task, especially when limited supervision is available.Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting by demonstrating how to output intermediate rationalizations while solving the complex question in a single pass.We introduce "Successive Prompting", where we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution.Successive prompting decouples the supervision for decomposing complex questions from the supervision for answering simple questions, allowing us to (1) have multiple opportunities to query in-context examples at each reasoning step (2) learn question decomposition separately from question answering, including using synthetic data, and (3) use bespoke (fine-tuned) components for reasoning steps where a large LM does not perform well.The intermediate supervision is typically manually written, which can be expensive to collect.We introduce a way to generate a synthetic dataset which can be used to bootstrap a model's ability to decompose and answer intermediate questions.Our best model (with successive prompting) achieves an improvement of ∼5% absolute F1 on a few-shot version of the DROP dataset when compared with a stateof-the-art model with the same supervision.
Understanding Domain Learning in Language Models Through Subpopulation Analysis

We investigate how different domains are encoded in modern neural network architectures.We analyze the relationship between natural language domains, model size, and the amount of training data used.The primary analysis tool we develop is based on subpopulation analysis with Singular Vector Canonical Correlation Analysis (SVCCA), which we apply to Transformer-based language models (LMs).We compare the latent representations of such a language model at its different layers from a pair of models: a model trained on multiple domains (an experimental model) and a model trained on a single domain (a control model).Through our method, we find that increasing the model capacity impacts how domain information is stored in upper and lower layers differently.In addition, we show that larger experimental models simultaneously embed domainspecific information as if they were conjoined control models.These findings are confirmed qualitatively, demonstrating the validity of our method.
Pneg: Prompt-based Negative Response Generation for Dialogue Response Selection Task

In retrieval-based dialogue systems, a response selection model acts as a ranker to select the most appropriate response among several candidates.However, such selection models tend to rely on context-response content similarity, which makes models vulnerable to adversarial responses that are semantically similar but not relevant to the dialogue context.Recent studies have shown that leveraging these adversarial responses as negative training samples is useful for improving the discriminating power of the selection model.Nevertheless, collecting human-written adversarial responses is expensive, and existing synthesizing methods often have limited scalability.To overcome these limitations, this paper proposes a simple but efficient method for generating adversarial negative responses leveraging a large-scale language model.Experimental results on dialogue selection tasks show that our method outperforms other methods of synthesizing adversarial negative responses.These results suggest that our method can be an effective alternative to human annotators in generating adversarial responses.Our dataset and generation code is available at https://github.com/leenw23/generating-negatives-by-gpt3.
Privacy-Preserving Models for Legal Natural Language Processing

Pre-training large transformer models with indomain data improves domain adaptation and helps gain performance on the domain-specific downstream tasks.However, sharing models pre-trained on potentially sensitive data is prone to adversarial privacy attacks.In this paper, we asked to which extent we can guarantee privacy of pre-training data and, at the same time, achieve better downstream performance on legal tasks without the need of additional labeled data.We extensively experiment with scalable self-supervised learning of transformer models under the formal paradigm of differential privacy and show that under specific training configurations we can improve downstream performance without sacrifying privacy protection for the in-domain data.Our main contribution is utilizing differential privacy for large-scale pre-training of transformer language models in the legal NLP domain, which, to the best of our knowledge, has not been addressed before. 1
Narrate Dialogues for Better Summarization

Dialogue summarization models aim to generate a concise and accurate summary for multiparty dialogue.The complexity of dialogue, including coreference, dialogue acts, and interspeaker interactions bring unique challenges to dialogue summarization.Most recent neural models achieve state-of-art performance following the pretrain-then-finetune recipe, where the large-scale language model (LLM) is pretrained on large-scale single-speaker written text, but later finetuned on multi-speaker dialogue text.To mitigate the gap between pretraining and finetuning, we propose several approaches to convert the dialogue into a third-person narrative style and show that the narration serves as a valuable annotation for LLMs.Empirical results on three benchmark datasets show our simple approach achieves higher scores on the ROUGE and a factual correctness metric.
Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation

A growing body of research has demonstrated the inability of NLP models to generalize compositionally and has tried to alleviate it through specialized architectures, training schemes, and data augmentation, among other approaches.In this work, we study a different approach: training on instances with diverse structures.We propose a model-agnostic algorithm for subsampling such sets of instances from a labeled instance pool with structured outputs.Evaluating on both compositional template splits and traditional IID splits of 5 semantic parsing datasets of varying complexity, we show that structurally diverse training using our algorithm leads to comparable or better generalization than prior algorithms in 9 out of 10 dataset-split type pairs.In general, we find structural diversity to consistently improve sample efficiency compared to random train sets.Moreover, we show that structurally diverse sampling yields comprehensive test sets that are a lot more challenging than IID test sets.Finally, we provide two explanations for improved generalization from diverse train sets: 1) improved coverage of output substructures, and 2) a reduction in spurious correlations between these substructures.
ASQA: Factoid Questions Meet Long-Form Answers

An abundance of datasets and availability of reliable evaluation metrics have resulted in strong progress in factoid question answering (QA).This progress, however, does not easily transfer to the task of long-form QA, where the goal is to answer questions that require in-depth explanations.The hurdles include (i) a lack of high-quality data, and (ii) the absence of a well-defined notion of the answer's quality.In this work, we address these problems by (i) releasing a novel dataset and a task that we call ASQA (Answer Summaries for Questions which are Ambiguous); and (ii) proposing a reliable metric for measuring performance on ASQA.Our task focuses on factoid questions that are ambiguous, that is, have different correct answers depending on interpretation.Answers to ambiguous questions should synthesize factual information from multiple sources into a long-form summary that resolves the ambiguity.In contrast to existing long-form QA tasks (such as ELI5), ASQA admits a clear notion of correctness: a user faced with a good summary should be able to answer different interpretations of the original ambiguous question.We use this notion of correctness to define an automated metric of performance for ASQA.Our analysis demonstrates an agreement between this metric and human judgments, and reveals a considerable gap between human performance and strong baselines.
AutoCAD: Automatically Generate Counterfactuals for Mitigating Shortcut Learning

Recent studies have shown the impressive efficacy of counterfactually augmented data (CAD) for reducing NLU models' reliance on spurious features and improving their generalizability.However, current methods still heavily rely on human efforts or task-specific designs to generate counterfactuals, thereby impeding CAD's applicability to a broad range of NLU tasks.In this paper, we present Au-toCAD, a fully automatic and task-agnostic CAD generation framework.AutoCAD first leverages a classifier to unsupervisedly identify rationales as spans to be intervened, which disentangles spurious and causal features.Then, AutoCAD performs controllable generation enhanced by unlikelihood training to produce diverse counterfactuals.Extensive evaluations on multiple out-of-domain and challenge benchmarks demonstrate that AutoCAD consistently and significantly boosts the out-of-distribution performance of powerful pre-trained models across different NLU tasks, which is comparable or even better than previous state-of-theart human-in-the-loop or task-specific CAD methods.
Learning When and What to Quote: A Quotation Recommender System with Mutual Promotion of Recommendation and Generation

This work extends the current quotation recommendation task to a more realistic quotation recommender system that learns to predict when to quote and what to quote jointly.The system consists of three modules (tasks), a prediction module to predict whether to quote given conversation contexts, a recommendation module to recommend suitable quotations and a generation module generating quotations or sentences in ordinary language to continue the conversation.We benchmark several competitive models for the two newly introduced tasks (i.e., when-to-quote and what-to-continue).For quotation recommendation, compared with previous work that is either generation-based or ranking-based recommendation, we propose a novel framework with mutual promotion of generation module and ranking-based recommendation module 1 .Experiments show that our framework achieves significantly better performance than baselines on two datasets.Further experiments and analyses validate the effectiveness of the proposed mechanisms and get a better understanding of the quotation recommendation task.
Multilingual Sentence Transformer as A Multilingual Word Aligner

Multilingual pretrained language models (mPLMs) have shown their effectiveness in multilingual word alignment induction.However, these methods usually start from mBERT or XLM-R.In this paper, we investigate whether multilingual sentence Transformer LaBSE is a strong multilingual word aligner.This idea is non-trivial as LaBSE is trained to learn language-agnostic sentence-level embeddings, while the alignment extraction task requires the more fine-grained word-level embeddings to be language-agnostic.We demonstrate that the vanilla LaBSE outperforms other mPLMs currently used in the alignment task, and then propose to finetune LaBSE on parallel corpus for further improvement.Experiment results on seven language pairs show that our best aligner outperforms previous state-of-theart models of all varieties.In addition, our aligner supports different language pairs in a single model, and even achieves new state-ofthe-art on zero-shot language pairs that does not appear in the finetuning process.
G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks

Recently, domain-specific PLMs have been proposed to boost the task performance of specific domains (e.g., biomedical and computer science) by continuing to pre-train general PLMs with domain-specific corpora.However, this Domain-Adaptive Pre-Training (DAPT; Gururangan et al. ( 2020)) tends to forget the previous general knowledge acquired by general PLMs, which leads to a catastrophic forgetting phenomenon and sub-optimal performance.To alleviate this problem, we propose a new framework of General Memory-Augmented Pre-trained Language Model (G-MAP), which augments the domain-specific PLM by a memory representation built from the frozen general PLM without losing any general knowledge.Specifically, we propose a new memory-augmented layer, and based on it, different augmented strategies are explored to build the memory representation and then adaptively fuse it into the domain-specific PLM.We demonstrate the effectiveness of G-MAP on various domains (biomedical and computer science publications, news, and reviews) and different kinds (text classification, QA, NER) of tasks, and the extensive results show that the proposed G-MAP 1 can achieve SOTA results on all tasks.
Cross-stitching Text and Knowledge Graph Encoders for Distantly Supervised Relation Extraction

Bi-encoder architectures for distantlysupervised relation extraction are designed to make use of the complementary information found in text and knowledge graphs (KG).However, current architectures suffer from two drawbacks.They either do not allow any sharing between the text encoder and the KG encoder at all, or, in case of models with KG-to-text attention, only share information in one direction.Here, we introduce cross-stitch bi-encoders, which allow full interaction between the text encoder and the KG encoder via a cross-stitch mechanism.The cross-stitch mechanism allows sharing and updating representations between the two encoders at any layer, with the amount of sharing being dynamically controlled via cross-attention-based gates.Experimental results on two relation extraction benchmarks from two different domains show that enabling full interaction between the two encoders yields strong improvements.
Measurement Extraction with Natural Language Processing: A Review

Quantitative data is important in many domains.Information extraction methods draw structured data from documents.However, the extraction of quantities and their contexts has received little attention in the history of information extraction.In this review, an overview of prior work on measurement extraction is presented.We describe different approaches to measurement extraction and outline the challenges posed by this task.The review concludes with an outline of potential future research.Research strains in measurement extraction tend to be isolated and lack a common terminology.Improvements in numerical reasoning, more extensive datasets, and the consideration of wider contexts may lead to significant improvements in measurement extraction.
Dealing with Abbreviations in the Slovenian Biographical Lexicon

Abbreviations present a significant challenge for NLP systems because they cause tokenization and out-of-vocabulary errors.They can also make the text less readable, especially in reference printed books, where they are extensively used.Abbreviations are especially problematic in low-resource settings, where systems are less robust to begin with.In this paper, we propose a new method for addressing the problems caused by a high density of domainspecific abbreviations in a text.We apply this method to the case of a Slovenian biographical lexicon and evaluate it on a newly developed gold-standard dataset of 51 Slovenian biographies.Our abbreviation identification method performs significantly better than commonly used ad-hoc solutions, especially at identifying unseen abbreviations.We also propose and present the results of a method for expanding the identified abbreviations in context.
Towards Unifying Reference Expression Generation and Comprehension

Reference Expression Generation (REG) and Comprehension (REC) are two highly correlated tasks. Modeling REG and REC simultaneously for utilizing the relation between them is a promising way to improve both. However, the problem of distinct inputs, as well as building connections between them in a single model, brings challenges to the design and training of the joint model. To address the problems, we propose a unified model for REG and REC, named UniRef. It unifies these two tasks with the carefully-designed Image-Region-Text Fusion layer (IRTF), which fuses the image, region and text via the image cross-attention and region cross-attention. Additionally, IRTF could generate pseudo input regions for the REC task to enable a uniform way for sharing the identical representation space across the REC and REG. We further propose Vision-conditioned Masked Language Modeling (VMLM) and Text-Conditioned Region Prediction (TRP) to pre-train UniRef model on multi-granular corpora. The VMLM and TRP are directly related to REG and REC, respectively, but could help each other. We conduct extensive experiments on three benchmark datasets, RefCOCO, RefCOCO+ and RefCOCOg. Experimental results show that our model outperforms previous state-of-the-art methods on both REG and REC.