Reviewerly/exHarmony huggingface dataset:
Class Incremental Learning for Intent Classification with Limited or No Old Data

In this paper, we explore class-incremental learning for intent classification (IC) in a setting with limited old data available.IC is the task of mapping user utterances to their corresponding intents.Even though classincremental learning without storing the old data yields high potential of reducing human and computational resources in industry NLP model releases, to the best of our knowledge, it hasn't been studied for NLP classification tasks in the literature before.In this work, we compare several contemporary class-incremental learning methods, i.e., BERT warm start, L2, Elastic Weight Consolidation, RecAdam and Knowledge Distillation within two realistic class-incremental learning scenarios: one where only the previous model is assumed to be available, but no data corresponding to old classes, and one in which limited unlabeled data for old classes is assumed to be available.Our results indicate that among the investigated continual learning methods, Knowledge Distillation worked best for our class-incremental learning tasks, and adding limited unlabeled data helps the model in both adaptability and stability.
Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models

How to usefully encode compositional task structure has long been a core challenge in AI.Recent work in chain of thought prompting has shown that for very large neural language models (LMs), explicitly demonstrating the inferential steps involved in a target task may improve performance over end-to-end learning that focuses on the target task alone.However, chain of thought prompting has significant limitations due to its dependency on huge pretrained LMs.In this work, we present compositional fine-tuning (CFT): an approach based on explicitly decomposing a target task into component tasks, and then fine-tuning smaller LMs on a curriculum of such component tasks.We apply CFT to recommendation tasks in two domains, world travel and local dining, as well as a previously studied inferential task (sports understanding).We show that CFT outperforms end-to-end learning even with equal amounts of data, and gets consistently better as more component tasks are modeled via fine-tuning.Compared with chain of thought prompting, CFT performs at least as well using LMs only 7.4% of the size, and is moreover applicable to task domains for which data are not available during pretraining.
SMSMix: Sense-Maintained Sentence Mixup for Word Sense Disambiguation

Word Sense Disambiguation (WSD) is an NLP task aimed at determining the correct sense of a word in a sentence from discrete sense choices.Although current systems have attained unprecedented performances for such tasks, the nonuniform distribution of word senses during training generally results in systems performing poorly on rare senses.To this end, we consider data augmentation to increase the frequency of these least frequent senses (LFS) to reduce the distributional bias of senses during training.We propose Sense-Maintained Sentence Mixup (SMSMix), a novel word-level mixup method that maintains the sense of a target word.SMSMix smoothly blends two sentences using mask prediction while preserving the relevant span determined by saliency scores to maintain a specific word's sense.To the best of our knowledge, this is the first attempt to apply mixup in NLP while preserving the meaning of a specific word.With extensive experiments, we validate that our augmentation method can effectively give more information about rare senses during training with maintained target sense label.
Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems

Large transformer models can highly improve Answer Sentence Selection (AS2) tasks, but their high computational costs prevent their use in many real-world applications.In this paper, we explore the following research question: How can we make the AS2 models more accurate without significantly increasing their model complexity?To address the question, we propose a Multiple Heads Student architecture (named CERBERUS), an efficient neural network designed to distill an ensemble of large transformers into a single smaller model.CER-BERUS consists of two components: a stack of transformer layers that is used to encode inputs, and a set of ranking heads; unlike traditional distillation technique, each of them is trained by distilling a different large transformer architecture in a way that preserves the diversity of the ensemble members.The resulting model captures the knowledge of heterogeneous transformer models by using just a few extra parameters.We show the effectiveness of CERBERUS on three English datasets for AS2; our proposed approach outperforms all single-model distillations we consider, rivaling the state-of-the-art large AS2 models that have 2.7× more parameters and run 2.5× slower.Code for our model is available at https:
EnDex: Evaluation of Dialogue Engagingness at Scale

We propose ENDEX, the first human-reaction based model to evaluate dialogue engagingness.ENDEX is trained on 80k Reddit-based Engagement Dataset (RED) curated using a novel distant-supervision framework.Engagingness is a key measure that captures high-level quality of AI dialogue systems and closely reflects actual user experience.However, data shortage, plus the abstract and extensive definition of engagingness makes it challenging to develop an automatic metric.Our work departs from mainstream approaches that use synthetic negative examples to train binary classifiers, and instead, proposes a solution using distant-supervision from human-reaction feedback.To support the soundness of our ENDEX metric, we offer a theoretical foundation for engagement, an extensive ablation study, and empirical evidence of high correlation on five engagingness related datasets. 1
M3: Multi-level dataset for Multi-document summarisation of Medical studies

We present M3 (Multi-level dataset for Multidocument summarisation of Medical studies), a benchmark dataset for evaluating the quality of summarisation systems in the biomedical domain.The dataset contains sets of multiple input documents and target summaries of three levels of complexity: documents, sentences, and propositions.The dataset also includes several levels of annotation, including biomedical entities, direction, and strength of relations between them, and the discourse relationships between the input documents ("contradiction" or "agreement").We showcase usage scenarios of the dataset by testing 10 generic and domainspecific summarisation models in a zero-shot setting, and introduce a probing task based on counterfactuals to test if models are aware of the direction and strength of the conclusions generated from input studies.
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation

The full power of human language-based communication cannot be realized without negation.All human languages have some form of negation.Despite this, negation remains a challenging phenomenon for current natural language understanding systems.To facilitate the future development of models that can process negation effectively, we present CONDAQA, the first English reading comprehension dataset which requires reasoning about the implications of negated statements in paragraphs.We collect paragraphs with diverse negation cues, then have crowdworkers ask questions about the implications of the negated statement in the passage.We also have workers make three kinds of edits to the passage-paraphrasing the negated statement, changing the scope of the negation, and reversing the negation-resulting in clusters of question-answer pairs that are difficult for models to answer with spurious shortcuts.CONDAQA features 14,182 questionanswer pairs with over 200 unique negation cues and is challenging for current state-ofthe-art models.The best performing model on CONDAQA (UNIFIEDQA-V2-3B) achieves only 42% on our consistency metric, well below human performance which is 81%.We release our dataset, along with fully-finetuned, few-shot, and zero-shot evaluations, to facilitate the development of future NLP methods that work on negated language.
MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences

Existing multimodal tasks mostly target at the complete input modality setting, i.e., each modality is either complete or completely missing in both training and test sets.However, the randomly missing situations have still been underexplored.In this paper, we present a novel approach named MM-Align to address the missing-modality inference problem.Concretely, we propose 1) an alignment dynamics learning module based on the theory of optimal transport (OT) for indirect missing data imputation; 2) a denoising training algorithm to simultaneously enhance the imputation results and backbone network performance.Compared with previous methods which devote to reconstructing the missing inputs, MM-Align learns to capture and imitate the alignment dynamics between modality sequences.Results of comprehensive experiments on three datasets covering two multimodal tasks empirically demonstrate that our method can perform more accurate and faster inference and relieve overfitting under various missing conditions.
The Effects of Corpus Choice and Morphosyntax on Multilingual Space Induction

In an effort to study the inductive biases of language models, numerous studies have attempted to use linguistically motivated tasks as a proxy of sorts, wherein performance on these tasks would imply an inductive bias towards a specific linguistic phenomenon.In this study, we attempt to analyse the inductive biases of language models with respect to natural language phenomena in the context of building multilingual embedding spaces.We sample corpora from 2 sources in 15 languages and train language models on pseudo-bilingual variants of each corpus, created by duplicating each corpus and shifting token indices for half the resulting corpus.We evaluate the cross-lingual capabilities of these LMs, and show that while correlations with language families tend to be weak, other corpus-level characteristics, such as type-token ratio, tend to be more strongly correlated.Finally, we show that multilingual spaces can be built, albeit less effectively, even when additional destructive perturbations are applied to the training corpora, implying that (effectively) bag-of-words models also have an inductive bias that is sufficient for inducing multilingual spaces.
Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations

Implicit discourse relation recognition is a challenging task that involves identifying the sense or senses that hold between two adjacent spans of text, in the absense of an explicit connective between them.In both PDTB-2 (Prasad et al., 2008) and PDTB-3 (Webber et al., 2019), discourse relational senses are organized into a three-level hierarchy ranging from four broad top-level senses, to more specific senses below them.Most previous work on implicitf discourse relation recognition have used the sense hierarchy simply to indicate what sense labels were available.Here we do more -incorporating the sense hierarchy into the recognition process itself and using it to select the negative examples used in contrastive learning.With no additional effort, the approach achieves stateof-the-art performance on the task.Our code is released in https://github.com/wanqiulong0923/Contrastive_IDRR.
Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attacks

Natural language processing (NLP) models are known to be vulnerable to backdoor attacks, which poses a newly arisen threat to NLP models.Prior online backdoor defense methods for NLP models only focus on the anomalies at either the input or output level, still suffering from fragility to adaptive attacks and high computational cost.In this work, we take the first step to investigate the unconcealment of textual poisoned samples at the intermediate-feature level and propose a feature-based efficient online defense method.Through extensive experiments on existing attacking methods, we find that the poisoned samples are far away from clean samples in the intermediate feature space of a poisoned NLP model.Motivated by this observation, we devise a distance-based anomaly score (DAN) to distinguish poisoned samples from clean samples at the feature level.Experiments on sentiment analysis and offense detection tasks demonstrate the superiority of DAN, as it substantially surpasses existing online defense methods in terms of defending performance and enjoys lower inference costs.Moreover, we show that DAN is also resistant to adaptive attacks based on feature-level regularization.Our code is available at https: //github.com/lancopku/DAN.
Multi-View Active Learning for Short Text Classification in User-Generated Data

Mining user-generated data often suffers from the lack of enough labeled data, short document lengths, and the informal user language.In this paper, we propose a novel active learning model to overcome these obstacles in the tasks tailored for query phrases-e.g., detecting positive reports of natural disasters.Our model has three novelties: 1) It is the first approach to employ multi-view active learning in this domain.2) It uses the Parzen-Rosenblatt window method to integrate the representativeness measure into multi-view active learning.3) It employs a query-by-committee strategy, based on the agreement between predictors, to address the usually noisy language of the documents in this domain.We evaluate our model in four publicly available Twitter datasets with distinctly different applications.We also compare our model with a wide range of baselines including those with multiple classifiers.The experiments testify that our model is highly consistent and outperforms existing models. 1 2 3 4 5 1 2 3 4 5 (a) Query-Based Tasks 1 2 3 4 5 1 2 3 4 5 Just experienced my first earthquake lol crazy
Calibrating Factual Knowledge in Pretrained Language Models

Previous literature has proved that Pretrained Language Models (PLMs) can store factual knowledge.However, we find that facts stored in the PLMs are not always correct.It motivates us to explore a fundamental question: How do we calibrate factual knowledge in PLMs without re-training from scratch?In this work, we propose a simple and lightweight method CA-LINET to achieve this goal.To be specific, we first detect whether PLMs can learn the right facts via a contrastive score between right and fake facts.If not, we then use a lightweight method to add and adapt new parameters to specific factual texts.Experiments on the knowledge probing task show the calibration effectiveness and efficiency.In addition, through closed-book question answering, we find that the calibrated PLM possesses knowledge generalization ability after fine-tuning.Beyond the calibration performance, we further investigate and visualize the knowledge calibration mechanism.The code and data are available at
Acceptability Judgements via Examining the Topology of Attention Maps

The role of the attention mechanism in encoding linguistic knowledge has received special interest in NLP. However, the ability of the attention heads to judge the grammatical acceptability of a sentence has been underexplored. This paper approaches the paradigm of acceptability judgments with topological data analysis (TDA), showing that the geometric properties of the attention graph can be efficiently exploited for two standard practices in linguistics: binary judgments and linguistic minimal pairs. Topological features enhance the BERT-based acceptability classifier scores by $8$%-$24$% on CoLA in three languages (English, Italian, and Swedish). By revealing the topological discrepancy between attention maps of minimal pairs, we achieve the human-level performance on the BLiMP benchmark, outperforming nine statistical and Transformer LM baselines. At the same time, TDA provides the foundation for analyzing the linguistic functions of attention heads and interpreting the correspondence between the graph features and grammatical phenomena.
ExpertPLM: Pre-training Expert Representation for Expert Finding

Expert Finding is an important task in Community Question Answering (CQA) platforms, which could help route questions to potential users to answer.The key is to learn representations of experts based on their historical answered questions accurately.In this paper, inspired by the strong text understanding ability of Pretrained Language modelings (PLMs), we propose a pre-training and fine-tuning expert finding framework.The core is that we design an expert-level pre-training paradigm, that effectively integrates expert interest and expertise simultaneously.Specifically different from the typical corpus-level pre-training, we treat each expert as the basic pre-training unit including all the historical answered question titles of the expert, which could fully indicate the expert interests for questions.Besides, we integrate the vote score information along with each answer of the expert into the pre-training phrase to model the expert ability explicitly.Finally, we propose a novel reputation-augmented Masked Language Model (MLM) pre-training strategy to capture the expert reputation information.In this way, our method could learn expert representation comprehensively, which then will be adopted and fine-tuned in the down-streaming expert-finding task.Extensive experimental results on six real-world CQA datasets demonstrate the effectiveness of our method.1 https://stackexchange.comQiyao Peng and Hongtao Liu are equal contribution.Qing Yang is the corresponding author.What types of jobs did upper class Romans do?Kublai Khan's army consist largely of Mongol warriors or native Chinese?How were books published in Ancient Greece?What knowledge may have been lost in … Alexandria?In Mongolian Conquests, how did they screen the population for engineers?
Towards Autoformalization of Mathematics and Code Correctness: Experiments with Elementary Proofs

The ever-growing complexity of mathematical proofs makes their manual verification by mathematicians very cognitively demanding.Autoformalization seeks to address this by translating proofs written in natural language into a formal representation that is computer-verifiable via interactive theorem provers.In this paper, we introduce a semantic parsing approach, based on the Universal Transformer architecture, that translates elementary mathematical proofs into an equivalent formalization in the language of the Coq interactive theorem prover.The same architecture is also trained to translate simple imperative code decorated with Hoare triples into formally verifiable proofs of correctness in Coq.Experiments on a limited domain of artificial and human-written proofs show that the models generalize well to intermediate lengths not seen during training and variations in natural language.
Extracting Operator Trees from Model Embeddings

Transformer-based language models are able to capture several linguistic properties such as hierarchical structures like dependency or constituency trees.Whether similar structures for mathematics are extractable from language models has not yet been explored.This work aims to probe current state-of-the-art models for the extractability of Operator Trees from their contextualized embeddings using the structure probe designed by (Hewitt and Manning, 2019).We release the code and our data set for future analyses 1 .
A Localized Geometric Method to Match Knowledge in Low-dimensional Hyperbolic Space

Matching equivalent entities across Knowledge graphs is a pivotal step for knowledge fusion.Previous approaches usually study the problem in Euclidean space.However, recent works have shown that hyperbolic space has a higher capacity than Euclidean space and hyperbolic embedding can represent the hierarchical structure in a knowledge graph.In this paper, we propose a localized geometric method to find equivalent entities in hyperbolic space.Specifically, we use a hyperbolic neural network to encode the lingual information of entities and the structure of both knowledge graphs into a low-dimensional hyperbolic space.To address the asymmetry of structure on different KGs and the localized nature of relations, we learn an instance-specific geometric mapping function based on rotation to match entity pairs.A contrastive loss function is used to train the model.The experiment verifies the power of low-dimensional hyperbolic space for entity matching and shows that our method outperforms the state of the art by a large margin.
Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts

Prompt tuning is a parameter-efficient tuning (PETuning) method for utilizing pre-trained models (PTMs) that simply prepends a soft prompt to the input and only optimizes the prompt to adapt PTMs to downstream tasks.Although it is parameter-and deploymentefficient, its performance still lags behind other state-of-the-art PETuning methods.Besides, the training cost of prompt tuning is not significantly reduced due to the back-propagation through the entire model.Through empirical analyses, we shed some light on the lagging performance of prompt tuning and recognize a trade-off between the propagation distance from label signals to the inserted prompt and the influence of the prompt on model outputs.Further, we present Late Prompt Tuning (LPT) that inserts a late prompt into an intermediate layer of the PTM instead of the input layer or all layers.The late prompt is obtained by a neural prompt generator conditioned on the hidden states before the prompt insertion layer and therefore is instance-dependent.Through extensive experimental results across various tasks and PTMs, we show that LPT can achieve competitive performance to full model tuning and other PETuning methods under both fulldata and few-shot scenarios while possessing faster training speed and lower memory cost.
Conditioned Masked Language and Image Modeling for Image-Text Dense Retrieval

Image-text retrieval is a fundamental crossmodal task that takes image/text as a query to retrieve relevant data of another type.The large-scale two-stream pre-trained models like CLIP have achieved tremendous success in this area.They embed the images and texts into instance representations with two separate encoders, aligning them on the instance-level with contrastive learning.Beyond this, the following works adopt the fine-grained token-level interaction (Masked Language and Image Modeling) to boost performance further.However, the vanilla token-level objectives are not designed to aggregate the image-text alignment information into the instance representations, but the token representations, causing a gap between pre-training and application.To address this issue, we carefully design two novel conditioned token-level pre-training objectives, Conditioned Masked Language and Image Modeling (ConMLM and ConMIM), forcing models to aggregate the token-level alignment information into the instance representations.Combing with the instance-level contrastive learning, we propose our cross-modal dense retrieval framework, Conditioned Language-Image Pretraining (ConLIP).Experimental results on two popular cross-modal retrieval benchmarks (MSCOCO and Flickr30k) reveal the effectiveness of our methods.