Reviewerly/exHarmony huggingface dataset:
Exploring Mode Connectivity for Pre-trained Language Models

Recent years have witnessed the prevalent application of pre-trained language models (PLMs) in NLP.From the perspective of parameter space, PLMs provide generic initialization, starting from which high-performance minima could be found.Although plenty of works have studied how to effectively and efficiently adapt PLMs to high-performance minima, little is known about the connection of various minima reached under different adaptation configurations.In this paper, we investigate the geometric connections of different minima through the lens of mode connectivity, which measures whether two minima can be connected with a low-loss path.We conduct empirical analyses to investigate three questions: (1) how could hyperparameters, specific tuning methods, and training data affect PLM's mode connectivity?(2) How does mode connectivity change during pretraining?(3) How does the PLM's task knowledge change along the path connecting two minima?In general, exploring the mode connectivity of PLMs conduces to understanding the geometric connection of different minima, which may help us fathom the inner workings of PLM downstream adaptation.The codes are publicly available at https://github.com/thunlp/Mode-Connectivity-PLM.
Byte-Level Massively Multilingual Semantic Parsing

Token free approaches have been successfully applied to a series of word and span level tasks.In this work, we compare a bytelevel (ByT5) and a wordpiece based (mT5) sequence to sequence model on the 51 languages of the MASSIVE multilingual semantic parsing dataset.We examine multiple experimental settings: (i) zero-shot, (ii) full gold data and (iii) zero-shot with synthetic data.By leveraging a state-of-the-art label projection method for machine translated examples, we are able to reduce the gap in exact match accuracy to only 5 points with respect to a model trained on gold data from all the languages.We additionally provide insights on the cross-lingual transfer of ByT5 and show how the model compares with respect to mT5 across all parameter sizes.
R-AT: Regularized Adversarial Training for Natural Language Understanding

Currently, adversarial training has become a popular and powerful regularization method in the natural language domain.In this paper, we Regularized Adversarial Training (R-AT) via dropout, which forces the output probability distributions of different sub-models generated by dropout to be consistent under the same adversarial samples.Specifically, we generate adversarial samples by perturbing the word embeddings.For each adversarial sample fed to the model, R-AT minimizes both the adversarial risk and the bidirectional KL-divergence between the adversarial output distributions of two sub-models sampled by dropout.Through extensive experiments on 13 public natural language understanding datasets, we found that R-AT has improvements for many models (e.g., rnn-based, cnn-based, and transformer-based models).For the GLUE benchmark, when R-AT is only applied to the fine-tuning stage, it is able to improve the overall test score of the BERT-base model from 78.3 to 79.6 and the RoBERTa-large model from 88.1 to 88.6.Theoretical analysis reveals that R-AT has potential gradient regularization during the training process.Furthermore, R-AT can reduce the inconsistency between training and testing of models with dropout 1 .
Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems

The rapidly growing market demand for automatic dialogue agents capable of goal-oriented behavior has caused many tech-industry leaders to invest considerable efforts into taskoriented dialog systems.The success of these systems is highly dependent on the accuracy of their intent identification -the process of deducing the goal or meaning of the user's request and mapping it to one of the known intents for further processing.Gaining insights into unrecognized utterances -user requests the systems fail to attribute to a known intent -is therefore a key process in continuous improvement of goal-oriented dialog systems.We present an end-to-end pipeline for processing unrecognized user utterances, deployed in a real-world, commercial task-oriented dialog system, including a specifically-tailored clustering algorithm, a novel approach to cluster representative extraction, and cluster naming.We evaluated the proposed components, demonstrating their benefits in the analysis of unrecognized user requests.
On Neurons Invariant to Sentence Structural Changes in Neural Machine Translation

We present a methodology that explores how sentence structure is reflected in neural representations of machine translation systems.We demonstrate our model-agnostic approach with the Transformer English-German translation model.We analyze neuron-level correlation of activations between paraphrases while discussing the methodology challenges and the need for confound analysis to isolate the effects of shallow cues.We find that similarity between activation patterns can be mostly accounted for by similarity in word choice and sentence length.Following that, we manipulate neuron activations to control the syntactic form of the output.We show this intervention to be somewhat successful, indicating that deep models capture sentence-structure distinctions, despite finding no such indication at the neuron level.To conduct our experiments, we develop a semi-automatic method to generate meaning-preserving minimal pair paraphrases (active-passive voice and adverbial clause-noun phrase) and compile a corpus of such pairs. 1
Towards Improved Distantly Supervised Multilingual Named-Entity Recognition for Tweets

Recent low-resource named-entity recognition (NER) work has shown impressive gains by leveraging a single multilingual model trained using distantly supervised data derived from cross-lingual knowledge bases.In this work, we investigate such approaches by leveraging Wikidata to build large-scale NER datasets of Tweets and propose two orthogonal improvements for low-resource NER in the Twitter social media domain: (1) leveraging domainspecific pre-training on Tweets; and (2) building a model for each language family rather than an all-in-one single multilingual model.For (1), we show that mBERT with Tweet pretraining outperforms the state-of-the-art multilingual transformer-based language model, LaBSE, by a relative increase of 34.6% in F1 when evaluated on Twitter data in a languageagnostic multilingual setting.For (2), we show that learning NER models for language families outperforms a single multilingual model by relative increases of 14.1%, 15.8% and 45.3% in F1 when utilizing mBERT, mBERT with Tweet pre-training and LaBSE, respectively.We conduct analyses and present examples for these observed improvements.
Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method &amp; Data Scarcity

We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages.By training models on sub-sampled datasets in three different languages, we assess the quality of estimates from a wide array of approaches and their dependence on the amount of available data.We find that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates can surprisingly suffer with more data.We also perform a qualitative analysis of uncertainties on sequences, discovering that a model's total uncertainty seems to be influenced to a large degree by its data uncertainty, not model uncertainty.All model implementations are opensourced in a software package. 1 The model zoo is available under https://github.com/Kaleidophon/nlp-uncertainty-zoo, with the code for the experiments available under https://github.com/Kaleidophon/nlp-low-resource-uncertainty.2 That is, unless the model class we chose is too restrictive.
McPhraSy: Multi-Context Phrase Similarity and Clustering

Phrase similarity is a key component of many NLP applications.Current phrase similarity methods focus on embedding the phrase itself and use the phrase context only during training of the pretrained model.To better leverage the information in the context, we propose McPhraSy (Multi-context Phrase Similarity), a novel algorithm for estimating the similarity of phrases based on multiple contexts.At inference time, McPhraSy represents each phrase by considering multiple contexts in which it appears and computes the similarity of two phrases by aggregating the pairwise similarities between the contexts of the phrases.Incorporating context during inference enables McPhraSy to outperform current state-of-theart models on two phrase similarity datasets by up to 13.3%.Finally, we also present a new downstream task that relies on phrase similarity -keyphrase clustering -and create a new benchmark for it in the product reviews domain.We show that McPhraSy surpasses all other baselines for this task.
Identifying Condescending Language: A Tale of Two Distinct Phenomena?

Patronizing and condescending language is characterized, among others, by its subtle nature.It thus seems reasonable to assume that detecting condescending language in text would be harder than detecting more explicitly harmful language, such as hate speech.However, the results of a SemEval-2022 Task devoted to this topic paint a different picture, with the top-performing systems achieving remarkably strong results.In this paper, we analyse the surprising effectiveness of standard text classification methods in more detail.In particular, we highlight the presence of two rather different types of condescending language in the dataset from the SemEval task.Some inputs are condescending because of the way they talk about a particular subject, i.e. condescending language in this case is a linguistic phenomenon, which can, in principle, be learned from training examples.However, other inputs are condescending because of the nature of what is said, rather than the way in which it is expressed, e.g. by emphasizing stereotypes about a given community.In such cases, our ability to detect condescending language, with current methods, largely depends on the presence of similar examples in the training data.
Topic Taxonomy Expansion via Hierarchy-Aware Topic Phrase Generation

Topic taxonomies display hierarchical topic structures of a text corpus and provide topical knowledge to enhance various NLP applications.To dynamically incorporate new topic information, several recent studies have tried to expand (or complete) a topic taxonomy by inserting emerging topics identified in a set of new documents.However, existing methods focus only on frequent terms in documents and the local topic-subtopic relations in a taxonomy, which leads to limited topic term coverage and fails to model the global topic hierarchy.In this work, we propose a novel framework for topic taxonomy expansion, named Topic-Expan, which directly generates topic-related terms belonging to new topics.Specifically, TopicExpan leverages the hierarchical relation structure surrounding a new topic and the textual content of an input document for topic term generation.This approach encourages newly-inserted topics to further cover important but less frequent terms as well as to keep their relation consistency within the taxonomy.Experimental results on two real-world text corpora show that TopicExpan significantly outperforms other baseline methods in terms of the quality of output taxonomies.
Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense

Visual commonsense understanding requires Vision Language (VL) models to not only understand image and text but also crossreference in-between to fully integrate and achieve comprehension of the visual scene described.Recently, various approaches have been developed and have achieved high performance on visual commonsense benchmarks.However, it is unclear whether the models really understand the visual scene and underlying commonsense knowledge due to limited evaluation data resources.To provide an indepth analysis, we present a Multimodal Evaluation (ME) pipeline to automatically generate question-answer pairs to test models' understanding of the visual scene, text, and related knowledge.We then take a step further to show that training with the ME data boosts model's performance in standard VCR evaluation.Lastly, our in-depth analysis and comparison reveal interesting findings: (1) semantically low-level information can assist learning of high-level information but not the opposite;(2) visual information is generally under utilization compared with text.
“Covid vaccine is against Covid but Oxford vaccine is made at Oxford!” Semantic Interpretation of Proper Noun Compounds

Proper noun compounds, e.g., "Covid vaccine", convey information in a succinct manner (a "Covid vaccine" is a "vaccine that immunizes against the Covid disease").These are commonly used in short-form domains, such as news headlines, but are largely ignored in information-seeking applications.To address this limitation, we release a new manually annotated dataset, PRONCI, consisting of 22.5K proper noun compounds along with their freeform semantic interpretations.PRONCI is 60 times larger than prior noun compound datasets and also includes non-compositional examples, which have not been previously explored.We experiment with various neural models for automatically generating the semantic interpretations from proper noun compounds, ranging from few-shot prompting to supervised learning, with varying degrees of knowledge about the constituent nouns.We find that adding targeted knowledge, particularly about the common noun, results in performance gains of upto 2.8%.Finally, we integrate our model generated interpretations with an existing Open IE system and observe an 7.5% increase in yield at a precision of 85%.The dataset and code are available at https://github.com/dair-iitd/pronci.
Adapting Multilingual Models for Code-Mixed Translation

The scarcity of gold standard code-mixed to pure language parallel data makes it difficult to train translation models reliably.Prior work has addressed the paucity of parallel data with data augmentation techniques.Such methods rely heavily on external resources making systems difficult to train and scale effectively for multiple languages.We present a simple yet highly effective two-stage back-translation based training scheme for adapting multilingual models to the task of code-mixed translation which eliminates dependence on external resources.We show a substantial improvement in translation quality (measured through BLEU), beating existing prior work by up to +3.8 BLEU on code-mixed Hi→En, Mr→En, and Bn→En tasks.On the LinCE Machine Translation leader board, we achieve the highest score for code-mixed Es→En, beating existing best baseline by +6.5 BLEU, and our own stronger baseline by +1.1 BLEU.
CycleKQR: Unsupervised Bidirectional Keyword-Question Rewriting

Users expect their queries to be answered by search systems, regardless of the query's surface form, which include keyword queries and natural questions.Natural Language Understanding (NLU) components of Search and QA systems may fail to correctly interpret semantically equivalent inputs if this deviates from how the system was trained, leading to suboptimal understanding capabilities.We propose the keyword-question rewriting task to improve query understanding capabilities of NLU systems for all surface forms.To achieve this, we present CycleKQR, an unsupervised approach, enabling effective rewriting between keyword and question queries using non-parallel data.Empirically we show the impact on QA performance of unfamiliar query forms for open domain and Knowledge Base QA systems (trained on either keywords or natural language questions).We demonstrate how CycleKQR significantly improves QA performance by rewriting queries into the appropriate form, while at the same time retaining the original semantic meaning of input queries, allowing CycleKQR to improve performance by up to 3% over supervised baselines.Finally, we release a dataset of 66k keyword-question pairs. 1 1 https://github.com/amzn/kqrHow much is iPhone 13?What is the price of iPhone 13? How much does iPhone 13 cost?iPhone 13 cost, cost of iPhone 13 iPhone 13 price, price of iPhone 13 iPhone 13 offer, iPhone 13deal
Natural Language Deduction with Incomplete Information

A growing body of work studies how to answer a question or verify a claim by generating a natural language "proof": a chain of deductive inferences yielding the answer based on a set of premises.However, these methods can only make sound deductions when they follow from evidence that is given.We propose a new system that can handle the underspecified setting where not all premises are stated at the outset; that is, additional assumptions need to be materialized to prove a claim.By using a natural language generation model to abductively infer a premise given another premise and a conclusion, we can impute missing pieces of evidence needed for the conclusion to be true.Our system searches over two fringes in a bidirectional fashion, interleaving deductive (forward-chaining) and abductive (backwardchaining) generation steps.We sample multiple possible outputs for each step to achieve coverage of the search space, at the same time ensuring correctness by filtering lowquality generations with a round-trip validation procedure.Results on a modified version of the EntailmentBank dataset and a new dataset called Everyday Norms: Why Not? show that abductive generation with validation can recover premises across in-and out-of-domain settings. 11 Code and data publicly available at https: //github.com/Zayne-sprague/Natural_Language_Deduction_with_Incomplete_Information.git An animal requires oxygen to breathe.A human is a kind of animal Astronauts are human. An astronaut is a kind of animal.An astronaut requires oxygen to breathe. Animals require oxygen to breathe.
Collateral facilitation in humans and language models

Are the predictions of humans and language models affected by similar things?Research suggests that while comprehending language, humans make predictions about upcoming words, with more predictable words being processed more easily.However, evidence also shows that humans display a similar processing advantage for highly anomalous words when these words are semantically related to the preceding context or to the most probable continuation.Using stimuli from 3 psycholinguistic experiments, we find that this is also almost always also the case for 8 contemporary transformer language models (BERT, ALBERT, RoBERTa, XLM-R, GPT-2, GPT-Neo, GPT-J, and XGLM).We then discuss the implications of this phenomenon for our understanding of both human language comprehension and the predictions made by language models.
Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances

The primary purpose of dialogue state tracking (DST), a critical component of an end-toend conversational system, is to build a model that responds well to real-world situations.Although we often change our minds from time to time during ordinary conversations, current benchmark datasets do not adequately reflect such occurrences and instead consist of over-simplified conversations, in which no one changes their mind during a conversation.As the main question inspiring the present study, "Are current benchmark datasets sufficiently diverse to handle casual conversations in which one changes their mind after a certain topic is over?"We found that the answer is "No" because DST models cannot refer to previous user preferences when template-based turnback utterances are injected into the dataset.Even in the the simplest mind-changing (turnback) scenario, the performance of DST models significantly degenerated.However, we found that this performance degeneration can be recovered when the turnback scenarios are explicitly designed in the training set, implying that the problem is not with the DST models but rather with the construction of the benchmark dataset.
MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure

In this paper, we propose a comprehensive benchmark to investigate models' logical reasoning capabilities in complex real-life scenarios.Current explanation datasets often employ synthetic data with simple reasoning structures.Therefore, it cannot express more complex reasoning processes, such as the rebuttal to a reasoning step and the degree of certainty of the evidence.To this end, we propose a comprehensive logical reasoning explanation form.Based on the multi-hop chain of reasoning, the explanation form includes three main components:(1) The condition of rebuttal that the reasoning node can be challenged; (2) Logical formulae that uncover the internal texture of reasoning nodes; (3) Reasoning strength indicated by degrees of certainty.The fine-grained structure conforms to the real logical reasoning scenario, better fitting the human cognitive process but, simultaneously, is more challenging for the current models.We evaluate the current best models' performance on this new explanation form.The experimental results show that generating reasoning graphs remains a challenging task for current models, even with the help of giant pre-trained language models.
Improving abstractive summarization with energy-based re-ranking

Current abstractive summarization systems present important weaknesses which prevent their deployment in real-world applications, such as the omission of relevant information and the generation of factual inconsistencies (also known as hallucinations).At the same time, automatic evaluation metrics such as CTC scores (Deng et al., 2021) have been recently proposed that exhibit a higher correlation with human judgments than traditional lexicaloverlap metrics such as ROUGE.In this work, we intend to close the loop by leveraging the recent advances in summarization metrics to create quality-aware abstractive summarizers.Namely, we propose an energy-based model that learns to re-rank summaries according to one or a combination of these metrics.We experiment using several metrics to train our energy-based re-ranker and show that it consistently improves the scores achieved by the predicted summaries.Nonetheless, human evaluation results show that the re-ranking approach should be used with care for highly abstractive summaries, as the available metrics are not yet sufficiently reliable for this purpose.
Generalizing over Long Tail Concepts for Medical Term Normalization

Medical term normalization consists in mapping a piece of text to a large number of output classes.Given the small size of the annotated datasets and the extremely long tail distribution of the concepts, it is of utmost importance to develop models that are capable to generalize to scarce or unseen concepts.An important attribute of most target ontologies is their hierarchical structure.In this paper we introduce a simple and effective learning strategy that leverages such information to enhance the generalizability of both discriminative and generative models.The evaluation shows that the proposed strategy produces state-of-the-art performance on seen concepts and consistent improvements on unseen ones, allowing also for efficient zero-shot knowledge transfer across text typologies and datasets.