Reviewerly/exHarmony huggingface dataset:
Don’t Copy the Teacher: Data and Model Challenges in Embodied Dialogue

Embodied dialogue instruction following requires an agent to complete a complex sequence of tasks from a natural language exchange.The recent introduction of benchmarks (Padmakumar et al., 2022) raises the question of how best to train and evaluate models for this multi-turn, multi-agent, long-horizon task.This paper contributes to that conversation, by arguing that imitation learning (IL) and related low-level metrics are actually misleading and do not align with the goals of embodied dialogue research and may hinder progress.We provide empirical comparisons of metrics, analysis of three models, and make suggestions for how the field might best progress.First, we observe that models trained with IL take spurious actions during evaluation.Second, we find that existing models fail to ground query utterances, which are essential for task completion.Third, we argue evaluation should focus on higher-level semantic goals. 1
Long Text Generation with Topic-aware Discrete Latent Variable Model

Generating coherent long texts is an important yet challenging task, particularly for the openended generation task.Prior work based on discrete latent codes focuses on the modeling of discourse relation, resulting in discrete codes only learning shallow semantics (Ji and Huang, 2021).A natural text always revolves around several related topics and the transition across them is natural and smooth.In this work, we investigate whether discrete latent codes can learn information of topics.To this end, we build a topic-aware latent code-guided text generation model.To encourage discrete codes to model information about topics, we propose a span-level bag-of-words training objective for the model.Automatic and manual evaluation experiments show that our method can generate more topic-relevant and coherent texts.
MatchPrompt: Prompt-based Open Relation Extraction with Semantic Consistency Guided Clustering

Relation clustering is a general approach for open relation extraction (OpenRE).Current methods have two major problems.One is that their good performance relies on large amounts of labeled and pre-defined relational instances for pre-training, which are costly to acquire in reality.The other is that they only focus on learning a high-dimensional metric space to measure the similarity of novel relations and ignore the specific relational representations of clusters.In this work, we propose a new prompt-based framework named Match-Prompt, which can realize OpenRE with efficient knowledge transfer from only a few predefined relational instances as well as mine the specific meanings for cluster interpretability.To our best knowledge, we are the first to introduce a prompt-based framework for unlabeled clustering.Experimental results on different datasets show that MatchPrompt achieves the new SOTA results for OpenRE.
Interventional Training for Out-Of-Distribution Natural Language Understanding

Out-of-distribution (OOD) settings are used to measure a model's performance when the distribution of the test data is different from that of the training data.NLU models are known to suffer in OOD settings (Utama et al., 2020b).We study this issue from the perspective of causality, which sees confounding bias as the reason for models to learn spurious correlations.While a common solution is to perform intervention, existing methods handle only known and single confounder (Pearl and Mackenzie, 2018), but in many NLU tasks the confounders can be both unknown and multifactorial.In this paper, we propose a novel interventional training method called Bottom-up Automatic Intervention (BAI) that performs multi-granular intervention with identified multifactorial confounders.Our experiments on three NLU tasks, namely, natural language inference, fact verification and paraphrase identification, show the effectiveness of BAI for tackling different OOD settings.
Zero-shot Cross-Language Transfer of Monolingual Entity Linking Models

Most entity linking systems, whether mono or multilingual, link mentions to a single English knowledge base.Few have considered linking non-English text to a non-English KB, and therefore, transferring an English entity linking model to both a new document and KB language.We consider the task of zero-shot crosslanguage transfer of entity linking systems to a new language and KB.We find that a system trained with multilingual representations does reasonably well, and propose improvements to system training that lead to improved recall in most datasets, often matching the in-language performance.We further conduct a detailed evaluation to elucidate the challenges of this setting.
Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation

Overconfidence has been shown to impair generalization and calibration of a neural network.Previous studies remedy this issue by adding a regularization term to a loss function, preventing a model from making a peaked distribution.Label smoothing smoothes target labels with a pre-defined prior label distribution; as a result, a model is learned to maximize the likelihood of predicting the soft label.Nonetheless, the amount of smoothing is the same in all samples and remains fixed in training.In other words, label smoothing does not reflect the change in probability distribution mapped by a model over the course of training.To address this issue, we propose a regularization scheme that brings dynamic nature into the smoothing parameter by taking model probability distribution into account, thereby varying the parameter per instance.A model in training self-regulates the extent of smoothing on the fly during forward propagation.Furthermore, inspired by recent work in bridging label smoothing and knowledge distillation, our work utilizes self-knowledge as a prior label distribution in softening target labels, and presents theoretical support for the regularization effect by knowledge distillation and the dynamic smoothing parameter.Our regularizer is validated comprehensively, and the result illustrates marked improvements in model generalization and calibration, enhancing robustness and trustworthiness of a model.
Chaining Simultaneous Thoughts for Numerical Reasoning

Given that rich information is hidden behind ubiquitous numbers in text, numerical reasoning over text should be an essential skill of AI systems.To derive precise equations to solve numerical reasoning problems, previous work focused on modeling the structures of equations, and has proposed various structured decoders.Though structure modeling proves to be effective, these structured decoders construct a single equation in a pre-defined autoregressive order, potentially placing an unnecessary restriction on how a model should grasp the reasoning process.Intuitively, humans may have numerous pieces of thoughts popping up in no pre-defined order; thoughts are not limited to the problem at hand, and can even be concerned with other related problems.By comparing diverse thoughts and chaining relevant pieces, humans are less prone to errors.In this paper, we take this inspiration and propose CAN-TOR, a numerical reasoner that models reasoning steps using a directed acyclic graph where we produce diverse reasoning steps simultaneously without pre-defined decoding dependencies, and compare and chain relevant ones to reach a solution.Extensive experiments demonstrated the effectiveness of CANTOR under both fully-supervised and weakly-supervised settings.
Bringing the State-of-the-Art to Customers: A Neural Agent Assistant Framework for Customer Service Support

Building Agent Assistants that can help improve customer service support requires inputs from industry users and their customers, as well as knowledge about state-of-the-art Natural Language Processing (NLP) technology.We combine expertise from academia and industry to bridge the gap and build task/domainspecific Neural Agent Assistants (NAA) with three high-level components for: (1) Intent Identification, (2) Context Retrieval, and (3) Response Generation.In this paper, we outline the pipeline of the NAA's core system and also present three case studies in which three industry partners successfully adapt the framework to find solutions to their unique challenges.Our findings suggest that a collaborative process is instrumental in spurring the development of emerging NLP models for Conversational AI tasks in industry.The full reference implementation code and results are available at https:
Self-Distillation with Meta Learning for Knowledge Graph Completion

In this paper, we propose a self-distillation framework with meta learning (MetaSD) for knowledge graph completion with dynamic pruning, which aims to learn compressed graph embeddings and tackle the long-tail samples.Specifically, we first propose a dynamic pruning technique to obtain a small pruned model from a large source model, where the pruning mask of the pruned model could be updated adaptively per epoch after the model weights are updated.The pruned model is supposed to be more sensitive to difficult-to-memorize samples (e.g., long-tail samples) than the source model.Then, we propose a one-step meta selfdistillation method for distilling comprehensive knowledge from the source model to the pruned model, where the two models co-evolve in a dynamic manner during training.In particular, we exploit the performance of the pruned model, which is trained alongside the source model in one iteration, to improve the source model's knowledge transfer ability for the next iteration via meta learning.Extensive experiments show that MetaSD achieves competitive performance compared to strong baselines, while being 10x smaller than baselines.
StuBot: Learning by Teaching a Conversational Agent Through Machine Reading Comprehension

This paper proposes StuBot, a text-based conversational agent that provides adaptive feedback for learning by teaching.StuBot first asks the users to teach the learning content by summarizing and explaining it in their own words.After the users inputted the explanation text for teaching, StuBot uses a machine reading comprehension (MRC) engine to provide adaptive feedback with further questions about the insufficient parts of the explanation text.We conducted a within-subject study to evaluate the effectiveness of adaptive feedback by StuBot.Both the quantitative and qualitative results showed that learning by teaching with adaptive feedback can improve learning performance, immersion, and overall experience.
Don’t Just Clean It, Proxy Clean It: Mitigating Bias by Proxy in Pre-Trained Models

Transformer-based pre-trained models are known to encode societal biases, not only in their contextual representations but also in their downstream predictions when fine-tuned on task-specific data.We present D-BIAS, an approach that selectively eliminates stereotypical associations (e.g, co-occurrence statistics) at fine-tuning, such that the model doesn't learn to excessively rely on those signals.D-BIAS attenuates biases from both identity words and frequently co-occurring proxies, which we select using pointwise mutual information.We apply D-BIAS to a) occupation classification, and b) toxicity classification and find that our approach substantially reduces downstream biases (> 60% in toxicity classification for identities that are most frequently flagged as toxic on online platforms).In addition, we show that D-BIAS dramatically improves upon scrubbing, i.e., removing only the identity words in question.We also demonstrate that D-BIAS easily extends to multiple identities and achieves competitive performance with two recently proposed debiasing approaches: R-LACE and INLP.
Multiple Instance Learning for Offensive Language Detection

Automatic offensive language detection has become a crucial issue in recent years. Existing researches on this topic are usually based on a large amount of data annotated at sentence level to train a robust model. However, sentence-level annotations are expensive in practice as the scenario expands, while there exist a large amount of natural labels from historical information on online platforms such as reports and punishments. Notably, these natural labels are usually in bag-level corresponding to the whole documents (articles, user profiles, conversations, etc.). Therefore, we target at proposing an approach capable of utilizing the bag-level labeled data for offensive language detection in this study. For this purpose, we formalize this task into a multiple instance learning (MIL) problem. We break down the design of existing MIL methods and propose a hybrid fusion MIL model with mutual-attention mechanism. In order to verify the validity of the proposed method, we present two new bag-level labeled datasets for offensive language detection: OLID-bags and MINOR. Experimental results based on the proposed datasets demonstrate the effectiveness of the mutual-attention method at both sentence level and bag level.
PRINCE: Prefix-Masked Decoding for Knowledge Enhanced Sequence-to-Sequence Pre-Training

Pre-trained Language Models (PLMs) have shown effectiveness in various Natural Language Processing (NLP) tasks. Denoising autoencoder is one of the most successful pre-training frameworks, learning to recompose the original text given a noise-corrupted one. The existing studies mainly focus on injecting noises into the input. This paper introduces a simple yet effective pre-training paradigm, equipped with a knowledge-enhanced decoder that predicts the next entity token with noises in the prefix, explicitly strengthening the representation learning of entities that span over multiple input tokens. Specifically, when predicting the next token within an entity, we feed masks into the prefix in place of some of the previous ground-truth tokens that constitute the entity. Our model achieves new state-of-the-art results on two knowledge-driven data-to-text generation tasks with up to 2% BLEU gains.
AstBERT: Enabling Language Model for Financial Code Understanding with Abstract Syntax Trees

Using the pre-trained language models to understand source codes has attracted increasing attention from financial institutions owing to the great potential to uncover financial risks.However, there are several challenges in applying these language models to solve programming language related problems directly.For instance, the shift of domain knowledge between natural language (NL) and programming language (PL) requires understanding the semantic and syntactic information from the data from different perspectives.To this end, we propose the AstBERT model, a pre-trained PL model aiming to better understand the financial codes using the abstract syntax tree (AST).Specifically, we collect a sheer number of source codes (both Java and Python) from the Alipay code repository and incorporate both syntactic and semantic code knowledge into our model through the help of code parsers, in which AST information of the source codes can be interpreted and integrated.We evaluate the performance of the proposed model on three tasks, including code question answering, code clone detection and code refinement.Experiment results show that our AstBERT achieves promising performance on three different downstream tasks.
Predicting Long-Term Citations from Short-Term Linguistic Influence

A standard measure of the influence of a research paper is the number of times it is cited.However, papers may be cited for many reasons, and citation count offers limited information about the extent to which a paper affected the content of subsequent publications.We therefore propose a novel method to quantify linguistic influence in timestamped document collections.There are two main steps: first, identify lexical and semantic changes using contextual embeddings and word frequencies; second, aggregate information about these changes into per-document influence scores by estimating a high-dimensional Hawkes process with a low-rank parameter matrix.We show that this measure of linguistic influence is predictive of future citations: the estimate of linguistic influence from the two years after a paper's publication is correlated with and predictive of its citation count in the following three years.This is demonstrated using an online evaluation with incremental temporal training/test splits, in comparison with a strong baseline that includes predictors for initial citation counts, topics, and lexical features.
Consecutive Question Generation via Dynamic Multitask Learning

In this paper, we propose the task of consecutive question generation (CQG), which generates a set of logically related question-answer pairs to understand a whole passage, with a comprehensive consideration of the aspects including accuracy, coverage, and informativeness.To achieve this, we first examine the four key elements of CQG, i.e., question, answer, rationale 1 , and context history 2 , and propose a novel dynamic multitask framework with one main task generating a question-answer pair, and four auxiliary tasks generating other elements.It directly helps the model generate good questions through both joint training and self-reranking.At the same time, to fully explore the worth-asking information in a given passage, we make use of the reranking losses to sample the rationales and search for the best question series globally.Finally, we measure our strategy by QA data augmentation and manual evaluation, as well as a novel application of generated question-answer pairs on DocNLI.We prove that our strategy can improve question generation significantly and benefit multiple related NLP tasks.S: [Once upon a time in Greece, there lived a young man called Narcissus.]stc 1 [He lived in a small village on the sea and was famous in the land because he was quite handsome.]stc 2 ...
Using Roark-Hollingshead Distance to Probe BERT’s Syntactic Competence

Probing BERT's general ability to reason about syntax is no simple endeavour, primarily because of the uncertainty surrounding how large language models represent syntactic structure.Many prior accounts of BERT's agility as a syntactic tool (Clark et al.,
TCS WITM 2022@FinSim4-ESG: Augmenting BERT with Linguistic and Semantic features for ESG data classification

Advanced neural network architectures have provided several opportunities to develop systems to automatically capture information from domain-specific unstructured text sources.The FinSim4-ESG shared task, collocated with the FinNLP workshop, proposed two sub-tasks.In sub-task1, the challenge was to design systems that could utilize contextual word embeddings along with sustainability resources to elaborate an ESG taxonomy.In the second subtask, participants were asked to design a system that could classify sentences into sustainable or unsustainable sentences.In this paper, we utilize semantic similarity features along with BERT embeddings to segregate domain terms into a fixed number of class labels.The proposed model not only considers the contextual BERT embeddings but also incorporates Word2Vec, cosine, and Jaccard similarity which gives word-level importance to the model.For sentence classification, several linguistic elements along with BERT embeddings were used as classification features.We have shown a detailed ablation study for the proposed models.
Identifying Human Strategies for Generating Word-Level Adversarial Examples

Adversarial examples in NLP are receiving increasing research attention.One line of investigation is the generation of word-level adversarial examples against fine-tuned Transformer models that preserve naturalness and grammaticality.Previous work found that human-and machine-generated adversarial examples are comparable in their naturalness and grammatical correctness.Most notably, humans were able to generate adversarial examples much more effortlessly than automated attacks.In this paper, we provide a detailed analysis of exactly how humans create these adversarial examples.By exploring the behavioural patterns of human workers during the generation process, we identify statistically significant tendencies based on which words humans prefer to select for adversarial replacement (e.g., word frequencies, word saliencies, sentiment) as well as where and when words are replaced in an input sequence.With our findings, we seek to inspire efforts that harness human strategies for more robust NLP models.
Summarizing Community-based Question-Answer Pairs

Community-based Question Answering (CQA), which allows users to acquire their desired information, has increasingly become an essential component of online services in various domains such as E-commerce, travel, and dining.However, an overwhelming number of CQA pairs makes it difficult for users without particular intent to find useful information spread over CQA pairs.To help users quickly digest the key information, we propose the novel CQA summarization task that aims to create a concise summary from CQA pairs.To this end, we first design a multi-stage data annotation process and create a benchmark dataset, CO-QASUM, based on the Amazon QA corpus.We then compare a collection of extractive and abstractive summarization methods and establish a strong baseline approach DedupLED for the CQA summarization task.Our experiment further confirms two key challenges, sentencetype transfer and deduplication removal, towards the CQA summarization task.Our data and code are publicly available. 1* Work done while at Megagon Labs. 1 https://github.com/megagonlabs/qa-summarization Q: Is this actually a rigid board or more of a floppy mat?A: It is rigid.themain board is rigid,the two sides are semi.Q: Is this actually a rigid board or more of a floppy mat?A: The main area is very sturdy.Then there are two work area pads that are more flexible so when moving those I keep two hands on them.
Open-domain Video Commentary Generation

Live commentary plays an important role in sports broadcasts and video games, making spectators more excited and immersed.In this context, though approaches for automatically generating such commentary have been proposed in the past, they have been generally concerned with specific fields, where it is possible to leverage domain-specific information.In light of this, we propose the task of generating video commentary in an open-domain fashion.We detail the construction of a new large-scale dataset of transcribed commentary aligned with videos containing various human actions in a variety of domains, and propose approaches based on well-known neural architectures to tackle the task.To understand the strengths and limitations of current approaches, we present an in-depth empirical study based on our data.Our results suggest clear trade-offs between textual and visual inputs for the models and highlight the importance of relying on external knowledge in this open-domain setting, resulting in a set of robust baselines for our task.
Most NLG is Low-Resource: here’s what we can do about it

Many domains and tasks in natural language generation (NLG) are inherently 'lowresource', where training data, tools and linguistic analyses are scarce.This poses a particular challenge to researchers and system developers in the era of machine-learning-driven NLG.In this position paper, we initially present the challenges researchers & developers often encounter when dealing with low-resource settings in NLG.We then argue that it is unsustainable to collect large aligned datasets or build large language models from scratch for every possible domain due to cost, labour, and time constraints, so researching and developing methods and resources for low-resource settings is vital.We then discuss current approaches to low-resource NLG, followed by proposed solutions and promising avenues for future work in NLG for low-resource settings.