Reviewerly/exHarmony huggingface dataset:
Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions

This paper considers continual learning of large-scale pretrained neural machine translation model without accessing the previous training data or introducing model separation.We argue that the widely used regularization-based methods, which perform multi-objective learning with an auxiliary loss, suffer from the misestimate problem and cannot always achieve a good balance between the previous and new tasks.To solve the problem, we propose a twostage training method based on the local features of the real loss.We first search low forgetting risk regions, where the model can retain the performance on the previous task as the parameters are updated, to avoid the catastrophic forgetting problem.Then we can continually train the model within this region only with the new training data to fit the new task.Specifically, we propose two methods to search the low forgetting risk regions, which are based on the curvature of loss and the impacts of the parameters on the model output, respectively.We conduct experiments on domain adaptation and more challenging language adaptation tasks, and the experimental results show that our method can achieve significant improvements compared with several strong baselines.
Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning

Prefix-tuning, or more generally continuous prompt tuning, has become an essential paradigm of parameter-efficient transfer learning.Using a large pre-trained language model (PLM), prefix-tuning can obtain strong performance by training only a small portion of parameters.In this paper, we propose to understand and further develop prefix-tuning through the kernel lens.Specifically, we make an analogy between prefixes and inducing variables in kernel methods and hypothesize that prefixes serving as inducing variables would improve their overall mechanism.From the kernel estimator perspective, we suggest a new variant of prefix-tuning-inducer-tuning, which shares the exact mechanism as prefix-tuning while leveraging the residual form found in adaptertuning.This mitigates the initialization issue in prefix-tuning.Through comprehensive empirical experiments on natural language understanding and generation tasks, we demonstrate that inducer-tuning can close the performance gap between prefix-tuning and fine-tuning.
Semantic-aware Contrastive Learning for More Accurate Semantic Parsing

Since the meaning representations are detailed and accurate annotations which express finegrained sequence-level semtantics, it is usually hard to train discriminative semantic parsers via Maximum Likelihood Estimation (MLE) in an autoregressive fashion.In this paper, we propose a semantic-aware contrastive learning algorithm, which can learn to distinguish fine-grained meaning representations and take the overall sequence-level semantic into consideration.Specifically, a multilevel online sampling algorithm is proposed to sample confusing and diverse instances.Three semantic-aware similarity functions are designed to accurately measure the distance between meaning representations as a whole.And a ranked contrastive loss is proposed to pull the representations of the semanticidentical instances together and push negative instances away.Experiments on two standard datasets show that our approach achieves significant improvements over MLE baselines and gets state-of-the-art performances by simply applying semantic-aware contrastive learning on a vanilla SEQ2SEQ model.
A Speaker-Aware Co-Attention Framework for Medical Dialogue Information Extraction

With the development of medical digitization, the extraction and structuring of Electronic Medical Records (EMRs) have become challenging but fundamental tasks.How to accurately and automatically extract structured information from medical dialogues is especially difficult because the information needs to be inferred from complex interactions between the doctor and the patient.To this end, in this paper, we propose a speaker-aware coattention framework for medical dialogue information extraction.To better utilize the pretrained language representation model to perceive the semantics of the utterance and the candidate item, we develop a speaker-aware dialogue encoder with multi-task learning, which considers the speaker's identity into account.To deal with complex interactions between different utterances and the correlations between utterances and candidate items, we propose a co-attention fusion network to aggregate the utterance information.We evaluate our framework on the public medical dialogue extraction datasets to demonstrate the superiority of our method, which can outperform the state-of-theart methods by a large margin.
Kernel-Whitening: Overcome Dataset Bias with Isotropic Sentence Embedding

Dataset bias has attracted increasing attention recently for its detrimental effect on the generalization ability of fine-tuned models.The current mainstream solution is designing an additional shallow model to pre-identify biased instances.However, such two-stage methods scale up the computational complexity of training process and obstruct valid feature information while mitigating bias.To address this issue, we utilize the representation normalization method which aims at disentangling the correlations between features of encoded sentences.We find it also promising in eliminating the bias problem by providing isotropic data distribution.We further propose Kernel-Whitening, a Nyström kernel approximation method to achieve more thorough debiasing on nonlinear spurious correlations.Our framework is end-to-end with similar time consumption to fine-tuning.Experiments show that Kernel-Whitening significantly improves the performance of BERT on out-of-distribution datasets while maintaining in-distribution accuracy.
PASTA: Table-Operations Aware Fact Verification via Sentence-Table Cloze Pre-training

Fact verification has attracted a lot of research attention recently, e.g., in journalism, marketing, and policymaking, as misinformation and disinformation online can sway one's opinion and affect one's actions.While fact-checking is a hard task in general, in many cases, false statements can be easily debunked based on analytics over tables with reliable information.Hence, table-based fact verification has recently emerged as an important and growing research area.Yet, progress has been limited due to the lack of datasets that can be used to pre-train language models (LMs) to be aware of common table operations, such as aggregating a column or comparing tuples.To bridge this gap, in this paper we introduce PASTA, a novel state-of-the-art framework for table-based fact verification via pre-training with synthesized sentence-table cloze questions.In particular, we design six types of common sentence-table cloze tasks, including Filter, Aggregation, Superlative, Comparative, Ordinal, and Unique, based on which we synthesize a large corpus consisting of 1.2 million sentence-table pairs from WikiTables.PASTA uses a recent pre-trained LM, DeBERTaV3, and further pretrains it on our corpus.Our experimental results show that PASTA achieves new state-ofthe-art performance on two table-based fact verification benchmarks: TabFact and SEM-TAB-FACTS.In particular, on the complex set of TabFact, which contains multiple operations, PASTA largely outperforms the previous state of the art by 4.7 points (85.6% vs. 80.9%), and the gap between PASTA and human performance on the small TabFact test set is narrowed to just 1.5 points (90.6% vs. 92.1%).
NewsClaims: A New Benchmark for Claim Detection from News with Attribute Knowledge

Claim detection and verification are crucial for news understanding and have emerged as promising technologies for mitigating misinformation and disinformation in the news.However, most existing work has focused on claim sentence analysis while overlooking additional crucial attributes (e.g., the claimer and the main object associated with the claim).In this work, we present NEWSCLAIMS, a new benchmark for attribute-aware claim detection in the news domain.We extend the claim detection problem to include extraction of additional attributes related to each claim and release 889 claims annotated over 143 news articles.NEWSCLAIMS 1 aims to benchmark claim detection systems in emerging scenarios, comprising unseen topics with little or no training data.To this end, we see that zero-shot and prompt-based baselines show promising performance on this benchmark, while still considerably behind human performance.
A Template-based Method for Constrained Neural Machine Translation

Machine translation systems are expected to cope with various types of constraints in many practical scenarios.While neural machine translation (NMT) has achieved strong performance in unconstrained cases, it is non-trivial to impose pre-specified constraints into the translation process of NMT models.Although many approaches have been proposed to address this issue, most existing methods can not satisfy the following three desiderata at the same time: (1) high translation quality, (2) high match accuracy, and (3) low latency.In this work, we propose a template-based method that can yield results with high translation quality and match accuracy and the inference speed of our method is comparable with unconstrained NMT models.Our basic idea is to rearrange the generation of constrained and unconstrained tokens through a template.Our method does not require any changes in the model architecture and the decoding algorithm.Experimental results show that the proposed template-based approach can outperform several representative baselines in both lexically and structurally constrained translation tasks.
EventGraph: Event Extraction as Semantic Graph Parsing

Event extraction involves the detection and extraction of both the event triggers and corresponding event arguments.Existing systems often decompose event extraction into multiple subtasks, without considering their possible interactions.In this paper, we propose Event-Graph, a joint framework for event extraction, which encodes events as graphs.We represent event triggers and arguments as nodes in a semantic graph.Event extraction therefore becomes a graph parsing problem, which provides the following advantages: 1) performing event detection and argument extraction jointly; 2) detecting and extracting multiple events from a piece of text; and 3) capturing the complicated interaction between event arguments and triggers.Experimental results on ACE2005 show that our model is competitive to state-of-the-art systems and has substantially improved the results on argument extraction.Additionally, we create two new datasets from ACE2005 where we keep the entire text spans for event arguments, instead of just the head word(s).Our code and models are released as open-source. 1
DuReader-Retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine

In this paper, we present DuReader retrieval , a large-scale Chinese dataset for passage retrieval.DuReader retrieval contains more than 90K queries and over 8M unique passages from a commercial search engine.To alleviate the shortcomings of other datasets and ensure the quality of our benchmark, we (1) reduce the false negatives in development and test sets by manually annotating results pooled from multiple retrievers, and (2) remove the training queries that are semantically similar to the development and testing queries.Additionally, we provide two outof-domain testing sets for cross-domain evaluation, as well as a set of human translated queries for for cross-lingual retrieval evaluation.The experiments demonstrate that DuReader retrieval is challenging and a number of problems remain unsolved, such as the salient phrase mismatch and the syntactic mismatch between queries and paragraphs.These experiments also show that dense retrievers do not generalize well across domains, and cross-lingual retrieval is essentially challenging.DuReader retrieval is publicly
QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance

Existing metrics for assessing question generation not only require costly human reference but also fail to take into account the input context of generation, rendering the lack of deep understanding of the relevance between the generated questions and input contexts.As a result, they may wrongly penalize a legitimate and reasonable candidate question when it (i) involves complicated reasoning with the context or (ii) can be grounded by multiple evidences in the context.In this paper, we propose QRelScore, a context-aware Relevance evaluation metric for Question Generation.Based on off-the-shelf language models such as BERT and GPT2, QRelScore employs both word-level hierarchical matching and sentence-level prompt-based generation to cope with the complicated reasoning and diverse generation from multiple evidences, respectively.Compared with existing metrics, our experiments demonstrate that QRelScore is able to achieve a higher correlation with human judgments while being much more robust to adversarial samples.
Title2Event: Benchmarking Open Event Extraction with a Large-scale Chinese Title Dataset

Event extraction (EE) is crucial to downstream tasks such as new aggregation and event knowledge graph construction.Most existing EE datasets manually define fixed event types and design specific schema for each of them, failing to cover diverse events emerging from the online text.Moreover, news titles, an important source of event mentions, have not gained enough attention in current EE research.In this paper, We present Title2Event, a large-scale sentence-level dataset benchmarking Open Event Extraction without restricting event types.Title2Event contains more than 42,000 news titles in 34 topics collected from Chinese web pages.To the best of our knowledge, it is currently the largest manuallyannotated Chinese dataset for open event extraction.We further conduct experiments on Ti-tle2Event with different models and show that the characteristics of titles make it challenging for event extraction, addressing the significance of advanced study on this problem.The dataset and baseline codes are available at https:// open-event-hub.github.io/title2event.
Disentangling Uncertainty in Machine Translation Evaluation

Trainable evaluation metrics for machine translation (MT) exhibit strong correlation with human judgements, but they are often hard to interpret and might produce unreliable scores under noisy or out-of-domain data.Recent work has attempted to mitigate this with simple uncertainty quantification techniques (Monte Carlo dropout and deep ensembles), however these techniques (as we show) are limited in several ways -for example, they are unable to distinguish between different kinds of uncertainty, and they are time and memory consuming.In this paper, we propose more powerful and efficient uncertainty predictors for MT evaluation, and we assess their ability to target different sources of aleatoric and epistemic uncertainty.To this end, we develop and compare training objectives for the COMET metric to enhance it with an uncertainty prediction output, including heteroscedastic regression, divergence minimization, and direct uncertainty prediction.Our experiments show improved results on uncertainty prediction for the WMT metrics task datasets, with a substantial reduction in computational costs.Moreover, they demonstrate the ability of these predictors to address specific uncertainty causes in MT evaluation, such as low quality references and outof-domain data. 1
Augmenting Multi-Turn Text-to-SQL Datasets with Self-Play

The task of context-dependent text-to-SQL aims to convert multi-turn user utterances to formal SQL queries.This is a challenging task due to both the scarcity of training data from which to learn complex contextual dependencies and to generalize to unseen databases.In this paper we explore augmenting the training datasets using self-play, which leverages contextual information to synthesize new interactions to adapt the model to new databases.We first design a SQL-to-text model conditioned on a sampled goal query, which represents a user's intent, that then converses with a textto-SQL semantic parser to generate new interactions.We then filter the synthesized interactions and retrain the models with the augmented data.We find that self-play improves the accuracy of a strong baseline on SParC and CoSQL, two widely used cross-domain text-to-SQL datasets.Our analysis shows that self-play simulates various conversational thematic relations, enhances cross-domain generalization and improves beam-search.
Exploring The Landscape of Distributional Robustness for Question Answering Models

We conduct a large empirical evaluation to investigate the landscape of distributional robustness in question answering. Our investigation spans over 350 models and 16 question answering datasets, including a diverse set of architectures, model sizes, and adaptation methods (e.g., fine-tuning, adapter tuning, in-context learning, etc.). We find that, in many cases, model variations do not affect robustness and in-distribution performance alone determines out-of-distribution performance.Moreover, our findings indicate thati) zero-shot and in-context learning methods are more robust to distribution shifts than fully fine-tuned models;ii) few-shot prompt fine-tuned models exhibit better robustness than few-shot fine-tuned span prediction models;iii) parameter-efficient and robustness enhancing training methods provide no significant robustness improvements.In addition, we publicly release all evaluations to encourage researchers to further analyze robustness trends for question answering models.
DEER: Descriptive Knowledge Graph for Explaining Entity Relationships

We propose DEER (Descriptive Knowledge Graph for Explaining Entity Relationships)an open and informative form of modeling entity relationships.In DEER, relationships between entities are represented by free-text relation descriptions.For instance, the relationship between entities of machine learning and algorithm can be represented as "Machine learning explores the study and construction of algorithms that can learn from and make predictions on data."To construct DEER, we propose a self-supervised learning method to extract relation descriptions with the analysis of dependency patterns and generate relation descriptions with a transformer-based relation description synthesizing model, where no human labeling is required.Experiments demonstrate that our system can extract and generate highquality relation descriptions for explaining entity relationships.The results suggest that we can build an open and informative knowledge graph without human annotation.
XLM-D: Decorate Cross-lingual Pre-training Model as Non-Autoregressive Neural Machine Translation

Pre-training language models have achieved thriving success in numerous natural language understanding and autoregressive generation tasks, but non-autoregressive generation in applications such as machine translation has not sufficiently benefited from the pre-training paradigm.In this work, we establish the connection between a pre-trained masked language model (MLM) and non-autoregressive generation on machine translation.From this perspective, we present XLM-D, which seamlessly transforms an off-the-shelf cross-lingual pre-training model into a non-autoregressive translation (NAT) model with a lightweight yet effective decorator.Specifically, the decorator ensures the representation consistency of the pre-trained model and brings only one additional trainable parameter.Extensive experiments on typical translation datasets show that our models obtain state-of-the-art performance while realizing the inference speedup by 19.9×.One striking result is that on WMT14 En⇒De, our XLM-D obtains 29.80 BLEU points with multiple iterations, which outperforms the previous mask-predict model by 2.77 points.
MetaTKG: Learning Evolutionary Meta-Knowledge for Temporal Knowledge Graph Reasoning

Reasoning over Temporal Knowledge Graphs (TKGs) aims to predict future facts based on given history.One of the key challenges for prediction is to learn the evolution of facts.Most existing works focus on exploring evolutionary information in history to obtain effective temporal embeddings for entities and relations, but they ignore the variation in evolution patterns of facts, which makes them struggle to adapt to future data with different evolution patterns.Moreover, new entities continue to emerge along with the evolution of facts over time.Since existing models highly rely on historical information to learn embeddings for entities, they perform poorly on such entities with little historical information.To tackle these issues, we propose a novel Temporal Meta-learning framework for TKG reasoning, MetaTKG for brevity.Specifically, our method regards TKG prediction as many temporal meta-tasks, and utilizes the designed Temporal Meta-learner to learn evolutionary metaknowledge from these meta-tasks.The proposed method aims to guide the backbones to learn to adapt quickly to future data and deal with entities with little historical information by the learned meta-knowledge.Specially, in temporal meta-learner, we design a Gating Integration module to adaptively establish temporal correlations between meta-tasks.Extensive experiments on four widely-used datasets and three backbones demonstrate that our method can greatly improve the performance.
Metric-guided Distillation: Distilling Knowledge from the Metric to Ranker and Retriever for Generative Commonsense Reasoning

Commonsense generation aims to generate a realistic sentence describing a daily scene under the given concepts, which is very challenging, since it requires models to have relational reasoning and compositional generalization capabilities.Previous work focuses on retrieving prototype sentences for the provided concepts to assist generation.They first use a sparse retriever to retrieve candidate sentences, then re-rank the candidates with a ranker.However, the candidates returned by their ranker may not be the most relevant sentences, since the ranker treats all candidates equally without considering their relevance to the reference sentences of the given concepts.Another problem is that re-ranking is very expensive, but only using retrievers will seriously degrade the performance of their generation models.To solve these problems, we propose the metric distillation rule to distill knowledge from the metric (e.g., BLEU) to the ranker.We further transfer the critical knowledge summarized by the distilled ranker to the retriever.In this way, the relevance scores of candidate sentences predicted by the ranker and retriever will be more consistent with their quality measured by the metric.Experimental results on the CommonGen benchmark verify the effectiveness of our proposed method: (1) Our generation model with the distilled ranker achieves a new state-of-the-art result.(2) Our generation model with the distilled retriever even surpasses the previous SOTA.