Reviewerly/exHarmony huggingface dataset:
What Is Overlap Knowledge in Event Argument Extraction? APE: A Cross-datasets Transfer Learning Model for EAE

The EAE task extracts a structured event record from an event text.Most existing approaches train the EAE model on each dataset independently and ignore the overlap knowledge across datasets.However, insufficient event records in a single dataset often prevent the existing model from achieving better performance.In this paper, we clearly define the overlap knowledge across datasets and split the knowledge of the EAE task into overlap knowledge across datasets and specific knowledge of the target dataset.We propose APE model to learn the two parts of knowledge in two serial learning phases without causing catastrophic forgetting.In addition, we formulate both learning phases as conditional generation tasks and design Stressing Entity Type Prompt to close the gap between the two phases.The experiments show APE achieves new state-of-the-art with a large margin in the EAE task.When only ten records are available in the target dataset, our model dramatically outperforms the baseline model with average 27.27%F1 gain.
Distilling Calibrated Knowledge for Stance Detection

Stance detection aims to determine the position of an author toward a target and provides insights into people's views on controversial topics such as marijuana legalization.Despite recent progress in this task, most existing approaches use hard labels (one-hot vectors) during training, which ignores meaningful signals among categories offered by soft labels.In this work, we explore knowledge distillation for stance detection and present a comprehensive analysis.Our contributions are: 1) we propose to use knowledge distillation over multiple generations in which a student is taken as a new teacher to transfer knowledge to a new fresh student; 2) we propose a novel dynamic temperature scaling for knowledge distillation to calibrate teacher predictions in each generation step.Extensive results on three stance detection datasets show that knowledge distillation benefits stance detection and a teacher is able to transfer knowledge to a student more smoothly via calibrated guiding signals.We publicly release our code to facilitate future research.
Arabic Word-level Readability Visualization for Assisted Text Simplification

This demo paper presents a Google Docs addon for automatic Arabic word-level readability visualization.The add-on includes a lemmatization component that is connected to a five-level readability lexicon and Arabic WordNet-based substitution suggestions.The add-on can be used for assessing the reading difficulty of a text and identifying difficult words as part of the task of manual text simplification.We make our add-on and its code publicly available. 1,2
Normalized Contrastive Learning for Text-Video Retrieval

Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness.In this work, however, we reveal that cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance.Specifically, we show that many test instances are either overor under-represented during retrieval, significantly hurting the retrieval performance.To address this problem, we propose Normalized Contrastive Learning (NCL) which utilizes the Sinkhorn-Knopp algorithm to compute the instance-wise biases that properly normalize the sum retrieval probabilities of each instance so that every text and video instance is fairly represented during cross-modal retrieval.Empirical study shows that NCL brings consistent and significant gains in text-video retrieval on different model architectures, with new stateof-the-art multimodal retrieval metrics on the ActivityNet, MSVD, and MSR-VTT datasets without any architecture engineering.
Neural Machine Translation with Contrastive Translation Memories

Retrieval-augmented Neural Machine Translation models have been successful in many translation scenarios.Different from previous works that make use of mutually similar but redundant translation memories (TMs), we propose a new retrieval-augmented NMT to model contrastively retrieved translation memories that are holistically similar to the source sentence while individually contrastive to each other providing maximal information gains in three phases.First, in TM retrieval phase, we adopt a contrastive retrieval algorithm to avoid redundancy and uninformativeness of similar translation pieces.Second, in memory encoding stage, given a set of TMs we propose a novel Hierarchical Group Attention module to gather both local context of each TM and global context of the whole TM set.Finally, in training phase, a Multi-TM contrastive learning objective is introduced to learn salient feature of each TM with respect to target sentence.Experimental results show that our framework obtains improvements over strong baselines on the benchmark datasets.
Infinite SCAN: An Infinite Model of Diachronic Semantic Change

In this study, we propose a Bayesian model that can jointly estimate the number of senses of words and their changes through time.The model combines a dynamic topic model on Gaussian Markov random fields (Frermann and Lapata, 2016) with a logistic stick-breaking process that realizes the Dirichlet process.In the experiments, we evaluated the proposed model in terms of interpretability, accuracy in estimating the number of senses, and tracking their changes using both artificial data and real data.We quantitatively verified that the model behaves as expected through evaluation using artificial data.Using the CCOHA corpus, we showed that our model outperforms the baseline model and investigated the semantic changes of several well-known target words.
Federated Model Decomposition with Private Vocabulary for Text Classification

With the necessity of privacy protection, it becomes increasingly vital to train deep neural models in a federated learning manner for natural language processing (NLP) tasks.However, recent studies show eavesdroppers (i.e., dishonest servers) can still reconstruct the private input in federated learning (FL).Such a data reconstruction attack relies on the mappings between vocabulary and associated word embedding in NLP tasks, which are unfortunately less studied in current FL methods.In this paper, we propose a fedrated model decomposition method that protects the privacy of vocabularies, shorted as FEDEVOCAB.In FEDEVOCAB, each participant keeps the local embedding layer in the local device and detaches the local embedding parameters from federated aggregation.However, it is challenging to train an accurate NLP model when the private mappings are unknown and vary across participants in a cross-device FL setting.To address this problem, we further propose an adaptive updating technique to improve the performance of local models.Experimental results show that FEDEVOCAB maintains competitive performance and provides better privacy-preserving capacity compared to status quo methods.* Co-corresponding author Hi, recently I feel very thirsty and urinate more.And I get hungry easily and eat more than usual.But being 165cm tall, my weight even dropped to 40kg.It has continued for half a month.Hi! Does your family have a history of diabetes?If so, there is a high probability of diabetes.
Open-Topic False Information Detection on Social Networks with Contrastive Adversarial Learning

Current works about false information detection based on conversation graphs on social networks focus primarily on two research streams from the standpoint of topic distribution: intopic and cross-topic techniques, which assume that the data topic distribution is identical or cross, respectively.This signifies that all test data topics are seen or unseen by the model.However, these assumptions are too harsh for actual social networks that contain both seen and unseen topics simultaneously, hence restricting their practical application.In light of this, this paper develops a novel open-topic scenario that is better suited to actual social networks.In this open-topic scenario, we empirically find that the existing models suffer from impairment in the detection performance for seen or unseen topic data, resulting in poor overall model performance.To address this issue, we propose a novel Contrastive Adversarial Learning Network, CALN, that employs an unsupervised topic clustering method to capture topic-specific features to enhance the model's performance for seen topics and an unsupervised adversarial learning method to align data representation distributions to enhance the model's generalisation to unseen topics.Experiments on two benchmark datasets and a variety of graph neural networks demonstrate the effectiveness of our approach.
Domain Adaptation of Machine Translation with Crowdworkers

Although a machine translation model trained with a large in-domain parallel corpus achieves remarkable results, it still works poorly when no in-domain data are available.This situation restricts the applicability of machine translation when the target domain's data are limited.However, there is great demand for high-quality domain-specific machine translation models for many domains.We propose a framework that efficiently and effectively collects parallel sentences in a target domain from the web with the help of crowdworkers.With the collected parallel data, we can quickly adapt a machine translation model to the target domain.Our experiments show that the proposed method can collect target-domain parallel data over a few days at a reasonable cost.We tested it with five domains, and the domain-adapted model improved the BLEU scores to +19.7 by an average of +7.8 points compared to a general-purpose translation model.
Automatic Generation of Socratic Subquestions for Teaching Math Word Problems

Socratic questioning is an educational method that allows students to discover answers to complex problems by asking them a series of thoughtful questions.Generation of didactically sound questions is challenging, requiring understanding of the reasoning process involved in the problem.We hypothesize that such questioning strategy can not only enhance the human performance, but also assist the math word problem (MWP) solvers.In this work, we explore the ability of large language models (LMs) in generating sequential questions for guiding math word problem-solving.We propose various guided question generation schemes based on input conditioning and reinforcement learning.On both automatic and human quality evaluations, we find that LMs constrained with desirable question properties generate superior questions and improve the overall performance of a math word problem solver.We conduct a preliminary user study to examine the potential value of such question generation models in the education domain.Results suggest that the difficulty level of problems plays an important role in determining whether questioning improves or hinders human performance.We discuss the future of using such questioning strategies in education.https://github.com/eth-nlped/scaffolding-generation
DSM: Question Generation over Knowledge Base via Modeling Diverse Subgraphs with Meta-learner

Existing methods on knowledge base question generation (KBQG) learn a one-size-fits-all model by training together all subgraphs without distinguishing the diverse semantics of subgraphs.In this work, we show that making use of the past experience on semantically similar subgraphs can reduce the learning difficulty and promote the performance of KBQG models.To achieve this, we propose a novel approach to model diverse subgraphs with metalearner (DSM).Specifically, we devise a graph contrastive learning-based retriever to identify semantically similar subgraphs, so that we can construct the semantics-aware learning tasks for the meta-learner to learn semanticsspecific and semantics-agnostic knowledge on and across these tasks.Extensive experiments on two widely-adopted benchmarks for KBQG show that DSM derives new state-of-the-art performance and benefits the question answering tasks as a means of data augmentation.Codes and datasets are available online 1 .
Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts

Question-answering datasets require a broad set of reasoning skills.We show how to use question decompositions to teach language models these broad reasoning skills in a robust fashion.Specifically, we use widely available QDMR representations to programmatically create hard-to-cheat synthetic contexts for real questions in six multi-step reasoning datasets.These contexts are carefully designed to avoid common reasoning shortcuts prevalent in real contexts that prevent models from learning the right skills.This results in a pretraining dataset, named TeaBReaC, containing 525K multi-step questions (with associated formal programs) covering about 900 reasoning patterns.We show that pretraining standard language models (LMs) on TeaBReaC before fine-tuning them on target datasets improves their performance by up to 13 F1 points across 4 multi-step QA datasets, with up to 21 point gain on more complex questions.The resulting models also demonstrate higher robustness, with a 5-8 F1 point improvement on two contrast sets.Furthermore, TeaBReaC pretraining substantially improves model performance and robustness even when starting with numerate LMs pretrained using recent methods (e.g., PReasM, POET).Our work thus shows how to effectively use decomposition-guided contexts to robustly teach multi-step reasoning.
Knowledge-augmented Self-training of A Question Rewriter for Conversational Knowledge Base Question Answering

The recent rise of conversational applications such as online customer service systems and intelligent personal assistants has promoted the development of conversational knowledge base question answering (ConvKBQA).Different from the traditional single-turn KBQA, ConvKBQA usually explores multi-turn questions around a topic, where ellipsis and coreference pose great challenges to the single-turn KBQA systems which require self-contained questions.In this paper, we propose a rewriteand-reason framework to first produce a fullfledged rewritten question based on the conversation history and then reason the answer by existing single-turn KBQA models.To overcome the absence of the rewritten supervision signals, we introduce a knowledge-augmented self-training mechanism to transfer the question rewriter from another dataset to adapt to the current knowledge base.Our question rewriter is decoupled from the subsequent QA process, which makes it easy to be united with either retrieval-based or semantic parsing-based KBQA models.Experiment results demonstrate the effectiveness of our method and a new state-of-the-art result is achieved.The code and dataset are available online now 1 .
Enhancing Automatic Readability Assessment with Pre-training and Soft Labels for Ordinal Regression

The readability assessment task aims to assign a difficulty grade to a text.While neural models have recently demonstrated impressive performance, most do not exploit the ordinal nature of the difficulty grades, and make little effort for model initialization to facilitate fine-tuning.We address these limitations with soft labels for ordinal regression, and with model pre-training through prediction of pairwise relative text difficulty.We incorporate these two components into a model based on hierarchical attention networks, and evaluate its performance on both English and Chinese datasets.Experimental results show that our proposed model outperforms competitive neural models and statistical classifiers on most datasets.
Specializing Multi-domain NMT via Penalizing Low Mutual Information

Multi-domain Neural Machine Translation (NMT) trains a single model with multiple domains.It is appealing because of its efficacy in handling multiple domains within one model.An ideal multi-domain NMT should learn distinctive domain characteristics simultaneously, however, grasping the domain peculiarity is a non-trivial task.In this paper, we investigate domain-specific information through the lens of mutual information (MI) and propose a new objective that penalizes low MI to become higher.Our method achieved the state-of-theart performance among the current competitive multi-domain NMT models.Also, we empirically show our objective promotes low MI to be higher resulting in domain-specialized multidomain NMT.
Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator

Task-Oriented Dialogue (TOD) systems are drawing more and more attention in recent studies.Current methods focus on constructing pre-trained models or fine-tuning strategies while the evaluation of TOD is limited by a policy mismatch problem.That is, during evaluation, the user utterances are from the annotated dataset while these utterances should interact with previous responses which can have many alternatives besides annotated texts.Therefore, in this work, we propose an interactive evaluation framework for TOD.We first build a goaloriented user simulator based on pre-trained models and then use the user simulator to interact with the dialogue system to generate dialogues.Besides, we introduce a sentence-level and a session-level score to measure the sentence fluency and session coherence in the interactive evaluation.Experimental results show that RL-based TOD systems trained by our proposed user simulator can achieve nearly 98% inform and success rates in the interactive evaluation of MultiWOZ dataset and the proposed scores measure the response quality besides the inform and success rates.We are hoping that our work will encourage simulator-based interactive evaluations in the TOD task 1 .1 https://github.com/xiami2019/
Doc2Bot: Accessing Heterogeneous Documents via Conversational Bots

This paper introduces Doc2Bot, a novel dataset for building machines that help users seek information via conversations.This is of particular interest for companies and organizations that own a large number of manuals or instruction books.Despite its potential, the nature of our task poses several challenges: (1) documents contain various structures that hinder the ability of machines to comprehend, and (2) user information needs are often underspecified.Compared to prior datasets that either focus on a single structural type or overlook the role of questioning to uncover user needs, the Doc2Bot dataset is developed to target such challenges systematically.Our dataset contains over 100,000 turns based on Chinese documents from five domains, larger than any prior document-grounded dialog dataset for information seeking.We propose three tasks in Doc2Bot: (1) dialog state tracking to track user intentions, (2) dialog policy learning to plan system actions and contents, and (3) response generation which generates responses based on the outputs of the dialog policy.Baseline methods based on the latest deep learning models are presented, indicating that our proposed tasks are challenging and worthy of further research.
Proxy-based Zero-Shot Entity Linking by Effective Candidate Retrieval

A recent advancement in the domain of biomedical Entity Linking is the development of powerful two-stage algorithms -an initial candidate retrieval stage that generates a shortlist of entities for each mention, followed by a candidate ranking stage.However, the effectiveness of both stages are inextricably dependent on computationally expensive components.Specifically, in candidate retrieval via dense representation retrieval it is important to have hard negative samples, which require repeated forward passes and nearest neighbour searches across the entire entity label set throughout training.In this work, we show that pairing a proxy-based metric learning loss with an adversarial regularizer provides an efficient alternative to hard negative sampling in the candidate retrieval stage.In particular, we show competitive performance on the recall@1 metric, thereby providing the option to leave out the expensive candidate ranking step.Finally, we demonstrate how the model can be used in a zero-shot setting to discover out of knowledge base biomedical entities.
Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialogue

Lifelong learning (LL) is vital for advanced task-oriented dialogue (ToD) systems.To address the catastrophic forgetting issue of LL, generative replay methods are widely employed to consolidate past knowledge with generated pseudo samples.However, most existing generative replay methods use only a single taskspecific token to control their models.This scheme is usually not strong enough to constrain the generative model due to insufficient information involved.In this paper, we propose a novel method, prompt conditioned VAE for lifelong learning (PCLL), to enhance generative replay by incorporating tasks' statistics.PCLL captures task-specific distributions with a conditional variational autoencoder, conditioned on natural language prompts to guide the pseudo-sample generation.Moreover, it leverages a distillation process to further consolidate past knowledge by alleviating the noise in pseudo samples.Experiments on natural language understanding tasks of ToD systems demonstrate that PCLL significantly outperforms competitive baselines in building lifelong learning models.We release the code and data at GitHub.
Unsupervised Opinion Summarisation in the Wasserstein Space

Opinion summarisation synthesises opinions expressed in a group of documents discussing the same topic to produce a single summary.Recent work has looked at opinion summarisation of clusters of social media posts.Such posts are noisy and have unpredictable structure, posing additional challenges for the construction of the summary distribution and the preservation of meaning compared to online reviews, which has been so far the focus of opinion summarisation.To address these challenges we present WassOS, an unsupervised abstractive summarization model which makes use of the Wasserstein distance.A Variational Autoencoder is used to get the distribution of documents/posts, and the distributions are disentangled into separate semantic and syntactic spaces.The summary distribution is obtained using the Wasserstein barycenter of the semantic and syntactic distributions.A latent variable sampled from the summary distribution is fed into a GRU decoder with a transformer layer to produce the final summary.Our experiments on multiple datasets including Twitter clusters, Reddit threads, and reviews show that WassOS almost always outperforms the state-of-the-art on ROUGE metrics and consistently produces the best summaries with respect to meaning preservation according to human evaluations.
Text Style Transferring via Adversarial Masking and Styled Filling

Text style transfer is an important task in natural language processing with broad applications.Existing models following the masking and filling scheme suffer two challenges: the word masking procedure may mistakenly remove unexpected words and the selected words in the word filling procedure may lack diversity and semantic consistency.To tackle both challenges, in this study, we propose a style transfer model, with an adversarial masking approach and a styled filling technique (AMSF).Specifically, AMSF first trains a mask predictor by adversarial training without manual configuration.Then two additional losses, i.e. an entropy maximization loss and a consistency regularization loss, are introduced in training the word filling module to guarantee the diversity and semantic consistency of the transferred texts.Experimental results and analysis on two benchmark text style transfer data sets demonstrate the effectiveness of the proposed approaches.