Reviewerly/exHarmony huggingface dataset:
SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data

This paper describes the results of Se-mEval 2023 task 7 -Multi-Evidence Natural Language Inference for Clinical Trial Data (NLI4CT) -consisting of 2 tasks, a Natural Language Inference (NLI) task, and an evidence selection task on clinical trial data.The proposed challenges require multi-hop biomedical and numerical reasoning, which are of significant importance to the development of systems capable of large-scale interpretation and retrieval of medical evidence, to provide personalized evidence-based care.Task 1, the entailment task, received 643 submissions from 40 participants, and Task 2, the evidence selection task, received 364 submissions from 23 participants.The tasks are challenging, with the majority of submitted systems failing to significantly outperform the majority class baseline on the entailment task, and we observe significantly better performance on the evidence selection task than on the entailment task.Increasing the number of model parameters leads to a direct increase in performance, far more significant than the effect of biomedical pre-training.Future works could explore the limitations of large models for generalization and numerical inference, and investigate methods to augment clinical datasets to allow for more rigorous testing and to facilitate finetuning.We envisage that the dataset, models, and results of this task will be useful to the biomedical NLI and evidence retrieval communities.The dataset 1 , competition leaderboard 2 , and website 3 are publicly available.
TemplateGEC: Improving Grammatical Error Correction with Detection Template

Grammatical error correction (GEC) can be divided into sequence-to-edit (Seq2Edit) and sequence-to-sequence (Seq2Seq) frameworks, both of which have their pros and cons.To utilize the strengths and make up for the shortcomings of these frameworks, this paper proposes a novel method, TemplateGEC, which capitalizes on the capabilities of both Seq2Edit and Seq2Seq frameworks in error detection and correction respectively.TemplateGEC utilizes the detection labels from a Seq2Edit model, to construct the template as the input.A Seq2Seq model is employed to enforce consistency between the predictions of different templates by utilizing consistency learning.Experimental results on the Chinese NLPCC18, English BEA19 and CoNLL14 benchmarks show the effectiveness and robustness of TemplateGEC.Further analysis reveals the potential of our method in performing human-in-the-loop GEC.
Patton: Language Model Pretraining on Text-Rich Networks

A real-world text corpus sometimes comprises not only text documents, but also semantic links between them (e.g., academic papers in a bibliographic network are linked by citations and co-authorships).Text documents and semantic connections form a text-rich network, which empowers a wide range of downstream tasks such as classification and retrieval.However, pretraining methods for such structures are still lacking, making it difficult to build one generic model that can be adapted to various tasks on text-rich networks.Current pretraining objectives, such as masked language modeling, purely model texts and do not take interdocument structure information into consideration.To this end, we propose our PretrAining on TexT-Rich NetwOrk framework PATTON.PATTON 1 includes two pretraining strategies: network-contextualized masked language modeling and masked node prediction, to capture the inherent dependency between textual attributes and network structure.We conduct experiments on four downstream tasks in five datasets from both academic and e-commerce domains, where PATTON outperforms baselines significantly and consistently.
Modeling User Satisfaction Dynamics in Dialogue via Hawkes Process

Dialogue systems have received increasing attention while automatically evaluating their performance remains challenging.User satisfaction estimation (USE) has been proposed as an alternative.It assumes that the performance of a dialogue system can be measured by user satisfaction and uses an estimator to simulate users.The effectiveness of USE depends heavily on the estimator.Existing estimators independently predict user satisfaction at each turn and ignore satisfaction dynamics across turns within a dialogue.In order to fully simulate users, it is crucial to take satisfaction dynamics into account.To fill this gap, we propose a new estimator ASAP (sAtisfaction eStimation via HAwkes Process) that treats user satisfaction across turns as an event sequence and employs a Hawkes process to effectively model the dynamics in this sequence.Experimental results on four benchmark dialogue datasets demonstrate that ASAP can substantially outperform state-of-the-art baseline estimators.
Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text

Self-supervised representation learning has proved to be a valuable component for outof-distribution (OoD) detection with only the texts of in-distribution (ID) examples.These approaches either train a language model from scratch or fine-tune a pre-trained language model using ID examples, and then take the perplexity output by the language model as OoD scores.In this paper, we analyze the complementary characteristics of both OoD detection methods and propose a multi-level knowledge distillation approach that integrates their strengths while mitigating their limitations.Specifically, we use a fine-tuned model as the teacher to teach a randomly initialized student model on the ID examples.Besides the prediction layer distillation, we present a similarity-based intermediate layer distillation method to thoroughly explore the representation space of the teacher model.In this way, the learned student can better represent the ID data manifold while gaining a stronger ability to map OoD examples outside the ID data manifold with the regularization inherited from pre-training.Besides, the student model sees only ID examples during parameter learning, further promoting more distinguishable features for OoD detection.We conduct extensive experiments over multiple benchmark datasets, i.e., CLINC150, SST, ROSTD, 20 NewsGroups, and AG News; showing that the proposed method yields new state-of-the-art performance 1 .We also explore its application as an AIGC detector to distinguish between answers generated by ChatGPT and human experts.It is observed that our model exceeds human evaluators in the pair-expert task on the Human ChatGPT Comparison Corpus.
Constructing Code-mixed Universal Dependency Forest for Unbiased Cross-lingual Relation Extraction

Latest efforts on cross-lingual relation extraction (XRE) aggressively leverage the languageconsistent structural features from the universal dependency (UD) resource, while they may largely suffer from biased transfer (e.g., either target-biased or source-biased) due to the inevitable linguistic disparity between languages.In this work, we investigate an unbiased UDbased XRE transfer by constructing a type of code-mixed UD forest.We first translate the sentence of the source language to the parallel target-side language, for both of which we parse the UD tree respectively.Then, we merge the source-/target-side UD structures as a unified code-mixed UD forest.With such forest features, the gaps of UD-based XRE between the training and predicting phases can be effectively closed.We conduct experiments on the ACE XRE benchmark datasets, where the results demonstrate that the proposed code-mixed UD forests help unbiased UD-based XRE transfer, with which we achieve significant XRE performance gains.
The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges

Code-Switching, a common phenomenon in written text and conversation, has been studied over decades by the natural language processing (NLP) research community.Initially, code-switching is intensively explored by leveraging linguistic theories and, currently, more machine-learning oriented approaches to develop models.We introduce a comprehensive systematic survey on code-switching research in natural language processing to understand the progress of the past decades and conceptualize the challenges and tasks on the codeswitching topic.Finally, we summarize the trends and findings and conclude with a discussion for future direction and open questions for further investigation.* The work was done while at Bloomberg.
Contrastive Decoding: Open-ended Text Generation as Optimization

Given a language model (LM), maximum probability is a poor decoding objective for open-ended generation, because it produces short and repetitive text.On the other hand, sampling can often produce incoherent text that drifts from the original topics.We propose contrastive decoding (CD), a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint.The contrastive objective returns the difference between the likelihood under a large LM (called the expert, e.g.OPT-13B) and a small LM (called the amateur, e.g.OPT-125M), and the constraint ensures that the outputs are plausible.CD is inspired by the fact that the failures of larger LMs (e.g., repetition, incoherence) are even more prevalent in smaller LMs, and that this difference signals which texts should be preferred.CD requires zero additional training, and produces higher quality text than decoding from the larger LM alone.It also works across model scales (OPT-13B and GPT2-1.5B)and significantly outperforms four strong decoding algorithms (e.g., nucleus, top-k) in automatic and human evaluations across wikipedia, news and story domains. 1
Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition

Due to the absence of explicit connectives, implicit discourse relation recognition (IDRR) remains a challenging task in discourse analysis.The critical step for IDRR is to learn highquality discourse relation representations between two arguments.Recent methods tend to integrate the whole hierarchical information of senses into discourse relation representations for multi-level sense recognition.Nevertheless, they insufficiently incorporate the static hierarchical structure containing all senses (defined as global hierarchy), and ignore the hierarchical sense label sequence corresponding to each instance (defined as local hierarchy).For the purpose of sufficiently exploiting global and local hierarchies of senses to learn better discourse relation representations, we propose a novel GlObal and Local Hierarchy-aware Contrastive Framework (GOLF), to model two kinds of hierarchies with the aid of multi-task learning and contrastive learning.Experimental results on PDTB 2.0 and PDTB 3.0 datasets demonstrate that our method remarkably outperforms current state-of-the-art models at all hierarchical levels. 1
MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies

Autoregressive language models are trained by minimizing the cross-entropy of the model distribution Q θ relative to the data distribution Pthat is, minimizing the forward cross-entropy, which is equivalent to maximum likelihood estimation (MLE).We have observed that models trained in this way may "over-generalize", in the sense that they produce non-human-like text.Moreover, we believe that reverse crossentropy, i.e., the cross-entropy of P relative to Q θ , is a better reflection of how a human would evaluate text generated by a model.Hence, we propose learning with MIXCE, an objective that mixes the forward and reverse crossentropies.We evaluate models trained with this objective on synthetic data settings (where P is known) and real data, and show that the resulting models yield better generated text without complex decoding strategies.https://github.com/bloomberg/mixce-acl2023 * Work done during an internship at Bloomberg. 1 Unbiased sampling is vanilla random sampling, i.e., sampling with temperature=1.0.It is also called ancestral sampling (Eikema and Aziz, 2020) or pure sampling (Holtzman
OpineSum: Entailment-based self-training for abstractive opinion summarization

A typical product or place often has hundreds of reviews, and summarization of these texts is a challenging problem.Recent progress on abstractive summarization in domains such as news has been driven by supervised systems trained on hundreds of thousands of news articles paired with human-written summaries.However for opinion texts, such large scale datasets are rarely available.Unsupervised methods, self-training, and few-shot learning approaches bridge that gap.In this work, we present a novel self-training approach, OPINE-SUM for abstractive opinion summarization.The self-training summaries in this approach are built automatically using a novel application of textual entailment and capture the consensus of opinions across the various reviews for an item.This method can be used to obtain silver-standard summaries on a large scale and train both unsupervised and few-shot abstractive summarization systems.OPINESUM outperforms strong peer systems in both settings.
PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts

Perceiving multi-modal information and fulfilling dialogues with humans is a long-term goal of artificial intelligence.Pre-training is commonly regarded as an effective approach for multi-modal dialogue.However, due to the limited availability of multi-modal dialogue data, there is still scarce research on multi-modal dialogue pre-training.Yet another intriguing challenge emerges from the encompassing nature of multi-modal dialogue, which involves various modalities and tasks.Moreover, new forms of tasks may arise at unpredictable points in the future.Hence, it is essential for designed multi-modal dialogue models to possess sufficient flexibility to adapt to such scenarios.This paper proposes PaCE, a unified, structured, compositional multi-modal dialogue pretraining framework.It utilizes a combination of several fundamental experts to accommodate multiple dialogue-related tasks and can be pre-trained using limited dialogue and extensive non-dialogue multi-modal data.Furthermore, we propose a progressive training method where old experts from the past can assist new experts, facilitating the expansion of their capabilities.Experimental results demonstrate that PaCE achieves state-of-the-art results on eight multi-modal dialog benchmarks.
WYWEB: A NLP Evaluation Benchmark For Classical Chinese

To fully evaluate the overall performance of different NLP models in a given domain, many evaluation benchmarks are proposed, such as GLUE, SuperGLUE and CLUE.The field of natural language understanding has traditionally focused on benchmarks for various tasks in languages such as Chinese, English, and multilingual, however, there has been a lack of attention given to the area of classical Chinese, also known as "wen yan wen (文言文)", which has a rich history spanning thousands of years and holds significant cultural and academic value.For the prosperity of the NLP community, in this paper, we introduce the WYWEB evaluation benchmark, which consists of nine NLP tasks in classical Chinese, implementing sentence classification, sequence labeling, reading comprehension, and machine translation.We evaluate the existing pre-trained language models, which are all struggling with this benchmark.We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on classical Chinese NLU.
Document-Level Event Argument Extraction With a Chain Reasoning Paradigm

Document-level event argument extraction aims to identify event arguments beyond sentence level, where a significant challenge is to model long-range dependencies.Focusing on this challenge, we present a new chain reasoning paradigm for the task, which can generate decomposable first-order logic rules for reasoning.This paradigm naturally captures long-range interdependence due to the chains' compositional nature, which also improves interpretability by explicitly modeling the reasoning process.We introduce T-norm fuzzy logic for optimization, which permits end-toend learning and shows promise for integrating the expressiveness of logical reasoning with the generalization of neural networks.In experiments, we show that our approach outperforms previous methods by a significant margin on two standard benchmarks (over 6 points in F1).Moreover, it is data-efficient in lowresource scenarios and robust enough to defend against adversarial attacks.
Distilling Script Knowledge from Large Language Models for Constrained Language Planning

In everyday life, humans often plan their actions by following step-by-step instructions in the form of goal-oriented scripts.Previous work has exploited language models (LMs) to plan for abstract goals of stereotypical activities (e.g., "make a cake"), but leaves more specific goals with multi-facet constraints understudied (e.g., "make a cake for diabetics").In this paper, we define the task of constrained language planning for the first time.We propose an overgenerate-then-filter approach to improve large language models (LLMs) on this task, and use it to distill a novel constrained language planning dataset, CoScript, which consists of 55,000 scripts.Empirical results demonstrate that our method significantly improves the constrained language planning ability of LLMs, especially on constraint faithfulness.Furthermore, CoScript is demonstrated to be quite effective in endowing smaller LMs with constrained language planning ability.
SCOTT: Self-Consistent Chain-of-Thought Distillation

Large language models (LMs) beyond a certain scale, demonstrate the emergent capability of generating free-text rationales for their predictions via chain-of-thought (CoT) prompting.While CoT can yield dramatically improved performance, such gains are only observed for sufficiently large LMs.Even more concerning, there is little guarantee that the generated rationales are consistent with LM's predictions or faithfully justify the decisions.In this work, we propose SCOTT, a faithful knowledge distillation method to learn a small, self-consistent CoT model from a teacher model that is orders of magnitude larger.To form better supervision, we elicit rationales supporting the gold answers from a large LM (teacher) by contrastive decoding, which encourages the teacher to generate tokens that become more plausible only when the answer is considered.To ensure faithful distillation, we use the teacher-generated rationales to learn a student LM with a counterfactual reasoning objective, which prevents the student from ignoring the rationales to make inconsistent predictions.Experiments show that, while yielding comparable end-task performance, our method can generate CoT rationales that are more faithful than baselines do.Further analysis suggests that such a model respects the rationales more when making decisions; thus, we can improve its performance more by refining its rationales.
Retrieval-free Knowledge Injection through Multi-Document Traversal for Dialogue Models

Dialogue models are often enriched with extensive external knowledge to provide informative responses through a retrieval-augmented pipeline.Nevertheless, retrieval-augmented approaches rely on finely annotated retrieval training data and knowledge-grounded response generation data, making it costly to transfer.To tackle this challenge, this paper proposed a retrieval-free approach, KiDG, by automatically turning knowledge documents into simulated multi-turn dialogues through a Multi-Document Traversal algorithm.The simulated knowledge-intensive dialogues constructed by KiDG in one domain can be easily used to train and enhance pre-trained dialogue models' knowledge w.r.t.this domain without costly annotation.We conduct extensive experiments comparing retrieval-augmented models and a variety of retrieval-free models.We found that dialogue models enhanced with data simulated with KiDG largely outperform state-ofthe-art retrieval-free methods, and it achieves comparable performance compared to retrievalaugmented methods while being better, and cheaper at domain transfer.We have released the code and data at https://github.com/DevoAllen/KiDG.
CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors

Large language models (LLMs) pre-trained on massive corpora have demonstrated impressive few-shot learning ability on many NLP tasks.A common practice is to recast the task into a text-to-text format such that generative LLMs of natural language (NL-LLMs) like GPT-3 can be prompted to solve it.However, it is nontrivial to perform information extraction (IE) tasks with NL-LLMs since the output of the IE task is usually structured and therefore is hard to be converted into plain text.In this paper, we propose to recast the structured output in the form of code instead of natural language and utilize generative LLMs of code (Code-LLMs) such as Codex to perform IE tasks, in particular, named entity recognition and relation extraction.In contrast to NL-LLMs, we show that Code-LLMs can be well-aligned with these IE tasks by designing code-style prompts and formulating these IE tasks as code generation tasks.Experiment results on seven benchmarks show that our method consistently outperforms fine-tuning moderate-size pre-trained models specially designed for IE tasks (e.g., UIE) and prompting NL-LLMs under few-shot settings.We further conduct a series of in-depth analyses to demonstrate the merits of leveraging Code-LLMs for IE tasks.
Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter

Federated Multilingual Neural Machine Translation (Fed-MNMT) has emerged as a promising paradigm for institutions with limited language resources.This approach allows multiple institutions to act as clients and train a unified model through model synchronization, rather than collecting sensitive data for centralized training.This significantly reduces the cost of corpus collection and preserves data privacy.However, as pre-trained language models (PLMs) continue to increase in size, the communication cost for transmitting parameters during synchronization has become a training speed bottleneck.In this paper, we propose a communication-efficient Fed-MNMT framework that addresses this issue by keeping PLMs frozen and only transferring lightweight adapter modules between clients.Since different language pairs exhibit substantial discrepancies in data distributions, adapter parameters of clients may conflict with each other.To tackle this, we explore various clustering strategies to group parameters for integration and mitigate the negative effects of conflicting parameters.Experimental results demonstrate that our framework reduces communication cost by over 98% while achieving similar or even better performance compared to competitive baselines.Further analysis reveals that clustering strategies effectively solve the problem of linguistic discrepancy and pruning adapter modules further improves communication efficiency. 1 lv-pl