Reviewerly/exHarmony huggingface dataset:
Effective Contrastive Weighting for Dense Query Expansion

Verbatim queries submitted to search engines often do not sufficiently describe the user's search intent.Pseudo-relevance feedback (PRF) techniques, which modify a query's representation using the top-ranked documents, have been shown to overcome such inadequacies and improve retrieval effectiveness for both lexical methods (e.g., BM25) and dense methods (e.g., ANCE, ColBERT).For instance, the recent ColBERT-PRF approach heuristically chooses new embeddings to add to the query representation using the inverse document frequency (IDF) of the underlying tokens.However, this heuristic potentially ignores the valuable context encoded by the embeddings.In this work, we present a contrastive solution that learns to select the most useful embeddings for expansion.More specifically, a deep language model-based contrastive weighting model, called CWPRF, is trained to learn to discriminate between relevant and non-relevant documents for semantic search.Our experimental results show that our contrastive weighting model can aid to select useful expansion embeddings and outperform various baselines.In particular, CWPRF can improve nDCG@10 by upto to 4.1% compared to an existing PRF approach for ColBERT while maintaining its efficiency.
Toward Expanding the Scope of Radiology Report Summarization to Multiple Anatomies and Modalities

Radiology report summarization (RRS) is a growing area of research.Given the Findings section of a radiology report, the goal is to generate a summary (called an Impression section) that highlights the key observations and conclusions of the radiology study.However, RRS currently faces essential limitations.First, many prior studies conduct experiments on private datasets, preventing the reproduction of results and fair comparisons across different systems and solutions.Second, most prior approaches are evaluated solely on chest Xrays.To address these limitations, we propose a dataset (MIMIC-RRS) involving three new modalities and seven new anatomies based on the MIMIC-III and MIMIC-CXR datasets.We then conduct extensive experiments to evaluate the performance of models both within and across modality-anatomy pairs in MIMIC-RRS.In addition, we evaluate their clinical efficacy via RadGraph, a factual correctness metric.
Unsupervised Keyphrase Extraction by Learning Neural Keyphrase Set Function

We create a paradigm shift concerning building unsupervised keyphrase extraction systems in this paper.Instead of modeling the relevance between an individual candidate phrase and the document as in the commonly used framework, we formulate the unsupervised keyphrase extraction task as a document-set matching problem from a set-wise perspective, in which the document and the candidate set are globally matched in the semantic space to particularly take into account the interactions among all candidate phrases.Since it is intractable to exactly extract the optimal subset by the document-set matching function during the inference, we propose an approximate approach, which obtains the candidate subsets via a set extractor agent learned by reinforcement learning.Exhaustive experimental results demonstrate the effectiveness of our model, which outperforms the recent state-of-the-art unsupervised keyphrase extraction baselines by a large margin.
Plug-and-Play Knowledge Injection for Pre-trained Language Models

Injecting external knowledge can improve the performance of pre-trained language models (PLMs) on various downstream NLP tasks.However, massive retraining is required to deploy new knowledge injection methods or knowledge bases for downstream tasks.In this work, we are the first to study how to improve the flexibility and efficiency of knowledge injection by reusing existing downstream models.To this end, we explore a new paradigm plugand-play knowledge injection, where knowledge bases are injected into frozen existing downstream models by a knowledge plugin.Correspondingly, we propose a plug-and-play injection method map-tuning, which trains a mapping of knowledge embeddings to enrich model inputs with mapped embeddings while keeping model parameters frozen.Experimental results on three knowledge-driven NLP tasks show that existing injection methods are not suitable for the new paradigm, while maptuning effectively improves the performance of downstream models.Moreover, we show that a frozen downstream model can be well adapted to different domains with different mapping networks of domain knowledge.Our code and models are available at https://github.com/THUNLP/Knowledge-Plugin.
Aligning Recommendation and Conversation via Dual Imitation

Human conversations of recommendation naturally involve the shift of interests which can align the recommendation actions and conversation process to make accurate recommendations with rich explanations.However, existing conversational recommendation systems (CRS) ignore the advantage of user interest shift in connecting recommendation and conversation, which leads to an ineffective loose coupling structure of CRS.To address this issue, by modeling the recommendation actions as recommendation paths in a knowledge graph (KG), we propose DICR (Dual Imitation for Conversational Recommendation), which designs a dual imitation to explicitly align the recommendation paths and user interest shift paths in a recommendation module and a conversation module, respectively.By exchanging alignment signals, DICR achieves bidirectional promotion between recommendation and conversation modules and generates high-quality responses with accurate recommendations and coherent explanations.Experiments demonstrate that DICR outperforms the state-of-theart models on recommendation and conversation performance with automatic, human, and novel explainability metrics.
Generating Literal and Implied Subquestions to Fact-check Complex Claims

Verifying political claims is a challenging task, as politicians can use various tactics to subtly misrepresent the facts for their agenda.Existing automatic fact-checking systems fall short here, and their predictions like "half-true" are not very useful in isolation, since it is unclear which parts of a claim are true or false.In this work, we focus on decomposing a complex claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim.We present CLAIMDECOMP, a dataset of decompositions for over 1000 claims.Given a claim and its verification paragraph written by fact-checkers, our trained annotators write subquestions covering both explicit propositions of the original claim and its implicit facets, such as additional political context that changes our view of the claim's veracity.We study whether state-of-the-art pre-trained models can learn to generate such subquestions.Our experiments show that these models generate reasonable questions, but predicting implied subquestions based only on the claim (without consulting other evidence) remains challenging.Nevertheless, we show that predicted subquestions can help identify relevant evidence to fact-check the full claim and derive the veracity through their answers, suggesting that claim decomposition can be a useful piece of a fact-checking pipeline. 1Joe Biden stated on August 31, 2020 in a speech: "When I was vice president, violent crime fell 15% in this country.... The murder rate now is up 26% across the nation this year under Donald Trump.
SafeText: A Benchmark for Exploring Physical Safety in Language Models

Understanding what constitutes safe text is an important issue in natural language processing and can often prevent the deployment of models deemed harmful and unsafe.One such type of safety that has been scarcely studied is commonsense physical safety, i.e. text that is not explicitly violent and requires additional commonsense knowledge to comprehend that it leads to physical harm.We create the first benchmark dataset, SAFETEXT, comprising real-life scenarios with paired safe and physically unsafe pieces of advice.We utilize SAFE-TEXT to empirically study commonsense physical safety across various models designed for text generation and commonsense reasoning tasks.We find that state-of-the-art large language models are susceptible to the generation of unsafe text and have difficulty rejecting unsafe advice.As a result, we argue for further studies of safety and the assessment of commonsense physical safety in models before release.
Decomposing Natural Logic Inferences for Neural NLI

In the interest of interpreting neural NLI models and their reasoning strategies, we carry out a systematic probing study which investigates whether these models capture the crucial semantic features central to natural logic: monotonicity and concept inclusion.Correctly identifying valid inferences in downward-monotone contexts is a known stumbling block for NLI performance, subsuming linguistic phenomena such as negation scope and generalized quantifiers.To understand this difficulty, we emphasize monotonicity as a property of a context and examine the extent to which models capture relevant monotonicity information in the vector representations which are intermediate to their decision making process.Drawing on the recent advancement of the probing paradigm, we compare the presence of monotonicity features across various models.We find that monotonicity information is notably weak in the representations of popular NLI models which achieve high scores on benchmarks, and observe that previous improvements to these models based on fine-tuning strategies have introduced stronger monotonicity features together with their improved performance on challenge sets.
Unsupervised Dense Retrieval for Scientific Articles

In this work, we build a dense retrieval based semantic search engine on scientific articles from Elsevier. The major challenge is that there is no labeled data for training and testing. We apply a state-of-the-art unsupervised dense retrieval model called Generative Pseudo Labeling that generates high-quality pseudo training labels. Furthermore, since the articles are unbalanced across different domains, we select passages from multiple domains to form balanced training data. For the evaluation, we create two test sets: one manually annotated and one automatically created from the meta-information of our data. We compare the semantic search engine with the currently deployed lexical search engine on the two test sets. The results of the experiment show that the semantic search engine trained with pseudo training labels can significantly improve search performance.
Non-Parametric Domain Adaptation for End-to-End Speech Translation

The end-to-end speech translation (E2E-ST) has received increasing attention due to the potential of its less error propagation, lower latency and fewer parameters.However, the effectiveness of neural-based approaches to this task is severely limited by the available training corpus, especially for domain adaptation where in-domain triplet data is scarce or nonexistent.In this paper, we propose a novel nonparametric method that leverages in-domain text translation corpus to achieve domain adaptation for E2E-ST systems.To this end, we first incorporate an additional encoder into the pre-trained E2E-ST model to realize text translation modeling, based on which the decoder's output representations for text and speech translation tasks are unified by reducing the correspondent representation mismatch in available triplet training data.During domain adaptation, a k-nearest-neighbor (kNN) classifier is introduced to produce the final translation distribution using the external datastore built by the domain-specific text translation corpus, while the universal output representation is adopted to perform a similarity search.Experiments on the Europarl-ST benchmark demonstrate that when in-domain text translation data is involved only, our proposed approach significantly improves baseline by 12.82 BLEU on average in all translation directions, even outperforming the strong in-domain fine-tuning strategy.
Mask-then-Fill: A Flexible and Effective Data Augmentation Framework for Event Extraction

We present Mask-then-Fill, a flexible and effective data augmentation framework for event extraction.Our approach allows for more flexible manipulation of text and thus can generate more diverse data while keeping the original event structure unchanged as much as possible.Specifically, it first randomly masks out an adjunct sentence fragment and then infills a variable-length text span with a fine-tuned infilling model.The main advantage lies in that it can replace a fragment of arbitrary length in the text with another fragment of variable length, compared to the existing methods which can only replace a single word or a fixed-length fragment.On trigger and argument extraction tasks, the proposed framework is more effective than baseline methods and it demonstrates particularly strong results in the low-resource setting.Our further analysis shows that it achieves a good balance between diversity and distributional similarity.
Inductive Relation Prediction with Logical Reasoning Using Contrastive Representations

Relation prediction in knowledge graphs (KGs) aims at predicting missing relations in incomplete triples, whereas the dominant embedding paradigm has a restriction on handling unseen entities during testing.In the realworld scenario, the inductive setting is more common because entities in the training process are finite.Previous methods capture an inductive ability by implicit logic in KGs.However, it would be challenging to preciously acquire entity-independent relational semantics of compositional logic rules and to deal with the deficient supervision of logic caused by the scarcity of relational semantics.To this end, we propose a novel graph convolutional network (GCN)-based model LogCo with logical reasoning by contrastive representations.LogCo firstly extracts enclosing subgraphs and relational paths between two entities to supply the entity-independence.Then a contrastive strategy for relational path instances and the subgraph is proposed for the issue of deficient supervision.The contrastive representations are learned for a joint training regime.Finally, prediction results and logic rules for reasoning are attained.Comprehensive experiments on twelve inductive datasets show that LogCo achieves outstanding performance comparing with SOTA inductive baselines.
PromptEHR: Conditional Electronic Healthcare Records Generation with Prompt Learning

Accessing longitudinal multimodal Electronic Healthcare Records (EHRs) is challenging due to privacy concerns, which hinders the use of ML for healthcare applications.Synthetic EHRs generation bypasses the need to share sensitive real patient records.However, existing methods generate single-modal EHRs by unconditional generation or by longitudinal inference, which falls short of low flexibility and makes unrealistic EHRs.In this work, we propose to formulate EHRs generation as a text-to-text translation task by language models (LMs), which suffices to highly flexible event imputation during generation.We also design prompt learning to control the generation conditioned by numerical and categorical demographic features.We evaluate synthetic EHRs quality by two perplexity measures accounting for their longitudinal pattern (longitudinal imputation perplexity, lpl) and the connections cross modalities (cross-modality imputation perplexity, mpl).Moreover, we utilize two adversaries: membership and attribute inference attacks for privacy-preserving evaluation.Experiments on MIMIC-III data demonstrate the superiority of our methods on realistic EHRs generation (53.1% decrease of lpl and 45.3% decrease of mpl on average compared to the best baselines) with low privacy risks. 1 1 Software is available at https://github.com/RyanWangZf/PromptEHR.
E-NER — An Annotated Named Entity Recognition Corpus of Legal Text

Identifying named entities such as a person, location or organization, in documents can highlight key information to readers.Training Named Entity Recognition (NER) models requires an annotated data set, which can be a time-consuming labour-intensive task.Nevertheless, there are publicly available NER data sets for general English.Recently there has been interest in developing NER for legal text.However, prior work and experimental results reported here indicate that there is a significant degradation in performance when NER methods trained on a general English data set are applied to legal text.We describe a publicly available legal NER data set, called E-NER, based on legal company filings available from the US Securities and Exchange Commission's EDGAR data set.Training a number of different NER algorithms on the general English CoNLL-2003 corpus but testing on our test collection confirmed significant degradations in accuracy, as measured by the F1-score, of between 29.4% and 60.4%, compared to training and testing on the E-NER collection.
Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change

Recent research has revealed that neural language models at scale suffer from poor temporal generalization capability, i.e., language model pre-trained on static data from past years performs worse over time on emerging data.Existing methods mainly perform continual training to mitigate such a misalignment.While effective to some extent but is far from being addressed on both the language modeling and downstream tasks.In this paper, we empirically observe that temporal generalization is closely affiliated with lexical semantic change, which is one of the essential phenomena of natural languages.Based on this observation, we propose a simple yet effective lexical-level masking strategy to post-train a converged language model.Experiments on two pre-trained language models, two different classification tasks, and four benchmark datasets demonstrate the effectiveness of our proposed method over existing temporal adaptation methods, i.e., continual training with new data.Our code is available at https: //github.com/zhaochen0110/LMLM.
CLLE: A Benchmark for Continual Language Learning Evaluation in Multilingual Machine Translation

Continual Language Learning (CLL) in multilingual translation is inevitable when new languages are required to be translated.Due to the lack of unified and generalized benchmarks, the evaluation of existing methods is greatly influenced by experimental design which usually has a big gap from the industrial demands.In this work, we propose the first Continual Language Learning Evaluation benchmark CLLE in multilingual translation.CLLE consists of a Chinese-centric corpus -CN-25 and two CLL tasks -the close-distance language continual learning task and the language family continual learning task designed for real and disparate demands.Different from existing translation benchmarks, CLLE considers several restrictions for CLL, including domain distribution alignment, content overlap, language diversity, and the balance of corpus.Furthermore, we propose a novel framework COMETA based on Constrained Optimization and META-learning to alleviate catastrophic forgetting and dependency on historical training data by using a meta-model to retain the important parameters for old languages.Our experiments prove that CLLE is a challenging CLL benchmark and that our proposed method is effective when compared with other strong baselines.Due to the construction of corpus, the task designing and the evaluation method are independent of the central language, we also construct and release the English-centric corpus EN-25 to facilitate academic research 1 .
GNN-encoder: Learning a Dual-encoder Architecture via Graph Neural Networks for Dense Passage Retrieval

Recently, retrieval models based on dense representations are dominant in passage retrieval tasks, due to their outstanding ability in terms of capturing semantics of input text compared to the traditional sparse vector space models.A common practice of dense retrieval models is to exploit a dual-encoder architecture to represent a query and a passage independently.Though efficient, such a structure loses interaction between the query-passage pair, resulting in inferior accuracy.To enhance the performance of dense retrieval models without loss of efficiency, we propose a GNN-encoder model in which query (passage) information is fused into passage (query) representations via graph neural networks that are constructed by queries and their top retrieved passages.By this means, we maintain a dual-encoder structure, and retain some interaction information between querypassage pairs in their representations, which enables us to achieve both efficiency and efficacy in passage retrieval.Evaluation results indicate that our method significantly outperforms the existing models on MSMARCO, Natural Questions and TriviaQA datasets, and achieves the new state-of-the-art on these datasets.
Sequentially Controlled Text Generation

While GPT-2 generates sentences that are remarkably human-like, longer documents can ramble and do not follow human-like writing structure.We study the problem of imposing structure on long-range text.We propose a novel controlled text generation task, sequentially controlled text generation, and identify a dataset, NewsDiscourse as a starting point for this task.We develop a sequential controlled text generation pipeline with generation and editing.We test different degrees of structural awareness and show that, in general, more structural awareness results in higher controlaccuracy, grammaticality, coherency and topicality, approaching human-level writing performance.
Invernet: An Inversion Attack Framework to Infer Fine-Tuning Datasets through Word Embeddings

Word embedding aims to learn the dense representation of words and has become a regular input preparation in many NLP tasks.Due to the data and computation intensive nature of learning embeddings from scratch, a more affordable way is to borrow the pretrained embedding available in public and fine-tune the embedding through a domain specific downstream dataset.A privacy concern can arise if a malicious owner of the pretrained embedding gets access to the fine-tuned embedding and tries to infer the critical information from the downstream datasets.In this study, we propose a novel embedding inversion framework called Invernet that materializes the privacy concern by inferring the context distribution in the downstream dataset, which can lead to key information breach.With extensive experimental studies on two real-world news datasets: Antonio Gulli's News and New York Times, we validate the feasibility of proposed privacy attack and demonstrate the effectiveness of Invernet on inferring downstream datasets based on multiple word embedding methods.
GPS: Genetic Prompt Search for Efficient Few-Shot Learning

Prompt-based techniques have demostrated great potential for improving the few-shot generalization of pretrained language models. However, their performance heavily relies on the manual design of prompts and thus requiring a lot of human efforts. In this paper, we introduce Genetic Prompt Search (GPS) to improve few-shot learning with prompts, which utilizes a genetic algorithm to automatically search for the best prompt.GPS is gradient-free and requires no update of model parameters but only a small validation set. Experiments on diverse datasets proved the effectiveness of GPS, which outperforms manual prompts by a large margin of 2.6 points. Our method is also better than other parameter-efficient tuning methods such as prompt tuning.