Reviewerly/exHarmony huggingface dataset:
FII SMART at SemEval 2023 Task7: Multi-evidence Natural Language Inference for Clinical Trial Data

The "Multi-evidence Natural Language Inference for Clinical Trial Data" task at SemEval 2023 competition focuses on extracting essential information on clinical trial data, by posing two subtasks on textual entailment and evidence retrieval.In the context of SemEval, we present a comparison between a method based on the BioBERT model and a CNN model.The task is based on a collection of breast cancer Clinical Trial Reports (CTRs), statements, explanations, and labels annotated by domain expert annotators.We achieved F1 scores of 0.69 for determining the inference relation (entailment vs contradiction) between CTR -statement pairs.The implementation of our system is made available via Github 1 .
Uni-Encoder: A Fast and Accurate Response Selection Paradigm for Generation-Based Dialogue Systems

Sample-and-rank is a key decoding strategy for modern generation-based dialogue systems.It helps achieve diverse and high-quality responses by selecting an answer from a small pool of generated candidates.The current stateof-the-art ranking methods mainly use an encoding paradigm called Cross-Encoder, which separately encodes each context-candidate pair and ranks the candidates according to their fitness scores.However, Cross-Encoder repeatedly encodes the same lengthy context for each candidate, resulting in high computational costs.Poly-Encoder addresses the above problems by reducing the interaction between context and candidates, but with a price of performance drop.In this work, we develop a new paradigm called Uni-Encoder 1 , that keeps the full attention over each pair as in Cross-Encoder while only encoding the context once, as in Poly-Encoder.Uni-Encoder encodes all the candidates with the context in one forward pass.We use the same positional embedding for all candidates to ensure they are treated equally and design a new attention mechanism to avoid confusion.Our Uni-Encoder can simulate other ranking paradigms using different attention and response concatenation methods.Extensive experiments show that our proposed paradigm achieves new state-of-theart results on four benchmark datasets with high computational efficiency.For instance, it improves R 10 @1 by 2.9% with an approximately 4× faster inference speed on the Ubuntu V2 dataset.
X-RiSAWOZ: High-Quality End-to-End Multilingual Dialogue Datasets and Few-shot Agents

Task-oriented dialogue research has mainly focused on a few popular languages like English and Chinese, due to the high dataset creation cost for a new language.To reduce the cost, we apply manual editing to automatically translated data.We create a new multilingual benchmark, X-RiSAWOZ, by translating the Chinese RiSAWOZ to 4 languages: English, French, Hindi, Korean; and a code-mixed English-Hindi language.X-RiSAWOZ has more than 18,000 human-verified dialogue utterances for each language, and unlike most multilingual prior work, is an end-to-end dataset for building fully-functioning agents.The many difficulties we encountered in creating X-RiSAWOZ led us to develop a toolset to accelerate the post-editing of a new language dataset after translation.This toolset improves machine translation with a hybrid entity alignment technique that combines neural with dictionary-based methods, along with many automated and semi-automated validation checks.We establish strong baselines for X-RiSAWOZ by training dialogue agents in the zero-and few-shot settings where limited gold data is available in the target language.Our results suggest that our translation and post-editing methodology and toolset can be used to create new high-quality multilingual dialogue agents cost-effectively.Our dataset, code, and toolkit are released open-source. 1
Universal Information Extraction with Meta-Pretrained Self-Retrieval

Universal Information Extraction (Universal IE) aims to solve different extraction tasks in a uniform text-to-structure generation manner.Such a generation procedure tends to struggle when there exist complex information structures to be extracted.Retrieving knowledge from external knowledge bases may help models to overcome this problem but it is impossible to construct a knowledge base suitable for various IE tasks.Inspired by the fact that large amount of knowledge are stored in the pretrained language models (PLM) and can be retrieved explicitly, in this paper, we propose MetaRetriever to retrieve task-specific knowledge from PLMs to enhance universal IE.As different IE tasks need different knowledge, we further propose a Meta-Pretraining Algorithm which allows MetaRetriever to quicktly achieve maximum task-specific retrieval performance when fine-tuning on downstream IE tasks.Experimental results show that MetaRetriever achieves the new state-of-the-art on 4 IE tasks, 12 datasets under fully-supervised, low-resource and few-shot scenarios.
Learning Action Conditions from Instructional Manuals for Instruction Understanding

The ability to infer pre-and postconditions of an action is vital for comprehending complex instructions, and is essential for applications such as autonomous instruction-guided agents and assistive AI that supports humans to perform physical tasks.In this work, we propose a task dubbed action condition inference, which extracts mentions of preconditions and postconditions of actions in instructional manuals.We propose a weakly supervised approach utilizing automatically constructed large-scale training instances from online instructions, and curate a densely human-annotated and validated dataset to study how well the current NLP models do on the proposed task.We design two types of models differ by whether contextualized and global information is leveraged, as well as various combinations of heuristics to construct the weak supervisions.Our experiments show a >20% F1-score improvement with considering the entire instruction contexts and a > 6% F1-score benefit with the proposed heuristics.However, the best performing model is still well-behind human performance. 1Heuristics Examples Descriptions Entity-Tracing & Coref.… Slice 500 grams of onions.… … Heat the pan with olive oil.… … Place them in the frying pan.… Precondition 1 Precondition 2The shared entities are pan and onions (linked via co-references to them).
How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese

This paper investigates the effect of tokenizers on the downstream performance of pretrained language models (PLMs) in scriptio continua languages where no explicit spaces exist between words, using Japanese as a case study.The tokenizer for such languages often consists of a morphological analyzer and a subword tokenizer, requiring us to conduct a comprehensive study of all possible pairs.However, previous studies lack this comprehensiveness.We therefore train extensive sets of tokenizers, build a PLM using each, and measure the downstream performance on a wide range of tasks.Our results demonstrate that each downstream task has a different optimal morphological analyzer, and that it is better to use Byte-Pair-Encoding or Unigram rather than WordPiece as a subword tokenizer, regardless of the type of task.
Text Augmentation Using Dataset Reconstruction for Low-Resource Classification

In the deployment of real-world text classification models, label scarcity is a common problem.As the number of classes increases, this problem becomes even more complex.One way to address this problem is by applying text augmentation methods.One of the more prominent methods involves using the text-generation capabilities of language models.We propose Text AUgmentation by Dataset Reconstruction (TAU-DR), a novel method of data augmentation for text classification.We conduct experiments on several multi-class datasets, showing that our approach improves the current state-of-the-art techniques for data augmentation.
Building a Buzzer-quiz Answering System

A buzzer quiz is a genre of quiz in which multiple players simultaneously listen to a quiz being read aloud and respond it by buzzing in as soon as they can predict the answer.Because incorrect answers often result in penalties, a buzzer-quiz answering system must not only predict the answer from only part of a question but also estimate the predicted answer's accuracy.In this paper, we introduce two types of buzzer-quiz answering systems: (1) a system that directly generates an answer from part of a question by using an autoregressive language model; and (2) a system that first reconstructs the entire question by using an autoregressive language model and then determines the answer according to the reconstructed question.We then propose a method to estimate the accuracy of the answers for each system by using the internal scores of each model.
Context-Aware Transformer Pre-Training for Answer Sentence Selection

Answer Sentence Selection (AS2) is a core component for building an accurate Question Answering pipeline.AS2 models rank a set of candidate sentences based on how likely they answer a given question.The state of the art in AS2 exploits pre-trained transformers by transferring them on large annotated datasets, while using local contextual information around the candidate sentence.In this paper, we propose three pre-training objectives designed to mimic the downstream fine-tuning task of contextual AS2.This allows for specializing LMs when fine-tuning for contextual AS2.Our experiments on three public and two large-scale industrial datasets show that our pre-training approaches (applied to RoBERTa and ELECTRA) can improve baseline contextual AS2 accuracy by up to 8% on some datasets.
PREADD: Prefix-Adaptive Decoding for Controlled Text Generation

We propose Prefix-Adaptive Decoding (PREADD), a flexible method for controlled text generation.Unlike existing methods that use auxiliary expert models to control for attributes, PREADD does not require an external model, instead relying on linearly combining output logits from multiple prompts.Specifically, PREADD contrasts the output logits generated using a raw prompt against those generated using a prefix-prepended prompt, enabling both positive and negative control with respect to any attribute encapsulated by the prefix.We evaluate PREADD on three tasks-toxic output mitigation, gender bias reduction, and sentiment control-and find that PREADD outperforms not only prompting baselines, but also an auxiliary-expert control method, by 12% or more in relative gain on our main metrics for each task.
Benchmarking Diverse-Modal Entity Linking with Generative Models

Entities can be expressed in diverse formats, such as texts, images, or column names and cell values in tables.While existing entity linking (EL) models work well on per modality configuration, such as text-only EL, visual grounding, or schema linking, it is more challenging to design a unified model for diverse modality configurations.To bring various modality configurations together, we constructed a benchmark for diverse-modal EL (DMEL) from existing EL datasets, covering all three modalities including text, image, and table.To approach the DMEL task, we proposed a generative diverse-modal model (GDMM) following a multimodal-encoder-decoder paradigm.Pre-training GDMM with rich corpora builds a solid foundation for DMEL without storing the entire KB for inference.Fine-tuning GDMM builds a stronger DMEL baseline, outperforming state-of-the-art task-specific EL models by 8.51 F1 score on average.Additionally, extensive error analyses are conducted to highlight the challenges of DMEL, facilitating future research on this task.
Temporal Relation Classification using Boolean Question Answering

Classifying temporal relations between a pair of events is crucial to natural language understanding and a well-known natural language processing task.Given a document and two event mentions, the task is aimed at finding which one started first.We propose an efficient approach for temporal relation classification (TRC) using a boolean question answering (QA) model which we fine-tune on questions that we carefully design based on the TRC annotation guidelines, thereby mimicking the way human annotators approach the task.Our new QA-based TRC model outperforms previous state-of-the-art results by 2.4%.
Stanford MLab at SemEval 2023 Task 7: Neural Methods for Clinical Trial Report NLI

We present a system for natural language inference in breast cancer clinical trial reports, as framed by SemEval 2023 Task 7: Multievidence Natural Language Inference for Clinical Trial Data.In particular, we propose a suite of techniques for two related inference subtasks: entailment and evidence retrieval.The purpose of the textual entailment identification subtask is to determine the inference relation (either entailment or contradiction) between given statement pairs, while the goal of the evidence retrieval task is to identify a set of sentences that support this inference relation.To this end, we propose fine-tuning Bio+Clinical BERT, a BERT-based model pre-trained on clinical data.Along with presenting our system, we analyze our architectural decisions in the context of our model's accuracy and conduct an error analysis.Overall, our system ranked 20 / 40 on the entailment subtask.
On the Interpretability and Significance of Bias Metrics in Texts: a PMI-based Approach

In recent years, word embeddings have been widely used to measure biases in texts.Even if they have proven to be effective in detecting a wide variety of biases, metrics based on word embeddings lack transparency and interpretability.We analyze an alternative PMIbased metric to quantify biases in texts.It can be expressed as a function of conditional probabilities, which provides a simple interpretation in terms of word co-occurrences.We also prove that it can be approximated by an odds ratio, which allows estimating confidence intervals and statistical significance of textual biases.This approach produces similar results to metrics based on word embeddings when capturing gender gaps of the real world embedded in large corpora. 1
Codex to corpus: Exploring annotation and processing for an open and extensible machine-readable edition of the Florentine Codex

This paper describes an ongoing effort to create, from the original hand-written text, a machine-readable, linguistically-annotated, and easily-searchable corpus of the Nahuatl portion of the Florentine Codex, a 16 th century Mesoamerican manuscript written in Nahuatl and Spanish.The Codex consists of 12 books and over 300,000 tokens.We describe the process of annotating 3 of these books, the steps of text preprocessing undertaken, our approach to efficient manual processing and annotation, and some of the challenges faced along the way.We also report on a set of experiments evaluating our ability to automate the text processing tasks to aid in the remaining annotation effort, and find the results promising despite the relatively low volume of training data.Finally, we briefly present a real use case from the humanities that would benefit from the searchable, linguistically annotated corpus we describe.
Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer

Cross-lingual natural language inference is a fundamental problem in cross-lingual language understanding.Many recent works have used prompt learning to address the lack of annotated parallel corpora in XNLI.However, these methods adopt discrete prompting by simply translating the templates to the target language and need external expert knowledge to design the templates.Besides, discrete prompts of human-designed template words are not trainable vectors and can not be migrated to target languages in the inference stage flexibly. In this paper, we propose a novel Soft prompt learning framework with the Multilingual Verbalizer (SoftMV) for XNLI. SoftMV first constructs cloze-style question with soft prompts for the input sample.Then we leverage bilingual dictionaries to generate an augmented multilingual question for the original question. SoftMV adopts a multilingual verbalizer to align the representations of original and augmented multilingual questions into a unified semantic space with consistency regularization.Experimental results on XNLI demonstrate that SoftMV can achieve state-of-the-art performance and significantly outperform the previous methods under the few-shot and full-shot cross-lingual transfer settings.
Multilingual Knowledge Graph Completion with Language-Sensitive Multi-Graph Attention

Multilingual Knowledge Graph Completion (KGC) aims to predict missing links with multilingual knowledge graphs.However, existing approaches suffer from two main drawbacks: (a) alignment dependency: the multilingual KGC is always realized with joint entity or relation alignment, which introduces additional alignment models and increases the complexity of the whole framework; (b) training inefficiency: the trained model will only be used for the completion of one target KG, although the data from all KGs are used simultaneously.To address these drawbacks, we propose a novel multilingual KGC framework with languagesensitive multi-graph attention such that the missing links on all given KGs can be inferred by a universal knowledge completion model.Specifically, we first build a relational graph neural network by sharing the embeddings of aligned nodes to transfer language-independent knowledge.Meanwhile, a language-sensitive multi-graph attention (LSMGA) is proposed to deal with the information inconsistency among different KGs.Experimental results show that our model achieves significant improvements on the DBP-5L and E-PKG datasets.
A Robust Information-Masking Approach for Domain Counterfactual Generation

Domain shift is a big challenge in NLP, thus, many approaches resort to learning domaininvariant features to mitigate the inference phase domain shift.Such methods, however, fail to leverage the domain-specific nuances relevant to the task at hand.To avoid such drawbacks, domain counterfactual generation aims to transform a text from the source domain to a given target domain.However, due to the limited availability of data, such frequency-based methods often miss and lead to some valid and spurious domain-token associations.Hence, we employ a three-step domain obfuscation approach that involves frequency and attention norm-based masking, to mask domain-specific cues, and unmasking to regain the domain generic context.Our experiments empirically show that the counterfactual samples sourced from our masked text lead to improved domain transfer on 10 out of 12 domain sentiment classification settings, with an average of 2% accuracy improvement over the state-of-the-art for unsupervised domain adaptation (UDA).Further, our model outperforms the state-of-the-art by achieving 1.4% average accuracy improvement in the adversarial domain adaptation (ADA) setting.Moreover, our model also shows its domain adaptation efficacy on a large multi-domain intent classification dataset where it attains state-of-the-art results.We release the codes publicly at https: //github.com/declare-lab/remask.
APA-RST: A Text Simplification Corpus with RST Annotations

We present a corpus of parallel Germanlanguage simplified newspaper articles.The articles have been aligned at sentence level and annotated according to the Rhetorical Structure Theory (RST) framework.These RST annotated texts could shed light on structural aspects of text complexity and how simplifications work on a text level.
Negation Scope Refinement via Boundary Shift Loss

Negation in language may affect many NLP applications, e.g., information extraction and sentiment analysis.The key sub-task of negation detection is negation scope resolution which aims to extract the portion of a sentence that is being negated by a negation cue (e.g., keyword "not" and "never") in the sentence.Due to the long spans, existing methods tend to make wrong predictions around the scope boundaries.In this paper, we propose a simple yet effective model named R-BSL which engages the Boundary Shift Loss to refine the predicted boundary. 1On multiple benchmark datasets, we show that the extremely simple R-BSL achieves best results.
The distribution of discourse relations within and across turns in spontaneous conversation

Time pressure and topic negotiation may impose constraints on how people leverage discourse relations (DRs) in spontaneous conversational contexts.In this work, we adapt a system of DRs for written language to spontaneous dialogue using crowdsourced annotations from novice annotators.We then test whether discourse relations are used differently across several types of multi-utterance contexts.We compare the patterns of DR annotation within and across speakers and within and across turns.Ultimately, we find that different discourse contexts produce distinct distributions of discourse relations, with single-turn annotations creating the most uncertainty for annotators.Additionally, we find that the discourse relation annotations are of sufficient quality to predict from embeddings of discourse units.Relation Discourse Units Acknowledgement A: || it starts recording now.|| B: || Okay.|| Background A: || I'm, we're originally from another state || and I know || in the state we were from that they did that t-, similar type thing.|| Clarification Question A: || We live in the Saginaw area.|| B: || Saginaw?|| Comment B: || They seem to be having a real good response.|| A: || That's pretty good.|| Continuation A: || I work off and on just temporarily || and usually find friends to babysit, || Contrast A: || I don't work, though, || but I used to work and, || Elaboration A: || in the state we were from that they did that t-, similar type thing.|| The city brought ought, || you know, || set tr-, separate trash cans || and you separated your stuff || Explanation A: || and they discontinued them || because people were coming and dumping their trash in them.|| Narration A: || and you put it in there || and they took it, || Question-Answer Pair B: || Saginaw?|| A: || Uh-huh.||Result B: || No, || I just, I noticed || it Iowa and other cities like that, it's a nickel per aluminum can.
On Evaluating Multilingual Compositional Generalization with Translated Datasets

Compositional generalization allows efficient learning and human-like inductive biases.Since most research investigating compositional generalization in NLP is done on English, important questions remain underexplored.Do the necessary compositional generalization abilities differ across languages?Can models compositionally generalize crosslingually?As a first step to answering these questions, recent work used neural machine translation to translate datasets for evaluating compositional generalization in semantic parsing.However, we show that this entails critical semantic distortion.To address this limitation, we craft a faithful rule-based translation of the MCWQ dataset (Cui et al., 2022) from English to Chinese and Japanese.Even with the resulting robust benchmark, which we call MCWQ-R, we show that the distribution of compositions still suffers due to linguistic divergences, and that multilingual models still struggle with cross-lingual compositional generalization.Our dataset and methodology will be useful resources for the study of cross-lingual compositional generalization in other tasks. 1