Reviewerly/exHarmony huggingface dataset:
Check-COVID: Fact-Checking COVID-19 News Claims with Scientific Evidence

We present a new fact-checking benchmark, Check-COVID, that requires systems to verify claims about COVID-19 from news using evidence from scientific articles.This approach to fact-checking is particularly challenging as it requires checking internet text written in everyday language against evidence from journal articles written in formal academic language.Check-COVID contains 1, 504 expertannotated news claims about the coronavirus paired with sentence-level evidence from scientific journal articles and veracity labels.It includes both extracted (journalist-written) and composed (annotator-written) claims.Experiments using both a fact-checking specific system and GPT-3.5, which respectively achieve F1 scores of 76.99 and 69.90 on this task, reveal the difficulty of automatically fact-checking both claim types and the importance of indomain data for good performance.Our data and models are released publicly at https: //github.com/posuer/Check-COVID.
Few Shot Rationale Generation using Self-Training with Dual Teachers

Self-rationalizing models that also generate a free-text explanation for their predicted labels are an important tool to build trustworthy AI applications.Since generating explanations for annotated labels is a laborious and costly process, recent models rely on large pretrained language models (PLMs) as their backbone and few-shot learning.In this work we explore a self-training approach leveraging both labeled and unlabeled data to further improve few-shot models, under the assumption that neither human written rationales nor annotated task labels are available at scale.We introduce a novel dual-teacher learning framework, which learns two specialized teacher models for task prediction and rationalization using self-training and distills their knowledge into a multi-tasking student model that can jointly generate the task label and rationale.Furthermore, we formulate a new loss function, Masked Label Regularization (MLR) which promotes explanations to be strongly conditioned on predicted labels.Evaluation on three public datasets demonstrate that the proposed methods are effective in modeling task labels and generating faithful rationales.
Exploiting Biased Models to De-bias Text: A Gender-Fair Rewriting Model

Natural language generation models reproduce and often amplify the biases present in their training data.Previous research explored using sequence-to-sequence rewriting models to transform biased model outputs (or original texts) into more gender-fair language by creating pseudo training data through linguistic rules.However, this approach is not practical for languages with more complex morphology than English.We hypothesise that creating training data in the reverse direction, i.e. starting from gender-fair text, is easier for morphologically complex languages and show that it matches the performance of state-of-the-art rewriting models for English.To eliminate the rule-based nature of data creation, we instead propose using machine translation models to create gender-biased text from real gender-fair text via round-trip translation.Our approach allows us to train a rewriting model for German without the need for elaborate handcrafted rules.The outputs of this model increased genderfairness as shown in a human evaluation study. 1
CICL_DMS at SemEval-2023 Task 11: Learning With Disagreements (Le-Wi-Di)

In this system paper, we describe our submission for the 11 th task of SemEval2023: Learning with Disagreements, or Le-Wi-Di for short.In the task, the assumption that there is a single gold label in NLP tasks such as hate speech or misogyny detection is challenged, and instead the opinions of multiple annotators are considered.The goal is instead to capture the agreements/disagreements of the annotators.For our system, we utilize the capabilities of modern large-language models as our backbone and investigate various techniques built on top, such as ensemble learning, multi-task learning, or Gaussian processes.Our final submission shows promising results and we achieve an upper-half finish.
Neural Machine Translation Methods for Translating Text to Sign Language Glosses

State-of-the-art techniques common to low resource Machine Translation (MT) are applied to improve MT of spoken language text to Sign Language (SL) glosses.In our experiments, we improve the performance of the transformer-based models via (1) data augmentation, (2) semi-supervised Neural Machine Translation (NMT), (3) transfer learning and (4) multilingual NMT.The proposed methods are implemented progressively on two German SL corpora containing gloss annotations.Multilingual NMT combined with data augmentation appear to be the most successful setting, yielding statistically significant improvements as measured by three automatic metrics (up to over 6 points BLEU), and confirmed via human evaluation.Our best setting outperforms all previous work that report on the same test-set and is also confirmed on a corpus of the American Sign Language (ASL).
SimOAP: Improve Coherence and Consistency in Persona-based Dialogue Generation via Over-sampling and Post-evaluation

Language models trained on large-scale corpora can generate remarkably fluent results in open-domain dialogue.However, for the persona-based dialogue generation task, consistency and coherence are also key factors, which are great challenges for language models.Existing works mainly focus on valuable data filtering, model structure modifying, or objective function designing, while their improvements are limited and hard to generalize to all types of pre-trained language models.However, we find that language models can produce consistent and coherent responses if we consider enough generations.Thus, the problems lay in largescale response generation and target response selection.In this work, a simple but effective two-stage SimOAP strategy is proposed, i.e., over-sampling and post-evaluation.The over-sampling stage takes large-scale responses from existing trained models efficiently via offthe-shelf distilling and compressing methods, and the post-evaluation stage selects a good response based on multiple well-designed evaluation metrics from large-scale candidates.Experimental results show that the proposed plug-in SimOAP strategy improves the backbone models and outperforms the baseline strategies in both automatic and human evaluations.
Choosing What to Mask: More Informed Masking for Multimodal Machine Translation

Pre-trained language models have achieved remarkable results on several NLP tasks.Most of them adopt masked language modeling to learn representations by randomly masking tokens and predicting them based on their context.However, this random selection of tokens to be masked is inefficient to learn some language patterns as it may not consider linguistic information that can be helpful for many NLP tasks, such as multimodal machine translation (MMT).Hence, we propose three novel masking strategies for cross-lingual visual pre-training -more informed visual masking, more informed textual masking, and more informed visual and textual masking -each one focusing on learning different linguistic patterns.We apply them to Vision Translation Language Modelling for video subtitles (Sato et al., 2022) and conduct extensive experiments on the Portuguese-English MMT task.The results show that our masking approaches yield significant improvements over the original random masking strategy for downstream MMT performance.Our models outperform the MMT baseline and we achieve state-of-theart accuracy (52.70 in terms of BLEU score) on the How2 dataset, indicating that more informed masking helps in acquiring an understanding of specific language structures and has great potential for language understanding 1 .
SIMSUM: Document-level Text Simplification via Simultaneous Summarization

Document-level text simplification is a specific type of simplification which involves simplifying documents consisting of several sentences by rewriting them into fewer or more sentences.In this paper, we propose a new two-stage framework SIMSUM for automated document-level text simplification.Our model is designed with explicit summarization and simplification models and guides the generation using the main keywords of a source text.In order to evaluate our new model, we use two existing benchmark datasets for simplification, namely D-Wikipedia and Wiki-Doc.We compare our model's performance with state of the art and show that SIMSUM achieves top results on the D-Wikipedia dataset SARI (+1.20),D-SARI (+1.64), and FKGL (-0.35) scores, improving over the best baseline models.In order to evaluate the quality of the generated text, we analyze the outputs from different models qualitatively and demonstrate the merit of our new model.Our code and datasets are available 1 .
Pseudo Outlier Exposure for Out-of-Distribution Detection using Pretrained Transformers

For real-world language applications, detecting an out-of-distribution (OOD) sample is helpful to alert users or reject such unreliable samples.However, modern over-parameterized language models often produce overconfident predictions for both in-distribution (ID) and OOD samples.In particular, language models suffer from OOD samples with a similar semantic representation to ID samples since these OOD samples lie near the ID manifold.A rejection network can be trained with ID and diverse outlier samples to detect test OOD samples, but explicitly collecting auxiliary OOD datasets brings an additional burden for data collection.In this paper, we propose a simple but effective method called Pseudo Outlier Exposure (POE) that constructs a surrogate OOD dataset by sequentially masking tokens related to ID classes.The surrogate OOD sample introduced by POE shows a similar representation to ID data, which is most effective in training a rejection network.Our method does not require any external OOD data and can be easily implemented within off-the-shelf Transformers.A comprehensive comparison with state-of-theart algorithms demonstrates POE's competitiveness on several text classification benchmarks.
JHU IWSLT 2023 Multilingual Speech Translation System Description

We describe the Johns Hopkins ACL 60-60 Speech Translation systems submitted to the IWSLT 2023 Multilingual track, where we were tasked to translate ACL presentations from English into 10 languages.We developed cascaded speech translation systems for both the constrained and unconstrained subtracks.Our systems make use of pre-trained models as well as domain-specific corpora for this highly technical evaluation-only task.We find that the specific technical domain which ACL presentations fall into presents a unique challenge for both ASR and MT, and we present an error analysis and an ACL-specific corpus we produced to enable further work in this area.
Learning to Generalize for Cross-domain QA

There have been growing concerns regarding the out-of-domain generalization ability of natural language processing (NLP) models, particularly in question-answering (QA) tasks.Current synthesized data augmentation methods for QA are hampered by increased training costs.To address this issue, we propose a novel approach that combines prompting methods and linear probing then fine-tuning strategy, which does not entail additional cost.Our method has been theoretically and empirically shown to be effective in enhancing the generalization ability of both generative and discriminative models.Our approach outperforms state-of-the-art baselines, with an average increase in F1 score of 4.5%-7.9%.Furthermore, our method can be easily integrated into any pre-trained models and offers a promising solution to the underexplored cross-domain QA task.We release our source code at Github * .
Multi-Relational Probabilistic Event Representation Learning via Projected Gaussian Embedding

Event representation learning has been shown beneficial in various downstream tasks.Current event representation learning methods, which mainly focus on capturing the semantics of events via deterministic vector embeddings, have made notable progress.However, they ignore two important properties: the multiple relations between events and the uncertainty within events.In this paper, we propose a novel approach to learning multi-relational probabilistic event embeddings based on contrastive learning.Specifically, the proposed method consists of three major modules, a multi-relational event generation module to automatically generate multi-relational training data, a probabilistic event encoding module to model uncertainty of events by Gaussian density embeddings, and a relation-aware projection module to adapt unseen relations by projecting Gaussian embeddings into relationaware subspaces.Moreover, a novel contrastive learning loss is elaborately designed for learning the multi-relational probabilistic embeddings.Since the existing benchmarks for event representation learning ignore relations and uncertainty of events, a novel dataset named MR-PES is constructed to investigate whether multiple relations between events and uncertainty within events are learned.Experimental results show that the proposed approach outperforms other state-of-the-art baselines on both existing and newly constructed datasets.
Weaker Than You Think: A Critical Look at Weakly Supervised Learning

Weakly supervised learning is a popular approach for training machine learning models in low-resource settings.Instead of requesting high-quality yet costly human annotations, it allows training models with noisy annotations obtained from various weak sources.Recently, many sophisticated approaches have been proposed for robust training under label noise, reporting impressive results.In this paper, we revisit the setup of these approaches and find that the benefits brought by these approaches are significantly overestimated.Specifically, we find that the success of existing weakly supervised learning approaches heavily relies on the availability of clean validation samples which, as we show, can be leveraged much more efficiently by simply training on them.After using these clean labels in training, the advantages of using these sophisticated approaches are mostly wiped out.This remains true even when reducing the size of the available clean data to just five samples per class, making these approaches impractical.To understand the true value of weakly supervised learning, we thoroughly analyze diverse NLP datasets and tasks to ascertain when and why weakly supervised approaches work.Based on our findings, we provide recommendations for future research.
FII_Better at SemEval-2023 Task 2: MultiCoNER II Multilingual Complex Named Entity Recognition

The "MultiCoNER II Multilingual Complex Named Entity Recognition" task 1 at SemEval 2023 competition focuses on identifying complex named entities (NEs), such as the titles on creative works (songs, books, movies), in several languages.In the context of SemEval, our team, FII_Better, presents an exploration of a base transformer model's capabilities regarding the task, focused more specifically on five languages (English, Spanish, Swedish, German, Italian).We take DistilBERT and BERT as two examples of basic transformer models, using DistilBERT as a baseline and BERT as the platform to create an improved model.In this process, we learned a lot about working with transformers and, on top of that, we managed to get fair results on the chosen languages.
I2C-Huelva at SemEval-2023 Task 9: Analysis of Intimacy in Multilingual Tweets Using Resampling Methods and Transformers

Nowadays, intimacy is a fundamental aspect of how we relate to other people in social settings.The most frequent way in which we can determine a high level of intimacy is in the use of certain emoticons, curse words, verbs, etc.This paper presents the approach developed to solve SemEval 2023 task 9: Multiligual Tweet Intimacy Analysis.To address the task, a transferlearning approach was conducted by finetuning various pre-trained language models.Since the dataset supplied by the organizer was highly imbalanced, our main strategy to obtain high prediction values was the implementation of round-trip translation for oversampling and a random approach for undersampling on the training set.Our final submission achieved an overall Pearson's r of 0.497.
Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step

Chain-of-thought prompting (e.g., "Let's think step-by-step") primes large language models to verbalize rationalization for their predictions.While chain-of-thought can lead to dramatic performance gains, benefits appear to emerge only for sufficiently large models (beyond 50B parameters).We show that ordersof-magnitude smaller models (125M-1.3Bparameters) can still benefit from chain-ofthought prompting.To achieve this, we introduce Symbolic Chain-of-Thought Distillation (SCoTD), a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model.Experiments across several commonsense benchmarks show that: 1) SCoTD enhances the performance of the student model in both supervised and few-shot settings, and especially for challenge sets; 2) sampling many reasoning chains per instance from the teacher is paramount; and 3) after distillation, student chain-of-thoughts are judged by humans as comparable to the teacher, despite orders of magnitude fewer parameters.We test several hypotheses regarding what properties of chain-of-thought samples are important, e.g., diversity vs. teacher likelihood vs. open-endedness.We release our corpus of chain-of-thought samples and code.
NLP Reproducibility For All: Understanding Experiences of Beginners

As natural language processing (NLP) has recently seen an unprecedented level of excitement, and more people are eager to enter the field, it is unclear whether current research reproducibility efforts are sufficient for this group of beginners to apply the latest developments.To understand their needs, we conducted a study with 93 students in an introductory NLP course, where students reproduced the results of recent NLP papers.Surprisingly, we find that their programming skill and comprehension of research papers have a limited impact on their effort spent completing the exercise.Instead, we find accessibility efforts by research authors to be the key to success, including complete documentation, better coding practice, and easier access to data files.Going forward, we recommend that NLP researchers pay close attention to these simple aspects of open-sourcing their work, and use insights from beginners' feedback to provide actionable ideas on how to better support them.
SenteCon: Leveraging Lexicons to Learn Human-Interpretable Language Representations

Although deep language representations have become the dominant form of language featurization in recent years, in many settings it is important to understand a model's decisionmaking process.This necessitates not only an interpretable model but also interpretable features.In particular, language must be featurized in a way that is interpretable while still characterizing the original text well.We present SEN-TECON, a method for introducing human interpretability in deep language representations.Given a passage of text, SENTECON encodes the text as a layer of interpretable categories in which each dimension corresponds to the relevance of a specific category.Our empirical evaluations indicate that encoding language with SENTECON provides high-level interpretability at little to no cost to predictive performance on downstream tasks.Moreover, we find that SENTECON outperforms existing interpretable language representations with respect to both its downstream performance and its agreement with human characterizations of the text.
CELDA: Leveraging Black-box Language Model as Enhanced Classifier without Labels

Utilizing language models (LMs) without internal access is becoming an attractive paradigm in the field of NLP as many cutting-edge LMs are released through APIs and boast a massive scale.The de-facto method in this type of black-box scenario is known as prompting, which has shown progressive performance enhancements in situations where data labels are scarce or unavailable.Despite their efficacy, they still fall short in comparison to fully supervised counterparts and are generally brittle to slight modifications.In this paper, we propose Clustering-Enhanced Linear Discriminative Analysis (CELDA), a novel approach that improves the text classification accuracy with a very weak-supervision signal (i.e., name of the labels).Our framework draws a precise decision boundary without accessing weights or gradients of the LM model or data labels.The core ideas of CELDA are twofold: (1) extracting a refined pseudo-labeled dataset from an unlabeled dataset, and (2) training a lightweight and robust model on the top of LM, which learns an accurate decision boundary from an extracted noisy dataset.Throughout in-depth investigations on various datasets, we demonstrated that CELDA reaches new state-of-theart in weakly-supervised text classification and narrows the gap with a fully-supervised model.Additionally, our proposed methodology can be applied universally to any LM and has the potential to scale to larger models, making it a more viable option for utilizing large LMs.
I2R at SemEval-2023 Task 7: Explanations-driven Ensemble Approach for Natural Language Inference over Clinical Trial Data

In this paper, we describe our system for SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data. Given a CTR premise, and a statement, this task involves 2 sub-tasks (i) identifying the inference relation between CTR - statement pairs (Task 1: Textual Entailment), and (ii) extracting a set of supporting facts, from the premise, to justify the label predicted in Task 1 (Task 2: Evidence Retrieval). We adopt an explanations driven NLI approach to tackle the tasks. Given a statement to verify, the idea is to first identify relevant evidence from the target CTR(s), perform evidence level inferences and then ensemble them to arrive at the final inference. We have experimented with various BERT based models and T5 models. Our final model uses T5 base that achieved better performance compared to BERT models. In summary, our system achieves F1 score of 70.1% for Task 1 and 80.2% for Task 2. We ranked 8th respectively under both the tasks. Moreover, ours was one of the 5 systems that ranked within the Top 10 under both tasks.