Reviewerly/exHarmony huggingface dataset:
NeuroX Library for Neuron Analysis of Deep NLP Models

Neuron analysis provides insights into how knowledge is structured in representations and discovers the role of neurons in the network.In addition to developing an understanding of our models, neuron analysis enables various applications such as debiasing, domain adaptation and architectural search.We present NeuroX, a comprehensive open-source toolkit to conduct neuron analysis of natural language processing models.It implements various interpretation methods under a unified API, and provides a framework for data processing and evaluation, thus making it easier for researchers and practitioners to perform neuron analysis.The Python toolkit is available at https://www.github.com/fdalvi/NeuroX.
DIAGRAPH: An Open-Source Graphic Interface for Dialog Flow Design

In this work, we present DIAGRAPH, an opensource 1 graphical dialog flow editor built on the ADVISER toolkit.Our goal for this tool is threefold: 1) To support subject-experts to intuitively create complex and flexible dialog systems, 2) To support rapid prototyping of dialog system behavior, e.g., for research, and 3) To provide a hands-on test bed for students learning about dialog systems.To facilitate this, DIAGRAPH aims to provide a clean and intuitive graphical interface for creating dialog systems without requiring any coding knowledge.Once a dialog graph has been created, it is automatically turned into a dialog system using state of the art language models.This allows for rapid prototyping and testing.Dialog designers can then distribute a link to their finished dialog system or embed it into a website.Additionally, to support scientific experiments and data collection, dialog designers can access chat logs.Finally, to verify the usability of DIAGRAPH, we performed evaluation with subject-experts who extensively worked with the tool and users testing it for the first time, receiving above average System Usability Scale (SUS) scores from both (82 out 100 and 75 out of 100, respectively).In this way, we hope DI-AGRAPH helps reduce the barrier to entry for creating dialog interactions.
Question-Answering in a Low-resourced Language: Benchmark Dataset and Models for Tigrinya

Question-Answering (QA) has seen significant advances recently, achieving near human-level performance over some benchmarks.However, these advances focus on high-resourced languages such as English, while the task remains unexplored for most other languages, mainly due to the lack of annotated datasets.This work presents a native QA dataset for an East African language, Tigrinya.The dataset contains 10.6K question-answer pairs spanning 572 paragraphs extracted from 290 news articles on various topics.The dataset construction method is discussed, which is applicable to constructing similar resources for related languages.We present comprehensive experiments and analyses of several resource-efficient approaches to QA, including monolingual, cross-lingual, and multilingual setups, along with comparisons against machine-translated silver data.Our strong baseline models reach 76% in the F1 score, while the estimated human performance is 92%, indicating that the benchmark presents a good challenge for future work.We make the dataset, models, and leaderboard publicly available.
Pre-training Multi-party Dialogue Models with Latent Discourse Inference

Multi-party dialogues are more difficult for models to understand than one-to-one twoparty dialogues, since they involve multiple interlocutors, resulting in interweaving reply-to relations and information flows.To step over these obstacles, an effective way is to pre-train a model that understands the discourse structure of multi-party dialogues, namely, to whom each utterance is replying.However, due to the lack of explicitly annotated discourse labels in multi-party dialogue corpora, previous works fail to scale up the pre-training process by putting aside the unlabeled multi-party conversational data for nothing.To fully utilize the unlabeled data, we propose to treat the discourse structures as latent variables, then jointly infer them and pre-train the discourse-aware model by unsupervised latent variable inference methods.Experiments on multiple downstream tasks show that our pre-trained model outperforms strong baselines by large margins and achieves state-of-the-art (SOTA) results, justifying the effectiveness of our method.The official implementation of this paper is available at https://github.com/EricLee8/MPD_EMVI.
PEEP-Talk: A Situational Dialogue-based Chatbot for English Education

English is acknowledged worldwide as a mode of communication.However, due to the absence of realistic practicing scenarios, students learning English as a foreign language (EFL) typically have limited chances to converse and share feedback with others.In this paper, we propose PEEP-Talk, a real-world situational dialogue-based chatbot designed for English education.It also naturally switches to a new topic or situation in response to out-of-topic utterances, which are common among English beginners.Furthermore, PEEP-Talk provides feedback score on conversation and grammar error correction.We performed automatic and user evaluations to validate performance and education efficiency of our system.The results show that PEEP-Talk generates appropriate responses in various real-life situations while providing accurate feedback to learners.Moreover, we demonstrate a positive impact on Englishspeaking, grammar, and English learning anxiety, implying that PEEP-Talk can lower the barrier to learning natural conversation in effective ways.
BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition

Using language models (LMs) pre-trained in a self-supervised setting on large corpora and then fine-tuning for a downstream task has helped to deal with the problem of limited label data for supervised learning tasks such as Named Entity Recognition (NER).Recent research in biomedical language processing has offered a number of biomedical LMs pretrained using different methods and techniques that advance results on many BioNLP tasks, including NER.However, there is still a lack of a comprehensive comparison of pre-training approaches that would work more optimally in the biomedical domain.This paper aims to investigate different pre-training methods, such as pre-training the biomedical LM from scratch and pre-training it in a continued fashion.We compare existing methods with our proposed pre-training method of initializing weights for new tokens by distilling existing weights from the BERT model inside the context where the tokens were found.The method helps to speed up the pre-training stage and improve performance on NER.In addition, we compare how masking rate, corruption strategy, and masking strategies impact the performance of the biomedical LM.Finally, using the insights from our experiments, we introduce a new biomedical LM (BIOptimus), which is pre-trained using Curriculum Learning (CL) and contextualized weight distillation method.Our model sets new states of the art on several biomedical Named Entity Recognition (NER) tasks.We release our code and all pre-trained models. 1
The OPUS-MT Dashboard – A Toolkit for a Systematic Evaluation of Open Machine Translation Models

The OPUS-MT dashboard is a web-based platform that provides a comprehensive overview of open translation models.We focus on a systematic collection of benchmark results with verifiable translation performance and large coverage in terms of languages and domains.We provide results for in-house OPUS-MT and Tatoeba models as well as external models from the Huggingface repository and usercontributed translations.The functionalities of the evaluation tool include summaries of benchmarks for over 2,300 models covering 4,560 language directions and 294 languages, as well as the inspection of predicted translations against their human reference.We focus on centralization, reproducibility and coverage of MT evaluation combined with scalability.The dashboard can be accessed live at https://opus.nlpl.eu/dashboard/.
CB2: Collaborative Natural Language Interaction Research Platform

CB2 is a multi-agent platform to study collaborative natural language interaction in a grounded task-oriented scenario.It includes a 3D game environment, a backend server designed to serve trained models to human agents, and various tools and processes to enable scalable studies.We deploy CB2 at https://cb2.aias a system demonstration with a learned instruction following model.
SPEECH: Structured Prediction with Energy-Based Event-Centric Hyperspheres

Event-centric structured prediction involves predicting structured outputs of events.In most NLP cases, event structures are complex with manifold dependency, and it is challenging to effectively represent these complicated structured events.To address these issues, we propose Structured Prediction with Energybased Event-Centric Hyperspheres (SPEECH).SPEECH models complex dependency among event structured components with energybased modeling, and represents event classes with simple but effective hyperspheres.Experiments on two unified-annotated event datasets indicate that SPEECH is predominant in event detection and event-relation extraction tasks.
Transfer and Active Learning for Dissonance Detection: Addressing the Rare-Class Challenge

While transformer-based systems have enabled greater accuracies with fewer training examples, data acquisition obstacles still persist for rare-class tasks -when the class label is very infrequent (e.g., < 5% of samples).Active learning has in general been proposed to alleviate such challenges, but choice of selection strategy, the criteria by which rare-class examples are chosen, has not been systematically evaluated.Further, transformers enable iterative transfer-learning approaches.We propose and investigate transfer-and active learning solutions to the rare class problem of dissonance detection through utilizing models trained on closely related tasks and the evaluation of acquisition strategies, including a proposed probability-of-rare-class (PRC) approach.We perform these experiments for a specific rare class problem: collecting language samples of cognitive dissonance from social media.We find that PRC is a simple and effective strategy to guide annotations and ultimately improve model accuracy, and while transferlearning in a specific order can improve the cold-start performance of the learner but does not benefit iterations of active learning.
WeLT: Improving Biomedical Fine-tuned Pre-trained Language Models with Cost-sensitive Learning

Fine-tuning biomedical pre-trained language models (BioPLMs) such as BioBERT has become a common practice dominating leaderboards across various natural language processing tasks.Despite their success and wide adoption, prevailing fine-tuning approaches for named entity recognition (NER) naively train BioPLMs on targeted datasets without considering class distributions.This is problematic especially when dealing with imbalanced biomedical gold-standard datasets for NER in which most biomedical entities are underrepresented.In this paper, we address the class imbalance problem and propose WeLT, a costsensitive fine-tuning approach based on new rescaled class weights for the task of biomedical NER.We evaluate WeLT's fine-tuning performance on mixed-domain and domain-specific BioPLMs using eight biomedical gold-standard datasets.We compare our approach against vanilla fine-tuning and three other existing reweighting schemes.Our results show the positive impact of handling the class imbalance problem.WeLT outperforms all the vanilla fine-tuned models.Furthermore, our method demonstrates advantages over other existing weighting schemes in most experiments.
UINAUIL: A Unified Benchmark for Italian Natural Language Understanding

This paper introduces the Unified Interactive Natural Understanding of the Italian Language (UINAUIL), a benchmark of six tasks for Italian Natural Language Understanding.We present a description of the tasks and software library that collects the data from the European Language Grid, harmonizes the data format, and exposes functionalities to facilitates data manipulation and the evaluation of custom models.We also present the results of tests conducted with available Italian and multilingual language models on UINAUIL, providing an updated picture of the current state of the art in Italian NLU.
Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms

This paper studies a new task of federated learning (FL) for semantic parsing, where multiple clients collaboratively train one global model without sharing their semantic parsing data.By leveraging data from multiple clients, the FL paradigm can be especially beneficial for clients that have little training data to develop a data-hungry neural semantic parser on their own.We propose an evaluation setup to study this task, where we re-purpose widely-used single-domain text-to-SQL datasets as clients to form a realistic heterogeneous FL setting and collaboratively train a global model.As standard FL algorithms suffer from the high client heterogeneity in our realistic setup, we further propose a novel LOss Reduction Adjusted Reweighting (Lorar) mechanism to mitigate the performance degradation, which adjusts each client's contribution to the global model update based on its training loss reduction during each round.Our intuition is that the larger the loss reduction, the further away the current global model is from the client's local optimum, and the larger weight the client should get.By applying Lorar to three widely adopted FL algorithms (FedAvg, FedOPT and FedProx), we observe that their performance can be improved substantially on average (4%-20% absolute gain under MacroAvg) and that clients with smaller datasets enjoy larger performance gains.In addition, the global model converges faster for almost all the clients.
Predicting Customer Satisfaction with Soft Labels for Ordinal Classification

In a typical call center, only up to 8% of callers leave a Customer Satisfaction (CSAT) survey response at the end of the call, and these tend to be customers with strongly positive or negative experiences.To manage this data sparsity and response bias, we outline a predictive CSAT deep learning algorithm that infers CSAT on the 1-5 scale on inbound calls to the call center with minimal latency.The key metric to maximize is the precision for CSAT = 1 (lowest CSAT).We maximize this metric in two ways.First, reframing the problem as a binary class, rather than five-class problem during model fine-tuning, and then mapping binary outcomes back to five classes using temperature-scaled model probabilities.Second, using soft labels to represent the classes.The result is a production model that supports key customer workflows with high accuracy over millions of calls a month.
Unsupervised Extractive Summarization of Emotion Triggers

Understanding what leads to emotions during large-scale crises is important as it can provide groundings for expressed emotions and subsequently improve the understanding of ongoing disasters.Recent approaches (Zhan et al., 2022) trained supervised models to both detect emotions and explain emotion triggers (events and appraisals) via abstractive summarization.However, obtaining timely and qualitative abstractive summaries is expensive and extremely time-consuming, requiring highlytrained expert annotators.In time-sensitive, high-stake contexts, this can block necessary responses.We instead pursue unsupervised systems that extract triggers from text.First, we introduce COVIDET-EXT, augmenting (Zhan et al., 2022)'s abstractive dataset (in the context of the COVID-19 crisis) with extractive triggers.Second, we develop new unsupervised learning models that can jointly detect emotions and summarize their triggers.Our best approach, entitled Emotion-Aware Pagerank, incorporates emotion information from external sources combined with a language understanding module, and outperforms strong baselines.We release our data and code at
Discourse Analysis via Questions and Answers: Parsing Dependency Structures of Questions Under Discussion

Automatic discourse processing is bottlenecked by data: current discourse formalisms pose highly demanding annotation tasks involving large taxonomies of discourse relations, making them inaccessible to lay annotators.This work instead adopts the linguistic framework of Questions Under Discussion (QUD) for discourse analysis and seeks to derive QUD structures automatically.QUD views each sentence as an answer to a question triggered in prior context; thus, we characterize relationships between sentences as free-form questions, in contrast to exhaustive fine-grained taxonomies.We develop the first-of-its-kind QUD parser that derives a dependency structure of questions over full documents, trained using a large, crowdsourced question-answering dataset DCQA (Ko et al., 2022).Human evaluation results show that QUD dependency parsing is possible for language models trained with this crowdsourced, generalizable annotation scheme.We illustrate how our QUD structure is distinct from RST trees, and demonstrate the utility of QUD analysis in the context of document simplification.Our findings show that QUD parsing is an appealing alternative for automatic discourse processing.
Combining Transformers with Natural Language Explanations

Many NLP applications require models to be interpretable. However, many successful neural architectures, including transformers, still lack effective interpretation methods. A possible solution could rely on building explanations from domain knowledge, which is often available as plain, natural language text. We thus propose an extension to transformer models that makes use of external memories to store natural language explanations and use them to explain classification outputs. We conduct an experimental evaluation on two domains, legal text analysis and argument mining, to show that our approach can produce relevant explanations while retaining or even improving classification performance.
Neural Machine Translation through Active Learning on low-resource languages: The case of Spanish to Mapudungun

Active learning is an algorithmic approach that strategically selects a subset of examples for labeling, with the goal of reducing workload and required resources.Previous research has applied active learning to Neural Machine Translation (NMT) for high-resource or wellrepresented languages, achieving significant reductions in manual labor.In this study, we explore the application of active learning for NMT in the context of Mapudungun, a lowresource language spoken by the Mapuche community in South America.Mapudungun was chosen due to the limited number of fluent speakers and the pressing need to provide access to content predominantly available in widely represented languages.We assess both model-dependent and model-agnostic active learning strategies for NMT between Spanish and Mapudungun in both directions, demonstrating that we can achieve over 40% reduction in manual translation workload in both cases.
Mitigating the Burden of Redundant Datasets via Batch-Wise Unique Samples and Frequency-Aware Losses

Datasets used to train deep learning models in industrial settings often exhibit skewed distributions with some samples repeated a large number of times.This paper presents a simple yet effective solution to reduce the increased burden of repeated computation on redundant datasets.Our approach eliminates duplicates at the batch level, without altering the data distribution observed by the model, making it model-agnostic and easy to implement as a plug-and-play module.We also provide a mathematical expression to estimate the reduction in training time that our approach provides.Through empirical evidence, we show that our approach significantly reduces training times on various models across datasets with varying redundancy factors, without impacting their performance on the Named Entity Recognition task, both on publicly available datasets and in real industrial settings.In the latter, the approach speeds training by up to 87%, and by 46% on average, with a drop in model performance of 0.2% relative at worst.We finally release a modular and reusable codebase to further advance research in this area.
nav-nlp at RadSum23: Abstractive Summarization of Radiology Reports using BART Finetuning

This paper describes the experiments undertaken and their results as part of the BioNLP 2023 workshop.We took part in Task 1B: Radiology Report Summarization (Delbrouck et al., 2023).Multiple runs were submitted for evaluation from solutions utilizing transfer learning from pre-trained transformer models, which were then fine-tuned on the MIMIC-III dataset, for abstractive report summarization.The task was evaluated using different evaluation metrics such as ROUGEL, Bertscore, and F1-RadGraph, and the corresponding scores of our best-performing system are 32.33,54.49, and 32.68 respectively.
Learn to Not Link: Exploring NIL Prediction in Entity Linking

Entity linking models have achieved significant success via utilizing pretrained language models to capture semantic features.However, the NIL prediction problem, which aims to identify mentions without a corresponding entity in the knowledge base, has received insufficient attention.We categorize mentions linking to NIL into Missing Entity and Non-Entity Phrase, and propose an entity linking dataset NEL that focuses on the NIL prediction problem.NEL takes ambiguous entities as seeds, collects relevant mention context in the Wikipedia corpus, and ensures the presence of mentions linking to NIL by human annotation and entity masking.We conduct a series of experiments with the widely used biencoder and cross-encoder entity linking models, results show that both types of NIL mentions in training data have a significant influence on the accuracy of NIL prediction.Our code and dataset can be accessed at https: //github.com/solitaryzero/NIL_EL.
NatLogAttack: A Framework for Attacking Natural Language Inference Models with Natural Logic

Reasoning has been a central topic in artificial intelligence from the beginning.The recent progress made on distributed representation and neural networks continues to improve the state-of-the-art performance of natural language inference.However, it remains an open question whether the models perform real reasoning to reach their conclusions or rely on spurious correlations.Adversarial attacks have proven to be an important tool to help evaluate the Achilles' heel of the victim models.In this study, we explore the fundamental problem of developing attack models based on logic formalism.We propose NatLogAttack to perform systematic attacks centring around natural logic, a classical logic formalism that is traceable back to Aristotle's syllogism and has been closely developed for natural language inference.The proposed framework renders both label-preserving and label-flipping attacks.We show that compared to the existing attack models, NatLogAttack generates better adversarial examples with fewer visits to the victim models.The victim models are found to be more vulnerable under the label-flipping setting.NatLogAttack provides a tool to probe the existing and future NLI models' capacity from a key viewpoint and we hope more logicbased attacks will be further explored for understanding the desired property of reasoning. 1