Reviewerly/exHarmony huggingface dataset:
Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation

Pre-trained language models (PLMs) have achieved great success in NLP and have recently been used for tasks in computational semantics.However, these tasks do not fully benefit from PLMs since meaning representations are not explicitly included in the pretraining stage.We introduce multilingual pretrained language-meaning models based on Discourse Representation Structures (DRSs), including meaning representations besides natural language texts in the same model, and design a new strategy to reduce the gap between the pre-training and fine-tuning objectives.Since DRSs are language neutral, crosslingual transfer learning is adopted to further improve the performance of non-English tasks.Automatic evaluation results show that our approach achieves the best performance on both the multilingual DRS parsing and DRS-to-text generation tasks.Correlation analysis between automatic metrics and human judgements on the generation task further validates the effectiveness of our model.Human inspection reveals that out-of-vocabulary tokens are the main cause of erroneous results.
DuluthNLP at SemEval-2023 Task 12: AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twitter Dataset

This paper describes the DuluthNLP system that participated in Task 12 of SemEval-2023 on AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twitter Dataset.Given a set of tweets, the task requires participating systems to classify each tweet as negative, positive or neutral.We evaluate a range of monolingual and multilingual pre-trained models on the Twi language dataset, one among the 14 African languages included in the SemEval task.We introduce TwiBERT, a new pretrained model trained from scratch.We show that TwiBERT, along with mBERT, generally perform best when trained on the Twi dataset, achieving an F1 score of 64.29% on the official evaluation test data, which ranks 14 out of 30 of the total submissions for Track 10.The TwiBERT model is released at https://huggingface.co/sakrah/TwiBERT
Team Converge at ProbSum 2023: Abstractive Text Summarization of Patient Progress Notes

In this paper, we elaborate on our approach for the shared task 1A issued by BioNLP Workshop 2023 titled "Problem List Summarization."With an increase in the digitization of health records, a need arises for quick and precise summarization of large amounts of records.With the help of summarization, medical professionals can sieve through multiple records in a short span of time without overlooking any crucial point.We use abstractive text summarization for this task and experiment with multiple stateof-the-art models like Pegasus, BART, and T5, along with various pre-processing and data augmentation techniques to generate summaries from patients' progress notes.For this task, the metric used was the ROUGE-L score.From our experiments, we conclude that Pegasus is the best-performing model on the dataset, achieving a ROUGE-L F1 score of 0.2744 on the test dataset (3rd rank on the leaderboard).
HistRED: A Historical Document-Level Relation Extraction Dataset

Despite the extensive applications of relation extraction (RE) tasks in various domains, little has been explored in the historical context, which contains promising data across hundreds and thousands of years.To promote the historical RE research, we present HistRED constructed from Yeonhaengnok.Yeonhaengnok is a collection of records originally written in Hanja, the classical Chinese writing, which has later been translated into Korean.HistRED provides bilingual annotations such that RE can be performed on Korean and Hanja texts.In addition, HistRED supports various self-contained subtexts with different lengths, from a sentence level to a document level, supporting diverse context settings for researchers to evaluate the robustness of their RE models.To demonstrate the usefulness of our dataset, we propose a bilingual RE model that leverages both Korean and Hanja contexts to predict relations between entities.Our model outperforms monolingual baselines on HistRED, showing that employing multiple language contexts supplements the RE predictions.
Towards Extracting and Understanding the Implicit Rubrics of Transformer Based Automatic Essay Scoring Models

By aligning the functional components derived from the activations of transformer models trained for AES with external knowledge such as human-understandable feature groups, the proposed method improves the interpretability of a Longformer Automated Essay Scoring (AES) system and provides tools for performing such analyses on further neural AES systems.The analysis focuses on models trained to score essays based on ORGANIZATION, MAIN IDEA, SUPPORT, and LANGUAGE.The findings provide insights into the models' decisionmaking processes, biases, and limitations, contributing to the development of more transparent and reliable AES systems.
Multilingual Multi-Figurative Language Detection

Figures of speech help people express abstract concepts and evoke stronger emotions than literal expressions, thereby making texts more creative and engaging. Due to its pervasive and fundamental character, figurative language understanding has been addressed in Natural Language Processing, but it’s highly understudied in a multilingual setting and when considering more than one figure of speech at the same time. To bridge this gap, we introduce multilingual multi-figurative language modelling, and provide a benchmark for sentence-level figurative language detection, covering three common figures of speech and seven languages. Specifically, we develop a framework for figurative language detection based on template-based prompt learning. In so doing, we unify multiple detection tasks that are interrelated across multiple figures of speech and languages, without requiring task- or language-specific modules. Experimental results show that our framework outperforms several strong baselines and may serve as a blueprint for the joint modelling of other interrelated tasks.
Maximum Entropy Loss, the Silver Bullet Targeting Backdoor Attacks in Pre-trained Language Models

Pre-trained language model (PLM) can be stealthily misled to target outputs by backdoor attacks when encountering poisoned samples, without performance degradation on clean samples.The stealthiness of backdoor attacks is commonly attained through minimal crossentropy loss fine-tuning on a union of poisoned and clean samples.Existing defense paradigms provide a workaround by detecting and removing poisoned samples at pre-training or inference time.On the contrary, we provide a new perspective where the backdoor attack is directly reversed.Specifically, maximum entropy loss is incorporated in training to neutralize the minimal cross-entropy loss fine-tuning on poisoned data.We defend against a range of backdoor attacks on classification tasks and significantly lower the attack success rate.In extension, we explore the relationship between intended backdoor attacks and unintended dataset bias, and demonstrate the feasibility of the maximum entropy principle in de-biasing.
In and Out-of-Domain Text Adversarial Robustness via Label Smoothing

Recently it has been shown that state-of-the-art NLP models are vulnerable to adversarial attacks, where the predictions of a model can be drastically altered by slight modifications to the input (such as synonym substitutions).While several defense techniques have been proposed, and adapted, to the discrete nature of text adversarial attacks, the benefits of general-purpose regularization methods such as label smoothing for language models, have not been studied.In this paper, we study the adversarial robustness provided by label smoothing strategies in foundational models for diverse NLP tasks in both in-domain and out-of-domain settings.Our experiments show that label smoothing significantly improves adversarial robustness in pre-trained models like BERT, against various popular attacks.We also analyze the relationship between prediction confidence and robustness, showing that label smoothing reduces over-confident errors on adversarial examples.* The first two authors contributed equally to this paper.Most of the work done while Soham Dan was at the University of Pennsylvania. 1 Confidence on an example is the highest softmax score of the classifier prediction on that example.
A Unified Evaluation Framework for Novelty Detection and Accommodation in NLP with an Instantiation in Authorship Attribution

State-of-the-art natural language processing models have been shown to achieve remarkable performance in 'closed-world' settings where all the labels in the evaluation set are known at training time.However, in real-world settings, 'novel' instances that do not belong to any known class are often observed.This renders the ability to deal with novelties crucial.To initiate a systematic research in this important area of 'dealing with novelties', we introduce NoveltyTask, a multi-stage task to evaluate a system's performance on pipelined novelty 'detection' and 'accommodation' tasks.We provide mathematical formulation of NoveltyTask and instantiate it with the authorship attribution task that pertains to identifying the correct author of a given text.We use Amazon reviews corpus and compile a large dataset (consisting of 250k instances across 200 authors/labels) for NoveltyTask.We conduct comprehensive experiments and explore several baseline methods for the task.Our results show that the methods achieve considerably low performance making the task challenging and leaving sufficient room for improvement.Finally, we believe our work will encourage research in this underexplored area of dealing with novelties, an important step en route to developing robust systems.
Trigger-Argument based Explanation for Event Detection

A critical task for constructing event knowledge graphs is event detection (ED), which aims to identify events of certain types in plain text.Neural models have achieved great success on ED, thus coming with a desire for higher interpretability.Existing works mainly exploit words or phrases of the input text to explain models' inner mechanisms.However, for ED, the event structure, comprising of an event trigger and a set of arguments, provides more enlightening clues to explain model behaviors.To this end, we propose a Trigger-Argument based Explanation method (TAE), which can utilize event structure knowledge to uncover a faithful interpretation for the existing ED models at neuron level.Specifically, we design group, sparsity, support mechanisms to construct the event structure from structuralization, compactness, and faithfulness perspectives.We evaluate our model on the large-scale MAVEN and the widely-used ACE 2005 datasets, and observe that TAE is able to reveal the process by which the model predicts.Experimental results also demonstrate that TAE can not only improve the interpretability on standard evaluation metrics, but also effectively facilitate the human understanding.
MultiEMO: An Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations

Emotion Recognition in Conversations (ERC) is an increasingly popular task in the Natural Language Processing community, which seeks to achieve accurate emotion classifications of utterances expressed by speakers during a conversation.Most existing approaches focus on modeling speaker and contextual information based on the textual modality, while the complementarity of multimodal information has not been well leveraged, few current methods have sufficiently captured the complex correlations and mapping relationships across different modalities.Furthermore, existing state-ofthe-art ERC models have difficulty classifying minority and semantically similar emotion categories.To address these challenges, we propose a novel attention-based correlation-aware multimodal fusion framework named MultiEMO, which effectively integrates multimodal cues by capturing cross-modal mapping relationships across textual, audio and visual modalities based on bidirectional multi-head crossattention layers.The difficulty of recognizing minority and semantically hard-to-distinguish emotion classes is alleviated by our proposed Sample-Weighted Focal Contrastive (SWFC) loss.Extensive experiments on two benchmark ERC datasets demonstrate that our MultiEMO framework consistently outperforms existing state-of-the-art approaches in all emotion categories on both datasets, the improvements in minority and semantically similar emotions are especially significant.
Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model

The recently released NLLB-200 is a set of multilingual Neural Machine Translation models that cover 202 languages.The largest model is based on a Mixture of Experts architecture and achieves SoTA results across many language pairs.It contains 54.5B parameters and requires at least four 32GB GPUs just for inference.In this work, we propose a pruning method that enables the removal of up to 80% of experts without further finetuning and with a negligible loss in translation quality, which makes it feasible to run the model on a single 32GB GPU.Further analysis suggests that our pruning metrics can identify language-specific experts.8 12 layers with 128 experts each = 1536 experts 9 We always use beam search with a beam size of 4.
Enhancing Human Summaries for Question-Answer Generation in Education

We address the problem of generating highquality question-answer pairs for educational materials.Previous work on this problem showed that using summaries as input improves the quality of question generation (QG) over original textbook text and that human-written summaries result in higher quality QG than automatic summaries.In this paper, a) we show that advances in Large Language Models (LLMs) are not yet sufficient to generate quality summaries for QG and b) we introduce a new methodology for rewriting bullet point student notes into fully-fledged summaries and find that our methodology yields higher quality QG.We conducted a large-scale human annotation study of generated question-answer pairs for the evaluation of our methodology.In order to aid in future research, we release a novel dataset of 9.2K human annotations of generated questions.
Assessing the efficacy of large language models in generating accurate teacher responses

Tack et al., 2023) organized the shared task hosted by the 18th Workshop on Innovative Use of NLP for Building Educational Applications on generation of teacher language in educational dialogues.Following the structure of the shared task, in this study, we attempt to assess the generative abilities of large language models in providing informative and helpful insights to students, thereby simulating the role of a knowledgeable teacher.To this end, we present an extensive evaluation of several benchmarking generative models, including GPT-4 (few-shot, in-context learning), fine-tuned GPT-2, and fine-tuned DialoGPT.Additionally, to optimize for pedagogical quality, we fine-tuned the Flan-T5 model using reinforcement learning.Our experimental findings on the Teacher-Student Chatroom Corpus subset indicate the efficacy of GPT-4 over other fine-tuned models, measured using BERTScore and DialogRPT.We hypothesize that several dataset characteristics, including sampling, representativeness, and dialog completeness, pose significant challenges to fine-tuning, thus contributing to the poor generalizability of the fine-tuned models.Finally, we note the need for these generative models to be evaluated with a metric that relies not only on dialog coherence and matched language modeling distribution but also on the model's ability to showcase pedagogical skills.
MoQA: Benchmarking Multi-Type Open-Domain Question Answering

Previous research on open-domain question answering (QA) focuses mainly on shortanswered questions.However, informationseeking QA often requires various formats of answers depending on the nature of the questions, e.g., why/how questions typically require a long answer.In this paper, we present MOQA 1 , a benchmark for opendomain QA that requires building one system that can provide short, medium, long, and yes/no answers to different questions accordingly.MOQA builds upon Natural Questions (Kwiatkowski et al., 2019) with multiple types of questions and additional crowdsourcing efforts to ensure high data quality.We adapt state-of-the-art models, and reveal unique findings in multi-type open-domain QA: (1) For retriever-reader models, training one retriever on all types achieves the overall best performance, but it is challenging to train one reader model to output answers of different formats, or to train a question classifier to distinguish between types; (2) An end-to-end closed-book QA model trained on multiple types struggles with the task across the board; (3) State-of-theart large language models such as the largest GPT-3 models (Brown et al., 2020;Ouyang et al., 2022) also lag behind open-book QA models.Our benchmark and analysis call for more effort to build versatile open-domain QA models in the future. 21 MoQA = Multi-type Open-domain Question Answering.It is pronounced as mocha.2
Automatic Named Entity Obfuscation in Speech

Sharing data containing personal information often requires its anonymization, even when consent for sharing was obtained from the data originator.While approaches exist for automated anonymization of text, the area is not as thoroughly explored in speech.This work focuses on identifying, replacing and inserting replacement named entities synthesized using voice cloning into original audio thereby retaining prosodic information while reducing the likelihood of deanonymization.The approach employs a novel named entity recognition (NER) system built directly on speech by training HuBERT (Hsu et al., 2021) using the English speech NER dataset (Yadav et al., 2020).Name substitutes are found using a masked language model and are synthesized using text to speech voice cloning (Eren and Team, 2021), upon which the substitute named entities are re-inserted into the original text.The approach is prototyped on a sample of the LibriSpeech corpus (Panayotov et al., 2015) with each step evaluated individually.
Reconciling Adaptivity and Task Orientation in the Student Dashboard of an Intelligent Language Tutoring System

In intelligent language tutoring systems, student dashboards should display the learning progress and performance and support the navigation through the learning content.Designing an interface that transparently offers information on students' learning in relation to specific learning targets while linking to the overarching functional goal, that motivates and organizes the practice in current foreign language teaching, is challenging.This becomes even more difficult in systems that adaptively expose students to different learning material and individualize system interactions.If such a system is used in an ecologically valid setting of blended learning, this generates additional requirements to incorporate the needs of students and teachers for control and customizability.We present the conceptual design of a student dashboard for a task-based, user-adaptive intelligent language tutoring system intended for use in real-life English classes in secondary schools.We highlight the key challenges and spell out open questions for future research.
QCon at SemEval-2023 Task 10: Data Augmentation and Model Ensembling for Detection of Online Sexism

Weston Feely, Prabhakar Gupta, Manas Ranjan Mohanty, Timothy Chon, Tuhin Kundu, Vijit Singh, Sandeep Atluri, Tanya Roosta, Viviane Ghaderi, Peter Schulam. Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023). 2023.
SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT

Second language acquisition (SLA) research has extensively studied cross-linguistic transfer, the influence of linguistic structure of a speaker's native language [L1] on the successful acquisition of a foreign language [L2].Effects of such transfer can be positive (facilitating acquisition) or negative (impeding acquisition).We find that NLP literature has not given enough attention to the phenomenon of negative transfer.To understand patterns of both positive and negative transfer between L1 and L2, we model sequential second language acquisition in LMs.Further, we build a Mutlilingual Age Ordered CHILDES (MAO-CHILDES)-a dataset consisting of 5 typologically diverse languages, i.e., German, French, Polish, Indonesian, and Japanese-to understand the degree to which native Child-Directed Speech (CDS) [L1] can help or conflict with English language acquisition [L2].To examine the impact of native CDS, we use the TILT-based cross lingual transfer learning approach established by Papadimitriou and Jurafsky (2020) and find that, as in human SLA, language family distance predicts more negative transfer.Additionally, we find that conversational speech data shows greater facilitation for language acquisition than scripted speech data.Our findings call for further research using our novel Transformer-based SLA models and we would like to encourage it by releasing our code, data, and models.
Effect Graph: Effect Relation Extraction for Explanation Generation

Argumentation is an important means of communication.For describing especially arguments about consequences, the notion of effect relations has been introduced recently.We propose a method to extract effect relations from large text resources and apply it on encyclopedic and argumentative texts.By connecting the extracted relations, we generate a knowledge graph which we call effect graph.For evaluating the effect graph, we perform crowd and expert annotations and create a novel dataset.We demonstrate a possible use case of the effect graph by proposing a method for explaining arguments from consequences.
OPI at SemEval-2023 Task 1: Image-Text Embeddings and Multimodal Information Retrieval for Visual Word Sense Disambiguation

The goal of visual word sense disambiguation is to find the image that best matches the provided description of the word's meaning.It is a challenging problem, requiring approaches that combine language and image understanding.In this paper, we present our submission to SemEval-2023 visual word sense disambiguation shared task.The proposed system integrates multimodal embeddings, learning to rank methods, and knowledge-based approaches.We build a classifier based on the CLIP model, whose results are enriched with additional information retrieved from Wikipedia and lexical databases.Our solution was ranked third in the multilingual task and won in the Persian track, one of the three language subtasks.