Reviewerly/exHarmony huggingface dataset:
Prefix Propagation: Parameter-Efficient Tuning for Long Sequences

Parameter-efficient tuning aims to mitigate the large memory requirements of adapting pretrained language models for downstream tasks.For example, one popular method, prefix-tuning (Li and Liang, 2021;Liu et al., 2022), prepends trainable tokens to sequences while freezing the rest of the model's parameters.Although such models attain comparable performance with fine-tuning when applied to sequences with short to moderate lengths, we show their inferior performance when modelling long sequences.To bridge this gap, we propose prefix-propagation, a simple but effective approach that conditions prefixes on previous hidden states.We empirically demonstrate that prefix-propagation outperforms prefix-tuning across long-document tasks, while using ∼50% fewer parameters.To further investigate the proposed architecture, we also show its advantage in calibration, and perform additional study on its relationship with kernel attention.To the best of our knowledge, this work is the first to focus on parameter-efficient learning for long-sequence language tasks. 1
Learning Neuro-Symbolic World Models with Conversational Proprioception

The recent emergence of Neuro-Symbolic Agent (NeSA) approaches to natural languagebased interactions calls for the investigation of model-based approaches.In contrast to modelfree approaches, which existing NeSAs take, learning an explicit world model has an interesting potential especially in the explainability, which is one of the key selling points of NeSA.To learn useful world models, we leverage one of the recent neuro-symbolic architectures, Logical Neural Networks (LNN).Here, we describe a method that can learn neuro-symbolic world models on the TextWorld-Commonsense set of games.We then show how this can be improved further by taking inspiration from the concept of proprioception, but for conversation.This is done by enhancing the internal logic state with a memory of previous actions while also guiding future actions by augmenting the learned model with constraints based on this memory.This greatly improves the gamesolving agents performance in a TextWorld setting, where the advantage over the baseline is an 85% average steps reduction and ×2.3 average score.
Conformal Nucleus Sampling

Language models generate text based on successively sampling the next word.A decoding procedure based on nucleus (top-p) sampling chooses from the smallest possible set of words whose cumulative probability exceeds the probability p.In this work, we assess whether a top-p set is indeed aligned with its probabilistic meaning in various linguistic contexts.We employ conformal prediction, a calibration procedure that focuses on the construction of minimal prediction sets according to a desired confidence level, to calibrate the parameter p as a function of the entropy of the next word distribution.We find that OPT models are overconfident, and that calibration shows a moderate inverse scaling with model size.https://github.com/shauli-ravfogel/
NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Tasks

Finetuning large language models inflates the costs of NLU applications and remains the bottleneck of development cycles.Recent works in computer vision use data pruning to reduce training time.Pruned data selection with static methods is based on a score calculated for each training example prior to finetuning, which involves important computational overhead.Moreover, the score may not necessarily be representative of sample importance throughout the entire training duration.We propose to address these issues with a refined version of dynamic data pruning, a curriculum which periodically scores and discards unimportant examples during finetuning.Our method leverages an EL2N metric that we extend to the joint intent and slot classification task, and an initial finetuning phase on the full train set.Our results on the GLUE benchmark and four joint NLU datasets show a better timeaccuracy trade-off compared to static methods.Our method preserves full accuracy while training on 50% of the data points and reduces computational times by up to 41%.If we tolerate instead a minor drop of accuracy of 1%, we can prune 80% of the training examples for a reduction in finetuning time reaching 66%.
Score It All Together: A Multi-Task Learning Study on Automatic Scoring of Argumentative Essays

When scoring argumentative essays in an educational context, not only the presence or absence of certain argumentative elements but also their quality is important.On the recently published student essay dataset PERSUADE, we first show that the automatic scoring of argument quality benefits from additional information about context, writing prompt and argument type.We then explore the different combinations of three tasks: automated span detection, type and quality prediction.Results show that a multi-task learning approach combining the three tasks outperforms sequential approaches that first learn to segment and then predict the quality/type of a segment.
Massively Multilingual Lexical Specialization of Multilingual Transformers

While pretrained language models (PLMs) primarily serve as general-purpose text encoders that can be fine-tuned for a wide variety of downstream tasks, recent work has shown that they can also be rewired to produce highquality word representations (i.e., static word embeddings) and yield good performance in type-level lexical tasks.While existing work primarily focused on the lexical specialization of single monolingual PLMs, in this work we expose massively multilingual transformers (MMTs, e.g., mBERT or XLM-R) to multilingual lexical knowledge at scale, leveraging BabelNet as the readily available rich source of multilingual and cross-lingual type-level lexical knowledge.Concretely, we use Babel-Net's multilingual synsets to create synonym pairs (or synonym-gloss pairs) across 50 languages and then subject the MMTs (mBERT and XLM-R) to a lexical specialization procedure guided by a contrastive objective.We show that such multilingual lexical specialization brings substantial gains in two standard cross-lingual lexical tasks, bilingual lexicon induction and cross-lingual word similarity, as well as in cross-lingual sentence retrieval.Crucially, we observe gains for languages unseen in specialization, indicating that multilingual lexical specialization enables generalization to languages with no lexical constraints.In a series of controlled experiments, we show that the number of specialization constraints plays a much greater role than the set of languages from which they originate.
VLN-Trans: Translator for the Vision and Language Navigation Agent

Language understanding is essential for the navigation agent to follow instructions.We observe two kinds of issues in the instructions that can make the navigation task challenging: 1.The mentioned landmarks are not recognizable by the navigation agent due to the different vision abilities of the instructor and the modeled agent.2. The mentioned landmarks are applicable to multiple targets, thus not distinctive for selecting the target among the candidate viewpoints.To deal with these issues, we design a translator module for the navigation agent to convert the original instructions into easy-tofollow sub-instruction representations at each step.The translator needs to focus on the recognizable and distinctive landmarks based on the agent's visual abilities and the observed visual environment.To achieve this goal, we create a new synthetic sub-instruction dataset and design specific tasks to train the translator and the navigation agent.We evaluate our approach on Room2Room (R2R), Room4room (R4R), and Room2Room Last (R2R-Last) datasets and achieve state-of-the-art results on multiple benchmarks.
An Active Learning Pipeline for NLU Error Detection in Conversational Agents

High-quality labeled data is paramount to the performance of modern machine learning models.However, annotating data is a timeconsuming and costly process that requires human experts to examine large collections of raw data.For conversational agents in production settings with access to large amounts of user-agent conversations, the challenge is to decide what data should be annotated first.We consider the Natural Language Understanding (NLU) component of a conversational agent deployed in a real-world setup with limited resources.We present an active learning pipeline for offline detection of classification errors that leverages two strong classifiers.Then, we perform topic modeling on the potentially mis-classified samples to ease data analysis and to reveal error patterns.In our experiments, we show on a realworld dataset that by using our method to prioritize data annotation we reach 100% of the performance annotating only 36% of the data.Finally, we present an analysis of some of the error patterns revealed and argue that our pipeline is a valuable tool to detect critical errors and reduce the workload of annotators.
Navigating Data Scarcity: Pretraining for Medical Utterance Classification

Pretrained language models leverage selfsupervised learning to use large amounts of unlabeled text for learning contextual representations of sequences.However, in the domain of medical conversations, the availability of large, public datasets is limited due to issues of privacy and data management.In this paper, we study the effectiveness of dialog-aware pretraining objectives and multiphase training in using unlabeled data to improve LMs training for medical utterance classification.The objectives of pretraining for dialog awareness involve tasks that take into account the structure of conversations, including features such as turn-taking and the roles of speakers.The multiphase training process uses unannotated data in a sequence that prioritizes similarities and connections between different domains.We empirically evaluate these methods on conversational dialog classification tasks in the medical and counseling domains, and find that multiphase training can help achieve higher performance than standard pretraining or finetuning.
PingAnLifeInsurance at SemEval-2023 Task 12: Sentiment Analysis for Low-resource African Languages with Multi-Model Fusion

This paper describes our system used in the SemEval-2023 Task12: Sentiment Analysis for Low-resource African Languages using Twitter Dataset.The AfriSenti-SemEval Shared Task 12 is based on a collection of Twitter datasets in 14 African languages for sentiment classification.It consists of three sub-tasks.Task A is a monolingual sentiment classification which covered 12 African languages.Task B is a multilingual sentiment classification which combined training data from Task A (12 African languages).Task C is a zero-shot sentiment classification.We utilized various strategies, including monolingual training, multilingual mixed training, and translation technology, and proposed a weighted voting method that combined the results of different strategies.Substantially, in the monolingual subtask, our system achieved Top-1 in two languages (Yoruba and Twi) and Top-2 in four languages (Nigerian Pidgin, Algerian Arabic, and Swahili, Multilingual).In the multilingual subtask, Our system achived Top-2 in publish leaderBoard.
An Inclusive Notion of Text

Natural language processing (NLP) researchers develop models of grammar, meaning and communication based on written text.Due to task and data differences, what is considered text can vary substantially across studies.A conceptual framework for systematically capturing these differences is lacking.We argue that clarity on the notion of text is crucial for reproducible and generalizable NLP.Towards that goal, we propose common terminology to discuss the production and transformation of textual data, and introduce a two-tier taxonomy of linguistic and non-linguistic elements that are available in textual sources and can be used in NLP modeling.We apply this taxonomy to survey existing work that extends the notion of text beyond the conservative language-centered view.We outline key desiderata and challenges of the emerging inclusive approach to text in NLP, and suggest community-level reporting as a crucial next step to consolidate the discussion.
ELQA: A Corpus of Metalinguistic Questions and Answers about English

We present ELQA, a corpus of questions and answers in and about the English language.Collected from two online forums, the >70k questions (from English learners and others) cover wide-ranging topics including grammar, meaning, fluency, and etymology.The answers include descriptions of general properties of English vocabulary and grammar as well as explanations about specific (correct and incorrect) usage examples.Unlike most NLP datasets, this corpus is metalinguistic-it consists of language about language.As such, it can facilitate investigations of the metalinguistic capabilities of NLU models, as well as educational applications in the language learning domain.To study this, we define a free-form question answering task on our dataset and conduct evaluations on multiple LLMs (Large Language Models) to analyze their capacity to generate metalinguistic answers.
PMCoders at SemEval-2023 Task 1: RAltCLIP: Use Relative AltCLIP Features to Rank

Visual Word Sense Disambiguation (VWSD) task aims to find the most related image among 10 images to an ambiguous word in some limited textual context.In this work, we use Alt-CLIP features and a 3-layer standard transformer encoder to compare the cosine similarity between the given phrase and different images.Also, we improve our model's generalization by using a subset of LAION-5B.The best official baseline achieves 37.20% and 54.39% macro-averaged hit rate and MRR (Mean Reciprocal Rank) respectively.Our best configuration reaches 39.61% and 56.78% macro-averaged hit rate and MRR respectively.The code will be made publicly available on GitHub.
John-Arthur at SemEval-2023 Task 4: Fine-Tuning Large Language Models for Arguments Classification

This paper presents the system submissions of the John-Arthur team to the SemEval Task 4 “ValueEval: Identification of Human Values behind Arguments”. The best system of the team was ranked 3rd and the overall rank of the team was 2nd (the first team had the two best systems). John-Arthur team models the ValueEval problem as a multi-class, multi-label text classification problem. The solutions leverage recently proposed large language models that are fine-tuned on the provided datasets. To boost the achieved performance we employ different best practises whose impact on the model performance we evaluate here. The code ispublicly available at github and the model onHuggingface hub.
Cross-lingual Data Augmentation for Document-grounded Dialog Systems in Low Resource Languages

This paper proposes a framework to address the issue of data scarcity in Document-Grounded Dialogue Systems(DGDS).Our model leverages high-resource languages to enhance the capability of dialogue generation in low-resource languages.Specifically, We present a novel pipeline CLEM (Cross-Lingual Enhanced Model) including adversarial training retrieval (Retriever and Reranker), and Fid (fusion-in-decoder) generator.To further leverage high-resource language, we also propose an innovative architecture to conduct alignment across different languages with translated training.Extensive experiment results demonstrate the effectiveness of our model and we achieved 4th place in the DialDoc 2023 Competition.Therefore, CLEM can serve as a solution to resource scarcity in DGDS and provide useful guidance for multilingual alignment tasks.
SUTNLP at SemEval-2023 Task 10: RLAT-Transformer for explainable online sexism detection

There is no simple definition of sexism, butit can be described as prejudice, stereotyping,or discrimination, especially against women,based on their gender. In online interactions,sexism is common. One out of ten Americanadults says that they have been harassed be-cause of their gender and have been the targetof sexism, so sexism is a growing issue. TheExplainable Detection of Online Sexism sharedtask in SemEval-2023 aims at building sexismdetection systems for the English language. Inorder to address the problem, we use largelanguage models such as RoBERTa and De-BERTa. In addition, we present Random LayerAdversarial Training (RLAT) for transformers,and show its significant impact on solving allsubtasks. Moreover, we use virtual adversar-ial training and contrastive learning to improveperformance on subtask A. Upon completionof subtask A, B, and C test sets, we obtainedmacro-F1 of 84.45, 67.78, and 52.52, respec-tively outperforming proposed baselines on allsubtasks. Our code is publicly available onGithub.
Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data

Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language.Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool.In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient parallel data resources.We show that under such data-deficient circumstances, the unlabeled data can significantly vary in domain from the supervised data, which results in pseudo-label quality degradation.We investigate two categories of remedies that require no additional supervision and target the domain mismatch: pseudo-label filtering and data augmentation.We show that pseudo-label analysis and processing in this way results in additional gains on top of the vanilla pseudolabeling setup providing a total improvement of up to 0.4% absolute WER and 2.1 BLEU points for En-De and 0.6% absolute WER and 2.2 BLEU points for En-Zh.* Work done during an internship at Apple. 1 We use descriptors "(un)labeled" and "(un)supervised" interchangeably throughout this paper.
A Meta-dataset of German Medical Corpora: Harmonization of Annotations and Cross-corpus NER Evaluation

Over the last years, an increasing number of publicly available, semantically annotated medical corpora have been released for the German language.While their annotations cover comparable semantic classes, the synergies of such efforts have not been explored, yet.This is due to substantial differences in the data schemas (syntax) and annotated entities (semantics), which hinder the creation of common meta-datasets.For instance, it is unclear whether named entity recognition (NER) taggers trained on one or more of such datasets are useful to detect entities in any of the other datasets.In this work, we create harmonized versions of German medical corpora using the BIGBIO framework, and make them available to the community.Using these as a meta-dataset, we perform a series of cross-corpus evaluation experiments on two settings of aligned labels.These consist in fine-tuning various pre-trained Transformers on different combinations of training sets, and testing them against each dataset separately.We find that a) trained NER models generalize poorly, with F 1 scores dropping approx.20 pp. on unseen test data, and b) current pre-trained Transformer models for the German language do not systematically alleviate this issue.However, our results suggest that models benefit from additional training corpora in most cases, even if these belong to different medical fields or text genres.
bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark

We present bgGLUE (Bulgarian General Language Understanding Evaluation), a benchmark for evaluating language models on Natural Language Understanding (NLU) tasks in Bulgarian.Our benchmark includes NLU tasks targeting a variety of NLP problems (e.g., natural language inference, fact-checking, named entity recognition, sentiment analysis, question answering, etc.) and machine learning tasks (sequence labeling, document-level classification, and regression).We run the first systematic evaluation of pre-trained language models for Bulgarian, comparing and contrasting results across the nine tasks in the benchmark.The evaluation results show strong performance on sequence labeling tasks, but there is a lot of room for improvement for tasks that require more complex reasoning.We make bgGLUE publicly available together with the fine-tuning and the evaluation code, as well as a public leaderboard at https://bgglue.github.io,and we hope that it will enable further advancements in developing NLU models for Bulgarian.* * Work done while Momchil was in the Sofia University, prior to joining Amazon.† Meta AI is not involved in the creation, release, or hosting of the datasets in the benchmark.
MPCHAT: Towards Multimodal Persona-Grounded Conversation

In order to build self-consistent personalized dialogue agents, previous research has mostly focused on textual persona that delivers personal facts or personalities.However, to fully describe the multi-faceted nature of persona, image modality can help better reveal the speaker's personal characteristics and experiences in episodic memory (Rubin et al., 2003;Conway, 2009).In this work, we extend persona-based dialogue to the multimodal domain and make two main contributions.First, we present the first multimodal persona-based dialogue dataset named MPCHAT, which extends persona with both text and images to contain episodic memories.Second, we empirically show that incorporating multimodal persona, as measured by three proposed multimodal persona-grounded dialogue tasks (i.e., next response prediction, grounding persona prediction, and speaker identification), leads to statistically significant performance improvements across all tasks.Thus, our work highlights that multimodal persona is crucial for improving multimodal dialogue comprehension, and our MPCHAT serves as a high-quality resource for this research.
Grammatical information in BERT sentence embeddings as two-dimensional arrays

Sentence embeddings induced with various transformer architectures encode much semantic and syntactic information in a distributed manner in a one-dimensional array.We investigate whether specific grammatical information can be accessed in these distributed representations.Using data from a task developed to test rule-like generalizations, our experiments on detecting subject-verb agreement yield several promising results.First, we show that while the usual sentence representations encoded as onedimensional arrays do not easily support extraction of rule-like regularities, a two-dimensional reshaping of these vectors allows various learning architectures to access such information.Next, we show that various architectures can detect patterns in these two-dimensional reshaped sentence embeddings and successfully learn a model based on smaller amounts of simpler training data, which performs well on more complex test data.This indicates that current sentence embeddings contain information that is regularly distributed, and which can be captured when the embeddings are reshaped into higher dimensional arrays.Our results cast light on representations produced by language models and help move towards developing fewshot learning approaches.