Reviewerly/exHarmony huggingface dataset:
Auxiliary Learning for Named Entity Recognition with Multiple Auxiliary Biomedical Training Data

Named entity recognition (NER) is one of the core technologies for knowledge acquisition from text and has been used for knowledge extraction of chemicals and medicine.As one of the NER improvement approaches, multitask learning that learns a model from multiple training data has been used.Among multitask learning, an auxiliary learning method, which uses training data of an auxiliary task for improving its target task, has shown higher NER performance than conventional multitask learning for improving all the tasks simultaneously.The conventional auxiliary learning method uses only one auxiliary training dataset.We propose Multiple Utilization of NER Corpora Helpful for Auxiliary BLESsing (MUNCHABLES).MUNCHABLES utilizes multiple training datasets as auxiliary training data by the following methods : the first one is to fine-tune the NER model of the target task by sequentially performing auxiliary learning for each auxiliary training dataset, and the other is to use all training datasets in one auxiliary learning.We evaluate MUNCHABLES on eight chemical/biomedical/scientific domain NER tasks, where seven training datasets are used as auxiliary training data.The experiment results show that our proposed methods achieve higher NER performance than conventional multi-task learning methods on average and that NER performance can be improved by using multiple auxiliary training data.Furthermore, the proposed models outperform stateof-the-art models on the datasets.
BPE vs. Morphological Segmentation: A Case Study on Machine Translation of Four Polysynthetic Languages

Morphologically-rich polysynthetic languages present a challenge for NLP systems due to data sparsity, and a common strategy to handle this issue is to apply subword segmentation.We investigate a wide variety of supervised and unsupervised morphological segmentation methods for four polysynthetic languages: Nahuatl, Raramuri, Shipibo-Konibo, and Wixarika.Then, we compare the morphologically inspired segmentation methods against Byte-Pair Encodings (BPEs) as inputs for machine translation (MT) when translating to and from Spanish.We show that for all language pairs except for Nahuatl, an unsupervised morphological segmentation algorithm outperforms BPEs consistently and that, although supervised methods achieve better segmentation scores, they under-perform in MT challenges.Finally, we contribute two new morphological segmentation datasets for Raramuri and Shipibo-Konibo, and a parallel corpus for Raramuri-Spanish.
Deep Inductive Logic Reasoning for Multi-Hop Reading Comprehension

Multi-hop reading comprehension requires an ability to reason across multiple documents.On the one hand, deep learning approaches only implicitly encode query-related information into distributed embeddings which fail to uncover the discrete relational reasoning process to infer the correct answer.On the other hand, logic-based approaches provide interpretable rules to infer the target answer, but mostly work on structured data where entities and relations are well-defined.In this paper, we propose a deep-learning based inductive logic reasoning method that firstly extracts query-related (candidate-related) information, and then conducts logic reasoning among the filtered information by inducing feasible rules that entail the target relation.The reasoning process is accomplished via attentive memories with novel differentiable logic operators.To demonstrate the effectiveness of our model, we evaluate it on two reading comprehension datasets, namely WikiHop and MedHop.
Early Stopping Based on Unlabeled Samples in Text Classification

Early stopping, which is widely used to prevent overfitting, is generally based on a separate validation set.However, in low resource settings, validation-based stopping can be risky because a small validation set may not be sufficiently representative, and the reduction in the number of samples by validation split may result in insufficient samples for training.In this study, we propose an early stopping method that uses unlabeled samples.The proposed method is based on confidence and class distribution similarities.To further improve the performance, we present a calibration method to better estimate the class distribution of the unlabeled samples.The proposed method is advantageous because it does not require a separate validation set and provides a better stopping point by using a large unlabeled set.Extensive experiments are conducted on five text classification datasets and several stop-methods are compared.Our results show that the proposed model even performs better than using an additional validation set as well as the existing stop-methods, in both balanced and imbalanced data settings.Our code is available at https://github.com/DMCB-GIST/BUS-stop.
Computational Historical Linguistics and Language Diversity in South Asia

South Asia is home to a plethora of languages, many of which severely lack access to new language technologies.This linguistic diversity also results in a research environment conducive to the study of comparative, contact, and historical linguistics-fields which necessitate the gathering of extensive data from many languages.We claim that data scatteredness (rather than scarcity) is the primary obstacle in the development of South Asian language technology, and suggest that the study of language history is uniquely aligned with surmounting this obstacle.We review recent developments in and at the intersection of South Asian NLP and historical-comparative linguistics, describing our and others' current efforts in this area.We also offer new strategies towards breaking the data barrier.
[CASPI] Causal-aware Safe Policy Improvement for Task-oriented Dialogue

The recent success of reinforcement learning's (RL) in solving complex tasks is most often attributed to its capacity to explore and exploit an environment where it has been trained. Sample efficiency is usually not an issue since cheap simulators are available to sample data on-policy. On the other hand, task oriented dialogues are usually learnt from offline data collected using human demonstrations. Collecting diverse demonstrations and annotating them is expensive. Unfortunately, use of RL methods trained on off-policy data are prone to issues of bias and generalization, which are further exacerbated by stochasticity in human response and non-markovian belief state of a dialogue management system. To this end, we propose a batch RL framework for task oriented dialogue policy learning: causal aware safe policy improvement (CASPI). This method gives guarantees on dialogue policy's performance and also learns to shape rewards according to intentions behind human responses, rather than just mimicking demonstration data; this couple with batch-RL helps overall with sample efficiency of the framework. We demonstrate the effectiveness of this framework on a dialogue-context-to-text Generation and end-to-end dialogue task of the Multiwoz2.0 dataset. The proposed method outperforms the current state of the art on these metrics, in both case. In the end-to-end case, our method trained only on 10\% of the data was able to out perform current state in three out of four evaluation metrics.
Integrating Question Rewrites in Conversational Question Answering: A Reinforcement Learning Approach

Resolving dependencies among dialogue history is one of the main obstacles in the research on conversational question answering (CQA).The conversational question rewrites (QR) task has been shown to be effective to solve this problem by reformulating questions in a selfcontained form.However, QR datasets are limited and existing methods tend to depend on the assumption of the existence of corresponding QR datasets for every CQA dataset.This paper proposes a reinforcement learning approach that integrates QR and CQA tasks without corresponding labeled QR datasets.We train a QR model based on the reward signal obtained from the CQA, and the experimental results show that our approach can bring improvement over the pipeline approaches.The code is available at https://github.com/HLTCHKUST/cqr4cqa.
Hate speech: a systematic review of scientific production and educational considerations.

Hate speech is a specific form of expression targeting certain vulnerable or minority groups. It is a social phenomenon which has gained strength in recent years due to its normalisation from the public sphere and escalation on social media, particularly i n contexts of crisis such as the circumstances that have arisen around COVID systematic review19. This paper presents a of the scientific literature produced over the past two decades in English, Portuguese and Spanish as a tool to examine, from social sc ience, the landscape of studies around one of the main threats democratic societies face today. In addition to an evidencebased idea of the state of the art, the results show a substantial increase of research into hate speech over the past five years and how focal points and disciplines have evolved from those approaches and are gaining in importance, along with the analysis of regulations and laws, computational linguistics and discourse analysis. Furthermore, the dearth of research from the field of com munication and education is confirmed, which would allow us to examine not only its management and analyse its impact, but rather its understanding from critical pedagogical perspectives committed to the deepening and extension of democracy
Data Augmentation for Biomedical Factoid Question Answering

We study the effect of seven data augmentation (da) methods in factoid question answering, focusing on the biomedical domain, where obtaining training instances is particularly difficult.We experiment with data from the bioasq challenge, which we augment with training instances obtained from an artificial biomedical machine reading comprehension dataset, or via back-translation, information retrieval, word substitution based on word2vec embeddings or masked language modeling, question generation, or extending the given passage with additional context.We show that da can lead to very significant performance gains, even when using large pretrained Transformers, contributing to a broader discussion of if/when da benefits large pretrained models.One of the simplest da methods, word2vec-based word substitution, performed best and is recommended.We release our artificial training instances and code.
Combining Static and Contextualised Multilingual Embeddings

Static and contextual multilingual embeddings have complementary strengths.Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages.We combine the strengths of static and contextual models to improve multilingual representations.We extract static embeddings for 40 languages from XLM-R, validate those embeddings with cross-lingual word retrieval, and then align them using VecMap.This results in high-quality, highly multilingual static embeddings.Then we apply a novel continued pre-training approach to XLM-R, leveraging the high quality alignment of our static embeddings to better align the representation space of XLM-R.We show positive results for multiple complex semantic tasks.We release the static embeddings and the continued pretraining code. 1 Unlike most previous work, our continued pre-training approach does not require parallel text.
Revisiting the Compositional Generalization Abilities of Neural Sequence Models

Compositional generalization is a fundamental trait in humans, allowing us to effortlessly combine known phrases to form novel sentences.Recent works have claimed that standard seqto-seq models severely lack the ability to compositionally generalize.In this paper, we focus on one-shot primitive generalization as introduced by the popular SCAN benchmark.We demonstrate that modifying the training distribution in simple and intuitive ways enables standard seq-to-seq models to achieve nearperfect generalization performance, thereby showing that their compositional generalization abilities were previously underestimated.We perform detailed empirical analysis of this phenomenon.Our results indicate that the generalization performance of models is highly sensitive to the characteristics of the training data which should be carefully considered while designing such benchmarks in future.
Virtual Reality Platform for Sustainable Road Education among Users of Urban Mobility in Cuenca, Ecuador

A traffic accident is an event generated in an unforeseen way that is beyond the control of the people involved, which can produce bodily, functional, or organic injuries, leading to death or disability in the worst cases.According to the Empresa P ública Municipal de Movilidad, Tránsito y Transporte de Cuenca (EMOV-EP), the total accidents recorded in 2021 were, 24.97% due to ignoring traffic signs, 21.11% due to not paying attention to traffic, and 16.94% due to driving under the influence of alcohol.The EMOV-EP, is the responsible for the regulation of human mobility.Thus, the EMOV-EP in conjunction with the Universidad Politécnica Salesiana (UPS) have introduced the next research question: How can a road safety education strategy, supported by Information and Communication Technologies (ICTs), can be developed to contribute to the improvement of the behavior of citizens to increase their knowledge of the traffic laws and regulations, and thus reduce the number of accidents in the city of Cuenca?Furthermore, in this paper we present the development of a Virtual Reality (VR) platform designed for road safety education.The platform is composed of a Web system, and 4 VR systems (games) that have been designed for 4 common causes of accidents respectively (drunk drivers, highspeed drivers, cyclists riding in bicycle lanes, and users of the tram transport system), using a serious games approach and the Oculus Rift/Quest technology.We have found that more than 80% of users have had a very good experience of playing and learning through the VR systems.Hence, this virtual reality platform constitutes a technological proposal with social impact because it creates an entertainment environment that can raise awareness among citizens, thereby strengthening road safety education and reducing the number of accidents in the city of Cuenca.
Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training

Generalising to unseen domains is underexplored and remains a challenge in neural machine translation.Inspired by recent research in parameter-efficient transfer learning from pretrained models, this paper proposes a fusionbased generalisation method that learns to combine domain-specific parameters.We propose a leave-one-domain-out training strategy to avoid information leaking to address the challenge of not knowing the test domain during training time.Empirical results on three language pairs show that our proposed fusion method outperforms other baselines up to +0.8 BLEU score on average.
CMCL 2022 Shared Task on Multilingual and Crosslingual Prediction of Human Reading Behavior

We present the second shared task on eyetracking data prediction of the Cognitive Modeling and Computational Linguistics Workshop (CMCL).Differently from the previous edition, participating teams are asked to predict eyetracking features from multiple languages, including a surprise language for which there were no available training data.Moreover, the task also included the prediction of standard deviations of feature values in order to account for individual differences between readers.A total of six teams registered to the task.For the first subtask on multilingual prediction, the winning team proposed a regression model based on lexical features, while for the second subtask on cross-lingual prediction, the winning team used a hybrid model based on a multilingual transformer embeddings as well as statistical features.
Explaining Classes through Stable Word Attributions

Input saliency methods have recently become a popular tool for explaining predictions of deep learning models in NLP.Nevertheless, there has been little work investigating methods for aggregating prediction-level explanations to the class level, nor has a framework for evaluating such class explanations been established.We explore explanations based on XLM-R and the Integrated Gradients input attribution method, and propose 1) the Stable Attribution Class Explanation method (SACX) to extract keyword lists of classes in text classification tasks, and 2) a framework for the systematic evaluation of the keyword lists.We find that explanations of individual predictions are prone to noise, but that stable explanations can be effectively identified through repeated training and explanation.We evaluate on web register data and show that the class explanations are linguistically meaningful and distinguishing of the classes.
AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level

Large Pre-trained Language Models (PLMs) have become ubiquitous in the development of language understanding technology and lie at the heart of many artificial intelligence advances.While advances reported for English using PLMs are unprecedented, reported advances using PLMs for Hebrew are few and far between.The problem is twofold.First, so far, Hebrew resources for training large language models are not of the same magnitude as their English counterparts.Second, most benchmarks available to evaluate progress in Hebrew NLP require morphological boundaries which are not available in the output of PLMs.In this work we remedy both aspects.We present AlephBERT, a large PLM for Modern Hebrew, trained on larger vocabulary and a larger dataset than any Hebrew PLM before.Moreover, we introduce a novel neural architecture that recovers the morphological segments encoded in contextualized embedding vectors.Based on this new morphological component we offer an evaluation suite consisting of multiple tasks and benchmarks that cover sentencelevel, word-level and sub-word level analyses.On all tasks, AlephBERT obtains state-of-theart results beyond contemporary Hebrew stateof-the-art models.We make our AlephBERT model, the morphological extraction component, and the Hebrew evaluation suite publicly available, for future investigations and evaluations of Hebrew PLMs.
Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation

Gender bias is largely recognized as a problematic phenomenon affecting language technologies, with recent studies underscoring that it might surface differently across languages.However, most of current evaluation practices adopt a word-level focus on a narrow set of occupational nouns under synthetic conditions.Such protocols overlook key features of grammatical gender languages, which are characterized by morphosyntactic chains of gender agreement, marked on a variety of lexical items and parts-of-speech (POS).To overcome this limitation, we enrich the natural, gender-sensitive MuST-SHE corpus (Bentivogli et al., 2020) with two new linguistic annotation layers (POS and agreement chains), and explore to what extent different lexical categories and agreement phenomena are impacted by gender skews.Focusing on speech translation, we conduct a multifaceted evaluation on three language directions (English-French/Italian/Spanish), with models trained on varying amounts of data and different word segmentation techniques.By shedding light on model behaviours, gender bias, and its detection at several levels of granularity, our findings emphasize the value of dedicated analyses beyond aggregated overall results.
Answer Uncertainty and Unanswerability in Multiple-Choice Machine Reading Comprehension

Machine reading comprehension (MRC) has drawn a lot of attention as an approach for assessing the ability of systems to understand natural language.Usually systems focus on selecting the correct answer to a question given a contextual paragraph.However, for many applications of multiple-choice MRC systems there are two additional considerations.For multiplechoice exams there is often a negative marking scheme; there is a penalty for an incorrect answer.This means that the system is required to have an idea of the uncertainty in the predicted answer.The second consideration is that many multiple-choice questions have the option of none of the above (NOA) indicating that none of the answers is applicable, rather than there always being the correct answer in the list of choices.This paper investigates both of these issues by making use of predictive uncertainty.It is shown that uncertainty does allow questions that the system is not confident about to be detected.Additionally we show that uncertainty outperforms a system explicitly built with an NOA option for the ReClor corpus.
Detection, Disambiguation, Re-ranking: Autoregressive Entity Linking as a Multi-Task Problem

We propose an autoregressive entity linking model, that is trained with two auxiliary tasks, and learns to re-rank generated samples at inference time.Our proposed novelties address two weaknesses in the literature.First, a recent method proposes to learn mention detection and then entity candidate selection, but relies on predefined sets of candidates.We use encoder-decoder autoregressive entity linking in order to bypass this need, and propose to train mention detection as an auxiliary task instead.Second, previous work suggests that re-ranking could help correct prediction errors.We add a new, auxiliary task, match prediction, to learn re-ranking.Without the use of a knowledge base or candidate sets, our model sets a new state of the art in two benchmark datasets of entity linking: COMETA in the biomedical domain, and AIDA-CoNLL in the news domain.We show through ablation studies that each of the two auxiliary tasks increases performance, and that re-ranking is an important factor to the increase.Finally, our low-resource experimental results suggest that performance on the main task benefits from the knowledge learned by the auxiliary tasks, and not just from the additional training data.
Test-Time Adaptation for Egocentric Action Recognition

Egocentric action recognition is becoming an increasingly researched topic thanks to the rising popularity of wearable cameras. Despite the numerous publications in the field, the learned representations still suffers from an intrinsic “environmental bias”. To address this issue, domain adaptation and generalization approaches have been proposed, which operate by either adapting the model to target data during training or by learning a model able to generalize to unseen videos by exploiting the knowledge from multiple source domains. In this work, we propose to adapt a model trained on source data to novel environments at test time, making adaptation practical to real-world scenarios where target data are not available at training time. On the popular EPIC-Kitchens dataset, we present a new benchmark for Test-Time Adaptation (TTA) in egocentric action recognition. Moreover, we propose a new multi-modal TTA approach, which we call RNA $$^{++}$$ , and combine it with a new set of losses aiming at reducing classifier’s uncertainty, showing remarkable results w.r.t. existing TTA methods inherited from image classification. Code available: https://github.com/EgocentricVision/RNA-TTA .