Reviewerly/exHarmony huggingface dataset:
Large-scale Lifelong Learning of In-context Instructions and How to Tackle It

Jointly fine-tuning a Pre-trained Language Model (PLM) on a pre-defined set of tasks with in-context instructions has been proven to improve its generalization performance, allowing us to build a universal language model that can be deployed across task boundaries.In this work, we explore for the first time whether this attractive property of in-context instruction learning can be extended to a scenario in which tasks are fed to the target PLM in a sequential manner.The primary objective of so-called lifelong in-context instruction learning is to improve the target PLM's instance-and task-level generalization performance as it observes more tasks.DYNAINST, the proposed method to lifelong in-context instruction learning, achieves noticeable improvements in both types of generalization, nearly reaching the upper bound performance obtained through joint training.
Exploring Variation of Results from Different Experimental Conditions

It might reasonably be expected that running multiple experiments for the same task using the same data and model would yield very similar results.Recent research has, however, shown this not to be the case for many NLP experiments.In this paper, we report extensive coordinated work by two NLP groups to run the training and testing pipeline for three neural text simplification models under varying experimental conditions, including different random seeds, run-time environments, and dependency versions, yielding a large number of results for each of the three models using the same data and train/dev/test set splits.From one perspective, these results can be interpreted as shedding light on the reproducibility of evaluation results for the three NTS models, and we present an indepth analysis of the variation observed for different combinations of experimental conditions.From another perspective, the results raise the question of whether the averaged score should be considered the 'true' result for each model.
Temporal and Second Language Influence on Intra-Annotator Agreement and Stability in Hate Speech Labelling

Much work in natural language processing (NLP) relies on human annotation.The majority of this implicitly assumes that annotator's labels are temporally stable, although the reality is that human judgements are rarely consistent over time.As a subjective annotation task, hate speech labels depend on annotator's emotional and moral reactions to the language used to convey the message.Studies in Cognitive Science reveal a 'foreign language effect', whereby people take differing moral positions and perceive offensive phrases to be weaker in their second languages.Does this affect annotations as well?We conduct an experiment to investigate the impacts of (1) time and (2) different language conditions (English and German) on measurements of intra-annotator agreement in a hate speech labelling task.While we do not observe the expected lower stability in the different language condition, we find that overall agreement is significantly lower than is implicitly assumed in annotation tasks, which has important implications for dataset reproducibility in NLP.
Enriching WayúnaikiSpanish Neural Machine Translation with Linguistic Information

We present the first neural machine translation system for the low-resource language pair Wayúunaiki-Spanish and explore strategies to inject linguistic knowledge into the model to improve translation quality.We explore a wide range of methods and combine complementary approaches.Results indicate that incorporating linguistic information through linguistically motivated subword segmentation, factored models, and pretrained embeddings helps the system to generate improved translations, with the segmentation contributing most.In order to evaluate translation quality in a general domain and go beyond the available religious domain data, we gather and make publicly available a new test set and supplementary material.Although translation quality as measured with automatic metrics is low, we hope these resources will facilitate and support further research on Wayúunaiki.
The Diminishing Returns of Masked Language Models to Science

Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks.It has also been demonstrated that the downstream task performance of such models can be improved by pretraining larger models for longer on more data.In this work, we empirically evaluate the extent to which these results extend to tasks in science.We use 14 domain-specific transformer-based models (including SCHOLARBERT, a new 770Mparameter science-focused masked language model pretrained on up to 225B tokens) to evaluate the impact of training data, model size, pretraining and finetuning time on 12 downstream scientific tasks.Interestingly, we find that increasing model size, training data, or compute time does not always lead to significant improvements (i.e., > 1% F1), if any, in scientific information extraction tasks.We offer possible explanations for this surprising result.
NollySenti: Leveraging Transfer Learning and Machine Translation for Nigerian Movie Sentiment Classification

Africa has over 2000 indigenous languages but they are under-represented in NLP research due to lack of datasets.In recent years, there have been progress in developing labelled corpora for African languages.However, they are often available in a single domain and may not generalize to other domains.In this paper, we focus on the task of sentiment classification for crossdomain adaptation.We create a new dataset, NollySenti-based on the Nollywood movie reviews for five languages widely spoken in Nigeria (English, Hausa, Igbo, Nigerian-Pidgin, and Yorùbá).We provide an extensive empirical evaluation using classical machine learning methods and pre-trained language models.Leveraging transfer learning, we compare the performance of cross-domain adaptation from Twitter domain, and cross-lingual adaptation from English language.Our evaluation shows that transfer from English in the same target domain leads to more than 5% improvement in accuracy compared to transfer from Twitter in the same language.To further mitigate the domain difference, we leverage machine translation (MT) from English to other Nigerian languages, which leads to a further improvement of 7% over cross-lingual evaluation.While MT to low-resource languages are often of low quality, through human evaluation, we show that most of the translated sentences preserve the sentiment of the original English reviews.
STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions

We present STT4SG-350 (Speech-to-Text for Swiss German), a corpus of Swiss German speech, annotated with Standard German text at the sentence level.The data is collected using a web app in which the speakers are shown Standard German sentences, which they translate to Swiss German and record.We make the corpus publicly available.It contains 343 hours of speech from all dialect regions and is the largest public speech corpus for Swiss German to date.Application areas include automatic speech recognition (ASR), text-to-speech, dialect identification, and speaker recognition.Dialect information, age group, and gender of the 316 speakers are provided.Genders are equally represented and the corpus includes speakers of all ages.Roughly the same amount of speech is provided per dialect region, which makes the corpus ideally suited for experiments with speech technology for different dialects.We provide training, validation, and test splits of the data.The test set consists of the same spoken sentences for each dialect region and allows a fair evaluation of the quality of speech technologies in different dialects.We train an ASR model on the training set and achieve an average BLEU score of 74.7 on the test set.The model beats the best published BLEU scores on 2 other Swiss German ASR test sets, demonstrating the quality of the corpus.
Language Model Analysis for Ontology Subsumption Inference

Investigating whether pre-trained language models (LMs) can function as knowledge bases (KBs) has raised wide research interests recently.However, existing works focus on simple, triple-based, relational KBs, but omit more sophisticated, logic-based, conceptualised KBs such as OWL ontologies.To investigate an LM's knowledge of ontologies, we propose ONTOLAMA, a set of inferencebased probing tasks and datasets from ontology subsumption axioms involving both atomic and complex concepts 1 .We conduct extensive experiments on ontologies of different domains and scales, and our results demonstrate that LMs encode relatively less background knowledge of Subsumption Inference (SI) than traditional Natural Language Inference (NLI) but can improve on SI significantly when a small number of samples are given.We will open-source our code and datasets. 2
Geographic Citation Gaps in NLP Research

In a fair world, people have equitable opportunities to education, to conduct scientific research, to publish, and to get credit for their work, regardless of where they live.However, it is common knowledge among researchers that a vast number of papers accepted at top NLP venues come from a handful of western countries and (lately) China; whereas, very few papers from Africa and South America get published.Similar disparities are also believed to exist for paper citation counts.In the spirit of "what we do not measure, we cannot improve", this work asks a series of questions on the relationship between geographical location and publication success (acceptance in top NLP venues and citation impact).We first created a dataset of 70,000 papers from the ACL Anthology, extracted their meta-information, and generated their citation network.We then show that not only are there substantial geographical disparities in paper acceptance and citation but also that these disparities persist even when controlling for a number of variables such as venue of publication and sub-field of NLP.Further, despite some steps taken by the NLP community to improve geographical diversity, we show that the disparity in publication metrics across locations is still on an increasing trend since the early 2000s.We release our code and dataset here:
Neural-based Mixture Probabilistic Query Embedding for Answering FOL queries on Knowledge Graphs

Query embedding (QE)-which aims to embed entities and first-order logical (FOL) queries in a vector space, has shown great power in answering FOL queries on knowledge graphs (KGs).Existing QE methods divide a complex query into a sequence of mini-queries according to its computation graph and perform logical operations on the answer sets of mini-queries to get answers.However, most of them assume that answer sets satisfy an individual distribution (e.g., Uniform, Beta, or Gaussian), which is often violated in real applications and limit their performance.In this paper, we propose a Neural-based Mixture Probabilistic Query Embedding Model (NMP-QEM) that encodes the answer set of each mini-query as a mixed Gaussian distribution with multiple means and covariance parameters, which can approximate any random distribution arbitrarily well in real KGs.Additionally, to overcome the difficulty in defining the closed solution of negation operation, we introduce neural-based logical operators of projection, intersection and negation for a mixed Gaussian distribution to answer all the FOL queries.Extensive experiments demonstrate that NMP-QEM significantly outperforms existing stateof-the-art methods on benchmark datasets.In NELL995, NMP-QEM achieves a 31% relative improvement over the state-of-the-art.
Explaining Translationese: why are Neural Classifiers Better and what do they Learn?

Recent work has shown that neural feature-and representation-learning, e.g.BERT, achieves superior performance over traditional manual feature engineering based approaches, with e.g.SVMs, in translationese classification tasks.Previous research did not show (i) whether the difference is because of the features, the classifiers or both, and (ii) what the neural classifiers actually learn.To address (i), we carefully design experiments that swap features between BERT-and SVM-based classifiers.We show that an SVM fed with BERT representations performs at the level of the best BERT classifiers, while BERT learning and using handcrafted features performs at the level of an SVM using handcrafted features.This shows that the performance differences are due to the features.To address (ii) we use integrated gradients and find that (a) there is indication that information captured by hand-crafted features is only a subset of what BERT learns, and (b) part of BERT's top performance results are due to BERT learning topic differences and spurious correlations with translationese.
On Language Spaces, Scales and Cross-Lingual Transfer of UD Parsers

Cross-lingual transfer of parsing models has been shown to work well for several closelyrelated languages, but predicting the success in other cases remains hard.Our study is a comprehensive analysis of the impact of linguistic distance on the transfer of Universal Dependencies (UD) parsers.As an alternative to syntactic typological distances extracted from URIEL, we propose three text-based feature spaces and show that they can be more precise predictors, especially on a more local scale, when only shorter distances are taken into account.Our analysis also reveals that the good coverage in typological databases is not among the factors that explain good transfer.
OTSeq2Set: An Optimal Transport Enhanced Sequence-to-Set Model for Extreme Multi-label Text Classification

Extreme multi-label text classification (XMTC) is the task of finding the most relevant subset labels from an extremely large-scale label collection.Recently, some deep learning models have achieved state-of-the-art results in XMTC tasks.These models commonly predict scores for all labels by a fully connected layer as the last layer of the model.However, such models can't predict a relatively complete and variablelength label subset for each document, because they select positive labels relevant to the document by a fixed threshold or take top k labels in descending order of scores.A less popular type of deep learning models called sequenceto-sequence (Seq2Seq) focus on predicting variable-length positive labels in sequence style.However, the labels in XMTC tasks are essentially an unordered set rather than an ordered sequence, the default order of labels restrains Seq2Seq models in training.To address this limitation in Seq2Seq, we propose an autoregressive sequence-to-set model for XMTC tasks named OTSeq2Set.Our model generates predictions in student-forcing scheme and is trained by a loss function based on bipartite matching which enables permutationinvariance.Meanwhile, we use the optimal transport distance as a measurement to force the model to focus on the closest labels in semantic label space.Experiments show that OT-Seq2Set outperforms other competitive baselines on 4 benchmark datasets.Especially, on the Wikipedia dataset with 31k labels, it outperforms the state-of-the-art Seq2Seq method by 16.34% in micro-F1 score.The code is available at https://github.com/caojie54/OTSeq2Set.
Evidence &gt; Intuition: Transferability Estimation for Encoder Selection

With the increase in availability of large pre-trained language models (LMs) in Natural Language Processing (NLP), it becomes critical to assess their fit for a specific target task a priori-as fine-tuning the entire space of available LMs is computationally prohibitive and unsustainable.However, encoder transferability estimation has received little to no attention in NLP.In this paper, we propose to generate quantitative evidence to predict which LM, out of a pool of models, will perform best on a target task without having to fine-tune all candidates.We provide a comprehensive study on LM ranking for 10 NLP tasks spanning the two fundamental problem types of classification and structured prediction.We adopt the state-of-the-art Logarithm of Maximum Evidence (LogME) measure from Computer Vision (CV) and find that it positively correlates with final LM performance in 94% of the setups.In the first study of its kind, we further compare transferability measures with the de facto standard of human practitioner ranking, finding that evidence from quantitative metrics is more robust than pure intuition and can help identify unexpected LM candidates.
Guiding Neural Story Generation with Reader Models

Automated storytelling has long captured the attention of researchers for the ubiquity of narratives in everyday life.However, it is challenging to maintain coherence and stay on-topic toward a specific ending when generating narratives with neural language models.In this paper, we introduce Story generation with Reader Models (StoRM) 1 , a framework in which a reader model is used to reason about the story should progress.A reader model infers what a human reader believes about the concepts, entities, and relations about the fictional story world.We show how an explicit reader model represented as a knowledge graph affords story coherence and provides controllability in the form of achieving a given story world state goal.Experiments show that our model produces significantly more coherent and on-topic stories, outperforming baselines in dimensions including plot plausibility and staying on topic.Jenny took shower.Jenny went home.Jenny moved to Miami for finding a job.
Momentum Contrastive Pre-training for Question Answering

Existing pre-training methods for extractive Question Answering (QA) generate cloze-like queries different from natural questions in syntax structure, which could overfit pre-trained models to simple keyword matching.In order to address this problem, we propose a novel Momentum Contrastive pRe-training fOr queStion anSwering (MCROSS) method for extractive QA.Specifically, MCROSS introduces a momentum contrastive learning framework to align the answer probability between cloze-like and natural query-passage sample pairs.Hence, the pre-trained models can better transfer the knowledge learned in cloze-like samples to answering natural questions.Experimental results on three benchmarking QA datasets show that our method achieves noticeable improvement compared with all baselines in both supervised and zero-shot scenarios.
What Makes Data-to-Text Generation Hard for Pretrained Language Models?

Expressing natural language descriptions of structured facts or relations -data-to-text generation (D2T) -increases the accessibility of structured knowledge repositories.Previous work (Nan et al., 2020) shows that pre-trained language models (PLMs) perform remarkably well on this task after fine-tuning on a significant amount of task-specific training data.On the other hand, while auto-regressive PLMs can generalize from a few task examples, their efficacy at D2T is largely unexplored.Furthermore, we have an incomplete understanding of the limits of PLMs on D2T.In this work, we conduct an empirical study of both finetuned and auto-regressive PLMs on the DART multi-domain D2T dataset.We consider their performance as a function of the amount of task-specific data and how the data is incorporated into the models: zero and few-shot learning, and fine-tuning of model weights.In addition, we probe the limits of PLMs by measuring performance on subsets of the evaluation data: novel predicates and abstractive test examples.To improve the performance on these subsets, we investigate two techniques: providing predicate descriptions in the context and re-ranking generated candidates by information reflected in the source.Finally, we conduct a human evaluation of model errors and show that D2T generation tasks would benefit from datasets with more careful manual curation.
Distinguishing between focus and background entities in biomedical corpora using discourse structure and transformers

Scientific documents typically contain numerous entity mentions, while only a subset are directly relevant to the key contributions of the paper.Distinguishing these focus entities from background ones effectively could improve the recovery of relevant documents and the extraction of information from documents.To study the identification of focus entities, we developed two large datasets of diseasecausing biological pathogens using MEDLINE, the largest collection of biomedical citations, and PubMed Central, a collection of full text articles.The focus entities were identified using human-curated indexing on these collections.Experiments with machine learning methods to identify focus entities show that transformer methods achieve high precision and recall and that document discourse information is relevant.The work lays the foundation for more targeted retrieval/summarisation of entity-relevant documents.
On the Impact of Temporal Concept Drift on Model Explanations

Explanation faithfulness of model predictions in natural language processing is typically evaluated on held-out data from the same temporal distribution as the training data (i.e.synchronous settings).While model performance often deteriorates due to temporal variation (i.e.temporal concept drift), it is currently unknown how explanation faithfulness is impacted when the time span of the target data is different from the data used to train the model (i.e.asynchronous settings).For this purpose, we examine the impact of temporal variation on model explanations extracted by eight feature attribution methods and three select-then-predict models across six text classification tasks.Our experiments show that (i) faithfulness is not consistent under temporal variations across feature attribution methods (e.g. it decreases or increases depending on the method), with an attention-based method demonstrating the most robust faithfulness scores across datasets; and (ii) select-then-predict models are mostly robust in asynchronous settings with only small degradation in predictive performance.Finally, feature attribution methods show conflicting behavior when used in FRESH (i.e. a selectand-predict model) and for measuring sufficiency/comprehensiveness (i.e. as post-hoc methods), suggesting that we need more robust metrics to evaluate post-hoc explanation faithfulness. 1
Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference

It has been shown that NLI models are usually biased with respect to the word-overlap between premise and hypothesis; they take this feature as a primary cue for predicting the entailment label.In this paper, we focus on an overlooked aspect of the overlap bias in NLI models: the reverse word-overlap bias.Our experimental results demonstrate that current NLI models are highly biased towards the non-entailment label on instances with low overlap, and the existing debiasing methods, which are reportedly successful on existing challenge datasets, are generally ineffective in addressing this category of bias.We investigate the reasons for the emergence of the overlap bias and the role of minority examples in its mitigation.For the former, we find that the word-overlap bias does not stem from pre-training, and for the latter, we observe that in contrast to the accepted assumption, eliminating minority examples does not affect the generalizability of debiasing methods with respect to the overlap bias.All the code and relevant data are available at: https: //github.com/sara-rajaee/reverse_biasOverlap Sample Label Full (1.0) P: A little kid in blue is sledding down a snowy hill.H: A little kid in blue sledding.Entailment P: The young lady is giving the old man a hug.H: The young man is giving the old man a hug.Non-Entailment 12 13 = 0.923 P: A woman in a blue shirt and green hat looks up at the camera.H: A woman wearing a blue shirt and green hat looks at the camera Entailment 11 12 = 0.917 P: Two men in wheelchairs are reaching in the air for a basketball.H: Two women in wheelchairs are reaching in the air for a basketball.Non