Reviewerly/exHarmony huggingface dataset:
A Continuum of Generation Tasks for Investigating Length Bias and Degenerate Repetition

Language models suffer from various degenerate behaviors.These differ between tasks: machine translation (MT) exhibits length bias, while tasks like story generation exhibit excessive repetition.Recent work has attributed the difference to task constrainedness, but evidence for this claim has always involved many confounding variables.To study this question directly, we introduce a new experimental framework that allows us to smoothly vary task constrainedness, from MT at one end to fully open-ended generation at the other, while keeping all other aspects fixed.We find that:(1) repetition decreases smoothly with constrainedness, explaining the difference in repetition across tasks; (2) length bias surprisingly also decreases with constrainedness, suggesting some other cause for the difference in length bias; (3) across the board, these problems affect the mode, not the whole distribution; (4) the differences cannot be attributed to a change in the entropy of the distribution, since another method of changing the entropy, label smoothing, does not produce the same effect.
Wider &amp; Closer: Mixture of Short-channel Distillers for Zero-shot Cross-lingual Named Entity Recognition

Zero-shot cross-lingual named entity recognition (NER) aims at transferring knowledge from annotated and rich-resource data in source languages to unlabeled and lean-resource data in target languages.Existing mainstream methods based on the teacher-student distillation framework ignore the rich and complementary information lying in the intermediate layers of pre-trained language models, and domaininvariant information is easily lost during transfer.In this study, a mixture of short-channel distillers (MSD) method is proposed to fully interact the rich hierarchical information in the teacher model and to transfer knowledge to the student model sufficiently and efficiently.Concretely, a multi-channel distillation framework is designed for sufficient information transfer by aggregating multiple distillers as a mixture.Besides, an unsupervised method adopting parallel domain adaptation is proposed to shorten the channels between the teacher and student models to preserve domaininvariant features.Experiments on four datasets across nine languages demonstrate that the proposed method achieves new state-of-the-art performance on zero-shot cross-lingual NER and shows great generalization and compatibility across languages and fields.
Unsupervised Term Extraction for Highly Technical Domains

Term extraction is an information extraction task at the root of knowledge discovery platforms.Developing term extractors that are able to generalize across very diverse and potentially highly technical domains is challenging, as annotations for domains requiring indepth expertise are scarce and expensive to obtain.In this paper, we describe the term extraction subsystem of a commercial knowledge discovery platform that targets highly technical fields such as pharma, medical, and material science.To be able to generalize across domains, we introduce a fully unsupervised annotator (UA).It extracts terms by combining novel morphological signals from sub-word tokenization with term-to-topic and intra-term similarity metrics, computed using general-domain pre-trained sentence-encoders.The annotator is used to implement a weaklysupervised setup, where transformer-models are fine-tuned (or pre-trained) over the training data generated by running the UA over large unlabeled corpora.Our experiments demonstrate that our setup can improve the predictive performance while decreasing the inference latency on both CPUs and GPUs.Our annotators provide a very competitive baseline for all the cases where annotations are not available.
Balanced Adversarial Training: Balancing Tradeoffs between Fickleness and Obstinacy in NLP Models

Traditional (fickle) adversarial examples involve finding a small perturbation that does not change an input's true label but confuses the classifier into outputting a different prediction.Conversely, obstinate adversarial examples occur when an adversary finds a small perturbation that preserves the classifier's prediction but changes the true label of an input.Adversarial training and certified robust training have shown some effectiveness in improving the robustness of machine learnt models to fickle adversarial examples.We show that standard adversarial training methods focused on reducing vulnerability to fickle adversarial examples may make a model more vulnerable to obstinate adversarial examples, with experiments for both natural language inference and paraphrase identification tasks.To counter this phenomenon, we introduce Balanced Adversarial Training, which incorporates contrastive learning to increase robustness against both fickle and obstinate adversarial examples.
Camelira: An Arabic Multi-Dialect Morphological Disambiguator

We present Camelira, a web-based Arabic multi-dialect morphological disambiguation tool that covers four major variants of Arabic: Modern Standard Arabic, Egyptian, Gulf, and Levantine.Camelira offers a user-friendly web interface that allows researchers and language learners to explore various linguistic information, such as part-of-speech, morphological features, and lemmas.Our system also provides an option to automatically choose an appropriate dialect-specific disambiguator based on the prediction of a dialect identification component.
Generative Multi-hop Retrieval

A common practice for text retrieval is to use an encoder to map the documents and the query to a common vector space and perform a nearest neighbor search (NNS); multi-hop retrieval also often adopts the same paradigm, usually with a modification of iteratively reformulating the query vector so that it can retrieve different documents at each hop.However, such a biencoder approach has limitations in multi-hop settings; (1) the reformulated query gets longer as the number of hops increases, which further tightens the embedding bottleneck of the query vector, and (2) it is prone to error propagation.In this paper, we focus on alleviating these limitations in multi-hop settings by formulating the problem in a fully generative way.We propose an encoder-decoder model that performs multi-hop retrieval by simply generating the entire text sequences of the retrieval targets, which means the query and the documents interact in the language model's parametric space rather than L2 or inner product space as in the bi-encoder approach.Our approach, Generative Multi-hop Retrieval (GMR), consistently achieves comparable or higher performance than bi-encoder models in five datasets while demonstrating superior GPU memory and storage footprint.
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection

Language models increasingly rely on massive web crawls for diverse text data.However, these sources are rife with undesirable content.As such, resources like Wikipedia, books, and news often serve as anchors for automatically selecting web text most suitable for language modeling, a process typically referred to as quality filtering.Using a new dataset of U.S. high school newspaper articles-written by students from across the country-we investigate whose language is preferred by the quality filter used for GPT-3.We find that newspapers from larger schools, located in wealthier, educated, and urban zones (ZIP codes) are more likely to be classified as high quality.We also show that this quality measurement is unaligned with other sensible metrics, such as factuality or literary acclaim.We argue that privileging any corpus as high quality entails a language ideology, and more care is needed to construct training corpora for language models, with better transparency and justification for the inclusion or exclusion of various texts.
Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU

Curriculum Learning (CL) is a technique of training models via ranking examples in a typically increasing difficulty trend with the aim of accelerating convergence and improving generalisability.Current approaches for Natural Language Understanding (NLU) tasks use CL to improve in-distribution data performance often via heuristic-oriented or task-agnostic difficulties.In this work, instead, we employ CL for NLU by taking advantage of training dynamics as difficulty metrics, i.e. statistics that measure the behavior of the model at hand on specific task-data instances during training and propose modifications of existing CL schedulers based on these statistics.Differently from existing works, we focus on evaluating models on in-distribution (ID), out-of-distribution (OOD) as well as zero-shot (ZS) cross-lingual transfer datasets.We show across several NLU tasks that CL with training dynamics can result in better performance mostly on zero-shot cross-lingual transfer and OOD settings with improvements up by 8.5% in certain cases.Overall, experiments indicate that training dynamics can lead to better performing models with smoother training compared to other difficulty metrics while being 20% faster on average.In addition, through analysis we shed light on the correlations of task-specific versus task-agnostic metrics 1 .
SLING: Sino Linguistic Evaluation of Large Language Models

To understand what kinds of linguistic knowledge are encoded by pretrained Chinese language models (LMs), we introduce the benchmark of Sino LINGuistics (SLING), which consists of 38K minimal sentence pairs in Mandarin Chinese grouped into 9 high-level linguistic phenomena.Each pair demonstrates the acceptability contrast of a specific syntactic or semantic phenomenon (e.g., The keys are lost vs.The keys is lost), and an LM should assign lower perplexity to the acceptable sentence.In contrast to the CLiMP dataset (Xiang et al., 2021), which also contains Chinese minimal pairs and was created by translating the vocabulary of the English BLiMP dataset, the minimal pairs in SLING are derived primarily by applying syntactic and lexical transformations to naturally-occurring, linguist-annotated sentences from the Chinese Treebank 9.0, thus addressing severe issues in CLiMP's data generation process.We test 18 publicly available pretrained monolingual (e.g., BERT-base-zh, CPM) and multi-lingual (e.g., mT5, XLM) language models on SLING.Our experiments show that the average accuracy for LMs is far below human performance (69.7% vs. 97.1%),while BERT-base-zh achieves the highest accuracy (84.8%) of all tested LMs, even much larger ones.Additionally, we find that most LMs have a strong gender and number (singular/plural) bias, and they perform better on local phenomena than hierarchical ones. 1
Improving Precancerous Case Characterization via Transformer-based Ensemble Learning

The application of natural language processing (NLP) to cancer pathology reports has been focused on detecting cancer cases, largely ignoring precancerous cases.Improving the characterization of precancerous adenomas assists in developing diagnostic tests for early cancer detection and prevention, especially for colorectal cancer (CRC).Here we developed transformerbased deep neural network NLP models to perform the CRC phenotyping, with the goal of extracting precancerous lesion attributes and distinguishing cancer and precancerous cases.We achieved 0.914 macro-F1 scores for classifying patients into negative, non-advanced adenoma, advanced adenoma and CRC.We further improved the performance to 0.923 using an ensemble of classifiers for cancer status classification and lesion size named entity recognition (NER).Our results demonstrated the potential of using NLP to leverage real-world health record data to facilitate the development of diagnostic tests for early cancer prevention.
1Cademy @ Causal News Corpus 2022: Enhance Causal Span Detection via Beam-Search-based Position Selector

In this paper, we present our approach and empirical observations for Cause-Effect Signal Span Detection-Subtask 2 of Shared task 3 (Tan et al., 2022a) at CASE 2022.The shared task aims to extract the cause, effect, and signal spans from a given causal sentence.We model the task as a reading comprehension (RC) problem and apply a token-level RC-based span prediction paradigm to the task as the baseline.We explore different training objectives to fine-tune the model, as well as data augmentation (DA) tricks based on the language model (LM) for performance improvement.Additionally, we propose an efficient beam-search post-processing strategy to due with the drawbacks of span detection to obtain a further performance gain.Our approach achieves an average F 1 score of 54.15 and ranks 1 st in the CASE competition.Our code is available at https://github.com/Gzhang-umich/1CademyTeamOfCASE.
End-to-End Speech Translation of Arabic to English Broadcast News

Speech translation (ST) is the task of directly translating acoustic speech signals in a source language into text in a foreign language.ST task has been addressed, for a long time, using a pipeline approach with two modules : first an Automatic Speech Recognition (ASR) in the source language followed by a text-totext Machine translation (MT).In the past few years, we have seen a paradigm shift towards the end-to-end approaches using sequence-tosequence deep neural network models.This paper presents our efforts towards the development of the first Broadcast News end-to-end Arabic to English speech translation system.Starting from independent ASR and MT LDC releases, we were able to identify about 92 hours of Arabic audio recordings for which the manual transcription was also translated into English at the segment level.These data was used to train and compare pipeline and end-toend speech translation systems under multiple scenarios including transfer learning and data augmentation techniques.
EventGraph at CASE 2021 Task 1: A General Graph-based Approach to Protest Event Extraction

This paper presents our submission to the 2022 edition of the CASE 2021 shared task 1, subtask 4. The EventGraph system adapts an endto-end, graph-based semantic parser to the task of Protest Event Extraction and more specifically subtask 4 on event trigger and argument extraction.We experiment with various graphs, encoding the events as either "labeled-edge" or "node-centric" graphs.We show that the "nodecentric" approach yields best results overall, performing well across the three languages of the task, namely English, Spanish, and Portuguese.EventGraph is ranked 3rd for English and Portuguese, and 4th for Spanish.
BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems

We present BotSIM, a data-efficient end-toend Bot SIMulation framework for commercial text-based task-oriented dialog (TOD) systems.BotSIM consists of three major components: 1) a Generator that can infer semanticlevel dialog acts and entities from bot definitions and generate user queries via model-based paraphrasing; 2) an agenda-based dialog user Simulator (ABUS) to simulate conversations with the dialog agents; 3) a Remediator to analyze the simulated conversations, visualize the bot health reports and provide actionable remediation suggestions for bot troubleshooting and improvement.We demonstrate BotSIM's effectiveness in end-to-end evaluation, remediation and multi-intent dialog generation via case studies on two commercial bot platforms.BotSIM's "generation-simulation-remediation" paradigm accelerates the end-to-end bot evaluation and iteration process by: 1) reducing manual test cases creation efforts; 2) enabling a holistic gauge of the bot in terms of NLU and end-to-end performance via extensive dialog simulation; 3) improving the bot troubleshooting process with actionable suggestions.A demo of our system can be found at https: //tinyurl.com
A Comprehensive Comparison of Neural Networks as Cognitive Models of Inflection

Neural networks have long been at the center of a debate around the cognitive mechanism by which humans process inflectional morphology.This debate has gravitated into NLP by way of the question: Are neural networks a feasible account for human behavior in morphological inflection?We address that question by measuring the correlation between human judgments and neural network probabilities for unknown word inflections.We test a larger range of architectures than previously studied on two important tasks for the cognitive processing debate: English past tense, and German number inflection.We find evidence that the Transformer may be a better account of human behavior than LSTMs on these datasets, and that LSTM features known to increase inflection accuracy do not always result in more human-like behavior.
Machine Translation Robustness to Natural Asemantic Variation

Current Machine Translation (MT) models still struggle with more challenging input, such as noisy data and tail-end words and phrases.Several works have addressed this robustness issue by identifying specific categories of noise and variation then tuning models to perform better on them.An important yet under-studied category involves minor variations in nuance (non-typos) that preserve meaning w.r.t. the target language.We introduce and formalize this category as Natural Asemantic Variation (NAV) and investigate it in the context of MT robustness.We find that existing MT models fail when presented with NAV data, but we demonstrate strategies to improve performance on NAV by fine-tuning them with human-generated variations.We also show that NAV robustness can be transferred across languages and find that synthetic perturbations can achieve some but not all of the benefits of organic NAV data.
Linearizing Transformer with Key-Value Memory

Efficient transformer variants with linear time complexity have been developed to mitigate the quadratic computational overhead of the vanilla transformer.Among them are lowrank projection methods such as Linformer and kernel-based Transformers.Despite their unique merits, they usually suffer from a performance drop comparing with the vanilla transformer on many sequence generation tasks, and often fail to obtain computation gain when the generation is short.We propose Mem-Sizer, an approach towards closing the performance gap while improving the efficiency even with short generation.It projects the source sequences into lower dimension representations like Linformer, while enjoying efficient recurrent-style incremental computation similar to kernel-based transformers.This yields linear computation time and constant memory complexity at inference time.Mem-Sizer also employs a lightweight multi-head mechanism which renders the computation as light as a single-head model.We demonstrate that MemSizer provides an improved balance between efficiency and accuracy over the vanilla transformer and other efficient transformer variants in three typical sequence generation tasks, including machine translation, abstractive text summarization, and language modeling.
1Cademy @ Causal News Corpus 2022: Leveraging Self-Training in Causality Classification of Socio-Political Event Data

This paper details our participation in the Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) workshop @ EMNLP 2022, where we take part in Subtask 1 of Shared Task 3 (Tan et al., 2022a).We approach the given task of event causality detection by proposing a self-training pipeline that follows a teacherstudent classifier method.More specifically, we initially train a teacher model on the true, original task data, and use that teacher model to self-label data to be used in the training of a separate student model for the final task prediction.We test how restricting the number of positive or negative self-labeled examples in the self-training process affects classification performance.Our final results show that using self-training produces a comprehensive performance improvement across all models and self-labeled training sets tested within the task of event causality sequence classification.On top of that, we find that selftraining performance did not diminish even when restricting either positive/negative examples used in training.Our code is be publicly available at https://github.com/Gzhangumich/1CademyTeamOfCASE.
SQU-CS @ NADI 2022: Dialectal Arabic Identification using One-vs-One Classification with TF-IDF Weights Computed on Character n-grams

In this paper, I present an approach using one-vs-one classification scheme with TF-IDF term weighting on character n-grams for identifying Arabic dialects used in social media.The scheme was evaluated in the context of the third Nuanced Arabic Dialect Identification (NADI 2022) shared task for identifying Arabic dialects used in Twitter messages.The approach was implemented with logistic regression loss and trained using stochastic gradient decent (SGD) algorithm.This simple method achieved a macro F1 score of 22.89% and 10.83% on TEST A and TEST B, respectively, in comparison to an approach based on AraBERT pretrained transformer model which achieved a macro F1 score of 30.01%and 14.84%, respectively.My submission based on AraBERT scored a macro F1 average of 22.42% and was ranked 10 out of the 19 teams who participated in the task.
Not to Overfit or Underfit the Source Domains? An Empirical Study of Domain Generalization in Question Answering

Machine learning models are prone to overfitting their training (source) domains, which is commonly believed to be the reason why they falter in novel target domains.Here we examine the contrasting view that multi-source domain generalization (DG) is first and foremost a problem of mitigating source domain underfitting: models not adequately learning the signal already present in their multi-domain training data.Experiments on a reading comprehension DG benchmark show that as a model learns its source domains better-using familiar methods such as knowledge distillation (KD) from a bigger model-its zero-shot out-of-domain utility improves at an even faster pace.Improved source domain learning also demonstrates superior out-of-domain generalization over three popular existing DG approaches that aim to limit overfitting.Our implementation of KD-based domain generalization is available via PrimeQA at: https://ibm.biz/domain-generalization-with-kd.
Data-Efficient Auto-Regressive Document Retrieval for Fact Verification

Document retrieval is a core component of many knowledge-intensive natural language processing task formulations such as fact verification and question answering.Sources of textual knowledge, such as Wikipedia articles, condition the generation of answers from the models.Recent advances in retrieval use sequence-to-sequence models to incrementally predict the title of the appropriate Wikipedia page given a query.However, this method requires supervision in the form of human annotation to label which Wikipedia pages contain appropriate context.This paper introduces a distant-supervision method that does not require any annotation to train autoregressive retrievers that attain competitive R-Precision and Recall in a zero-shot setting.Furthermore we show that with task-specific supervised finetuning, autoregressive retrieval performance for two Wikipedia-based fact verification tasks can approach or even exceed full supervision using less than 1/4 of the annotated data indicating possible directions for data-efficient autoregressive retrieval.