Reviewerly/exHarmony huggingface dataset:
Controlling for Stereotypes in Multimodal Language Model Evaluation

We propose a methodology and design two benchmark sets for measuring to what extent language-and-vision language models use the visual signal in the presence or absence of stereotypes.The first benchmark is designed to test for stereotypical colors of common objects, while the second benchmark considers gender stereotypes.The key idea is to compare predictions when the image conforms to the stereotype to predictions when it does not.Our results show that there is significant variation among multimodal models: the recent Transformer-based FLAVA seems to be more sensitive to the choice of image and less affected by stereotypes than older CNN-based models such as VisualBERT and LXMERT.This effect is more discernible in this type of controlled setting than in traditional evaluations where we do not know whether the model relied on the stereotype or the visual signal.
Arabic Sentiment Analysis by Pretrained Ensemble

This paper presents the 259 team’s BERT ensemble designed for the NADI 2022 Subtask 2 (sentiment analysis) (Abdul-Mageed et al., 2022). Twitter Sentiment analysis is one of the language processing (NLP) tasks that provides a method to understand the perception and emotions of the public around specific topics. The most common research approach focuses on obtaining the tweet’s sentiment by analyzing its lexical and syntactic features. We used multiple pretrained Arabic-Bert models with a simple average ensembling and then chose the best-performing ensemble on the training dataset and ran it on the development dataset. This system ranked 3rd in Subtask 2 with a Macro-PN-F1-score of 72.49%.
GGNN@Causal News Corpus 2022:Gated Graph Neural Networks for Causal Event Classification from Social-Political News Articles

The discovery of causality mentions from text is a core cognitive concept and appears in many natural language processing (NLP) applications.In this paper, we study the task of Event Causality Identification (ECI) from social-political news.The aim of the task is to detect causal relationships between event mention pairs in text.Although deep learning models have recently achieved a state-of-theart performance on many tasks and applications in NLP, most of them still fail to capture rich semantic and syntactic structures within sentences which is key for causality classification.We present a solution for causal event detection from social-political news that captures semantic and syntactic information based on gated graph neural networks (GGNN) and contextualized language embeddings.Experimental results show that our proposed method outperforms the baseline model (BERT (Bidirectional Embeddings from Transformers) in terms of f 1-score and accuracy.
DiffG-RL: Leveraging Difference between Environment State and Common Sense

Taking into account background knowledge as the context has always been an important part of solving tasks that involve natural language.One representative example of such tasks is text-based games, where players need to make decisions based on both description text previously shown in the game, and their own background knowledge about the language and common sense.In this work, we investigate not simply giving common sense, as can be seen in prior research, but also its effective usage.We assume that a part of the environment states different from common sense should constitute one of the grounds for action selection.We propose a novel agent, DiffG-RL, which constructs a Difference Graph that organizes the environment states and common sense by means of interactive objects with a dedicated graph encoder.DiffG-RL also contains a framework for extracting the appropriate amount and representation of common sense from the source to support the construction of the graph.We validate DiffG-RL in experiments with text-based games that require common sense and show that it outperforms baselines by 17% of scores.We will make our code publicly available.
Fine-grained Multi-lingual Disentangled Autoencoder for Language-agnostic Representation Learning

Encoding both language-specific and languageagnostic information into a single highdimensional space is a common practice of pre-trained Multi-lingual Language Models (pMLM).Such encoding has been shown to perform effectively on natural language tasks requiring semantics of the whole sentence (e.g., translation).However, its effectiveness appears to be limited on tasks requiring partial information of the utterance (e.g., multi-lingual entity retrieval, template retrieval, and semantic alignment).In this work, a novel Fine-grained Multilingual Disentangled Autoencoder (FMDA) is proposed to disentangle fine-grained semantic information from language-specific information in a multi-lingual setting.FMDA is capable of successfully extracting the disentangled template semantic and residual semantic representations.Experiments conducted on the MASSIVE dataset demonstrate that the disentangled encoding can boost each other during the training, thus consistently outperforming the original pMLM and the strong language disentanglement baseline on monolingual template retrieval and cross-lingual semantic retrieval tasks across multiple languages.
Comparative Analysis of Cross-lingual Contextualized Word Embeddings

Contextualized word embeddings have emerged as the most important tool for performing NLP tasks in a large variety of languages.In order to improve the crosslingual representation and transfer learning quality, contextualized embedding alignment techniques, such as mapping and model fine-tuning, are employed.Existing techniques however are time-, data-and computational resource-intensive.In this paper we analyze these techniques by utilizing three tasks: bilingual lexicon induction (BLI), word retrieval and cross-lingual natural language inference (XNLI) for a high resource (German-English) and a low resource (Bengali-English) language pair.In contrast to previous works which focus only on a few popular models, we compare five multilingual and seven monolingual language models and investigate the effect of various aspects on their performance, such as vocabulary size, number of languages used for training and number of parameters.Additionally, we propose a parameter-, dataand runtime-efficient technique which can be trained with 10% of the data, less than 10% of the time and have less than 5% of the trainable parameters compared to model fine-tuning.We show that our proposed method is competitive with resource heavy models, even outperforming them in some cases, even though it relies on less resources.
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis

Is it possible to build a general and automatic natural language generation (NLG) evaluation metric?Existing learned metrics either perform unsatisfactorily or are restricted to tasks where large human rating data is already available.We introduce SESCORE, a model-based metric that is highly correlated with human judgements without requiring human annotation, by utilizing a novel, iterative error synthesis and severity scoring pipeline.This pipeline applies a series of plausible errors to raw text and assigns severity labels by simulating human judgements with entailment.We evaluate SESCORE against existing metrics by comparing how their scores correlate with human ratings.SESCORE outperforms all prior unsupervised metrics on multiple diverse NLG tasks including machine translation, image captioning, and WebNLG text generation.For WMT 20/21 En-De and Zh-En, SESCORE improve the average Kendall correlation with human judgement from 0.154 to 0.195.SESCORE even achieves comparable performance to the best supervised metric COMET, despite receiving no humanannotated training data.
C5L7: A Zero-Shot Algorithm for Intent and Slot Detection in Multilingual Task Oriented Languages: A Zero-Shot Algorithm for Intent and Slot Detection in Multilingual Task Oriented Languages

Voice assistants are becoming central to our lives.The convenience of using voice assistants to do simple tasks has created an industry for voice-enabled devices like TVs, thermostats, air conditioners, etc.It has also improved the quality of life of elders by making the world more accessible.Voice assistants engage in task-oriented dialogues using machinelearned language understanding models.However, training deep-learned models take a lot of training data, which is time-consuming and expensive.Furthermore, it is even more problematic if we want the voice assistant to understand hundreds of languages.In this paper, we present a zero-shot deep learning algorithm that uses only the English part of the Massive dataset and achieves a high level of accuracy across 51 languages.The algorithm uses delexicalized translation to generate a multilingual parallel corpus with intent and slot labels for data augmentation.The training data is further weighted to improve the accuracy of the worst-performing languages.We report on our experiments with code-switching, word order, multilingual ensemble methods and other techniques and their impact on overall accuracy.
Factual Error Correction for Abstractive Summaries Using Entity Retrieval

Despite the recent advancements in abstractive summarization systems leveraged from largescale datasets and pre-trained language models, the factual correctness of the summary is still insufficient.One line of trials to mitigate this problem is to include a post-editing process that can detect and correct factual errors in the summary.In building such a system, it is strongly required that 1) the process has a high success rate and interpretability and 2) it has a fast running time.Previous approaches focus on the regeneration of the summary, resulting in low interpretability and high computing resources.In this paper, we propose an efficient factual error correction system RFEC based on entity retrieval.RFEC first retrieves the evidence sentences from the original document by comparing the sentences with the target summary to reduce the length of the text to analyze.Next, RFEC detects entity-level errors in the summaries using the evidence sentences and substitutes the wrong entities with the accurate entities from the evidence sentences.Experimental results show that our proposed error correction system shows more competitive performance than baseline methods in correcting factual errors with a much faster speed. 1 1 https://github.com/hwanheelee1993/RFEC
NaturalAdversaries: Can Naturalistic Adversaries Be as Effective as Artificial Adversaries?

While a substantial body of prior work has explored adversarial example generation for natural language understanding tasks, these examples are often unrealistic and diverge from the real-world data distributions.In this work, we introduce a two-stage adversarial example generation framework (NaturalAdversaries), for designing adversaries that are effective at fooling a given classifier and demonstrate naturallooking failure cases that could plausibly occur during in-the-wild deployment of the models.At the first stage a token attribution method is used to summarize a given classifier's behaviour as a function of the key tokens in the input.In the second stage a generative model is conditioned on the key tokens from the first stage.NaturalAdversaries is adaptable to both black-box and white-box adversarial attacks based on the level of access to the model parameters.Our results indicate these adversaries generalize across domains, and offer insights for future research on improving robustness of neural text classification models.
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation

Counterfactual data augmentation (CDA) -i.e., adding minimally perturbed inputs during training -helps reduce model reliance on spurious correlations and improves generalization to out-of-distribution (OOD) data.Prior work on generating counterfactuals only considered restricted classes of perturbations, limiting their effectiveness.We present COunterfactual Generation via Retrieval and Editing (CORE), a retrieval-augmented generation framework for creating diverse counterfactual perturbations for CDA.For each training example, CORE first performs a dense retrieval over a taskrelated unlabeled text corpus using a learned bi-encoder and extracts relevant counterfactual excerpts.CORE then incorporates these into prompts to a large language model with fewshot learning capabilities, for counterfactual editing.Conditioning language model edits on naturally occurring data results in diverse perturbations.Experiments on natural language inference and sentiment analysis benchmarks show that CORE counterfactuals are more effective at improving generalization to OOD data compared to other DA approaches.We also show that the CORE retrieval framework can be used to encourage diversity in manually authored perturbations 1 .
Towards Fair Dataset Distillation for Text Classification

With the growing prevalence of large-scale language models, their energy footprint and potential to learn and amplify historical biases are two pressing challenges.Dataset distillation (DD) -a method for reducing the dataset size by learning a small number of synthetic samples which encode the information in the original dataset -is a method for reducing the cost of model training, however its impact on fairness has not been studied.We investigate how DD impacts on group bias in the context of text classification tasks, with experiments over two data sets, concluding that vanilla DD preserves the bias of the dataset.We then show how existing debiasing methods can be combined with DD to produce models that are fair and accurate, at reduced training cost.
Improving Dialogue Act Recognition with Augmented Data

We present our work on augmenting dialogue act recognition capabilities utilizing synthetically generated data.Our work is motivated by the limitations of current dialogue act datasets, and the need to adapt for new domains as well as ambiguity in utterances written by humans.We list our observations and findings towards how synthetically generated data can contribute meaningfully towards more robust dialogue act recognition models extending to new domains.Our major finding shows that synthetic data, which is linguistically varied, can be very useful towards this goal and increase the performance from 0.39, 0.16 to 0.85, 0.88 for AF-FIRM and NEGATE dialogue acts respectively.
CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structure

Code pre-trained models (CodePTMs) have recently demonstrated significant success in code intelligence.To interpret these models, some probing methods have been applied.However, these methods fail to consider the inherent characteristics of codes.In this paper, to address the problem, we propose a novel probing method CAT-probing to quantitatively interpret how CodePTMs attend code structure.We first denoise the input code sequences based on the token types pre-defined by the compilers to filter those tokens whose attention scores are too small.After that, we define a new metric CAT-score to measure the commonality between the tokenlevel attention scores generated in CodePTMs and the pair-wise distances between corresponding AST nodes.The higher the CATscore, the stronger the ability of CodePTMs to capture code structure.We conduct extensive experiments to integrate CAT-probing with representative CodePTMs for different programming languages.Experimental results show the effectiveness of CAT-probing in CodePTM interpretation.Our codes and
One size does not fit all: Investigating strategies for differentially-private learning across NLP tasks

Preserving privacy in contemporary NLP models allows us to work with sensitive data, but unfortunately comes at a price.We know that stricter privacy guarantees in differentiallyprivate stochastic gradient descent (DP-SGD) generally degrade model performance.However, previous research on the efficiency of DP-SGD in NLP is inconclusive or even counter-intuitive.In this short paper, we provide an extensive analysis of different privacy preserving strategies on seven downstream datasets in five different 'typical' NLP tasks with varying complexity using modern neural models based on BERT and XtremeDistil architectures.We show that unlike standard non-private approaches to solving NLP tasks, where bigger is usually better, privacypreserving strategies do not exhibit a winning pattern, and each task and privacy regime requires a special treatment to achieve adequate performance.
AMAL: Meta Knowledge-Driven Few-Shot Adapter Learning

NLP has advanced greatly together with the proliferation of Transformer-based pre-trained language models.To adapt to a downstream task, the pre-trained language models need to be fine-tuned with a sufficient supply of annotated examples.In recent years, Adapter-based fine-tuning methods have expanded the applicability of pre-trained language models by substantially lowering the required amount of annotated examples.However, existing Adapterbased methods still fail to yield meaningful results in the few-shot regime where only a few annotated examples are provided.In this study, we present a meta-learning-driven low-rank adapter pooling method, called AMAL, for leveraging pre-trained language models even with just a few data points.We evaluate our method on five text classification benchmark datasets.The results show that AMAL significantly outperforms previous few-shot learning methods and achieves a new state-of-the-art.
EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching

Accurate alignment between languages is fundamental for improving cross-lingual pretrained language models (XLMs).Motivated by the natural phenomenon of code-switching (CS) in multilingual speakers, CS has been used as an effective data augmentation method that offers language alignment at word-or phraselevel, in contrast to sentence-level via parallel instances.Existing approaches either use dictionaries or parallel sentences with wordalignment to generate CS data by randomly switching words in a sentence.However, such methods can be suboptimal as dictionaries disregard semantics, and syntax might become invalid after random word switching.In this work, we propose ENTITYCS, a method that focuses on ENTITY-level Code-Switching to capture fine-grained cross-lingual semantics without corrupting syntax.We use Wikidata and the English Wikipedia to construct an entitycentric CS corpus by switching entities to their counterparts in other languages.We further propose entity-oriented masking strategies during intermediate model training on the ENTI-TYCS corpus for improving entity prediction.Evaluation of the trained models on four entitycentric downstream tasks shows consistent improvements over the baseline with a notable increase of 10% in Fact Retrieval.We release the corpus and models to assist research on codeswitching and enriching XLMs with external knowledge 1 .
Holistic Sentence Embeddings for Better Out-of-Distribution Detection

Detecting out-of-distribution (OOD) instances is significant for the safe deployment of NLP models.Among recent textual OOD detection works based on pretrained language models (PLMs), distance-based methods have shown superior performance.However, they estimate sample distance scores in the last-layer CLS embedding space and thus do not make full use of linguistic information underlying in PLMs.To address the issue, we propose to boost OOD detection by deriving more holistic sentence embeddings.On the basis of the observations that token averaging and layer combination contribute to improving OOD detection, we propose a simple embedding approach named Avg-Avg, which averages all token representations from each intermediate layer as the sentence embedding and significantly surpasses the state-of-the-art on a comprehensive suite of benchmarks by a 9.33% FAR95 margin.Furthermore, our analysis demonstrates that it indeed helps preserve general linguistic knowledge in fine-tuned PLMs and substantially benefits detecting background shifts.The simple yet effective embedding method can be applied to fine-tuned PLMs with negligible extra costs, providing a free gain in OOD detection.Our code is available at https: //github.com/lancopku/Avg-Avg.
LawngNLI: A Long-Premise Benchmark for In-Domain Generalization from Short to Long Contexts and for Implication-Based Retrieval

Natural language inference has trended toward studying contexts beyond the sentence level.An important application area is law: past cases often do not foretell how they apply to new situations and implications must be inferred.This paper introduces LawngNLI, constructed from U.S. legal opinions with automatic labels with high humanvalidated accuracy.Premises are long and multigranular.Experiments show two use cases.First, LawngNLI can benchmark for indomain generalization from short to long contexts.It has remained unclear if large-scale long-premise NLI datasets actually need to be constructed: near-top performance on long premises could be achievable by fine-tuning using short premises.Without multigranularity, benchmarks cannot distinguish lack of finetuning on long premises versus domain shift between short and long datasets.In contrast, our long and short premises share the same examples and domain.Models fine-tuned using several past NLI datasets and/or our short premises fall short of top performance on our long premises.So for at least certain domains (such as ours), large-scale long-premise datasets are needed.Second, LawngNLI can benchmark for implication-based retrieval.Queries are entailed or contradicted by target documents, allowing users to move between arguments and evidence.Leading retrieval models perform reasonably zero shot on a LawngNLI-derived retrieval task.We compare different systems for re-ranking, including lexical overlap and cross-encoders finetuned using a modified LawngNLI or past NLI datasets.LawngNLI can train and test systems for implication-based case retrieval and argumentation.
Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes

Providing better language tools for lowresource and endangered languages is imperative for equitable growth.Recent progress with massively multilingual pretrained models has proven surprisingly effective at performing zero-shot transfer to a wide variety of languages.However, this transfer is not universal, with many languages not currently understood by multilingual approaches.It is estimated that only 72 languages possess a "small set of labeled datasets" on which we could test a model's performance, the vast majority of languages not having the resources available to simply evaluate performances on.In this work, we attempt to clarify which languages do and do not currently benefit from such transfer.To that end, we develop a general approach that requires only unlabelled text to detect which languages are not well understood by a cross-lingual model.Our approach is derived from the hypothesis that if a model's understanding is insensitive to perturbations to text in a language, it is likely to have a limited understanding of that language.We construct a cross-lingual sentence similarity task to evaluate our approach empirically on 350, primarily low-resource, languages.