Reviewerly/exHarmony huggingface dataset:
Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models

English pretrained language models, which make up the backbone of many modern NLP systems, require huge amounts of unlabeled training data.These models are generally presented as being trained only on English text but have been found to transfer surprisingly well to other languages.We investigate this phenomenon and find that common English pretraining corpora actually contain significant amounts of non-English text: even when less than 1% of data is not English (well within the error rate of strong language classifiers), this leads to hundreds of millions of foreign language tokens in large-scale datasets.We then demonstrate that even these small percentages of non-English data facilitate cross-lingual transfer for models trained on them, with target language performance strongly correlated to the amount of in-language data seen during pretraining.In light of these findings, we argue that no model is truly monolingual when pretrained at scale, which should be considered when evaluating cross-lingual transfer.
How (Un)Faithful is Attention?

Although attention weights have been commonly used as a means to provide explanations for deep learning models, the approach has been widely criticized due to its lack of faithfulness.In this work, we present a simple approach to compute the newly proposed metric AtteFa, which can quantitatively represent the degree of faithfulness of the attention weights.Using this metric, we further validate the effect of the frequency of informative input elements and the use of contextual vs. noncontextual encoders on the faithfulness of the attention mechanism.Finally, we apply the approach on several real-life binary classification datasets to measure the faithfulness of attention weights in real-life settings.
LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models

The opaque nature and unexplained behavior of transformer-based language models (LMs) have spurred a wide interest in interpreting their predictions.However, current interpretation methods mostly focus on probing models from outside, executing behavioral tests, and analyzing salience input features, while the internal prediction construction process is largely not understood.In this work, we introduce LM-Debugger, an interactive debugger tool for transformer-based LMs, which provides a fine-grained interpretation of the model's internal prediction process, as well as a powerful framework for intervening in LM behavior.For its backbone, LM-Debugger relies on a recent method that interprets the inner token representations and their updates by the feed-forward layers in the vocabulary space.We demonstrate the utility of LM-Debugger for single-prediction debugging, by inspecting the internal disambiguation process done by GPT2.Moreover, we show how easily LM-Debugger allows to shift model behavior in a direction of the user's choice, by identifying a few vectors in the network and inducing effective interventions to the prediction process.We release LM-Debugger as an open-source tool and a demo over GPT2 models.
Visual Grounding of Inter-lingual Word-Embeddings

Visual grounding of Language aims at enriching textual representations of language with multiple sources of visual knowledge such as images and videos.Although visual grounding is an area of intense research, inter-lingual aspects of visual grounding have not received much attention.The present study investigates the inter-lingual visual grounding of word embeddings.We propose an implicit alignment technique between the two spaces of vision and language in which inter-lingual textual information interact in order to enrich pre-trained textual word embeddings.We focus on three languages in our experiments, namely, English, Arabic, and German.We obtained visually grounded vector representations for these languages and studied whether visual grounding on one or multiple languages improved the performance of embeddings on word similarity and categorization benchmarks.Our experiments suggest that inter-lingual knowledge improves the performance of grounded embeddings in similar languages such as German and English.However, inter-lingual grounding of German or English with Arabic led to a slight degradation in performance on word similarity benchmarks.On the other hand, we observed an opposite trend on categorization benchmarks where Arabic had the most improvement on English.In the discussion section, several reasons for those findings are laid out.We hope that our experiments provide a baseline for further research on inter-lingual visual grounding.
On Measuring the Intrinsic Few-Shot Hardness of Datasets

While advances in pre-training have led to dramatic improvements in few-shot learning of NLP tasks, there is limited understanding of what drives successful few-shot adaptation in datasets.In particular, given a new dataset and a pre-trained model, what properties of the dataset make it few-shot learnable and are these properties independent of the specific adaptation techniques used?We consider an extensive set of recent few-shot learning methods, and show that their performance across a large number of datasets is highly correlated, showing that few-shot hardness may be intrinsic to datasets, for a given pre-trained model.To estimate intrinsic few-shot hardness, we then propose a simple and lightweight metric called Spread that captures the intuition that fewshot learning is made possible by exploiting feature-space invariances between training and test samples.Our metric better accounts for few-shot hardness compared to existing notions of hardness, and is ~8-100x faster to compute.
A Stacking-based Efficient Method for Toxic Language Detection on Live Streaming Chat

In a live streaming chat on a video streaming service, it is crucial to filter out toxic comments with online processing to prevent users from reading comments in real-time.However, recent toxic language detection methods rely on deep learning methods, which can not be scalable considering inference speed.Also, these methods do not consider constraints of computational resources expected depending on a deployed system (e.g., no GPU resource).This paper presents an efficient method for toxic language detection that is aware of real-world scenarios.Our proposed architecture is based on partial stacking that feeds initial results with low confidence to meta-classifier.Experimental results show that our method achieves a much faster inference speed than BERT-based models with comparable performance.
Mixed-effects transformers for hierarchical adaptation

Language differs dramatically from context to context.To some degree, large language models like GPT-3 account for such variation by conditioning on strings of initial input text, or prompts.However, prompting can be ineffective when contexts are sparse, out-of-sample, or extra-textual.In this paper, we introduce the mixed-effects transformer (MET), a novel approach for learning hierarchically-structured prefixes-lightweight modules prepended to an input sequence-to account for structured variation in language use.Specifically, we show how the popular class of mixedeffects regression models may be extended to transformer-based architectures using a regularized prefix-tuning procedure with dropout.We evaluate this approach on several domainadaptation benchmarks, finding that it learns contextual variation from minimal data while generalizing well to unseen contexts.
Turning Fixed to Adaptive: Integrating Post-Evaluation into Simultaneous Machine Translation

Simultaneous machine translation (SiMT) starts its translation before reading the whole source sentence and employs either fixed or adaptive policy to generate the target sentence.Compared to the fixed policy, the adaptive policy achieves better latency-quality tradeoffs by adopting a flexible translation policy.If the policy can evaluate rationality before taking action, the probability of incorrect actions will also decrease.However, previous methods lack evaluation of actions before taking them.In this paper, we propose a method of performing the adaptive policy via integrating post-evaluation into the fixed policy.Specifically, whenever a candidate token is generated, our model will evaluate the rationality of the next action by measuring the change in the source content.Our model will then take different actions based on the evaluation results.Experiments on three translation tasks show that our method can exceed strong baselines under all latency 1 .
Contrastive Learning enhanced Author-Style Headline Generation

Headline generation is a task of generating an appropriate headline for a given article, which can be further used for machine-aided writing or enhancing the click-through ratio.Current works only use the article itself in the generation, but have not taken the writing style of headlines into consideration.In this paper, we propose a novel Seq2Seq model called CLH3G (Contrastive Learning enhanced Historical Headlines based Headline Generation) which can use the historical headlines of the articles that the author wrote in the past to improve the headline generation of current articles.By taking historical headlines into account, we can integrate the stylistic features of the author into our model, and generate a headline not only appropriate for the article, but also consistent with the author's style.In order to efficiently learn the stylistic features of the author, we further introduce a contrastive learning based auxiliary task for the encoder of our model.Besides, we propose two methods to use the learned stylisic features to guide both the pointer and the decoder during the generation.Experimental results show that historical headlines of the same user can improve the headline generation significantly, and both the contrastive learning module and the two style features fusion methods can further boost the performance.
CEM: Machine-Human Chatting Handoff via Causal-Enhance Module

Aiming to ensure chatbot quality by predicting chatbot failure and enabling human-agent collaboration, Machine-Human Chatting Handoff (MHCH) has attracted lots of attention from both industry and academia in recent years.However, most existing methods mainly focus on the dialogue context or assist with global satisfaction prediction based on multi-task learning, which ignore the grounded relationships among the causal variables, like the user state and labor cost.These variables are significantly associated with handoff decisions, resulting in prediction bias and cost increase.Therefore, we propose Causal-Enhance Module (CEM) by establishing the causal graph of MHCH based on these two variables, which is a simple yet effective module and can be easy to plug into the existing MHCH methods.For the impact of users, we use the user state to correct the prediction bias according to the causal relationship of multi-task.For the labor cost, we train an auxiliary cost simulator to calculate unbiased labor cost through counterfactual learning so that a model becomes cost-aware.Extensive experiments conducted on four real-world benchmarks demonstrate the effectiveness of CEM in generally improving the performance of existing MHCH methods without any elaborated model crafting.
Sarcasm Detection is Way Too Easy! An Empirical Comparison of Human and Machine Sarcasm Detection

Sarcasm Detection is Way Too Easy! Machine Co Incorrect 5. Well it's a good question 6. Wow, Chelsea bean beaten by West Ham, that was a day I had been looking forward to for a while.7. Well this is
An Alignment-based Approach to Text Segmentation Similarity Scoring

Text segmentation is a natural language processing task with popular applications, such as topic segmentation, element discourse extraction, and sentence tokenization.Much work has been done to develop accurate segmentation similarity metrics, but even the most advanced metrics used today, B, and WindowDiff, exhibit incorrect behavior due to their evaluation of boundaries in isolation.In this paper, we present a new segment-alignment based approach to segmentation similarity scoring and a new similarity metric A. We show that A does not exhibit the erratic behavior of B and WindowDiff, quantify the likelihood of B and WindowDiff misbehaving through simulation, and discuss the versatility of alignment-based approaches for segmentation similarity scoring.We make our implementation of A publicly available and encourage the community to explore more sophisticated approaches to text segmentation similarity scoring.
AGReE: A system for generating Automated Grammar Reading Exercises

We describe the AGREE system, which takes user-submitted passages as input and automatically generates grammar practice exercises that can be completed while reading.Multiplechoice practice items are generated for a variety of different grammar constructs: punctuation, articles, conjunctions, pronouns, prepositions, verbs, and nouns.We also conducted a large-scale human evaluation with around 4,500 multiple-choice practice items.We notice for 95% of items, a majority of raters out of five were able to identify the correct answer and for 85% of cases, raters agree that there is only one correct answer among the choices.Finally, the error analysis shows that raters made the most mistakes for punctuation and conjunctions.
Ahmed and Khalil at NADI 2022: Transfer Learning and Addressing Class Imbalance for Arabic Dialect Identification and Sentiment Analysis

In this paper, we present our findings in the two subtasks of the 2022 NADI shared task.First, in the Arabic dialect identification subtask, we find that there is heavy class imbalance, and propose to address this issue using focal loss.Our experiments with the focusing hyperparameter confirm that focal loss improves performance.Second, in the Arabic tweet sentiment analysis subtask, we deal with a smaller dataset, where text includes both Arabic dialects and Modern Standard Arabic.We propose to use transfer learning from both pre-trained MSA language models and our own model from the first subtask.Our system ranks in the 5th and 7th best spots of the leaderboards of first and second subtasks respectively.
Opinion Summarization by Weak-Supervision from Mix-structured Data

Opinion summarization of multiple reviews suffers from the lack of reference summaries for training.Most previous approaches construct multiple reviews and their summary based on textual similarities between reviews, resulting in information mismatch between the review input and the summary.In this paper, we convert each review into a mix of structured and unstructured data, which we call opinion-aspect pairs (OAs) and implicit sentences (ISs).We propose a new method to synthesize training pairs of such mix-structured data as input and the textual summary as output, and design a summarization model with OA encoder and IS encoder.Experiments show that our approach outperforms previous methods on Yelp, Amazon and RottenTomatos datasets.
End-to-End Neural Discourse Deixis Resolution in Dialogue

We adapt Lee et al.'s (2018) span-based entity coreference model to the task of end-to-end discourse deixis resolution in dialogue, specifically by proposing extensions to their model that exploit task-specific characteristics.The resulting model, dd-utt, achieves state-ofthe-art results on the four datasets in the CODI-CRAC 2021 shared task.
Cross-Domain Sentiment Classification using Semantic Representation

Previous studies on cross-domain sentiment classification depend on the pivot features or utilize the target data for representation learning, which ignore the semantic relevance among different domains.To this end, we exploit Abstract Meaning Representation (AMR) to help with cross-domain sentiment classification.Compared with the textual input, AMR reduces data sparsity and explicitly provides core semantic knowledge and correlations among different domains.In particular, we develop an algorithm to construct a sentimentdriven semantic graph from sentence-level AMRs.We further design two strategies to linearize the semantic graph and propose a text-graph interaction model to fully explore the correlations between the text and semantic graph representations for cross-domain sentiment classification.Empirical studies show the effectiveness of our proposed model over several strong baselines.The results also indicate the importance of the proposed sentimentdriven semantic graph for cross-domain sentiment classification.
Does Simultaneous Speech Translation need Simultaneous Models?

In simultaneous speech translation (SimulST), finding the best trade-off between high output quality and low latency is a challenging task.To meet the latency constraints posed by different application scenarios, multiple dedicated SimulST models are usually trained and maintained, generating high computational costs.In this paper, also motivated by the increased sensitivity towards sustainable AI, we investigate whether a single model trained offline can serve both offline and simultaneous applications under different latency regimes without additional training or adaptation.Experiments on en→{de, es} show that, aside from facilitating the adoption of well-established offline architectures and training strategies without affecting latency, the offline solution achieves similar or better quality compared to the standard SimulST training protocol, also being competitive with the state-of-the-art system.
On Parsing as Tagging

There are many proposals to reduce constituency parsing to tagging. To figure out what these approaches have in common, we offer a unifying pipeline, which consists of three steps: linearization, learning, and decoding. We prove that classic shift–reduce parsing can be reduced to tetratagging—the state-of-the-art constituency tagger—under two assumptions: right-corner transformation in the linearization step and factored scoring in the learning step. We ask what is the most critical factor that makes parsing-as-tagging methods accurate while being efficient. To answer this question, we empirically evaluate a taxonomy of tagging pipelines with different choices of linearizers, learners, and decoders. Based on the results in English as well as a set of 8 typologically diverse languages, we conclude that the linearization of the derivation tree and its alignment with the input sequence is the most critical factor in achieving accurate parsers as taggers.
Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models

Deep Neural Networks (DNNs) are known to be vulnerable to backdoor attacks.In Natural Language Processing (NLP), DNNs are often backdoored during the fine-tuning process of a large-scale Pre-trained Language Model (PLM) with poisoned samples.Although the clean weights of PLMs are readily available, existing methods have ignored this information in defending NLP models against backdoor attacks.In this work, we take the first step to exploit the pre-trained (unfine-tuned) weights to mitigate backdoors in fine-tuned language models.Specifically, we leverage the clean pre-trained weights via two complementary techniques: (1) a two-step Fine-mixing technique, which first mixes the backdoored weights (fine-tuned on poisoned data) with the pre-trained weights, then fine-tunes the mixed weights on a small subset of clean data; (2) an Embedding Purification (E-PUR) technique, which mitigates potential backdoors existing in the word embeddings.We compare Fine-mixing with typical backdoor mitigation methods on three singlesentence sentiment classification tasks and two sentence-pair classification tasks and show that it outperforms the baselines by a considerable margin in all scenarios.We also show that our E-PUR method can benefit existing mitigation methods.Our work establishes a simple but strong baseline defense for secure fine-tuned NLP models against backdoor attacks.
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives

In this paper, we investigate the instability in the standard dense retrieval training, which iterates between model training and hard negative selection using the being-trained model.We show the catastrophic forgetting phenomena behind the training instability, where models learn and forget different negative groups during training iterations.We then propose ANCE-Tele, which accumulates momentum negatives from past iterations and approximates future iterations using lookahead negatives, as "teleportations" along the time axis to smooth the learning process.On web search and OpenQA, ANCE-Tele outperforms previous state-of-theart systems of similar size, eliminates the dependency on sparse retrieval negatives, and is competitive among systems using significantly more (50x) parameters.Our analysis demonstrates that teleportation negatives reduce catastrophic forgetting and improve convergence speed for dense retrieval training.
SLICER: Sliced Fine-Tuning for Low-Resource Cross-Lingual Transfer for Named Entity Recognition

Large multilingual language models generally demonstrate impressive results in zero-shot cross-lingual transfer, yet often fail to successfully transfer to low-resource languages, even for token-level prediction tasks like named entity recognition (NER).In this work, we introduce a simple yet highly effective approach for improving zero-shot transfer for NER to lowresource languages.We observe that NER finetuning in the source language decontextualizes token representations, i.e., tokens increasingly attend to themselves.This increased reliance on token information itself, we hypothesize, triggers a type of overfitting to properties that NE tokens within the source languages share, but are generally not present in NE mentions of target languages.As a remedy, we propose a simple yet very effective sliced fine-tuning for NER (SLICER) that forces stronger token contextualization in the Transformer: we divide the transformed token representations and classifier into disjoint slices that are then independently classified during training.We evaluate SLICER on two standard benchmarks for NER that involve low-resource languages, WikiANN and MasakhaNER, and show that it (i) indeed reduces decontextualization (i.e., extent to which NE tokens attend to themselves), consequently (ii) yielding consistent transfer gains, especially prominent for low-resource target languages distant from the source language.
Learning a Grammar Inducer from Massive Uncurated Instructional Videos

Video-aided grammar induction aims to leverage video information for finding more accurate syntactic grammars for accompanying text.While previous work focuses on building systems for inducing grammars on text that are well-aligned with video content, we investigate the scenario, in which text and video are only in loose correspondence.Such data can be found in abundance online, and the weak correspondence is similar to the indeterminacy problem studied in language acquisition.Furthermore, we build a new model that can better learn video-span correlation without manually designed features adopted by previous work.Experiments show that our model trained only on large-scale YouTube data with no textvideo alignment reports strong and robust performances across three unseen datasets, despite domain shift and noisy label issues.Furthermore our model yields higher F1 scores than the previous state-of-the-art systems trained on in-domain data.
On the Curious Case of l2 norm of Sense Embeddings

We show that the ℓ 2 norm of a static sense embedding encodes information related to the frequency of that sense in the training corpus used to learn the sense embeddings.This finding can be seen as an extension of a previously known relationship for word embeddings to sense embeddings.Our experimental results show that, in spite of its simplicity, the ℓ 2 norm of sense embeddings is a surprisingly effective feature for several word sense related tasks such as (a) most frequent sense prediction, (b) Word-in-Context (WiC), and (c) Word Sense Disambiguation (WSD).In particular, by simply including the ℓ 2 norm of a sense embedding as a feature in a classifier, we show that we can improve WiC and WSD methods that use static sense embeddings.