Reviewerly/exHarmony huggingface dataset:
Hybrid Uncertainty Quantification for Selective Text Classification in Ambiguous Tasks

Many text classification tasks are inherently ambiguous, which results in automatic systems having a high risk of making mistakes, in spite of using advanced machine learning models.For example, toxicity detection in usergenerated content is a subjective task, and notions of toxicity can be annotated according to a variety of definitions that can be in conflict with one another.Instead of relying solely on automatic solutions, moderation of the most difficult and ambiguous cases can be delegated to human workers.Potential mistakes in automated classification can be identified by using uncertainty estimation (UE) techniques.Although UE is a rapidly growing field within natural language processing, we find that stateof-the-art UE methods estimate only epistemic uncertainty and show poor performance, or under-perform trivial methods for ambiguous tasks such as toxicity detection.We argue that in order to create robust uncertainty estimation methods for ambiguous tasks it is necessary to account also for aleatoric uncertainty.In this paper, we propose a new uncertainty estimation method that combines epistemic and aleatoric UE methods.We show that by using our hybrid method, we can outperform state-of-the-art UE methods for toxicity detection and other ambiguous text classification tasks 1 .
Local Interpretation of Transformer Based on Linear Decomposition

In recent years, deep neural networks (DNNs) have achieved state-of-the-art performance on a wide range of tasks.However, limitations in interpretability have hindered their applications in the real world.This work proposes to interpret neural networks by linear decomposition and finds that the ReLU-activated Transformer can be considered as a linear model on a single input.We further leverage the linearity of the model and propose a linear decomposition of the model output to generate local explanations.Our evaluation of sentiment classification and machine translation shows that our method achieves competitive performance in efficiency and fidelity of explanation.In addition, we demonstrate the potential of our approach in applications with examples of error analysis on multiple tasks.
Pipeline for modeling causal beliefs from natural language

Causal reasoning is a core cognitive function and is central to how people learn and update their beliefs.Causal information is also central to how people represent and use language.Natural Language Processing algorithms that detect people's causal representations can illuminate the considerations shaping their beliefs and reasoning.We present a causal language analysis pipeline that leverages a Large Language Model to identify causal claims in natural language documents, and aggregates claims across a corpus to produce a causal claim network.The pipeline then applies a clustering algorithm that groups causal claims according to their semantic topics.We demonstrate the pipeline by modeling causal belief systems surrounding the Covid-19 vaccine from tweets.
DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering

Question answering models commonly have access to two sources of "knowledge" during inference time: (1) parametric knowledgethe factual knowledge encoded in the model weights, and (2) contextual knowledge -external knowledge (e.g., a Wikipedia passage) given to the model to generate a grounded answer.Having these two sources of knowledge entangled together is a core issue for generative QA models as it is unclear whether the answer stems from the given non-parametric knowledge or not.This unclarity has implications on issues of trust, interpretability and factuality.In this work, we propose a new paradigm in which QA models are trained to disentangle the two sources of knowledge.Using counterfactual data augmentation, we introduce a model that predicts two answers for a given question: one based on given contextual knowledge and one based on parametric knowledge.Our experiments on the Natural Questions dataset show that this approach improves the performance of QA models by making them more robust to knowledge conflicts between the two knowledge sources, while generating useful disentangled answers.
Empowering Conversational Agents using Semantic In-Context Learning

Language models are one of the biggest game changers in downstream NLP applications, especially in conversational agents.In spite of their awesome capabilities to generated responses to solve the inquiries, there are still some big challenges to using them.One challenge is how to enable the LLMs to use the private internal data to solve inquires.And secondly, how to keep the LLMs updated with newly incoming data without the burden of finetuning as it is not only expensive but also not an available option for some commercial LLMs, such as ChatGPT.In this work, we propose Semantic In-Context Learning (S-ICL) to address the aforementioned challenges.Our proposed approach participated in the BEA 2023 shared task 1 and ended up achieving the fourth place in both the development and evaluation phases.
Detecting Label Errors by Using Pre-Trained Language Models

We show that large pre-trained language models are inherently highly capable of identifying label errors in natural language datasets: simply examining out-of-sample data points in descending order of finetuned task loss significantly outperforms more complex error-detection mechanisms proposed in previous work.To this end, we contribute a novel method for introducing realistic, human-originated label noise into existing crowdsourced datasets such as SNLI and TweetNLP.We show that this noise has similar properties to real, handverified label errors, and is harder to detect than existing synthetic noise, creating challenges for model robustness.We argue that human-originated noise is a better standard for evaluation than synthetic noise.Finally, we use crowdsourced verification to evaluate the detection of real errors on IMDB, Amazon Reviews, and Recon, and confirm that pre-trained models perform at a 9-36% higher absolute Area Under the Precision-Recall Curve than existing models.
Trustworthiness-aware knowledge graph representation for recommendation

Incorporating knowledge graphs (KGs) into recommender systems (RS) has recently attracted increasing attention. For large-scale KGs, due to limited labour supervision, noises are inevitably introduced during automatic construction. However, the effects of such noises as untrustworthy information in KGs on RS are unclear, and how to retain RS performing well while encountering such untrustworthy information has yet to be solved. Motivated by them, we study the effects of the trustworthiness of the KG on RS and propose a novel method trustworthiness-aware knowledge graph representation (KGR) for recommendation (TrustRec). TrustRec introduces a trustworthiness estimator into noise-tolerant KGR methods for collaborative filtering. Specifically, to assign trustworthiness, we leverage internal structures of KGs from microscopic to macroscopic levels: motifs, communities and global information, to reflect the true degree of triple expression. Building on this estimator, we then propose trustworthiness integration to learn noise-tolerant KGR and item representations for RS. We conduct extensive experiments to show the superior performance of TrustRec over state-of-the-art recommendation methods.
An Analysis of Acknowledgments in NLP Conference Proceedings

While acknowledgments are often overlooked and sometimes entirely missing from publications, this short section of a paper can provide insights on the state of a field.We characterize and perform a textual analysis of acknowledgments in NLP conference proceedings across the last 17 years, revealing broader trends in funding and research directions in NLP as well as interesting phenomena including career incentives and the influence of defaults.
Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5

We compare sequential fine-tuning with a model for multi-task learning in the context where we are interested in boosting performance on two tasks, one of which depends on the other.We test these models on the FigLang2022 shared task which requires participants to predict language inference labels on figurative language along with corresponding textual explanations of the inference predictions.Our results show that while sequential multi-task learning can be tuned to be good at the first of two target tasks, it performs less well on the second and additionally struggles with overfitting.Our findings show that simple sequential fine-tuning of text-to-text models is an extraordinarily powerful method for crosstask knowledge transfer while simultaneously predicting multiple interdependent targets.So much so, that our best model achieved the (tied) highest score on the task 1 .
Modeling Context With Linear Attention for Scalable Document-Level Translation

Document-level machine translation leverages inter-sentence dependencies to produce more coherent and consistent translations.However, these models, predominantly based on transformers, are difficult to scale to long documents as their attention layers have quadratic complexity in the sequence length.Recent efforts on efficient attention improve scalability, but their effect on document translation remains unexplored.In this work, we investigate the efficacy of a recent linear attention model by Peng et al. ( 2021) on document translation and augment it with a sentential gate to promote a recency inductive bias.We evaluate the model on IWSLT 2015 and OpenSubtitles 2018 against the transformer, demonstrating substantially increased decoding speed on long sequences with similar or better BLEU scores.We show that sentential gating further improves translation quality on IWSLT. 1
Probing Cross-modal Semantics Alignment Capability from the Textual Perspective

In recent years, vision and language pretraining (VLP) models have advanced the stateof-the-art results in a variety of cross-modal downstream tasks.Aligning cross-modal semantics is claimed to be one of the essential capabilities of VLP models.However, it still remains unclear about the inner working mechanism of alignment in VLP models.In this paper, we propose a new probing method that is based on image captioning to first empirically study the cross-modal semantics alignment of VLP models.Our probing method is built upon the fact that given an image-caption pair, the VLP models will give a score, indicating how well two modalities are aligned; maximizing such scores will generate sentences that VLP models believe are of good alignment.Analyzing these sentences thus will reveal in what way different modalities are aligned and how well these alignments are in VLP models.We apply our probing method to five popular VLP models, including UNITER, ROSITA, ViLBERT, CLIP, and LXMERT, and provide a comprehensive analysis of the generated captions guided by these models.Our results show that VLP models (1) focus more on just aligning objects with visual words, while neglecting global semantics; (2) prefer fixed sentence patterns, thus ignoring more important textual information including fluency and grammar; and (3) deem the captions with more visual words are better aligned with images.These findings indicate that VLP models still have weaknesses in crossmodal semantics alignment and we hope this work will draw researchers' attention to such problems when designing a new VLP model. 1
CODER: An efficient framework for improving retrieval through COntextual Document Embedding Reranking

Contrastive learning has been the dominant approach to training dense retrieval models.In this work, we investigate the impact of ranking context -an often overlooked aspect of learning dense retrieval models.In particular, we examine the effect of its constituent parts: jointly scoring a large number of negatives per query, using retrieved (query-specific) instead of random negatives, and a fully list-wise loss.To incorporate these factors into training, we introduce Contextual Document Embedding Reranking (CODER), a highly efficient retrieval framework.When reranking, it incurs only a negligible computational overhead on top of a firststage method at run time (∼ 5 ms delay per query), allowing it to be easily combined with any state-of-the-art dual encoder method.Models trained through CODER can also be used as stand-alone retrievers.Evaluating CODER in a large set of experiments on the MS MARCO and TripClick collections, we show that the contextual reranking of precomputed document embeddings leads to a significant improvement in retrieval performance.This improvement becomes even more pronounced when more relevance information per query is available, shown in the TripClick collection, where we establish new state-of-the-art results by a large margin.
Does Your Model Classify Entities Reasonably? Diagnosing and Mitigating Spurious Correlations in Entity Typing

Entity typing aims at predicting one or more words that describe the type(s) of a specific mention in a sentence.Due to shortcuts from surface patterns to annotated entity labels and biased training, existing entity typing models are subject to the problem of spurious correlations.To comprehensively investigate the faithfulness and reliability of entity typing methods, we first systematically define distinct kinds of model biases that are reflected mainly from spurious correlations.Particularly, we identify six types of existing model biases, including mention-context bias, lexical overlapping bias, named entity bias, pronoun bias, dependency bias, and overgeneralization bias.To mitigate model biases, we then introduce a counterfactual data augmentation method.By augmenting the original training set with their debiased counterparts, models are forced to fully comprehend sentences and discover the fundamental cues for entity typing, rather than relying on spurious correlations for shortcuts.Experimental results on the UFET dataset show our counterfactual data augmentation approach helps improve generalization of different entity typing models with consistently better performance on both the original and debiased test sets 1 .
Joint Completion and Alignment of Multilingual Knowledge Graphs

Knowledge Graph Completion (KGC) predicts missing facts in an incomplete Knowledge Graph (KG).Multilingual KGs associate entities and relations with surface forms written in different languages.An entity or relation may be associated with distinct IDs in different KGs, necessitating entity alignment (EA) and relation alignment (RA).Many effective algorithms have been proposed for completion and alignment as separate tasks.Here we show that these tasks are synergistic and best solved together.Our multitask approach starts with a state-of-the-art KG embedding scheme, but adds a novel relation representation based on sets of embeddings of (subject, object) entity pairs.This representation leads to a new relation alignment loss term based on a maximal bipartite matching between two sets of embedding vectors.This loss is combined with traditional KGC loss and optionally, losses based on text embeddings of entity (and relation) names.In experiments over KGs in seven languages, we find that our system achieves large improvements in KGC compared to a strong completion model that combines known facts in all languages.It also outperforms strong EA and RA baselines, underscoring the value of joint alignment and completion.
HIT-SCIR at MMNLU-22: Consistency Regularization for Multilingual Spoken Language Understanding

Multilingual spoken language understanding (SLU) consists of two sub-tasks, namely intent detection and slot filling.To improve the performance of these two sub-tasks, we propose to use consistency regularization based on a hybrid data augmentation strategy.The consistency regularization enforces the predicted distributions for an example and its semantically equivalent augmentation to be consistent.We conduct experiments on the MASSIVE dataset under both full-dataset and zero-shot settings.Experimental results demonstrate that our proposed method improves the performance on both intent detection and slot filling tasks.Our system 1 ranked 1st in the MMNLU-22 competition under the full-dataset setting.
International Workshop on Multimodal Learning - 2023 Theme: Multimodal Learning with Foundation Models

The recent advancements in machine learning and artificial intelligence (particularly foundation models such as BERT, GPT-3, T5, ResNet, etc.) have demonstrated remarkable capabilities and driven significant revolutionary changes to the way we make inferences from complex data. These models represent a fundamental shift in the way data are approached and offer exciting new research directions and opportunities for multimodal learning and data fusion. Given the potential of foundation models to transform the field of multimodal learning, there is a need to bring together experts and researchers to discuss the latest developments in this area, exchange ideas, and identify key research questions and challenges that need to be addressed. By hosting this workshop, we aim to create a forum for researchers to share their insights and expertise on multimodal data fusion and learning using foundation models, and to explore potential new research directions and applications in the rapidly evolving field. We expect contributions from interdisciplinary researchers to study and model interactions between (but not limited to) modalities of language, graphs, time-series, vision, tabular data, sensors, and more. Our workshop will emphasize interdisciplinary work and aim at seeding cross-team collaborations around new tasks, datasets, and models.
Learning Non-linguistic Skills without Sacrificing Linguistic Proficiency

The field of Math-NLP has witnessed significant growth in recent years, motivated by the desire to expand LLM performance to the learning of non-linguistic notions (numerals, and subsequently, arithmetic reasoning).However, non-linguistic skill injection typically comes at a cost for LLMs: it leads to catastrophic forgetting of core linguistic skills, a consequence that often remains unaddressed in the literature.As Math-NLP has been able to create LLMs that can closely approximate the mathematical skills of a grade-schooler or the arithmetic reasoning skills of a calculator, the practicality of these models fail if they concomitantly shed their linguistic capabilities.In this work, we take a closer look into the phenomena of catastrophic forgetting as it pertains to LLMs and subsequently offer a novel framework for non-linguistic skill injection for LLMs based on information-theoretic interventions and skill-specific losses that enable the learning of strict arithmetic reasoning.Our model outperforms the state-of-the-art both on injected non-linguistic skills and on linguistic knowledge retention, and does so with a fraction of the non-linguistic training data (1/4) and zero additional synthetic linguistic training data.Our pre-trained models and experimentation codebases are hosted online 1 .
Explaining How Transformers Use Context to Build Predictions

Language Generation Models produce words based on the previous context.Although existing methods offer input attributions as explanations for a model's prediction, it is still unclear how prior words affect the model's decision throughout the layers.In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation.Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines.Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable.Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions.
Reward Gaming in Conditional Text Generation

To align conditional text generation model outputs with desired behaviors, there has been an increasing focus on training the model using reinforcement learning (RL) with reward functions learned from human annotations.Under this framework, we identify three common cases where high rewards are incorrectly assigned to undesirable patterns: noise-induced spurious correlation, naturally occurring spurious correlation, and covariate shift.We show that even though learned metrics achieve high performance on the distribution of the data used to train the reward function, the undesirable patterns may be amplified during RL training of the text generation model.While there has been discussion about reward gaming in the RL or safety community, in this discussion piece, we would like to highlight reward gaming in the natural language generation (NLG) community using concrete conditional text generation examples and discuss potential fixes and areas for future work.
Relationality and Offensive Speech: A Research Agenda

We draw from the framework of relationality as a pathway for modeling social relations to address gaps in text classification, generally, and offensive language classification, specifically.We use minoritized language, such as queer speech, to motivate a need for understanding and modeling social relationsboth among individuals and among their social communities.We then point to socio-ethical style as a research area for inferring and measuring social relations as well as propose additional questions to structure future research on operationalizing social context.
PRACTICAL COMPONENTS OF STUDENTS' PROFESSIONAL TRAINING FOR ENHANCING FUTURE EMPLOYABILITY

Formulation of the problem.The full-scale Russian aggression against Ukraine changed the scale of Ukrainians' planning and affected all the spheres of life.Issues related to the need to restore the economy and ensure the country's survival in the long term come to the fore.The ambitious goal of the reconstruction of Ukraine is the comprehensive modernization of the country, instead of only the restoration of the pre-war state.The main asset of the modern economy is human capital, so the need to continue reforming the education system is of paramount importance.In addition, in June 2022 Ukraine received the status of a candidate state for EU membership.From now on, certain reforms should be carried out in Ukraine, the implementation of which will enable the country to become a full-fledged member
What’s New? Identifying the Unfolding of New Events in a Narrative

Narratives include a rich source of events unfolding over time and context.Automatic understanding of these events provides a summarised comprehension of the narrative for further computation (such as reasoning).In this paper, we study the Information Status (IS) of the events and propose a novel challenging task: the automatic identification of new events in a narrative.We define an event as a triplet of subject, predicate, and object.The event is categorized as new with respect to the discourse context and whether it can be inferred through commonsense reasoning.We annotated a publicly available corpus of narratives with the new events at sentence level using human annotators.We present the annotation protocol and study the quality of the annotation and the difficulty of the task.We publish the annotated dataset, annotation materials, and machine learning baseline models for the task of new event extraction for narrative understanding.
BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric

Mingda Chen, Paul-Ambroise Duquenne, Pierre Andrews, Justine Kao, Alexandre Mourachko, Holger Schwenk, Marta R. Costa-jussà. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.