Reviewerly/exHarmony huggingface dataset:
Counterfactual Debiasing for Fact Verification

Fact verification aims to automatically judge the veracity of a claim according to several pieces of evidence.Due to the manual construction of datasets, spurious correlations between claim patterns and its veracity (i.e., biases) inevitably exist.Recent studies show that models usually learn such biases instead of understanding the semantic relationship between the claim and evidence.Existing debiasing works can be roughly divided into dataaugmentation-based and weight-regularizationbased pipeline, where the former is inflexible and the latter relies on the uncertain output on the training stage.Unlike previous works, we propose a novel method from a counterfactual view, namely CLEVER, which is augmentationfree and mitigates biases on the inference stage.Specifically, we train a claim-evidence fusion model and a claim-only model independently.Then, we obtain the final prediction via subtracting output of the claim-only model from output of the claim-evidence fusion model, which counteracts biases in two outputs so that the unbiased part is highlighted.Comprehensive experiments on several datasets have demonstrated the effectiveness of CLEVER.
Task-aware Retrieval with Instructions

We study the problem of retrieval with instructions, where users provide explicit descriptions of their intent along with their queries to guide a retrieval system.Our solution is a generalpurpose task-aware retrieval system, trained using multi-task instruction tuning and can follow human-written instructions to find relevant documents to a given query.We introduce the first large-scale collection of 37 retrieval datasets with instructions, BERRI, and present TART, a single multi-task retrieval system trained on BERRI with instructions that can adapt to a new task without any parameter updates.TART advances the state of the art on two zero-shot retrieval benchmarks, BEIR and LOTTE, outperforming models up to three times larger.We further introduce a new evaluation setup, X 2 -Retrieval, to better reflect real-world scenarios in which diverse domains and tasks are pooled.TART significantly outperforms competitive baselines in this setup, further highlighting the effectiveness of guiding retrieval with instructions.
Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering

Despite the impressive few-shot performance of in-context learning (ICL), it remains a common practice to randomly select examples to serve as the context.In this paper, we advocate self-adaptive in-context learning, a new principle for ICL, in which the self-adaption mechanism is introduced to help each input find an in-context example organization (i.e., selection and permutation) that can derive the correct output, thus maximizing performance.To validate the effectiveness of self-adaptive ICL, we propose a general select-then-rank framework and a set of novel selection and ranking algorithms.Upon extensive evaluation on eight different NLP datasets, our self-adaptive ICL method achieves a 40% relative improvement over the common practice setting.Further analysis reveals the great potential of selfadaptive ICL as a promising method to close the gap between ICL and finetuning.Our code will be released to facilitate future research.
MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models &amp; Tasks

Vision and language models (VL) are known to exploit unrobust indicators in individual modalities (e.g., introduced by distributional biases) instead of focusing on relevant information in each modality.That a unimodal model achieves similar accuracy on a VL task to a multimodal one, indicates that so-called unimodal collapse occurred.However, accuracybased tests fail to detect e.g., when the model prediction is wrong, while the model used relevant information from a modality.Instead, we propose MM-SHAP, a performance-agnostic multimodality score based on Shapley values that reliably quantifies in which proportions a multimodal model uses individual modalities.We apply MM-SHAP in two ways: (1) to compare models for their average degree of multimodality, and (2) to measure for individual models the contribution of individual modalities for different tasks and datasets.Experiments with six VL models -LXMERT, CLIP and four ALBEF variants -on four VL tasks highlight that unimodal collapse can occur to different degrees and in different directions, contradicting the wide-spread assumption that unimodal collapse is one-sided.Based on our results, we recommend MM-SHAP for analysing multimodal tasks, to diagnose and guide progress towards multimodal integration.
Sequence Parallelism: Long Sequence Training from System Perspective

Transformer achieves promising results on various tasks.However, self-attention suffers from quadratic memory requirements with respect to the sequence length.Existing work focuses on reducing time and space complexity from an algorithm perspective.In this work, we propose sequence parallelism, a memory-efficient parallelism to solve this issue from system perspective instead.Our approach is compatible with most existing parallelisms (e.g., data, pipeline, and tensor parallelism), which means our sequence parallelism makes 4D parallelism possible.More importantly, we no longer require a single device to hold the whole sequence.Besides, using efficient attention with linear complexity, our sequence parallelism enables us to train transformer with infinite long sequence.Specifically, we split the input sequence into multiple chunks and feed each chunk into its corresponding device (i.e., GPU).To compute the attention output, we integrated ring-style communication with self-attention calculation and proposed Ring Self-Attention (RSA).Experiments show that sequence parallelism performs well when scaling with batch size and sequence length.Compared with tensor parallelism, our approach achieved 13.7× and 3.0× maximum batch size and sequence length respectively when scaling up to 64 NVIDIA P100 GPUs.With efficient attention, sequence can handle sequence with over 114K tokens, which is over 27× longer than existing efficient attention works holding the whole sequence on a single device.
Unified Language Representation for Question Answering over Text, Tables, and Images

When trying to answer complex questions, people often rely on multiple sources of information, such as visual, textual, and tabular data.Previous approaches to this problem have focused on designing input features or model structure in the multi-modal space, which is inflexible for cross-modal reasoning or dataefficient training.In this paper, we call for an alternative paradigm, which transforms the images and tables into unified language representations, so that we can simplify the task into a simpler textual QA problem that can be solved using three steps: retrieval, ranking, and generation, all within a language space.This idea takes advantage of the power of pre-trained language models and is implemented in a framework called Solar.Our experimental results show that Solar outperforms all existing methods by 10.6-32.3pts on two datasets, Multi-modalQA and MMCoQA, across ten different metrics.Additionally, Solar achieves the best performance on the WebQA leaderboard 1 .
Improving Grammatical Error Correction with Multimodal Feature Integration

Grammatical error correction (GEC) is a promising task aimed at correcting errors in a text.Many methods have been proposed to facilitate this task with remarkable results.However, most of them only focus on enhancing textual feature extraction without exploring the usage of other modalities' information (e.g., speech), which can also provide valuable knowledge to help the model detect grammatical errors.To shore up this deficiency, we propose a novel framework that integrates both speech and text features to enhance GEC.In detail, we create new multimodal GEC datasets for English and German by generating audio 1 from text using the advanced text-to-speech models.Subsequently, we extract acoustic and textual representations by a multimodal encoder that consists of a speech and a text encoder.A mixture-of-experts (MoE) layer is employed to selectively align representations from the two modalities, and then a dot attention mechanism is used to fuse them as final multimodal representations.Experimental results on CoNLL14, BEA19 English, and Falko-MERLIN German show that our multimodal GEC models achieve significant improvements over strong baselines and achieve a new stateof-the-art result on the Falko-MERLIN test set.
IRIT_IRIS_A at SemEval-2023 Task 6: Legal Rhetorical Role Labeling Supported by Dynamic-Filled Contextualized Sentence Chunks

This work presents and evaluates an approach to efficiently leverage the context exploitation ability of pre-trained Transformer models as a way of boosting the performance of models tackling the Legal Rhetorical Role Labeling task.The core idea is to feed the model with sentence chunks that are assembled in a way that avoids the insertion of padding tokens and the truncation of sentences and, hence, obtain better sentence embeddings.The achieved results show that our proposal is efficient, despite its simplicity, since models based on it overcome strong baselines by 3.76% in the worst case and by 8.71% in the best case.
ML Mob at SemEval-2023 Task 5: “Breaking News: Our Semi-Supervised and Multi-Task Learning Approach Spoils Clickbait”

Online articles using striking headlines that promise intriguing information are often used to attract readers.Most of the time, the information provided in the text is disappointing to the reader after the headline promised exciting news.As part of the SemEval-2023 challenge, we propose a system to generate a spoiler for these headlines.The spoiler provides the information promised by the headline and eliminates the need to read the full article.We consider Multi-Task Learning and generating more data using a distillation approach in our system.With this, we achieve an F1 score up to 51.48% on extracting the spoiler from the articles.
Life is a Circus and We are the Clowns: Automatically Finding Analogies between Situations and Processes

Analogy-making gives rise to reasoning, abstraction, flexible categorization and counterfactual inference -abilities lacking in even the best AI systems today.Much research has suggested that analogies are key to non-brittle systems that can adapt to new domains.Despite their importance, analogies received little attention in the NLP community, with most research focusing on simple word analogies.Work that tackled more complex analogies relied heavily on manually constructed, hard-to-scale input representations.In this work, we explore a more realistic, challenging setup: our input is a pair of natural language procedural texts, describing a situation or a process (e.g., how the heart works/how a pump works).Our goal is to automatically extract entities and their relations from the text and find a mapping between the different domains based on relational similarity (e.g., blood is mapped to water).We develop an interpretable, scalable algorithm and demonstrate that it identifies the correct mappings 87% of the time for procedural texts and 94% for stories from cognitive-psychology literature.We show it can extract analogies from a large dataset of procedural texts, achieving 79% precision (analogy prevalence in data: 3%).Lastly, we demonstrate that our algorithm is robust to paraphrasing the input texts 1 .
IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages

Natural Language Generation (NLG) for non-English languages is hampered by the scarcity of datasets in these languages.In this paper, we present the IndicNLG Benchmark, a collection of datasets for benchmarking NLG for 11 Indic languages.We focus on five diverse tasks, namely, biography generation using Wikipedia infoboxes, news headline generation, sentence summarization, paraphrase generation and, question generation.We describe the created datasets and use them to benchmark the performance of several monolingual and multilingual baselines that leverage pre-trained sequence-to-sequence models.Our results exhibit the strong performance of multilingual language-specific pre-trained models, and the utility of models trained on our dataset for other related NLG tasks.Our dataset creation methods can be easily applied to modest-resource languages as they involve simple steps such as scraping news articles and Wikipedia infoboxes, light cleaning, and pivoting through machine translation data.To the best of our knowledge, the IndicNLG Benchmark is the first NLG benchmark for Indic languages and the most diverse multilingual NLG dataset, with approximately 8M examples across 5 tasks and 11 languages.The datasets and models are publicly available 1 .
BotsTalk: Machine-sourced Framework for Automatic Curation of Large-scale Multi-skill Dialogue Datasets

To build open-domain chatbots that are able to use diverse communicative skills, we propose a novel framework BOTSTALK, where multiple agents grounded to the specific target skills participate in a conversation to automatically annotate multi-skill dialogues.We further present Blended Skill BotsTalk (BSBT), a large-scale multi-skill dialogue dataset comprising 300K conversations.Through extensive experiments, we demonstrate that our dataset can be effective for multi-skill dialogue systems which require an understanding of skill blending as well as skill grounding.Our code and
Using Contextual Sentence Analysis Models to Recognize ESG Concepts

This paper summarizes the joint participation of the Trading Central Labs and the L3i laboratory of the University of La Rochelle on both sub-tasks of the Shared Task FinSim-4 evaluation campaign.The first sub-task aims to enrich the 'Fortia ESG taxonomy' with new lexicon entries while the second one aims to classify sentences to either 'sustainable' or 'unsustainable' with respect to ESG (Environment, Social and Governance) related factors.For the first sub-task, we proposed a model based on pre-trained Sentence-BERT models to project sentences and concepts in a common space in order to better represent ESG concepts.The official task results show that our system yields a significant performance improvement compared to the baseline and outperforms all other submissions on the first sub-task.For the second sub-task, we combine the RoBERTa model with a feed-forward multi-layer perceptron in order to extract the context of sentences and classify them.Our model achieved high accuracy scores (over 92%) and was ranked among the top 5 systems.
Soft-Labeled Contrastive Pre-Training for Function-Level Code Representation

Code contrastive pre-training has recently achieved significant progress on code-related tasks.In this paper, we present SCodeR, a Soft-labeled contrastive pre-training framework with two positive sample construction methods to learn functional-level Code Representation.Considering the relevance between codes in a large-scale code corpus, the soft-labeled contrastive pre-training can obtain fine-grained soft-labels through an iterative adversarial manner and use them to learn better code representation.The positive sample construction is another key for contrastive pre-training.Previous works use transformation-based methods like variable renaming to generate semantically equal positive codes.However, they usually result in the generated code with a highly similar surface form, and thus mislead the model to focus on superficial code structure instead of code semantics.To encourage SCodeR to capture semantic information from the code, we utilize code comments and abstract syntax subtrees of the code to build positive samples.We conduct experiments on four code-related tasks over seven datasets.Extensive experimental results show that SCodeR achieves new stateof-the-art performance on all of them, which illustrates the effectiveness of the proposed pretraining method.
Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems

Do all instances need inference through the big models for a correct prediction?Perhaps not; some instances are easy and can be answered correctly by even small capacity models.This provides opportunities for improving the computational efficiency of systems.In this work, we present an explorative study on 'model cascading', a simple technique that utilizes a collection of models of varying capacities to accurately yet efficiently output predictions.Through comprehensive experiments in multiple task settings that differ in the number of models available for cascading (K value), we show that cascading improves both the computational efficiency and the prediction accuracy.For instance, in K=3 setting, cascading saves up to 88.93% computation cost and consistently achieves superior prediction accuracy with an improvement of up to 2.18%.We also study the impact of introducing additional models in the cascade and show that it further increases the efficiency improvements.Finally, we hope that our work will facilitate development of efficient NLP systems making their widespread adoption in real-world applications possible.
XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing

In-context learning using large language models has recently shown surprising results for semantic parsing tasks such as Text-to-SQL translation.Prompting GPT-3 or Codex using several examples of question-SQL pairs can produce excellent results, comparable to state-of-the-art finetuning-based models.However, existing work primarily focuses on English datasets, and it is unknown whether large language models can serve as competitive semantic parsers for other languages.To bridge this gap, our work focuses on cross-lingual Text-to-SQL semantic parsing for translating non-English utterances into SQL queries based on an English schema.We consider a zero-shot transfer learning setting with the assumption that we do not have any labeled examples in the target language (but have annotated examples in English).This work introduces the XRICL framework, which learns to retrieve relevant English exemplars for a given query to construct prompts.We also include global translation exemplars for a target language to facilitate the translation process for large language models.To systematically evaluate our model, we construct two new benchmark datasets, XSPI-DER and XKAGGLE-DBQA, which include questions in Chinese, Vietnamese, Farsi, and Hindi.Our experiments show that XRICL effectively leverages large pre-trained language models to outperform existing baselines.
The Devil in Linear Transformer

Linear transformers aim to reduce the quadratic space-time complexity of vanilla transformers.However, they usually suffer from degraded performances on various tasks and corpora.In this paper, we examine existing kernel-based linear transformers and identify two key issues that lead to such performance gaps: 1) unbounded gradients in the attention computation adversely impact the convergence of linear transformer models; 2) attention dilution which trivially distributes attention scores over long sequences while neglecting neighbouring structures.To address these issues, we first identify that the scaling of attention matrices is the devil in unbounded gradients, which turns out unnecessary in linear attention as we show theoretically and empirically.To this end, we propose a new linear attention that replaces the scaling operation with a normalization to stabilize gradients.For the issue of attention dilution, we leverage a diagonal attention to confine attention to only neighbouring tokens in early layers.Benefiting from the stable gradients and improved attention, our new linear transformer model, TRANSNORMER, demonstrates superior performance on text classification and language modeling tasks, as well as on the challenging Long-Range Arena benchmark, surpassing vanilla transformer and existing linear variants by a clear margin while being significantly more space-time efficient.The code is available at TRANSNORMER.
The Curious Case of Absolute Position Embeddings

Transformer language models encode the notion of word order using positional information.Most commonly, this positional information is represented by absolute position embeddings (APEs), that are learned from the pretraining data.However, in natural language, it is not absolute position that matters, but relative position, and the extent to which APEs can capture this type of information has not been investigated.In this work, we observe that models trained with APE over-rely on positional information to the point that they break-down when subjected to sentences with shifted position information.Specifically, when models are subjected to sentences starting from a non-zero position (excluding the effect of priming), they exhibit noticeably degraded performance on zero-to full-shot tasks, across a range of model families and model sizes.Our findings raise questions about the efficacy of APEs to model the relativity of position information, and invite further introspection on the sentence and word order processing strategies employed by these models.
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings

Learning sentence embeddings in an unsupervised manner is fundamental in natural language processing.Recent common practice is to couple pre-trained language models with unsupervised contrastive learning, whose success relies on augmenting a sentence with a semantically-close positive instance to construct contrastive pairs.Nonetheless, existing approaches usually depend on a monoaugmenting strategy, which causes learning shortcuts towards the augmenting biases and thus corrupts the quality of sentence embeddings.A straightforward solution is resorting to more diverse positives from a multiaugmenting strategy, while an open question remains about how to unsupervisedly learn from the diverse positives but with uneven augmenting qualities in the text field.As one answer, we propose a novel Peer-Contrastive Learning (PCL) with diverse augmentations.PCL constructs diverse contrastive positives and negatives at the group level for unsupervised sentence embeddings.PCL performs peer-positive contrast as well as peer-network cooperation, which offers an inherent anti-bias ability and an effective way to learn from diverse augmentations.Experiments on STS benchmarks verify the effectiveness of PCL against its competitors in unsupervised sentence embeddings.
CodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search

In this paper, we propose the CodeRetriever model, which learns the function-level code semantic representations through large-scale code-text contrastive pre-training.We adopt two contrastive learning schemes in CodeRetriever: unimodal contrastive learning and bimodal contrastive learning.For unimodal contrastive learning, we design an unsupervised learning approach to build semanticrelated code pairs based on the documentation and function name.For bimodal contrastive learning, we leverage the documentation and in-line comments of code to build code-text pairs.Both contrastive objectives can fully leverage large-scale code corpus for pre-training.Extensive experimental results show that CodeRetriever achieves new stateof-the-art with significant improvement over existing code pre-trained models, on eleven domain/language-specific code search tasks with six programming languages in different code granularity (function-level, snippet-level and statement-level).These results demonstrate the effectiveness and robustness of CodeRetriever.The codes and resources are available at https: