Reviewerly/exHarmony huggingface dataset:
Seals_Lab at SemEval-2023 Task 12: Sentiment Analysis for Low-resource African Languages, Hausa and Igbo

One of the most extensively researched applications in natural language processing (NLP) is sentiment analysis.While the majority of the study focuses on high-resource languages (e.g., English), this research will focus on lowresource African languages namely Igbo and Hausa.The annotated tweets of both languages have a significant number of code-mixed tweets.The curated datasets necessary to build complex AI applications are not available for the majority of African languages.To optimize the use of such datasets, research is needed to determine the viability of present NLP procedures as well as the development of novel techniques.This paper outlines our efforts to develop a sentiment analysis (for positive and negative as well as neutral) system for tweets from the Hausa, and Igbo languages.Sentiment analysis can computationally analyze and discover sentiments in a text or document.We worked on the first thorough compilation of AfriSenti-SemEval 2023 Shared Task 12 Twitter datasets that are human-annotated for the most widely spoken languages in Nigeria, such as Hausa and Igbo.Here we trained the modern pretrained language model AfriBERTa large on the AfriSenti-SemEval Shared Task 12 Twitter dataset to create sentiment classification.In particular, the results demonstrate that our model trained on AfriSenti-SemEval Shared Task 12 datasets and produced with an F1 score of 80.85% for Hausa and 80.82% for Igbo languages on the sentiment analysis test.In AfriSenti-SemEval 2023 shared task 12 (Task A), we consistently ranked top 10 by achieving a mean F1 score of more than 80% for both the Hausa and Igbo languages.
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations

Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswami, Changhan Wang, Juan Pino, Benoît Sagot, Holger Schwenk. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
Prompting Language Models for Linguistic Structure

Although pretrained language models (PLMs) can be prompted to perform a wide range of language tasks, it remains an open question how much this ability comes from generalizable linguistic understanding versus surface-level lexical patterns.To test this, we present a structured prompting approach for linguistic structured prediction tasks, allowing us to perform zero-and few-shot sequence tagging with autoregressive PLMs.We evaluate this approach on part-of-speech tagging, named entity recognition, and sentence chunking, demonstrating strong few-shot performance in all cases.We also find that while PLMs contain significant prior knowledge of task labels due to task leakage into the pretraining corpus, structured prompting can also retrieve linguistic structure with arbitrary labels.These findings indicate that the in-context learning ability and linguistic knowledge of PLMs generalizes beyond memorization of their training data.C: Alright what is this ?T: Alright_INTJ what_PRON is_AUX this_PRON ?
AfroLID: A Neural Language Identification Tool for African Languages

Language identification (LID) is a crucial precursor for NLP, especially for mining web data.Problematically, most of the world's 7000+ languages today are not covered by LID technologies.We address this pressing issue for Africa by introducing AfroLID, a neural LID toolkit for 517 African languages and varieties.AfroLID exploits a multi-domain web dataset manually curated from across 14 language families utilizing five orthographic systems.When evaluated on our blind Test set, AfroLID achieves 95.89 F 1 -score.We also compare AfroLID to five existing LID tools that each cover a small number of African languages, finding it to outperform them on most languages.We further show the utility of AfroLID in the wild by testing it on the acutely under-served Twitter domain.Finally, we offer a number of controlled case studies and perform a linguistically-motivated error analysis that allow us to both showcase AfroLID's powerful capabilities and limitations. 1
Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition

Continual Learning for Named Entity Recognition (CL-NER) aims to learn a growing number of entity types over time from a stream of data.However, simply learning Other-Class in the same way as new entity types amplifies the catastrophic forgetting and leads to a substantial performance drop.The main cause behind this is that Other-Class samples usually contain old entity types, and the old knowledge in these Other-Class samples is not preserved properly.Thanks to the causal inference, we identify that the forgetting is caused by the missing causal effect from the old data.To this end, we propose a unified causal framework to retrieve the causality from both new entity types and Other-Class.Furthermore, we apply curriculum learning to mitigate the impact of label noise and introduce a self-adaptive weight for balancing the causal effects between new entity types and Other-Class.Experimental results on three benchmark datasets show that our method outperforms the state-of-theart method by a large margin.Moreover, our method can be combined with the existing stateof-the-art methods to improve the performance in CL-NER.
Pair-Based Joint Encoding with Relational Graph Convolutional Networks for Emotion-Cause Pair Extraction

Emotion-cause pair extraction (ECPE) aims to extract emotion clauses and corresponding cause clauses, which have recently received growing attention.Previous methods sequentially encode features with a specified order.They first encode the emotion and cause features for clause extraction and then combine them for pair extraction.This lead to an imbalance in inter-task feature interaction where features extracted later have no direct contact with the former.To address this issue, we propose a novel Pair-Based Joint Encoding (PBJE) network, which generates pairs and clauses features simultaneously in a joint feature encoding manner to model the causal relationship in clauses.PBJE can balance the information flow among emotion clauses, cause clauses and pairs.From a multi-relational perspective, we construct a heterogeneous undirected graph and apply the Relational Graph Convolutional Network (RGCN) to capture the various relationship between clauses and the relationship between pairs and clauses.Experimental results show that PBJE achieves state-of-the-art performance on the Chinese benchmark corpus.
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning

Instruction tuning is an emergent paradigm in NLP wherein natural language instructions are leveraged with language models to induce zeroshot performance on unseen tasks.Dialogue is an especially interesting area in which to explore instruction tuning because dialogue systems perform multiple tasks related to language (e.g., natural language understanding and generation, domain-specific interaction), yet instruction tuning has not been systematically explored for dialogue-related tasks.We introduce INSTRUCTDIAL, an instruction tuning framework for dialogue, which consists of a repository of 48 diverse dialogue tasks in a unified text-to-text format created from 59 openly available dialogue datasets.We explore crosstask generalization ability on models tuned on INSTRUCTDIAL across diverse dialogue tasks.Our analysis reveals that INSTRUCTDIAL enables good zero-shot performance on unseen datasets and tasks such as dialogue evaluation and intent detection, and even better performance in a few-shot setting.To ensure that models adhere to instructions, we introduce novel meta-tasks.We establish benchmark zero-shot and few-shot performance of models trained using the proposed framework on multiple dialogue tasks 1 .
Reinforced Question Rewriting for Conversational Question Answering

Conversational Question Answering (CQA) aims to answer questions contained within dialogues, which are not easily interpretable without context.Developing a model to rewrite conversational questions into self-contained ones is an emerging solution in industry settings as it allows using existing single-turn QA systems to avoid training a CQA model from scratch.Previous work trains rewriting models using human rewrites as supervision.However, such objectives are disconnected with QA models and therefore more human-like rewrites do not guarantee better QA performance.In this paper we propose using QA feedback to supervise the rewriting model with reinforcement learning.Experiments show that our approach can effectively improve QA performance over baselines for both extractive and retrieval QA.Furthermore, human evaluation shows that our method can generate more accurate and detailed rewrites when compared to human annotations.
ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering

With the recent advance in large pre-trained language models, researchers have achieved record performances in NLP tasks that mostly focus on language pattern matching.The community is experiencing the shift of the challenge from how to model language to the imitation of complex reasoning abilities like human beings.In this work, we investigate the application domain of finance that involves realworld, complex numerical reasoning.We propose a new large-scale dataset, CONVFINQA, aiming to study the chain of numerical reasoning in conversational question answering.Our dataset poses great challenge in modeling longrange, complex numerical reasoning paths in real-world conversations.We conduct comprehensive experiments and analyses with both the neural symbolic methods and the promptingbased methods, to provide insights into the reasoning mechanisms of these two divisions.We believe our new dataset should serve as a valuable resource to push forward the exploration of real-world, complex reasoning tasks as the next research focus.Our dataset and code is publicly available 1 .
Probing GPT-3’s Linguistic Knowledge on Semantic Tasks

GPT-3 has attracted much attention from both academia and industry.However, it is still unclear what GPT-3 has understood or learned especially in linguistic knowledge.Some studies have shown linguistic phenomena including negation and tense are hard to be recognized by language models such as BERT.In this study, we conduct probing tasks focusing on semantic information.Specifically, we investigate GPT-3's linguistic knowledge on semantic tasks to identify tense, the number of subjects, and the number of objects for a given sentence.We also experiment with different prompt designs and temperatures of the decoding method.Our experiment results suggest that GPT-3 has acquired linguistic knowledge to identify certain semantic information in most cases, but still fails when there are some types of disturbance happening in the sentence.We also perform error analysis to summarize some common types of mistakes that GPT-3 has made when dealing with certain semantic information.
Formulating Few-shot Fine-tuning Towards Language Model Pre-training: A Pilot Study on Named Entity Recognition

Fine-tuning pre-trained language models is a common practice in building NLP models for various tasks, including the case with less supervision.We argue that under the few-shot setting, formulating fine-tuning closer to the pretraining objective shall be able to unleash more benefits from the pre-trained language models.In this work, we take few-shot named entity recognition (NER) for a pilot study, where existing fine-tuning strategies are much different from pre-training.We propose a novel few-shot fine-tuning framework for NER, FFF-NER.Specifically, we introduce three new types of tokens, "is-entity", "which-type" and bracket, so we can formulate the NER fine-tuning as (masked) token prediction or generation, depending on the choice of the pre-training objective.In our experiments, we apply FFF-NER to fine-tune both BERT and BART for fewshot NER on several benchmark datasets and observe significant improvements over existing fine-tuning strategies, including sequence labeling, prototype meta-learning, and promptbased approaches.We further perform a series of ablation studies, showing few-shot NER performance is strongly correlated with the similarity between fine-tuning and pre-training.
Calibrating Trust of Multi-Hop Question Answering Systems with Decompositional Probes

Multi-hop Question Answering (QA) is a challenging task since it requires an accurate aggregation of information from multiple context paragraphs and a thorough understanding of the underlying reasoning chains.Recent work in multi-hop QA has shown that performance can be boosted by first decomposing the questions into simpler, single-hop questions.In this paper, we explore one additional utility of the multi-hop decomposition from the perspective of explainable NLP: to create explanation by probing a neural QA model with them.We hypothesize that in doing so, users will be better able to predict when the underlying QA system will give the correct answer.Through human participant studies, we verify that exposing the decomposition probes and answers to the probes to users can increase their ability to predict system performance on a question instance basis.We show that decomposition is an effective form of probing QA systems as well as a promising approach to explanation generation.In-depth analyses show the need for improvements in decomposition systems. 1
Improving Semantic Matching through Dependency-Enhanced Pre-trained Model with Adaptive Fusion

Transformer-based pre-trained models like BERT have achieved great progress on Semantic Sentence Matching.Meanwhile, dependency prior knowledge has also shown general benefits in multiple NLP tasks.However, how to efficiently integrate dependency prior structure into pre-trained models to better model complex semantic matching relations is still unsettled.In this paper, we propose the Dependency-Enhanced Adaptive Fusion Attention (DAFA), which explicitly introduces dependency structure into pre-trained models and adaptively fuses it with semantic information.Specifically, (i) DAFA first proposes a structure-sensitive paradigm to construct a dependency matrix for calibrating attention weights.(ii) It adopts an adaptive fusion module to integrate the obtained dependency information and the original semantic signals.Moreover, DAFA reconstructs the attention calculation flow and provides better interpretability.By applying it on BERT, our method achieves state-of-the-art or competitive performance on 10 public datasets, demonstrating the benefits of adaptively fusing dependency structure in semantic matching task.
Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP

Textual adversarial samples play important roles in multiple subfields of NLP research, including security, evaluation, explainability, and data augmentation.However, most work mixes all these roles, obscuring the problem definitions and research goals of the security role that aims to reveal the practical concerns of NLP models.In this paper, we rethink the research paradigm of textual adversarial samples in security scenarios.We discuss the deficiencies in previous work and propose our suggestions that the research on the Security-oriented adversarial NLP (SoadNLP) should: (1) evaluate their methods on security tasks to demonstrate the real-world concerns; (2) consider realworld attackers' goals, instead of developing impractical methods.To this end, we first collect, process, and release a security datasets collection Advbench.Then, we reformalize the task and adjust the emphasis on different goals in SoadNLP.Next, we propose a simple method based on heuristic rules that can easily fulfill the actual adversarial goals to simulate real-world attack methods.We conduct experiments on both the attack and the defense sides on Advbench.Experimental results show that our method has higher practical value, indicating that the research paradigm in SoadNLP may start from our new benchmark.
ACENet: Attention Guided Commonsense Reasoning on Hybrid Knowledge Graph

Augmenting pre-trained language models (PLMs) with knowledge graphs (KGs) has demonstrated superior performance on commonsense reasoning.Given a commonsense based QA context (question and multiple choices), existing approaches usually estimate the plausibility of candidate choices separately based on their respective retrieved KGs, without considering the interference among different choices.In this paper, we propose an Attention guided Commonsense rEasoning Network (ACENet) 1 to endow the neural network with the capability of integrating hybrid knowledge.Specifically, our model applies the multilayer interaction of answer choices to continually strengthen correct choice information and guide the message passing of GNN.In addition, we also design a mix attention mechanism of nodes and edges to iteratively select supporting evidence on hybrid knowledge graph.Experimental results demonstrate the effectiveness of our proposed model through considerable performance gains across CommonsenseQA and OpenbookQA datasets.
Multi-Granularity Optimization for Non-Autoregressive Translation

Despite low latency, non-autoregressive machine translation (NAT) suffers severe performance deterioration due to the naive independence assumption.This assumption is further strengthened by cross-entropy loss, which encourages a strict match between the hypothesis and the reference token by token.To alleviate this issue, we propose multi-granularity optimization for NAT, which collects model behaviors on translation segments of various granularities and integrates feedback for backpropagation.Experiments on four WMT benchmarks show that the proposed method significantly outperforms the baseline models trained with cross-entropy loss, and achieves the best performance on WMT'16 En⇔Ro and highly competitive results on WMT'14 En⇔De for fully non-autoregressive translation.
Summarization as Indirect Supervision for Relation Extraction

Relation extraction (RE) models have been challenged by their reliance on training data with expensive annotations.Considering that summarization tasks aim at acquiring concise expressions of synoptical information from the longer context, these tasks naturally align with the objective of RE, i.e., extracting a kind of synoptical information that describes the relation of entity mentions.We present SURE, which converts RE into a summarization formulation.SURE leads to more precise and resource-efficient RE based on indirect supervision from summarization tasks.To achieve this goal, we develop sentence and relation conversion techniques that essentially bridge the formulation of summarization and RE tasks.We also incorporate constraint decoding techniques with Trie scoring to further enhance summarization-based RE with robust inference.Experiments on three RE datasets demonstrate the effectiveness of SURE in both full-dataset and low-resource settings, showing that summarization is a promising source of indirect supervision signals to improve RE models. 1
Improving Bilingual Lexicon Induction with Cross-Encoder Reranking

Bilingual lexicon induction (BLI) with limited bilingual supervision is a crucial yet challenging task in multilingual NLP.Current state-ofthe-art BLI methods rely on the induction of cross-lingual word embeddings (CLWEs) to capture cross-lingual word similarities; such CLWEs are obtained 1) via traditional static models (e.g., VECMAP), or 2) by extracting type-level CLWEs from multilingual pretrained language models (mPLMs), or 3) through combining the former two options.In this work, we propose a novel semi-supervised post-hoc reranking method termed BLICEr (BLI with Cross-Encoder Reranking), applicable to any precalculated CLWE space, which improves their BLI capability.The key idea is to 'extract' cross-lingual lexical knowledge from mPLMs, and then combine it with the original CLWEs.This crucial step is done via 1) creating a word similarity dataset, comprising positive word pairs (i.e., true translations) and hard negative pairs induced from the original CLWE space, and then 2) fine-tuning an mPLM (e.g., mBERT or XLM-R) in a cross-encoder manner to predict the similarity scores.At inference, we 3) combine the similarity score from the original CLWE space with the score from the BLI-tuned cross-encoder.BLICEr establishes new state-ofthe-art results on two standard BLI benchmarks spanning a wide spectrum of diverse languages: it substantially outperforms a series of strong baselines across the board.We also validate the robustness of BLICEr with different CLWEs.
FLUTE: Figurative Language Understanding through Textual Explanations

Figurative language understanding has been recently framed as a recognizing textual entailment (RTE) task (a.k.a.natural language inference, or NLI).However, similar to classical RTE/NLI datasets, the current benchmarks suffer from spurious correlations and annotation artifacts.To tackle this problem, work on NLI has built explanation-based datasets such as e-SNLI, allowing us to probe whether language models are right for the right reasons.Yet no such data exists for figurative language, making it harder to assess genuine understanding of such expressions.To address this issue, we release FLUTE, a dataset of 9,000 figurative NLI instances with explanations, spanning four categories: Sarcasm, Simile, Metaphor, and Idioms.We collect the data through a model-in-the-loop framework based on GPT-3, crowd workers, and expert annotators.We show how utilizing GPT-3 in conjunction with human annotators (novices and experts) can aid in scaling up the creation of datasets even for such complex linguistic phenomena as figurative language.The baseline performance of the T5 model fine-tuned on FLUTE shows that our dataset can bring us a step closer to developing models that understand figurative language through textual explanations.
Semantic Novelty Detection and Characterization in Factual Text Involving Named Entities

Much of the existing work on text novelty detection has been studied at the topic level, i.e., identifying whether the topic of a document or a sentence is novel or not.Little work has been done at the fine-grained semantic level (or contextual level).For example, given that we know Elon Musk is the CEO of a technology company, the sentence "Elon Musk acted in the sitcom The Big Bang Theory" is novel and surprising because normally a CEO would not be an actor.Existing topic-based novelty detection methods work poorly on this problem because they do not perform semantic reasoning involving relations between named entities in the text and their background knowledge.This paper proposes an effective model (called PAT-SND) to solve the problem, which can also characterize the novelty.An annotated dataset is also created.Evaluation shows that PAT-SND outperforms 10 baselines by large margins.′ 1 , e ′ 2 ) with respect to D tr , K, and pair (e ′ 1 , e ′ 2 ), i.e., classifying d ′ into one of the classes {NORMAL, NOVEL}.As F is built with only the "NORMAL" data, the task is an one-class classification problem.