Reviewerly/exHarmony huggingface dataset:
SAIDS: A Novel Approach for Sentiment Analysis Informed of Dialect and Sarcasm

Sentiment analysis becomes an essential part of every social network, as it enables decisionmakers to know more about users' opinions in almost all life aspects.Despite its importance, there are multiple issues it encounters like the sentiment of the sarcastic text which is one of the main challenges of sentiment analysis.This paper tackles this challenge by introducing a novel system (SAIDS) that predicts the sentiment, sarcasm and dialect of Arabic tweets.SAIDS uses its prediction of sarcasm and dialect as known information to predict the sentiment.It uses MARBERT as a language model to generate sentence embedding, then passes it to the sarcasm and dialect models, and then the outputs of the three models are concatenated and passed to the sentiment analysis model.Multiple system design setups were experimented with and reported.SAIDS was applied to the ArSarcasm-v2 dataset where it outperforms the state-of-the-art model for the sentiment analysis task.By training all tasks together, SAIDS achieves results of 75.98 FPN, 59.09 F1-score and 71.13 F1-score for sentiment analysis, sarcasm detection, and dialect identification respectively.The system design can be used to enhance the performance of any task which is dependent on other tasks.
RED-ACE: Robust Error Detection for ASR using Confidence Embeddings

ASR Error Detection (AED) models aim to post-process the output of Automatic Speech Recognition (ASR) systems, in order to detect transcription errors.Modern approaches usually use text-based input, comprised solely of the ASR transcription hypothesis, disregarding additional signals from the ASR model.Instead, we utilize the ASR system's word-level confidence scores for improving AED performance.Specifically, we add an ASR Confidence Embedding (ACE) layer to the AED model's encoder, allowing us to jointly encode the confidence scores and the transcribed text into a contextualized representation.Our experiments show the benefits of ASR confidence scores for AED, their complementary effect over the textual signal, as well as the effectiveness and robustness of ACE for combining these signals.To foster further research, we publish a novel AED dataset consisting of ASR outputs on the LibriSpeech corpus with annotated transcription errors.
Interpreting Language Models with Contrastive Explanations

Model interpretability methods are often used to explain NLP model decisions on tasks such as text classification, where the output space is relatively small.However, when applied to language generation, where the output space often consists of tens of thousands of tokens, these methods are unable to provide informative explanations.Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics.Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding.To disentangle the different decisions in language modeling, we focus on explaining language models contrastively: we look for salient input tokens that explain why the model predicted one token instead of another.We demonstrate that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena, and that they significantly improve contrastive model simulatability for human observers.We also identify groups of contrastive decisions where the model uses similar evidence, and we are able to characterize what input tokens models use during various language generation decisions. 1
PENTATRON: PErsonalized coNText-Aware Transformer for Retrieval-based cOnversational uNderstanding

Niranjan Uma Naresh, Ziyan Jiang, Ankit Ankit, Sungjin Lee, Jie Hao, Xing Fan, Chenlei Guo. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2022.
Transfer Learning from Semantic Role Labeling to Event Argument Extraction with Template-based Slot Querying

In this work, we investigate transfer learning from semantic role labeling (SRL) to event argument extraction (EAE), considering their similar argument structures.We view the extraction task as a role querying problem, unifying various methods into a single framework.There are key discrepancies on role labels and distant arguments between semantic role and event argument annotations.To mitigate these discrepancies, we specify natural language-like queries to tackle the label mismatch problem and devise argument augmentation to recover distant arguments.We show that SRL annotations can serve as a valuable resource for EAE, and a template-based slot querying strategy is especially effective for facilitating the transfer.In extensive evaluations on two English EAE benchmarks, our proposed model obtains impressive zero-shot results by leveraging SRL annotations, reaching nearly 80% of the fullysupervised scores.It further provides benefits in low-resource cases, where few EAE annotations are available.Moreover, we show that our approach generalizes to cross-domain and multilingual scenarios.
Generative Approach for Gender-Rewriting Task with ArabicT5

Addressing the correct gender in generative tasks (e.g., Machine Translation) has been an overlooked issue in the Arabic NLP.However, the recent introduction of the Arabic Parallel Gender Corpus (APGC) dataset has established new baselines for the Arabic Gender Rewriting task.To address the Gender Rewriting task, we first pre-train our new Seq2Seq Ara-bicT5 model on a 17GB of Arabic Corpora.Then, we continue pre-training our ArabicT5 model on the APGC dataset using a newly proposed method.Our evaluation shows that our ArabicT5 model, when trained on the APGC dataset, achieved competitive results against existing state-of-the-art methods.In addition, our ArabicT5 model shows better results on the APGC dataset compared to other Arabic and multilingual T5 models.
Dialect &amp; Sentiment Identification in Nuanced Arabic Tweets Using an Ensemble of Prompt-based, Fine-tuned, and Multitask BERT-Based Models

Dialect Identification is important to improve the performance of various application as translation, speech recognition, etc.In this paper, we present our findings and results in the Nuanced Arabic Dialect Identification Shared Task (NADI 2022) for country-level dialect identification and sentiment identification for dialectical Arabic.The proposed model is an ensemble between fine-tuned BERT-based models and various approaches of prompt-tuning.Our model secured first place on the leaderboard for subtask 1 with an 27.06 F1-macro score, and subtask 2 secured first place with 75.15 F1-PN score.Our findings show that prompt-tuning-based models achieved better performance when compared to fine-tuning and Multi-task based methods.Moreover, using an ensemble of different loss functions might improve model performance.
A Weak Supervised Transfer Learning Approach for Sentiment Analysis to the Kuwaiti Dialect

Developing a system for sentiment analysis is very challenging for the Arabic language due to the limitations in the available Arabic datasets.Many Arabic dialects are still not studied by researchers in Arabic sentiment analysis due to the complexity of annotators' recruitment process during dataset creation.This paper covers the research gap in sentiment analysis for the Kuwaiti dialect by proposing a weak supervised approach to develop a large labeled dataset.Our dataset consists of over 16.6k tweets with 7,905 negatives, 7,902 positives, and 860 neutrals that spans several themes and time frames to remove any bias that might affect its content.The annotation agreement between our proposed system's labels and human-annotated labels reports 93% for the pairwise percent agreement and 0.87 for Cohen's kappa coefficient.Furthermore, we evaluate our dataset using multiple traditional machine learning classifiers and advanced deep learning language models to test its performance.The results report 89% accuracy when applied to the testing dataset using the ARBERT model.
EvEntS ReaLM: Event Reasoning of Entity States via Language Models

This paper investigates models of event implications.Specifically, how well models predict entity state-changes, by targeting their understanding of physical attributes.Nominally, Large Language models (LLM) have been exposed to procedural knowledge about how objects interact, yet our benchmarking shows they fail to reason about the world.Conversely, we also demonstrate that existing approaches often misrepresent the surprising abilities of LLMs via improper task encodings and that proper model prompting can dramatically improve performance of reported baseline results across multiple tasks.In particular, our results indicate that our prompting technique is especially useful for unseen attributes (out-of-domain) or when only limited data is available.
The Effect of Arabic Dialect Familiarity on Data Annotation

Data annotation is the foundation of most natural language processing (NLP) tasks.However, data annotation is complex and there is often no specific correct label, especially in subjective tasks.Data annotation is affected by the annotators' ability to understand the provided data.In the case of Arabic, this is important due to the large dialectal variety.In this paper, we analyse how Arabic speakers understand other dialects in written text.Also, we analyse the effect of dialect familiarity on the quality of data annotation, focusing on Arabic sarcasm detection.This is done by collecting third-party labels and comparing them to high-quality first-party labels.Our analysis shows that annotators tend to better identify their own dialect and they are prone to confuse dialects they are unfamiliar with.For task labels, annotators tend to perform better on their dialect or dialects they are familiar with.Finally, females tend to perform better than males on the sarcasm detection task.We suggest that to guarantee high-quality labels, researchers should recruit native dialect speakers for annotation.
Robustness of Demonstration-based Learning Under Limited Data Scenario

Demonstration-based learning has shown great potential in stimulating pretrained language models' ability under limited data scenario.Simply augmenting the input with some demonstrations can significantly improve performance on few-shot NER.However, why such demonstrations are beneficial remains unclear since there is no explicit alignment between the demonstrations and the predictions.In this paper, we design pathological demonstrations by gradually removing intuitively useful information from the standard ones to take a deep dive of the robustness of demonstration-based sequence labeling and show that (1) demonstrations composed of random tokens still make the model a better few-shot learner; (2) the length of random demonstrations and the relevance of random tokens are the main factors affecting the performance; (3) demonstrations increase the confidence of model predictions on captured superficial patterns.We have publicly released our code at https://github.com/SALT-NLP/RobustDemo.
SpeechNet: Weakly Supervised, End-to-End Speech Recognition at Industrial Scale

End-to-end automatic speech recognition systems represent the state of the art, but they rely on thousands of hours of manually annotated speech for training, as well as heavyweight computation for inference.Of course, this impedes commercialization since most companies lack vast human and computational resources.In this paper, we explore training and deploying an ASR system in the label-scarce, computelimited setting.To reduce human labor, we use a third-party ASR system as a weak supervision source, supplemented with labeling functions derived from implicit user feedback.To accelerate inference, we propose to route production-time queries across a pool of CUDA graphs of varying input lengths, the distribution of which best matches the traffic's.Compared to our third-party ASR, we achieve a relative improvement in word-error rate of 8% and a speedup of 600%.Our system, called Speech-Net, currently serves 12 million queries per day on our voice-enabled smart television.To our knowledge, this is the first time a large-scale, Wav2vec-based deployment has been described in the academic literature.
Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning

In the field of multimodal sentiment analysis (MSA), a few studies have leveraged the inherent modality correlation information stored in samples for self-supervised learning.However, they feed the training pairs in a random order without consideration of difficulty.Without human annotation, the generated training pairs of self-supervised learning often contain noise.If noisy or hard pairs are used for training at the easy stage, the model might be stuck in bad local optimum.In this paper, we inject curriculum learning into weakly supervised modality correlation learning.The weakly supervised correlation learning leverages the label information to generate scores for negative pairs to learn a more discriminative embedding space, where negative pairs are defined as two unimodal embeddings from different samples.To assist the correlation learning, we feed the training pairs to the model according to difficulty by the proposed curriculum learning, which consists of elaborately designed scoring and feeding functions.The scoring function computes the difficulty of pairs using pre-trained and current correlation predictors, where the pairs with large losses are defined as hard pairs.Notably, the hardest pairs are discarded in our algorithm, which are assumed as noisy pairs.Moreover, the feeding function takes the difference of correlation losses as feedback to determine the feeding actions ('stay', 'step back', or 'step forward').The proposed method reaches state-of-the-art performance on MSA.
Reproducibility Issues for BERT-based Evaluation Metrics

Reproducibility is of utmost concern in machine learning and natural language processing (NLP).In the field of natural language generation (especially machine translation), the seminal paper of Post (2018) has pointed out problems of reproducibility of the dominant metric, BLEU, at the time of publication.Nowadays, BERT-based evaluation metrics considerably outperform BLEU.In this paper, we ask whether results and claims from four recent BERT-based metrics can be reproduced.We find that reproduction of claims and results often fails because of (i) heavy undocumented preprocessing involved in the metrics, (ii) missing code and (iii) reporting weaker results for the baseline metrics.(iv) In one case, the problem stems from correlating not to human scores but to a wrong column in the csv file, inflating scores by 5 points.Motivated by the impact of preprocessing, we then conduct a second study where we examine its effects more closely (for one of the metrics).We find that preprocessing can have large effects, especially for highly inflectional languages.In this case, the effect of preprocessing may be larger than the effect of the aggregation mechanism (e.g., greedy alignment vs. Word Mover Distance).
Identifying Code-switching in Arabizi

We describe a corpus of social media posts that include utterances in Arabizi, a Romanscript rendering of Arabic, mixed with other languages, notably English, French, and Arabic written in the Arabic script.We manually annotated a subset of the texts with word-level language IDs; this is a non-trivial task due to the nature of mixed-language writing, especially on social media.We developed classifiers that can accurately predict the language ID tags.Then, we extended the word-level predictions to identify sentences that include Arabizi (and code-switching), and applied the classifiers to the raw corpus, thereby harvesting a large number of additional instances.The result is a large-scale dataset of Arabizi, with precise indications of code-switching between Arabizi and English, French, and Arabic.
KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models

Knowledge Graphs (KGs) store information in the form of (head, predicate, tail)-triples.To augment KGs with new knowledge, researchers proposed models for KG Completion (KGC) tasks such as link prediction; i.e., answering (h; p; ?) or (?; p; t) queries.Such models are usually evaluated with averaged metrics on a held-out test set.While useful for tracking progress, averaged single-score metrics cannot reveal what exactly a model has learned-or failed to learn.To address this issue, we propose KGxBoard 1 : an interactive framework for performing fine-grained evaluation on meaningful subsets of the data, each of which tests individual and interpretable capabilities of a KGC model.In our experiments, we highlight the findings that we discovered with the use of KGxBoard, which would have been impossible to detect with standard averaged single-score metrics.
Continued Pretraining for Better Zero- and Few-Shot Promptability

Recently introduced language model prompting methods can achieve high accuracy in zeroand few-shot settings while requiring few to no learned task-specific parameters.Nevertheless, these methods still often trail behind full model finetuning.In this work, we investigate if a dedicated continued pretraining stage could improve "promptability", i.e., zero-shot performance with natural language prompts or few-shot performance with prompt tuning.We reveal settings where existing continued pretraining methods lack promptability.We also identify current methodological gaps, which we fill with thorough large-scale experiments.We demonstrate that a simple recipe, continued pretraining that incorporates a trainable prompt during multi-task learning, leads to improved promptability in both zero-and fewshot settings compared to existing methods, up to 31% relative.On the other hand, we find that continued pretraining using MAML-style metalearning, a method that directly optimizes fewshot promptability, yields subpar performance.We validate our findings with two prompt tuning methods, and, based on our results, we provide concrete recommendations to optimize promptability for different use cases.
RelU-Net: Syntax-aware Graph U-Net for Relational Triple Extraction

Relational triple extraction is a critical task for natural language processing.Existing methods mainly focused on capturing semantic information, but suffered from ignoring the syntactic structure of the sentence, which is proved in the relation classification task to contain rich relational information.This is due to the absence of entity locations, which is the prerequisite for pruning noisy edges from the dependency tree, when extracting relational triples.In this paper, we propose a unified framework to tackle this challenge and incorporate syntactic information for relational triple extraction.First, we propose to automatically contract the dependency tree into a core relational topology and eliminate redundant information with graph pooling operations.Then, we propose a symmetrical expanding path with graph unpooling operations to fuse the contracted core syntactic interactions with the original sentence context.We also propose a bipartite graph matching objective function to capture the reflections between the core topology and golden relational facts.Since our model shares similar contracting and expanding paths with encoder-decoder models like U-Net, we name our model as Relation U-Net (RelU-Net).We conduct experiments on several datasets and the results prove the effectiveness of our method.
Factorizing Content and Budget Decisions in Abstractive Summarization of Long Documents

We argue that disentangling content selection from the budget used to cover salient content improves the performance and applicability of abstractive summarizers.Our method, FAC-TORSUM 1 , does this disentanglement by factorizing summarization into two steps through an energy function: (1) generation of abstractive summary views covering salient information in subsets of the input document (document views) ; (2) combination of these views into a final summary, following a budget and content guidance.This guidance may come from different sources, including from an advisor model such as BART or BigBird, or in oracle modefrom the reference.This factorization achieves significantly higher ROUGE scores on multiple benchmarks for long document summarization, namely PubMed, arXiv, and GovReport.Notably, our model is effective for domain adaptation.When trained only on PubMed, it achieves a 46.29 ROUGE-1 score on arXiv, outperforming PEGASUS trained in domain by a large margin.Our experimental results indicate that the performance gains are due to more flexible budget adaptation and processing of shorter contexts provided by partial document views.
Domain-Adapted BERT-based Models for Nuanced Arabic Dialect Identification and Tweet Sentiment Analysis

This paper summarizes the solution of the Nuanced Arabic Dialect Identification (NADI) 2022 shared task.It consists of two subtasks: a country-level Arabic Dialect Identification (ADID) and an Arabic Sentiment Analysis (ASA).Our work shows the importance of using domain-adapted models and languagespecific pre-processing in NLP task solutions.We implement a simple but strong baseline technique to increase the stability of fine-tuning settings to obtain a good generalization of models.Our best model for the Dialect Identification subtask achieves a Macro F-1 score of 25.54% as an average of both Test-A (33.89%) and Test-B (19.19%)F-1 scores.We also obtained a Macro F-1 score of 74.29% of positive and negative sentiments only, in the Sentiment Analysis task 1 .
MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation

As generic machine translation (MT) quality has improved, the need for targeted benchmarks that explore fine-grained aspects of quality has increased (Freitag et al., 2021;Isabelle et al., 2017).In particular, gender accuracy in translation (Choubey et al., 2021; Saunders and Byrne, 2020) can have implications in terms of output fluency, translation accuracy, and ethics.In this paper, we introduce MT-GenEval, a benchmark for evaluating gender accuracy in translation from English into eight widely-spoken languages.MT-GenEval complements existing benchmarks by providing realistic, gender-balanced, counterfactual data in eight language pairs where the gender of individuals is unambiguous in the input segment, including multi-sentence segments requiring inter-sentential gender agreement.
Mitigating Inconsistencies in Multimodal Sentiment Analysis under Uncertain Missing Modalities

For the missing modality problem in Multimodal Sentiment Analysis (MSA), the inconsistency phenomenon occurs when the sentiment changes due to the absence of a modality.The absent modality that determines the overall semantic can be considered as a key missing modality.However, previous works all ignored the inconsistency phenomenon, simply discarding missing modalities or solely generating associated features from available modalities.The neglect of the key missing modality case may lead to incorrect semantic results.To tackle the issue, we propose an Ensemble-based Missing Modality Reconstruction (EMMR) network to detect and recover semantic features of the key missing modality.Specifically, we first learn joint representations with remaining modalities via a backbone encoder-decoder network.Then, based on the recovered features, we check the semantic consistency to determine whether the absent modality is crucial to the overall sentiment polarity.Once the inconsistency problem due to the key missing modality exists, we integrate several encoder-decoder approaches for better decision making.Extensive experiments and analyses are conducted on CMU-MOSI and IEMOCAP datasets, validating the superiority of the proposed method.