Reviewerly/exHarmony huggingface dataset:
Efficient Coreset Selection with Cluster-based Methods

Coreset selection is a technique for efficient machine learning, which selects a subset of the training data to achieve similar model performance as using the full dataset. It can be performed with or without training machine learning models. Coreset selection with training, which iteratively trains the machine model and updates data items in the coreset, is time consuming. Coreset selection without training can select the coreset before training. Gradient approximation is the typical method, but it can also be slow when dealing with large training datasets as it requires multiple iterations and pairwise distance computations for each iteration. The state-of-the-art (SOTA) results w.r.t. effectiveness are achieved by the latter approach, i.e. gradient approximation.
A Federated Deep Reinforcement Learning-Based Trust Model in Underwater Acoustic Sensor Networks

Underwater acoustic sensor networks (UASNs) have been widely deployed in many areas, such as marine ranching, naval applications, and marine disaster warning systems. The security of UASNs, particularly insider threats, is of growing concern. Internal attacks carried out via compromised normal nodes are more damaging and stealthy than external attacks, such as signal stealing, data decryption, and identity forgery. As a security mechanism for internal threat detection based on interaction data, trust models have proven to enhance the security of UASNs. However, traditional trust models lack sufficient scalability when faced with movable underwater devices, heterogeneous network environments, and variable attack patterns. Therefore, in this paper, a novel trust model based on federated deep reinforcement learning is proposed for UASNs. First, the evidence acquisition mechanism, including communication, energy, and data evidence, is improved based on existing ones to better accommodate the topological dynamics of UASNs. Second, acquired trust evidence is fed into the corresponding deep reinforcement learning-based local trust model to accomplish trust prediction and model training. Finally, a federated learning-based update method periodically aggregates and updates the parameters of the local models. The experimental results prove that the proposed scheme exhibits satisfactory performance in terms of improving trust prediction accuracy and energy efficiency.
Forgotten Knowledge: Examining the Citational Amnesia in NLP

Citing papers is the primary method through which modern scientific writing discusses and builds on past work.Collectively, citing a diverse set of papers (in time and area of study) is an indicator of how widely the community is reading.Yet there is little work looking at broad temporal patterns of citation.This work, systematically and empirically examines: How far back in time do we tend to go to cite papers?How has that changed over time, and what factors correlate with this citational attention/amnesia?We chose NLP as our domain of interest, and analyzed ∼71.5K papers to show and quantify several key trends in citation.Notably, ∼62% of cited papers are from the immediate five years prior to publication, whereas only ∼17% are more than ten years old.Furthermore, we show that the median age and age diversity of cited papers was steadily increasing from 1990 to 2014, but since then the trend has reversed, and current NLP papers have an all-time low temporal citation diversity.Finally, we show that unlike the 1990s, the highly cited papers in the last decade were also papers with the least citation diversity; likely contributing to the intense (and arguably harmful) recency focus.Code, data, and a demo are available at the project homepage.
Resources for Automated Identification of Online Gender-Based Violence: A Systematic Review

Online Gender-Based Violence (GBV), such as misogynistic abuse, is an increasingly prevalent problem that technological approaches have struggled to address.Through the lens of the GBV framework, which is rooted in social science and policy, we systematically review 63 available resources for automated identification of such language.We find the datasets are limited in a number of important ways, such as their lack of theoretical grounding and stakeholder input, static nature, and focus on certain media platforms.Based on this review, we recommend development of future resources rooted in sociological expertise and centering stakeholder voices, namely GBV experts and people with lived experience of GBV.* Now at Google DeepMind.We particularly focus on the extent to which stakeholders, namely GBV experts and people with lived experience of GBV have been included in the design and production of these resources.Databases Following a scoping study to establish coverage of GBV-related publications and datasets, we searched two databases: the DBLP Computer Science Bibliography 2 and hatespeechdata.com. 3 We found that these were sufficient to cover all papers published at typical NLP venues such as the ACL Anthology. 4 Keyword selection We used the primary search keywords misogyn*, sexis*, and "gender based violence".For DBLP, to capture publications that concern hate speech and abusive language more generally, but that include categories relevant to GBV, we also search using the secondary keywords hate speech | detection | rhetoric, abuse,
Tackling Modality Heterogeneity with Multi-View Calibration Network for Multimodal Sentiment Detection

With the popularity of social media, detecting sentiment from multimodal posts (e.g.imagetext pairs) has attracted substantial attention recently.Existing works mainly focus on fusing different features but ignore the challenge of modality heterogeneity.Specifically, different modalities with inherent disparities may bring three problems: 1) introducing redundant visual features during feature fusion; 2) causing feature shift in the representation space; 3) leading to inconsistent annotations for different modal data.All these issues will increase the difficulty in understanding the sentiment of the multimodal content.In this paper, we propose a novel Multi-View Calibration Network (MVCN) to alleviate the above issues systematically.We first propose a text-guided fusion module with novel Sparse-Attention to reduce the negative impacts of redundant visual elements.We then devise a sentiment-based congruity constraint task to calibrate the feature shift in the representation space.Finally, we introduce an adaptive loss calibration strategy to tackle inconsistent annotated labels.Extensive experiments demonstrate the competitiveness of MVCN against previous approaches and achieve state-of-the-art results on two public benchmark datasets.
Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training

Dense retrievers have achieved impressive performance, but their demand for abundant training data limits their application scenarios.Contrastive pre-training, which constructs pseudopositive examples from unlabeled data, has shown great potential to solve this problem.However, the pseudo-positive examples crafted by data augmentations can be irrelevant.To this end, we propose relevance-aware contrastive learning.It takes the intermediatetrained model itself as an imperfect oracle to estimate the relevance of positive pairs and adaptively weighs the contrastive loss of different pairs according to the estimated relevance.Our method consistently improves the SOTA unsupervised Contriever model (Izacard et al., 2022) on the BEIR and open-domain QA retrieval benchmarks.Further exploration shows that our method can not only beat BM25 after further pre-training on the target corpus but also serves as a good few-shot learner.Our code is publicly available at https://github.
Do Question Answering Modeling Improvements Hold Across Benchmarks?

Do question answering (QA) modeling improvements (e.g., choice of architecture and training procedure) hold consistently across the diverse landscape of QA benchmarks?To study this question, we introduce the notion of concurrence-two benchmarks have high concurrence on a set of modeling approaches if they rank the modeling approaches similarly.We measure the concurrence between 32 QA benchmarks on a set of 20 diverse modeling approaches and find that human-constructed benchmarks have high concurrence amongst themselves, even if their passage and question distributions are very different.Surprisingly, even downsampled human-constructed benchmarks (i.e., collecting less data) and programmatically-generated benchmarks (e.g., cloze-formatted examples) have high concurrence with human-constructed benchmarks.These results indicate that, despite years of intense community focus on a small number of benchmarks, the modeling improvements studied hold broadly.
Sentiment Analysis using the Relationship between Users and Products

In product reviews, user and product aspects are useful in sentiment analysis.Nevertheless, previous studies mainly focus on modeling user and product aspects without considering the relationship between users and products.The relationship between users and products is typically helpful in estimating the bias of a user toward a product.In this paper, we, therefore, introduce the Graph Neural Network-based model with the pre-trained Language Model (GNNLM), where the relationship between users and products is incorporated.We conducted experiments on three well-known benchmarks for sentiment classification with the user and product information.The experimental results show that the relationship between users and products improves the performance of sentiment analysis.Furthermore, GNNLM achieves state-of-the-art results on yelp-2013 and yelp-2014 datasets.
Divide, Conquer, and Combine: Mixture of Semantic-Independent Experts for Zero-Shot Dialogue State Tracking

Zero-shot transfer learning for Dialogue State Tracking (DST) helps to handle a variety of task-oriented dialogue domains without the cost of collecting in-domain data.Existing works mainly study common data-or modellevel augmentation methods to enhance the generalization but fail to effectively decouple the semantics of samples, limiting the zero-shot performance of DST.In this paper, we present a simple and effective "divide, conquer and combine" solution, which explicitly disentangles the semantics of seen data, and leverages the performance and robustness with the mixtureof-experts mechanism.Specifically, we divide the seen data into semantically independent subsets and train corresponding experts, the newly unseen samples are mapped and inferred with mixture-of-experts with our designed ensemble inference.Extensive experiments on MultiWOZ2.1 upon the T5-Adapter show our schema significantly and consistently improves the zero-shot performance, achieving the SOTA on settings without external knowledge, with only 10M trainable parameters 1 .
Masakhane-Afrisenti at SemEval-2023 Task 12: Sentiment Analysis using Afro-centric Language Models and Adapters for Low-resource African Languages

In this paper, we describe our submission for the AfriSenti-SemEval Shared Task 12 of SemEval-2023 .The task aims to perform monolingual sentiment classification (sub-task A) for 12 African languages, multilingual sentiment classification (sub-task B), and zero-shot sentiment classification (task C).For sub-task A, we conducted experiments using classical machine learning classifiers, Afro-centric language models, and language-specific models.For task B, we fine-tuned multilingual pretrained language models that support many of the languages in the task.For task C, we made use of a parameter-efficient Adapter approach that leverages monolingual texts in the target language for effective zero-shot transfer.Our findings suggest that using pre-trained Afrocentric language models improves performance for low-resource African languages.We also ran experiments using adapters for zero-shot tasks, and the results suggest that we can obtain promising results by using adapters with limited resources.
AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing

Developing monolingual large Pre-trained Language Models (PLMs) is shown to be very successful in handling different tasks in Natural Language Processing (NLP).In this work, we present AraMUS, the largest Arabic PLM with 11B parameters trained on 529GB of highquality Arabic textual data.AraMUS achieves state-of-the-art performances on a diverse set of Arabic classification and generative tasks.Moreover, AraMUS shows impressive few-shot learning abilities compared with the best existing Arabic PLMs.
Topic-Guided Self-Introduction Generation for Social Media Users

Millions of users are active on social media.To allow users to better showcase themselves and network with others, we explore the autogeneration of social media self-introduction, a short sentence outlining a user's personal interests.While most prior work profiles users with tags (e.g., ages), we investigate sentencelevel self-introductions to provide a more natural and engaging way for users to know each other.Here we exploit a user's tweeting history to generate their self-introduction.The task is non-trivial because the history content may be lengthy, noisy, and exhibit various personal interests.To address this challenge, we propose a novel unified topic-guided encoderdecoder (UTGED) framework; it models latent topics to reflect salient user interest, whose topic mixture then guides encoding a user's history and topic words control decoding their self-introduction.For experiments, we collect a large-scale Twitter dataset, and extensive results show the superiority of our UTGED to the advanced encoder-decoder models without topic modeling.
Multiple Evidence Combination for Fact-Checking of Health-Related Information

Fact-checking of health-related claims has become necessary in this digital age, where any information posted online is easily available to everyone.The most effective way to verify such claims is by using evidences obtained from reliable sources of medical knowledge, such as PubMed.Recent advances in the field of NLP have helped automate such fact-checking tasks.In this work, we propose a domainspecific BERT-based model using a transfer learning approach for the task of predicting the veracity of claim-evidence pairs for the verification of health-related facts.We also improvise on a method to combine multiple evidences retrieved for a single claim, taking into consideration conflicting evidences as well.We also show how our model can be exploited when labelled data is available and how backtranslation can be used to augment data when there is data scarcity.
Follow the leader(board) with confidence: Estimating p-values from a single test set with item and response variance

Among the problems with leaderboard culture in NLP has been the widespread lack of confidence estimation in reported results.In this work, we present a framework and simulator for estimating p-values for comparisons between the results of two systems, in order to understand the confidence that one is actually better (i.e.ranked higher) than the other.What has made this difficult in the past is that each system must itself be evaluated by comparison to a gold standard.We define a null hypothesis that each system's metric scores are drawn from the same distribution, using variance found naturally (though rarely reported) in test set items and individual labels on an item (responses) to produce the metric distributions.We create a test set that evenly mixes the responses of the two systems under the assumption the null hypothesis is true.Exploring how to best estimate the true p-value from a single test set under different metrics, tests, and sampling methods, we find that the presence of response variance (from multiple raters or multiple model versions) has a profound impact on p-value estimates for model comparison, and that choice of metric and sampling method is critical to providing statistical guarantees on model comparisons.
Contrastive Bootstrapping for Label Refinement

Traditional text classification typically categorizes texts into pre-defined coarse-grained classes, from which the produced models cannot handle the real-world scenario where finer categories emerge periodically for accurate services.In this work, we investigate the setting where fine-grained classification is done only using the annotation of coarse-grained categories and the coarse-to-fine mapping.We propose a lightweight contrastive clustering-based bootstrapping method to iteratively refine the labels of passages.During clustering, it pulls away negative passage-prototype pairs under the guidance of the mapping from both global and local perspectives.Experiments on NYT and 20News show that our method outperforms the state-of-the-art methods by a large margin. 1
A Dataset for Physical and Abstract Plausibility and Sources of Human Disagreement

We present a novel dataset for physical and abstract plausibility of events in English.Based on naturally occurring sentences extracted from Wikipedia, we infiltrate degrees of abstractness, and automatically generate perturbed pseudoimplausible events.We annotate a filtered and balanced subset for plausibility using crowdsourcing, and perform extensive cleansing to ensure annotation quality.In-depth quantitative analyses indicate that annotators favor plausibility over implausibility and disagree more on implausible events.Furthermore, our plausibility dataset is the first to capture abstractness in events to the same extent as concreteness, and we find that event abstractness has an impact on plausibility ratings: more concrete event participants trigger a perception of implausibility.
Model Intrinsic Features of Fine-tuning based Text Summarization Models for Factual Consistency

In this study, we analyze the model intrinsic features of a summarization model by varying the fine-tuning objectives and datasets.We finetune BART models combining three fine-tuning objectives (negative log-likelihood, unlikelihood, and contrastive loss) and two datasets (CNN/DailyMail and XSum) and provide shuffled or aligned documents to observe changes in the model predictions and intrinsic features.We find that (i) the inductive bias for factual consistency during the fine-tuning procedure depends on both the objectives and datasets, and (ii) summarization models with relatively low factual consistency are more likely to model summaries that are not conditional to the documents.We demonstrate that splitting data based on the unconditional and conditional summary modeling difficulty affects the factual consistency and intrinsic features of the summarization models.Our experimental results highlight the importance of studying the inductive bias during fine-tuning for factual consistency.
What about “em”? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns

As 3rd-person pronoun usage shifts to include novel forms, e.g., neopronouns, we need more research on identity-inclusive NLP.Exclusion is particularly harmful in one of the most popular NLP applications, machine translation (MT).Wrong pronoun translations can discriminate against marginalized groups, e.g., non-binary individuals (Dev et al., 2021).In this "reality check", we study how three commercial MT systems translate 3rd-person pronouns.Concretely, we compare the translations of gendered vs. gender-neutral pronouns from English to five other languages (Danish, Farsi, French, German, Italian), and vice versa, from Danish to English.Our error analysis shows that the presence of a gender-neutral pronoun often leads to grammatical and semantic translation errors.Similarly, gender neutrality is often not preserved.By surveying the opinions of affected native speakers from diverse languages, we provide recommendations to address the issue in future MT research.
EmbedTextNet: Dimension Reduction with Weighted Reconstruction and Correlation Losses for Efficient Text Embedding

The size of embeddings generated by large language models can negatively affect system latency and model size in certain downstream practical applications (e.g.KNN search).In this work, we propose EmbedTextNet, a light add-on network that can be appended to an arbitrary language model to generate a compact embedding without requiring any changes in its architecture or training procedure.Specifically, we use a correlation penalty added to the weighted reconstruction loss that better captures the informative features in the text embeddings, which improves the efficiency of the language models.We evaluated Embed-TextNet on three different downstream tasks: text similarity, language modelling, and text retrieval.Empirical results on diverse benchmark datasets demonstrate the effectiveness and superiority of EmbedTextNet compared to state-ofart methodologies in recent works, especially in extremely low dimensional embedding sizes.The developed code for reproducibility is included in the supplementary material.
Defending against Insertion-based Textual Backdoor Attacks via Attribution

Textual backdoor attack, as a novel attack model, has been shown to be effective in adding a backdoor to the model during training.Defending against such backdoor attacks has become urgent and important.In this paper, we propose AttDef, an efficient attribution-based pipeline to defend against two insertion-based poisoning attacks, BadNL and InSent Specifically, we regard the tokens with larger attribution scores as potential triggers since larger attribution words contribute more to the false prediction results and therefore are more likely to be poison triggers.Additionally, we further utilize an external pre-trained language model to distinguish whether input is poisoned or not.We show that our proposed method can generalize sufficiently well in two common attack scenarios (poisoning training data and testing data), which consistently improves previous methods.For instance, AttDef can successfully mitigate both attacks with an average accuracy of 79.97% (56.59%↑) and 48.34% (3.99%↑) under pre-training and post-training attack defense respectively, achieving the new state-of-the-art performance on prediction recovery over four benchmark datasets. 1
Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs

Abishek Komma, Nagesh Panyam Chandrasekarasastry, Timothy Leffel, Anuj Goyal, Angeliki Metallinou, Spyros Matsoukas, Aram Galstyan. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track). 2023.
XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE

The recent introduction of large-scale datasets for the WiC (Word in Context) task enables the creation of more reliable and meaningful contextualized word embeddings.However, most of the approaches to the WiC task use crossencoders, which prevent the possibility of deriving comparable word embeddings.In this work, we introduce XL-LEXEME, a Lexical Semantic Change Detection model.XL-LEXEME extends SBERT, highlighting the target word in the sentence.We evaluate XL-LEXEME on the multilingual benchmarks for SemEval-2020 Task 1 -Lexical Semantic Change (LSC) Detection and the RuShiftEval shared task involving five languages: English, German, Swedish, Latin, and Russian.XL-LEXEME outperforms the state-of-the-art in English, German and Swedish with statistically significant differences from the baseline results and obtains state-of-the-art performance in the RuShiftEval shared task.
Ebhaam at SemEval-2023 Task 1: A CLIP-Based Approach for Comparing Cross-modality and Unimodality in Visual Word Sense Disambiguation

Zeinab Taghavi, Parsa Haghighi Naeini, Mohammad Ali Sadraei Javaheri, Soroush Gooran, Ehsaneddin Asgari, Hamid Reza Rabiee, Hossein Sameti. Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023). 2023.