Venue: EMNLP 2023

# The Skipped Beat:

A Study of Sociopragmatic Understanding in LLMs for 64 Languages

Chiyu Zhang\({}^{\xi}\)   Khai Duy Doan\({}^{\lambda,*}\)   Qisheng Liao\({}^{\lambda,*}\)   Muhammad Abdul-Mageed\({}^{\xi,\lambda}\)

\({}^{\xi}\)Deep Learning & Natural Language Processing Group, The University of British Columbia

\({}^{\lambda}\)Department of Natural Language Processing & Department of Machine Learning, MBZUAI

{chiyuzh@mail,muhammad.mageed@}.ubc.ca,

{duy.doan,qisheng.liao}@mbzuai.ac.ae

 Equal contribution

###### Abstract

Instruction tuned large language models (LLMs), such as ChatGPT, demonstrate remarkable performance in a wide range of tasks. Despite numerous recent studies that examine the performance of instruction-tuned LLMs on various NLP benchmarks, there remains a lack of comprehensive investigation into their ability to understand cross-lingual sociopragmatic meaning (SM), i.e., meaning embedded within social and interactive contexts. This deficiency arises partly from SM not being adequately represented in any of the existing benchmarks. To address this gap, we present SPARROW, an extensive multi-lingual benchmark specifically designed for SM understanding. SPARROW comprises \(169\) datasets covering \(13\) task types across six primary categories (e.g., anti-social language detection, emotion recognition). SPARROW datasets encompass \(64\) different languages originating from \(12\) language families representing \(16\) writing scripts. We evaluate the performance of various multilingual pretrained language models (e.g., mT5) and instruction-tuned LLMs (e.g., BLOOMZ, ChatGPT) on SPARROW through fine-tuning, zero-shot, and/or few-shot learning. Our comprehensive analysis reveals that existing open-source instruction tuned LLMs still struggle to understand SM across various languages, performing close to a random baseline in some cases. We also find that although ChatGPT outperforms many LLMs, it still falls behind task-specific finetuned models with a gap of \(12.19\) SPARROW score. Our benchmark is available at: [https://github.com/UBC-NLP/SPARROW](https://github.com/UBC-NLP/SPARROW)

## 1 Introduction

Multilingual LLMs have recently transformed NLP, due to their powerful capabilities on a wide range of tasks Xue et al. (2021); Scao et al. (2022). Methods such instruction tuning and reinforcement learning from human feedback (RLHF) Ouyang et al. (2022) have further enhanced the zero-shot generalizability of these models. Notably, ChatGPT exhibits impressive capabilities in this regard. Human language, however, is intrinsically ambiguous and far from solved. In fact, some forms of meaning are deeply embedded in social interactions. We collectively refer to this type of meaning as _sociopragmatic meaning_ (SM). To appreciate SM, consider how the meaning of an utterance in social interaction (e.g., on social media) can be highly subtle and how it incorporates both the social variation related to language users (from a sociolinguistics perspective Tagliamonte (2015) and their communicative intentions (from a pragmatics perspective) Boxer and Cortes-Conde (2021). Although SM is quite established within linguistics, NLP systems still struggle with this type of meaning that is intertwined in social and interactive contexts Zhang and Abdul-Mageed (2022). The extent to which instruction tuned models such as ChatGPT can capture SM across languages re

Figure 1: Comparison of SM benchmarks with leaderboards. The bubble size indicates the number of datasets. Previous works: TweetEval Barbieri et al. (2020), UMASB Barbieri et al. (2022), Nusax Winata et al. (2022), UTCC Risch et al. (2021), NajaSenti Muhammad et al. (2022), AfriSenti Muhamad et al. (2023a), SentiEval Zhang et al. (2023b).



mains largely unclear as these models are yet to be evaluated on appropriate datasets under standardized conditions easy to replicate.

To facilitate evaluation of LLMs and enhance fairness of model comparisons and reproducibility, early work introduces evaluation benchmarks. Most existing benchmarks, however, focus on the monolingual setting. These include GLUE (Wang et al., 2019), SentEval (Conneau and Kiela, 2018), and TweetEval (Barbieri et al., 2020) for English, ARLUE (Abdul-Mageed et al., 2021) for Arabic, CLUE (Xu et al., 2020a) for Chinese, and IndoNLU (Wilie et al., 2020) for Indonesian. Although XTREME (Hu et al., 2020) and XGLUE (Liang et al., 2020) introduce multilingual benchmarks, they only include a few SM tasks for a limited number of languages. They are also limited to standard language use (e.g., Wikipedia). Barbieri et al. (2022) propose a multilingual sentiment analysis benchmark (UMSAB), but it solely contains tweet sentiment analysis datasets in only eight languages. As such, absence of a unified, diverse, and comprehensive benchmark and a fragmented evaluation landscape hamper NLP work for cross-lingual SM.

Another challenge for SM research is the issue of _data inaccessible_(Assenmacher et al., 2022). Although many studies release the IDs of posts (e.g., tweets), substantial amounts of these social posts become inaccessible over time due to deletion, etc. (Zhang et al., 2022). In our benchmark, we attempt to re-collect text contents of \(25\) datasets by using their IDs but can only retrieve \(58\%\) samples on average (see Table 8 in Appendix). This data decay also hinders fair comparisons in NLP for SM research. This issue has already become worse as corporations such as Twitter tighten access to their API, making it even harder to collect historical data. To address this bottleneck, we introduce a massively multilingual SM evaluation benchmark, dubbed _SPAROW_, that comprises \(169\) datasets covering \(64\) languages from \(12\) language families, \(16\) types of scripts, across diverse online platforms (e.g., Twitter, YouTube, and Weibo). We then perform an extensive evaluation of ChatGPT, comparing it to \(13\) other models ranging in size between \(110\)M-7B parameters. Our evaluations allow us to answer multiple questions related to how it is that LLMs fare across languages on SM. To facilitate future comparisons, we also design a modular, interactive leaderboard on top of our new benchmark.

To summarize, the contributions of this paper are as follows: **(1)** We collect, uniformize, and responsibly release massively multilingual SM datasets in a _newbenchmark_; **(2)** Our SPARROW benchmark is essentially an archive of SM datasets that alleviates the serious issue of _data decay_; **(3)** We evaluate a wide range of models on our SPARROW benchmark via fine-tuning SoTA encoder-only pretrained language models and zero-shot learning of a number of generative models, including instruction tuned models (e.g., BLOOMZ) as well as ChatGPT; and **(4)** We establish standard settings for future research in this area across a large number of languages and tasks through a _public leaderboard_.

## 2 Related Work

Evaluation of LLMs.There have been many attempts to evaluate ChatGPT and instruction tuned LLMs. Qin et al. (2023); Laskar et al. (2023); Zhong et al. (2023); Wu et al. (2023) utilize existing English evaluation benchmarks, such as GLUE (Wang et al., 2019) and BigBench (Srivastava et al., 2022), to evaluate LLMs' capacities on various NLP tasks. These studies find that although ChatGPT performs less effectively than the models finetuned specifically for each task, it demonstrates superior capabilities compared to other instruction tuned LLMs (e.g., FLAN Chung et al., 2022)). Ahuja et al. (2023); Bang et al. (2023); Lakkar et al. (2023); Lai et al. (2023); Huang et al. (2023) evaluate LLMs on more diverse languages using existing multilingual benchmarks (e.g., XNLI, PAWS-X, XLSum) involving monolingual NLP tasks and crosslinkingal tasks (e.g., machine translation). Their findings point to a large gap in performance of instruction tuned LLMs and ChatGPT, especially on low-resource languages and those with non-Latin scripts.

\begin{table}
\begin{tabular}{l c r r r r r} \hline \hline
**Studies** & **Lang.** & **Tasks** & **SM Tasks** & **Dataset** & **Models** & **LeaderBrd** \\ \hline Zhong et al. (2023) & en & 5 & 1 & 8 & 5 & ✗ \\ Qin et al. (2023) & en & 7 & 1 & 20 & 29 & ✗ \\ Ahuja et al. (2023) & 70 & 10 & 3 & 16 & 11 & ✗ \\ Lakkar et al. (2023) & 12 & 12 & 2 & 140 & 27 & ✗ \\ Bang et al. (2023) & 8 & 8 & 1 & 23 & 3 & ✗ \\ Lai et al. (2023) & 37 & 7 & 0 & 8 & 7 & ✗ \\ Das et al. (2023) & 11 & 2 & 2 & 2 & 1 & ✗ \\ Wang et al. (2023) & en & 5 & 5 & 18 & 3 & ✗ \\ Zhang et al. (2023) & en & 13 & 13 & 26 & 5 & ✓ \\ Ziems et al. (2023) & en & 24 & 18 & 24 & 13 & ✗ \\ \hline Ours & 64 & 13 & 13 & 169 & 14 & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 1: Our work in comparison.

 

SM is still not adequately represented in existing benchmarks, hindering comprehensive evaluations on more languages. As we summarize in Table 1, benchmarks used for listed evaluations only include a few SM tasks focusing on sentiment analysis. Wang et al. (2023); Zhang et al. (2023b) investigate LLMs on a number of SM tasks (e.g., offensive language detection), but only on English. Ziems et al. (2023) investigate Chat-GPT performance on a range of computational social science tasks covering subjects such as sociology, psychology, and linguistics, but they again focus only on English. Das et al. (2023) extend evaluation of ChatGPT on hate speech detection to 11 languages. Compared to these works, our objective is to investigate more diverse SM tasks on a massively multilingual setting.

**Sociopragmatic Meaning Benchmarks.** Many previous works introduce unified benchmarks such as GLUE (Wang et al., 2019), SentEval (Conneau and Kiela, 2018), XTREME (Hu et al., 2020), and XGLUE (Liang et al., 2020). These benchmarks include a wide range of NLP tasks, but comprise a sole SM task (i.e., sentiment analysis). Some recent studies started to construct benchmarks focusing on SM: Barbieri et al. (2020) introduce TweetEval benchmark that contains seven English datasets of six SM tasks; Zhang et al. (2023b) develop SentiEval that involves \(26\) English datasets of \(13\) sentiment-related tasks. Beyond English, NusaX (Winata et al., 2022), NaijaSenti (Muhammad et al., 2022), and AfriSenti (Muhammad et al., 2023a) propose benchmarks for sentiment analysis with eight Indonesian languages, four African languages, and \(14\) African languages, respectively. UMASAB introduced by Barbieri et al. (2022) contains \(11\) sentiment analysis datasets in \(11\) languages. For detecting antisocial online comments, Risch et al. (2021) introduces a toxic comment collection that contains \(43\) datasets of six antisocial detection tasks in \(14\) languages. Compared to these works, our SPARROW benchmark includes significantly more SM tasks and languages, from more diverse sources (refer to Figure 1 for a comparison).

## 3 SPARROW Benchmark

In this section, we describe clusters of tasks in our benchmark as well as our preprocessing and unification. SPARROW consists of \(13\) types of tasks in six main categories. It contains \(169\) datasets from diverse online platforms and covers a wide period of time (\(1986\)-\(2022\)). We group different tasks in our benchmark by what we perceive to be an affinity between these tasks. For example, we group tasks of hate speech, offensive language, and dangerous language detection as anti-social language detection. Meanwhile, we keep particular tasks (such as sentiment analysis and emotion recognition) distinct due to the popularity of these tasks and since there are multiple datasets representing each of them. Table 2 summarizes statistics of SPARROW. We now briefly introduce our task clusters. We provide more information about languages in SPARROW in Table 7 of the Appendix. We also provide detailed descriptions with full citations of all our datasets in Tables 9, 10, 11, 12, 13, and 14 in Appendix.

### Task Clusters

Antisocial Language Detection.The proliferation of antisocial language (e.g., hate speech) toxifies public discourse, incites violence, and undermines civil society (Sap et al., 2019; Vidgen and Derczynski, 2020). Antisocial language detection is thus a useful task. We include under the umbrella of antisocial language the following: **(1)** aggressive language (Kumar et al., 2018), **(2)** dangerous language (Alshehri et al., 2020), **(3)** hate speech (e.g., Waseem and Hovy (2016); Deng et al. (2022)), **(4)** offensive language (e.g., Mubarak et al. (2020); Kralj Novak et al. (2021)), **(5)** offense type identification (e.g., Zampieri et al. (2019)), and **(6)** offense target identification (e.g., Ousidhoum et al. (2019); Jeong et al. (2022)).

\begin{table}
\begin{tabular}{l l r r r r} \hline \hline  & \multicolumn{1}{c}{**Tasks**} & \multicolumn{1}{c}{**Dataset**} & \multicolumn{1}{c}{**Lang.**} & \multicolumn{1}{c}{**LF**} & \multicolumn{1}{c}{**Ser**} \\ \cline{3-6}  & Aggressive & 1 & 1 & 1 & 1 \\  & Dungeons & 1 & 1 & 1 & 1 \\  & Hate & 16 & 11 & 6 & 5 \\  & Offence & 7 & 6 & 3 & 3 \\  & HO-Group & 3 & 3 & 2 & 3 \\  & HO-Target & 8 & 8 & 4 & 7 \\  & Aalitical & 36 & 20 & 7 & 10 \\ \hline \multirow{2}{*}{**Emotion**} & \multirow{2}{*}{26} & \multirow{2}{*}{17} & \multirow{2}{*}{5} \\ \cline{2-6}  & & & & & \\ \cline{2-6}  & & & & & \\ \hline \multirow{4}{*}{**Humor**} & \multirow{2}{*}{4} & \multirow{2}{*}{1} & \multirow{2}{*}{2} \\ \cline{2-6}  & Irony & 9 & 7 & 3 & 3 \\ \cline{1-1}  & Sacramm & 10 & 4 & 3 & 3 \\ \cline{1-1}  & Irony-Type & 1 & 1 & 1 & 1 \\ \cline{1-1}  & Iony\&Scarcasm & 20 & 8 & 3 & 3 \\ \cline{1-1} \cline{2-6}  & Sentiment & 77 & 58 & 10 & 15 \\ \hline
**Subjectivity** & 6 & 5 & 2 & 2 \\ \hline
**SPARROW** & 169 & 64 & 12 & 16 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Summary of datasets in SPARROW. **Lang:** number of languages, **LF:** number of language families, **Ser:** number of scripts.

 

Emotion Recognition.Emotion affects our decision-making as well as mental and physical health (Abdul-Mageed and Ungar, 2017). SPARROW includes \(26\) emotion datasets in \(17\) languages (e.g., Kajava (2018); Bianchi et al. (2021)).

Humor Detection.Humor is a type of figurative language which induces amusing effects, such as laughter or well-being sensations. We include four humor detection datasets in four languages (e.g., Blinov et al. (2019); Meaney et al. (2021)).

Irony & Sarcasm Detection.Irony and sarcasm also involve figurative language. An ironic/sarcastic expression intentionally uses diametric language to signify implied meaning. We include **(1)** nine irony detection datasets in seven languages (e.g., Xiang et al. (2020)), **(2)** ten sarcasm detection datasets in four languages (e.g., Walker et al. (2012)), and **(3)** an irony type identification dataset (Van Hee et al., 2018).

Subjectivity and Sentiment Analysis.Subjectivity analysis aims to understand the opinions, feelings, judgments, and speculations expressed via language Abdul-Mageed et al. (2014). Our benchmark includes six subjectivity analysis datasets in five different languages (e.g., Pang and Lee (2004); Priban and Steinberger (2022)). Subjectivity incorporates sentiment. Sentiment analysis Poria et al. (2020) is one of the most popular tasks in SM understanding where the objective is to identify the underlying sentiment of a given text. Our benchmark contains \(77\) sentiment analysis datasets in \(58\) languages (e.g., Pang and Lee (2005); Marreddy et al. (2022)).

### Preprocessing, Splits, and Metrics

We apply light normalization on all the samples by converting user mentions and hyperlinks to 'USER' and 'URL', respectively. We standardize label names for consistency across datasets without reassigning nor aggregating the original labels of the datasets. For instance, in certain sentiment analysis datasets, we map _'0'_ and _'1'_ to _'Negative'_ and _'Positive'_ respectively. Regarding data splits, if the dataset already has Train, Dev, and Test sets, we maintain the same splits. If the original dataset does not include a Dev set, we randomly select \(10\%\) of training data to be a Dev set. In cases without pre-defined splits, we use an \(80\%\) Train, \(10\%\) Dev, and \(10\%\) Test random split. For computing constraints, we also prepare a smaller Test set for each dataset by randomly sampling \(500\) samples from Test (if its size exceeds \(500\)). We refer to this smaller test set as Test-S.

We evaluate on each dataset using its original metric as Tables 9, 10, 11, 12, 13, and 14 in Appendix summarize.1 We report the performance on individual datasets, aggregate datasets into \(13\) tasks, and report an average score over each task. Moreover, we introduce a metric for each main category, calculated as the average of dataset-specific metrics within that category. Inspired by previous evaluation benchmarks like GLUE Wang et al. (2019), we define a global metric called _SPARROW score_, which represents the unweighted average of all dataset-specific metrics. The SPARROW score provides an overall indication of performance on SM tasks.

Footnote 1: We select the macro-average F\({}_{1}\) score as the main metric if the original paper utilizes more than one metric.

## 4 Evaluation Methods

### Finetuning on Encoder-Only Models

We evaluate the following Transformer-encoder-based multilingual models on SPARROW: **(1) Multilingual-BERT** (mBERT) Devlin et al. (2019) Base, \(110\)M parameters), **(2) XLM-RoBERTBase** (XLM-R) Conneau et al. (2020) (\(270\)M parameters), **(3) Bernice**DeLucia et al. (2022), a \(270\)M-parameter model trained with \(2.5\)B tweets in \(66\) languages, and **(4) InfoDCL**Zhang et al. (2023a), a SoTA for SM understanding, which further trains XLM-R with \(100\)M tweets in \(66\) languages with contrastive learning. More details about all models are in Appendix B.

### Zero- and Few-Shot on LLMs

We investigate zero-shot performance on a wide range of generative models, including _pre-trained_**generative models**: **(1)****BLOOM**Scao et al. (2022), **(2)****mT5**Xue et al. (2021), **(3)****LLaMA**Touvron et al. (2023), _instruction tuned_**models**: **(4)****BLOOMZ**Scao et al. (2022), a BLOOM-initialized model tuned with multilingual xP3 corpus, **(5)****BLOOMZ-P3**Muennighoff et al. (2022), a BLOOM-initialized model tuned with English-only P3 corpus, **(6)****BLOOM-Bactrian**Li et al. (2023), a BLOOM-initialized model tuned with \(3.4\)M instruction-following samples in \(52\) languages, **(7)****mT0**Muennighoff et al. (2022), an mT5 model tuned with xP3 corpus, **(8)****Alpaca**Taori et al. (2023), a 

LLaMA-initialized model tuned with \(52\)K English instruction-following samples, **(9) Vicuna**[14], a LLaMA-initialized model on \(70\)K conversational data, and **(10) ChatGPT**, for which we use the gpt-3.5-turbo-0301 version via OpenAI API.2 We use 7B-size version of BLOOM- and LLaMA-based models and \(4\)B-size version of mT5-based models. We also evaluate six open-source LLMs (i.e, BLOOM, BLOOMZ-P3, mT5, mT0, LLaMA, and Vicuna) via few-shot in-context learning.

Footnote 2: In rest of this paper, ChatGPT refers to gpt–3.5-turbo-0301.

## 5 Experiments

### Implementation

Finetuning.To keep computation reasonable, we randomly select \(45\) datasets for hyperparameter tuning and only tune the learning rate of each model.3 For all experiments, we finetune a pretrained model with an arbitrary batch size of \(32\) and sequence length of \(128\) tokens. Each model is finetuned on the full Train set of each dataset for \(20\) epochs (with patience \(=5\)) based on performance on Dev set. We run each experiment _three_ times with different random seeds and identify the best model on Dev in each run. We report the average performance on Test-S over three runs.4

Zero-shot Evaluation.We perform a zero-shot evaluation on SPARROW for BLOOM-, mT5-, and LLaMA-based models using language model evaluation harness (lm-evaluation-harness Gao et al. (2021)).5 While we do not tailor prompts specifically for each model, customized prompts are employed for each set of tasks. These prompts follow the structure of question-and-answer tasks, where we present sample content alongside a task-specific question, as shown in Figure 2. The prompts are summarized in Appendix Table 15. We then instruct the model to generate an answer based on the provided option labels. Each option label represents a potential answer, and we calculate the log-likelihood for each candidate. The prediction with the highest log-likelihood is chosen as the model's final prediction. For the evaluation of ChatGPT, we draw inspiration from previous practices for prompt design Ziems et al. (2023), and incorporate additional instructions to guide its generation of the desired labels. As shown in Figure 2, we provide an instruction that compels ChatGPT to select a single label for the given input text without providing any additional explanation. We set the temperature to \(0\) to generate _deterministic and reproducible results_ from ChatGPT. For a few instances, we observe that ChatGPT is unable to provide a direct answer. In these cases, we randomly assign a false label to each sample. In addition, we also use machine translation to translate English prompts and label names to the corresponding language of each dataset.6

Footnote 6: We use Google Translate for most languages. NLLB model is used to translate the languages of ace, ban, bjn, bug, and min because Google Translate does not cover these. The prompts of pcm datasets are translated by a native speaker.

Few-shot Evaluation.We utilize lm-evaluation-harness tool with the same prompts employed in zero-shot evaluation to explore the few-shot incontext learning abilities of open-source LLMs. Before the actual test examples, we prepend \(m\) examples from the Train set. Each example consists

Figure 2: Examples of prompts used for zero-shot evaluation with lm-evaluation-harness (yellow ) and ChatGPT (green ). We use an English prompt (Figures a, c) and machine translated the prompt in the corresponding language (Figures b, d), repectively. The prompts construct each task as question-and-answer tasks. The actual input sample is in blue, and the label options are in red.

 

of an input text, task-specific instruction, and the corresponding answer. We set \(m\) to either 3 or 5.

### Results

We present the aggregated performance of TestS on each task and main category, respectively, in Table 3. We also present test results on all datasets and compare to dataset-specific SoTA performance in Tables 17, 18, 19, 20, 21, and 22 in Appendix.

**(1) How is the overall performance over different models?**_All the fully finetuned models surpass the zero-shot generative models as well as ChatGPT, as shown in Table 3._ The most superior among the finetuned models is InfoDCL, which achieves a SPARROW score of \(71.60\) and outperforms ChatGPT with \(11.56\) points SPARROW score. On the other hand, the open-source models (i.e., BLOOM, mT5 and LLaMA) still significantly lag behind on multilingual SM understanding with performance close to a random baseline. Meanwhile, the instruction tuned multilingual LLMs (BLOOMZ and mT0) only slightly perform better than the random baseline.

**(2) Can instruction tuning enhance LLMs' ability on SM understanding?**_Yes, but it depends on the instruction training data._ Following instruction tuning on the English-only P3 dataset, BLOOMZ-P3 demonstrates an improvement of \(7.76\) SPARROW score compared to BLOOM. Also, BLOOMZ improves \(5.85\) points over BLOOM (but falls short of BLOOMZ-P3). MT0 also outperforms mT5. However, there remains a substantial gap between all instruction tuned models and finetuned models. BLOOM-Bactrian performs worse than BLOOMZ and BLOOMZ-P3, which are instruction tuned with NLP tasks. This indicates that the general purpose instruction-response dataset is not very useful for SM understanding.

To further probe how instruction tuning improves BLOOM-based models, we compare BLOOM with BLOOMZ-P3 and BLOOMZ in terms of individual tasks, finding sentiment analysis to exhibit the most significant improvement. BLOOMZ-P3 and BLOOMZ achieve a sentiment score improvement of \(16.37\) and \(12.36\), respectively, based on average calculation across \(77\) sentiment analysis datasets. However, BLOOM-Bactrian obtains an improvement of only \(1.79\) sentiment score, perhaps implying that the Backrian instruction-response data is not all that useful for some SM tasks. After tuning mT5 on xP3 dataset, mT0 also experiences a \(13.88\) improvement in the sentiment score. These may be stemming from inclusion of five English sentiment analysis datasets in both P3 and xP3 during the training phase. For example, we observe that BLOOM, BLOOMZ, BLOOMZ-P3, mT5, and mT0 obtain an accuracy of \(56.4\), \(92.2\), \(93.00\), \(49.00\), and \(76.8\) on SentengSoc (not included in either xP3 or P3), respectively and that BLOOM-Bactrian still performs poorly (accuracy\(=53.60\)) after instruction tuning. Again, these results indicate that it is still important to include task-related datasets in the instruction tuning stage.

**(3) How do LLMs perform across different SM tasks?**_They are inferior at humor and antisocial language detection while being relatively better at sentiment and emotion recognition tasks._ BLOOMZ-P3, BLOOMZ, and mT0 exhibit considerable enhancements (\(>5\) points) on sentiment and emotion when compared to their respective initialization models. On the other hand, we find that instruction tuned models perform significantly worse on aggressive language detection and humor detection tasks. BLOOMZ-P3, BLOOMZ, BLOOM-Bactrian, and mT0 all incur a loss of more than \(5\) points on these two tasks. Upon investigating the predictions of instruction tuned models, we find that they tend to assign negative labels (i.e., non-aggressive or non-humor) which results in many false negative predictions. For a concrete example, we show that BLOOMZ-P3 predict most samples as non-humor in Figure 3a shows.

ChatGPT outperforms the open-source LLMs on all tasks except dangerous language detection. Comparing ChatGPT to InfoDCL, we find gaps favoring InfoDCL in subjectivity analysis (a difference of \(9.47\)), emotion recognition (a difference of \(10.68\)), and irony & sarcasm detection (a difference of \(10.70\)). ChatGPT also largely lags behind InfoDCL in humor detection (a difference of \(15.40\)) and antisocial language detection (a difference of \(14.06\)). As the example shows in Figure 3b, ChatGPT makes more false positive errors (classifies non-hateful as hateful).

**(4) How do LLMs perform across different languages?** We now examine the impact of instruction finetuning on the model's language-wise performance. We categorize the performance of each 

dataset based on language and calculate the average language scores across all datasets within a language. Since each language contains different tasks and datasets, a direct comparison across languages is not feasible. Therefore, we compare the relative performance between different models for each language. By comparing the instruction tuned models to their initial models, we observe that _most languages experience improvement_. However, we also observe a significant decline in performance for the Amharic (amh) dataset among these models. Specifically, BLOOMZ-P3, BLOOMZ, and mT0 experience a deterioration of \(36.07\), \(24.99\), and \(26.12\) points, respectively, compared to their respective initial models. We hypothesize that this deterioration can be attributed to catastrophic forgetting after instruction tuning, where Amharic was not included in the training set and does not share the writing scripts with the other included languages. Similarly, the Filipino (fil) tasks exhibit an average decline of approximately \(11\) points on both BLOOMZ-P3 and BLOOMZ, as Filipino is not included in the xP3 dataset. Although Hindi is included in the xP3 dataset, the three instruction tuned models still show a decline in performance. Upon examining the individual performance of Hindi datasets, we find that the major deteriorations occur in the aggressive language detection and humor detection tasks, while the emotion recognition and sentiment analysis tasks show improvement. The instruction-response data for training Alpaca and Vicuna consist solely of English language. Therefore, we compare the performance of Alpaca and Vicuna to that of LLaMA using both English and non-English datasets. We observe that Alpaca and Vicuna outperform LLaMA when evaluated on English datasets, achieving

\begin{table}
\begin{tabular}{l l|c c c c c c c c c c c c c c c c c c c c c c c c c} \hline \multicolumn{15}{c}{**Land**} & \multicolumn{8}{c}{**Random**} & \multicolumn{8}{c}{**InfoDCI**} & \multicolumn{8}{c}{**BMZ-P3**} & \multicolumn{8}{c}{**mT0**} & \multicolumn{8}{c}{**Viciuna**} & \multicolumn{8}{c}{**CG**} & \multicolumn{8}{c}{**CG-MT**} \\ \cline{3-15} \multicolumn{15}{c}{} & **math** & 37.95 & **65.68** & 16.05 & 22.49 & 2.99 & 20.62 & **46.82** & \\ bug & 30.77 & **71.55** & 34.60 & 18.27 & 12.90 & 34.63 & 30.86 & & & & & & & & & & & & & & \\ ell & 41.24 & **79.13** & 46.71 & 45.47 & 48.21 & 60.94 & 34.98 & eng & 37.90 & **54.78** & 43.32 & 39.32 & 39.75 & 66.51 & & & & & & & & & & & & \\ fil & 52.37 & **79.01** & 34.47 & 34.47 & 34.47 & 60.13 & 66.67 & heb & 47.60 & **95.80** & 71.20 & 76.60 & 40.80 & 84.20 & 57.40 & & & & & & & & & & & & \\ bin & 35.24 & **67.55** & 28.92 & 26.20 & 29.06 & 52.63 & 48.30 & & & & & & & & & & & & & & & & & & \\ mal & 31.68 & **82.70** & 43.84 & 41.65 & 24.85 & & & & & & & & & & & & & & & & & & & & \\ \end{tabular}
\end{table}
Table 4: Language-wise model performance for sample languages. The complete results are in Table 23 in Appendix. Best performance in each language is **bold**, and the second best is in [green highlight]. The red font denotes a performance lower than the random baseline.

Figure 3: Confusion matrices of two datasets.

 

average scores of \(8.30\) and \(5.51\), respectively. However, their performance declines when tested on non-English datasets, resulting in average decreases of \(1.53\) and \(0.33\), respectively. Compared to task-specific InfoDCL, ChatGPT performs poorly in \(63\) out of \(64\) languages, sometimes with a large gap (e.g., \(45.06\) lower on Amharic, \(38.67\) lower on Malayalam, and \(36.91\) lower on Buginese), as Table 4 shows.

We also investigate how different models perform on SM tasks across various languages. Results for two tasks, hate speech detection (top) and humor detection (bottom), are presented in Figure 4. The dataset for each task is grouped according to language, and the average score of each language is obtained. The relative gain of each model against the random baseline is shown, allowing us to compare across these languages.7 We observe that InfoDCL is the best model across various tasks and languages, with the exception of hate speech in Polish where ChatGPT outperforms it. As Figure 4 shows, ChatGPT performs better for Western languages on hate speech detection. We can also observe wider gaps in hate speech detection between ChatGPT and InfoDCL on Arabic and Korean. Similarly, while ChatGPT demonstrates satisfactory performance in English humor, it remains at significant distance behind InfoDCL in Hindi humor.

Footnote 7: We note that different annotation artifacts across the different languages still make direct comparisons challenging.

**(5) Do machine translated prompts help LLMs?_Not in general, but they do help in a few cases_. We find, in Table 3, that the SPARROW score of ChatGPT with machine translated prompts is \(6.14\) points lower than ChatGPT with English prompts. Meanwhile, a few tasks such as humor and sarcasm acquire improvements. We also observe a similar pattern for BLOOMZ and mT0, as Table 3 shows. The low-resource languages with non-Latin scripts experience more performance drops in general, which is in line with findings by Lai et al. (2023). Hebrew (heb) and Greek (ell) get the largest performance drops (over \(25\) points in each case), as shown in Table 4.

**(6) Does GPT-4 outperform ChatGPT?_Yes, it does._ We provide a study on probing GPT-4's capacities. We exploit \(20\) datasets from two tasks (i.e., hate speech and humor detection) in \(12\) languages, only choosing samples whose labels ChatGPT predicted incorrectly. We refer to this test set as GPTHard and provide samples from it to GPT-4 in their original language, employing the same English prompts as those used by ChatGPT. As Table 5 shows, GPT-4 significantly outperforms ChatGPT (McNemar's test with \(\alpha<0.01\)) on \(19\) datasets.8

Footnote 8: An exception is one dataset where a significance test is not possible due to small sample size (\(n=2\)).

**(7) Can translating input samples into English help improve ChatGPT's predictions?_Yes, it can._ Here, we use the non-English part of GPTHard (\(16\) datasets). We translate these test samples into English using ChatGPT and subsequently employ the translated text and English prompt for classification. As Table 5 shows, we acquire a noteworthy enhancement in ChatGPT's performance (McNemar's test with \(\alpha<0.01\)

Figure 4: A comparison of different models on multiple tasks across various languages. We show the relative gain of each model compared to the random baseline.

 

when using the translated input. We also observe that when fed with these English-translated samples, ChatGPT is able to surpass GPT4 with the original inputs in three datasets (i.e., Hate-\({\text{ara}_{\text{Ala}}}\), Hate-\({\text{spa}_{\text{Bas}}}\), Hate-\({\text{ara}_{\text{Mub}}}\)). These results suggest that although ChatGPT has inferior ability on several languages in terms of detecting SM, a translate-then-detect approach may be possible.

**(8) How do open-source LLMs perform with few-shot in-context learning?** As Table 6 shows, we compare three-shot and five-shot results with zero-shot results. Based on SPARROW score, we observe that few-shot learning does enhance the performance of BLOOM, mT5, LLaMA, and Vicuna. With the increasing number of shots, the performance of LLaMA and Vicuna increases. Vicuna obtains SPARROW scores of \(29.36\), \(39.44\), and \(41.97\) with zero, three, and five shots, respectively. However, BLOOMZ-P3 and mT0 do not improve with few-shot learning. We suspect this is because the instruction fine-tuning of these two models only uses a zero-shot template that hurts their few-shot learning capacities. BLOOMZ-P3 and mT0 are also different from BLOOM and LLaMA in that they are fine-tuned on several NLP tasks only one of which is an SM task (i.e., sentiment analysis). This probably biases the behavior of these two models.

**(9) Are the open-source LLMs sensitive to prompts used?** We carry out a study to probe the open-source LLMs' sensitivity to prompts. We curate \(55\) datasets across four tasks from SPARROW and evaluate six models with the prompts we used for evaluating ChatGPT. As Table 24 in Appendix shows, we find that BLOOM, LLaMA, and Vicuna incur sizable performance drops (\(>6\) points decrease across \(55\) datasets), while BLOOMZ-P3, mT5, and mT0 demonstrate performance levels akin to those observed in previous experiments (\(<2\) points different). We leave a more comprehensive evaluation of prompt sensitivity as future work.

## 6 Public Leaderboard

To facilitate future work, we design a public leaderboard for scoring models on SPARROW. Our leaderboard is _interactive_ and offers _rich metadata_ about the various datasets in our benchmark. It also encourages users to submit information about their models (e.g., number of parameters, time to convergence, pretraining datasets). We also distribute a new _modular toolkit_ for finetuning or evaluating models on SPARROW.

## 7 Conclusion

In order to understand the abilities of ChatGPT and other instruction tuned LLMs on capturing sociopragmatic meaning, we introduced a massively multilingual evaluation benchmark, dubbed SPARROW. The benchmark involves \(169\) datasets covering \(64\) languages from \(12\) language families and \(16\) scripts. Evaluating ChatGPT on SPARROW, we find it struggles with different languages. We also reveal that task-specific models finetuned on SM (much smaller than ChatGPT) consistently outperform larger models by a significant margin even on English.

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Tasks**} & \multicolumn{3}{c}{**Zero-shot**} & \multicolumn{3}{c}{**Three-shot**} & \multicolumn{3}{c}{**Five-shot**} \\ \cline{3-14}  & \multicolumn{3}{c}{**BM**} & \multicolumn{3}{c}{**mT5**} & \multicolumn{3}{c}{**mT0**} & \multicolumn{3}{c}{**LLLA. Vie.**} & \multicolumn{3}{c}{**WM**} & \multicolumn{3}{c}{**BZZ**} & \multicolumn{3}{c}{**mT0**} & \multicolumn{3}{c}{**LLLA. Vie.**} & \multicolumn{3}{c}{**BM**} & \multicolumn{3}{c}{**BMZ**} \\ \cline{3-14}  & **BM** & **P3** & **mT5** & **mT0** & **LLA. Vie.** & **BM** & **P3** & **mT0** & **LLA. Vie.** & **BM** & **P3** & **mT5** & **mT0** & **LLa. Vie.** \\ \hline \multirow{14}{*}{**LMa**} & Aggressive & 51.06 & 17.2 & 53.67 & 15.82 & 18.31 & 25.07 & 46.43 & 43.93 & 40.73 & 41.88 & 43.70 & 44.53 & 47.92 & 47.63 & 33.80 & 37.52 & 50.66 & **55.27** \\  & Datagrams & 46.87 & 46.87 & **49.31** & 46.87 & 46.87 & 46.87 & 46.87 & 46.84 & 45.68 & 46.87 & 46.86 & 46.87 & 46.87 & 46.87 & 46.87 & 46.87 & 46.87 & 46.87 & 46.87 & 46.87 & 46.87 & 46.87 \\  & Hate & 39.83 & 88.52 & 32.97 & 37.33 & 37.80 & 41.59 & 88.83 & 38.36 & 39.43 & 37.82 & 43.51 & **49.17** & 37.95 & 37.14 & 39.53 & 37.70 & 41.87 & 48.37 \\  & Offense & 41.06 & 38.95 & 24.99 & 39.90 & 38.55 & 48.70 & 43.94 & 40.25 & 21.99 & 45.13 & 46.36 & **54.49** & 42.42 & 40.59 & 34.10 & 41.67 & 43.83 & 51.72 \\  & HO-Group & 103.63 & **12.23** & **7.02** & 16.25 & 12.55 & 9.26 & 11.43 & 13.04 & 17.92 & 15.98 & 11.84 & 11.69 & 6.88 & 11.16 & 7.23 & 14.50 & 25.85 & 16.27 \\  & HO-Target & 18.73 & 16.89 & 6.69 & -20.58 & 19.32 & 17.01 & 17.09 & 18.94 & 10.48 & 16.55 & 20.56 & **24.84** & 17.44 & 17.45 & 9.32 & 16.56 & 20.09 & 23.60 \\ \cline{2-14}  & **AS** & 57.30 & 51.57 & 20.14 & 52.02 & 51.68 & 54.50 & 35.41 & 25.55 & 27.19 & 52.56 & 36.04 & **24.69** & 52.43 & 51.88 & 50.77 & 52.09 & 35.35 & 40.99 \\ \hline \multirow{14}{*}{**Fantion**} & 97.1 & 15.07 & 17.55 & 27.87 & 15.14 & 16.12 & 17.08 & 27.18 & 17.16 & 26.16 & 23.21 & 32.02 & 40.28 & 18.48 & 23.55 & 10.07 & 25.57 & 34.04 & **24.79** \\ \cline{1-1} \cline{2-14}  & **Humor** & 41.78 & 33.04 & 43.60 & 33.12 & 39.78 & 46.19 & 33.67 & 33.12 & 44.70 & 38.19 & 52.50 & 57.15 & 34.06 & 33.12 & 49.20 & 37.08 & 53.86 & **58.75** \\ \cline{1-1} \cline{2-14}  & Ivony & 36.63 & 44.46 & 36.32 & 34.69 & 40.78 & **47.82** & 42.14 & 45.81 & 45.61 & 36.76 & **37.96** & 44.34 & 44.14 & 96.74 & 38.28 & 36.04 & 41.61 \\ \cline{1-1}  & Sarcasm & 43.09 & 41.68 & 36.09 & 41.44 & 41.77 & 46.64 & 42.41 & 46.72 & 46.72 & 48.42 & 49.75 & **52.55** & 44.53 & 43.05 & 45.75 & 39.88 & 40.93 & 52.51 \\ \cline{1-1} \cline{2-14}  & Inors-Type & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 & 18.83 \\ \cline{1-1} \cline{2-14}  & IMS-S & 36.74 & 21.70 & 54.23 & 73.36 & 39.76 & **26.14** & 43.01 & 47.13 & 43.48 & 40.98 & 44.75 & 34.17 & 43.67 & 54.52 & 47.47 & 45.92 \\ \hline \multicolumn{14}{l}{**Serment**} & 26.67 & **43.03** & 20.77 & 34.65 & 27.55 & 25.02 & 34.81 & 35.79 & 24.76 & 31.37 & 3 

Limitations

Benchmark Construction.Our SPARROW benchmark only includes text classification tasks related to SM. Despite our best efforts, we acknowledge that our benchmark has not covered existing SM datasets exhaustively. We will continue expanding this benchmark and welcome future datasets or metric contributions to it. We also plan to extend SPARROW to more types of tasks related to SM, such as span-based sentiment analysis Xu et al. (2020), affective language generation Goswamy et al. (2020), and conversational sentiment analysis Ojamaa et al. (2015). We only include text-based SM tasks. Another improvement direction is to extend this benchmark to more tasks that involve more modalities, such affective image captioning Mohamed et al. (2022) and multi-modal emotion recognition Firdaus et al. (2020).

Model Selection.Due to computation constraints, we cannot evaluate on model sizes \(>7\)B. However, we hope SPARROW will be used in the future to evaluate larger-sized models. Again, due to budget constraints, we only conduct a relatively small case study on GPT-4 and do not evaluate more diverse commercial instruction tuned models that are more expensive (e.g., text-davinci-003 by OpenAI).

Experiments.While we customize prompts employed for each task, we do not tailor prompts specifically for each model. We acknowledge that the performance of models may be influenced by different prompt variants. In future work, we will test diverse prompt variations for more robust results. We only experiment with machine translated prompts in our analyses and acknowledge that the performance drop may stem from the poor quality of machine translation. We will investigate the utility of human translated prompts in a future study. In this paper, we only evaluate LLMs on zero-shot learning. The adoption of few-shot context learning may enhance performance, which we also leave to future work.

## Ethics Statement and Broad Impacts

Data Collection and Releasing.All the \(169\) datasets are produced by previous research. Since there are large numbers of datasets and languages in SPARROW, it is hard to manually verify the quality of all the datasets. As a quality assurance measure, we only include in SPARROW datasets that are introduced in peer-reviewed published research. To facilitate access to information about each dataset, we link to each published paper describing each of these datasets inTables 9, 10, 11, 12, 13, and 14.

Following privacy protection policies, we anonymize all SPARROW data as described in Section 3.2. With reference to accessibility of the original individual dataset, SPARROW data can be categorized into three releasing strategies: **(1)** In the case of datasets requiring approval by the original authors, we require future researchers to obtain approval first and will share our splits once approval has been obtained. We indicate these nine datasets in our data description tables. **(2)** For the \(25\) datasets (see Table 8 in Appendix) that are shared via tweet IDs, we share our obtained data for research use. By doing so, we expect to mitigate the issue of data decay and allow fair comparisons. **(3)** We will share the other \(135\) publicly accessible datasets upon request. We will also require a justification for responsible use of the datasets. Each dataset will be shared in our Train, Dev, and Test splits along with a dataset card to indicate the original publication of the dataset.

Intended Use.The intended use of SPARROW benchmark is to construct a scoring board to facilitate model comparisons as well as enhance fairness and reproducibility across different languages and tasks. We also aim to mitigate data decay issues in social media research. SPARROW could help researchers investigate model's capacity on SM tasks across languages. SPARROW may also be used to investigate model transferability across a wide range of tasks and diverse languages in different settings (such as zero- or few-shot settings and prompting).

Potential Misuse and Bias.We notice that some annotations in the datasets of SPARROW (e.g., for hate speech task Waseem and Hovy (2016)) can carry annotation and temporal biases. We recommend that any dataset in SPARROW not be used for research or in applications without careful consideration of internal biases of the datasets and potential biases of the resulting systems. We also suggest that users of SPARROW not only focus on the overall SPARROW score but also a model performance on each task and 

dataset. The SPARROW score is an unweighted average score over all the dataset-specific metrics, which may lose the fine-grained information and be dominated by the largest task cluster (i.e., sentiment analysis) or languages (e.g., languages from Indo-European language family).

## Acknowledgements

We acknowledge support from Canada Research Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN-2018-04267), the Social Sciences and Humanities Research Council of Canada (SSHRC; 435-2018-0576; 895-2020-1004; 895-2021-1008), Canadian Foundation for Innovation (CFI; 37771), Digital Research Alliance of Canada,9 and UBC ARC-Sockeye.10

Footnote 9: [https://alliancecan.ca](https://alliancecan.ca)

Footnote 10: [https://arc.ubc.ca/ubc-arc-sockeye](https://arc.ubc.ca/ubc-arc-sockeye)

## References

* Abdul-Mageed et al. (2014) Muhammad Abdul-Mageed, Mona T. Diab, and Sandra Kubler. 2014. SAMAR: subjectivity and sentiment analysis for arabic social media. _Comput. Speech Lang._, 28(1):20-37.
* Abdul-Mageed et al. (2021) Muhammad Abdul-Mageed, AbdelRahim A. Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: deep bidirectional transformers for arabic. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACLIJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 7088-7105. Association for Computational Linguistics.
* Abdul-Mageed and Ungar (2017) Muhammad Abdul-Mageed and Lyle Ungar. 2017. EmoNet: Fine-grained emotion detection with gated recurrent neural networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 718-728, Vancouver, Canada. Association for Computational Linguistics.
* Abdul-Mageed et al. (2022) Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The third nuanced Arabic dialect identification shared task. In _Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP)_, pages 85-97, Abu Dhabi, United Arab Eminates (Hybrid). Association for Computational Linguistics.
* Abdul-Mageed et al. (2020) Muhammad Abdul-Mageed, Chiyu Zhang, Azadeh Hashemi, and El Moatez Billah Nagoudi. 2020. AraNet: A deep learning toolkit for Arabic social media. In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_, pages 16-23, Marseille, France. European Language Resource Association.
* Farha and Magdy (2020) Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic sentiment analysis to sarcasm detection: The ArSacram dataset. In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_, pages 32-39, Marseille, France. European Language Resource Association.
* Aggarwal et al. (2020)Akshita Aggarwal, Anshul Wadhawan, Anshima Chaudhary, and Kavita Maurya. 2020. "did you really mean what you said?" : Sarcasm detection in Hindi-English code-mixed data using bilingual word embeddings. In _Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)_, pages 7-15, Online. Association for Computational Linguistics.
* Ahuja et al. (2023) Kabir Ahuja, Rishaw Hada, Millicent Ochieng, Prachi Jain, Harishta Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: multilingual evaluation of generative AI. _CoRR_, abs/2303.12528.
* Alakrot et al. (2018)Azladen Alakrot, Liam Murray, and Nikola S. Nikolov. 2018. Dataset construction for the detection of antisocial behaviour in online communication in arabic. In _Fourth International Conference On Arabic Computational Linguistics, ACLING 2018, November 17-19, 2018, Dubai, United Arab Eminates_, volume 142 of _Procedia Computer Science_, pages 174-181. Elsevier.
* Alshehri et al. (2020) Ali Alshehri, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2020. Understanding and detecting dangerous speech in social media. In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_, pages 40-47, Marseille, France. European Language Resource Association.
* Amram et al. (2018) Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from Modern Hebrew. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 2242-2252, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Asli et al. (2020) Seyed Arad Ashrafi Asli, Behnam Sabei, Zahra Majdabadi, Preni Golazizian, reza fahmi, and Omid Momenzadeh. 2020. Optimizing annotation effort using active learning strategies: A sentiment analysis case study in Persian. In _Proceedings of The 12th Language Resources and Evaluation Conference_, pages 2855-2861, Marseille, France. European Language Resources Association.
* Assenmacher et al. (2018) Dennis Assenmacher, Derek Weber, Mike Preuss, Andre Calero Valdez, Alison Bradshaw, Bjorn Ross, 

Stefano Cresci, Heike Trautmann, Frank Neumann, and Christian Grimme. 2022. Benchmarking crisis in social media analytics: A solution for the data-sharing problem. _Social Science Computer Review_, 40(6):1496-1522.
* Bamman and Smith [2015] David Bamman and Noah A. Smith. 2015. Contextualized sarcam detection on twitter. In _Proceedings of the Ninth International Conference on Web and Social Media, ICWSM 2015, University of Oxford, Oxford, UK, May 26-29, 2015_, pages 574-577. AAAI Press.
* Bang et al. [2023] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. _CoRR_, abs/2302.04023.
* Barbieri et al. [2016] Francesco Barbieri, Valerio Basile, Danilo Croce, Malvina Nissim, Nicole Novielli, and Viviana Patti. 2016. Overview of the evalita 2016 sentiment polarity classification task. In _Proceedings of Third Italian Conference on Computational Linguistics (CLiC-it 2016) & Fifth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2016), Napoli, Italy, December 5-7, 2016_, volume 1749 of _CEUR Workshop Proceedings_. CEUR-WS.org.
* Barbieri et al. [2020] Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1644-1650, Online. Association for Computational Linguistics.
* Barbieri et al. [2022] Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 258-266, Marseille, France. European Language Resources Association.
* Basile et al. [2014] Valerio Basile, Andrea Bolioli, Viviana Patti, Paolo Rosso, and Malvina Nissim. 2014. Overview of the evalita 2014 sentiment polarity classification task. _Overview of the Evalita 2014 SENTIIment POLarity Classification Task_, pages 50-57.
* Basile et al. [2019] Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Noza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In _Proceedings of the 13th International Workshop on Semantic Evaluation_, pages 54-63, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
* Bianchi et al. [2021] Federico Bianchi, Debora Nozza, and Dirk Hovy. 2021. FEEL-IT: Emotion and sentiment classification for the Italian language. In _Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_, pages 76-83, Online. Association for Computational Linguistics.
* Bianchi et al. [2022] Federico Bianchi, Debora Nozza, and Dirk Hovy. 2022. XLM-EMO: Multilingual emotion prediction in social media text. In _Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis_, pages 195-203, Dublin, Ireland. Association for Computational Linguistics.
* Blinov et al. [2019] Vladislav Blinov, Valeria Bolotova-Baranova, and Pavel Braslavski. 2019. Large dataset and language model fun-tuning for humor recognition. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 4027-4032. Association for Computational Linguistics.
* Bosco et al. [2018] Cristina Bosco, Felice Dell'Orletta, Fabio Poletto, Manuela Sanguinetti, and Maurizio Tesconi. 2018. Overview of the EVALITA 2018 hate speech detection task. In _Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018) co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018), Turin, Italy, December 12-13, 2018_, volume 2263 of _CEUR Workshop Proceedings_. CEUR-WS.org.
* Boxer and Cortes-Conde [2021] Diana Boxer and Florencia Cortes-Conde. 2021. _Soci Groups and Relational Networks_, Cambridge Handbooks in Language and Linguistics, page 227-246. Cambridge University Press.
* Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Brum [2018] Henrico Bertini Brum. 2018. _Expansao de recursos para analise de sentimentos usando aprendizado semi-supervisionado_. Ph.D. thesis, Universidade de Sao Paulo.
* Brum and Marcas Gracas Volpe [2018] Henrico Bertini Brum and Maria das Gracas Volpe. 2018. Building a sentiment corpus of tweets in Brazilian Portuguese. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, 

Japan, May 7-12, 2018_. European Language Resources Association (ELRA).
* Cabasag et al. [2019] Neil Vicente Cabasag, Vicente Raphael Chan, Sean Christian Lim, Mark Edward Gonzales, and Charibeth Cheng. 2019. Hate speech in philippine election-related tweets: Automatic detection and classification using natural language processing. _Philippine Computing Journal, XIV No_, 1.
* Chakravarthi et al. [2022] Bharathi Raja Chakravarthi, Ruba Priyadharshini, Vigneshwaran Muralidaran, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, and John P. McCrae. 2022. Dravidiancodemix: sentiment analysis and offensive language identification dataset for dravidian languages in code-mixed text. _Lang. Resour. Evaluation_, 56(3):765-806.
* Chen et al. [2022] Qianben Chen, Richong Zhang, Yaowei Zheng, and Yongyi Mao. 2022. Dual contrastive learning: Text classification via label-aware data augmentation. _CoRR_, abs/2201.08702.
* Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressions gpt-4 with 90%* chatgtp quality.
* Chiril et al. [2020] Patricia Chiril, Veronique Moriceau, Farah Benamara, Alda Mari, Gloria Origgi, and Marlene Coulomb-Gully. 2020. An annotated corpus for sexism detection in french tweets. In _Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020_, pages 1397-1403. European Language Resources Association.
* Chiruzzo et al. [2021] Luis Chiruzzo, Santiago Castro, Santiago Gongora, Aiala Rosa, J. A. Meaney, and Rada Mihalcea. 2021. Overview of HAHA at IberLEF 2021: Detecting, rating and analyzing humor in Spanish. _Process. del Lang. Natural_, 67:257-268.
* Christiano et al. [2017] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 4299-4307.
* Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Akakanksha Chowdhery, Sharma Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chijt Def Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. _CoRR_, abs/2210.11416.
* Cignarella et al. [2018] Alessandra Teresa Cignarella, Simona Frencha, Valerio Basile, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2018. Overview of the EVALITA 2018 task on irony detection in italian tweets (imorita). In _Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018) co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018), Turin, Italy, December 12-13, 2018_, volume 2263 of _CEUR Workshop Proceedings_. CEUR-WS.org.
* Ciobotaru and Dinu [2021] Alexandra Ciobotaru and Liviu P. Dinu. 2021. Red: A novel dataset for romanian emotion detection from tweets. In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)_, pages 296-305, Varna, Bulgaria. INCOMA Ltd.
* Conneau et al. [2020] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Conneau and Kiela [2018] Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Cortiz et al. [2021] Diogo Cortiz, Jefferson O. Silva, Newton Calegari, Ana Luisa Freitas, Ana Angelica Soares, Carolina Botelho, Gabriel Gaudencio Rego, Waldir Sampaio, and Paulo Sergio Boggio. 2021. A weak supervised dataset of fine-grained emotions in portuguese. _CoRR_, abs/2108.07638.
* Das et al. [2023]Mithun Das, Saurabh Kumar Pandey, and Animeesh Mukherjee. 2023. Evaluating chatgptt's performance for multilingual and emoji-based hate speech detection. _CoRR_, abs/2305.13276.
* Davidson et al. [2017] Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In _Proceedings of the Eleventh International Conference on Web and Social Media, ICWSM 2017, Montreal, Quebec, Canada, May 15-18, 2017_, pages 512-515. AAAI Press.
* DeLucia et al. [2022] Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Mark Dredze, and Philip Resnik. 2022. Bernice: A multilingual pre-trained encoder for Twitter. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December, 2022_, pages 6191-6205. Association for Computational Linguistics 

Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4040-4054, Online. Association for Computational Linguistics.
* Deng et al. (2022) Jiawen Deng, Jingyan Zhou, Hao Sun, Fei Mi, and Minlie Huang. 2022. COLD: A benchmark for chinese offensive language detection. _CoRR_, abs/2201.06025.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Diallo et al. (2021) Mountaga Diallo, Chayma Fourati, and Hatem Haddad. 2021. Bambara language dataset for sentiment analysis.
* Dingli and Sant (2016) Alexiei Dingli and Nicole Sant. 2016. Sentiment analysis on maltese using machine learning. In _Proceedings of The Tenth International Conference on Advances in Semantic Processing (SEMAPRO 2016)_, pages 21-25.
* Dumitrescu et al. (2020) Stefan Daniel Dumitrescu, Andrei-Marius Avram, and Sampo Pyysalo. 2020. The birth of romanian BERT. In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_, volume EMNLP 2020 of _Findings of ACL_, pages 4324-4328. Association for Computational Linguistics.
* Elmadany et al. (2022) AbdelRahim A. Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2022. ORCA: A challenging benchmark for arabic language understanding. _CoRR_, abs/2212.10758.
* Farha et al. (2021) Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcam and sentiment detection in arabic. In _Proceedings of the Sixth Arabic Natural Language Processing Workshop, WANLP 2021, Kyiv, Ukraine (Virtual), April 9, 2021_, pages 296-305. Association for Computational Linguistics.
* Felbo et al. (2017) Bjarke Felbo, Alan Mislove, Anders Sogaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1615-1625, Copenhagen, Denmark. Association for Computational Linguistics.
* Firdaus et al. (2020) Mauajima Firdaus, Hardik Chauhan, Asif Ekbal, and Pushpak Bhattacharyya. 2020. MEISD: A multimodal multi-label emotion, intensity and sentiment dialogue dataset for emotion recognition and sentiment analysis in conversations. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 4441-4453, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Fortuna et al. (2019) Paula Fortuna, Joao Rocha da Silva, Juan SolerCompany, Leo Wanner, and Sergio Nunes. 2019. A hierarchically-labeled Portuguese hate speech dataset. In _Proceedings of the Third Workshop on Abusive Language Online_, pages 94-104, Florence, Italy. Association for Computational Linguistics.
* Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.
* Ghanem et al. (2019) Bilal Ghanem, Jihen Karoui, Farah Benamara, Veronique Moriceau, and Paolo Rosso. 2019. IDAT at FIRE2019: overview of the track on irony detection in arabic tweets. In _FIRE '19: Forum for Information Retrieval Evaluation, Kolkata, India, December, 2019_, pages 10-13. ACM.
* Golazizian et al. (2020) Preni Golazizian, Behnam Sabei, Seyed Arad Ashrafi Asli, Zahra Majdabadi, Omid Momenzadeh, and Reza Fahmi. 2020. Irony detection in Persian language: A transfer learning approach using emoji prediction. In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 2839-2845, Marseille, France. European Language Resources Association.
* Gong et al. (2020) Xiaochang Gong, Qin Zhao, Jun Zhang, Ruibin Mao, and Ruifeng Xu. 2020. The design and construction of a chinese sarcasm dataset. In _Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020_, pages 5034-5039. European Language Resources Association.
* Gordon Jr (2005) Raymond G Gordon Jr. 2005. Ethnologue, languages of the world. _http://www. ethnologue. com/_.
* Goswamy et al. (2020) Tsuhar Goswamy, Ishika Singh, Ahshan Barkati, and Ashutosh Modi. 2020. Adapting a language model for controlled affective text generation. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 2787-2801, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Guven et al. (2020) Zekeriya Guven, Banu Diri, and Tolgahan Cakaoglu. 2020. Comparison of n-stage latent dirichlet allocation versus other topic modeling methods for emotion analysis. _Journal of the Faculty of Engineering and Architecture of Gazil University_, 35(4).
* Ho et al. (2019) Vong Anh Ho, Duong Huynh-Cong Nguyen, Danh Hoang Nguyen, Linh Thi-Van Pham, 

Duc-Vu Nguyen, Kiet Van Nguyen, and Ngan Luuthuyen. 2019. Emotion recognition for Vietnamese social media text. In Computational Linguistics - 16th International Conference of the Pacific Association for Computational Linguistics, PACLING 2019, Hanoi, Vietnam, October 11-13, 2019, Revised Selected Papers, Vol. 1215 of Communications in Computer and Information Science, pp. 319-333. Cited by: SS4.
* J. Hu, S. Ruder, A. Siddhaant, G. Neubig, O. Firat, and M. Johnson (2020)XTREME: a massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, V. K. Leibler, R. Fergus, D. Fox, and C. E. Manning (Eds.), pp. 4411-4421. External Links: Link, Document Cited by: SS4.
* H. Huang, T. Tang, D. Zhang, W. Xin Zhao, Ting Song, Y. Xia, and F. Wei (2023)Not all languages are created equal in llms: improving multilingual capability by cross-lingual-thought prompting. CoRRabs/2305.07004. External Links: Link, 2305.07004 Cited by: SS4.
* Md. Asif Iqbal, A. Das, O. Sharif, M. Moque, and I. H. Sarker (2022)Bemoc: a corpus for identifying emotion in bengali texts. SN Comput. Sci.3 (2), pp. 135. Cited by: SS4.
* K. Istham, S. Kar, M. Saiful Islam, and M. Anin (2021)SentNoB: a dataset for analysing sentiment on noisy bangla texts. In Findings of the Association for Computational Linguistics: EMNLP 2021, Punta Cana, Dominican Republic, pp. 3265-3271. External Links: Link, Document Cited by: SS4.
* H. Jang, M. Kim, and H. Shin (2013)KOSAC: a full-fledged korean sentiment analysis corpus. In Proceedings of the 27th Pacific Asia Conference on Language, Information and Computation, PACLIC 27, Taipei, Taiwan, November 21-24, 2013, External Links: Link, Document Cited by: SS4.
* Younghoon Jeong, J. Oh, J. Ahn, J. Lee, J. Moon, S. Park, and A. Oh (2022)KOLD: korean offensive language dataset. CoRRabs/2205.11315. External Links: Link, 2205.11315 Cited by: SS4.
* Kaisla Kajava (2018)Cross-lingual sentiment preservation and transfer learning in binary and multiclass classification. Master's thesis, University of Helsinki. Cited by: SS4.
* M. Rezault Karim, S. K. Dey, T. Islam, S. Sarker, M. Hasan Menon, K. Hosseian, M. Ozam, and S. Decker (2021)DeepHarexpalinear: explainable hate speech detection in under-resourced pongali language. In 8th IEEE International Conference on Data Science and Advanced Analytics, DSAA 2021, Porto, Portugal, October 6-9, 2021, P. Ke, H. Ji, S. Liu, X. Zhu, and M. Huang (Eds.), pp. 6975-6988. External Links: Link, Document Cited by: SS4.
* P. Koc, B. Mikowski, and M. Zasko-Zielinska (2019)Multi-level sentiment analysis of poleno2.0: extended corpus of multi-domain consumer reviews. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), Hong Kong, China, pp. 980-991. External Links: Link, Document Cited by: SS4.
* P. Kralj Novak, I. Mozetic, and N. Ljubesic (2021)Slovenian twitter hate speech dataset imsvppfl-sl. Slovenian language resource repository CLARIN.SI. Cited by: SS4.
* A. Kulkarni, M. Mandhane, M. Likhtkar, G. Kshirsagar, and R. Joshi (2021)L3cubemahasment: a marathi tweet-based sentiment analysis dataset. CoRRabs/2103.11408. External Links: Link, 2103.11408 Cited by: SS4.
* R. Kumar, A. N. Reganti, A. Bhatia, and T. Maheshwari (2018)Aggression-annotated corpus of hindi-engliish code-mixed data. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018, External Links: Link, Document Cited by: SS4.
* V. Lacai, N. Trung Ngo, A. Pourran Ben Veyseh, H. Man, F. Dermoncourt, T. Bui, and T. Huu Nguyen (2023)CatBPU beyond english: towards a comprehensive evaluation of large language models in multilingual learning. CoRRabs/2304.05613. External Links: Link, 2304.05613 Cited by: SS4.
* M. Tahmid Rahman Taskar, M. Bairi, M. Rahman, M. Amrani Hossen Bruiyan, Shafiq Joty, and J. Kitangji Huang (2023)A systematic study and comprehensive evaluation of chatapt on benchmark datasets. CoRRabs/2305.18486. External Links: Link, 2305.18486 Cited by: SS4.
* S. Lee and Z. Wang (2015)Emotion in code-switching texts: Corpus construction and analysis. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, Beijing, China, pp. 91-99. External Links: Link, Document Cited by: SS4.
* Haonan Li, F. Koto, M. Wu, A. Kikri Aji, and T. Baldwin (2023)Bactrian-x : a multilingual replicable instruction-following model with low-rank adaptation. CoRRabs/2305.15011. External Links: Link, 2305.15011 Cited by: SS4.
* Yaobo Liang, N. Duan, Y. Gong, N. Wu, F. Guo, Weizhen Qi, M. Gong, L. Shou, D. Jiang, G. Cao, X. Fan, R. Zhang, R. Agrawal, E. Cui, S. Wei, T. Bharti, Y. Qiao, J. Chen, W. Wu, S. Liu, F. Yang, D. Campos, R. Majumder, and M. Zhou (2020) 

XGLUE: A new benchmark datasetfor cross-lingual pre-training, understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 6008-6018. Cited by: SS4.
* M. Marreddy, S. R. Oota, L. Sireesha Vakada, V. Charan Chinni, and R. Mamidi (2022)Am i a resource-poor language? data sets, embeddings, models and analysis for four different nlp tasks in telugu language. ACM Trans. Asian Low-Resour. Lang. Inf. Process.22 (1), pp. . External Links: Document Cited by: SS3.
* J. A. Meaney, S. Wilson, L. Chiruzzo, A. Lopez, and W. Magdy (2021)SemEval 2021 task 7: haHackathon, detecting and rating humor and offense. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), Online, pp. 105-119. External Links: Link, Document Cited by: SS4.
* Y. Mohamed, F. F. Khan, K. Haydarov, and M. Elhoseiny (2022)It is okay to not be okay: overcoming emotional bias in affective image captioning by contrastive data collection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 21231-21240. External Links: Link, Document Cited by: SS4.
* S. Mohammad, F. Bravo-Marquez, M. Salameh, and S. Kiritchenko (2018)SemEval-2018 task 1: affect in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, Men, Dohaha, India, pp. 1-17. External Links: Link, Document Cited by: SS4.
* J. Moon, W. I. Cho, and J. Lee (2020)BEEP! Korean corpus of online news comments for toxic speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, Online, Association for Computational Linguistics, Online, pp. 25-31. External Links: Link, Document Cited by: SS4.
* I. Mozetic, M. Grcar, and J. Smailovic (2016)Multilingual twitter sentiment classification: the role of human annotators. PloS one11 (5), pp. e0155036. External Links: Link, Document Cited by: SS4.
* H. Mubarak, K. Darwish, N. Magdy, T. Elsayed, and H. Al-Khalifa (2020)Overview of OSACT4 Arabic offensive language detection shared task. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, Marseille, France, pp. 48-52. External Links: Link, Document Cited by: SS4.
* N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. Le Scao, M. Bari, S. Shen, Z. X. Yong, H. Schoelkopf, X. Tang, D. Radev, A. Fikri Aji, K. Almubarak, S. Albaniez, Z. Alyafeai, A. Webson, E. Raff, and C. Raffel (2022)Crosslingual generalization through multitask finetuning. CoRRabs/2211.01786. External Links: Link, 2111.01786 Cited by: SS4.
* S. Husam Muhammad, I. Abdulmunin, A. Aiyele, N. Osidhoum, D. Heoluwa Adelani, S. Muhie Yimam, I. Said Ahmad, M. Beloucif, S. M. Mohammad, S. Ruder, Q. Hourane, P. Brazil, F. Dario Mario Antoin Ali, D. David, S. Oseil, B. Shelhu bello, F. Ibrahim, T. Gwaldeb, S. Rutunda, T. Destaw Belay, W. Baye Messelle, H. B. Balcha, S. Adugna Chala, H. Sefabrun Gebremichael, B. Opoku, and S. Arthur (2023a)Afrisenti: a twitter sentiment analysis benchmark for african languages. CoRRabs/2302.08956. External Links: Link, 2302.08956 Cited by: SS4.
* S. Hassan Muhammad, I. Abdulmu-min, S. Mihm, D. Afoulahoun, J. A. A. Aplee, S. M. Mohammad, M. Boloucif, and S. Ruder (2023b)SemEval-2023 task 1: efficient analysis for african Languages (afriSenti-SemEval). In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), External Links: Link, Document Cited by: SS4.
* S. Hussan Muhammad, D. Adelani, S. Ruder, I. Sah'id Ahmad, I. Abdulmunin, S. Bello Bello, M. Choudhury, C. Emezue, S. Salahuddeen Abdullah, A. Arenu, A. Jeorge, and P. Brazdil (2022)Najisensti: an nigerian twitter sentiment corpus for multilingual sentiment analysis. In Proceedings of the 13th Language Resources and Evaluation Conference, Florence, Italy, pp. 590-602. External Links: Link, Document Cited by: SS4.
* H. Mulki, H. Haddad, C. Becikh Ali, and H. Alshabani (2019)L-HSAB: a reventine Twitter dataset for hate speech and abusive language. In Proceedings of the Third Workshop on Abusive Language Online, Florence, Italy, pp. 111-118. External Links: Link, Document Cited by: SS4.
* S. Nordhoff and H. Hammstrom (2011)Glottolog/langdoc: defining dialects, languages, and language families as collections of resources. In Proceedings of the First International Workshop on Linked Science 2011, Bonn, Germany, October 24, 2011, Vol. 783, CEUR Workshop Proceedings, CEUR-WS.org, External Links: ISBN 978-1-4503-3895-3 Cited by: SS4.
* E. Ohman, M. Pamies, K. Kajava, and J. Tiedemann (2020)XED: a multilingual dataset for sentiment analysis and emotion detection. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pp. 6542-6552. External Links: Link, Document Cited by: SS4.

 

Briggitta Ojamaa, Paivi Kristina Jokinen, and Kadri Muischenk. 2015 Sentiment analysis on conversational texts. In Proceedings of the 20th Nordic Conference of Computational Linguistics, NODAL-IDA 2015, Institute of the Lithuanian Language, Vilnius, Lithuania, May 11-13, 2015, Vol. 109 of Linkoping Electronic Conference Proceedings, pp. 233-237. External Links: ISBN 978-3-642-46556-5 Cited by: SS1.
* S. Oraby, V. Harrison, L. Reed, E. Hernandez, E. Riloff, and M. Walker (2016)Creating and characterizing a diverse corpus of sarcasm in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Los Angeles, pp. 31-41. External Links: Document, ISSN 1558-1221 Cited by: SS1.
* R. Ortega-Bueno, F. Rangel, D. Hemafez Faras, P. Rosso, M. Montes-y Gomez, and J. E. Medina Pagola (2019)Overview of the task on irony detection in spanish variants. In Proceedings of the Iberian Languages Evaluation Forum co-located with 35th Conference of the Spanish Society for Natural Language Processing, IberLEF@SEPLN 2019, Bilbao, Spain, September 24th, 2019, B. Rosso, A. Montes-y Gomez, and J. E. Medina Pagola (Eds.), CEUR-WS.org, Cham 

the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 704-714, Seattle, Washington, USA. Association for Computational Linguistics.
* Risch et al. (2021) Julian Risch, Philipp Schmidt, and Ralf Krestel. 2021. Data integration for toxic comment classification: Making more than 40 datasets easily accessible in one unified format. In _Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021)_, pages 157-163, Online. Association for Computational Linguistics.
* Rosenthal et al. (2017) Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_, pages 502-518, Vancouver, Canada. Association for Computational Linguistics.
* Rybak et al. (2020) Piotr Rybak, Robert Mroozkowski, Janusz Tracz, and Ireneusz Gawlik. 2020. KLEJ: Comprehensive benchmark for polish language understanding. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1191-1201, Online. Association for Computational Linguistics.
* Sabri et al. (2021) Nazanin Sabri, Reyhane Akhavan, and Behnam Bahrak. 2021. EmoPars: A collection of 30k emotion-annotated persian social media texts. pages 167-173.
* Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multitask prompted training enables zero-shot task generalization. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
* Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1668-1678, Florence, Italy. Association for Computational Linguistics.
* Saputri et al. (2018) Mei Silviana Saputri, Rahmad Mahendra, and Mirna Adriani. 2018. Emotion classification on indonesian twitter dataset. In _2018 International Conference on Asian Language Processing, IALP 2018, Bandung, Indonesia, November 15-17, 2018_, pages 90-95. IEEE.
* Sboev et al. (2020) Alexander G. Sboev, Aleksandr Naumov, and Roman B. Rybka. 2020. Data-driven model for emotion detection in russian texts. In _Proceedings of the 2020 Annual International Conference on Brain-Inspired Cognitive Architectures for Artificial Intelligence, BICA 2020, Eleventh Annual Meeting of the BICA Society, November 10-15, 2020, Virtual Event / Natal, Rio Grande do Norte, Brazil_, volume 190 of _Procedia Computer Science_, pages 637-642. Elsevier.
* De Scao et al. (2020) Teven Le Scao, Angela Fan, Christopher Akiki, Elie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandros Sasha Luccioni, Francois Yvon, Matthias Galle, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamachi, Thomas Wang, Benouti Sagot, Niklas Muennighoff, Alber Villanova del Moral, Olatuji Ruwase, Rachel Bawden, Staas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulinier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurencion, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokasaln, Adi Simhi, Attor Soroa, Alham Flixi Aj, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adellani, and et al. 2022. BLOOM: A 176b-parameter open-access multilingual language model. _CoRR_, abs/2211.05100.
* Shode et al. (2022) Iyanuoluwa Shode, David Ifeoluwa Adelani, and Anna Feldman. 2022. YOSM: A new Yoruba Sentiment Corpus for Movie Reviews. _AfricaNLP 2022 @ICLR_.
* Shome (2021) Debaditya Shome. 2021. Emonind: Fine-grained multilabel emotion recognition from hindi texts with deep learning. In _12th International Conference on Computing Communication and Networking Technologies, ICCCNT 2021, Kharagpur, India, July 6-8, 2021_, pages 1-5. IEEE.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
* Srivastava et al. (2018) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, AKhat Agarwal, Aletena Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tzariv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Asskell, Amanda Do Souza, Ameet Rahane, Ananthharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas 

Stuhlmuller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Artul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _CoRR_, abs/2206.04615.
* Sun et al. (2020) Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. ERNIE 2.0: A continual pre-training framework for language understanding. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_, pages 8968-8975. AAAI Press.
* Suresh and Ong (2021) Varsha Suresh and Desmond C. Ong. 2021. Not all negatives are equal: Label-aware contrastive loss for fine-grained text classification. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_, pages 4381-4394. Association for Computational Linguistics.
* Suriyawongkul et al. (2019) Arthit Suriyawongkul, Ekapol Chuangsuwanich, Pattarawat Chormai, and Charin Polpanumas. 2019. Pythainlp/wisesight-sentiment: First release.
* Suzuki et al. (2022) Haruya Suzuki, Yuto Miyauchi, Kazuki Akiyama, Tomoyuki Kaijwara, Takashi Ninomiya, Noriko Takemura, Yuta Nakashima, and Hajime Nagahara. 2022. A Japanese dataset for subjective and objective sentiment polarity classification in micro blog domain. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 7022-7028, Marseille, France. European Language Resources Association.
* Tagliamonte (2015) Sali A Tagliamonte. 2015. _Making waves: The story of variationist sociolinguistics_. John Wiley & Sons.
* Tan and Zhang (2008) Songbo Tan and Jin Zhang. 2008. An empirical study of sentiment analysis for chinese documents. _Expert Syst. Appl._, 34(4):2622-2629.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. 2023. Stanford aplica: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Thelwall et al. (2012) Mike Thelwall, Kevan Buckley, and Georgios Palotoglou. 2012. Sentiment strength detection for the social web. _J. Assoc. Inf. Sci. Technol._, 63(1):163-173.
* Tian et al. (2020) Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, and Feng Wu. 2020. SKEP: Sentiment knowledge enhanced pre-training for sentiment analysis. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4067-4076, Online. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gaudier Izacard, Xavier Martinet, Marie-Anne Lacchaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLama: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* Lee et al. (2018) Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In _Proceedings of The 12th International Workshop on Semantic Evaluation_, pages 39-50, New Orleans, Louisiana. Association for Computational Linguistics.
* Velldal et al. (2018) Erik Velldal, Lilja Ovrelid, Eivind Alexander Bergem, Cathrine Stadsnes, Samia Touileb, and Fredrik Jorgensen. 2018. NoReC: The Norwegian review corpus. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Vidgen and Derczynski (2020) Bertie Vidgen and Leon Derczynski. 2020. Directions in abusive language training data, a systematic review: Gabrage in, garbage out. _Plos one_, 15(12):e0243300.
* Vijay et al. (2018) Deepanshu Vijay, Aditya Bohra, Vinay Singh, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018. A dataset for detecting irony in hindi-engishi code-mixed social media text. In _Proceedings of 4th Workshop on Sentic Computing, Sentiment Analysis, Opinion Mining, and Emotion Detection (EMSASW 2018) Co-located with the 15th Extended Semantic Web Conference 2018 (ESWC 2018), Heraklion, Greece, June 4, 2018_, volume 2111 of _CEUR Workshop Proceedings_, pages 38-46. CEUR-WS.org.
* Walker et al. (2012) Marilyn Walker, Jean Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for research on deliberation and debate. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)_, pages 812-817, Istanbul, Turkey. European Language Resources Association (ELRA).
* Wallbott and Scherer (1986) Harald G Wallbott and Klaus R Scherer. 1986. How universal and specific is emotional experience? evidence from 27 countries on five continents. _Social science information_, 25(4):763-795.
* 25th International Conference, DASFAA 2020, Jeju, South Korea, September
 

24-27, 2020, Proceedings, Part III_, volume 12114 of _Lecture Notes in Computer Science_, pages 745-749. Springer.
* Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Wang et al. (2023) Zengzhi Wang, Qiming Xie, Zixiang Ding, Yi Feng, and Rui Xia. 2023. Is chatgpt a good sentiment analyzer? A preliminary study. _CoRR_, abs/2304.04339.
* Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In _Proceedings of the NAACL Student Research Workshop_, pages 88-93, San Diego, California. Association for Computational Linguistics.
* Wilie et al. (2020) Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. Indonlu: Benchmark and resources for evaluating indonesian natural language understanding. In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, AACL/IJCNLP 2020, Suzhou, China, December 4-7, 2020_, pages 843-857. Association for Computational Linguistics.
* Winata et al. (2022) Genta Indra Winata, Alham Fikeri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2022. Nusax: Multilingual parallel sentiment dataset for 10 Indonesian local languages. _CoRR_, abs/2205.15960.
* Wu et al. (2023) Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Albamir Kikri Aji. 2023. Lamini-lm: A diverse herd of distilled models from large-scale instructions. _CoRR_, abs/2304.14402.
* Xiang et al. (2020) Rong Xiang, Xuefeng Gao, Yunfei Long, Anran Li, Emmanuele Chersoni, Qin Lu, and Chu-Ren Huang. 2020. Crion: a new benchmark dataset for chinese irony detection. In _Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020_, pages 5714-5720. European Language Resources Association.
* Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihu Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020a. CLUE: A chinese language understanding evaluation benchmark. In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_, pages 4762-4772. International Committee on Computational Linguistics.
* Xu et al. (2020b) Lu Xu, Lidong Bing, Wei Lu, and Fei Huang. 2020b. Aspect sentiment classification with aspect-specific opinion spans. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3561-3567, Online. Association for Computational Linguistics.
* Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihur Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_, pages 483-498. Association for Computational Linguistics.
* Yimam et al. (2020) Seid Muhrie Yimam, Hizkiel Mitiku Alemayehu, Abinev Ayele, and Chris Biemann. 2020. Exploring Ambaric sentiment analysis from social media texts: Building annotation tools and classification models. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 1048-1060, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Notra Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1415-1420, Minneapolis. Association for Computational Linguistics.
* Zampieri et al. (2020) Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdyub Murzak, Leon Derczynski, Zeses Pitenis, and Cagri Coltekin. 2020. Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020). In _Proceedings of the Fourteenth Workshop on Semantic Evaluation, SemEval@COLING 2020, Barcelona (online), December 12-13, 2020_, pages 1425-1447. International Committee for Computational Linguistics.
* Zhang and Abdul-Mageed (2022) Chiyu Zhang and Muhammad Abdul-Mageed. 2022. Improving social meaning detection with pragmatic masking and surrogate fine-tuning. In _Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis_, pages 141-156, Dublin, Ireland. Association for Computational Linguistics 

Chiyu Zhang, Muhammad Abdul-Mageed, and Ganesh Jawahar. 2023a. Contrastive learning of sociopragmatic meaning in social media. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 2405-2439, Toronto, Canada. Association for Computational Linguistics.
* Zhang et al. (2022) Chiyu Zhang, Muhammad Abdul-Mageed, and El Moatez Billash Nagoudi. 2022. Decay no more: A persistent twitter dataset for learning social meaning. In _Workshop Proceedings of the 16th International AAAI Conference on Web and Social Media, ICWSM 2022 Workshops, Atlanta, Georgia, USA__[hybrid], June 6, 2022_.
* Zhang et al. (2023b) Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023b. Sentiment analysis in the era of large language models: A reality check. _CoRR_, abs/2305.15005.
* Zhong et al. (2023) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? A comparative study on chatgtp and fine-tuned BERT. _CoRR_, abs/2302.10198.
* Ziems et al. (2023) Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large language models transform computational social science? _CoRR_, abs/2305.03514.

## Appendix A Benchmark

Table 7 summarizes language distribution of datasets in SPARROW and taxonomy of these language according to Ethnologue Gordon Jr (2005) and Glottolog Nordhoff and Hammarstrom (2011). Tables 9, 10, 11, 12, 13, and 14 describe the datasets in tasks of antisocial language detection, emotion recognition, humor detection, irony and sarcasm detection, sentiment analysis, and subjectivity analysis, respectively.

We empirically characterize the issue of data inaccessibility by re-collecting tweets content via tweet IDs. Table 8 shows the data decay issue of \(25\) datasets.

 



**(1) BLOOM**[25] is a Transformer decoder-only model trained on the ROOTS corpus consisting of \(46\) natural and \(13\) programming languages. BLOOM uses a multilingual vocabulary with \(250\)K tokens and is trained with auto-regressive language modelling objectives.

**(2) Multilingual T5 (mT5)**[28] is Transformer encoder-decoder model trained on CommonCrawl data involving \(101\) languages and contains a vocabulary with \(250\)K tokens. It trained with sequence-to-sequence MLM objective.

**(3) LLaMA**[79] is a Transformer decoder-only model pretrained on 1.4T tokens where the majority are English and a small amount of data in 20 other languages. We utilize LLaMA with 7B parameters and a vocabulary with 30K tokens.

**(4) BLOOMZ**[79] is also an instruction finetune model. It further fine-tunes BLOOM on xP3 corpus that contains \(13\) type of tasks in \(46\) languages with English prompt. We benchmark SPARROW on the BLOOM-based models with a size of \(7.1\)B parameters.

**(5) BLOOMZ-P3**[79] is an instruction finetuned model. It is initialized by BLOOM and further finetunes on English-only P3 corpus [26] containing \(2,073\) natural language prompts for eight types of NLP tasks.

**(6) BLOOM-Bacteria**[10] tune BLOOM on a \(3.4\)M instruction-following dataset in \(52\) languages with low-rank adaptation modules. Li et al. li:2013 translate the English \(67\)K instructions from Alpaca and Dolly datasets into \(51\) languages and utilize ChatGPT API to generate responses in the corresponding language.

**(7) mT0**[79] is instruction fine-tuned mT5 model with xP3 corpus. We evaluate the mT5-based models with XL size (with 3.7B parameters).

**(8) Alpaca**[79] further tune LLaMA on a \(52\)K instruction-following dataset that is generated by gpt-3.5-turbo of OpenAI API. The dataset includes diverse English instruction-following tasks, e.g., question answering and programming.

**(9) Vicuna**[10] further tune LLaMA on \(70\)K diverse user-shared conversations with ChatGPT in English.

**(10) ChatGPT** is a conversation-based LLM trained GTP-3 [7] through reinforcement learning with human feedback [79, 80]. We exploit gpt-3.5-turbo-0301 via OpenAI API.11

Footnote 11: [https://openai.com/](https://openai.com/)

## Appendix C Experiments

### Hyperparameters

To be computation friendly, we only tune the peak learning rate of each model in a set of \(\{1e-4,5e-5,3e-5,1e-5\}\) and randomly select \(45\) datasets for hyper-parameter tuning. We fine-tune a PLM with an arbitrary batch size of \(32\), sequence length of \(128\) tokens, and \(20\) epochs with patience of five epochs based on the model performance on Dev set. We fine-tune each dataset three time with different seeds and identify the best model based on Dev set performance. The best learning rate for each model is identified based on the average score of Dev set of the \(45\) datasets. The best peak learning rate is \(3e-5\) for mBERT, XLM-T, and Bernice and \(1e-5\) for other models.

### Prompts

The prompts we use in our experiments are summarized in Table 15.

### Results

Table 16 shows aggregated performance of fine-tuned models on Dev and Test-S. We report the average of dataset-specific metrics and standard deviation in a task and a category. We also report the Test-S performance of tasks of antisocial language detection, emotion recognition, humor detection, irony and sarcasm detection, sentiment analysis, and subjectivity analysis in Tables 17, 18, 19, 20, 21, and 22, respectively.

We provide a concise study to probe the sensitivity of open-source LLMs to prompts and present the results in Table 24.





\begin{table}
\begin{tabular}{l l r c c c} \hline \hline
**Lang. Family** & **Lang.** & **Code** & **\# dataset** & **Script** \\ \hline  & Ankhate & amh & 1 & Eilicioie \\ Archbe & mraja & any & 1 & Arabic \\  & Darius & areq & 1 & Arabic \\ Afte-Axiatic & Dzitna & arq & 1 & Arabic \\  & Brassa & hasn & 1 & Latin \\  & Hebrew & heb & 1 & Hebrew \\  & Malese & mit & 1 & Latin \\ \hline  & Banbara & hamn & 1 & Latin \\  & Iibo & bio & hio & 1 & Latin \\  & Kinyuranda & kkin & 1 & Latin \\ Atlanta-Congo & Souhlri & swh & 1 & Latin \\  & Tsui & twi & 1 & Latin \\  & Tsonga & tsos & 1 & Latin \\  & Yeohts & yor & 2 & Latin \\ \hline Abstomatic & Vietnamese & vie & 1 & Latin \\ \hline  & Acchinese & ace & 1 & Latin \\  & Balinese & bam & 1 & Latin \\  & Baiparase & bin & 1 & Latin \\  & Borginee & bug & 1 & Latin \\  & Filipson & fi & 1 & Latin \\  & Indonesian & ind & 3 & Latin \\  & Iwrence & jwv & 1 & Latin \\  & Madurese & mod & 1 & Latin \\  & Minzaghaban & min & 1 & Latin \\  & Nguiu & inj & 1 & Latin \\  & Sundzone & sum & 1 & Latin \\  & Tiba bark & ibc & 1 & Latin \\ \hline  & Karunaka & kan & z Kamakata, Latin \\  & Malayham & mal & 2 & Maiiyalma, Latin \\  & Tamil & tam & 2 & Tamil, Latin \\  & Totegue & id & 2 & Telogu \\ \hline  & Albanian & sqj & 1 & Latin \\  & Bosain & bos & 1 & Latin \\  & Bulgarian & bal & 1 & Cyrille \\  & Berngahi & ben & 4 & Bengzli, Latin \\  & Croatian & hrv & 1 & Latin \\  & Czech & cs & 2 & Latin \\  & Danish & dan & 1 & Latin \\  & English & eng & 27 & Latin \\  & French & fra & 6 & Latin \\  & German & da & 3 & Latin \\  & Greek & ell & 1 & Greek \\  & Hindi & him & 5 & Dorangari, Latin \\ Indo-European & Italian & ita & 11 & Latin \\  & Marathi & marr & 1 & Devangari \\  & Nigerian Pdglem & pcm & 2 & Latin \\  & Norwegian & por & 1 & Latin \\  & Persian & fns & 3 & Arabic \\  & Portuguese & prpr & 4 & Latin \\  & Polish & pal & 4 & Latin \\  & Romanian & rom & 2 & Latin \\  & Russian & rrss & 3 & Cyrille \\  & Spanish & spa & 9 & Latin \\  & Sofstan & snp & 1 & Cynilic \\  & Slovak & sk & 1 & Latin \\  & Slovenia & sylv & 2 & Latin \\  & Swedish & swe & 1 & Latin \\ \hline Japonic & Japanese & jpn & 1 & Han, Hir, Kat. \\ Koreanic & Korean & kar & 5 & Hangl \\ \hline Sino-Thetan & Chinese & zho & 6 & Hun \\ \hline Tat-Kadai & Thai & thia & 1 & Thai \\ \hline Turkic & Turkish & tur & 2 & Latin \\ Uralic & Finnish & fin & 3 & Latin \\  & Hungarian & hun & 1 & Latin \\ \hline \hline \end{tabular}
\end{table}
Table 7: Summary of languages covered in SPARROW. **Lang.:** Language code is marked by ISO 639-3 code. Language information is retrieved from Ethnologue (Gordon Jr, 2005) and Glottolog (Nordhoff and Hammarström, 2011). The column **# dataset** shows the number of datasets covered by SPARROW per language. **Hir.:** Hiragana, **Kat.:** Katakana

\begin{table}
\begin{tabular}{l|r r r r r} \hline \hline
* Jan. 2022 and find that \(42\%\) samples are inaccessible.




\begin{table}
\begin{tabular}{l l c c r r r r|c c c} \hline \hline
**Dataset** & **Study** & **Lang.** & **Source / Domain** & **Year** & **\#Lb** & **Labels** & **Data Slipt** & **Metric** \\ \hline Aggr-InSiRm & Kumar et al.(2018) & bin & Twitter, Facebook & 2018 & 2 & [Aggressive, Net] & 9.306/1.637/1.164 & M-F1 \\ \hline Dang-arnaMh & Alibehti et al.(2020) & ara & Twitter & 2020 & 2 & [Dangerous, Not] & 3,474/156/5663 & M-F1 \\ \hline Hate-engWas & Waseem and Huovy(2016) & eng & Twitter & 2016 & 3 & [Not, Racism, Sexism] & 8.683/1.086/1.085 & W-F1 \\ Hate-engWas & Davidson et al.(2017) & eng & Twitter & 2017 & 3 & [Hate, Not] & 19.826/2.472/4.729 & W-F1 \\ Hate-muMh & Albster et al.(2018) & ara & YouTube comment & 2017 & 2 & [Hate, Not] & 9.014/1.71/1.127 & M-F1 \\ Hate-tioBno* & Boco et al.(2018) & ita & Twitter & 2018 & 2 & [Hate, Not] & 2,700/300/1.000 & M-F1 \\ Hate-tlClo & Cabasag et al.(2019) & fil & Twitter & 2016 & 2 & [Hate, Not] & 10,000/2.423/2.423 & M-F1 \\ Hate-mtCaMh & Multii et al.(2019) & ara & Twitter & 2019 & 3 & [Abusive, Hate, Not] & 4,208/4.086/1.710 & M-F1 \\ Hate-engHaEns & Basile et al.(2019) & eng & Twitter & 2019 & 2 & [Hate, Not] & 9,000/1.000/300/1.000 & M-F1 \\ Hate-paPaPori & Fortuna et al.(2019) & por & Twitter & 2019 & 2 & [Hate, Not] & 4,536/567/567 & M-F1 \\ Hate-polPu & Paszysnski et al.(2019) & pol & Twitter & 2019 & 2 & [Hate, Not] & 9,037/1.004/1.000 & M-F1 \\ Hate-kotMoos & Monan et al.(2020) & koc & News comment & 2020 & 3 & [Hate, Not] & 7,106/7094/1.047 & M-F1 \\ Hate-maMon & Muhar et al.(2020) & ara & Twitter & 2020 & 2 & [Hate, Not] & 6,899/1.000/2.000 & M-F1 \\ Hate-zhOen & Deng et al.(2022) & zho & Weibo & 2022 & 2 & [Hate, Not] & 25,726/6.431/5.232 & M-F1 \\ Hate-korJoo & Jeong et al.(2022) & koc & News and YouTube & & 2022 & 2 & [Hate, Not] & 32,343/0.403/4.043 & M-F1 \\  & & & & & & & & & \\ Hate-telMarke & Marreddly et al.(2022) & tel & Misce & 2022 & 2 & [Hate, Not] & 24,599/3.5107/0.033 & M-F1 \\ Sexi-trCls & Chini et al.(2020) & fra & Twitter & 2018 & 2 & [Not, Sexism] & 7,670/959/959 & M-F1 \\ \hline Offe-engZam & Zampieri et al.(2019) & eng & Twitter & 2019 & 2 & [Not, Offensive] & 11,916/1.324/860 & M-F1 \\ Offe-argZam & Zampieri et al.(2020) & ara & Twitter & 2019 & 2 & [Not, Offensive] & 7,055/784/1.827 & M-F1 \\ Offe-dazzam & Zampieri et al.(2020) & dan & Misce & 2019 & 2 & [Not, Offensive] & 2,664/296/239 & M-F1 \\ Offe-ellZam & Zampieri et al.(2020) & ell & Twitter & 2019 & 2 & [Not, Offensive] & 7,899/874/1.544 & M-F1 \\ Offe-tuzzEnt & Zampieri et al.(2020) & tur & Twitter & 2019 & 2 & [Not, Offensive] & 28,149/3.128/5.515 & M-F1 \\ Offe-arsha & Mbursak et al.(2020) & ara & Twitter & 2020 & 2 & [Not, Offensive] & 6,839/1.002/.000 & M-F1 \\ Offe-slsync & Kralj Nowak et al.(2021) & slv & Twitter & 2020 & 4 & [Appropriate, Inappropriate, Not, Offensive] & 65,012/1.827/1.828 & M-F1 \\ \hline Offe-G-engZam & Zampieri et al.(2019) & eng & Twitter & 2019 & 3 & [Group, Individual, Others] & 3,485/391/213 & M-F1 \\ Hate-G-araOus & Ouishduum et al.(2019) & ara & Twitter & 2019 & 13 & [African, descent, Arab, Asians, Christian, Gay, Dimmigrants, Indian, Individual, Jews, Left_wing_people, Muslim, Obres, Refugees, Special_needs, Women] & 3,485/391/213 & M-F1 \\ Hate-T-ringCon & Ouishduum et al.(2019) & fra & Twitter & 2019 & 16 & [Alrravan,descent, Arabs, Asians, Christian, Gay, Gispans, Immorrisants, India/handia, Individuals, Lewis,Left_wing_people, Muslim, Obres, Refugees, Special_needs, Women] & 3,211/401/402 & M-F1 \\ \hline Office-TenzEm & Zampieri et al.(2019) & eng & Twitter & 2019 & 2 & [Targeted, Untargeted] & 3,963/437/240 & M-F1 \\ Hate-T-ztApIons & Ouishduum et al.(2019) & ara & Twitter & 2019 & 4 & [Grander, Origin, Others, Religion] & 2,682/334/336 & M-F1 \\ Hate-T-fr+PoMaN & Ouishduum et al.(2019) & fra & Twitter & 2019 & 6 & [Disability, Gender, Onigin, Others, Religion, Sexual, Orientation] & 3,211/401/402 & M-F1 \\ Hate-T-benSaff & Karim et al.(2021) & ben & Misce & 2020 & 4 & [Geopolitical, Personal, Political, Religian] & 4,558/5705/70 & M-F1 \\ Offe-T-KancLu & Chakravarthi et al.(2022) & kan & YouTube comment & 2019 & 5 & [Group, Individual, Not, Others, Untargeted] & 4,694/586/593 & M-F1 \\ Offe-T-malCu & Chakvararthi et al.(2022) & mal & YouTube comment & 2019 & 4 & [Group, Individual, Not, Untargeted] & 14,723/1.836/1.844 & M-F1 \\ Offe-TeimmCu & Chakravarthi et al.(2022) & tam & YouTube comment & 2019 & 5 & [Group, Individual, Not, Others, Untargeted] & 33,685/4.2164/2.322 & M-F1 \\ Hate-T-korJoo & Jeong et al.(2022) & koc & News and YouTube comment & 2022 & 4 & [Group, Individual, Other, Uttargeted] & 16,239/2.049/2.022 & M-F1 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Description of \(36\) antisocial language detection datasets. **Lang.:** Language is marked by ISO 639-3, **#Lb:** the label size of a dataset. **M-F1:** Macro-F1, **W-F1:** Weighted-F1. \({}^{\star}\) indicates that data sharing needs approval from the original authors.





\begin{table}
\begin{tabular}{l l c c c c|c c c c}
**Dataset** & **Study** & **Lang.** & **Source / Domain** & **Year** & **H.D.** & **Labels** & **Data Slipt** & **Metric** \\ \hline Emo-eng\({}_{\text{Full}}\) & Wallbott and Scheerer (1986) & eng & Questionnaire & 1986 & 7 & [Auger, Digusut, Fear, Guilt, Joy, Saduras, Shame] & 6,13276/767 & M-F1 \\ Emo-db\({}_{\text{Low}}\) & Lee and Wang (2015) & zho & Weibo & 2015 & 5 & [Auger, Fear, Happy, Saduras, Surprise] & 3,1223/47418 & Accuracy \\ Emo-dm\({}_{\text{High}}\) & Kajura (2018) & fin & Skettle & 2016 & 8 & [Auger, Anticipation, Digusat, Fear, Joy, Saduras, Suprise, Trust] & 5,1975/77653 & M-F1 \\ Emo-frg\({}_{\text{CBI}}\) & Kajura (2018) & fra & Skettle & 2016 & 8 & [Auger, Anticipation, Digusat, Fear, Serups, Trust] & 5,1975/77653 & M-F1 \\ Emo-id\({}_{\text{High}}\) & Kajura (2018) & ta & Skettle & 2016 & 8 & [Auger, Anticipation, Digusat, Fear, Serups, Trust] & 5,1975/77653 & M-F1 \\ Emo-ars\({}_{\text{Alld}}\) & Abdul-Mageed et al. (2020) & ara & Twitter & 2016 & 8 & [Auger, Anticipation, Digusat, Fear, Serups, Trust] & 50,0009/10941 & M-F1 \\ Emo-enrega\({}_{\text{Medium}}\) & Mohammad et al. (2018) & eme & Twitter & 2018 & 4 & [Auger, Joy, Optimations, Schwets] & 3,2572/471421 & M-F1 \\ Emo-sr\({}_{\text{High}}\) & Mohammad et al. (2018) & asra & Twitter & 2018 & 4 & [Auger, For, Few, Saduras] & 2,2684/901818 & M-F1 \\ Emo-m\({}_{\text{High}}\) & Subramil et al. (2018) & ja & Twitter & 2018 & 4 & [Auger, Jorg, Fear, Saduras] & 2,2084/90186 & M-F1 \\ Emo-mr\({}_{\text{Upper}}\) & Guyon et al. (2020) & tar & Twitter & 2019 & 5 & [Auger, Fear, Hagley, Saduras, Surprise] & 3,2004/0400 & Accuracy \\ Emo-in\({}_{\text{high}}\) & Wilie et al. (2020) & ind & Twitter & 2018 & 5 & [Auger, Fear, Happy, Low, Sadness] & 3,1693/352440 & M-F1 \\ Emo-irgs\({}_{\text{Lio}}\) & Ho et al. (2019) & via & Facebook & 2019 & 7 & [Auger, Inspati, Fact, Joy, Others, Sudaries, Simple] & 5,5486/866693 & W-F1 \\ Emo-esrg\({}_{\text{An}}\) & Plaza del Aaco et al. (2020) & eme & Twitter & 2020 & 7 & [Auger, Unspati, Fact, Joy, Others, Sudares, Surprise] & 5,8427/30731 & M-F1 \\ Emo-epgr\({}_{\text{A}}\) & Plaza del Aaco et al. (2020) & spa & Twitter & 2020 & 7 & [Auger, Dispati, Fact, Joy, Others, Sudares, Surprise] & 6,727/841841 & M-F1 \\ Emo-fring\({}_{\text{Onman}}\) & Ohman et al. (2020) & fin & Skettle & 2020 & 8 & [Auger, Anticipation, Digusat, Fear, Joy, Sadnes, Surprise, Trust] & 8,864/1,1187/086 & M-F1 \\ Emo-eps\({}_{\text{Inn}}\) & Dermaddy et al. (2020) & eng & Reddit & 2020 & [Admintion, Amennment, Auger, Annymace, Approval, Carting, Conscious, Design, Disappointement, Discrupointement, Discrupointement, Discrupointement, Excitement, Fear, Gradits, Gerber, Joy, Lowe, Nervousues, Outset, Dirk, Reakt, Resistance, Refier, Griers, Schades, Surprise] & 1,629/204/024 & M-F1 \\ Emo-tag\({}_{\text{Min}}\) & Bianchi et al. (2021) & ia & Twitter & 2021 & 4 & [Auger, Four, Jay, Sadnes] & 1,629/204/024 & M-F1 \\ Emo-enc\({}_{\text{Cor}}\) & Colobarts and Dina (2021) & rou & Twitter & 2020 & 4 & [Auger, Four, Joy, Sadnes] & 2,6003/813/024 & M-F1 \\ Emo-himpub\({}_{\text{Shone}}\) & Shoev (2021) & lin & Machine Translation & 2021 & 27 & [Admintion, Amennment, Angry, Amphone, Appraisal, Carting, Conscious, Design, Disappointement, Dissapointement, Escientement, Pear, Gradits, Goset, Joy, Love, Nervousues, Outset, Dirk, Reakt, Resistance, Refer, Griers, Schades, Surprise] & 2,485/2562/884 & M-F1 \\ Emo-perCore & Cortiz et al. (2021) & por & Twitter & 2021 & 8 & [Admintion, Amineinetient, Agrey, Amphone, Appraisal, Comission, Confusion, Encous, Design, Disappointement, Disapproointement, Denysport, Disappointement, Envy, Exoticament, Fear, Grinditude, Grief, Joy, Longning, Lowe, Newosomes, Optimum, Prike, Reject, Remove, Salenes, Surprise] & 4,1802/3523 & M-F1 \\ Emo-fring\({}_{\text{Sub}}\) & Sabiv et al. (2020) & fus & Twitter & 2021 & 6 & [Auger, Fear, Hayred, Hard, Sadnes, Woodard] & 4,1807/352/523 & M-F1 \\ Emo-resp\({}_{\text{Body}}\) & Shoev et al. (2020) & ruu & Mice & 2021 & 5 & [Auger, Fear, Joy, Sadnes, Surprise] & 3,9514/4277/1,128 & M-F1 \\ Emo-sheng\({}_{\text{Expl}}\) & kjbal et al. (2022) & ben & Mice & 2022 & 6 & [Auger, Benign, Fear, Joy, Sadnes, Surprise] & 5,6007/09070 & M-F1 \\ Emo-feng\({}_{\text{Int}}\)* & Bianchi et al. (2022) & fr & Machine Translation & 2018 & 4 & [Auger, Four, Joy, Sadnes] & 3,7984/76476 & M-F1 \\ Emo-deeufa\({}^{*}\) & Bianchi et al. (2022) & deu & Machine Translation & 2018 & 4 & [Auger, Four, Joy, Sadnes] & 3,7984/76476 & M-F1 \\ \end{tabular}
\end{table}
Table 10: Description of \(26\) emotion recognition datasets. **Lang.:** Language is marked by ISO 639-3, **#LDb:** the label size of a dataset. **M-F1:** Macro-F1.

\begin{table}
\begin{tabular}{l l l l l l l l l l}
**Dataset** & **Study** & **Lang** & **Source / Domain** & **Year** & **H.D.** & **Labels** & **Data Slipt** & **Metric** \\ \hline Emo-im\({}_{\text{High}}\) & Aggurved et al. (2020) & him & Twitter & 2018 & 2 & [Homeot, Not] & 4,1875/324/523 & Accuracy \\ Homo-mpia & Bilov et al. (2019) & rns & Mike & 2018 & 2 & [Homeot, Not] & 2514/1681/7941/877 & M-F1 \\ Homo-mpia & Chuzzo et al. (2021) & spa & Twitter & 2019 & 2 & [Homeot, Not] & 2,0005/0009/0000 & M-F1 \\ Homo-mpia & Meaney et al. (2021) & eng & Twitter & 2021 & 2 & [Humanot, Not] & 8,0007/1001/0000 & M-F1 \\ \end{tabular}
\end{table}
Table 11: Description of four humor detection datasets. **Lang.:** Language is marked by ISO 639-3, **#LDb:** the label size of a dataset. **M-F1:** Macro-F1.





\begin{table}
\begin{tabular}{l l c r r r r|r r r} \hline
**Dataset** & **Study** & **Lang.** & **Source / Domain** & **Year** & **\#Lb** & **Labels** & **Data Slipt** & **Metric** \\ \hline Iron-tiqaa & Basile et al.(2014) & ita & Twitter & 2014 & 2 & [Iromy, Not] & & 4,0624/3591.936 & M-F1 \\ Iron-rqaa & Barbieri et al.(2016) & spra & Twitter & 2014 & 2 & [Iromy, Not] & & 6,8697/411.997 & M-F1 \\ Iron-epnscu & Van Hee et al.(2018) & eng & Twitter & 2018 & 2 & [Iromy, Not] & 3,4503/874 & F1-trony \\ Iron-tiqac & Cignarelta et al.(2018) & ita & Twitter & 2018 & 2 & [Iromy, Not] & 3,5793/872 & M-F1 \\ Iron-tiqav & Vijay et al.(2018) & him & Twitter & 2018 & 2 & [Iromy, Not] & 2,2172/7777 & M-F1 \\ Iron-mqcaa & Ghanem et al.(2019) & ara & Twitter & 2019 & 2 & [Iromy, Not] & 3,624/201/1.006 & M-F1 \\ Iron-fapaon & Ortega-Buono et al.(2019) & spra & Twitter & 2019 & 2 & [Iromy, Not] & 2,1602/20606 & M-F1 \\ Iron-facade* & Golazizian et al.(2020) & fas & Twitter & 2019 & 2 & [Iromy, Not] & 2,352/952/924 & Accuracy \\ Iron-chovia* & Xiang et al.(2020) & zho & Weibo & 2020 & 5 & [Insufficient_Evidence, Irw, Not, Unlikely\_Tronic, Weakly\_Torny] & 7,014/876/876 & M-F1 \\  & & & & & & & & \\ \hline Sarc-epspi & Waller et al.(2012) & eng & Debate Forum & 2012 & 2 & [Not, Saracsm] & 900/100/995 & M-F1 \\ Sarc-epspi & Rindler et al.(2013) & eng & Twitter & 2013 & 2 & [Not, Saracsm] & 1,4317/1777/17 & F1-strasm \\ Sarc-epspi & Plack et al.(2014) & cres & Twitter & 2013 & 2 & [Not, Saracsm] & 3,774/947/947 & M-F1 \\ Sarc-epspi & Prakic et al.(2014) & eng & Twitter & 2013 & 2 & [Not, Saracsm] & 17,438/989/830 & M-F1 \\ Sarc-epspi & Bamman and Smith(2015) & eng & Twitter & 2015 & 2 & [Not, Saracsm] & 1,8601/483/1.484 & Accuracy \\ Sarc-epspi & Rajadesgani et al.(2015) & eng & Twitter & 2015 & 2 & [Not, Saracsm] & 41,261/1585/158 & Accuracy \\ Sarc-epspi+coq & Oralby et al.(2016) & eng & Debate Forum & 2016 & 2 & [Not, Saracsm] & 900/100/2260 & M-F1 \\ Sarc-china* & Gong et al.(2020) & zho & News comment & 2019 & 2 & [Not, Saracsm] & 3,798/847/947/947 & M-F1 \\ Sarc-anuAshu & Abu Farha and Magdy(2020) & ara & Twitter & 2020 & 2 & [Not, Saracsm] & 7,593/844/21.10 & M-F1 \\ Sarc-antpre & Furha et al.(2021) & ara & Twitter & 2020 & 2 & [Not, Saracsm] & 11,293/1.255/3,000 & M-F1 \\ Iron-T-engrBase & Van Hee et al.(2018) & eng & Twitter & 2018 & 4 & [Ironic_by_clash, Not, Other_inuty, Sitational_tropy] & 3,450384/784 & M-F1 \\  & & & & & & & \\ \end{tabular}
\end{table}
Table 12: Description of \(20\) irony and sarcasm detection datasets. **Lang.:** Language is marked by ISO 639-3, **#Lb:** the label size of a dataset. **M-F1:** Macro-F1. \({}^{*}\) indicates that data sharing needs an approval from the original authors.





\begin{table}
\begin{tabular}{l l c r r r r r r r r}
**Dataset** & **Study** & **Lang.** & **Source/Domain** & **Year** & **ff.b.** & **Labels** & **Data Slipt** & **Metric** \\ \hline Sent-org**me** & Pang and Lee (2005) & eng & Movie review & 2005 & 2 & [Negative, Positive] & 8,5291,0667,1067 & Accuracy \\ Sent-off**me** & Tan and Zhang (2008) & rho & Mice & 2008 & 2 & [Negative, Positive] & 9690,12001,200 & M-F1 \\ Sent-erg**me** & Thlehall et al. (2012) & eng & Twitter & 2012 & 2 & [Negative, Positive] & 9004,1131, AFCuracy \\ Sent-Yeneg** & Thehual et al. (2012) & eng & YouTube comment & 2012 & 2 & [Negative, Positive] & 90091001,142 & Accuracy \\ Sent-sem**eps** & Socher et al. (2013) & eng & Movie review & 2013 & 5 & [Negative, Neutral, Positive] & 8,544,1101/2,210 & Accuracy \\  & & & & & & Very_Positive & & \\ Sent-corr**n* & Jang et al. (2013) & ker & News article & 2013 & 4 & [Complete, Negative, Neutral, Positive] & 4,187523524 & M-F1 \\ Sent-eff**sco** & Socher et al. (2013) & eng & Movie review & 2013 & 2 & [Negative, Positive] & 6,9208721,821 & Accuracy \\ Sent-inf**sco** & Snule et al. (2014) & its & Twitter & 2014 & 2 & [Negative, Positive] & 2,7626521,207 & M-F1 \\ Sent-info** & Barbieri et al. (2016) & ita & Twitter & 2016 & 2 & [Negative, Positive] & 3,73844161,018 & M-F1 \\ Sent-info** & Dingli and Sant (2016) & mt & Movie review & 2016 & 2 & [Negative, Positive] & 59687881,M & M-F1 \\ Sent-shot** & Material et al. (2016) & bd & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 2,1874,7732,773 & M-F1 \\ Sent-sub** & Morelli et al. (2016) & bcs & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 16,35272242,047 & M-F1 \\ Sent-ads** & Morelli et al. (2016) & des & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 4,2019,52452,5252 & M-F1 \\ Sent-emb** & Morelli et al. (2016) & mg & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 4,548,137413,118 & M-F1 \\ Sent-spit** & Morelli et al. (2016) & bye & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 122,419,5015,10320 & M-F1 \\ Sent-strv** & Morelli et al. (2016) & hvc & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 52,971,6216,62 & M-F1 \\ Sent-sm**ss** & Morelli et al. (2016) & hsa & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 7,419,48940 & M-F1 \\ Sent-plk** & Morelli et al. (2016) & pd & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 97,941,09931,9992 & RFI \\ Sent-ep** & Mazuela et al. (2016) & pod & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 3,524,941,4940 & M-F1 \\ Sent-stv** & Mazuela et al. (2016) & ns & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 3,24,914,1174,118 & M-F1 \\ Sent-slk** & Mazuela et al. (2016) & ski & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 3,694,93737,337 & MF-F1 \\ Sent-srv** & Mazuela et al. (2016) & syu & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 3,957,4797,4790 & MF-F1 \\ Sent-sph** & Mazuela et al. (2016) & sei & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 29,757,782/72,7267 & MF-F1 \\ Sent-syn**ss** & Morelli et al. (2016) & sp & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 22,142,762,762,762 & MF-F1 \\ Sent-sync** & Morelli et al. (2016) & spe & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 2,77,277,089,410 & MF-F1 \\ Sent-sync** & Refi et al. (2016) & das & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 2,721,731,7313 & MF-F1 \\ Sent-sync** & Refi et al. (2016) & pba & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 2,701,731,731,731 & MF-F1 \\ Sent-sync** & Refi et al. (2016) & sta & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 3,699,783,762 & MF-F1 \\ Sent-sync** & Refi et al. (2016) & tia & Twitter & 2016 & 3 & [Negative, Neutral, Positive] & 6,818,935,762 & MF-F1 \\ Sent-cr**ss** & Rouchowski et al. (2017) & oog & Twitter & 2017 & 3 & [Negative, Neutral, Positive] & 42,756,744,712,284 & Me-Roodes \\ Sent-wr** & Patra et al. (2018) & ben & Twitter & 2015 & 3 & [Negative, Neutral, Positive] & 2,252,0520/3,528 & M-F1 \\ Sent-sn**ss** & Pats et al. (2018) & lin & Twitter & 2015 & 3 & [Negative, Neutral, Positive] & 11,642,129,352,525 & MF-F1 \\ Sent-bc**ss** & Arment et al. (2018) & bcb & Facebook & 2018 & 2 & [Negative, Positive] & 8,591,999,983,683 & Across \\ Sent-per** & Román and Grosse** & por & Twitter & 2017 & 3 & [Negative, Neutral, Positive] & 48,127,5585/537 & MF-F1 \\  & Joyce Nunes (2018) & fa & Sibolite & 2016 & 2 & [Negative, Positive] & 5,197572663 & MF-F1 \\ Sent-fm**ss** & Kajpars (2018) & fa & Sibolite & 2016 & 2 & [Negative, Positive] & 5,198,97572653 & MF-F1 \\ Sent-tr**ss** & Kajpars (2018) & fa & Sibolite & 2016 & 2 & [Negative, Positive] & 5,198,97572653 & MF-F1 \\ Sent-ar**ss** & Kabara (2018) & ia & Siddieu & 2016 & 2 & [Negative, Positive] & 5,4,579,78727653 & MF-F1 \\ Sent-error** & Vudski et al. (2018) & nor & Online review & 2018 & 6 & [Negative, Negative], Negret,atives & 3,904,934,964,951,51 & SF-T1 \\  & & & & & attreps, Possilet, Positives/5 & & \\ Sent-pol**cco** & Kosci et al. (2019) & pol & Customer review & 2019 & 4 & [Complete, Negative, Neutral, Positive] & 5,17,3075741,217 & MF-F1 \\ Sent-bl**ss** & Samiyaswandaki et al. (2019) & tha & Facebook & 2019 & 3 & [Negative, Neutral, Positive] & 21,152,362,62,614 & MF-F1 \\ Sent-obs**ss** & Suniyaswandaki et al. (2020) & who & Weibo & 2019 & 2 & [Negative, Neutral, Positive] & 95,990,991,9991,999,991 & MF-F1 \\ Sent-ss**ss** & Adharfi



\begin{table}
\begin{tabular}{l l c c c|c c c c} \hline \hline
**Dataset** & **Study** & **Lang.** & **Source/Domain** & **Year** & **fLb.** & **Labels** & **Data Slip** & **Metric** \\ \hline Sabj+argm\({}_{\text{lang}}\) & Pang and Lee Lee [2004] & eng & Moving review & 2004 & 2 & [Objective, Subjective] & 8.1009001,000 & Accuracy \\ Sabj+acr\({}_{\text{31a}}\) & Jang et al. [2013] & kar & News article & 2013 & 7 & [Agreement, Argument, Argument, & 4.2845/55356 & M-F1 \\  & & & & & Emerson, Intention, Judge- & & Emerson, Intention, Judge- & \\ Sabj+n\({}_{\text{lang}}\) & Basile et al. [2014] & itia & Twitter & 2014 & 2 & [Objective, Subjective] & 4.0614/852/935 & M-F1 \\ Subj+n\({}_{\text{lang}}\) & Barbieri et al. [2016] & its & Twitter & 2016 & 2 & [Objective, Subjective] & 6.6967/41/943 & M-F1 \\ Sibj+spike & Barbieri et al. [2016] & spra & Twitter & 2014 & 2 & [Objective, Subjective] & 6.6967/41/983 & M-F1 \\ Sabi+exp\({}_{\text{31b}}\) & Polish and Steinberg [2022] & cos & Movie review & 2021 & 2 & [Objective, Subjective] & 7.5005/002/000 & Accuracy \\ \hline \hline \end{tabular}
\end{table}
Table 14: Description of four subjectivity analysis datasets. **Lang.** Language is marked by ISO 639-3, **#LDb:** the label size of a dataset. **M-F1:** Macro-F1. \({}^{*}\) indicates that data sharing needs an approval from the original authors.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline




\begin{table}
\begin{tabular}{l l c c c c c c c c c c c c c} \hline \multicolumn{12}{c}{**Dev Set**} & \multicolumn{6}{c}{**Test-S5**} \\ \cline{3-14}  & mBERT & XLM+R & Berrite & InfoDCL & mBERT & XLM+R & Berrite & InfoDCL \\ \hline \multirow{9}{*}{Artificial Attack} & Aggressive & 73.39\(\pm\)0.30 & 73.92\(\pm\)0.50 & 76.79\(\pm\)0.52 & 76.09\(\pm\)0.38 & 72.71\(\pm\)1.92 & 74.64\(\pm\)0.14 & 75.45\(\pm\)0.73 & 73.96\(\pm\)0.91 \\  & Dangerous & 69.76\(\pm\)1.6 & 69.33\(\pm\)1.04 & 69.42\(\pm\)1.88 & 73.42\(\pm\)0.82 & 63.26\(\pm\)1.08 & 68.73\(\pm\)1.15 & 67.31\(\pm\)0.66 & 75.31\(\pm\)0.66 & 75.31\(\pm\)0.66 \\  & Rate & 77.73\(\pm\)0.76 & 79.40\(\pm\)0.58 & 81.16\(\pm\)1.15 & 80.06\(\pm\)0.60 & 2.97\(\pm\)1.20 & 74.37\(\pm\)1.48 & 76.76\(\pm\)2.43 & 75.85\(\pm\)0.90 \\  & Offense & 78.96\(\pm\)1.04 & 80.21\(\pm\)0.92 & 85.12\(\pm\)0.68 & 85.15\(\pm\)0.66 & 88.55\(\pm\)0.46 & 77.53\(\pm\)1.27 & 78.88\(\pm\)2.43 & 76.45\(\pm\)1.89 & 78.88\(\pm\)2.63 \\  & HO-Group & 51.57\(\pm\)2.33 & 41.42\(\pm\)1.36 & 48.30\(\pm\)1.90 & 46.22\(\pm\)0.45 & 46.18\(\pm\)0.42 & 43.29\(\pm\)3.30 & 51.15\(\pm\)2.01 & 50.44\(\pm\)1.99 & 50.24\(\pm\)1.99 \\  & HO-Target & 40.21\(\pm\)3.00 & 50.92\(\pm\)1.17 & 40.83\(\pm\)1.26 & 60.41\(\pm\)1.21 & 36.49\(\pm\)1.21 & 36.49\(\pm\)0.77 & 75.71\(\pm\)1.99 & 40.96\(\pm\)2.06 & 60.97\(\pm\)1.38 \\  & AS & 70.15\(\pm\)5.9 & 71.50\(\pm\)1.74 & 71.50\(\pm\)1.74 & 73.31\(\pm\)0.45 & 76.63\(\pm\)2.79 & 76.92\(\pm\)1.78 & 71.74\(\pm\)1.25 & 76.71\(\pm\)1.64 \\ \hline Emotion & A & 63.09\(\pm\)0.90 & 67.43\(\pm\)0.68 & 68.50\(\pm\)0.58 & 69.44\(\pm\)0.53 & 64.12\(\pm\)1.51 & 68.73\(\pm\)0.99 & 68.13\(\pm\)1.28 & 69.27\(\pm\)1.03 \\ \hline Human & S & 85.16\(\pm\)0.32 & 85.38\(\pm\)0.57 & 85.22\(\pm\)0.63 & 86.74\(\pm\)0.42 & 84.35\(\pm\)1.21 & 85.19\(\pm\)1.62 & 86.75\(\pm\)1.13 & 87.05\(\pm\)0.82 \\ \hline \multirow{3}{*}{IKS} & Irony & 69.16\(\pm\)1.29 & 69.94\(\pm\)1.02 & 72.19\(\pm\)1.24 & 71.12\(\pm\)0.72 & 64.24\(\pm\)1.16 & 65.63\(\pm\)1.57 & 66.88\(\pm\)1.23 & 63.63\(\pm\)1.00 \\  & Succession & 46.55\(\pm\)1.14 & 87.41\(\pm\)1.92 & 78.21\(\pm\)1.56 & 77.21\(\pm\)1.56 & 77.21\(\pm\)1.48 & 77.49\(\pm\)2.42 & 74.78\(\pm\)1.69 & 7.49\(\pm\)1.13 \\  & Iony-Type & 53.11\(\pm\)2.11 & 53.28\(\pm\)1.25 & 55.85\(\pm\)1.37 & 77.21\(\pm\)1.92 & 77.45\(\pm\)1.89 & 6.33\(\pm\)0.63 & 56.01\(\pm\)1.77 & 66.95\(\pm\)1.29 & 7.58\(\pm\)1.42 \\  & RS & 71.13\(\pm\)2.66 & 79.01\(\pm\)5.25 & 74.32\(\pm\)1.30 & 73.49\(\pm\)1.00 & 76.74\(\pm\)1.67 & 63.74\(\pm\)1.95 & 70.52\(\pm\)1.69 & 7.11\(\pm\)2.10 \\ \hline Sentiment & 69.99\(\pm\)1.14 & 71.34\(\pm\)1.76 & 77.23\(\pm\)0.38 & 77.31\(\pm\)0.80 & 66.63\(\pm\)1.92 & 74.01\(\pm\)0.92 & 74.04\(\pm\)1.61 & 77.58\(\pm\)1.33 \\ \hline Subjectivity & 75.18\(\pm\)0.63 & 77.78\(\pm\)0.74 & 76.97\(\pm\)0.69 & 77.78\(\pm\)0.87 & 72.54\(\pm\)1.46 & 74.45\(\pm\)1.18 & 74.80\(\pm\)1.08 & 75.73\(\pm\)1.33 \\ \hline SM & & 69.29\(\pm\)1.17 & 71.50\(\pm\)0.93 & 73.16\(\pm\)0.98 & 73.46\(\pm\)0.72 & 66.60\(\pm\)1.83 & 69.38\(\pm\)1.51 & 70.85\(\pm\)1.63 & 71.60\(\pm\)1.30 \\ \hline \end{tabular}
\end{table}
Table 16: Performance of finetuned models on Dev and Test-S set. We fineute each model on dataset for three runs with different random seeds and calculate the mean and deviation of dataset-specific metrics over the three runs. We report the average of dataset-specific metrics and standard deviation in a task and a category.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c c} \hline \hline
**Dataset** & **Movie** & **Random** & **mBERT** & **XLM+R** & Berrite** & **InfoDCL** & **HOOTAL** & **FBOME** & **EMODL** & **HOOTAL** & **noFF** & **all-Action** & **Apps** & **Value** & **CoDFT** & **Self-Action** \\ \hline Zero-Image & MF1 & 12.0 & 45.3 & 10.97 & 72.04 & **79.45** & 73.76 & 51.08 & 55.22 & 18.72 & 18.72 & 55.53 & 12.20 & 18.31 & 69.29 & 12.07 & 69.53 & 13.40 & 30.98 & Game +/-2.019 \\ Data-driven & MF1 & 2, 49.6 & 72.0 & **73.73** & 63.23 & 64.07 & 68.57 & 50.84 & 84.67 & 65.03 & 68.47 & 68.57 & 68.57 & 68.67 & 35.79 & 37.06 & 22.84 & Global-Global-2.02 \\  & Moi-type & MF1 & 45.1 & 51.72 & **70.03** & **83.93** & 89.70 & 89.26 & 53.96 & 66.06 & 60.07 & 1.073 & 50.95 & 97.93 & 69.07 & 51.08 & 69.78 & 79.06 & 28.51 & Global-Global-2.02 \\  & H-class & MF1 & 45.3 & 11.28 & **70.03** & **83.93** & 89.92 & 89.12 & 7.45 & 99.12 & 7.08 & 7.52 & 5.03 & 7.14 & 67.07 &



\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c c c c c c c}
**Dataset** & **Movie** & **Random** & **anBERT** & **XL-R** & **Bronix** & **IndDCCAL** & **BLOOM** & **BLOOME** & **BLOOME** & **BLOOME** & **MLBOME** & **MLBOME** & **MLBOME** & **MLBOME** & **VAT5** & **m/s** & **mt/s** \\  &



\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c c c c c c c}
**Dataset** & **Movie** & **Random** & **mNET** & **X1.0-R** & **Bounce** & **LoDST** & **BOOM** & **HO



\begin{table}
\begin{tabular}{c c l|r r r r|r r r} \hline
**Lang Fam.** & **Lang** & **Random** & **InfoDCL** & **BMZ-PY** & **mT0** & **Vienna** & **CG** & **CG-MT** \\ \hline  & ara & 34.05 & **73.53** & 36.78 & 35.61 & 33.61 & 60.81 & 52.53 \\  & amh & 37.95 & **65.68** & 16.05 & 22.49 & 2.99 & 20.62 & **64.82** \\  & argu & 34.23 & **71.25** & 52.02 & 18.05 & 5.33 & 63.89 & **67.58** \\ Afro-Asiatic & any & 35.94 & **53.44** & 37.40 & 23.41 & 16.64 & 52.19 & 51.66 \\  & hau & 35.22 & **72.18** & 30.14 & 20.93 & 17.39 & 55.52 & 34.13 \\  & heb & 47.60 & **95.80** & 71.20 & _76.60_ & 48.00 & 54.20 & 57.40 \\  & mtlt & 47.51 & 68.01 & 47.70 & 39.25 & 48.98 & **78.47** & **77.45** \\ \hline  & bam & 27.61 & **65.57** & 36.70 & 24.31 & 14.40 & 40.27 & 37.11 \\  & ibo & 32.90 & **76.75** & 23.21 & 11.52 & 28.37 & 57.55 & 33.46 \\ Atlantic-C. & kin & 35.83 & **56.69** & 28.06 & 14.84 & 22.23 & 53.78 & 29.03 \\  & swh & 33.72 & **61.29** & 17.60 & 14.02 & 45.55 & 54.39 & 53.84 \\  & twi & 34.04 & **64.51** & 46.56 & 29.34 & 5.28 & 51.12 & 32.06 \\  & toq & 34.03 & **52.55** & 45.84 & 30.74 & 6.50 & 42.58 & 35.70 \\  & yor & 41.68 & **74.88** & 43.10 & 35.83 & 26.38 & 56.47 & 42.31 \\ \hline Austrosai. & vie & 16.12 & **64.58** & 21.22 & 27.81 & 9.52 & 54.69 & 33.96 \\ \cline{2-10}  & ace & 34.78 & **77.36** & 37.89 & 24.73 & 12.79 & 52.63 & 58.05 \\  & ban & 30.14 & **79.49** & 41.09 & 29.91 & 13.82 & 60.01 & 42.28 \\  & bijn & 30.77 & **84.50** & 50.51 & 27.78 & 14.89 & 60.34 & 15.43 \\  & bug & 30.77 & **71.55** & 34.60 & 18.27 & 12.90 & 34.63 & 30.86 \\  & fil & 52.37 & **79.01** & 34.47 & 34.47 & 34.47 & 69.13 & 66.67 \\ Austrones. & ind & 22.85 & **83.05** & 42.86 & 36.77 & 20.69 & **75.29** & 64.14 \\  & jav & 31.62 & **84.79** & 48.06 & 37.65 & 15.21 & 73.03 & **71.56** \\  & mud & 28.64 & **78.36** & 45.44 & 21.85 & 13.14 & 61.07 & 61.14 \\  & min & 34.41 & **84.07** & 49.93 & 32.41 & 14.95 & 69.80 & 62.91 \\  & mij & 34.86 & **77.22** & 42.86 & 22.89 & 15.21 & 57.64 & 57.07 \\  & sum & 32.18 & **81.71** & 44.83 & 37.65 & 12.93 & 64.97 & 68.76 \\  & blbc & 30.60 & **73.58** & 36.42 & 19.20 & 13.86 & 38.43 & 40.65 \\ \hline  & kan & 32.14 & **61.79** & 39.23 & 27.80 & 19.12 & 44.81 & 30.39 \\ Dravidian & mal & 31.68 & **82.70** & 43.84 & 41.65 & 24.58 & **44.03** & 31.44 \\  & tram & 29.45 & **57.81** & 38.53 & 34.27 & 17.60 & 48.85 & 33.20 \\  & tol & 32.29 & **59.61** & 40.99 & 35.15 & 36.61 & 53.33 & 38.68 \\ \hline  & sqi & 29.39 & **47.04** & 31.78 & 26.59 & 16.11 & 46.82 & 33.84 \\  & bos & 34.37 & **63.31** & 38.01 & 21.16 & 16.72 & 63.22 & 48.15 \\  & ball & 32.48 & **65.09** & 27.35 & 19.37 & 23.69 & 59.32 & 54.39 \\  & benn & 24.53 & **69.49** & 24.71 & 25.18 & 15.62 & 53.25 & 37.33 \\ hrv & 30.23 & **67.83** & 35.18 & 27.47 & 13.67 & 62.02 & 55.66 \\  & cres & 41.06 & **80.15** & 50.91 & 46.61 & 51.00 & 56.80 & 63.24 \\  & dan & 41.14 & **82.09** & 46.68 & 46.68 & 51.84 & 66.91 & 66.79 \\  & eng & 37.90 & **75.48** & 43.32 & 39.23 & 39.75 & 66.51 & — \\ fra & 24.65 & **66.58** & 23.29 & 32.20 & 30.46 & 62.08 & 56.84 \\  & deu & 24.40 & **67.55** & 20.00 & 26.51 & 29.31 & 52.24 & 35.41 \\  & ell & 41.24 & **79.13** & 46.71 & 45.47 & 48.21 & 60.94 & 34.98 \\ Indo-Euro. & lin & 35.24 & **67.55** & 28.92 & 26.20 & 29.06 & 52.63 & 48.30 \\ Irido-Euro. & ita & 39.83 & **74.78** & 38.76 & 41.45 & 45.05 & 70.29 & 64.37 \\  & mar & 29.60 & **86.47** & 43.80 & 35.40 & 34.40 & 68.60 & 50.20 \\  & pcm & 33.36 & **66.65** & 47.16 & 24.25 & 13.92 & 61.13 & 42.75 \\ nor & i



\begin{table}
\begin{tabular}{l c r r r r|r r r r r r} \hline \hline \multirow{2}{*}{**Task**} & \multicolumn{8}{c}{**Int-evaluation-harness Prompts**} \\ \cline{3-14}  & **BLOOM** & **BLOOMZ** & **mT5** & **mtD** & **LAaMA** & **Vicuna** & **BLOOM** & **BLOOMZ** & **mT5** & **mtD** & **LLaMA** & **Vicuna** \\ \hline Hate & 39.83 & 38.52 & 23.29 & 37.33 & 37.80 & **41.59** & 18.39 & 31.34 & 28.96 & 38.19 & 18.37 & 37.33 \\ Emotion & 9.71 & 15.07 & 7.75 & 27.87 & 15.14 & 18.12 & 8.61 & 20.07 & 7.57 & **29.63** & 17.61 & 8.65 \\ Humanor & 41.78 & 33.04 & 43.60 & 33.12 & 39.78 & **46.19** & 41.59 & 44.99 & 41.34 & 34.13 & 41.59 & 33.12 \\ Irony & 36.63 & 44.46 & 36.52 & 34.69 & 40.78 & **47.48** & 27.33 & 26.02 & 36.08 & 34.31 & 26.02 & 34.70 \\
**Average** & 25.21 & 28.01 & 19.58 & 32.12 & 27.72 & 31.80 & 16.92 & 26.14 & 20.91 & **33.21** & 20.95 & 23.03 \\ \hline \end{tabular}
\end{table}
Table 24: Study on model sensitivity to prompts used for zero-shot evaluation.