The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages
Authors: Chiyu Zhang
Khai Duy Doan
Qisheng Liao
Muhammad Abdul-Mageed
Venue: EMNLP 2023 Main (2023)
Keywords: sociopragmatics
benchmark
large language models
social media
ChatGPT
multilinguality
TL;DR: We introduce a massively multilingual evaluation benchmark for sociopragmatics and evaluate a wide range of LLMs on this benchmark.

Abstract: Instruction tuned large language models (LLMs), such as ChatGPT, demonstrate remarkable performance in a wide range of tasks. Despite numerous recent studies that examine the performance of instruction-tuned LLMs on various NLP benchmarks, there remains a lack of comprehensive investigation into their ability to understand cross-lingual sociopragmatic meaning (SM), i.e., meaning embedded within social and interactive contexts. This deficiency arises partly from SM not being adequately represented in any of the existing benchmarks. To address this gap, we present SPARROW, an extensive multilingual benchmark specifically designed for SM understanding. SPARROW comprises 169 datasets covering 13 task types across six primary categories (e.g., anti-social language detection, emotion recognition). SPARROW datasets encompass 64 different languages originating from 12 language families representing 16 writing scripts. We evaluate the performance of various multilingual pretrained language models (e.g., mT5) and instruction-tuned LLMs (e.g., BLOOMZ, ChatGPT) on SPARROW through fine-tuning, zero-shot, and/or few-shot learning. Our comprehensive analysis reveals that existing open-source instruction tuned LLMs still struggle to understand SM across various languages, performing close to a random baseline in some cases. We also find that although ChatGPT outperforms many LLMs, it still falls behind task-specific finetuned models with a gap of 12.19 SPARROW score. Our benchmark is available at: https://github.com/UBC-NLP/SPARROW

The Skipped Beat:
A Study of Sociopragmatie Understanding in LLMs for 64 Languages
Chiyu Zhang Khai Duy Doan Qisheng Li Muhammad Abdul-Mageed
8 Deep Learning & Natural Language Processing Group, The University of British Columbia *Department of Natural Language Processing & Department of Machine Learning, MBZUAI rchiyuzh@maiic muhammat mageede .bc. ca, {duy. doan, qi sheng liao @mbzuai ac.as ae
Abstract
Instruction tuned large language models (LLMs), such ChatGPT, demonstrate markable performance in a wide range of tasks. Despite numerous recent studies that examine the performance of Instruction-tuna L.MSs on various NLP benchmarks, there re- mains lack of comprehensive investigation into their ability to understand cross-lingual sociopragmatic meaning (SM), i.c., meaning embedded within social and interactive texts. This deficiency arises partly from SM not being adequately represented in any of the existing benchmarks. To address this gap, we present SPARROW, an extensive multi- lingual benchmark specifically designed SM understanding. SPARROW comprises 169 datasets covering 13 task types across six primary categories (e.g., anti-social lan- guage detection, emotion recognition). SPAR- ROW datasets encompass different lan- guages originating from 12 language fam- ilies representing 16 writing scripts. We evaluate the performance various multilin- gual pretrained language models (e.g.. mTS) and instruction-tunde LLMSs (e.g., BLOOMZ. CharGPT) on SPARROW through fine-ture tinentuning zero-shot, and/or few-shot learning. Our com- prehensive analysis reveals that existing open- source instruction tuned LLMs still struggle to understand SM across various languages, performing close to random baseline in cases. We also find that although Chat- GPT outperforms many LLMs, it still falls behind task-specific finetuned models with gap of 12.19 SPARROW score. Our bench- mark is available at: https: :ttps:///giiee /ggthtb...b.///// com/ UBC-NLP TCRP//PP /SPARRON
Totter
35
Figure I: Comparison of SM benchmarks with leader- boards. The bubble size indicates the number of
datasets. Previous works: TweetEval (Barbieri et al., 2020), UMSAB (Barbieri et al.. 2022), Nusax (Winata al., 2022), UTCC (Risch et al., 2021), Nai- jaSenti (Muhammad et al., 2022), AfriSenti (Muham- mad et al., 2023a), SentiEval (Zhang et al., 2023b).
wide range of tasks (Xue et al., 2021; Scao et al., 2022). Methods such instruction tun- ing and reinforcement learning from human feed- back (RLHF) (Ouyang al., 2022) have fur- ther enhanced the zero-shot generalizability of these models. Notably, ChatGPT exhibits impres- sive capabilities in this regard. Human language, however, is intrinsically ambiguous and far from solved. In fact, some forms of meaning are deeply embedded in social interactions. We collectively refer this type of meaning sociopragmatic meaning (SM). To appreciate SM, consider how the meaning of an utterance in social interaction (e.g., on social media) can be highly subtle and how incorporates both the social variation re- lated to language users (from a sociolinguistics perspective) (Tagliamonte, 2015) and their com- municative intentions (from a pragmatics perspec- tive) (Boxer and Cortés-Conde, 2021). Although SM is quite established within linguistics, NLP systems still struggle with this type of meaning that is intertwined in social and interactive con- texts (Zhang and Abdul-Mageed, 2022). The ex- tent to which instruction tuned models such as ChatGPT can capture SM across languages re-
1 Introduction
Multilingual LLMs have recently transformed NLP, due to their powerful capabilities on
Equal contribution

---

mains largely unclear these models yet be evaluated on appropriate datasets under standard- ized conditions easy to replicate.
Studies
Lang. Tasks
SM Tasks
Dutuset Models LeaderBid
(2023)
5
Qin et (2023)
en
7
20
29
Abuja et (2023)
70
10
16
11
Laskar et (2023)
12
12
2
140
27
To facilitate evaluation of LLMs and enhance fairness of model comparisons and reproducibil- ity, early work introduces evaluation benchmarks. Most existing benchmarks, however, focus on the monolingual setting. These include GLUE (Wang et al., 2019), SentEval (Conneau and Kiela, 2018), and TweetEval (Barbieri al., 2020) for En- glish, ARLUE (Abdul-Mageed
Bang (2023)
23
3
Lai (2023)
Das et (2023)
11
2
2
2
Wang
Zhang et (2023b)
en
13
13
26
5
Ziems al. (2023)
24
24
13
Ours
109
Table 1: Our work comparison.
al., 2021) for Arabic, CLUE (Xu et al., 2020a) for Chinese, and IndoNLU (Wilie et al., 2020) for Indone- sian. Although XTREME (Hu et al., 2020) and XGLUE (Liang et al., 2020) introduce multilin- gual benchmarks, they only include a few SM tasks for a limited number of languages. They are also limited to standard language use (e.g., Wikipedia). Barbieri al. (2022) propose multi- lingual sentiment analysis benchmark (UMSAB), but it solely contains tweet sentiment analysis datasets only eight languages. As such, absence a unified, diverse, and comprehensive bench- mark and a fragmented evaluation landscape ham- per NLP work for cross-lingual SM.
tive leaderboard on top of our new benchmark. To summarize, the contributions of this paper are as follows: (1) We collect, uniformize, and responsibly release massively multilingual SM datasets in new benchamark; (2) Our SPAR- ROW benchmark is essentially an archive of SM datasets that alleviates the serious issue of data decay; (3) We evaluate a wide range of mod- cls on our SPARROW benchmark via fine-tuning SoTA encoder-only pretrained language models and zero-shot learning of number of generative models, including instruction tuned models (e.g.. BLOOMZ) as well as ChatGPT; and (4) We es- tablish standard settings for future research in this area across large number of languages and tasks. through public leaderboard.
Another challenge for SM research is the is- sue of data inaccessibility (Assenmacher et al., 2022). Although many studies release the IDs of posts (c.g., tweets), substantial amounts of these social posts become inaccessible over time due to deletion, etc. (Zhang al., 2022). In our bench- mark, we attempt re-collect text contents of 25 datasets by using their IDs but can only re- trieve 58% samples on average (see Table in Ap- pendix). This data decay also hinders fair com- parisons in NLP for SM research. This issue has already become worse corporations such Twitter tighten access to their API, making even harder to collect historical data. To address this bottleneck, we introduce a massively multi- lingual SM evaluation benchmark, dubbed SPAR- ROW, that comprises 169 datasets covering 64 lan- guages from 12 language families, 16 types of scripts, across diverse online platforms (e.g., Twit- ter, YouTube, and Weibo). We then perform an extensive evaluation of ChatGPT, comparing it to 13 other models ranging in size between 110M-7B parameters. Our evaluations allow multiple questions related to how it is that L.MMs fare across languages on SM. To facilitate future comparisons, we also design a modular, interac-
Related Work
Evaluation of LLMs. There have been many at- tempts evaluate ChatGPT and instruction tuned LLMs. Qin et al. (2023); Laskar et al. (2023); Zhong et al. (2023); Wu et al. (2023) utilize ex- isting English evaluation benchmarks, such as GLUE (Wang et al., 2019) and BigBench (Sri- vastava et al., 2022), to evaluate LLMs' capaci- ties on various NLP tasks. These studies find that although ChatGPT performs less effectively than the models finetuned specifically for each task, demonstrates superior capabilities compared other instruction tuned LLMs (e.g.. FLAN (Chung et al., 2022)). Ahuja et al. (2023); Bang et al. (2023); Laskar et al. (2023); Lai et al. (2023); Huang et al. (2023) evaluate LLMs on more di- verse languages using existing multilingual bench- marks (c.g., XNLI, PAWS-X, XLSum) involv- ing monolingual NLP tasks and crosslingual tasks (e.g., machine translation). Their findings point to a large gap in performance of instruction tuned LLMs and ChatGPT, especially on low-resource languages and those with non- non-Latin scripts.
answer

---

SM is still not adequately represented in exist- ing benchmarks, hindering comprehensive evalu- ations on more languages. As we summarize in Table 1, benchmarks used for listed evaluations only include a few SM tasks focusing on senti- ment analysis. Wang et al. (2023); Zhang et al. (2023b) investigate LLMs on number of SM tasks (e.g., offensive language detection), but only English. Ziems al. (2023) investigate Chat- GPT performance on range of computational so- cial science tasks covering subjects such as soci- ology, psychology, and linguistics, but they again focus only on English. Das et al. (2023) extend evaluation of ChatGPT on hate speech detection to 11 languages. Compared to these works, our objective is to investigate more diverse SM tasks massively multilingual setting.
Tasks
Dataset
Lang.
LF
Ser
Aggressive Dangerous
1
Anthooial
Hale
16
11
6
Oflense
3
HHO-Grroup
2
Target
Antisocial
36
20
10
Emotion
Humor
Im&&
Irony
Sarcasm
10
Irony-Type
Trany&Sarcaen
Sentiment
77
58
Subjectivity
6
2
SPARROW
169 64
16
Table Summary of datasets in SPARROW. Lang: number of languages, LF: number of language fami- lies, Ser: number of scripts.
from diverse online platforms and covers wide period of time (1986-2022). We group different tasks in our benchmark by what we perceive be an affinity between these tasks. For example, we group tasks of hate speech, offensive language, and dangerous language detection as anti-social language detection. Meanwhile, we keep partic- ular tasks (such as sentiment analysis and emo- tion recognition) distinct due to the popularity of these tasks and since there are multiple datasets representing each of them. Table 2 summarizes statistics of SPARROW. We now briefly intro- duce our task clusters. We provide more informa- tion about languages in SPARROW in Table 7 of the Appendix. We also provide detailed descrip- tions with full citations of all our datasets in Ta- bles 9, 10, 11, 12, 13, and 14 in Appendix.
Sociopragmatic Meaning Benchmarks. Many previous works introduce unified benchmarks such as GLUE (Wang et al., 2019), SentE- val (Conneau and Kiela, 2018), XTREME (Hu et al., 2020), and XGLUE (Liang et al., 2020). These benchmarks include wide range of NLP tasks, but comprise sole SM task (i.e., senti- ment analysis). Some recent studies started to construct benchmarks focusing on SM: Barbi- eri et al. (2020) introduce TweetEval benchmark that contains seven English datasets of six SM tasks; Zhang et al. (2023b) develop SentiEvall that involves 26 English datasets of 13 sentiment- related tasks. Beyond English, NusaX (Winata et al., 2022), NaijaSenti (Muhammad et al., 2022), and AfriSenti (Muhammad al., 2023a) propose benchmarks for sentiment analysis with eight In- donesian languages, four African languages, and 14 African languages, respectively. UMSAB in- troduced by Barbieri et al. (2022) contains 11 sen- timent analysis datasets in 11 languages. For de- tecting antisocial online comments, Risch et al. (2021) introduces toxic comment collection that contains 13 datasets of six antisocial detection tasks in 14 languages. Compared these works, our SPARROW benchmark includes significantly more SM tasks and languages, from more diverse sources (refer to Figure for a comparison).
3.1 Task Clusters
Antisocial Language Detection. The prolifera- tion of antisocial language (e.g.. hate speech) tox- ifies public discourse, incites violence, and un- dermines civil society (Sap al., 2019; Vidgen and Derczynski, 2020). Antisocial language de- tection thus a useful task. We include un- der the umbrella of antisocial language the fol- lowing: (1) aggressive language (Kumar et al.,
2018), (2) dangerous language (Alshehri et al.,
2020), (3) hate speech (e.g., Waseem and Hovy (2016); Deng et al. (2022)), (4) offensive lan- guage (e.g., Mubarak et al. (2020); Kralj No- vak al. (2021)), (5) offense type identification (e.g., Zampieri et (2019)), and (6) offense tar- get identification (c.g., Ousidhoum et al. (2019); Jeong et al. (2022)).
3 SPARROW Benchmark
In this section, we describe clusters of tasks in our benchmark as well as our preprocessing and unifi- cation. SPARROW consists of 13 types of tasks in six main categories. It contains 169 datasets

---

Emotion Recognition. Emotion affects our decision-making as well as mental and physical health (Abdul-Mageed and Ungar, 2017). SPAR- ROW includes 26 emotion datasets in 17 lan- guages (e.g., Kajava (2018); Bianchi et al. (2021))-
each dataset by randomly sampling 500 samples from Test (if its size exceeds 500). We refer to this smaller test set as Test-S.
We evaluate on each dataset using its origi- nal metric as Tables 9, 10, 11, 12, 13, and 14 in Appendix summarize." We report the perfor- mance on individual datasets, aggregate datasets into 13 tasks, and report average score over each task. Moreover, introduce metric for each main category, calculated as the average of dataset-specific metries within that category. In- spired by previous evaluation benchmarks like GLUE (Wang et al., 2019), we define global met- ric called SPARROW score, which represents the unweighted average of all dataset-specife metrics. The SPARROW score provides an overall indica- tion of performance SM tasks.
Humor Detection. Humor is type of figura- tive language which induces amusing effects, such as laughter or well-being sensations. We include four humor detection datasets in four languages (e.g.. Blinov et al. (2019); Meaney et al. (2021)).
Irony & Sareasm Detection. Irony and sar- casm also involve figurative language. An ironic/sarcastic expression intentionally uses di- ametric language to signify implied meaning. We include (1) nine irony detection datasets in seven languages (e.g.. Xiang et al. (2020)), (2) ten sarcasm detection datasets in four languages (e.g., Walker al. (2012)), and (3) an irony type identification dataset (Van Hee et al., 2018).
Evaluation Methods
4.1 Finetuning on Encoder-Only Models
Subjectivity and Sentiment Analysis. Subjec- tivity analysis aims to understand the opin- ions, feelings, judgments, and speculations ex- pressed via language (Abdul-Mageed et al., 2014). Our benchmark includes six subjectivity analysis datasets in five different languages (e.g., Pang and Lee (2004); Pribán and Steinberger (2022)). Sub- jectivity incorporates sentiment. Sentiment anal- ysis (Poria et al., 2020) is one of the most popu- lar tasks in SM understanding where the objective is identify the underlying sentiment of given text. Our benchmark contains 77 sentiment anal- ysis datasets in 58 languages (e.g., Pang and Lee (2005); Marreddy et al. (2022)).
We evaluate the following Transformer-eneodee based multilingual models on SPARROW: (I) Multilingual-BERT (mBERT) (Devlin et al., 2019) (Base, 110M parameters), (2) XLM-
RoBERTaBase (XLM-R) (Conneau et al., 2020)
(270M parameters), (3) Bernice (Delucia et al., 2022), 270M-parameter model trained with 2.5B tweets in 66 languages, and (4) InfoDCL (Zhang et al.. 2023a), SoTA for SM understanding. which further trains XLM-R with 100M tweets 66 languages with contrastive learning. More de- tails about all models are in Appendix B.
4.2 Zero- and Few-Shot on LLMs
We investigate zero-shot performance on wide range of generative models, including pre- trained generative models: (1) BLOOM (Scao et al., 2022), (2) mT5 (Xue et al., 2021), (3) LLaMA (Touvron et al., 2023), instruction tuned models: (4) BLOOMZ (Scao et al., 2022), a BLOOM-initialized model tuned with multi- lingual xP3 corpus, (5) BLOOMZ-P3 (Muen- nighoff et al., 2022), BL.OM-inittalized model tuned with English-only P3 corpus, (6) BLOOM- Bactrian (Li al., 2023), BL.OMM-iittialized model tuned with 3.4M instruction-followinng samples in 52 languages, (7) mTO (Muen- nighoff et al., 2022), an mT5 model tuned with xP3 corpus, (8) Alpaca (Taori et al., 2023), a
3.2 Preprocessing, Splits, and Metrics
We apply light normalization on all the samples by converting mentions and hyperlinks to "USER' and 'URL', respectively. We standardize label names for consistency across datasets with- out reassigning nor aggregating the original labels of the datasets. For instance, in certain sentiment analysis datasets, we map '0' and '/" to 'Negative' and 'Positive' respectively. Regarding data splits, if the dataset already has Train, Dev, and Test sets, we maintain the same splits. If the original dataset does not include Dev set, we randomly select 10% of training data to be Dev set. In cases with- out pre-defined splits, we use an 80% Train, 10% Dev, and 10% Test random split. For computing constraints, we also prepare a smaller Test set for
We seleet the macro-average Fi soore as the main metric the original paper utilizes more than one metric.

---

Subj-spajar
Ment-ambMuh
Hate-araAla
Emot-zho
alيa عاوي اللو deyou of ctilidd dzy wmrd
padre fosse state in cava sarebbe Infartol
Incrediblle a firence! Se mio
AA + OUE hP? 160 p09C?
Question:
is the language of this text hatetul
adss juù تعع wp aaoin
4938 7%
R5
JÉ 4
#FlarentinaJuve SSAyyeriied
4
answer question by selecting only one the following options: (hateful, not) You do not need provide any explanation. Even
18 00228-9
FS 2
3 HRS
Question: this sensence objectivee
Tp RHLU 9 17C negt hi 7AA+7
W1.. 18.6.
146. 58 d9
subjective?
ovAh:
agge hF7 yoy
if you uncertain, you options without using any other Answer:
one the
85 06 00
3B 1
(R-E
Rt.
7
Answer:
H#
(b)
(c)
(d)
Figure 2: Examples of prompts used for zero-shot evaluation with 1m-eval wwreevalaatiii wai.uuaanon--aeessss yellow and ChatGPT green We use an English prompt (Figures a, c) and machine translated the prompt in the correspond- ing language (Figures b, d), repectively. The prompts construct each task as guestion-and-eae and-answer tasks. The actual input sample is in blue, and the label options are red.
LLaMA-initialized model tuned with 52K English mnstucttion-oolowiin samples, (9) Vicuna (Chi- ang al., 2023), I.LaMA-intiialized model on 70K conversational data, and (10) ChatGPT, for which we use the gpt-3. 5-turbo-03301 ver- sion via OpenAl 2PL. We use 7B-size version of BLOOM- and LLaMA-based models and 4B-size version of mTS-based models. We also evaluate six open-source LLMs (i.e, BLOOM, BLOOMZ- P3, mT5, mTO LLaMA, and Vicuna) via few-shot in-context learning.
specifically for each model, customized prompts employed for each set of tasks. These prompts follow the structure of question-and-anssen tasks, where we present sample content alongside a task-
specific question, as shown in Figure 2. The prompts are summarized in Appendix Table 15. We then instruct the model to generate an answer based on the provided option labels. Each option label represents a potential answer, and we calcu- late the log-likelihood for each candidate. The pre- diction with the highest bog-likeldhood is chosen as the model's final prediction. For the evaluation of ChatGPT, we draw inspiration from previous practices for prompt design (Ziems et al., 2023), and incorporate additional instructions to guide its generation of the desired labels. As shown in Figure 2. we provide an instruction that compels ChatGPT to select a single label for the given input text without providing any additional explanation. We set the temperature 0 to generate determin- istic and reproducible results from ChatGPT. For few instances, we observe that ChatGPT is un- able to provide a direct answer. these cases, we randomly assign false label to each sample. In addition, we also use machine translation to trans- late English prompts and label names to the corre- sponding language of cach dataset.
5 Experiments
5.1 Implementation
Finetuning. To keep computation reasonable, we randomly select 45 datasets for hyper- parameter tuming and only tune the learning rate of each model. 3 For all experiments, we finetune pretrained model with an arbitrary batch size of 32 and sequence length of 128 tokens. Each model finetuned on the full Train set of each dataset for 20 epochs (with patience = 5) based on perfor- mance on Dev set. We run each experiment three times with different random seeds and identify the best model on Dev in each run. We report the av- erage performance Test-S over three runs.4
Zero-shot Evaluation. We perform zero-shot evaluation on SPARROW for BLOOM- mTT5, and LLaMA-based models using language model evaluation harness (Im-evaluation-hamm harness Gao et al. (2021)).5 While we do not tailor prompts
Few-shot Evaluation. We utilize Im-evaluation- harness tool with the same prompts employed in zero-shot evaluation to explore the few-shot in- context learning abilities of open-source LLMs. Before the actual test examples, we prepend amples from the Train set. Each example consists
IIn rest of this paper, ChaiGPT gpt- 3.5....bbtrr-3--0
refers to
'For more information, refer Section C.1 in Appendix. "We also report the performance Dev and standard de- viations Appendix Table 16.
"We use Google Translate for most languages. NLLB model is used to translate the languages of ace, ban, bjn, bug, and min because Google Translate does not enver these. The prompts of pem datascts are translated hy native speaker.
'https: //g thub. cot...com//t com/RI eutherAT/ Im-evaluse weataluutii Lon harnasssss

---

of an input text, task-specific instruction, and the corresponding answer. We set m either 3 or
instruction-responss data is not all that useful for some SM tasks, After tuning mT5 on xP3 dataset, m'TO also experiences a 13.88 improvement in the sentiment score. These may be stemming from in- clusion of five English sentiment analysis datasets in both P3 and xP3 during the training phase. For example, we observe that BLOOM. BLOOMZ, BLOOMZ-P3, mT5, and mTO obtain accuracy of 56.4, 92.2, 93.00, 19.00, and 76.8 on Sent- engsoc (not included in either xP3 P3), respec- tively and that BLOOM BLOOM-Bactrian still performs poorly (accuracy= 53.60) after instruction tuning. Again, these results indicate that it is still impor- tant to include task-related datasets in the instruc- tion tuning stage.
5.2 Results
We present the aggregated performance of Test- S on each task and main category, respectively, in Table 3. We also present test results on all datasets and compare to -ptesct- SoTA performance in Tables 17, 18, 19, 20, 21, and 22 in Appendix.
(1) How is the overall performance over differ- ent models? All the fully finetuned models sur- pass the zero-shot generative models as well as ChatGPT, as shown in Table 3. The most su- perior among the finetuned models is InfoDCL, which achieves a SPARROW score 71.60 and outperforms ChatGPT with 11.56 points SPAR- ROW score. On the other hand, the open-source models (i.e., BLOOM, mT5 and LLaMA) still significantly lag behind on multilingual SM un- derstanding with performance close to random baseline. Meanwhile, the instruction tuned multi- lingual LLMs (BLOOMZ and mTO) only slightly perform better than the random baseline.
(3) How do LLMs perform across different SM tasks? They are inferior at humor and antiso- cial language detection while being relatively bet- ter at sentiment and emotion recognition tasks. BLOOMZ-P3, BLOOMZ, and mTO exhibit con- siderable enhancements (> points) on sentiment and emotion when compared their respective initialization models. On the other hand, we find that instruction tuned models perform significantly worse on aggressive language detection and hu- mor detection tasks. BLOOMZ-P3, BLOOMZ, BLOOM-Bactrian, and mTO all incur loss of more than 5 points on these two tasks. Upon inves- tigating the predictions of instruction tuned mod- cls, we find that they tend assign negative labels (i.e., non aog-eggress or non-humor) which results in many false negative predictions. For a concrete example, we show that BLOOMZ-P3 predict most samples as non-humor in Figure 3a shows.
(2) Can instruction tuning enhance LLMs² ability on SM understanding? Yes, but it de- pends on the instruction training data. Fol- lowing instruction tuning the English-only P3 dataset, BLOOMZ-P3 demonstrates im- provement of 7.76 SPARROW score compared to BLOOM. Also, BLOOMZ improves 5.85 points over BLOOM (but falls short of BLOOMZ.P33. MTO also outperforms mT5. However, there re- mains a substantial gap between all instruction tuned models and finetuned models. BLOOM- Bactrian performs worse than BLOOMZ and BLOOMZ-P3, which are instruction tuned with NLP tasks. This indicates that the general purpose instruction-response dataset is not very useful for SM understanding.
ChatGPT outperforms the open-source LLMs on all tasks except dangerous language detection. Comparing ChatGPT to InfoDCL, we find gaps fa- voring InfoDCL in subjectivity analysis (a differ- ence of 9.47), emotion recognition (a difference of 10.68), and irony & sarcasm detection (a dif- ference of 10.70). ChatGPT also largely lags be- hind InfoDCL in humor detection (a difference of 15.40) 40) and antisocial language detection differ- ence of 14.06). As the example shows in Fig- ure 3b, ChatGPT makes more false positive errors (classifies non-hateful as hateful).
To further probe how instruction tuning im- proves BLOOM-based models, we compare BLOOM with BLOOMZ-P3 and BLOOMZ in terms of individual tasks, finding sentiment anal- ysis to exhibit the most significant improvement. BLOOMZ-P3 and BLOOMZ achieve a sentiment score improvement of 16.37 and 12.36, respec- tively, based on average calculation across 77 sentiment analysis datasets. However, BLOOM- Bactrian obtains an improvement of only 1.79 sen- timent score, perhaps implying that the Bactrian
How do LLMs perform across different lan- guages? We now examine the impact of instruc- tion finetuming on the model's language-wise per- formance. We categorize the performance of each

---

Ramd.
Finetuning
Zero-hoot
BMZ
BMZ
CG
Tacks
Infol>
BM
BMZ
(MT)
P3
Bac.
WTO
(MT)
Alp-
Vic.
CG
(MTT)
110M
270M
270M
270M
7B
7B
7B
7B
4B
4B
4B
7B
71
7B
175B
175B
Apgressive
43.14
72.21
74.64
75.45
73.96
51.06
82
15.82
18.72
16.37
53.67
15.82
22.00
18.
25, 25.07
63.53
54.36
Dangerours
42.06
62.36
63.57
67.13
65.23
46.87
46.87
50.84
46.87
46.87
49.31
46.87
46.87
46.87
46.87
46.87
33,99
33.68
Antiodal
Hate
43.62
72.97
74.37
76.76
75.85
39.83
39.44
37.76
38.52
42.25
23.29
37.33
39.05
37.55
44.31
41.59
66.06
58.74
39.48
77.53
15.88
78.45
78.88
41.06
40.42
20.28
38.59
40.43
244999
39:90
21.11
39.85
16.82
48.70
67.31
52.70
IIO Group
14.82
46.18
42.39
51.15
50.24
13.63
17.26
14.25
21.23
14.81
7.02
16.25
17.01
12.35
14.13
9.26
39.66
26.74
H/O-TTnget
60.96
18.77
19.32
16.83
.5.89
A8
35.20
66.92
67.9
71.14
70.61
33,70
322.60
27.95
51.97
33.79
20.14
32.02
228.1
31.66
50.55
34.50
36.55
47.20
Emottio
9.71
17.18
15.007
15.19
27.87
Humor
49.65
84.35
85.19
86.75
87.05
41.78
33.12
33.82
33.17
33.04
33.991
43.60
33.12
39.78
41.72
46.19
71.65
72.70
hony
68.38
35.15
44.46
34.69
27.49
Sareasm
45.48
72.41
73:40
74.78
74.94
43.00
41.62
52.25
52.22
41.68
46.34
36.09
41.62
41.17
52.48
47.67
65. 34
65.55
Trony-Type
18.83
18.83
30.81
1&8
42.93
67.48
B83T
70.29
71.12
38
3757
34,36
47.79
46.35
35.42
37 56
35.35
3937
29.56
46.14
60.41
59.63
Sentiment
34.08
66.34
09.58
70.44
71.64
26.67
39.03
28.61
43.03
46
20.37
34.05
32.76
27
27.55
25.84
25.02
60.34
54.94
Subjectivity
41.41
22.54
14.45
74,80
75.73
44.12
25.45
30.00
30.73
39.05
37,35
41.04
36.16
42.
42.40
30,44
38.7
73
66.26
59.33
SPARROW
33.47
66.60
69.58
7088
71.00
27.94
33.79
27.17
35.70
29.45
121.45
33.63
30.85
28.75
28.79
29.36
60.04
53.90
Table 3: SPARROW benchmark Test-S results. report the average of dptaset-specific metries in task and a cat- egory, respectively. Rand.: random baseline, mB.: mBERT, X-R: XLM-R, Ber.: Bernice, InfoD: InfoDCL, BM: BLOOM, LLa.: LLaMA, Alp.: Aplaca, Vic.: Vicuna, CG: ChatGPT, MT: using machine translated prompts. The best performance in each setting is Bold. The red font denotes a performance lower than the random baseline.
Dhefi
Lang Random
InfoDCL.
BMZ-P3
mTo
Vicuna
CG
CG-MT
annh
37.95
65.68
16.05
22.49
2.99
20.62
46.82
bug 30.77
71.55
34.60
18.27
12.90
34.63
30.86
ell 41.24
79.13
45.47
eng 37,90
75.48
43.32
39.23
39.75
66.51
BLOONE
fil 52.37
79.01
34.47
34.47
34.47
69.13
66.67
heb 35.24
95.80
71.20
76.60
84.20
hin
67.55
28. 92
26.20
29.06
52.63
48.30
mal 31.68
82.70
43.84
41.65
24.85
44.03
31.44
Indoad Love
Leod
Table 4: Language-wise model performance for sam- ple languages. The complete results are in Table 23 in Appendix. Best performance in each language is bold, and the second best is in green highlight The red font denotes a performance lower than the random baseline.
(a) Humo-spack
(b) Hate-araan
Figure 3: Confusion matrices of two datasets.
dataset based on language and calculate the av- crage language scores across all datasets within a language. Since each language contains differ- ent tasks and datasets, direct comparison across languages is not feasible. Therefore, we com- pare the relative performance between different models for each language. By comparing the instruction tuned models to their initial models, we observe that most languages experience im- provement. However, we also observe a sig- nificant decline in performance for the Amharic (amh) dataset among these models. Specifically, BLOOMZ-P3, BLOOMZ, and mTO experience a deterioration of 36.07, 24.99, and 26.12 points, respectively, compared to their respective initial models. We hypothesize that this deterioration can be attributed to catastrophic forgetting after instruction tuning, where Amharic was not in- cluded in the training set and does not share the writing scripts with the other included languages.
Similarly, the Filipino (fil) tasks exhibit an aver- age decline of approximately 11 points on both BL.OOMZ-P3 and BLOOMZ, as Filipino is not included in the xP3 dataset. Although Hindi is included in the xP3 dataset, the three instruc- tion tuned models still show decline in perfor- mance. Upon examining the individual perfor- mance of Hindi datasets, we find that the major deteriorations occur in the aggressive language de- tection and humor detection tasks, while the emo- tion recognition and sentiment analysis tasks show improvement. The instruction response data for training Alpaca and Vicuna consist solely of En- glish language. Therefore, we compare the perfor- mance of Alpaca and Vicuna to that of LLaMA us- ing both English and non-English datasets. We ob- serve that Alpaca and Vicuna outperform LLaMA when evaluated on English datasets, achieving

---

Spench Desectoo
Task
ASumple
CG (sample MT)
GPT-
96
Hate engoe
168
76.30
Hate-ara
153
38.82
35.76
Hate iape
104
29.75
38.44
Hate-Illew
145
15.76
23.96
amaos
Hate-
178
19:23
Hate-spaaa
174
17.35
Hate poree
142
35.37
37.34
Hate-polrs
54
35.62
49.57
Hate-kongoo
250
14.20
15.85
Hote-amann
89
20.54
16.74
zhor
Hate-kotge
169
27.75
44.84
Hate
100,00
100.00
Sexi frucs
113
34.10
43.69
Hume himage
220
22.73
30,91
Langargo1
Hume rusgii
136
26.89
47.31
Figure 4: A comparison of different models on multiple tasks across various languages. We show the relative gain of each model compared to the random baseline.
Humo-spicni
152
30,68
36.80
Humo-enghae
Table Case study on using machine translated input and GPT-4 on samples mispredicted by ChatGPT.
average scores of 8.30 and 5.51, respectively. However, their performance declines when tested on non-English datasets, resulting in average de- creases of 1.53 and 0.33, respectively. Com- pared task-specifie InfoDCL, ChatGPT per- forms poorly in 63 out of 64 languages, sometimes with a large gap (e.g., 45.06 lower on Amharic, 38.67 lower on Malayalam, and 36.91 lower on Buginese), as Table shows.
prompts 6.14 points lower than ChatGPT with English prompts. Meanwhile, few tasks such as humor and sarcasm acquire improvements. We also observe similar pattern for BLOOMZ and mTO, Table 3 shows. The low-resource lan- guages with non-Latin scripts experience more performance drops in general, which is in line with findings by Lai et al. (2023). Hebrew (heb) and Greek (ell) get the largest performance drops (over 25 points in each case), as shown in Table 4.
We also investigate how different models per- form on SM tasks across various languages. Re- sults for two tasks, hate speech detection (top) and humor detection (bottom), ure 4. The dataset for each task is grouped ac- cording to language, and the average score of each language obtained. The relative gain of each model against the random baseline is shown, al- lowing us to compare across these languages.? We observe that InfoDCL is the best model across var- ious tasks and languages, with the exception of hate speech in Polish where ChatGPT outperforms it. As Figure 4 shows, ChatGPT performs better for Western languages on hate speech detection. We can also observe wider gaps in hate speech de- tection between ChatGPT and InfoDCL on Arabic and Korean. Similarly, while ChatGPT demon- strates satisfactory performance in English humor, it remains at significant distance behind InfoDCL. in Hindi humor.
presented in Fig-
(6) Does GPT-4 outperform ChatGPT? Yes, it does. We provide a study on probing GPT-4's ca- pacities. We exploit 20 datasets from two tasks (i.e., hate speech and humor detection) in 12 lan- guages, only choosing samples whose labels Chat- GPT predicted incorrectly, We refer to this test set as GPTHard and provide samples from it to GPT- 4 in their original language, employing the same English prompts as those used by ChatGPT. As Table 5 shows, GPT-4 significantly outperforms ChatGPT (McNemar's test with a < 0.01) on 19 datasets. 8
(7) Can translating input samples into English help improve ChatGPT's predictions? Yes, it can. Here, we use the non-English part of GPTHard (16 datasets). We translate these test samples into English using ChatGPT and subse- quently employ the translated text and English prompt for classification. As Table 5 shows, we acquire a noteworthy enhancement in ChaiGPT's with a 0.01)
(5) Do machine translated prompts help LLMs? Not in general, but they do help in few cases. We find, in Table 3, that the SPAR- ROW score of ChatGPT with machine translated
performance (McNemar's
We note that different annotation artifacts across the dif- ferent languages still make direct comparisons challenging.
8 An exception is one dataset where a significance test is not possible due to small sample size (n = 2).

---

Zero-shot
Three-shot
Five-shol
Tasks
BMZ
BMZ
BMZ
P3
m/DD
L.a.a
BM
mTO
Vic.
P3
mTS
I.
Aggressive
$1.06
18.72
53:67
15.82
18.31
25 25.07
46.43
43,93
40.73
41.88
43.70
44.53
47.92
47.63
33.80
37.52
50.66
55.27
Dangerours
46.87
46.87
46.87
46.87
46.87
46.87
46.87
45.68
46.87
46.05
46.87
48.91
46.87
46.87
Antisocial
Hate
39.83
38.52
23.29
37.33
37.80
41.59
38.83
38.30
39.43
37.82
43.51
49.17
37.95
37.14
39.53
37.70
41.87
48.37
Offense
41.06
38.59
24.99
39.90
39.85
48.70
43.94
40.25
21.99
41.53
46.36
54.49
42.42
40.59
34,10
41.67
43.83
51.72
H/O-Grrup
13.63
21.23
16.25
12.38
9.26
13.04
7.92
15.98
14.19
9.68
11:76
7:23
14.50
12.58
16.27
IDO Target
18.73
16.89
6.69
20.58
19.32
17.01
17.09
18.41
10.48
16.55
20.56
24.84
17.44
17.45
9.32
16.56
20.09
23.60
33570
3137
30.74
33176
39.50
33.
33235
77.79
57.55
"364T
31.39
32.43
3T.68
7:7T
32.00
740700
Emotion
9.71
15.07
7.75
27.87
15.14
18.12
17.08
12.17
10.06
23.12
32,12
40.28
18.48
12:35
10.07
25.57
34.20
41.79
Humor
41.78
33.04
53.12
39.78
33.07
44.70
38.19
57.15
34.06
40.20
37.08
53.86
Iroey
36.63
24.46
36.52
52,09
40.78
47.48
42:21
41.58
42.61
35.18
36.76
19.78
44.34
44.14
39.67
14.82
38,40
41.61
Sarcwsm
41.68
41.62
41.17
47.67
46.14
48.42
52.55
45.75
49.03
Type
18.83
18.85
18.83
18.83
18.85
18.83
18:88
18.83
18.83
18.83
18.83
18. 85
18.83
18.83
18.85
18.83
18.83
18.83
18S
41.79
37.36
43.01
21.67
42.74
Sentiment
26.67
43,03
20.77
34.65
27.55
25.02
34.81
55.79
24.76
31.37
57.73
34.55
33.15
37.71
23.17
29.25
40.88
39.37
Subjectivity
30.73
41.64
42.30
36.50
30.66
31
34.36
54.77
33,70
30.72
46.53
SPARROW
27.94
35.70
21.45
33.63
28.75
29.36
32.76
31,91
26.12
31.71
37.82
39.44
32.05
32.84
25.47
30.44
39.48
41.97
Table 6: Evaluating open-source LLMs on SPARROW with few-shot in-context learning. The best performance in each setting is Bold. The red font denotes a performance lower than the random baseline.
when using the translated input. We also ob- serve that when fed with these English-translated samples, ChatGPT is able to surpass GPT4 with the original inputs in three datasets (i.e., Hate- araAla- Hate-spagas. Hate-araMub) These results suggest that although ChatGPT has inferior abil- ity on several languages in terms of detecting SM, a translate-then then-detect approach may be possible.
ble 24 in Appendix shows, we find that BLOOM, LLaMA, and Vicuna incur sizable performance drops (> 6 points decrease across 55 datasets), while BLOOMZ-P3, mT5, and mTO demonstrate performance levels akin to those observed in pre- vious experiments (< 2 points different). We leave more comprehensive evaluation of prompt sensi- tivity as future work.
(8) How do open-source LLMSs perform with few-shot in-context learning? As Table shows, we compare three-shot and five-shot re- sults with zero-shot results. Based on SPAR- ROW score, we observe that few-shot learning does enhance the performance of BLOOM, mT5, LLaMA, and Vicuna. With the increasing num- ber of shots, the performance of LLaMA and Vi- cuna increases. Vicuna obtains SPARROW scores of 29.36, 39.44, and 41.97 with zero, three, and five shots, respectively. However, BLOOMZ-P3 and mTO do not improve with few-shot learning. We suspect this is because the instruction fine- tuning of these two models only uses zero-shot template that hurts their few-shot learning capac- ities. BLOOMZ-P3 and mTO are also different from BLOOM and LLaMA in that they are fine- tuned on several NLP tasks only one of which is an SM task (i.e., sentiment analysis). This proba- bly biases the behavior of these two models.
Public Leaderboard
To facilitate future work, we design a public leaderboard for scoring models on SPARROW. Our leaderboard is interactive and offers rich metadata about the various datasets in our bench- mark. It also encourages users to submit informa- tion about their models (e.g.. number of param- eters, time to convergence, pretraining datasets). We also distribute a new modular tookiit for fine- tuning or evaluating models on SPARROW.
Conclusion
In order to understand the abilities of ChatGPT and other instruction tuned LLMs on capturing sociopragmatic meaning, we introduced a mas- sively multilingual evaluation benchmark, dubbed SPARROW. The benchmark involves 169 datasets covering 64 languages from 12 language families and 16 scripts. Evaluating ChatGPT on SPAR- ROW, find it struggles with different lan- guages. We also reveal that task-specific models finetuned on SM (much smaller than ChatGPT) consistently outperform larger models by a signif- icant margin even on English.
(9) Are the open-source LLMs sensitive to prompts used? We carry out study probe the open-source LLMs' sensitivity to prompts. We curate 55 datasets across four tasks from SPARROW and evaluate six models with the prompts we used for evaluating ChatGPT. As Ta-

---

8 Limitations
quality of all the datasets. As a quality assur- ance measure, we only include in SPARROW datasets that are introduced in peer-reviewed pub- lished research. To facilitate access to informa- tion about each dataset, we link to each pub- lished paper describing each of these datasets inTaales 9, 10, 11, 12, 13, and 14.
Benchmark Construction. Our SPARROW benchmark only includes text classification tasks related to SM. Despite our best efforts, we acknowledge that our benchmark has not covered existing SM datasets exhaustively. We will continue expanding this benchmark and welcome future datasets or metric contributions to it. We also plan to extend SPARROW to more types of tasks related SM, such as span-based sentiment analysis (Xu et al., 2020b), affective language generation (Goswamy et al., 2020), and conversational sentiment analysis (Ojamaa et al., 2015). We only include text-based SM tasks. Another improvement direction is to extend this benchmark to more tasks that involve more modalities, such affective image captioning (Mo- hamed et al., 2022) and multi-modas emotion recognition (Firdaus et al., 2020).
Following privacy protection policies, we anonymize all SPARROW data as described in Section 3.2. With reference to accessibility of the original individual dataset, SPARROW data can be categorized into three releasing strategies: (1) In the case of datasets requiring approval by the orig- inal authors, require future researchers to ob- tain approval first and will share our splits once approval has been obtained. We indicate these nine datasets our data description tables. (2) For the 25 datasets (see Table in Appendix) that are shared via tweet IDs, we share our obtained data for research use. By doing so, we expect to mit- igate the issue of data decay and allow fair com- parisons. (3) We will share the other 135 publicly accessible datasets upon request. We will also require a justification for responsible use of the datasets. Each dataset will be shared our Train, Dev, and Test splits along with a dataset card to indicate the original publication of the dataset.
Model Selection. Due to computation con- straints, we cannot evaluate on model sizes > 7B. However, we hope SPARROW will be used in the future to evaluate larger-sized models. Again, due to budget constraints, we only con- duct relatively small case study GPT-4 and do not evaluate more diverse commercial instruc- tion tuned models that are more expensive (e.g., text-davinci--03 by OpenAl).
Intended Use. The intended use of SPARROW benchmark is to construct a scoring board to facil- itate model comparisons as well as enhance fair- ness and reproducibility across different languages and tasks. We also aim to mitigate data decay is- sues in social media research. SPARROW could help researchers investigate model's capacity on SM tasks across languages. SPARROW may also be used to investigate model transferability across a wide range of tasks and diverse languages in dif- ferent settings (such as zero- or shw-shot settings and prompting).
Experiments. While we customize prompts em- ployed for each task, we do not tailor prompts specifically for each model. We acknowledge that the performance of models may be influenced by different prompt variants. In future work, we will test diverse prompt variations for more robust re- sults. We only experiment with machine translated prompts in our analyses and acknowledge that the performance drop may stem from the poor qual- ity of machine translation. We will investigate the utility of human translated prompts in a future study. In this paper, we only evaluate LLMs on zero-shot learning. The adoption of few-shot in- context learning may enhance performance, which we also leave to future work.
Potential Misuse and Bias. We notice that some annotations in the datasets of SPARROW (e.g., for hate speech task (Waseem and Hovy, 2016)) can carry annotation and temporal biases. We recommend that any dataset in SPARROW not be used for research in applications with- out careful consideration of internal biases of the datasets and potential biases of the resulting sys- tems. We also suggest that users of SPARROW not only focus on the overall SPARROW score but also a model performance on each task and
Ethics Statement and Broad Impacts
Data Collection and Releasing. All the 169 datasets are produced by previous research. Since there are large numbers of datasets and languages in SPARROW, is hard to manually verify the

---

dataset. The SPARROW score is an unweighted average score over all the datasct-specific metries, which may lose the fine-grained information and be dominated by the largest task cluster (i.e., senti- ment analysis) or languages (e.g., languages from Indo-European language family).
Open-Source Arabic Corpora and Processing Tools. with a Shared Task on Offensive Language Detec- tion, pages 16-23, Marseille, France. European Lan- guage Resource Association.
Ibrahim Abu Farha and Walid Magdy, 2020. From Ara- bic sentiment analysis to sarcasm detection: The Ar- Sarcasm dataset. In Proceedings of the 4th Work- shop on Open-Source Arabic Corpora and Process- ing Tools, with a Shared Tusk on Offensive Language Detection, pages 32-39, Marseille, France. Euro- pean Language Resource Association.
Acknowledgements
We acknowledge support from Canada Re- search Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN-2018-042677 the Social Sci- ences and Humanities Research Council of Canada (SSHRC; 435-2018-0576; 895-2020- 1004; 895-2021-1009), Canadian Foundation for Innovation (CFI; 37771), Digital Research Al- liance of Canada, 9 and UBC ARC-Sockeye." 10
Akshita Aggarwal, Anshul Wadhawan, Anshima Chaudhary, and Kavita Maurya. 2020. "did you re- ally mean what you said?" Sarcasm detection in Enggdih code-mixed data using bilingual word embeddings. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pages 7-15, Online. Association for Computational Lin- guistics.
Kabir Ahuja, Rishav Hada, Millicent Ochieng. Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: multilingual eval- uation of generative AL CoRR, abs/2303. 12528.
References
Muhammad Mbgul-Mageed, Mona T. Diab, and Sandra Kubler. 2014. SAMAR: subjectivity and sentiment analysis for arabic social media. Comput. Speech Lang., 28(1):20-33
Azalden Alakrot, Liam Murray, and Nikola S. Nikolov. 2018. Dataset construction for the detection of anti- social behaviour in online communication in ara- bic. In Fourth International Conference On Arabic Computational Linguisties, ACLING 2018, Novem- ber 17-19, 2018, Dubai, United Arab Emirates, vol- ume 142 of Procedia Computer Science, pages 174- 181. Elsevier.
Muhammad Abdul-Mageed, AbdelRahim A. EI- madany, and EI Moatez Billah Nagoudi. 2021. AR- BERT& & MARBERT: deep bidirectional transform- ers for arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing, ACL/UJCNLP 2021, (Volume 1: Long Pupers), Virtual Event, Au- gust 1-6, 2021, pages 7088-710S, Association for Computational Linguistics.
Ali Alshehri, EI Moatez Billah Nagoudi, and Muham- mad Abdul-Mageed. 2020. Understanding and de- tecting dangerous speech in social media. In Pro- ceedings of the 4th Workshop on Open-Source Ara- bic Corpora and Processing Tools, with Shared Task on Offensive Language Detection, pages 40- 47, Marseille, France. European Language Resource Association.
Muhammad Abdul-Mageed and Lyle Ungar. 2017. EmoNet: Fine-grained emotion detection with gated recurrent neural networks. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1; Long Papers), pages 718-728, Vancouver, Canada. Association for Com- putational Linguistics.
Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neu- ral sentiment analysis for morphologically rich lan- guages: A case study from Modern Hebrew. In Proceedings of the 27th International Conference on Computational Lingwistics, pages 2242-22 2222, Santa Fe, New Mexico, USA. Association for Computa- tional Linguistics.
Muhammad Abdul-Magecd Mageed, Chiyu Zhang, Abdel- Rahim Elmadany, Houda Bouamor. and Nizar Habash. 2022. NADI 2022: The third nuanced Ara- bic dialect identification shared task. In Proceed- ings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 85-97, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Seyed Arad Ashrafi Asli, Behnam Sabeti, Zahra Majd- abadi, Preni Golazizian, reza fahmi, and Omid Mo- menzadeh. 2020. Optimizing annotation effort us- ing active learning strategies: A sentiment analy- sis case study in Persian. In Proceedings of The 12th Language Resources and Evaluation Confer- ence, pages 2855-2861, Marseille, France. Euro- pean Language Resources Association.
Muhammad Abdul-Mageed,, Chiyu Zhang, Azadeh Hashemi, and El Moatez Billah Nagoudi. 2020. AraNet: A deep learning toolkit for Arabie social media. In Proceedings of the 4th Workshop on
10 "haess/alllanneeaa.ca
Dennis Assenmacher, Derek Weber, Mike Preuss, An- dré Calero Valdez, Alison Bradshaw, Björn Ross,

---

Stefano Cresci, Heike Trautmann, Frank Neumann, and Christian Grimme. 2022. Benchmarking crisis in social media analyties: A solution for the data- sharing problem. Social Science Computer Review, 40(6); 1492-1522.
the Italian language. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjec- rivity, Sentiment and Social Media Analysis, pages 76-83, Online. Association for Computational Lin- guisties.
David Bamman and Noah Smith. 2015. Contextual- ized sarcasm detection on twitter. In Proceedings of Web and So- cial Media, ICWSM 2015, University of Oxford, Ox- ford, UK, May 26-29, 2015, pages 574-577. AAAT Press.
Federico Bianchi, Debora Nozza, and Dirk Hovy, 2022. XLM-EMO: Multilingual emotion prediction in so- cial media text. Proceedings of the 12th Work- shop on Computational Approaches to Subjectiviry, Sentiment & Social Media Analysis, pages 195-203, Dublin, Ireland. Association for Computational Lin- guisties.
the Ninth International Conference
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su. Bryan Wilie, Holy Lovenia, Zi- wei Ji, Tiezheng Yu, Willy Chung, Quyet Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. CORR, abs/2302.04023.
Vladislav Blinov, Valeria Golotova-Baranone and Pavel Braslavski. 2019. Large dataset and language model fun-tuning for humor recognition. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Pa- pers, pages 4027-4032 Association for Computa- tional Linguistics.
Francesco Barbieri, Valerio Basile, Danilo Croce, Malvina Nissim, Nicole Novielli, and Viviana Patti. 2016. Overview of the evalita 2016 sentiment po- larity classification task. In Proceedings of Third Italian Conference on Computational Linguisties (CLiC-it 2016) & Fifth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2016), Napoli, Italy, December 5-7, 2016, volume 1749 of CEUR Workshop Proceedings. CEUR-WS.OO
Cristina Bosco, Felice DrlltOrletta, Fabio Poletto, Manuela Sanguinetti, and Maurizio Tesconi. 2018. Overview of the EVALITA 2018 hate speech de- tection task. In Proceedings of the Sixth Evalua- tion Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018) co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-il 2018). Turin, Italy, December 12- 2018, volume 2263 of CEUR Workshop Proceedings. CEUR-WS.org.
Francesco Barbieri, Jose Camacho Collados, Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetE- val: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2020, pages 1644-1650, Online. Association for Compu- tational Linguistics.
Diana Boxer and Florencia Cortés-Conde 2021. So- cial Groups and Relational Networks, Cambridge Handbooks in Language and Linguistics, page 227-246. Cambridge University Press.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Hersert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual.
Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 258-266, Marseille, France. European Language Resources Association.
Valerio Basile, Andrea Bolioli, Viviana Patti, Paolo Rosso, and Malvina Nissim. 2014. Overview of the evalita 2014 sentiment polarity classification task. Overview of the Evalita 2014 SENTiment POLarity Classification Task, pages 50-57.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela San- guinetti. 2019. SemfEval-2019 task 5: Multilin- gual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th inter- national Workshop on Semantie Evaluation, pages 54-63, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Henrico Bertini Brum. 2018. Expansão de recursos para análise de sentimentos usando aprendizado supervssppri Ph.D. thesis, Universidade de São Paulo.
Henrico Bertini Brum and Maria das Graças Volpe Nunes, 2018 Building sentiment corpus of tweets in Brazilian Portuguese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki,
Federico Bianchi, Debora Nozza, and Dirk Hovy. 2021. FEEL-IT: Emotion and sentiment classification for

---

Japan, May 7-12, 2018. European Language Resources Association (ELRA).
Alessandra Teresa Cignarella. Simona Frenda, Vale- rio Basile, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2018. Overview of the EVALITA 2018 task on irony detection in italian tweets (ironita). In Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Ialiin.. Final Workshop (EVALITA 2018) co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-il 2018), Turin, Italy, December 12-13, 2018, volume 2263 of CEUR Workshop Pro- ceedings. CEUR-WS.co
Neil Vicente Cabasag, Vicente Raphael Chan, Sean Christian Lim, Mark Edward Gonzales, and Charibeth Cheng. 2019. Hate speech in philippine election-related tweets: Automatic detection and classification using natural language processing. Philippine Computing Journal, XIV No, 1.
Bharathi Raja Chakravarthi, Ruba Priyadharshini, Vigneshwaran Muralidaran, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, and John P. McCrae. 2022. Dravidiancodemix: sentiment analysis and offensive language identification dataset for dravid- ian languages in code-mixed text. Lang. Resour. Evaluation, 56(3):66-8866
Alexandra Ciobotaru and Liviu P. Dinu. 2021. Red: A novel dataset for romamian emotion detection from tweets. In Proceedings of the International Confer- ence on Recent Advances in Natural Language Pro- cexsing (RANLP 2021), pages 296-305, Vamn,, Bul- garia. INCOMA Ltd.
Qianben Chen, Richong Zhang, Yaowei Zheng, and Yongyi Mao. 2022. Dual contrastive learning: Text classification via label-aware data augmenta- tion. CoRR, abs/2201.08702.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanow. 2020. Unsupervised cross-lingual representation learning at scale, In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu. Hao Zhang, Lianmin Zheng. Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Fric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Patricia Chiril, Véronique Moriceau, Farah Benamara, Alda Mari, Gloria Origgi, and Marlène Coulomb- Gully. 2020. An annotated corpus for sexism de- tection in french tweets. In Proceedings of The 12th Language Resources and Evaluation Confer- ence, LREC 2020, Marseille, France, May 11-16, 2020, pages 1397-1403. European Language Re- sources Association.
Diogo Cortiz, Jefferson O. Silva, Newton Calegari, Ana Luisa Freitas, Ana Angélica Soares, Carolina Botelho, Gabriel Gaudencio Rêgo, Waldir Sampaio, and Paulo Sergio Boggio. 2021. A weak super- vised dataset of fine- fine-grained emotions in portuguese. CoRR, abs/2108.0700 00.683...
Luis Chiruzzo, Santiago Castro, Santiago Góngora, Aiala Rosá, J. A. Meaney, and Rada Mihalcea. 2021. Overview of HAHA at IberLEF 2021: Detecting, rating and analyzing humor in Spanish. Proces. del Leng. Natural, 67:257-268.
Mithun Das, Saurabh Kumar Pandey, and Animesh Mukherjee. 2023, Evaluating chatgpt's performance for multilingual and emoji- based hate speech detec- tion. CoRR, abs/2305. 13276.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic. Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Pro- cessing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4399-4307
Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weher. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the Eleventh International Con- ference Web and Social Media, ICWSM 2017, Montréal, Québec, Canada, May 15-18, 2017, pages 512-515. AAAI Press.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, AL- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowd- hery, Sharan Narang, Gaurav Mishra, Adams Yu. Vincent Y. Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction- finetuned language models. CoRR, abs/2210. 1416.
Alexandra DeLucia, Shijie Wu, Aaron Mueller, Car- los Aguirre, Mark Dredze, and Philip Resnik. 2022. Bernice: A multilingual pre-trained encoder for Twitter. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2022, Abu Dhabi, United Arab Emi- rates, December. 2022, pages 6191-6205. Associa- tion for Computational Linguistics.

---

Dorottya Demszky, Dana Movshovitz-ANl Jeong- woo Ko, Alan Cowen, Gaurav Nemade, and Su- jith Ravi. 2020. GoFmotions: A dataset of fine- grained emotions. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguisties, pages 4040-4054, Online. Association for Computational Linguistics.
dialogue dataset for emotion recognition and sen- timent analysis in conversations. In Proceedings of the 28th International Conference on Compu- rational Linguistics, pages 4441-4453, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Paula Fortuna, João Rocha da Silva, Juan Soler- Company, Leo Wanner, and Sérgio Nunes. 2019. A dierarchically-labele dataset. In Proceedings of the Third Workshop Abusive Language Online, pages 94-104, Florence, Italy, Association for Computational Linguisties.
Jiawen Deng, Jingyan Zhou, Hao Sun, Fei Mi, and Minlic Huang. 2022. COLD: benchmark for chinese offensive language detection. CoRR. 0bs/2201.060223
Portuguese hate speech
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis. Minnesota. Associ- ation for Computational Linguistics.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Gold- ing. Jeffrey Hsu, Kyle McDonell, Niklas Muen- nighoff, Jason Phang. Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.
Bilal Ghanem, Jihen Karoui. Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. IDAT at FIRE2019: overview of the track on irony detec- tion in arabic tweets. In FIRE '19; Forum for In- formation Retrieval Evaluation, Kolkata, India, De- cember, 2019, pages 10-13, ACM.
Mountaga Diallo, Chayma Fourati, and Hatem Haddad. 2021. Bambara language dataset for sentiment anal- ysis.
Alexiei Dingli and Nicole Sant. 2016. Sentiment anal- ysis on maltese using machine learning. Proceed- ings of The Tenth International Conference on Ad- vances in Semantic Processing (SEMAPRO 2016), pages 21-25.
Preni Golazizian, Behnam Sabeti, Seyed Arad Ashrafi Asli, Zahra Majdabudi, Omid Momenzadeh. and Reza Fahmi. 2020. Irony detection in Persian language: A transfer learning approach using emoji prediction. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2839- 2845, Marseille, France. European Language Re- sources Association.
Stefan Daniel Dumitrescu, Andrei-Marius Avram, and Sampo Pyysalo. 2020. The birth of romanian BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACI., pages 4324-4328 Association for Compu- tational Linguistics.
Xiaochang Gong, Qin Zhao, Jun Zhang, Ruibin Mao, and Ruifeng Xu. 2020. The design and construc- tion of chinese sarcasm dataset. In Proceedings of The 12th Language Resources and Evaluation Con- ference, LREC 2020, Marseille, France, May 1 11-16, 2020, pages 5034-5039. European Language Re- sources Association.
AbdelRahim A. Elmadany, EI Moatez Billah Nagoudi, and Muhammad Mbdul-Mageed. 2022. ORCA: A challenging benchmark for arabic language under- standing. CORR, abs/2212.10758.
Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in arabic. In Proceedings of the Sixth Arabic Natural Lan- guage Processing Workshop, WANLP 2021, Kyiv, Ukraine (Virtual). April 9, 2021, pages 296-305. Association for Computational Linguistics.
Raymond G Gordon Jr. 2005. Ethnologue, languages of the world. http:///ww: ethnologue. com/.
Tushar Goswamy, Ishika Singh, Ahsan Barkati, and Ashutosh Modi. 2020. Adapting language model for controlled affective text generation. In Proceedings of the 28th International Conference Computational Linguistics, pages 2787-2801, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain represen- tations for detecting sentiment, emotion and sar- casm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Process- ing, pages 1625, Copenhagen, Denmark. As- sociation for Computational Linguistics.
Zekeriya Güven, Banu Diri, and Tolgahan Çakaloğlu. 2020. Comparison of n-stage latent dirichlet alloca- tion versus other topic modeling methods for emo- tion analysis. Journal of the Faculty of Engineering and Architecture of Gazi University, 35(4).
Mauajama Firdaus, Hardik Chauhan, Asif Ekbal, and Pushpak Bhattacharyya. 2020. MEISD: multi- modal multi-label emotion, intensity and sentiment
Vong Anh Ho. Duong Huynh-Cong Nguyen, Danh Hoang Nguyen, Linh Thi-Van Pham,

---

Duc-Vu Nguyen, Kiet Van Nguyen, and Ngan Luu-
Pei Ke, Haozhe Ji. Siyang Liu, Xiaoyan Zhu, and Min- lie Huang. 2020. SentiLARE: Sentiment- aware lan- guage representation learning with linguistic knowl- edge. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLPP, pages 6975-6988, Online. Associa- tion for Computational Linguistics.
Thuy Nguyen. 2019. Emotion recognition for
Vietnamese social media text. In Computational Linguistics 16th International Conference of the Pucific Association for Computational Linguistics. PACLING 2019, Hanoi, Vietnam, October 11-13, 2019, Revised Selected Papers, volume 1215 of Communications in Computer and Information Science, pages 319-333, Springer.
Jan Kocoń, Piotr Milkowski, and Monika Zaško- Zielińska. 2019. Multi-level sentiment analysis of PolEmo 2.0: Extended corpus of multi-domain con- sumer reviews. In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning (CoNLL), pages 980-991, Hong Kong, China. Asso- ciation for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13- 18 July 2020, Virtual Event, volume I 119 of Proceed- ings of Machine Learning Research, pages 4411- 4421. PMLR.
Petra Kralj Novak. Igor Mozetië, and Nikola Ljubešié. 2021. Slovenian twitter hate speech dataset IMSyPP-sl. Slovenian language resource repository CLARIN.S..
Haoyang Huang. Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross- lingual-thought prompting. CoRR, abs/2305.070004.
Atharva Kulkarni, Meet Mandhane, Manali Likhitkar, Gayatri Kshirsagar, and Raviraj Joshi. 2021. L3cubemahasent: A marathi tweet-based sentiment analysis dataset. CoRR, abs/2103. 14408
Ritesh Kumar, Aishwarya N. Reganti, Akshit Bha- tia, and Tushar Maheshwari. 2018 Aggression- annotated corpus of hindi-english code-mixed data. In Proceedings of the Fleventh International Confer- ence on Language Resources and Evaluation, LREC 2018. Miyazaki, Japan, May 7-12, 2018, European Language Resources Association (ELRA).
Md. Asif Iqhal, Avishek Das, Omar Sharif, Mo- hammed Moshiul Hoque, and Iqbal H. Sarker. 2022. Bemoc: A corpus for identifying emotion bengali texts. SN Comput. Sci., 3(2):135.
Khondoker Ittehadul Islam, Sudipta Kar, Md Saiful Is- lam, and Mohammad Ruhul Amin. 2021. SentNoB: noisy Bangla
A dataset for analysing sentiment texts. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 3265-3271, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond en- glish: Towards a comprehensive evaluation of large language models in multilingual learning. CoRR, abs/2304.05613.
Hayeon Jang, Munhyong Kim, and Hyopil Shin. 2013. KOSAC: A full-fledged korean sentiment analysis corpus. In Proceedings of the 27th Pacific Asia Conference on Language, Information and Compu- tation, PACLIC 27, Taipei, Taiwan, November 21- 24, 2013. National Chengehi University, Taiwan.
Md. Tahmid Rahman Laskar, M. Saiful Bari, Miza- nur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. 2023. A system- atic study and comprehensive evaluation of chatgpt on benchmark datasets. CoRR, htt./3355.8886.
Sophia Lee and Zhongqing Wang. 2015. Emotion in code-switching texts: Corpus construction and anal- ysis. In Proceedingy of the Eighth SIGHAN Work- shop on Chinese Language Processing, pages 91- 99, Beijing, China. Association for Computational Linguisties.
Younghoon Jeong, Juhyun Oh, Jaimeen Ahn, Jongwon Lec, Jihyung Moon, Sungjoon Park, and Alice Oh. 2022. KOLD; korean offensive language dataset. CoRR, 1ttp2225..113.....
Kaisla Kajava. 2018 Cross-lingual sentiment preser- vation and transfer learning in binary and multi- class classification. Master's thesis, University of Helsinki.
Haonan Li. Fajri Koto. Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-x A multi- lingual replicable instruction-folloiing model with low-rank adaptation. CoRR, abs/2305.15011.
Md. Rezaul Karim, Sumon Kanti Dey, Tanhim Islam, Sagor Sarker, Mchadi Hasan Menon, Kabir Hos- sain, Md. Azam Hossain, and Stefan Decker, 2021. Deephateexplainer Explainable hate speech detec- tion in under-resourced bengali language. In 8th IEEE International Conference on Data Science and Advanced Analyties, DSAA 2021, Porto, Portugal, October 6-9, 2021, pages 1-10. IEEE.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fen- fei Guo, Weizhen Qi. Ming Gong. Linjun Shou, Daxin Jiang, Guihong Cao. Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Fdwwar Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Win- nie Wu, Shuguang Liu, Fan Yang. Daniel Cam- pos, Rangan Majumder, and Ming Zhou. 2020.

---

XGLUE; new benchmark datasetfor cross-linge Pross-lingual pre traininnnn understanding and generation. In Pro- ceedings of the 2020 Conference Empirical Meth- ods Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6008-6018. Association for Computational Linguisties.
Crosslingual generalization through multitask finetuning. CoRR, abs/2211.01786.
Shamsuddeen Hassan Muhammad, Idris Abdulmu- min, Abinew Ali Ayele, Nedjma Ousidhoum, David Iffoluwa Adelami. Seid Muhic Yimam, Ibrahim Said Ahmad, Meriem Beloucif, Saif M. Mohammad, Sebastian Ruder, Oumaima Hourrane, Pavel Brazdil, Felermino Dário Mário António Ali, Davis David, Salomey Osei, Bello Shehu Bello, Falalu Ibrahim. Tajuddeen Gwadabe, Samuel Ru- tunda, Tadesse Destaw Belay, Wendimu Baye Mes- selle, Hailu Beshada Balcha, Sisay Adugna Chala, Hagos Tesfahun Gebremichael, Bernard Opoku, and Steven Arthur. 2023a. Afrisenti: A twitter sentiment analysis benchmark for african languages. CoRR, abs/2302.08956
Mounika Marreddy, Subba Reddy Oota, Lak- shmi Sireesha Vakada, Venkata Charan Chinni, and Radhika Mamidi. 2022. Am i resource-pos lan- guage? data sets, embeddings, models and analysis for four different nlp tasks in telugu language, ACM Trans. Asian Low-Resour. Lang, Inf. Process., 22(1).
J. A. Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, and Walid Magdy. 2021. SemEval 2021 task 7: HaHackathon, detecting and rating humor and offense. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021) pages 105-119, Online. Association for Computa- tional Linguisties.
Shamsuddeen Hassan Muhammad, Idris Abdulmu- min, Seid Muhie Yimam, David Ifeoluwa Ade- lani, Ibrahim Sa'id Ahmad, Nedjma Ousidhoum, Abinew Ali Ayele, Saif M. Mohammad, Meriem Beloucif, and Sebastian Ruder. 2023b. SemEval- 2023 Task 12: Sentiment Analysis for African Lan- In Proceedings of the 17th International Workshop on Semantic Eval- uation 202223)-20233. Association for Computa- tional Linguisties.
Youssef Mohamed, Faizan Farooq Khan, Kilichbek Haydarov, and Mohamed Elhoseiny. 2022. is okay to not be okay: Overcoming emotional bias in affec- tive image captioning by contrastive data collection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA USA, June 18-24, 2022, pages 21231-21240. IEEE.
guages (AfriSenti-SemFa)
Shamsuddeen Hassan Muhammad, David Adelani, Se- bastian Ruder, Ibrahim Sa'id Ahmad, Idris Abdul- mumin, Shehu Bello Bello, Monojit Choudhury, Chris Chinenye Emezue, Saheed Salahuddeen Ab- dullahi, Anuoluwapo Aremu, Alipio Jeorge, and Pavel Brazdil. 2022. Naijasenti: A nigerian twitter sentiment corpus for multilingual sentiment analy- sis. In Proceedings of the 13th Language Resources and Evaluation Conference, pages 590-602, Mar- seille, France. European Language Resources Asso- ciation.
Saif Mohammad, Felipe MarqueMa Moham- mad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 201822211000 task I: Affect in tweets. In Proceed- ings of The 12th International Workshop on Seman- tic Evaluation, pages 1-17, New Orleans, Louisiana. Association for Computational Linguistics.
Jihyung Moon, Won Ik Cho, and Junbum Lee. 2020. BEEP! Korean corpus of online news comments for toxic speech detection. In Proceedings of the Eighth International Workshop on Natural Language Pro- cessing for Social Media, pages 25-31, Online, As- sociation for Computational Linguistics.
Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Halima Alshahani. 2019. L-HHAB: A Levantine Twitter dataset for hate speech and abusive lan- guage.
Igor Mozetič. Miha Grčar, and Jasmina Smailović. 2016. Multilingual twitter sentiment classifica- tion: The role of human annotators. PloS one, 1155)300
Proceedings of the Third Workshop on Abusive Language Online, pages 111-118, Flo- rence, Italy, Association for Computational Linguis- tics.
Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020. Overview of OSACT4 Arabic offensive language detection shared task. In Proceedings of the 4th Workshop on
Sebastian Nordhoff and Harald Hammarström. 2011. Glottolog/angglo:: Defining dialects, languages, and language families as collections of resources. In Proceedings of the First International Workshop on Linked Science 2011, Bonn, Germany, October 24, 2011, I, volume 783 of CEUR Workshop Proceedings. CEUR-W SLUR-WS.oom WS.org.
Arabic Corpora and Pro- cessing Tools, with a Shared Task on Offensive Lan- guage Detection, pages 48-52, Marseille, France. European Language Resource Association.
Emily Öhman, Mare Pamies, Kaisla Kajava, and Jörg Tiedemann. 2020. XED: A multilingual dataset for sentiment analysis and emotion detec- tion. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 5542-6552. International Committee on Computational Linguisties.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika. Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022.

---

Birgitta Ojamaa, Päivi Kristiina Jokinen, and Kadri Muischenk. 2015. Sentiment analysis on conver- sational texts. In Proceedings of the 20th Nordic Conference of Computational Linguistics, NODAL- IDA 2015, Institate of the Lithuanian Language, Vilnius, Lithuania, May 11-13, 2015, volume 109 of Linköping Electronic Conference Proceedings, pages 233-237. Linköping University Electronic Press / Association for Computational Linguistics.
Computational Linguistics (ACL (55), pages 115- 124, Ann Arbor, Michigan. Association for Compu- tational Linguistics.
Braja Gopal Patra, Dipankar Das, and Amitava Das. 2018. Sentiment analysis of code-mixed indian lan- guages: An overview of sail_ caal_-odeemiig shared task @icon-2017. CoRR, abs/1803.06745.
Flor Miriam Plaza del Arco, Carlo Strapparava, L. Al- fonso Urena Lopez, and Maite Martin. 2020. Emo- Event: A multilingual emotion corpus based on dif- ferent events. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 1492-1498, Marseille, France. European Language Resources Association.
Shereen Oraby, Vrindavan Harrison, Lena Reed, Ernesto Hermandez, Ellen Riloff, and Marilyn Walker. 2016. Creating and characterizing diverse corpus of sarcasm in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 31-41, Los An- geles. Association for Computational Linguistics.
Soujanya Poria, Devamanyu Hazarika, Navonil Ma- jumder, and Rada Mihalcea. 2020. Beneath the tip of the iceherg: Current challenges and new direc- tions in sentiment analysis research. arXiv preprint arXiv:2005.003: 00357.
Reynier Ortega-Bueno, Francisco Rangel, D Hernán- dez Farras, Paolo Rosso, Manuel Montes-y Gómez, and José E Medina Pagola. 2019. Overview of the task on irony detection in spanish variants. In Pro- ceedings of the Iherian Languages Evaluation Fo- rum co-located with 35th Conference of the Span- ish Society for Natural Language Processing, Iber- L.F@SEEPIN 2019, Bilbao, Spain, September 24th, 2019, volume 2421 of CEUR Workshop Proceed- ings. pages 229-256. SEUR-WS.org
Pavel Pribán and Josef Steinberger. 2022. Czech dataset for cross-lingual subjectivity classification. In Proceedings of the Thirteemth Language Re- sources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022, pages 1381- 1391. European Language Resources Association.
Nedima Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Van Yeung. 2019. Multi- lingual and multi-aspect hate speech analysis. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EEMNIP-IJCNIP 2019, Hong Kong, China, November 3-7, 2019, pages 4674- 4683. Association for Computational Linguisties.
Tomá Práček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection Czech and English Twitter. In Proceedings of COLING 2014, the 25th Inter- national Conference on Computational Linguistics: Technical Papers, pages 213-223, Dublin, Ireland. Dublin City University and Association for Compu- tational Linguistics.
Michal Ptaszynski, Agata Pieciukiewicz, and Pawel Dybala. 2019. Results of the PolEval 2019 shared task 6: First dataset and open shared task for auto- matic cyberbullying detection in Polish twitter. Pro- ceedings of the PolEval 2019 Workshop, page 89.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In NewrIPS.
Chengwei Qin, Aston Zhang. Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general- natural language process- ing task solver? CoRR, hbs/2302.06477
Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A be- havioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM 2015, Shanghai, China, February 2-6, 2015, pages 97-106. ACM.
Wuraola Fisayo Oyewusi. Olubayo Adekanmbi, and Olalekan Akinsande. 2020. Semantic enrichment of nigerian pidgin english for contextual sentiment classification.
Bo Pang and Lillian Lee. 2004. sentimental edu- cation: Sentiment analysis using subjectivity sum- marization based minimum cuts. In Proceed- ings of the 42nd Annual Meeting of the Association for Computational Linguistics, 21-26 July, 2004, Barcelona, Spain, pages 271 ACL.
Lais Rei, Dunja Mladenic, and Simon Krek, 2016. A multilingual social media linguistic corpus. In Pro- ceedings of the 4th Conference on CMC and So- cial Media Corpora for the Humanities, Ljubljana, Slovenia.
Bo Pang and Lillian Lee. 2005. Seeing stars: Ex- ploiting class relationships for sentiment categoriza- tion with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for
Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin- dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sen- timent and negative situation. In Proceedings of

---

the 2013 Conference on Empirical Methods in Nat- ural Language Processing, pages 704-714, Seattle, Washington, USA. Association for Computational Linguistics.
Alexander G. Sboev, Alcksandr Naumov, and Ro- man B. Rybka, 2020. Data-driven model for emo- tion detection in russian texts. In Proceedings of the 2020 Annual International Conference on Brain- Inspired Cognitive Architectures for Artificial Intel- ligence, BICA 2020, Eleventh Annual Meeting of the BICA Society, November 10-15, 2020, Virtual Event Natal, Rio Grande do Norte, Brazil, volume 190 of Procedia Computer Science, pages 637-642. Else- vier.
Julian Risch, Philipp Schmidt, and Ralf Krestel. 2021.
Data integration for toxic comment classification: Making more than 40 datasets easily accessible in one unified format. In Proceedings of the 5th Work- shop on Online Abuse and Harms (WOAH 2021), pages 157-163, Online. Association for Computa- tional Linguisties.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilie, Daniel Hesslow, Ro- man Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexan- der M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang. Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pe- dro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav. Canwen Xu. Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifcoluwa Ade- lani, and et al. 2022. BLOOM: A 176b-parameter open-aceess multilingual language model. CoRR, 0hs/2211.001100
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment amalysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 518, Vancouver, Canada. Association for Computational Linguistics.
Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and Treneusz Gawlik. 2020. KLEJ: Comprehensive benchmark for polish language understanding. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1191 1201, Online. Association for Computational Lin- guistics.
Nazanin Sabri, Reyhane Akhavan, and Behnam Bahrak. 2021. EmoPars: A collection of 30k inottionannnttted persian social media texts. pages 167-173.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Tae- woon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica. Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Ab- heesht Sharma, Andrea Santilli. Thibault Févry, Ja- Alan Fries, Ryan Teehan, Teven Le Seao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexan- der M. Rush. 2022. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representa- tions, ICLR 2022 Virtual Event, April 25-29, 2022. OpenReview.net.
Iyanuoluwa Shode, David Ifeoluwa Adelani, and Anna Feldman. 2022. YOSM: A new Yorùbá Senti- ment Corpus for Movie Reviews. AfricaNLP 2022 @ @ICLR.
Debaditya Shome. 2021. Emohind: Fine-grained mul- tilabel emotion recognition from hindi texts with deep learning. In 12th International Conference on Computing Communication and Networking Tech- nologies, ICCCNT 2021, Kharagpur, India, July 6-8. 2021, pages 1-5, IEEE.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment tree- bank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Process- ing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Maarten Sap. Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 1668-1678, Florence, Italy. Association for Computational Linguistics.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- rish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman Iyer, Anders Andreassen, Andrea Santilli, Andreas
Mei Silviana Saputri. Rahmad Mahendra, and Mirna Adriani. 2018. Emotion classification on indonesian twitter dataset. In 2018 International Conference on Asian Language Processing, IALP 2018, Ban- dung. Indonesia, November 15-17, 2018, pages 90- 95. IEEE

---

Stuhlmüller, Andrew M. Dai, Andrew La, An- drew K. Lampinen, Andy Zou, Angela Jiang, Angel- ica Chen, Anh Vuong, Animesh Gupta, Anna Got- tardi, Antonio Norelli, Anu Venkatesh, Arash Gho- lamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabhar- wal, Austin Herrick, Avia Lifrat, Aykut Erdem, Ayla Karakas, and et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. CoRR, abs/2206.0461
Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei Нe, Hua Wu, Haifeng Wang, and Feng Wu. 2020. SKEP: Sentiment knowledge enhanced pre-training for sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 4067-4076, Online. Asso- ciation for Computational Linguisties.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.
Yu Sun, Shuohuan Wang, Yu-Kun Li. Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. ERNIE 2.0: A continual pre-training framework for language understanding. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAT 2020, New York, NY. USA, February 7-12, 2020, pages 8968-8975, AAAI Press.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 task 3: Irony detection in En- glish tweets. In Proceedings of The 12th Interna- tional Workshop on Semantic Evaluation, pages 39- 50, New Orleans, Louisiana. Association for Com- putational Linguistics.
Erik Velldal, Lilja Øvrelid, Eivind Alexander Bergem, Cathrine Stadsnes, Samia Touileb, and Fredrik Jør- gensen. 2018. NoReC: The Norwegian review cor- pus. In Proceedings of the Eleventh International Conference on Languaee Resources and Evaluation (LREC 2018). Miyazaki, Japan. European Language Resources Association (ELRA).
Varsha Suresh and Desmond C. Ong. 2021. Not all negatives are equal: Label-aware contrastive loss for fine- fine-grained text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021. Vir- tual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 4381-4394. Association for Computational Linguistics.
Bertic Vidgen and Leon Derczynski. 2020. Direc- tions in abusive language training data, a system- atic review: Garbage in, garbage out. Plos one, 1.5( (30233000333
Arthit Suriyawongkul, Ekapol Chuangsuwanich, Pat- tarawat Chormai, and Charin Polpanumas. 2019. Sehhainpwwisssightt-eeiiie First release.
Deepanshu Vijay, Aditya Bohra, Vinay Singh, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018. A dataset for detecting irony in hindi-english code-mixed social media text. In Proceedings of 4th Workshop on Sentic Computing, Sentiment Analysis, Opinion Mining, and Emotion Detection (EMSASW 2018) Co-located with the 15th Extended Seman- tic Web Conference 2018 (ESWC 2018), Heraklion, Greece, June 4, 2018, volume 2111 of CEUR Work- WS.org.
Haruya Suzuki, Yuto Miyauchi. Kazuki Akiyama, To- moyuki Kajiwara, Takashi Ninomiya, Noriko Take- mura. Yuta Nakashima, and Hajime Nagahara. 2022.
A Japanese dataset for subjective and objective sen- timent polarity classification in miero blog domain. In Proceedings of the Thirteenth Language Re- sources and Evaluation Conference, pages 7022- 7028, Marseille, France. European Language Re- sources Association.
shop Proceedings, pages 38 46.
Marilyn Walker, Jean Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for re- search on deliberation and debate. In Proceed- ings of the Eighth International Conference on Lan- guage Resources and Evaluation (LREC'I2) pages 812-817, Istanbul, Turkey, European Language Re- sources Association (ELRA).
Sali A Tagliamonte. 2015. Making waves: The story of variationist sociolinguistics. John Wiley & Sons.
Songbo Tan and Jin Zhang. 2008. An empirical study of sentiment analysis for chinese documents. Expert Syst. Appl., 34(4)2222 2629.
Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-follovinn Ilama model. https:// i www.thcoo/ttttttteoooooooot wwwoo/tatuuuuuoooeeooooooossaaat abb stantord_a Ipacaa
Harald G Wallbott and Klaus R Scherer. 1986. How universal and specific is emotional experience? ev- idence from 27 countries on five continents. Social science information, 25(4):663-75 795.
Shuo Wan, Bohan Li, Anman Zhang, Wenhuan Wang. and Donghai Guan. 2020. s2ap: Sequential senti- weibo analysis platform. In Database Systems for Advanced Applications 25th International Confer- ence, DASFAA 2020, Jeju, South Korea, September
Mike Thelwall, Kevan Buckley, and Georgios Pal- toglou. 2012. Sentiment strength detection for the social web. J. Assoc, Inf. Sci. Technol., 63(1):133- 173.

---

24-27, 2020, Proceedings, Part III, volume 12114 of Lecture Notes in Computer Science, pages 743-749. Springer.
Zuoyu Tian, Yiwen Zhang, He Zhou, Shaowei- hua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xin- nui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020a. CLUE: A chinese language understanding evaluation benchmark. In Proceed- ings of the 28th International Conference Com- putational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020. pages 4762- 4772. International Committee on Computational Linguistics.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUT:: A multi-task benchmark and analysis plat- form for natural language understanding. In 7th International Conference on Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9. 2019. OpenReview.net.
Lu Xu, Lidong Bing, Wei Lu, and Fei Huang. 2020b. Aspect sentiment classification with aspect-specific opinion spans. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages Association for Computational Linguistics.
Zengzhi Wang, Qiming Xie, Zixiang Ding, Yi Feng, and Rui Xia. 2023, Is chatgpt good sentiment ana- lyzer? A preliminary study. CoRR, abs/2304.04339.
Online.
Zeerak Waseem and Dirk Hovy. 2016. Hateful sym- bols or hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop, pages 88-93, San Diego, California. Association for Computa- tional Linguisties.
Linting Xue. Neah Constant, Adam Roberts, Mi- hir Kale, Rami AI-Rfou, Aditya Siddhant, Adilya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11. 2021, pages 483-498, Association for Computational Linguis- tics.
Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li. Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. Indonlu: Benchmark and resources for evaluating indonesian natural language understanding. In Proceedings of the Ist Conference of the Asta-Pacific Chapler of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, AACL/JCNLP 2020. Suzhou, China, December 4-7, 2020, pages 843-857, Asso- ciation for Computational Linguistics.
Seid Muhie Yimam, Hizkiel Mitiku Alemayehu, Abinew Ayele, and Chris Biemann. 2020. Exploring Amharic sentiment analysis from social media texts: Building annotation tools and classification models. In Proceedings of the 28th International Conference Computational Linguistics, pages 1048-1060, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyaw- jjya,, Rahmad Mahendra, Fajri Koto, Ade Ro- madhony, Kemal Kurniawan, David Moeljadi, Ra- dityo Eko Prasoja, Pascale Fung, Timothy Bald- win, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2022. Nusax: Multilingual parallel sen- timent dataset for 10 indonesian local languages.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offen- sive posts in social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. Volume 1 (Long and Short Papers), pages 1415-1420, Minneapolis, Min- nesota. Association for Computational Linguistics.
CoRR, abs/2205. 15960.
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muham- mad Abdul- Aagul-Mageed, and Alham Fikri Aji. 2023. Lamini-Im: A diverse herd of distilled models from large-scale instructions. CORR, abs/2304.14400
Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çagri Çöltekin. 2020. Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020). In Proceedings of the Fourteenth Workshop on Semantic Evaluation, SemEval @ COLING 2020, Barcelona (online). December 12-13, 2020. pages 1425 1447. International Committee for Computa- tional Linguistics.
Rong Xiang, Xuefeng Gao, Yunfei Long, Anran Li, Emmanuele Chersoni, Qin Lu, and Chu-Ren Huang. 2020. Ciron: new benchmark dataset for chinese irony detection. Proceedings of The 12th Lan- guage Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 5714-5720. European Language Resources Associ- ation.
Chiyu Zhang and Muhammad Abdul-Mageed. 2022. Improving social meaning detection with pragmatic masking and surrogate fine-tuning. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analy- sis, pages 141-156, Dublin, Ireland. Association for Computational Linguistics.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng. Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson,

---

Chiyu Zhang, Muhammad Mageee, and Ganesh Jawahar. 2023a. Contrastive learning socioprag- matic meaning in social media. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2405-2439, Toronto, Canada. Associa- tion for Computational Linguisties.
A Appendices
A Benchmark
Table 7 summarizes language distribution of datasets in SPARROW and taxonomy of these lan- guage according Ethnologue (Gordon Jr, 2005) and Glottolog (Nordhoff and Hammarström, 2011). 1). Tables 9, 10, 11, 12, 13, and 14 describe the datasets tasks of antisocial language detection, emotion recognition, humor detection, irony and sarcasm detection, sentiment analysis, and subjec- tivity analysis, respectively.
Chiyu Zhang, Muhammad Abdul-Mageed, and El Moatez Billah Nagoudi. 2022. Decay no more: persistent twitter dataset for learning social meaning. In Workshop Proceedings of the 16th International AAAI Conference on Web and Social Media, ICWSM 2022 Workshops, Atlanta, Georgia, USA [hybrid]] June 6, 2022.
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023b. Sentiment analysis in the era of large language models: A reality check.
We empirically characterize the issue of data inaccessibility by re-collecting tweets content via tweet IDs. Table 8 shows the data decay issuc of 25 datasets.
CORR,
Qihuang Zhong. Liang Ding, Juhua Liu, Bo Du and Dacheng Tao. 2023. Can chatgpt understand too? comparative study on chatgpt and fine-tuned BERT.
B Models
CoRR, abs/2302. 10198.
B.1 Finetuning on Encoder-only LLMs
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large lan- guage models transform computational social sci- ence? CoRR, abs/2305.03514.
We evaluate the following Transformer-encoodr-- based multilingual PLMs on SPARROW. We fine- tune each PLMs on the full training set and update all the parameters of the model during the training. (1) Multilingual-EERT (mBERT) (Devlin et al., 2019) is trained on a Wikipedia corpus including 104 languages with masked language modelling (MLM) and next sentence prediction objectives. It contains 10M parameters. mBERT tokenizes text by using WordPiece with vocabu- lary size of 172K.
(2) XLM-RoBERTaBan (XLM-R) (Conneau et al., 2020) is trained on CommonCrawi data in- volving 100 languages with MLM objective. It uses a SentencePiece tokenizer with a vocabulary size of 250K and contains 270M parameters.
(3) Bernice (DeLucia et al., 2022) is trained with 2.5B tweets in 66 languages and MLM objec- tive. Bernice consists of 270M parameters and tweet-specific SentencePiece tokenizer including a vocabulary size of 250K.
(4) InfoDCL (Zhang et al., 2023a) further trains XLM-R with 100M tweets in 66 languages with two contrastive learning, MLM, and distant label prediction objectives. InfoDCL shows that it ef- fectively learns language representations for un- derstanding SM.
B.2 Zero-shot Setting on LLMs
We also investigate the zero-shot performance on a wide range of LLMs:

---

(1) BLOOM (Scao al., 2022) is Trans- former decoder-only model trained on the ROOTS corpus consisting of 16 natural and 13 program- ming languages. BLOOM uses a multilingual vo- cabulary with 250K tokens and is trained with auto-segressive language modelling objectives.
et al., 2022; Christiano et al., 2017). We expolit 3pt-3.5-tune 000rbo00301 via OpenAl API.
Experiments
C.1 Hyperparameters
To be computation friendly, we only tune the peak learning rate of each model in a set of { {1e-44 4,50 5,3c 5, le 5) and randomly select 45 datasets for hyper-parameter tuning. We fine-tune a PLM with an arbitrary batch size of 32, sequence length of 128 tokens, and 20 epochs with patience of five epochs based on the model performance set. We fine-tune each dataset three time with dif- ferent seeds and identify the best model based Dev set performance. The best learning rate for each model is identified based on the average score of Dev set the 45 datasets. The best peak learn- ing rate is 3c 5 for mBERT, XLM-T, and Bernice and le - 5 for other models.
(2) Multilingual T5 (mT5) (Xue et al., 2021) is Transformer encoder-decoder model trained on CommonCrawl data involving 101 languages and contains a vocabulary with 250K tokens. trained with sequence-to-sequene MLM objective.
(3) LLaMA (Touvron al., 2023) is a Trans- former decoder-only model pretrained on 1.4T to- kens where the majority are English and a small amount data in 20 other languages. We utilize LLaMA with 7B parameters and a vocabulary with 30K tokens.
Dev
(4) BLOOMZ (Muennighoff al., 2022) is also an instruction finetune model. further fine- tunes BLOOM on xP3 corpus that contains 13 type of tasks in 16 languages with English prompt. We benchmark SPARROW on the 1..00M-basee models with a size of 7.1B parameters.
C.2 Prompts
The prompts we use in our experiments are sum- marized in Table 15.
(5) BLOOMZ-P3 (Muennighoff et al., 2022) is an instruction finetuned model. It is initialized by BLOOM and further finetunes on English-only P3 corpus (Sanh et al., 2022) containing 2, 073 natu- ral language prompts for eight types of NLP tasks. (6) BLOOM-Bactrian (Li et al., 2023) tune BLOOM on a 3.4M instruction-following dataset in 52 languages with low-rank adaptation mod- ules. Li et al. (2023) translate the English 67K instructions from Alpaca and Dolly datasets into 51 languages and utilize ChatGPT API to gener- ate responses in the corresponding language.
C.3 Results
Table 16 shows aggregated performance of fine- tuned models Dev and Test-S. We report the av- erage of speecifc metrics and standard de- viation in task and category. We also report the Test-S performance tasks antisocial language detection, emotion recognition, humor detection, irony and sarcasm detection, sentiment analysis, and subjectivity analysis in Tables 17, 18, 19, 20, 21, and 22, respectively.
We provide concise study probe the sen- sitivity of open-source LLMs to prompts and present the results Table 24.
(7) mTO (Muennighoff et al., 2022) is instruc- tion fine-tune tund mT5 model with xP3 corpus. We evaluate the m --bbeed models with XL size (with 3.7B parameters).
(8) Alpaca (Taori et al., 2023) further tune LLaMA on a 52K instruction-followiing dataset that is generated by gpt- -- turbo of Ope- nAI API. The dataset includes diverse English instruction-folloiinng tasks, e.g., question answer- ing and programming.
(9) Vicuna (Chiang et al., 2023) further tune LLaMA on 70K diverse user-shared conversations with ChatGPT in English.
(10) ChatGPT is conversation-basel LLM trained GTP-3 (Brown et al., 2020) through rein- forcement learning with human feedback (Ouyang
"nttps://openal.cr

---

Lang, Familyy
Tang-
Code
Script
Anthric
unb
Ethiopic
Arahic
am
15
Arbhic
Dania
ary
Arabic
Afm--Asiatic
Drina
Arabic
Heleow
Hebrow
Maltese
Rumbara
Kinyarwands
Allanic Congo
Swotth
swh
Latin
Tws
twi
Latin
Isonga
150
Latin
Yoruba
yox
Lotin
Ansirnaviatic
Viamnmmse
vic
Latin
Acchuese
002
Latin
Buliness
hun
1atin
Danjarese
bin
Latin
Rugines
Latin
Piápino
B1
Indumesiion
Austnseiia
Javanese
jay
mad
Datavet
Study
Year
Original
Retrisval
Theray
Minangkabun
zin
Latan
Ngsiaa
nij
Latin
Sane engar
Riledf et (2013)
2013
5K
1K
0.41
Sunchmee
Latin
Smn-ropu
Puinak et al. (2014)
2013
7K
4K
0.29
Tolu butak
Mcc
Latin
Pubek (2014)
100K
59K
San--mgaus
Smith
19K
14K
Kannada
kan
Kannada, Latin
bulade:
(2016)
27K
Dravidian
Tamil
Malayzlam
mal
Malayslan, Latin
Tamil,
Sent detistr
bosgau
Mozeed et (2016)
Mozetić (2016)
2016
109K
44K
32K
20K
052
Tolugu
Seni-cngata
Mozeté et (2016)
2016
103K
43K
0.56
Semi-spitte
Mosuri (2016)
2016
275K
151K
0.56
ATlomian
Latin
San--evvat
Mozztič (2016)
2016
978
66K
0.32
Rolgaion
Cyrillie
Sem
Mouuti (2016)
2016
109K
40K
063
English
Czech
Bengali
Duniihh
Creatim
dun
ong
ces
27
Bengali
Latin
Latin
latin
Sen sloMar
Sen ellatee
Sem--sibtte
consporstee
polduo
risste
Mozzed et (2016)
Mozsti et (2016)
Mozza2
Mozee2
Mozeté et (2016)
(2016)
2016
2015
2016
133K
107K
70K
53K
16K
74K
38K
4IK
0.36
0.46
0.1
Fiench
Gemman
Ein
Latin
Sent San--sppMoe
Moztid et (2016)
Mozeti et (2016)
2016
2016
73K
5BR
27K
34K
0.42
0.63
Grock
Greek
Waserm and Blwy (2016)
19K
0.36
Desiagee,
Sen--rrio
56K
Indo-Euntpean
Maralhi
Nigerian Palgin
pem
Devnagei
Latin
lron-hing
Sesi-fiecsi
engao
Chiril et (2020)
Vijay es al (2013)
Rosenthal
2018
20K
12K
9K
0.10
0.22
Norwegian
Pessian
Tas
Arbii
Latin
Humo-hinag
Agpimaal al (2020)
2018
7K
SK
0.30
Porugguse
Polish
Table 8: Data decay issue in social media data. These 25 datasets are distribute by tweet IDs. We retrieve
Russian
Cyclllic
Spanish
apx
Serbian
stp
Cyriliic
these tweet on Nov. 2020 Jan. 2022 and find that 42% samples are inaccessible.
Sloook
<l1
Latin
Sloveniat
slv
Lati
Swetish
Latin
Japonic
Japanese
Jp0
Han, Hir, Kat.
Koczsnic
Korean
kor
Hangul
Sin- Tibetan
Chincas
sbo
Turkish
Urali
Flanish
tin
Hangaria
Table 7: Summary of languages covered in SPARROW. Lang.: Language. Language code is marked by ISO 639-3 code. Language information retrieved from Ethnologue (Gordon Jr, 2005) and Glottolog (Nord- hoff and Hammarström, 2011). The column dataset shows the number of datasets covered by SPARROW per language. Hin: Hiragana, Kat.: Katakana

---

Datasel
Study
Lang
Source/ / Domaln
Year
#Lb
Labels
Duta Slipt
Metric
Aegr hittorn
Kuate (2015)
hin
Twater, Facebook
2018
[Ageresivve Not)
9,506/1 11661116611110 164
M-FI
Dong aoA
Alstetn al. (2720)
Twiner
20201
[inggewws, Noc)
3,679911500
SEUT
Hate angwas
Waseem and Hovy (2016)
CR2
Twater
2016
3
[Noc Rasisn, Sckism]
8,083/1 1086111660 1,885
W-F1
Hate-meno
Davisson al. (2017)
Twilter
2017
Hatte Not, Offersive)
19,836/24788 012662.77//24477
W-FI
Hate
Alkko al. (2015)
YouTube
2017
Haate
9,014/1.1
M.FL
Hate-itagos
al. (201))
im
2018
Hatte Noll
2,700/306V1 10000
fllc
Caterag et (2019)
Twiter
Hate.,
10,0004,23244, 2220,4222242232
Hate-rmmtal
Mulli (3019)
2019
Aluuvis, Hiso Nea)
4.205/468/1
M-FI
Hate enBire
Bacile et al. (2919)
eng
Tomter
2019
Hate. Not]
9,000/ 00003,0000
Hil- Hut--pagga speius
Basile et al. (2019)
spa
Twimer
2019
[Hate Notl
4.500050011,0 2000
M..1
Hate
Fortana at. (2019)
por
Twiter
2019
[Hate, Not)
4,536/567267
Hate Hute-polin
Paszyaskies et al. (2019)
pol
Twitter
2019
[Hate Not1
9,037/1 00341 .000
M.FI
Hotee
Mara et al. (2000)
ker
News
2000
Hate, Not, Officssioe)
7. 1.10/7994471
M.FI
Hate
Makankk al. (2020)
Hatte
6,8391 00002.000
Hate- Hate-dhppan
(3022)
2002
25.72666,4118
Hut-lorjo
Iooag ala (2023)
News
(Hate Not
22,3436404444004
M-F
YouTube
com-
ment
Thte-tehaa telau
Marroidly all (2022)
tel
Mise
2022
2
[Hate, Not] [Noc Sesism)
24.99//.51077.000
M-F
Soui fraçui
Chiril et al. (2020)
fra
Twater
2018
2
7,070/999999
ME
off--rgaass
Zumpieri cl (2019)
eng
Twitter
2019
[Nor. Offamie!
11916/13344660
M-E
Offo artzen
Zumpieri dl (2020)
aza
Twiter
2019
Nox. Offensee
7,255784/1.55 527
Offe-danges
Zampieri d (2030)
das
Mise
2019
NNx. Offenise
2,664/206/320
M.FI
olle-ellzan
Zampien ol. (2020)
ell
Twiter
2019
Offensive
7,868874/1 554
M-FL
Affe-urzann
Zampieri (2030)
Offamise)
140/3 28/1,515
Offe araaa
Mularak
Offensive)
10002.0000
M-F
Officalogss
Knij Newak (202))
Twilter
2000
AAprepriase, huppropiie, Not, (ensive)
65,021/8 127
M-FI
Offe-G- Gaag-an
Zumpieri (2019)
002
Twitter Twiller
2019
[Grons, Indvvddall Others)
3,485190/213
M-L
Hate- Gestooa
Quodlloam (2019)
2019
Afriia dincent. Arabs, Asims, Chistian, Gay. Immignatss la- dunthinds. Individtadl loss. Mar- lims, Others, Refigaes, Wamen! (Atrican discunt Arbbs Asime, Chaistaan. Gay. Gispanics, gants Jess. wing people. Others. Refugess, Special needs. Wemen))
2,682/34/335
M-FI
Onsddhomm et al. (2019)
fra
Twikter
2019
16
3,201401002
M-FI
Indivitial,
Ont--enngzoo
Zumpieri (2019)
ong
Twitter
2019
(Targeted, Unnrygtod) (Gender Origin. Others, Religion] [Dieahility Gender, Origin. Others, Religion. Sexual Orieniaion) (Geopeliiical Pessonal Folnical, Re-
3,96243773440
Thte-T-rripes
Qusdlhoam al. (2019)
Twitter
2019
2.8823344356
M-FI
Han-T-fiiqus
Qusidhoum et al. (3019)
fra
Twilter
2019
3,21 140114022
M-FI
Hate I Ebenga
Karita et al. (2021)
bes
Misc
2029
4,5505705700
MI.11
ligion)
Offe -ake-Ta
Clickrovrtti
kns
YouTube
2019
Individual, Not, Others, targeeed)
4,094596693
Cluktovrrtii ot (2022)
mud
comment
[Grong, Individial, Not, Uncargeted]
14,723/1 886-11344444
M-F
Choknwarthi (2022)
tm
Thlee commert
2019
[Gnory Indivihual, Not, Others, (agesed)
33,685/4211644200
M-FI
Hane--oorja
Isong al. (3022)
lcr
News
and
2002
(Cimugs Individiod, Other, Unme- 16, 16.2192.94020003
M-FI
You Tube
com-
geted)
Table 9: Description of 36 antisocial language detection datasets. Lang.: Language is marked by ISO 639-3, #Lb: the label size dataset. M-FI: Macro-F1, W-F1: Weighted-F1. * indicates that data sharing needs approval from the original authors.

---

Dataset
Study
Lang.
Source/ Domain
Labels
Data Slipt
Metric
Bonot-mpww
Wallbott Schener
Questionmane
(Anger Diggns, Ferr, Guilt, Any.
6,1322677677
Sudees, Shame]
Emot-zhloas
Loe Wing (2015)
(Asger. Fesr, Happy, Salnew,
3,122/47/418
Accuncy
peisel
Emoc-lingg
Kejoua (2018)
fi
Subelle
2016
(Arger. Articipation, Diggut, Fear,
5,1999977655
M-FI
Joy, Sodness, Suppise. Trust)
Funoe-frayai
Kajina (20 (8)
Tri
Sahelle
2016
(Anger. Anticipationn Diguud Fesr. Joy, Sudness, Surprie, Thust) (Anger Anticipution, Digitd, Joy,
5, 5,1999771651
M-FI
Fonct-iinaa
(2018)
Scheille
5,197/977053
M-FI
Supnise, Trust]
Abdul-Maage Mageed et al. (2020)
Toine:
(Asgar, Astelptton Diggut, Fear. Joy, Stdness, Suppise, Trust] [Asger, Joy. Optimisat, Sadaess] (Aager, Joy, Fear. Sodness)
50000810941
M-F1
Emot-cntaaas
Mohamandd et al. (2018)
cap
Twine:
2018
3,257/574/1421
MIFI
limot
Mohammad et al. (2018)
Twiter
2015
2,280999111
M
Enoot-ppogua
Motamaned et al. (2018)
Toines
2015
(Anger, Joy, Foss, Solness)
2,70844981.666
MIF
Finet-indsp
Sapurri et al. (2018)
Tuittr
2019
(Anger Fexr. Heppy Lave. Sulwoo)
1,5396440641
M-FI
Gives
Toine
(Aager, Fess, Happy. Sunees, peisel
3,2004000400
Accumey
Emoo--indyu
Will (2020)
Toitter
2018
(Anger. Four Hoppy, Lave, Sidsess
M-FI
Simo-vient
Ho et al. (2019)
Via
Fleebook
2019
(Augar. Disgust, Foar, Joy, Others, Salres, Suppise
5.5458866698
WIFI
Eimot engen
Plzza del Arco all (2020)
Twittrr
2020
(Anger. Diggst, Poor. Joy, Others.
53427700731
M-P1
Sudros. Surprise
Lmnot
Pizza Arco (2020)
Twitter
Diggut, F'err. Others.
6.727/84-/841
M-11
Sudaess, Suspiise (Anger, Antccppaton, Diguzz, Pear. loy, Sudnees, Surpris,, Trusi] (Adminaionn Amuseneat, Anger. Anteyanse, Agpinvzl. Cuing Con- tusion. Cuitosty. Desire, Disappoint mest. Disagpeoval Disgusi, Endee- Excitement, For. Grai- Grief,
(2020)
Suberde
82661,1111,0666
ME
Emot engtker
Demseky at al. (2020)
ccp
Reddi
2020 27
25.483/2,900029944
MI
Opiinism, Pride. Romerar, SaannssS, Suppisch) (Anger Feor. Juy, Sadness) (Asgar, Feuz. Joy, Sodness)
Emnot-iiapa iitiis Emot-roiiioo Limot hinpch
Bianchi et (2021)
iia
Twitter
2021
1,28/200/011
M-E1
Cloboraru and Dinu (2021)
ron
Twine:
2020
2.600318/524
M.F
Shome (2021)
hin
Machine Transia-
2021
27
(Adminsion. Amusement, Anger. Anneyance, Appioval Carngg Con- fusion, Curinsly. Drsine, Diuppiin- Distppooval, Disgust, Enka- Excitement, mdde, Grief. Love, Nerrousess. Opiinism, Prick,, Realiscation, Relief Remorse. Sadness, Surprze) (Adminsson.. Amusmment Anger. Annoyance, Appeoval, Compussion, Contusioa, Cuissity Desiie, Dissp- Disappoooal, burraenent, Exciieent Gratitdd, Joy. Longing, Love, Narvmanss, Optinien, Pride. Ro- lief, Remorse, Sxdtess, Surpeise) (Anger. Forr, Happy, Haree, Sul- ness, Wunder)
25.88/22990622994
Mi11
tinn
Grai-
Emmot- Conee
Cortic atal, (2001)
Twittrr
2021
28
24.91972 22..26/122.966 9,96
M-FI
Fince-faesuh
Sahri et al. (2021)
fax
Twitter
2021
4.1 4,1809228823
M-FI
Emoct rusoo
Sboev et (2023))
ru5
Mile
2021
(Aager, Feur, Joy, Sodness, Surprss) (Anger, Disgust, Fear. Sudness, Suspisel
3,85142771.288
M.EI
(2022)
Misc
500000000000
Mi11
limot fraps.
Biancti (2022)
Machine
Transla
(Anger, Feur. Juy, Sodness)
3,7981766/76
Funet-dbrqua
Binchi (3022)
Machine
Tranch--
2018
(Anger, Foxr, Juy, Studnene)
3,7984764466
M-FI
tinn
Table 10: Description of 26 emotion recognition datasets. Lang.: Language is marked by ISO 639-3, #Lb: the label size of a dataset. M-F1: Macro-F1, W-F1: Weighted-F1.
Dataset
Study
Lang
Source/ Domain
Labek
Data Slipt
Metric
Hume hig
Aggansall et
Twitter
[Humon, Not]
4,187:524/523
Aucuracy
Hune-maga
Blinew (2019)
mi
Mis::
2018
[Humoe, Not]
251.416611.9941
M-FI
Humo-1p9e3i
Chanzzo al. (2021)
4pa
Toiner
2019
[Humoe, Not)
24,000%%000000000 000
M-FI
Humosergtalo
Money all (2021)
sng
Twittr
2021
(Humoe, Not]
3,000 1.000/1 1000
M-FI
Table 11: Description of four humor detection datasets. Lang: Language is marked by ISO 639-3, #Lb: the label size of dataset. M-F1: Macro-F1.

---

Study
Domain
Slipt
Metric
Basile (2014)
Twitter
Not]
4,0624551
M:11
loon-ppina
Bulkeri (2016)
9pa
Twitler
Noc]
6,06NT41/1,
engrae
(2018)
Not]
344003647744
lon-itics
Cignarella
Hicry,
Iron binyg
Vijey et (2018)
Min
Twmer
2018
[Looy, Noc]
2.217/22//27
M.F1
tron amrm
Gianem et al. (2019)
Twitter
2019
[Iooy,, Not)
3,622/00/1.00 000
M-F1
Tron-qptox,
Outege-Burnoet sl. (30
Twitter
2019
[rony. Not)
2,1802466000
M-FI
Golanzian (2020)
Twitter
[rroy, Not)
2.380/29/294
aloogia
Walb
MusufiilenBBvvdeee
7.0148768766
M.FI
Uniikely
Imooy)
enewal
(2012)
Debate Foru
NNot, Sureen]
900/1009955
M-FI
San--mper
Ríluff (3013)
(Nol,
1.413/177177
Sare-ceiew
Noo, Sureean
Saro-mphuu
mng
Twitter
[Nu, Smmsen
71,4338,
Saro-cigRov
Bemman and Smith (201.)
cng
Twiter
2015
Noot, Sanvenn
11,86411 A88/1 .444
Accunacy
Sare ingsy
Regadeningm et (2015)
Twittrr
2015
[Nol, Smosen
41,261/5 15655 158
Accumaty
Saro-engrra
Onuby al. (2016)
Debate Furm
2016
[Not, Smnzen]
200100/2.266
M.FI
shocee
Gong (2020)
Nows
NNot,
3,97849974497
Ahu
Maply (2000)
(Nul, Sanzen]
7,902/44422110
arpuur
(2021)
[Nol,
11.2901.25530000
Tennsier
(20L8)
Twittr
Hosaicc by
3,450/394754
M.FI
Sinatiool
Table 12: Description of 20 irony and sarcasm detection datasets. Lang. Language is marked by ISO 639-3, #Lb: the label size of dataset. M-F1: Macro-F1. indicates that data sharing needs an approval from the original authors.

---

Datavt
Study
Lang.
Suuran// / Dommin
Year
UT
Labele
Data Slipt
Mitric
Sent engrae
Pang and Lee (2005)
eng
Moview review
2005
2
(Necative, Positie]
N.52W1,000 0006/ 1067
Accuracy
Seat-2hequ
The and Zhang (2008)
2210
Mise
2005
(Negative, Posiive]
9,66011,2001 1.00/12200 2000
M1.51
Seat I-enatho
Thetwall (2002)
eng
2012
(Negative, Postive]
900-100/1.13
Sea-Y-eeghy lne
Thetwall (2012)
cig
YouTube
2012
(Negative, Fosiive]
900/100/1142
Accuracy
Sechor (2013)
mg
Movieu
(Negaiie Neutal, Pue-
8,5447, 1002,3110
Ausiney
line.
Very Negaties
Very Postivo))
Sent-korao"
Jung al. (2013)
lor
News article
2013
[Complex, Negative Neural.
4,187/523524
M-FI
Postive]
Seat engsue
Socher al. (2013)
esg
Moview review
2013
2
[Necative, Positve]
6,92092211 (22
Accuracy
Sent
Basile al. (2014)
it
Twitter
2014
(Nepaivee Preitiee
1.396/265/1 1207
M-F1
Fanus
Bubien (2016)
Twiner
2016
(Negative, Fostive]
3,738646/11001
Sent-milia
Dingli Sunt
Movics review
2016
(Negatve, Rodiive]
506/851171
Seat-bulste
Mozetiž (2016)
Twiner
2016
(Negative, Neutta, Postiive
22.88422777322.772
Sent-borstee
Moretič al. (2016)
bos
Tuitier
2016
(Neptive. Nautal, Povilive]
16,3999.040/2.042
Seat-deusdee
Mozztič al. (2016)
deu
Twiner
2016
(Negative, Neutral, Positive!
42,01005,25 251.
ME
Sent-erggsou
Moretič at al (2016)
eng
Twiner
2016
(Negetie. Nasttol, Puxiivv]
54,53844 31 3177/.118
M-FI
Sent spladur
Muzetic al. (2016)
spe
Twiteer
2016
(Nepative Neutral, Posttive;
122.41OV1 2022
ME
Sent-hi Gett-hrydde
Moretič sd (2016)
lev
Twitur
2016
(Negeive, Noutal, Postive)
52,97 166 /22
M.FI
Sent
Mozatic (2016)
2016
Neutral, Postiie)
32,71 (00)) (880)
Scat- Sest-peltte
Mozati (2016)
pol
Twiner
2016
(Negative, Neutral Postiie]
87.941/10.99//0000
poraise
Muzeti (2016)
por
Twiter
2016
(Negative, Neutral Postiiv]
39,52544,441/45590
Seat-mistts
Mozetič (2016)
ris
Twiter
2016
(Negative, Neutial, Poillive]
32,84118,1 11774.118
MEEL
Seat slisteer
Mozzti et (2016)
sik
Twiner
2016
(Negative, Neutral Positive]
30,09443 857738857
Sent-cbrsto
Mosetič al. (2016)
sty
Twiner
2016
(Negative Neutna, Posilive!
59, 99,924/7-00 40177.400
Sant-uthate
Muratič all (2016)
upi
Twiner
2016
(Negetie, Nautna, Positive
2M, 10,775/ 67233.622
M
Sent-erpui
Mosetič al. (2016)
Twincr
3016
(Negrive, Nautal, Pusiive!
22, /24/2 766/2.766
MIFI
Sent-oweydue
Mozetič al. (2016)
Twitur
2016
(Neprive, Noutra, Potiive;
27,07723 400/3,11
Seat detRei
(2010)
2016
(Negative, Neutal, (Negatvee Neutral, Postivee
2.001/3572338
(2016)
spa
2016
6.09W7661/122
NET NET
Seat Zagei
Rei (2001)
ita
Twiner
3016
(Negative, Neutral Postliv]
6,8188558852
M
Sent-criuue
Rosenihal et (2017)
eng
Twitter
2017
(Neptive, Nautal, Posllive
42,75664 7,1/122844
M-Reccifi
Seat- beut-beirss"
Patm et al. (2018)
bea
Twiner
2015
(Negative, Neutral, Posilive!
2, 2,200/2503.008
MEL
Sent-hinps:
Patm as all. (2018)
hin
Twiner
2015
(Negrtive Nistaal, Pusiive]
M.FI
Seat-
Ancam et al. (2015)
libb
Freebook
2015
(Negative, Positvel (Negetie, Nautall Prvttiv]
8,055/995/2488
Accuncy
Sent Semteguravo
Beam and dis
Gragas
Twitter
3017
45,/2715, 585/56377
MIFI
Nunes (2018)
Kajoss (2018)
lin
Suberke
2016
(Nepeiie, Prestive]
5,1975778681
M-FI
Sent-
Kajosa (2018)
fra
Suberte
2016
(Negative Rositive]
5,1955778653
MFI
Seat itaci
Kajera (2018)
it
Subcile
2016
(Negative, Positive
5,197/571683
ME
Sent zungol
Validal al. (2018)
nor
Online review
2018
(Negativel, Negaiiet, Neg- atiwe, Postived, Posilive Postivet)
34,004
Scat-pelice
Kocod al. (2019)
pol
Customer rexiew
2019
[Congles, Negativo, Nantal. Poutire]
5,170574/12217
Seat-fingar Seat abewos
Sariywongkul et (2019)
tha
Freebook
2019
(Negative, Neutral, Positive]
21,052/23362/2.66
Wan et al. (2020)
zbo
Weibe
2019
(Negative, Fositivl
95.9MN11
Sent-lispua
Astzaai Asli et al. (2020)
fas
Cusomer resiew
3020
(Negative, Neutral, Povitive! (Negptive, Posétive]
75,09449 3679 387
Samt-rumpu
Diminescu et all (3020)
nn
Cusomer review
3020
16, 146/1.795/1 1005
Sent-pemoye Scat Scuz-pelayy
Oyewasi et sl. (2000)
Twintr
3020
(Nageiie, Nautaal, Puvitive) (Negative Neutal Pos- ilise,
11. 11,20001,4007 40000,400/..00 400
M.F
Rybak (2020)
pol
Customer review
2020
5.619995/000
Very Nagutios
Vary Postivel
Sent-indhast
Wille es (3020)
ind
Mise
2019
(Negrtiee Neutnd, Pavitise)
9,00001, 100/1
Sent amaad
Abdul Mapeed al. (2021)
ara
Pwitter
2021
(Negative Neutral, Neuiral.Positiee Posilive)
19,10144,4439.98
Sant-Pormpu
Dialluet al (2021)
hom
Misr
3021
3
(Negetie. Niutra, Paxilive)
2,436/105/005
Seat bettya
Islam et (2023)
ben
Twiter
2021
(Negative, Neutral, Posttive
12,575/1,57711 1.86
Semt-morgua
Kulkami (2021)
Twittr
2020
(Neprivee Neutral, Postive)
12,11441.500/ 500/500/22500
katteta
Chuknavatti (2002))
YouTube
2019
Postve]
3,995/05/5122
Seat-miile
Chukmarathi (2022)
mil
YouTule
2019
(Negative, Roslive
9,41001.04/11009 1099
MIP
samch
Chuknoratti et (2022)
tan
YouTube comment
2019
(Negative, Fosttie]
24,06//296663000
Sent-amtgaa
Abdul-Maaeed al. (3022)
sea
Twiner
(Negtive, Nautaal Poxitiee
1.500/900/..0000
Sunt-ambaad
Vimam et al. (2020)
amh
Twitter
2020
(Neprive Neutra, Positie)
5,98411 (017)4000111 599
Sent-arystch
Mohummad et al. (20236)
ary
Twiner
3021
(Negative Nautal, Poxilie!
5.5834404/.961
Sent cnnt-an
Mulammad et al. (20236)
ang
Twiner
3021
(Negrtive, Neutral, Posiiive
1.811414 14955
Sent-hamggu
Muismmal et al. (3022)
han
Twintrr
3021
(Negeive, Nantal, Poxtive]
14, (L./77/..67//... 3013
Sent
Muhammade al. (2022)
ibo
Twiter
(Negatiee Neutral, Postiie)
10,192/1, 84133682
peilkdui
Mutaamad (2022)
pem
(Negative, Neutral, Postive]
5,121/1.281 11221/2211144155
kintuu
Mutammad (2022)
kin
Twiter
(Negative, Neuttal, Postlive
3,300/E221110 1025
Seat-swligua
Mulasmadd et al. (2022)
swb
Twiter
(Negative, Ncuttal, Posllive]
1.8104537745
Sent-twran hash
Muhummal et al. (2022)
ba
Tuitter
(Negaive. Nautnd, Puvilise
804/201/254
Seat-twisthh
Motaamad et al. (2022)
Twiner
2021
(Negative, Neutral, Posilive!
3,481/388,9999
WHEL
Samt-yrraaa
Muhsrmad at al. (2022)
you
Twiner
3021
(Negetive. Niutral Posilve]
8,502/7.00044 515
W-F1
Seat yurgui
Stade et al. (2022)
yor
Misc
2021
(Negative, Positive]
50002000500
Sent-jrmt Sent-jrnauu
Sunkki al. (2022)
SNS pool
2021
(Negriive, Neutral Poe- live.
20.00007,500/2.500
Ausincy
Very_ Very_Nesative.
Very Postiiee
Seat azewin
Winata al. (2012)
aze
Trans onlize com.
3022
(Negative, Neutral, Positive]
5001000400
Seat-bettgi
Winats cl. (2022)
bun
Trans online com.
3023
(Negative. Nautral, Posilive]
5001003400
M.FI
Sear bbuwbbbo
Winata et (2022)
bbc
Trans of onlise com.
2022
(Nexative, Neutral, Positine
50V180400
Wintss cl. (2022)
lije
Trans of online com.
3022
(Negative, Nautal, Pusilive
500V1004400
M-F1
Sant hupwo.
Wintt et (2022)
bag
Trans of online com.
2022
(Negative Neutral, Positive
500V100H00
M-F1
Sent-javwin
Winats al. (2012)
Trans of ooliin
3022
(Negeive, Noutal, Povilive
50N1103400
M.FI
Sent Sent-moolpin
Winsta al. (2022)
mod
Trans online
2022
(Neprive Neutral, Postiiee
50AV100410
M-F1
Sam--ningg
Winns (2012)
Trane online
3022
(Nepeive Nautal, Pustise]
501V1039409
wi/ww
Winza al. (2022)
ni
Trans online com.
2022
(Negatiee Neutral, Postlive
5000100910
Seat-sumaja
Winata et (2012)
sun
Trans onlise com.
2023
(Negative, Neuttal, Postiiv]
5001103400
NI.1
Sent relstar
Marredit al. (2002)
tel.
Mise
2022
(Negative, Neutral, Posttive)
24.9995,510770000
M1F
Table 13: Description 77 sentiment analysis datasets. Lang.: Language is marked by ISO 639-3, #Lb: the label size of dataset. M-F1: Macro-F1, M-Recall: Macro- Recal, Trans. of online com.: Trans, of online com." indicates that data sharing needs approval from the original authors

---

Datnset
Stody
Source Domaln
#Lb
Labels
Metric
Subj-cigrrs
Pung and Loe (2004)
Maview review
2004
[Olyective Subjetive)
8, .000
Accumes
[Agreement, Arganeat, Emution.
4.284535/536
M-F1
Speculation)
Subj-htgg
Rusile (2014)
Twitter
[Oljective: Subyerieer
4,06
Sobj stauu
Barbken et al. (2016)
113
Twitter
2015
(Cloectivee Subectivee
0,669971111 9433
M-11 M-F1 Accurccy
Schj-spiee: Subj octpu
Barbiesi et al. (20LG)
9pa
Twiterr
2014
[Objective Subyctivv)
6.669744111
Ihibin and Stennherger (2022)
Moview TOVIEW
2021
[Obsective. Subsective
7,20020022000
Table 14: Description of four subjectivity analysis datasets. Lang.: Language is marked by ISO 639-3, #Lb: the label size dataset. M-F1: Macro-F1. indiccttss that data sharing needs an approval from the original authors.
Dalaset
Prompt
(ooment)
Dang-snia SPAA
langage of [labele]
Agar hingun elote- eng.a.. Halte-aRa Ragee fllp moyt Hare olgga. Hate spop.- porpee polpu- korsteo
(ooneest]
lhle mratt Hale zhrppes laate kanlee sebtan Sest tia.e. Olfe engzan
Question: languge of text [labelu)?
Offe arpznn Offe datzzan Offe clizan. Off--urzan Oft--aragg Offe-diNeer
Anower:
(Coment)
Olin-T-ancka Dile Dile-G-mgaan Toorgee
Question: Does this offoméve teat targe (laheis])
Answens
Ant-
Question: Does this hate speeci target [labels)?
(Cument)
Anower:
Cummra
Offe
Question: this ocfensive text (labels) insul??
(oomnni
speech agrinsi people (lubels]
Answer:
Consent)
Hhte Tbenga
Question:
Does this test expeess (labels]?
Answer:
(Coment)
Offi-Temlaa Offi-Tiommss
Question: this errienae hore spench on mott? If diess this ventence
larget individual, group not?
Answer:
Emoelen
Emok-zntwu: Emot-Shhuue Emot-linga: Emul-finga Emol-ingg,: Emnt--maigaa Fmnot-mais
(Coment)
Tonc--mpaa Emut-arnge -pitcc. Fiuttinilag Fmui-tinga Emui-pptss Emmet-inhgg
Question:
(labels)?
Emot-vieu.. Emot-ctgg,, Emut-sppp,, Emns-finann Emuteeggee Emoc-iispiy, Emot-ronc Emoo-hinpaa. Emoo-gottue Eimo--iegaa finga. Fimot migho. Emott-bmyp. Emot-frqu Emut-deung
Humee
(Contet)
Humo hinagr- Humo susms Humo Humo
Question: this arreenor (labels]
Answer:
Ioon Ioon-inngs Iron-engtee: Inoeviitçi Ironshing. Immeanicaaa Ineneppiie: Samo-nnwwu Surc-ndpa Strc-engp, cagRinn San-enpauy mie Sumwwhhran' Aarr Sanc-srapor
(Cumere)
loon-Sagaa Imn-thhoaa"
Questio:: (labets]?
Irony
(Conent)
Inon-Teee Temmpee
Question: type this lest [ahelky?
Answer:
Scnn maanen Sent thora. Sent lotiss Sentl eagsee- Seat-itane: Seat Sent-benryn Sent hitey Sem ihe... Sent migis. Sent-bllaeer bilate Sem botadan Sent-deibtte Sem engatee. Sent spistee Sont-hrittoo Sent-huiagar Sett-pohoser Sett
Sent-ttsysee Seat illystn shiyte Seni-sojttz: Sent-mthonn Sentoowigae Sent-driga Sem--page Sent-iliga: Seni-ennaa Scut-laky. Scat-tnxe, sont-iling. salt-polco.. Sem-thine Sem zhowae fataa, comminn Sent penty. polay, indan. 1taat
Sentimen Anaiysi
Sent-hebasrr
Questioa:
(Coment)
this [labels]?
Sem- hamau.. Sent- bette mon-mmhee. Sent-lanes Sent-mokeu Sen tames Sem orhaah
Son anbijat Seat aryseh Seat arjasa- Sem habisak Seat ibosud Seat. petiuta Seat kinsuc
Sem soltstkk Sent-tooyaa: Sen (Wistts Sumauu Sent youshu- Sent jpusu.. Sent arewis
Sem banwie Semt bbewie- Scat Seat-Sjngge Seat bugwar Sent joruse. Seat aiddy Sent Sent
Sem Sent
Sentó
Cummra
Somt
Question: [Iabets]] Higher
(Contert)
Seljestov
Subj Korte
Question: sentence express (labele))
Anower:
(Content)
Sabj eseros. Suby litt... Subj Itnte.. Sobj spitaar Subjicerps cerps
Question: amrenno (Ibbds]?
Answer
Table 15: Prompts use for zero-shot evaluation with inrevaluation-hareeee

---

Dev Set
Tos-S Sct
BBEET
XLM-
Bernice
InfeDCT.
BBEETT
XLM-R
Barrice
InfoDCL
Apgressive
73,2240,50
76.79 5.552
76.09 2..35
22:1+-192
74.64-0.114
75:45
7.996-00.91
Dangerours
69.76=1.56
62:53++1.04
74.92=1.188 1.18
73.42+0.80
62.36 12.3
68.57=+1.15
67.13-0.56
65.23-+11.0 1,00
Hole
72.7320076
79.401:..55
81.16:11:15
80621:000
72:97
74.372111 1.48
76 76.76-2433
75:85:10.00
Antiocial
Offanee
18.96==1.04
00.21
62.15:20.68
81.5540.466 46
77.58
75 25.84
78.45-1899
78.88-22..3
100-Grorp
$1.57.22.55
61 41.24:33.16
48 18.00:1::00 190
46.0112.25
46 182 10
42, 3.130
51.152001
50.24---.19
HOO Targer
54.02-3.00
59.25-11771
00.83-1126
60.14+1.21
53.16:24.499
57.67:41.79
00.96=2266
60.79:1.358
AS
10.18:1.559
11.47:11.24
13 1:15
7506
2.222222.99
67 1.84
21.14=2155
70611110
00.00
68, 68.56:20:35
68
69-2731 L.
Tlrmrr
35
56.721005
K3 1511111
56.75-1113
Iun
1.20
1.0
72.19:124
71121077
K5. 65.5551133
66:38 122
66.36ET000 1.00
IAS
Succsm
Type
$3.51-2.11
74.65=1114
52 12.1822.15
75 1..64+1.922
77.22+156
58 98.55433117
77.21+1055
27.7242.92
1.34++1355
47.35 189
46.41:20.61
73 23,40+2222
74.78-169
56.04-177
S7. 57.58:11.4 442
74.94+-1113
R&S
71.3321226
71 1.58
1..0911.00
67.0811331
19.9111995
70:29-1.49
71.1211.00
Sentisennt Sobjecmvit
40,77
72.95 +0088
66.30+1922
09.55:11411
70:4--101
75.1810055
7122110.4
76.9/10099
77.781088
72.5111.96
70.80 108
75.7511333
SM
19.39:1117
71.9010.91
73.66:09.8
73446:0.3
66
838
68
1.53
70.85_163
Table 16: Performance of finetuned models on Dev and Test-S set. We finetune each model on each dataset for three runs with different random seeds and calculate the mean and standard deviation of -ataset-specffic metrics over the three runs. We report the average of datasct-specific metrics and standard deviation in a task and a category. I&S: irony and sarcasm.
IELDOMI
(MF)
BLOONEZ.
BLOHINZ-
HCTH
Agghiee
MiP1
144
72.21
32,66
15.45
12.55
11.00
13:52
13:55
10:12
15:37
11.07
1392
22.09
13:31
0120
2um
62:55
$3.30
11.05
(2018)
45.37
14151
4U31
(3GE))
WiF1
$5.66
39:01
WiF1
1.69
19:25
1673
8-11
1536
64.55
Hunkue (511))
130
25.03
306
11.07
$-40
$1.36
8136
15.00
012
1635
0136
025
0L3
F7AL
15:03
Buic mal 12018)
00.38
1542
25.40
23.46
25.4
33.6
61.55
12:11
MaBics. 2009
21:01
3706
13,9
gal
56.0
13.33
37.50
26.1
31.58
31551
3096
6635
11
Balk (210))
Halpero
7018
300
14.49
4001
31.90
4000
3527
3166
85.43
15.06
Prse-cii (311
1011
15.34
35.10
63.1
64.33
63.9
300
15.00
11/4
20.5
35.30
1600
2633
1600
23.0
1630
33.00
40.0
Moad a (333
Ha-dras
35.55
9131
15.2
2011
36.2
12.54
14.56
(312)
Ba boh
7000
18:38
31.0
3566
(3.1
$7.00
Jung mul (362)
11:32
(207)
Gui
15.55
2910
65.55
1191
Carl (N69)
15.54
8521
91.55
55.5
1726
-400
1407
6-6
12.0
66-
$331
(7.5
2t (330)
41334
43.77
16.44
0.44
45.4
4521
0334
34.0
Zurknnc (232)
43:31
75.11
44.0
1902
44.3
4-30
3611
31.1
35.33
43.0
Zueeko (222)
6631
300
16:0
15.8
1596
1250
12.30
13.97
11.0
1681
M111
6.90
06.1
7211
23344
1100
1.3
121
811
1.5
606
633
140
15.5
11:0
10:00
(nla a (309)
1511
23:35
03.6
411
47.2
4000
-182
42.0
SUAT
45.7
45:7
6655
Zuzkkii (230))
23:42
1954
1558
1/4
44.5
Ocidboan ata (260)
1936
11.8
Ha-Tacite
2283
0.30
15.11
1334
111
15.2
22.0
135
12.0
11211
2.1
43341
15,0
1276
264
4333
4080
Satmici il (382))
12.11
4.12
1622
11.11
11.5
24/6
206
3081
24.6
3600
24.0
3-66
3606
SUH
1832
416
Chilorrriid di (2822)
3.10
1020
HasTirta
M.1
22311
65.55
2337
323
14.00
13.14
41.33
(2G2)
Anargg
25:30
66.00
$7.99
11.34
33.11
33.00
22.00
27.00
51.0
31.50
13114
3240
28771
31/8
31.5
31:50
55.55
(7.40
Table 17: Full Test-S results Antisocial task. SoTA: Previous SoTA performance on each respective dataset. Underscore indicates that we have different data splits to the SoTA model. Best model of cach datasct is in bold.
Dibocct
Netric: Ranatees
SLM.- Benate:
BLOON BLLDONE
BLDOMME BEDDMNZ
(347)
$25
TE. EELDONIZ Sucriia
1875
EATO (4T1
LLaAMA
Alpec
Vieasi ChaiGrt
ChaiGiT (XTT
SeITA
09:51
20.10
16:36
11,45
24.55
4.5
23:01
37:31
50.25
41.01
(8:55
18:15
$7.00
Search One (3I2L)
N-F
13.32
40.13
25
4154
2037
4.5
1.43
1.33
3.45
3.40
13.11
18.30
5/00
333
126
540
52.30
46:58
(rgu
$4.1
53.35
5016
9154
631
4.24
5.5
12215
21.55
1436
3426
11.54
(2:33
MGP
9077
66.44
$1.00
5.5
6.22
1095
8.55
13:58
5.0
319
$1.0
Akdd Magordl (200))
1246
81
$616
$3.0
19.13
31.55
3-401
31.07
23.8
12.50
16.33
3615
11.6
35.5
50:57
1100
41.00
Banci atall (S13))
12:27
11.23
54th
BUR
Banck mial
15.03
$3.56
7637
$1.27
135
1.0
15.55
15.25
25.03
3455
3634
24.6
11.1
66.38
(211))
19:00
9241
96.6
25.00
27/03
3175
15.55
171.5
19775
26.53
5410
Carnet a 2800
Guering
6:11
16.13
506
03.50
6:13
64.6
3.38
1730
553
13.50
1628
1.5
21.1
31.5
3.44
363
6-3
51.4
13.66
66.1
Hls at (3014)
TLa Araraia
11.47
51.9
1325
$138
56.5
1.4
12.00
337
352
1330
1354
13.45
54
27.46
12.31
40:5
44.0
LUB
(3223)
11:00
15m
334
11:3
245
113
A141
(201)
bariige M.T
23:54
(3%
N675
75.
1621
2707
15/0
11.46
23.6
3920
1135
554
514
2410
73465
73.18
TUP Banck mta (361))
MFF
1.00
15:02
1310
$2.66
410
Lm
4.5
152
241
616
128
116.
15.00
7239
3333
3670
528
125
638
224
472
2500 Catiz es. (20)))
28.27
3631
11.5
335
1230
536
1400
ba4ndyp M.P
1334
16.0
1317
11.6
6876
1588
12.96
1110
1134
11 17
833
3136
235
19
7633
200
at ai CX00
fiwe Tay. M.F
38.6
7600
1441
8615
15%
311
2415
9061
3000
18.1
51 15
0-1
51.6
1410
19
2125
71.38
3121
11.5
7618
13.4
21/0
14.22
31.6
1358
1223
4321
5135
3606
59.9
32.00
75.00
2459
Banci cial (2G22)
99.27
1500
3131
1511
Table 18: Full Test-S results on emotion recognition. SoTA: Previous SoTA performance on each respective dataset. Underscore indicates that we have different data splits to the SoTA model. Best model of each dataset is in bold.

---

Detect
Metric
Rosken
HRBET
XIME
Ronée
SADCE
BLOOM BLOOVZ
BL.00M BLOOZ BEOOMI
(48))
Battriaa
1115
(4T)
LLMA
alpen
Vinsss
ChetGP (41)
SoTA Medy
Apganvelie
MIFF
17:
11
80.00
35.0
54:13
9:13
05.03
1926
7141
(005)
50.56
05.66
1448
10.20
$7.66
6835
22.77
3330
1197
23:39
15625
1247
32,71
34.56
34.5
510
063
6833
1353
Chanase aud (23D)
10,00
Table 19: Full Test-S results on humor detection. SoTA: Previous SoTA performance on each respective dataset. Underscore indicates that have different data splits the SoTA model.
Bakwer
Merie Rasker
#RERT
VEMR
Brake TARECT.
DOM BLOOMZ
BLOEME BLOIME BLDOME
(IT)
(MT)
1LMA
Yerms ChaiGe
QuIGPT (MP)
SaTt
SATA
4112
13:91
13:00
$3.53
55.75
01:00
14.93
18.35
000
010
110
0.01
3.31
0101
5.66
5136
5610
Yes
1681
00.0
13.93
71.34
12:33
35.66
43.99
53.55
11.00
13.00
19.33
4353
48:31
1205
2151
57.55
3235
03.00
17.01
Vaa (M01)
boo -kn-
31.11
61.0
11:43
12.01
2417
ARMA Mazri (2IZ))
13.39
8155
0.11
3417
5630
13:0
11:5
Auz
40.30
14.34
76,0
13.55
16.33
11.00
23.45
23.40
11.30
5666
50.6
36.4
25.00
41.6
43.55
37:52
02.33
5635
1310
Ccaalia ca 2200
ter-appa
1691
19.33
00.05
03.85
03.5
15311
13.03
13.01
13.25
01.34
130m
3355
11011
53.
11.00
1839
10:1
Pob CB
tenctou
Hlum
0547
$3.09
10.00
14.03
1/6
14.30
1359
160
1/6
F-36
6.2
$5.9
5135
(200)
tax rxa-
51.34
19.16
15.55
13.7
15.0
18.07
1706
1131
1817
24.0
1an
016
535)
Ace.
40.00
94.30
02.33
56.37
15.57
7750
$1.30
11.01
$1.00
1768
1900
9131
11
17.0
15.0
$3.0
74.0
7166
22.6
Repledaaa (215))
ten-tAss
3491
40.23
11.31
12.5
1335
4136
33.39
33.31
15.30
1808
33.55
13
3625
4.41
41.0
5335
TUE
Ooggeia (2226
Sav seae
1457
3756
11.0
1515
1016
4117
11
1331
5136
71:1
ABSA Magri (3633)
17.6
11TH
3621
22.36
67.35
46.33
5654
33.56
15833
11:55
11:55
11.35
1333
15.0
1933
1133
15.0
1853
1543
3033
3031
5130 You the (200)
Aunage
02.4
03.58
08:11
16.02
12:0
12:52
11.96
4154
01.46
1930
15:4
60.11
Table 20: Full Test-S results on irony & sarcasm detection. SoTA: Previous SoTA performance on each respective
dataset. Underscore indicates that we have different data splits to the SoTA model. Best model each dataset is in bold.

---

Nostc Randees
XIM6 Binker
BLOOM BLOONZ
1973
LEGMA
Appca
Vansa ChaddT
2011
SITA Selu
M--1
04:11
3031
0138
2131
0111
ORET)
00.11
07.1
1163
6246
1331
0116
112
Intarial (h117)
See 01
13.34
0.50
11.6
285
2000
13:0
01
1641
1538
1533
531
(23)
3633
M03
Nodooa (910
bor: alyss
1.4
5164
1636
71:4
5113
Degios Sat0
Sun kera
16:11
24.37
04:35
5553
8331
14.38
3638
16:31
11.0
5165
2-10
58.39
2760
1933
16333
6523
48.5
2173
Moddie 2013
10:11
1671
72.55
1539
3656
7537
0066
(415)
San andw
16:11
33.33
64.3
J8.44
5133
1635
25.18
2633
1363
5340
56.5
202 Noddee 2015
16:1
33.00
0330
08.36
176
1868
14.0
56.55
5645
14311
2408
2633
1481
0184
08:5
Moddie 2015
bor
pous
26.31
5:35
8:37
357
1133
10.1
31.3
5335
1161
16337
11.3 Modiee 2313
San-douu
1631
28.53
0157
0133
915
5634
3636
2345
360
17.5
Nodi 2015
1671
13.6
16.55
135
365
350
2745
3108
2513
1965
5584
12.4
Nodil 23135
Sur Jinn
1631
13.90
1500
160
301
13:0
1639
636
301
3605
33.51
1671
4857
4.14
1131
1700
130
1345
1483
91.77
1531
3241
14:30
3736
1538
216
506
2.4
Pats (303)
San bab
47.00
10.37
02:7
15.5
3030
648
110
1330
71:3
6760
3-60
4033
(740
4 (2110
13.50
13.45
15.8
4134
4135
2638
5715
15m
03:1
19.11
6:17
15:00
1206
1111
11:11
13:33
6.5
2535
800
655
5056
16-11
1730
61:1
5056
100
3.6
31.6
1339
1631
15.55
3500
2540
37.56
06.66
(102)
31.44
654
1440
43.1
05.99
14.5
1571
2.36
(301)
A0C
3640
7832
1150
27-4
50.3
12:77
5125
1579
(2022)
26773
S0B
22.35
36.5
1725
1556
(2025
0333
34.0
11.0
300
3726
19
3861
201
18.99
1125
15.2
333
5385
(122))
2637
48:52
1.9
196
114
5635
1961
1053
9.31
(SD7)
16.0
16:30
14.1
1641
1000
363
6558
(200)
110
605
270
1094
1614
256
11.43
76.
5-38
2111
15:6
1366
2148
11
25
1966
194
1531
1500
3.5
1543
BEI (305)
11.3
10
5641
564
1958
0141
for
1310
11.1
8131
1530
1198
15
1453
21.1
19
25
13
1225
1341
1269
5:60
(8.16
bxik (3055
06.34
08.55
704
7154
260
2010
31:11
125.7
3485
32.01
2155
2504
2362
0024
Table 21: Full Test-S results on sentiment amalysis. SoTA: Previous SoTA performance on each respective dataset. Underscore indicates that we have different data splits to the SoTA model. Best model of each dataset is in bold.
Debaci
Nctric Runakes
SLM-K Boniex bEsDCI BLOOM BLOOADZ
BLDOAE BLOOMEZ
BLOOMZ
x11
LEaMA Alpas Vizza
Quarry
SITA soudy
25.00
55.15
$1.53
95.30
56.50
-516
-9.6
$1.31
14:00
11:0
4.30
53.3
(4:3
53:00
mtal (2002)
34.11
37.76
2.15
1096
12:30
35
77-1
48.53
$8.00
11.34
71,6
72.36
36.14
2614
27.0
2634
48.13
614
47.04
4734
2834
35:55
7351
9AST
14.44
Batticc csu CNI
46:33
3067
53.55
91.00
15.40
3120
33.00
32.00
30.86
47.31
12.90
44.30
53.00
41.00
7540
100
53.35
(VHR) and (312)
Avorgge
4144
72.54
34.45
7430
75:33
44.12
2645
3066
30.33
28,55
33.35
41.4
2.11
4330
30:44
26333
06.55
9633
Table 22: Full test results on subjectivity analysis. SoTA: Previous SoTA performance on each respective dataset. Underscore indicates that we have different data splits to the SoTA model. Best model of each datasct is in bold.

---

Lang Fam.
Random
InfoIC1
BMZ-P3
Vieuma
CG
CG-MT
34.05
73.53
36.78
35.61
33.61
60.81
52.55
amh
37.95
63.68
16.05
22.49
2.99
20.62
46.82
ang
Afro-Asiatic
ary
35,94
53.44
37,40
23.41
16.64
52.19
51.66
hal
35,2
72.18
30.14
20.93
17.39
55.52
34.13
heh 47.60
95.80
71.20
76.60
40.
84.20
57.40
37.11
ibo 32,90
76.75
23.21
11.52
28.37
57.55
33.46
kin 35.8.
$6.69
28.06
14,84
22.25
53.78
29.03
Atlatic-C. C,
swh
33,72
61.29
17.60
14.02
45.55
$4.39
53.84
34.03
52.55
45.54
30.74
6.50
42.58
35.20
41.68
74.88
43,10
35,83
26.38
664.7
42,31
Austroasi.
vie 16.12
64.58
12.22
27.83
9.52
$4..99
32,96
ace
34,78
77.36
37.89
24.73
12.79
52.63
58.05
30.77
84.50
50.51
21.78
14.89
75.43
hug 30.77
71.55
34,60
18.27
12.90
34.63
30.86
fil 52.37
79.01
34.47
34.47
34.47
69.13
66.67
Austrones.
22.85
83.05
42,86
36.77
20.60
75.29
64.
78
mad
28. 64
78.36
45.44
21.85
13:14
61.07
61.
min 34.41
84.07
49.93
32.41
14.95
66.80
62.91
nij 34.86
77.22
42.86
22.89
15.21
55..64
$7.07
37.65
bbe 30,60
73.58
36.42
19.20
13.86
38.43
40.65
Kan 32.14
61.79
39.23
27,80
19:12
144.1
30.39
Dravidian
mal
31.68
82.70
43.84
41.65
24.85
.4..0
31.44
13.60
32,29
59.61
35.15
36.61
53.33
38.68
29.39
47.04
31,78
26.59
16.11
46.82
33.84
bus
34.37
68.31
31.80
21.16
16.72
63.22
48.15
32.48
63.09
27.35
19.37
23.69
59.32
$4.39
30.23
67.83
35.18
27.47
13.67
62.00
58.66
41.06
$0.15
50.91
46.61
$1.00
65.30
63.24
dan 41.14
82.09
46.68
46.68
$1.84
66.91
66.79
eng
37,90
75.48
43.32
39.75
66.51
24.65
66.58
32,20
56.84
den 24.40
67.55
20,00
26.51
29.31
52.24
35.41
ell
41.24
79.13
46.71
45.47
48.21
60.94
34.98
hin 35.24
67.55
28.93
26, 26.20
29.06
$2.63
48.30
Indo-Baro.
70.29
29.60
$6.47
43.80
35.40
34.40
68.60
50,20
pem 33.36
66.65
47.16
24.25
13.92
61.13
42.75
nor 16.24
42.36
19.84
35.73
9.80
42.11
41.41
33.03
por 29.17
66.10
24.45
20.14
19.16
41.10
39.37
pol 30.43
68.04
31,82
31.96
19:15
58.25
49.43
ron 38.16
89.83
48.97
67.14
30.54
89.11
77.34
31.85
84..1
28.33
28:51
09.08
spa 35.98
70.20
32.17
37.71
56.64
34.09
$6.85
27.44
20.55
19.05
55.84
52.54
alk
28.57
75.71
31,89
27.85
10.46
57.46
56.
26.32
58.62
14.37
16.84
16.64
/46.25
36.97
33.35
Japonic
18.00
61,47
44,20
47.60
26.50
55.880
44.40
Korcanie
kor
28.55
57.46
19.48
21.24
16.26
42.90
37.06
Sino- Tib.
zbo
36.77
76.10
48.19
46.94
38.19
63.43
61.39
Kadai
31.26
75.17
20.16
23.94
24.40
52.81
36.32
Turkie
29.56
87.71
32.32
48.07
35.44
64.84
Uralic
hun
27.80
68.37
30.31
27.19
16.39
$3.22
43.89
Table 23: Language-wiss model performance. The best performance in each language is bold, and the second best is green highlight The font denotes performance lower than the random baseline. BMZ: BLOOMZ, CG: ChatGPT, MT: using machine translated prompts.

---

Task
Im-esaluaion-tummnn Prompts
ChaGGP Prompts
BLOOMZ
m/3
m110
1.aMA
Vicuna
BLOOM
BLOOMZ
mT5
mTD
LLaMA
Emotion
9:21
7.75
27.87
15:14
8.51
7.57
29.63
17.51
Humor
41.78
23.04
43.00
33.12
39:75
46.19
41.39
41.9
41.38
34.13
41.59
33.32
Ioory
36.87
11.46
16.52
34.69
00.78
47.48
27.35
26.02
36.08
31.31
26:00
34.70
Aveage
25.21
28.01
19.38
32.12
27.73
31.30
16.92
26.14
30.1
33.21
20.38
3301
Table 24: Study on model sensitivity to prompts used for zero-shot evaluation.


Reviews:
[{"review_id": "1fju500xF4", "reviewer": "Reviewer_fC9A", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful and valuable review. \n\nReasons To Reject:\n\n* We agree that we only define sociopragmatics briefly in the paper. To improve, we will provide a more appropriate grounding of sociopragmatic understanding to the camera-ready. We provide some background here as well. The term “sociopragmatic” can be understood as contextualized meaning in social interactions. An example of such a meaning is one that arises through human interaction, such as on social media platforms. This meaning is highly dependent on its particular context and transcends the literal interpretation of statements, encompassing insights about the individual's use of the language (investigated within the realm of \"sociolinguistics\" [1]) and their distinctive identities, as well as the underlying purposes guiding their communication (explored in \"pragmatics\" [2]) [3].\nWe group different tasks in our benchmark by what we perceive to be an affinity between these tasks. For example, we group tasks of hate speech, offensive language, and dangerous language detection as anti-social language detection. \nMeanwhile, we keep particular tasks (such as sentiment analysis and emotion recognition) distinct due to the popularity of these tasks and since there are multiple datasets representing each of them. We acknowledge that this grouping is not necessarily theoretically motivated, but we feel that it does facilitate modular evaluations through the benchmark. \n\n\n[1] Sali A Tagliamonte. 2015. Making waves: The story of variationist sociolinguistics. John Wiley & Sons.\n\n[2] Jenny A Thomas. 2014. Meaning in interaction: An introduction to pragmatics. Routledge.\n\n[3] Chiyu Zhang, Muhammad Abdul-Mageed, and Ganesh Jawahar. 2023. Contrastive Learning of Sociopragmatic Meaning in Social Media. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2405–2439, Toronto, Canada. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A comprehensive and well-curated benchmark for evaluation LLM's ability of sociopragmatic understanding\n2. The benchmark is beyond English, which is novel given most of the existing benchmarks are in English\n3. The to be release interactive benchmark would be very helpful to the community", "reasons_to_reject": "1. Lacking theoretical grounding of sociopragmatic understanding", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a unified multilingual evaluation benchmark which could be very helpful for evaluating the multilingual abilities of LLMs. The authors also tried to answer a series of important questions through thorough analysis. The authors found that ChatGPT outperforms open-sourcing LLMs on nearly all tasks and instruction tuning also helps the model performance in many cases. \n\nOverall I think this paper represents an important contribution to the evaluation of LLMs and I can see it being very helpful once the interactive leaderboard is released.\n\nOther than all the merits, one major weakness of this paper is lacking theoretical grounding of \"Sociopragmatic Understanding\". It is unclear to me how are these categories selected and why, instead of simply combining all these available datasets. A relevant paper is \"Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark\" (https://arxiv.org/abs/2305.14938).  ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BT4zeiErUP", "reviewer": "Reviewer_B91m", "summary": "", "questions": "A)How does few-shot in context learning perform? \nB) How sensitive is the sparrow score to the choice of prompt? \nC) How does the performance vary with source/domain? For example twitter vs youtube. \n\nPOST REBUTTAL: My concerns are addressed by the authors in detail. I recommend 'accept'.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review of our paper. \n\nReasons To Reject:\n* The focus of our paper is natural language understanding, as our title indicates. For this reason, datasets for natural language generation are outside of the scope of our work. We do not make any claims about them. \n\nQuestions For The Authors:\n\n**A.**  Thanks for your questions. We report a new experiment here to investigate the few-shot in-context learning with 55 datasets from 4 tasks. As this table shows, we compare the results of 3-shot and 5-shot in-context learning with 0-shot results included in the paper. We observe that few-shot in-context learning does enhance the performance of most models (i.e., BLOOM, mT0, LLaMA, and Vicuna). With the increasing number of shots, we can also find that the performances increase. For example, we can see that the Vicuna model obtains average scores over 55 datasets of 31.80, 44.01, and 44.91 with 0, 3, and 5 shots, respectively. However, BLOOMZ-P3 and mT0 do not improve with few-shot in-context learning. We suspect this is because BLOOMZ-P3 and mT0 were finetuned only on NLP datasets, which makes them different from Vicuna that is finetuned with open-ended instructions. These two models (BLOOMZ-P3 and mT0) are also different from BLOOM and LLaMA in that they are finetuned on several NLP tasks only one of which is the sociopragmtic task (i.e., sentiment analysis). We believe this biases the behavior of these two models. We note that we do not perform few-shot in-context learning on sentiment analysis as part of the current experiment because it involves a large number of datasets. \n\nIn the camera-ready, we will complete a few-shot in-context learning on all 169 datasets in SPARROW. \n\n|         | 0-Shot |        |       |       |       |           | 3-Shot |        |       |       |       |           | 5-Shot |        |       |       |       |           |\n|---------|--------|--------|-------|-------|-------|-----------|--------|--------|-------|-------|-------|-----------|--------|--------|-------|-------|-------|-----------|\n|         | BLOOM  | BLOOMZ-P3 | mT5   | mT0   | LLaMA | Vicuna    | BLOOM  | BLOOMZ-P3 | mT5   | mT0   | LLaMA | Vicuna    | BLOOM  | BLOOMZ-P3 | mT5   | mT0   | LLaMA | Vicuna    |\n| Hate    | 39.83  | 38.52  | 23.29 | 37.33 | 37.80 | 41.59     |\\| 38.39  | 37.16  | 39.67 | 37.38 | 43.51 | **49.17** |\\| 37.95  | 37.14  | 39.53 | 37.70 | 41.87 | 48.37     |\n| Emotion | 9.71   | 15.07  | 7.75  | 27.87 | 15.14 | 18.12     |\\| 17.08  | 12.17  | 10.66 | 23.12 | 32.12 | 40.28     |\\| 18.48  | 12.35  | 10.07 | 25.57 | 34.20 | **41.79** |\n| Humor   | 41.78  | 33.04  | 43.60 | 33.12 | 39.78 | 46.19     |\\| 33.67  | 33.12  | 44.70 | 38.19 | 55.20 | 57.15     |\\| 34.06  | 33.12  | 40.20 | 37.08 | 53.86 | **58.75** |\n| Irony   | 36.63  | 44.46  | 36.52 | 34.69 | 40.78 | **47.48** |\\| 42.21  | 41.58  | 42.61 | 35.18 | 36.76 | 39.78     |\\| 44.34  | 44.14  | 39.67 | 34.82 | 38.40 | 41.61     |\n| Average | 25.21  | 28.01  | 19.58 | 32.12 | 27.72 | 31.80     |\\| 28.60  | 25.78  | 26.80 | 30.34 | 37.87 | 44.01     |\\| 29.51  | 26.28  | 25.67 | 31.45 | 38.55 | **44.91** |\n\n**B.**  In our paper, we do not tailor prompts specifically for each model due to the large size of the datasets and models we investigated. Here, we provide a concise study to probe the model’s sensitivity to prompts. We curate 55 datasets across 4 tasks from SPARROW and evaluate 6 models with prompts used for evaluating ChatGPT. As results shown in this table, we can see that BLOOM, LLaMA and Vicuna get sizable performance drops (>6 points decrease across 55 datasets), while BLOOMZ-P3-P3, mT5, and mT0 demonstrated performance levels akin to those observed in previous experiments (<2 points different). We believe that the model performance is not only affected by the choice of prompts but also by which model is used and which dataset is tested on. We hope SPARROW will be a useful benchmark for investigating multilingual socio-pragmatic understanding in future work including opportunities to evaluate different types of prompts. We will add this analysis to the camera-ready. \n\n| Task    | lm-eval-harness | Prompts |       |       |       |        | ChatGPT | Prompts |       |       |       |        |\n|---------|-----------------|---------|-------|-------|-------|--------|---------|---------|-------|-------|-------|--------|\n|         | BLOOM           | BLOOMZ-P3  | mT5   | mT0   | LLaMA | Vicuna | BLOOM   | BLOOMZ-P3  | mT5   | mT0   | LLaMA | Vicuna |\n| Hate    | 39.83           | 38.52   | 23.29 | 37.33 | 37.80 | 41.59  | 18.39   | 31.34   | 28.96 | 38.19 | 18.37 | 37.33  |\n| Emotion | 9.71            | 15.07   | 7.75  | 27.87 | 15.14 | 18.12  | 8.61    | 20.07   | 7.57  | 29.63 | 17.61 | 8.65   |\n| Humor   | 41.78           | 33.04   | 43.60 | 33.12 | 39.78 | 46.19  | 41.59   | 44.99   | 41.34 | 34.13 | 41.59 | 33.12  |\n| Irony   | 36.63           | 44.46   | 36.52 | 34.69 | 40.78 | 47.48  | 27.33   | 26.02   | 36.08 | 34.31 | 26.02 | 34.70  |\n| Average  | 25.21           | 28.01   | 19.58 | 32.12 | 27.72 | 31.80  | 16.92   | 26.14   | 20.91 | 33.21 | 20.95 | 23.03  |\n\n\n**C.**  To investigate the cross-domain performance, we provide a study using binary sentiment analysis datasets that involve 8 different domains. We average the performance of each domain and present the relative gain compared to the random baseline. We can observe that the models’ performance varies across different domains. For example, ChatGPT performs better on Twitter data than YouTube comments. We also would like to mention that a direct comparison across domains is not feasible since each domain contains different tasks, datasets, and languages. While we try to have a fair comparison by using relative gain, we acknowledge the limitation of this experiment.\n\n| Source / Domain           | # Dataset | mBERT | InfoDCL | BLOOM  | BLOOMZ-P3 | mT5    | mT0   | LLaMA  | Vicuna | ChatGPT |\n|---------------------------|-----------|-------|---------|--------|-----------|--------|-------|--------|--------|---------|\n| Customer review           | 1         | 31.73 | 35.90   | -14.52 | 30.84     | -22.07 | 40.87 | 9.75   | -22.07 | 41.15   |\n| Facebook                  | 1         | 45.67 | 48.20   | 17.20  | 23.60     | -15.40 | 29.00 | 19.40  | -6.80  | 36.60   |\n| Moview review             | 3         | 27.70 | 32.52   | 0.42   | 30.46     | -2.12  | 15.58 | 7.69   | 2.69   | 37.52   |\n| Multiple online platforms | 2         | 43.37 | 42.50   | -13.10 | 23.20     | -15.31 | 16.90 | 0.91   | -8.39  | 33.15   |\n| Subtitle                  | 3         | 29.69 | 36.17   | -5.61  | 9.96      | -13.53 | 11.38 | -0.62  | 14.57  | 36.77   |\n| Twitter                   | 3         | 26.85 | 39.90   | -3.00  | 24.34     | -7.65  | 9.49  | 3.99   | 13.56  | 38.98   |\n| Weibo                     | 1         | 52.14 | 52.28   | -0.17  | 23.99     | -13.15 | 9.85  | 7.23   | 19.04  | 31.17   |\n| YouTube comment           | 4         | 31.20 | 35.25   | -18.16 | 20.05     | -26.00 | 10.31 | -18.77 | -23.56 | 24.54   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The proposed benchmark is huge and one of the most comprehensive ones.\n2) The benchmark consists of 64 languages which inculudes many low-resource languages.\n3) The paper evaluates and compares the perfromance of 14 LLMs on the benchmark. \n4) Detailed analysis of multiple models on the datasets across mutiple dimensions. For example overall perfromance of models, effect of instruction tuining, perfromance across tasks, languages etc.\n5) The paper shows that for some low resource languages, translating the prompt to english and then making the prediction can give better results.", "reasons_to_reject": "1) Finetuned LLMs are compared with zero-shot LLMs but evaluation of few-shot In-context learning is missing. \n2) All the datasets included are text classification tasks. Datasets on language generation are not included. \n\nPOST REBUTTAL: My concerns are addressed by the authors in detail. I recommend 'accept'.", "questions_to_authors": "A)How does few-shot in context learning perform? \nB) How sensitive is the sparrow score to the choice of prompt? \nC) How does the performance vary with source/domain? For example twitter vs youtube. \n\nPOST REBUTTAL: My concerns are addressed by the authors in detail. I recommend 'accept'.", "paper_topic_and_main_contributions": "The paper introduces a new benchmark called Sparrow, for multilingual sociopragmatic meaning understanding. The dataset covers 64 languages and 169 datasets across 13 types of tasks. The paper also introduces a SPARROW score based on the benchmark and maintains and public leaderboard.  Finetuned and zero-shot LLMs are evaluated on this benchmark, and it is shown that zero-shot chatgpt lags behind task specific finetuned models by 12 points. Further, the paper analyzes multiple models across dimensions like overall performance, language, task-type etc.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Footnote 1 is unnecessary"}, "extra_text": {}}, {"review_id": "iRquT15jc7", "reviewer": "Reviewer_GKU9", "summary": "", "questions": "A. Can there be any data leakage between different datasets? E.g. test set for some dataset and task contains very similar examples to the train set of another dataset. This is especially important for the finetuned models\n\nB. InfoDCL uses contrastive learning, it was somehow expected to have better performance compared to a transformer used as an encoder without contrastive learning or additional losses. I feel this should be highlighted when presenting the models.\n\nC. The discussions and results would have been more relevant if some of the larger models have been finetuned (not all of them, just pick the top 1-3) on some tasks or at least tested with few shots in context learning. Any comments for not including at least some few-shot results?\n\nD. For a benchmark containing so many different datasets for different languages and tasks, how do you feel about taking dataset quality into consideration? This seems especially important when computing aggregate scores in a benchmark. Maybe the benchmark should be divided into datasets that are somehow validated to have a good quality and other datasets where data quality is not clear yet. \n\nE. \"While we customize prompts employed for each task, we do not tailor prompts specifically for each model.\" -> Is the ChatGPT prompt providing better results for the other models? Have the authors tested this?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable review and questions about our paper.  \n\nReasons To Reject:\n* While it is true that our paper does not introduce new original datasets, the power of our new benchmark comes from the scale and the massively multilingual nature. SPARROW is based on 169 datasets covering 64 languages. We believe that, perceived as one massive resource whose whole is much larger than its individual components, SPARROW is novel. We also believe that it has the potential to accelerate research on multilingual socio-pragmatics and enable meaningful comparisons in the community.\n\nQuestions For The Authors:\n\n**A.** Thanks for the question. While we did not verify if the test samples of one dataset overlapped with the training samples of other datasets, we do not think that there is any data leakage in our experiments. For finetuned encoder-only baselines, we finetune and test each model on each individual dataset separately. We do not finetune and test across different datasets. The training and testing sets of each dataset are exclusive. We will emphasize these details in the camera-ready version. \n\n**B.** We use InfoDCL as the strongest encoder-only LM from previous studies. It serves as a SoTA encoder-only LM to compare with other LLMs. Due to the page limit, we provided detailed information on all models in Appendix B. We will clarify the model detail in the main paper in the camera-ready version.\n\n**C.** Due to the expensive cost of finetuning these large models, we provide a case study on few-shot in-context learning (but not finetuning): \nWe report a new experiment here to investigate the few-shot in-context learning with 55 datasets from 4 tasks. As this table shows, we compare the results of 3-shot and 5-shot in-context learning with 0-shot results included in the paper. We observe that few-shot in-context learning does enhance the performance of most models (i.e., BLOOM, mT0, LLaMA, and Vicuna). With the increasing number of shots, we can also find that the performances increase. For example, we can see that the Vicuna model obtains average scores over 55 datasets of 31.80, 44.01, and 44.91 with 0, 3, and 5 shots, respectively. However, BLOOMZ-P3 and mT0 do not improve with few-shot in-context learning. We suspect this is because BLOOMZ-P3 and mT0 were finetuned only on NLP datasets, which makes them different from Vicuna that is finetuned with open-ended instructions. These two models (BLOOMZ-P3 and mT0) are also different from BLOOM and LLaMA in that they are finetuned on several NLP tasks only one of which is the sociopragmtic task (i.e., sentiment analysis). We believe this biases the behavior of these two models. We note that we do not perform few-shot in-context learning on sentiment analysis as part of the current experiment because it involves a large number of datasets. \n\n\nIn the camera-ready, we will complete a few-shot in-context learning on all 169 datasets in SPARROW. \n\n|         | 0-Shot |        |       |       |       |           | 3-Shot |        |       |       |       |           | 5-Shot |        |       |       |       |           |\n|---------|--------|--------|-------|-------|-------|-----------|--------|--------|-------|-------|-------|-----------|--------|--------|-------|-------|-------|-----------|\n|         | BLOOM  | BLOOMZ-P3 | mT5   | mT0   | LLaMA | Vicuna    | BLOOM  | BLOOMZ-P3 | mT5   | mT0   | LLaMA | Vicuna    | BLOOM  | BLOOMZ-P3 | mT5   | mT0   | LLaMA | Vicuna    |\n| Hate    | 39.83  | 38.52  | 23.29 | 37.33 | 37.80 | 41.59     |\\| 38.39  | 37.16  | 39.67 | 37.38 | 43.51 | **49.17** |\\| 37.95  | 37.14  | 39.53 | 37.70 | 41.87 | 48.37     |\n| Emotion | 9.71   | 15.07  | 7.75  | 27.87 | 15.14 | 18.12     |\\| 17.08  | 12.17  | 10.66 | 23.12 | 32.12 | 40.28     |\\| 18.48  | 12.35  | 10.07 | 25.57 | 34.20 | **41.79** |\n| Humor   | 41.78  | 33.04  | 43.60 | 33.12 | 39.78 | 46.19     |\\| 33.67  | 33.12  | 44.70 | 38.19 | 55.20 | 57.15     |\\| 34.06  | 33.12  | 40.20 | 37.08 | 53.86 | **58.75** |\n| Irony   | 36.63  | 44.46  | 36.52 | 34.69 | 40.78 | **47.48** |\\| 42.21  | 41.58  | 42.61 | 35.18 | 36.76 | 39.78     |\\| 44.34  | 44.14  | 39.67 | 34.82 | 38.40 | 41.61     |\n| Average | 25.21  | 28.01  | 19.58 | 32.12 | 27.72 | 31.80     |\\| 28.60  | 25.78  | 26.80 | 30.34 | 37.87 | 44.01     |\\| 29.51  | 26.28  | 25.67 | 31.45 | 38.55 | **44.91** |\n\n\n**D.** Since there are large numbers of datasets (169 datasets) and languages (64 languages) in SPARROW, it is really hard to manually verify the quality of all the datasets. To ensure the quality of the dataset, we only select the datasets that are introduced in published papers (all the original papers are cited in Table 8-13), when we construct our benchmark. In our references, we also make sure that we link to each published paper describing each dataset to facilitate access to the annotation information of each dataset. \n\n\n**E.** We provide a concise study to evaluate open-source LLMs with prompts used for evaluating ChatGPT. We curate 55 datasets across 4 tasks from SPARROW and evaluate 6 models with ChatGPT prompts. As results shown in this table, we can see that BLOOM, LLaMA and Vicuna get sizable performance drops (>6 points decrease across 55 datasets), while BLOOMZ-P3, mT5, and mT0 demonstrated performance levels akin to those observed in previous experiments (<2 points different). We believe that the model performance is not only affected by the choice of prompts but also by which model is used and which dataset is tested on. We hope SPARROW will be a useful benchmark for investigating multilingual socio-pragmatic understanding in future work including opportunities to evaluate different types of prompts. We will add this analysis to the camera-ready.\n\n| Task    | lm-eval-harness | Prompts |       |       |       |        | ChatGPT | Prompts |       |       |       |        |\n|---------|-----------------|---------|-------|-------|-------|--------|---------|---------|-------|-------|-------|--------|\n|         | BLOOM           | BLOOMZ-P3  | mT5   | mT0   | LLaMA | Vicuna | BLOOM   | BLOOMZ-P3  | mT5   | mT0   | LLaMA | Vicuna |\n| Hate    | 39.83           | 38.52   | 23.29 | 37.33 | 37.80 | 41.59  | 18.39   | 31.34   | 28.96 | 38.19 | 18.37 | 37.33  |\n| Emotion | 9.71            | 15.07   | 7.75  | 27.87 | 15.14 | 18.12  | 8.61    | 20.07   | 7.57  | 29.63 | 17.61 | 8.65   |\n| Humor   | 41.78           | 33.04   | 43.60 | 33.12 | 39.78 | 46.19  | 41.59   | 44.99   | 41.34 | 34.13 | 41.59 | 33.12  |\n| Irony   | 36.63           | 44.46   | 36.52 | 34.69 | 40.78 | 47.48  | 27.33   | 26.02   | 36.08 | 34.31 | 26.02 | 34.70  |\n| Average  | 25.21           | 28.01   | 19.58 | 32.12 | 27.72 | 31.80  | 16.92   | 26.14   | 20.91 | 33.21 | 20.95 | 23.03  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The SPARROW benchmark, in 64 languages, covering for NLP tasks related to sociopragmatic understanding, that will be useful for assessing and comparing the performance of LLMs and maybe other models.\n- In depth discussion and experiments using various finetuned smaller transformer-based models and zero-shot LLMs (with over 1B parameters) on the benchmark\n- There seems lots of work in building the benchmark and performing all the experiments - the results and discussions are useful for the NLP community", "reasons_to_reject": "- The paper does not introduce any original datasets or methods\n- The discussions and results would have been more relevant if some of the larger models have been finetuned (not all of them, just pick the top 1-3) on some tasks or at least tested with few shots in context learning", "questions_to_authors": "A. Can there be any data leakage between different datasets? E.g. test set for some dataset and task contains very similar examples to the train set of another dataset. This is especially important for the finetuned models\n\nB. InfoDCL uses contrastive learning, it was somehow expected to have better performance compared to a transformer used as an encoder without contrastive learning or additional losses. I feel this should be highlighted when presenting the models.\n\nC. The discussions and results would have been more relevant if some of the larger models have been finetuned (not all of them, just pick the top 1-3) on some tasks or at least tested with few shots in context learning. Any comments for not including at least some few-shot results?\n\nD. For a benchmark containing so many different datasets for different languages and tasks, how do you feel about taking dataset quality into consideration? This seems especially important when computing aggregate scores in a benchmark. Maybe the benchmark should be divided into datasets that are somehow validated to have a good quality and other datasets where data quality is not clear yet. \n\nE. \"While we customize prompts employed for each task, we do not tailor prompts specifically for each model.\" -> Is the ChatGPT prompt providing better results for the other models? Have the authors tested this?", "paper_topic_and_main_contributions": "The paper presents a large multilingual benchmark for NLP tasks related to sociopragmatic understanding, covering topics like antisocial behavior, irony and sarcasm, emotion detection, humor detection, sentiment analysis, and subjectivity prediction. The benchmark should be useful for assessing and comparing the performance of LLMs and maybe other models at some point for tasks that require subjectivity, commonsense, and pragmatics. \n\nAt the same time, the authors offer an in-depth analysis and discussion of various finetuned smaller transformer-based models and zero-shot LLMs (with over 1B parameters) on the benchmark, and also have a good discussion related to each task, to each language, and more.\n\nWhile all these datasets are public, and the paper does not present any original models, there is an important contribution in the amount of work to gather the benchmark and to make it available to the public, together with the initial set of experiments.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents SPARROW, a multilingual benchmark dataset for evaluating cross-lingual sociopragmatic understanding.  SPARROW consists of 169 datasets for 64 different languages across 13 task types divided into six umbrella categories of sociopragmatic meaning.  Using a proposed SPARROW score, an unweighted average across dataset-specific metrics, the paper evaluates a number of LLMs in different settings (zero-shot for generative models, finetuning for encoder-only models) on their ability to perform sociopragmatic understanding tasks.  They found that overall, smaller, fully finetuned models surpassed larger generative models in the zero-shot setting.

Overall, the paper was well-received by reviewers, who all appreciated the authors’ work in curating a comprehensive benchmark for sociopragmatic tasks across multiple languages and releasing it to the public.  In particular, reviewer fC9A highlights how useful the proposed benchmark would be to the broader NLP community in evaluating the capabilities of LLMs.  Reviewers B91m and GKU9 also appreciated the detailed set of experiments and analyses across various dimensions, such as the impact of finetuning and instruction tuning, performance across tasks and languages, and the impact of using translated prompts and inputs.  

The major concern highlighted by reviewers B91m and GKU9, however, is that for the larger models, the submitted paper only presents results in a zero-shot setting without exploring few-shot in-context learning.  Additionally, both reviewers raised questions about how robust the results of their analyses were to the choice of prompt, as in the original paper, prompts were tailored to each task rather than to each model tested.  However, both of these concerns were addressed during the rebuttal/discussion period, with the authors presenting new results with few-shot ICL and prompt variation.  If accepted, the paper would be strengthened with the inclusion of those new results in the camera-ready version.  One potential concern, however, is how substantially the conclusions about overall performance over different models may change with the full set of results, as so far, the authors have not presented ICL results for their best generative model (ChatGPT) or one of the strongest tasks for generative LLMs.

The other concern raised by reviewer fC9A about the paper is the lack of theoretical grounding in how tasks were chosen and categorized for the benchmark.  The authors clarify their use of the term “sociopragmatic” and the motivation behind their grouping of selected tasks in their rebuttal.  While the rebuttal does not fully address the issue of the lack of theoretical motivation in how tasks were selected and grouped, the motivation behind the paper would nevertheless be strengthened by the additional discussion about the definition of sociopragmatics and the interaction between language, identity, and interaction in the rebuttal.
Decision: Accept-Main