Massively Multilingual Corpus of Sentiment Datasets and Multi-faceted Sentiment Classification Benchmark
Authors: Lukasz Augustyniak
Szymon Woźniak
Marcin Gruza
Piotr Gramacki
Krzysztof Rajda
Mikołaj Morzy
Tomasz Jan Kajdanowicz
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: sentiment analysis
multilingual
dataset
benchmark
nlp
TL;DR: The largest corpus of multi-lingual datasets for sentiment classification and a multi-faceted benchmark of models (pre-trained vs fine-tuned, multi-lingual vs single-language) for sentiment classification.

Abstract: Despite impressive advancements in multilingual corpora collection and model training, developing large-scale deployments of multilingual models still presents a significant challenge. This is particularly true for language tasks that are culture-dependent. One such example is the area of multilingual sentiment analysis, where affective markers can be subtle and deeply ensconced in culture.

This work presents the most extensive open massively multilingual corpus of datasets for training sentiment models. The corpus consists of 79 manually selected datasets from over 350 datasets reported in the scientific literature based on strict quality criteria. The corpus covers 27 languages representing 6 language families. Datasets can be queried using several linguistic and functional features. In addition, we present a multi-faceted sentiment classification benchmark summarizing hundreds of experiments conducted on different base models, training objectives, dataset collections, and fine-tuning strategies.

Massively Multilingual Corpus of Sentiment Datasets and Multi-faceted Sentiment Classification Benchmark
Lukasz Augustyniak
Szymon Wožniak
WUST (Wroctaw University of Science and Technology) lukasz, augustyniakkppr edu. pl
Brand24 AI
Marein Gruza Brand24 AI, WUST
Piotr Gramacki Brand24 AI, WUST
Krzysztof Rajda Brand24 AI, WUST
Mikolaj Morzy
Tomasz Kajdanowiez WUST
Pozmań University of Technology
Abstract
Despite impressive advancements in multilingual corpora collection and model training, developing large-scale deployments of multilingual models still presents significant challenge. This is particularly true for language tasks that culture- dependent. One such example is the area of multilingual sentiment analysis, where affective markers be subtle and deeply ensconced in culture. This work presents the most extensive open massively multilingual corpus of datasets for training sentiment models. The corpus consists of 79 manually selected datasets from over 350 datasets reported in the scientific literature based on strict quality criteria. The corpus covers 27 languages representing language families. Datasets can be queried using several linguistic and functional features. In addition, we present multi-faceted sentiment classification benchmark summarizing hundreds of experiments conducted on different base models, training objectives, dataset collections, and fine-tuning strategies.
Introduction
Consider hotel booking service that allows its customers to post reviews. You have found just the perfect accommodation to stay for couple of days with your family, but you browse through the reviews section of the website check the experiences of former guests. Suddenly, you encounter review in Polish: "hotel jak hotel, moglo byé gorzej." This review has the following sentiment scores 8neg 0.44, 8neu = 0.44, 8pns 0.12. Intrigued by the ambiguity scores, you translate the review into English: "hotel like hotel, all in all, it could have been worse," which scored as 8neg = 0.80, 8neu 0.16, 8pos = 0.04. Apparently, the stereotypically pessimistic Polish outlook life gets lost in translation. The next review is written in Czech: "ok, ale nic zajimavého" with scores negg = 0.32, Sneu = 0.54, Spos = 0.14. After translating into English ("ok, but nothing interesting") the sentiment classification model scores the review negative (Sneg = 0.500, Snen = 0.37, 5 pos = 0.13). After your stay, you decide to add very positive review of the hotel ("it was a killer place stay", Sneg 0.03, &new 0.05, 8pos 0.92). You would be very surprised to learn that the Czechs would be quite confused about your opinion ("ro bylo vražedné
Sentiment scores in this paragraph cardiffnlp/twitter--e xiaeer--lm--obbbbbe base-aseeeeeeeee sentimment model
produced
by
the multilingual
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

místo & pobytu", Sneg = 0,51, Snew 0.09, spos = 0.39), while the Poles would stay away from the hotel at all costs ("to byto zabójeze miejsce na pobyt". Sneg 0.78, Suen 0.09, Spos 0.13).
multilingual services become ubiquitous the modern global economy. As more websites begin to offer automatic translation of content, users do not bother to express themselves in the lingua franca of the Web, writing instead in their native languages. Despite impressive advancements in automatic translation, many NLP tasks are still difficult in the multilingual setting. And sentiment classification is among the most challenging. The expression of sentiment is highly culture-dependent [31] The emotional valence of individual words, the presence of specific phrasemes, and the expectations around sentiment values make sentiment classification across languages demanding task.
Models performing sentiment classifications have to cope with two independent sources of variance in the input data: cultural expressions of sentiment and errors in automatic translations. In addition, the productization of sentiment classification leads to several engineering choices which influence the efficiency of the model:
single multilingual model vs. dedicated monolingual models: deploying a single model in a production environment is much easier than orchestrating ensemble of models,
training vs. fine-tuning: sentiment classification model can be trained from scratch, both in the multi- and monolingual regimes, and the alternative is the fine-tuning of general- purpose pre-trained language model with sentiment data,
transfer learning between domains: a sentiment classification model trained for a specific domain (e.g., book reviews) be transferred to another domain (e.g., hotel reviews) under the assumption that sentiment expressions in a given language remain independent of the subject of sentiment,
transfer learning between languages: may hypothesize that related languages can utilize similar sentiment expression mechanisms regarding grammar, punctuation, and vocabulary, In theory, it is possible to use training data in language L1, fine-tune a multilingual model with this data, and as a result of fine-tuning, improve the performance of the model on language L2, provided languages L1 and L2 are sufficiently similar.
Working with multilingual datasets and models opens another opportumity. Most NLP research focuses on 20 major languages. Many languages native to significant human populations are not adequately studied. Although the term low-resource language is not precisely defined and can be understood in terms of computerization, privilege, the abundance of resources, or density [47], the existence of large chasm between languages with respect to linguistic resources is apparent. Our work aims at supporting low-resource languages performant sentiment classification. Finally, there is a lively debate in the scientific community about the inherent ability of neural models handle general linguistic phenomena. This paper tries answer this question the domain of sentiment classification.
Contribution presented in this paper threefold:
multilingual corpus: present the largest multilingual collection of sentiment classification datasets consisting of 79 high-quality datasets covering 27 languages. The collection of the corpus and detailed descriptions of collected datasets are presented in Section
multi- fucti-faceted benchmark: the dataset is supplemented with a benchmark containing detailed run statistics of hundreds experiments representing different training and testing scenarios.
All details relevant to the benchmark are presented in Section 4
library for dataset access: all datasets in the multilingual corpus are publicly available via library compatible with the HuggingFace library, along with the ability filter and select datasets, verify their licenses, etc
2 Linguistic typology
The similarity of languages and the main aspects of their differences is the field of study of language typology. The differences between languages can be phonological (differences in sounds used

---

by languages). syntactic (differences in language structures), lexical (differences in vocabulary). and theoretical (differences characterized as general properties of languages). Linguistic typology analyzes the current state of languages and is often contrasted with genealogical linguistics. The latter is concerned with historical relationships between languages established via historical records or with the help of comparative linguistics. The main focus of genealogical linguistics resolves around language families and language genera. The term language family refers to a group of languages sharing pedigree from a common ancestral language (the proto-language). As of today, linguists define over 7000 languages categorized into 150 families [1]. The largest families of Sino-Tibetan Turkic, Afro-Asiatic,
Niger-Congo and Eskimo-Aleut [24]. Main families are further divided into branches called genera. Examples of genera within the Indo Indo-European family of languages include Slavic, Romance. Germanic, and Indic This division of language families into genera closely follows the genetic family of humankind as attested by DNA similarity (43]. Some language families do not produce distinctive genera but form dialect continua defined by mutual intelligibility. Finally, an important concept is that of a sprachbund, geographical area occupied by languages sharing linguistic features. The linguistic similarities within a sprachbund result from cultural exchange and geographical contact rather than by chance or common origin, as described by Thomason and Kaufman (85]
languages include Indo-Europent
Languages can be described using hundreds of linguistic features. World Atlas of Language Struc- tures (27] lists almost 200 different features. Since our work focuses sentiment classification, select 10 features that seem to be the most relevant to the task of sentiment expression. These features are:
1. definite article: the morpheme associated with nouns and used to code the uniqueness or definiteness concept; almost half of the languages do not use the definite article.
2. indefinite article: the morpheme used together with nouns to signal that the related concept is unknown to the hearer; half of the languages do not use the indefinite article, some languages use separate article, and some the numeral "one" as the indefinite article.
3. number of cases: morphological cases are a common way to express various relationships between words; human languages vary greatly in the number of cases used by a language.
4. order of subject, verb, and object: some languages have the strict ordering of words, while languages that convey semantics through inflection may be much looser with the ordering; half of all languages use the SOV ssubject-ojeect-verb) ordering, one third uses the SVO ssubect-verb-oojeee
ordering, and a small fraction of languages use the remaining VSO, VOS, OVS, and OSV orderings. Interestingly, around 13% of world languages do not have any fixed word order.
5. negative morphemes:
negative morpheme is used to signal clausal negation in declarative sentences; this is usually achieved using negative affix a negative particle.
6. polar questions: a polar question is a question with only yes/no answers: these questions can be built using question particles, interrogative morphology. or intonation only.
7. position of the negative morpheme: languages differ by the position of the negative mor- pheme in relation to subjects and objects, with many variants such as SNeg VO, NegSVO, SVNegO, obligatory and optional double negations, etc.
8. prefixing vs. suffixing: languages differ significantly in their use of prefixes versus suffixes in inflectional morphology.
9. coding of nominal plurals: two major types of plural coding are present in languages, either by changing the morphological form of the noun or by using a plurality indicator morpheme somewhere in the noun phrase.
10. grammatical genders: there is significant variability among languages with respect to the number of grammatical genders, some languages do not use the concept at all, and some languages may have 5 or more grammatical genders.
All language features mentioned above are available as filtering features in our library. Thus, when training a sentiment classifier using our dataset, one may download different facets of the collection. For instance, one can download all datasets in Slavic languages in which polar questions are formed using the interrogative word order (Listing T or download all datasets from the Asiatiiiatic language family with no morphological case-making (Listing
3

---

3 Datasets
3.1
Quality criteria
The initial pool of sentiment datasets has been gathered using extensive search and consisted of 345 datasets found through Google Scholar, GitHub repositories, and the HuggingFace datasets library. This initial pool of datasets has been manually filtered based on the following set of quality assurance criteria:
1. strong annotations: we have rejected datasets containing weak annotations (e.g., datasets with labels based on emoji occurrence or generated automatically through classification by machine learning models) due to an extensive amount of noise [58]
2. well-defined annotation protocol: we have rejected datasets without sufficient information about the annotation protocol (e.g., whether annotation manual automatic, number of annotators) to avoid merging datasets with contradicting annotation instructions.
3. numerical ratings: we have accepted datasets with numerical ratings, mapping Likert-type 5-point scales into three class sentiment labels as follows: ratings and 2 were mapped to negative, rating was mapped to neutral, and ratings 4 and were mapped to positive.
4. three classes only: have rejected datasets annotated with binary sentiment labels their performance in three class settings was unsatisfactory.
5. monolingual datasets: when dataset contained samples in multiple languages, we opted to divide it into independent datasets in constituent languages.
3.2 Pre- Pre-processing of datasets
the datasets still contained conflicting entries, i.c., duplicated records with different sentiment labels. We have cross-refcrenced all datasets identify conflicts and have made the data coherent using majority-labos voting. Finally, labeling and rating schemes of all datasets have been mapped to 3-class scheme with negative, neutral, and positive labels only. For datasets with emotional annotations, we mapped positive emotions (joy, happiness) into positive sentiment and negative emotions (fear, sadness, disgust, anger) into negative sentiment. Texts with ambiguous emotions like anticipation and surprise were discarded. This pre- processing pipeline resulted in 79 datasets containing 6 164 762 text samples. Most of the datasets are English (2 330 486 samples across 17 datasets), Arabic (932 075 samples across 9 datasets), and Spanish (418 712 samples across 5 datasets). The datasets represent four different domains: social media (44 datasets), reviews (24 datasets), news (5 datasets), and others (6 datasets). In addition, all datasets were processed by the cleanlab library to produce self-confidence label-quality score for each data point.
Despite quality assurance criteria described in Section 3.1
Exhaustive testing of all configurations of the benchmark is unfeasible, but the total lack of baseline is unacceptable, either. We have decided to test the benchmark dataset by manually compiling a strong baseline. The main rationale behind this effort was the lack of coherence in annotation guidelines between considered datasets. Our baseline is built using strict annotation guidelines constructed iteratively over annotation batches, resulting in highly coherent annotations. The baseline dataset consists of 3 911 short text samples (trimmed to 350 characters) in Polish and English, annotated independently by 3 annotators fluent in these languages. Bascline texts cover multiple domains, such as social media, news sites, blogs, and Internet forums. For each instance, the majority label assigned by the annotators has been stored. The inter-rater agreement is 0.665 (average Cohen's 's kappa between three pairs of annotators) and a = 0.666 (Krippendorff's alpha).
One may think of the baseline dataset representative of current sentiment model training, where researchers have to build domain-alignes corpora of various languages and annotate them. The comparison of results between the benchmark and the baseline datasets is proxy of the performance trade-offs should one decide to use our benchmark dataset to train domain and language-specific sentiment classifiers.

---

Table 1: Summary of the corpus. Categories: N news, R reviews, SM
social media, other
#datasets
category
#samples NEU
mean
SM
0
NEG
POS
#words
#chars
English
17
6
4
304,939
290,823
1,734,724
62
339
Arabic
4
138.899
192,774
600,402
52
289
Spanish
2
0
108,733
122,493
187,486
26
150
Chinese
0
117,967
69.016
144,719
80
German
5
0
104,667
100,071
111,149
26
171
Polish
2
77,422
62,074
97,192
123
French
3
2
0
84,187
43,245
83,199
28
159
Japanese
0
83,982
41,979
83,819
101
Czech
4
2
0
39,674
59,200
97,413
34
212
Portuguese
4
56,827
55,165
45,842
63
Slovenian
0
33,694
50,553
29,296
41
269
Russian
31,770
48,106
31,054
11
70
Croatian
0
19,757
19,470
38.367
17
116
Serbian
25,089
32,283
18,996
269
Thai
9,326
28,616
34,377
22
381
Bulgarian
13,930
28,657
19,563
12
86
Hungarian
000
8,974
17,621
30.087
83
Slovak
14,431
12,842
29,350
13
98
Albanian
6,889
14,757
22,638
13
91
Swedish
1100
16,266
13,342
11,738
14
94
Bosnian
11,974
11,145
13.064
12
76
Urdu
5,239
8,585
5,836
13
69
Hindi
0
4,992
6,392
5,615
128
Persian
0
1,602
5,091
6,832
21
104
Italian
2
0
4.043
4,193
3,829
16
103
Hebrew
0
2,279
243
6,097
22
110
Latvian
0
1,378
2,618
1,794
20
138
Multi-faceted benchmark
As we have explained in Section 1 the deployment of multilingual sentiment classifier can be evaluated using several criteria leading different architecture choices. Thus, we do pub- lish a single benchmark, but we aggregate the results along several dimensions. The benchmark available at htttps://uuggiggee http:aaaaccccpccceeeeeeeeeeaa models according to:
co/ Sraaces/rrand24/mma (atraadD2//mm__bencaaaa333 benchmark allows compare
number of languages: multilingual models with monolingual models,
training procedure: models trained from scratch vs. fine-tuning,
domain language: language to which model is applied,
data modality: news, reviews, social media,
knowledge transfer: transfer between languages, transfer between domains.
Table 2 presents the list of models included in the benchmark. For each model, we include the number parameters and languages used pre-training and the base model. The results presented in the benchmark reflect three possible scenarios of model deployment. A pre-trained model can be used generate text representation only. In this scenario, denoted head-linear (HL), model serves as feature extractor followed by a small linear classification head. In the second scenario, linear classification head replaced by a BiLSTM classifier operating on features extracted by the pre-trained model. We refer to this scenario as head-bilstm (HB). Finally, each pre-trained transformer-based model (with the exception of (USF-srranforrme)) been fine-tuned to sentiment classification task. We refer this scenario as fine-tuning (FT).
5

---

Table Models included in the benchmark
Model
#params
#langs
base
reference
mT5
277M
101
T5
Xue al. 9
LASER
52M
93
BiLSTM
Artetxe and Schwenk
mBERT
177M
104
BERT
Devlin al.
26
MPNet
278M
53
XLM-R
Reimers and Gurevych
64
XLM-R-din
278M
53
XLM-R
Reimers and Gurevych
64
XLM-R
278M
100
XLM-R
Conneau et al. 22
LaBSE
470M
109
BERT
Feng al. 30
DiRTilmBERT
134M
104
BERT
Sanh al. 73
mUSE-dist
134M
53
DistilmBERT
Reimers and Gurevych
64
mUSB-transformer
85M
16
transformer
Yang et al.
995
mUSE-con
68M
16
CNN
Yang et al. 95
Fine-ture Fine-tuning F1 score per langwage
XLM-R R
61
61
60
66
04
64
72
67
70
68
00
34
>B
66
55
50
85
01
62
62
45
61
55
30
52
63
61
5B
05
L88SF
60
82
69
67
65
61
71
67
70
64
63
42
59
66
5B
50
63
61
64
63
47
59
48
65
62
57
61
MPNct
60
70
05
67
52
63
43
80
61
55
49
61
50
65
01
G3
5B
XUM R-dist
59
70
67
83
63
45
50
0Z
50
02
02
49
01
56
58
WBBRT
63
mUSF-dii
5
B
2 *
8
Figure 1: Detailed comparison of models. Legend: lang averaged by all languages, ds averaged by dataset, ar Arabic, bg Bulgarian, bs Bosnian, cs Czech, de German, en English, es Spanish, fa Persian, fr French, he Hebrew, hi Hindi, hr Croatian, hu Hungarian, it Italian, ja Japanese, ly Latvian, pl Polish, pt Portuguese, ru Russian, sk Slovak, sl Slovenian, sq Albanian, sr Serbian, sv Swedish. th Thai, ur Urdu, zh Chinese.
The hyperparameters models included the benchmark are as follows. The hidden size of the model was set to the embedding size of each model when used in HL and FT scenarios. The hidden size of the HB scenario was set
32. The learning rate for the HI scenario was f/ = 10-3 The batch size in HL and HB scenarios was = 200 and b = 6 for fine-tuning. Dropout for HB was d = 0.5 and
fine-tuning used 77 1 10 and HB scenario used 7) 5 x 10
0.2 for other scenarios. Training took 5 epochs for fine-tuning and 2 epochs for HL and HB scenarios (beyond 2 epochs most models started to overfit). The models are evaluated using the traditional F1 score computed three levels: the entire dataset, averaged over all datasets, and the internal dataset.
We performed our experiments using Python 3.9 and PyTorch (1.8.1) (and Tensorflow (2.3.0) for original mUSE). Our experimental setup consists of Intel(R) Xeon(R) CPU ES-2630 v4 @ 2.20GHz and Nvidia Tesla V100 16GB.
Here we present one result available through the benchmarks. Due to space constraints, we move the presentation of other case studies to Appendix Figure Figure]T contains Fi scores of all models fine-tuned on all datasets available for a given language. Interestingly, we see little variance in performance between models for high-resource languages and significant deterioration of performance for low- resource languages. An outlier is the aforementioned Portuguese, where the performance is caused by the lack of data points representing news and reviews. Figure [ directly shows the usefulness of our benchmark. When designing sentiment classification solutions in Spanish, the choice of model secondary (models' performances are very similar), and other model features can be considered (such ease of deployment, cost of inference, and memory requirements). However, if sentiment classification is to be applied Latvian, the performance difference between models can be high as 32 pp depending the choice of the model.

---

5 Related work
5.1 Multilingual text representations
One of the foundational discoveries in multilingual presentation learning was the fact that latent vector spaces seemed to encode very similar word relationships across a wide spectrum of languages. Needless to say, first monolingual static word embeddings were quickly followed by multilingual word embeddings which provided the basis for multilingual text representations [68]. A popular solution was to use pre-trained monolingual embeddings, such as word2vec [49] and align them via
linear transformations using parallel multilingual dictionaries [50]. Another approach advocated for joint learning of multilingual word embeddings on useudo-bilinguan datasets, where tokens in one
language would be randomly translated to another language [92].
Static multilingual embeddings were superseded by contextual embeddings produced by language models such as BiL.STM [S] and various Transformers 330
93 95 In the case of these more complex architectures, the multilingual capabilities of language models resulted from specific objective functions used during training. These objectives "pushed" the models toward universal multilingual representations by forcing the models to perform machine translation [5], translation
22
26
language modeling (TLM) [22 20). or translation ranking 30 94.
The evaluation of the quality of multilingual text representation is not trivial. Cross-lingual and multilingual tasks are actively developed to foster the development of multilingual models. Examples of such cross-lingual and multilingual tasks include cross-lingus natural language inference (21],
question answering 42). named entity recognition [87 [8] or parallel text extraction 97 96). For
example, results of comparison of LASER, mBERT, and XLM in tasks of named entity recognition and part-of- -frr-eh--peec--ehe tagging zero-shot settings suggests that LASER outperforms the latter methods in the case of knowledge transfer [89]. Another important benchmark is XTREME (34,, designed for testing the abilities of cross-lingual transfer across 40 languages and 9 tasks. Despite its massive character, XTREME lacks benchmarking task of sentiment analysis. Also, only mBERT, XLM, XLM-R, and MMTE are used as baseline models.
By far, the most popular pre-trained multilingual language model is Multilingual BERT (mBERT) [26]. has been used in several cross-linguat studies, for instance, in zero-shot knowledge transfer between Slovenian and Croatian languages [62]. exploring code-switching on Spanglish and Hinglish [6]],
and building hierarchical architecture for zero-shot setting 774). The properties and limitations of mBERT have been extensively studied. One of the studies showed that mBERT is not learning a joint representation of languages. Rather, it partitions the representation space between languages [79]. The authors used the Projection Weighted Canonical Correlation Analysis (PWCCA) to analyze how translations of the same sentence are represented mBERT layers. The correlations were stable in pre-trained and fine-tuned models, with the effect being more pronounced in deeper layers of the model. The hierarchical structure induced by correlations was similar to the structure produced by genealogical linguistics.
Another interesting finding was that mBERT encodes Sanguage-specific information within the parameter space, and this language component is not removed by fine-tuning [44]. The language- specific component of mBERT can be removed by estimating the centroid of the language (the mean of mBERT embeddings of a given language vocabulary) and subtracting this centroid from representations produced in the language. The existence of the language component has been proven in multiple tasks: language classification, language similarity, sentence retrieval, word alignment, and machine translation.
Several works tested the cross-lingual abilities of mBERT as compared to monolingual models, finding its performance on low-resource languages inferior to monolingual models [9]]. One experiment used a bilingual version of mBERT and trained it in multiple configurations to test the influence of the linguistic relationship between the source and the target language, the network architecture, and the input and learning objective (37 The authors found that structural similarity and depth of a model are the most significant factors behind mBERT's pross-lingual performance. On the other hand, multi-head attention was not particularly important. Regarding the low-resource languages, 99 compared mBERT with baseline models on tasks of named entity recognition, universal part-of- speech tagging, and universal dependency parsing. The authors focused on the comparison between low- and high-resource languages (determined by the size of the Wikipedia dump in each language). mBERT was found to work well on high-resource languages but under-performed on owwrresuucee
7

---

languages. Finally, Liu et al. [46] analyzed the relationship between the contextual aspect of mBERT and the training dataset size in the context of multilingual datasets. The authors compared contextual mBERT embeddings with non non-contextual models of Verd2Vec and GloVe. As it turns out, mBERT outperforms non-contextual models only when large datasets are available for training or fine-tuning. In the low data regime, as well as when the context window is short (limited input), multilingual mBERT is surpassed by static token embeddings.
5.2 Multilingual sentiment classification
Several surveys present an overview of traditional sentiment classification methods (lexicon- based approaches and shallow models with lexical features engineering) [25] 70). Recently, deep-learning approaches became more prevalent. Attia et al. [6] use convolutional neural networks on word-level embeddings of texts in English, German and Arabic. The approach is computationally expensive as it requires separate embedding dictionaries for each language. An alternative approach is to character-level embeddings. Wehrmann al. (90] trained such model for binary sentiment classification of English, German, Portuguese, and Spanish tweets. A similar model was used in (4) but the authors applied multi-stage pre- precceccsssng. including dictionary checks, Soundex algorithm, and lemmatization. Other examples of deep neural models for multilingual sentiment classification include recurrent neural networks supplied with machine translation. Can al. [16] trained a model on English reviews and evaluated it on tranineet reviews in Russian, Spanish, Turkish, and Dutch using the Google Translation API and pre-trained Glo Ve embeddings for English. A similar approach is presented in [38], where the authors used LASER sentence embeddings to train sentiment classifier on Polish reviews and used this classifier to score reviews translated into other languages.
5.3 Multilingual sentiment datasets
Multilingual sentiment datasets are crucial resources for training sentiment analysis models that can operate across various languages. Unfortunately, very few multilingual datasets contain sentiment polarity annotations. The Multilingual Amazon Reviews Corpus (57) is noteworthy example of such a dataset. It provides customer review data in English, Japanese, German, French, Spanish, and Chinese, annotated for sentiment, Another significant resource is the Multilingual Sentiment Lexicons developed by Chen and Skiena [18), which provides sentiment lexicons for 136 languages. Being a lexicon, it cannot be used to train sentiment models directly, but it can be used for weak supervision. The NTCIR corpus contains information on sentiment polarity for news related to sport and politics for articles in English, Chinese, and Japanese (76). The JMTC dataset [52) (Jigsaw Multilingual Toxic Comment Classification) dataset contains comments from Wikipedia's talk page edits and is available in multiple languages. It is annotated for toxic vs. non-toxic sentiment, which cannot be aligned with our 3-class annotation schema. An interesting resource is Task from SemEval providing English, Spanish, Dutch, Arabic, Russian, and Turkish tweets and SMS messages annotated for 3-class sentiment 67]. The most similar dataset to ours is XED compiled by Öhman et al. [59]. However, this dataset contains original annotations only for English and Finnish, and the remaining 30 languages are "projected" (i.c., annotated samples in English and Finnish are translated into multiple languages).
6 Discussion
The main focus of the paper is the introduction of a massively multilingual large collection of sentiment datasets and the extensive benchmark of model training and validation scenarios. During the construction of the benchmark, we have gathered valuable experiences that we want to share with the community. Our most important observation is that single multilingual sentiment classifier can perform approximately equally well for all languages. A small group of pre- pre-trained models (XML- R, LaBSE, MPNet) under fine-tuning scenario produces the best results relative to other models. Obviously, the absolute Fi score across languages differs significantly (due to data scarcity, data quality, or difficulty of samples), but these selected models always produce top classifiers. Secondly, all models perform better under the fine-tuning scenario, but the performance gain varies from model to model. When evaluated on the test dataset, models gained between pp. (mUSF-dist) and 9 pp. (mBERT, DEsTilmBERT), When tested on the internal datasets, Fi gains varied between 0 pp (mUSE-
8

---

dist) and 20 pp. (DistilBBERT), with mT5 and XML-R improving by 17 pp. and 15 pp., respectively. In general, the largest gains can be observed for models trained with the masked language modeling technique (XMI.-R, mBERT). We also observe that models which produce lower-quality sentence embeddings gain more from fine-tuning.
We also find that bigger models tend to achieve better performance in all data modalities and training scenarios. Of course, there are counterexamples, e.g., mUSE-dist is smaller than mBERT but achieves better performance in the HL scenario for all datasets. This indicates that the size of the model is an important factor for determining its performance, but other factors, like the domain and the type of pretraining task, may also affect the results. Moreover, we observe that the correlation between model size and model performance is weaker after fine-tuning. This means that one may often find a competitive model with similar performance to the state-of-thee model but significantly smaller and faster for the production environment.
Our final remark is a warning: in our experiments, we have encountered a significant variability in performance conditioned on particular data split. In one of the cross-validation fine-tuning experiments, we observed a data fold that produced consistently worse results (up to 4 pp.) for all models. This fold was produced by the same random seed and (most probably) consisted of samples with increased difficulty. Since our experiments involved multiple models, multiple training scenarios, multiple datasets, and multiple languages, we had rely on data subsampling. Conducting experiments on fixed seed for sample selection could lead to similar data fold and, consequently. to biased experimental results.
7 Limitations
Despite the fact that our collection is the largest public collection of multilingual sentiment datasets, still covers only 27 languages. The collection of datasets is highly biased towards the Indo- European family of languages, English in particular. We attribute this bias to the general culture of scientific publishing and its enforcement of English as the primary carrier of scientific discovery. Our work's 's main potential negative social impact is that the models developed and trained using the provided datasets may still exhibit better performance for the major languages. This could further perpetuate the existing language disparities and inequality in sentiment analysis capabilities across different languages. Addressing this limitation and working towards more equitable representation and performance across languages is crucial to avoid reinforcing language biases and the potential marginalization of underrepresented languages. The ethical implications of such disparities should be thoroughly discussed and considered.
An important limitation of our dataset collection is significant variance in sample quality across all datasets and all languages. preseetss the distribution of self-confidence label-quality score for each data point computed by the cleanlab [58]. The distribution of quality is skewed in favor of popular languages, with low-resource languages suffering from data quality issues. A related limitation is caused by an unequal distribution of data modalities across languages. For instance, our benchmark clearly shows that all models universally underperform when tested on Portuguese datasets. This is the direct result of the fact that data points for Portuguese almost exclusively represent the domain of social media. As a consequence, some combinations of filtering facets in our dataset collection produce very little data (i.c., asking for social media data in the Germanic genus of muro-uur languages will produce a significantly larger dataset than asking for news data representing Afro-Asiatic languages).
Finally, we acknowledge the lack of internal coherence of annotation protocols between datasets and languages. We have enforced strict quality criteria and rejected all datasets published without the annotation protocol, but we were unable, for obvious reasons, to unify annotation guidelines. The annotation of sentiment expressions and the assignment of sentiment labels are heavily subjective and, at the same time, influenced by cultural and linguistic features. Unfortunately, it is possible that semantically similar utterances will be assigned conflicting labels they come from different datasets or modalities.

---

Acknowledgments and Disclosure of Funding
The work presents results achieved in projects funded by (1) European Regional Development Fund (ERDF) in RPO WD 2014-2020 (project no. RPDS.01 1....00-0220005//200 (2) the Polish Ministry of Education and Science, CLARIN-PL; (3) the ERDF in the 2014-2020 Smart Growth Operational Programme, CLARIN Common Language Resources and Technology Infrastructure; (4) project CLARIN-Q (agreement no. 2022/WK (22/); (5) the Department of Artificial Intelligence at Wroclaw University of Science and Technology. Mikolaj Morzy is supported by (6) EEA Financial Mechanism
2014-2021 Project 2019/35//HS66/34998
References
[1] Marwan Al Omari, Moustafa AI-Hajj. Nacereddine Hammami, and Amani Sabra. Sentiment classifier: Logistic regression for arabic services' reviews in lebanon. In 2019 International Conference on Computer and Information Sciences (ICCIS), pages 1-5, 2019. doi: 10. 10.1109/ ICCISci.201..87163944
[2] Mohamed Aly and Amir Atiya. LABR: A large scale Arabic book reviews dataset. In Proceed- ings of the 51 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494-498, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https:tt///// /tt.ttts::/aacllaaaa http:/::/acaaathhlooooo
[3] Adam Amram, Anat Ben David, and Reut Tsarfaty. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from Modern Hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242- 2252, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL. pattts:://aclaneeetet
otg/C18- 1190
[4] Monika Arora and Vineet Kansal. Character level embedding with deep convolutional neural net- work for text normalization of unstructured data for twitter sentiment analysis. Social Network Analysis and Mining, 91): Mar 2019. ISSN 1869-5469. doi: 10. 10.11077411328--19--05777
URL (otLhhtps::/ //oi. org/ 10. 1007/ / 13wws-111---0- 019-0557-
[5] Mikel Artetxe and Holger Schwenk. Massively Multilingual Sentence Embeddings for Zero- Shot Cross-Lingual Transfer and Beyond. Transactions of the Association for Computational Linguistics, 7:597-610, 09 2019. ISSN 2307-387X. doi: 10.1 2alp:/aaal_0000555525ae2222 URL https //doi. 10. wttp///eeeeeeeeeeeeeee.e a_ a_0088
[6] Mohammed Attia, Younes Samih, Ali Elkahky, and Laura Kallmeyer. Multilingual multi- class sentiment classification using convolutional neural networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https /acclanthology. wrcpiloooyyorg.gg. org/LL8-1100
[7] Ramy Baly, Alaa Khaddaj. Hazem M. Hajj. Wassim El-Hajj. and Khaled Bashir Shaban. ArSentD-LEV: A Multi-Topic Corpus for Target-based Sentiment Analysis in Arabic Levantine Tweets. In Hend Al-Khalifa. King Saud University, KSA Walid Magdy, University of Ed- inburgh, UK Kareem Darwish, Qatar Computing Research Institute, Qatar Tamer Elsayed, Qatar University, and Qatar, editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France, may 2018. European Language Resources Association (ELRA). ISBN 979-10-95546-25-2.
[8] Francesco Barbieri, Valerio Basile, Danilo Croce, Malvina Nissim, Nicole Novielli, and Viviana Patti. Overview of the Evalita 2016 SENTIment POLarity Classification Task. In Proceedings of Third Italian Conference on Computational Linguisties (CLiC-it 2016) & Fifth Evaluation Cam- paign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2016), Naples, Italy, December 2016. URL http::ttps/// h/tp./hall.... inria. -1444714
[9] Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados Xlm-t: multilingual language model toolkit for twitter. arXiv e-prints, pages arXiv-2104, 2021.
10

---

[10] Mohaddeseh Bastan, Mahnaz Koupaee, Youngseo Son, Richard Sicoli, and Niranjan Balasubra- manian. Author's s sentiment prediction. Proceedings of the 28th International Conference on Computational Linguistics, pages (04,615, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguisties. doi: 10. 18653/v1/2000 main.52. wwwgg/2220.oom http:/220.ccooonnnooooooo.. main.main... 52
URL.tttps https://aaaaathhoooo
[11]
Vuk Batanović, Boško Nikolić, and Milan Milosavljević. Reliable baselines for sentiment
analysis in resource- languages: The Serbian movie review dataset. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'I6) 16), pages 2688-2692 Portorož. Slovenia, May 2016. European Language Resources Association (ELRA). 1427
URL haatttss:/aalaattoollasoeo
[12] Vuk Batanović, Miloš Cvetanović, and Boško Nikolić. versatile framework for resource- limited sentiment articulation, annotation, and analysis of short texts. PLOS ONE, 15(11):1-30. 022.4.00000....0
11 2020. doi: 10.1371 /3/1//jurnll.pooo 0242050. httphttss://do wttp:oo.//////...... 1371/journa pone. 0242050
[13] Henrico Brum and Maria das Graças Volpe Nunes. Building sentiment corpus of tweets in Brazilian Portuguese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https: aat.hooooaln Borg/L18-10 1658
[14] Jože Bučar, Martin Žnidaršič, and Janez Povh. Annotated news corpora and a lexicon for sentiment analysis in slovene. Language Resources and Evaluation, 52(3):895-919, September 2018. doi: 10. 0011007/11579-01--9990
[15] Lyle Campbell. Ethnologue: Languages of the world, 2008.
[16] Ethem Can, Aysu Ezen-Can, and Fazli Can. Multilingual sentiment analysis: An RNN-based framework for limited data. Computing Research Repository, UxXiv:180600451,, 2018. URL Version
http://arxiv.. org aws/sass// 04511
[17] Emile Chapuis, Pierre Colombo, Matteo Manica, Matthieu Labeau, and Chloé Clavel. Hier- archical tra-tiaining for sequence labelling in spoken dialog. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2636-2648, Online, November 2020. Association for Computational Linguistics. doi: 10. http://w/22....nnnggeeemmmm https://acclattholl
URL
org/ 2020. Siindings-emll 2399
[18] Yanqing Chen and Steven Skiena. Building sentiment lexicons for all major languages. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 383-389, 2014.
[19] Mark Cieliebak, Jan Milan Deriu, Dominic Egger, and Fatih Uzdilli. A Twitter corpus and benchmark resources for German sentiment analysis. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 45-51, Valencia, Spain, April 2017. Association for Computational Linguistics. doi: 10. 2...8853/1//W11-1100 URL htttps://aclantholoos org/
[20] Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In Pro- ceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc. URL battps://dil acm. www.aat.................t....e.. brrg/doi/10 5555/ 3454287 3454921
[21] Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium, October-Novemter 2018. Association for Computational Linguistics. doi: 10. (0.18653/11/D18-11 /
URL htthhttt:///// http//cccantooooyoooooo wwacchooogy.rrgg/e/ orwp://-/////001000
[22] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning scale. In Proceedings of the 58th Annual
11

---

Meeting of the Association for Computational Linguisties, pages 1440-8451 Online, July 2020. Association for Computational Linguistics. doi: 10.18653//1/2022..cc main.747. URL https://aallarthology org/ 2020 acl-main. 747
[23] Fermin L Cruz, Jose A Troyano, Fernando Enriquez, and Javier Ortega. Experiments in sentiment classification of movie reviews in spanish. Procesamiento del Lenguaje Natural, 41: 73-80, 2008.
[24] Peter T Daniels. The atlas of languages: The origin and development of languages throughout the world. Language, 81(2):117-511 2005.
[25] Kia Dashtipour, Soujanya Poria, Amir Hussain, Erik Cambria, Ahmad YA Hawalah, Alexan- der Gelbukh, and Qiang Zhou. Multilingual sentiment analysis: state of the art and in- dependent comparison of techniques. Cognitive computation, 8(4):757-771, 2016. doi: (55007//12559-00--9445555
[26] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171 1171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10. 1011886//111111111 1423. http::/ts:://cl http://cccaathhoogooooooo
1423
[27] Matthew S. Dryer and Martin Haspelmath, editors. WALS Online (v2020.3). Zenodo, 2013. doi: 22228//zenooo.3338
URL htRpshtps:/// //ttps://doi.... org/10 10. /281//eeod22 7385533
(28) Ashraf Elnagar and Omar Einea. BRAD 1.0: Book reviews in arabic dataset. In 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA), pages 1-8, 2016. doi: 10. 10.110000 10.10/ICCCSA. 20177995000..
[29] Ashraf Elnagar, Yasmin S. Khalifa, and Anas Einea. Hotel Cirabic-Reviewe Dataset Construction for Sentiment Analysis Applications. Springer International Publishing, Cham, 2018. ISBN 978-3-319-67056-0. doi:
3. URL http:: wte.ddi..orr....1.0 10. 1007/
978-3-319-67056
[30] Fangxiaoyu Feng, Yinfei Yang. Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language- agnostic BERT Sentence Embedding. Computing Research Repository, arXiv:2007.01 .0000007.115,. 2020. Version
[31] Steven L Gordon. The sociology of sentiments and emotion. In Social psychology, pages 162-592. Routledge, 2017.
[32] Ivan Habernal, Tomás Ptáček, and Josef Steinberger. Sentiment analysis in Czech social media using supervised machine learning. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 65-74, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL htttp:://ccaathhologyy org/ W13-1609
[33] Pedram Hosseini, Ali Ahmadian Ramaki, Hassan Maleki, Mansourch Anvari, and Seyed Abol- ghasem Mirroshandel. SentiPers: A sentiment analysis corpus for persian. Computing Research Repository, arXiv:1801.07737, 2018. uttp::/t::/aarttt /tap::v///xxiv.//// org/ absaaaas.//11 07737 2.
Version
[34] Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin John- son. XTREME: massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, URL httts:://proceeeeeeeee
pages 4411 4411-4421. PMLR, 13-18 Jul 2020.
mlr press/vi19/
hu20b. html
[35] Clayton
Hutto and Eric Gilbert. VADER: A parsimonious baled model for sentiment analysis of social media text. In Proceedings of the International AAAI Conference on Web and Social Media, volume 8, pages 216-225, May 2014. URL. http::///ssssssss.. baai.org// org//ndex iidexx
phppICCWM// wttp:///tttteeeeeeeeeeeeeeeeeeee wtt.tiieeeeeeeee555/00
12

---

[36] Crowdflower Inc. Twitter us airline sentiment, 2015. URL https: http:://wwwwkkglll...m com/
crowdfloRer/twitter http://owwrrtttttteeaarreeeeeee Sarline-sentimenn
[37]
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. Cross-lingual ability of multilin-
gual bert: An empirical study. In International Conference on Learning Representations, 2020.
URL httpps://ppenrrvier wtwp:rrvvieieeeeefrrrrnn wew.eeoooruiiddaeelllla.e......e
[38] Kamil Kanclerz, Piotr Milkowski, and Jan Kocoñ. Cross-lingual deep neural transfer learning in sentiment analysis. Procedia Computer Science, 176: 776:128-133 137, 2020. ISSN 1877-0509. doi: https://doi.or//1... 11111101111611..orrssss 2020. 10...08.0144 0.44.. URL.https: / /www. sciencedirect. com/ Knowledge- -aaeedeee-sedd and Intelligent Informa- tion & Engineering Systems: Proceedings of the 24th International Conference KES2020.
sci
[39] Brian Keith Norambuena, Exequiel Lettura, and Claudio Villegas. Sentiment analysis and opinion mining applied to scientific paper reviews. Intelligent Data Analysis, 23:191-214, 02 2019. doi: 0..2333/DA-173300
[40] Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. The multilingual Amazon reviews corpus. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4563-4568 Online, November 2020. Association for Computational Linguistics. doi: 10. (0.18653/v1/2000 10055///11220002mmmmmm.....333
URL https://
aclanthology aclarthology.orr/2200.. www.2.00.mmmpmmi.....mmmmmmmmm.. 369.
[41] Jan Kocoñ, Piotr Milkowski, and Monika Ziiko-Zieliíska. Multi-level sentiment analysis of PolEmo 2.0: Extended corpus of multi-domain consumer reviews. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 980- 991, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 0...8655/v1/K199100 1092.
ptattpp://acllatbbooog.ooo
1092
[42] Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315-7330, Online, July 2020. Association for Computational Linguistics. doi: 1..18653/112222..acll m53.maan..65. URL https://aclaattholog
2020 acl main. 653
[43] Jun Z Li, Devin M Absher, Hua Tang, Audrey M Southwick, Amanda M Casto, Sohini Ramachandran, Howard M Cann, Gregory S Barsh, Mareus Feldman, Luigi L Cavalli-Sforza, et al. Worldwide human relationships inferred from genome-wide patterns of variation. Science, 2008
31 319(5866):
100-1104, 1008666)1100110000
[44] Jindrich Libovický, Rudolf Rosa, and Alexander Fraser. How language-neutral is multilingual bert? Computing Research Repository, arXiv:1911.03310 2019. Version 1.
[45] Yiou Lin, Hang Lei, Jia Wu, and Xiaoyu Li. An empirical study on sentiment classification of Chinese review using word embedding. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation: Posters, pages 258-266, Shanghai, China, October 2015. URL hhtttp:://cclannhooogg.oo
[46] Chi-Liang Liu, Tsung- Yuan Hsu, Yung-Sung Chuang, and Hung yi Lee. What makes multi- lingual bert multilingual? Computing Research Repository, arXiv:2010. 10938, 2020. Version 1.
[47] Alexandre Magueresse, Vincent Carles, and Evan Heetderks. Low-resource languages: A review of past work and future challenges. arXiv preprint wrXiv:2006.07244 2020.
[48] Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4):782-796 apr 2014. ISSN 2330-1635. doi:
10. 10.1000//si.23062. URL httpt:://p/:oii org 10. 1002/ asi 23062
[49] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In Yoshua Bengio and Yann LeCun, editors, 1sl International Conference on Learning Representations, ICLR 2013, Scousdale,, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. URL http: // arxiv org/ abs/ 1301 3781
org/abs/
13

---

[50] Tomas Mikolov, Quoe V. Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. Computing Research Repository, arXiv: 309.4468, 2013. Version 1.
[51] Igor Mozetić, Miha Grëar, and Jasmina Smailović. Multilingual twitter sentiment classification: The role of human annotators. PLOS ONE, 11(5):1-26, 05 2016. doi: 10. 1/j/33//oaaallpoo
0155036. https://///// org/ 10. 1371/journal. pow...aa...o............ 0155036
[52] Jigsaw Multilingual. Jigsaw multilingual toxic comment classification, 2020.
[53] Mahmoud Nabil, Mohamed Aly, and Amir Atiya. ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pages 2515 2515-25194 19, Lisbon, Portugal, September 2015. Association for Computational Linguistics. URL https:tt///// httpss://aaaaatthoocoooo htwp:/ooooooo..-------00001 1299
doi: 11116653/1/DD1-1D1
[54] Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. Semeval- 2016 task 4: Sentiment analysis in twitter. arXiv preprint arXiv:1912.019: 2019.
[55] Sascha Narr, Michael Hülfenhaus, and Sahin Albayrak. Language-independent twitter sentiment analysis. In Workshop on Knowledge Discovery, Data Mining and Machine Learning (KDML- 2012), 2012.
[56] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN)
pages 188-197, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/1/DD19-100 1018. URL https //aclanthology.oo org.:llloooo.ggggggoooooo D19 0rg/ 1018
[57] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fine- aineggrined aspects. In Proceedings of the 2019 conference on empirical methods natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNIP), pages 188-197, 2019.
[58] Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 10:1373-1411, 2021.
[59] Emily Öhman, Mare Pamies, Kaisla Kajava, and Jörg Tiedemann. XED: A multilingual dataset for sentiment analysis and emotion detection. In Proceedings of the 28th International Conference Computational Linguistics. pages 6542-6552, Barcelona, Spain (Online), De- cember 2020. International Committee on Computational I Linguistics. doi: 10.1 8653/v1/2020. coling-main.575. URL httpp:://aclanthholllee
[60] Parth Patwa, Gustavo Aguilar, Sudipta Kar, Suraj Pandey, Srinivas PYKL., Björn Gambäck, Tanmoy Chakraborty, Thamar Solorio, and Amitava Das. SemEval-2020 task 9: Overview of sentiment analysis of code-mixed tweets. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 774-790, Barcelona (online), December 2020. International Committee for Computational Linguistics. doi: 10. 10.118633///222200mmmmvaaal https://aclarthologyy org/ wwg//////0022. 20020.eemeval-11
URL
[61] Parth Patwa, Gustavo Aguilar, Sudipta Kar, Suraj Pandey, Srinivas PYKL, Björ Gambäck, Tanmoy Chakraborty, Thamar Solorio, and Amitava Das. SemEval-2020 task 9: Overview of sentiment analysis of code-mixed tweets. In Proceedings of the Fourteenth Workshop on Seman- tic Evaluation, pages 774-790, Barcelona (online), December 2020. International Committee wrgg
for Computational Linguistics. URL httptttta//ccaaathheee
org/ 2020. semeval-1.100
[62] Andraž Pelicon, Marko Pranjić, Dragana Miljković, Blaž Škrlj, and Senja Pollak. Zero-shot learning for cross-lingual news sentiment classification. Applied Sciences, 10(17):5993, 2020.
[63] Andraž Pelicon, Marko Pranjié, Dragana Miljković, Blaž Škrlj, and Senja Pollak. Zero-shot learning for cross-lingual news sentiment classification. Applied Sciences, 10(17), 2020. ISSN 2076-3417. doi: 10.3390/app11159999 URL. https:: // www. mdpi com/2006--411110/11
5993
14

---

[64] Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),, pages 5512-4525, Online, November 2020. Association for Computational Linguistics. doi: 10.1 16833/ 8633/1122220eemmmmmmm...66.. URL https://
aclanthology org/ 2020. mmnlp-main. 365.
[65] Anna Rogers, Alexey Romanov, Anna Rumshisky, Svitlana Volkova, Mikhail Gronas, and Alex Gribov. RuSentiment: An enriched sentiment analysis dataset for social media in Russian. In Proceedings of the 27th International Conference on Computational Linguistics, pages 755-763, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL https://aclanthologgy org/ C16-1064
[66] Sara Rosenthal, Noura Farra, and Preslav Nakov. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 1ith International Workshop on Semantic Evaluation (SemEval- 2017). pages 502-518. Vancouver, Canada, August 2017. Association for Computational Linguistics, doi: 1188655/11/1172000
[67] Sara Rosenthal, Noura Farra, and Preslav Nakov. Semeval-2017 task 4: Sentiment analysis in twitter. arXiv preprint arXiv: /0/i:////20007/4 2019.
[68] Sebastian Ruder, Ivan Vulić, and Anders Søgaard. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569 631, Aug 2019. ISSN 1076-9757. doi: 10.1613/jair...1 w.t.jarr..1.111. 16400 URL http:: doi org/ 1613/jair 1. 11640
[69] Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and Ireneusz Gawlik. KLEJ: Compre- hensive benchmark for Polish language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1191-1201, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653//v12000.cclmmmaaa https: pttp:://accaanthlllee
URL
org/ 22g/2020.... acl main. 111
[70] Santwana Sagnika, Anshuman Pattanaik, Bhabani Shankar Prasad Mishra, and Saroj K Meher. A review on multi-lingual sentiment analysis by machine learning methods. Journal of Engineering Science & Technology Review, 13(2):154-166, 2020. doi: 10.25 00.25103/jestr11322 19..
[71] Mohammad Salameh, Saif Mohammad, and Svetlana Kiritchenko. Sentiment after translation: case-study on Arabic social media posts. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 767-777, Denver, Colorado, May-June 2015. Association for Computational Linguistics. doi: 10.3115/11/115-0078
wwg. -15-
[72] Niek J Sanders. Handers-Twitter Sentiment Corpus. Sanders Analytics LLC, 2011.
[73] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a dis- tilled version of BERT: smaller, faster, cheaper and lighter. Computing Research Repository, arXiv:1910.01108,, 2020. Version
[74] Anindya Sarkar, Sujeeth Reddy, and Raghu Sesha Iyengar. sero-shot multilingual sentiment analysis using hierarchical attentive network and bert. In Proceedings of the 2019 3rd Interna- tional Conference on Natural Language Processing and Information Retrieval, pages 49-56, 2019.
[75] Dietmar Schabus and Marcin Skowron. Academic-industiaa perspective the development and deployment of a moderation system for a newspaper website. In Proceedings of the Eleventh International Conference Language Resources and Evaluation (LREC 2018). Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https //aclanthology. . wwwp//gg/////// 1253
[76] Yohei Seki, David Kirk Evans, Lui Ku, Le Sun 0001, Hsin-Hsi Chen, and Noriko Kando. Overview of multilingual opinion analysis task at ntcir-7. In NTCIR, pages 185-203, 2008.
[77] Zareen Sharf and Saif Ur Rahman. Performing natural language processing roman urdu datasets. International Journal of Computer Science and Network Security, volume 18, pages 141-148, 2018. U //pp:er//pppppp//eeeeeee ijcsns
15

---

[78] Emily Sheng and David Uthus. Investigating societal biases in a poetry composition system. In Proceedings of the Second Workshop on Gender Bias Natural Language Processing, pages 93-106. Barcelona, Spain (Online), December 2020. Association for Computational Linguisties. http:/200022eee11..0.. gebnlp-1.9
[79] Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. BERT is not interlingua and the bias of tokenization. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 47-55, Hong Kong. China, November 2019. Association for Computational Linguisties. doi: 10. 1111666////D1--61666 URL https://
oclaattoology.org/D1----
[80] Pawel Sobkowicz and Antoni Sobkowicz. Two-year study of emotion and communication patterns in highly polarized political discussion forum. Social Science Computer Review, 30 (4):448 469, 2012. doi: 10. 10.11 011177008944991144331111
[81] Uga Sprogis and Matiss Rikters. What can we learn from almost a decade of food tweets. Computing Research Repository, M2Xiv:2007.051990 2020. [RLts:::: org/ abs/ 2007.05194 Version 2.
[82] Rachele Sprugnoli. Multiemotions-i:: new dataset for opinion polarity and emotion analysis for italian. In Proceedings of the Seventh Italian Conference on Computational Linguistics, 12 2020.
[83] Arthit Suriyawongkul, Ekapol Chuangsuwanich, Pattarawat Chormai, and Charin Polpanumas. wrthainlpwiisssihht--ettmm First release (v1.0), September 2019. https:s::://di org/
10. www.52220ooooo..o 34552/eeoooo...4447777 Zenodo.
[84] Mike Thelwall, Kevan Buckley, and Georgios Paltoglou. Sentiment strength detection for the social web. J. Am. Soc. Inf. Sei. Technol., 63(1): 173, January 2012. ISSN 1532-2882. doi:
10.1002/asi.21662 URI. https::/p//ooo org/ 10. 1002/000///s/ 21662
[85] Sarah Grey Thomason and Terrence Kaufman. Language contact, volume 22. Edinburgh University Press Edinburgh, 2001.
[86] Ekkalak Thongthanomkul. Tanapol Nearunchorn, and Yuwat Chuesathuchon. wongnai-corpe https://github http://ggithuu..cmooooo///g///// www.wo///oooooeeec hwtpm///wggaa///aaaiooooooroop
2019.
[87] Erik F. Tjong Kim Sang. Introduction to the CoNLL-2002 shared task: Language-indeeeendnt named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002), 2002. URL hetttp:://acaanhhoooy/// org/ WWW-20220
[88] Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CONI.L-2003 shared task: Langunge-independee named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147, 2003. URL https
/aaclantholoyy org/N03-0411 0419
[89] Niels van der Heijden, Samira Abnar, and Ekaterina Shutova. A comparison of architectures and pretraining methods for contextualized multilingual word embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9090-9097, 2020.
[90] Joonatas Wehrmann, Willian Becker, Henry F. L. Cagnini, and Rodrigo C. Barros. character- based convolutional neural network for language-agnosti twitter sentiment analysis. In 2017 International Joint Conference on Neural Networks (IJCNN) pages 2384-2391, 2017. doi: 1..1199/JCNNN.21177966
[91] Shijie Wu and Mark Dredze. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18655//122222eeee44hp..
[92] Min Xiao and Yuhong Guo. Distributed word representation learning for cross-lingus de- pendency parsing- In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 119-129, Ann Arbor, Michigan, June 2014. Association for Com- putational Linguistics. doi: 10.31155/11W114-1111 URI. htttss:///cclaeeeee htttps:/acllattioogg..oos
114-1613
16

---

[93] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text trans- former. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653//1/2021.maaa www.::/aaaaaalmaa..a.... URL htttp::/aclaattool.gg htwps/hoooooooogg/2222/ org/2021 maacc-mainn
[94] Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IICAI-19, pages 5370-5378. International Joint Conferences on Artificial Intelligence Organization, 7 2019. doi: 00124963/jcai.201197 URL https:///oi.org 10. 24963/1 2019/746
[95] Yinfei Yang. Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil, Multilingual universal sentence encoder for semantic retrieval. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 87-94, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.acl-demos.. 12. URL /ttpp
org/2020 acl-demos. 12
[96] Michal Ziemski, Marcin Junczys-Dowmunt and Bruno Pouliquen. The united nations parallel corpus v1.0. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). pages 3530-35344 Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL https: :ttp::///wwet /www://wwwc/oooo//// http::/wwwwwwwlla... acwweb. org/ wwtaaanoogg//l--15a000
[97] Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora. Proceedings of the 10th Workshop on Building and Using Comparable Corpora, pages 60-67, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 1/11122511111122212.. URL
https://wwes aclweb. org/ http://nhhoool//1oooooo N17-
17

---

A
Datasets
We present detailed lists of datasets included in our research in Tables B and They include language. category, dataset size, class balance, and basic dataset characteristics.
Table 3: List of all monolingual datasets used in experiments. Category (Cat.): R Reviews, SM Social Media, C Chats, N News, P Poems, M Mixed. HI. human laheled, #Words and #Chars are mean values
Paper
Lang
Cat.
HL
Samples
NEG/NEU/POS
#Words
#Char.
Omari et al.
ar
R
No
3096
13.0/10.2/76.8
Elnagar et al. 29
ar
R
No
400101
13.0/19.9/6.11
22
127
Aly and Atiya
ar
R
No
6250
11.6/17.9/70.5
65
343
Elnagar and Finea
28
R
No
504007
15.4/21 .0/63.6
77
424
Baly ct al.
ar
SM
Yes
2809
47.2/23.9/29.0
22
130
Nabil et al. [53
ar
SM
Yes
3224
50.9/25 10/24.1
Salameh et al.
ar
SM
Yes
1199
48.00/0../41.5
11
51
Salameh ct al.
ar
SM
Yes
1998
07.5/10.1/2244
20
107
Habernal ct al.
R
No
91140
7/33.9
50
311
Habernal ct al.
cs
R
No
92758
7.9/23 4/68.7
20
131
Habernal et al.
32
cs
SM
9752
20.4/53. 1/26.5
Habernal et al.
32
cs
SM
Yes
2637
30.8/60.6/8.6
33
170
Cieliebak et al. [1
de
SM
Yes
9948
2/.2449.2/24.6
11
86
Schabus and Skowron
75
de
SM
Yes
3598
17.3/51.5/1.
33
237
Chapuis
C
Yes
12138
31.8/46.5/21.7
12
48
Chapuis
en
C
Yes
4643
22.3/48. 9/28.8
15
Malo ct al. 48
cm
N
Yes
3448
12.2/62.
22
124
Bastan al. 10
en
N
Yes
5333
11.6/37.3/51.0
388
2129
Hutto and Gilbert
35
N
No
5190
29.3/52.9/17.
17
104
Sheng and Uthus 78
en
P
Yes
1052
18.3/15, 8165.9
Hutto and Gilbert
cn
R
No
3708
44.2/19.5/46,
16
87
Hutto and Gilbert
35
en
R
No
10605
19.6/1.5/48.
19
111
Ni et al. [56
en
R
No
1883238
8.3/8.0/83.7
70
382
Sanders 12
SM
Yes
3424
16.7/68. 1/15.2
14
97
Thelwall et al.
en
SM
Yes
11759
28.0/34.0/38.
26
147
Inc. 36
en
SM
Yes
14427
63.
17
104
Hutto and Gilhert
35
en
SM
No
4200
26.9/17.0/56.
13
79
Keith Norambuena al.
39
R
Yes
382
45.0/27.2/27.8
165
1033
Cruz al. 23
R
No
3871
32.9/32.3/34.9
511
3000
Hosseini
fa
R
13525
12.0/37.5/50.5
21
104
Amram et al.
he
SM
Yes
8619
26.5/2.8/70.8
22
110
Pelicon ct al.
63
hr
N
Yes
2025
22.5/61 4/11.00
161
1021
Barbieri ct al.
it
SM
Yes
8926
16,7/41.7/216
14
101
Sprugnoli [82
it
SM
Yes
3139
24.6/14.9/60.6
17
106
Sprogis and Rikters
Iv
SM
Yes
5790
23.8/45 1,3
20
138
Rybak et al.
669
pl
R
No
10074
10.8/13.2/56.0
80
494
Kocoń et al.
41
pl
R
Yes
57038
42.4/26. 8/30.8
30
175
Sobkowicz and Sobkowicz
80
pl
SM
Yes
645
50.7/47.3/2.1
33
230
Brum and Volpe Nunes
pt
SM
Yes
10109
28.8/25. 1/46.1
12
74
Rogers et al.
65
ru
SM
Yes
23226
8/.4..66//8.
12
Bučar et al.
[4]
sl
N
Yes
10417
32.0/52.0/16.0
309
2017
Batanović et al.
sr
R
No
4724
17.8/43, 7/38.5
498
3097
Batanović al.
12]
R
Yes
3948
30.3/18. 3/18.111 1/51
18
105
Thongthanomkul
th
R
No
46193
5,4/30.5164
29
544
Suriyawongkul
183
th
SM
Yes
26126
26. 1655 6/18..
90
Sharf and Rahman
ur
M
Yes
19660
26.7/43.6/2 6/299.7
13
69
Lin al.
45
zh
R
No
125725
28.6/21
51
128
18

---

Table List of all multilingual datasets used in experiments. Category (Cat.): Reviews, SM
Social Media, C Chats, N News, P Poems, M Mixed. human labeled
Paper
Cat.
Lang
HI.
Samples
(NEG/NEU/OOS)
#Words
#Char.
Narr et al. [55]
SM
de
Yes
953
10.0/75.1/14.9
12
80
Yes
1781
16.9/63.3/19.8
13
81
Yes
7073
17.4/60.0/22.6
14
78
Yes
685
23.4/53.4/23.2
14
82
fr
Yes
1786
25.0/54.3/20.8
15
83
Yes
759
28. 13.1333.2/38.7
14
78
Yes
1769
30.7/33.9/354
14
78
Keung et al.
40
R
de
No
209073
40.1/20.0/39.
33
208
No
209393
40.0/20.0/40.0
34
179
No
208127
40.2/20.0/39.8
27
152
No
208160
40.2/20. 1/39.7
28
160
ja
No
209780
40.0/20.0/40.0
101
No
205977
39 8/200.1/440
50
Rosenthal et al. [66]
M
Yes
9391
35.5/40.6/23.9
14
105
Yes
65071
19.1/45.7/35.2
18
111
Patwa et al. 600
SM
Yes
14920
16.8/33.1/50.0
16
86
Yes
16999
29.4/37.6/33.0
27
128
Mozetič et al. 51
SM
bg
Yes
62150
22.6/45.9/31.5
12
85
Yes
36183
33.4/30.5/36.1
12
75
de
Yes
90534
19.7/52.8/27. 44
12
94
en
Yes
85784
26.8/44.1/29.
12
77
Yes
191412
11.8/37.9/50.3
14
92
hr
Yes
75569
25.7/23.9/50.4
12
91
hu
Yes
56682
15.9/31.0/53.1
11
83
pl
Yes
168931
1..0/26.1/43.9
11
82
pt
Yes
145197
37.2/35.0/27..8
10
61
ru
Yes
87704
32.0/40.1/27.8
10
67
sk
Yes
56623
25.6/22.5/51.9
13
97
sl
Yes
103126
29.9/43.3/26.8
13
91
sq
Yes
44284
15.7/33.1/51.1
13
90
sr
Yes
67696
34.8/42.8/22.4
13
81
Yes
41346
40.3/31.2/28.5
14
94
B Experiments
B.1
Data quality
In this section, we expand upon the results of experiments included in the benchmark. We begin with the presentation of the distribution of data point quality across all datasets. The data point quality has been computed using the cl eleanlab library self-confidence label-quality score (58). Figure[]rrseem the cumulative distribution of data quality score. As we can see, the data quality still is a pressing issue, despite our efforts to publish well-curated collection. Over 40% of all data points have quality 0.6 or worse. As we have mentioned, low-quality data points are not distributed uniformly across all languages and datasets, but the impact of low-quality data is most pronounced for low-resource languages.
B.2 Pairwise ranked comparison of models
Of course, everyone is most interested in finding the answer to the following question: "Which model should use?" The answer, as usual, is: "It depends". Figure presents the results the Nemenyi pairwise rank comparison test. In neither of the considered training scenarios (HL, HB, TF). single model statistically better than others. However, these results allow us to draw partial conclusions. For instance, the MPNet performs best in the HI scenario, and the same model is not significantly worse than XMI-M, which the best model HB and TF scenarios. We can also notice that mBERT-based models (mBERT and DistillmBERT) proved to be the worst language models for
19

---

3.8
Prbaaiityy
0.6
0.4
0.2
0.0
Cleanlab self-confiddence
Figure 2: Cumulative distribution of sample quality across all datasets
10
MPNet L4552
DistinBERT
mUS2: cnn
NBERRT LASER
XMM-8
mUSE
(a) HL. scenario
(b) HB scenario
XLM4-
mUS8-dist DistlmBERT NBERT
LaBSE
MPNet
mI5
(c) FT scenario
Figure 3: Nemenyi diagrams of model ranks according to the F1-score on each dataset
tasks. Three models stand out as the most promising choices: XLM-R, LaBSE, and MPNet. They achieve comparable performance in all scenarios and test cases. Furthermore, they better than other models in almost all test cases.
20

---

Table 5: Aggregated F1 results of models.
The best results for each test set are highlighted. W
whole test, A avg. by dataset, I internal
XLM-R-dist
DistilmBERI
mUSE-trans.
mUSE-dist
mUSE-cnn
mUSE
XLM-R
LABSE
MPNet
MBERT
LASER
mT5
HL scenario
W
62
62
63
60
59
56
55
59
55
55
54
A
51
54
55
51
49
45
43
50
47
47
45
55
61
56
50
43
38
60
50
49
50
HB scenario
W
66
62
63
62
65
60
59
62
A
57
55
56
54
56
49
48
54
64
63
64
63
63
54
48
64
FT scenario
W
68
68
67
67
66
65
64
63
61
62
62
62
60
56
56
56
70
69
65
67
67
57
58
60
B.3 Detailed results for each model
5 shows the F1 scores of all models aggregated by datasets. The models improve with fine- tuning (up to 0.7 F2) as compared to linear (0.61) or BiLSTM (0.64) layers operating on embeddings produced by models. The performance gains are more pronounced for models trained with masked language modeling objectives (mBERT, XML-R) than for models trained with sentence classification or sentence similarity objectives (LaBSE). Fine-tuning reduces incqualities in the results between models (0.55 vs. 0.43 for best and worst models in the HL scenario). The additional BiLSTM layer on top of transformer-bast based token embeddings has the capacity improve the results compared to the linear classification layer model. The differences are most clear in the case of the results for our internal dataset, where the result improved even by pp. for the mT5 model.
21

---

B.4 Detailed results for each language
We assessed the performance of each model each experimental scenario per language. The texts were sampldd with stratification by language and class label so that language distribution in the test dataset follows the distribution in the whole dataset. We also include the total macro Fi score value in column "all". Results are presented in Figure 4 Those results confirm our previous findings regarding the advantage of XI.M-R, LaBSE, and MPNet models. They outperform other models in most languages, with no clear indication of superiority among them.
Linear Head
XI XLM-R
57
61
58
46
50
50
67
58
63
61
58
30
49
13
45
48
58
49
53
53
38
56
53
41
45
45
55
54
41
LaBSE
54
53
61
59
53
60
57
65
58
63
60
59
44
55
55
52
48
55
57
52
54
41
53
57
34
42
52
61
55
53
59
62
53
48
49
59
62
43
XLM-R-dIst
58
47
50
45
mT5
50
48
59
56
45
5B
56
65
63
54
57
39
49
52
39
44
59
52
47
41
39
54
49
35
40
48
52
57
4B
MBERT
46
55
53
40
55
49
5-
49
45
36
42
4B
39
29
47
51
50
30
41
34
mUSE-dist
50
59
59
54
60
53
50
53
46
47
50
50
51
57
41
43
LASER
48
46
55
52
50
55
50
5-
57
52
52
39
46
46
45
44
44
50
50
48
42
47
52
26
37
43
56
47
36
45
55
48
52
56
52
41
42
40
45
52
43
48
29
23
12
6
R
%
X
5
4
BiLSTM STM Head
XLM-R
57
66
66
62
61
70
66
63
42
50
48
56
62
65
53
41
60
50
46
LeBSE
54
54
61
61
56
58
57
65
60
62
57
53
44
53
57
51
46
58
57
53
57
59
53
37
46
53
59
59
51
MPNot
55
64
58
64
60
50
54
57
XLM-R-diCK
57
54
A6
50
55
60
MTS
55
55
655
66
63
63
60
68
65
58
58
42
52
57
52
44
63
50
58
59
50
36
52
54
52
57
53
MBEAT
50
49
59
59
56
59
51
62
56
56
49
38
48
56
4B
40
44
51
55
36
51
53
35
48
40
55
57
37
DistIIBERT
52
35
47
51
MUSE dist
56
49
44
54
50
lano
8
8
2
*
5
R
X5
5
Fine-tuning
XLM R
61
08
70
66
64
64
72
70
68
65
44
53
86
55
30
65
62
62
45
G1
55
50
51
03
61
58
65
LaBSE
60
62
67
69
67
61
71
70
64
63
42
159
66
58
50
63
64
63
47
61
59
43
48
05
62
G1
MPNet
XLM-R- R-d-R-
50
mm5
57
60
66
68
65
63
62
70
70
64
59
44
53
60
49
44
62
59
53
56
41
60
55
38
54
52
61
45
mBERT
55
64
66
63
60
58
66
63
60
43
54
62
57
42
51
55
59
54
49
56
56
33
DiS
mUSE-dist
54
63
64
64
60
59
63
63
55
43
51
56
49
43
53
55
58
54
54
35
47
47
5
2
%
5
Figure 4: Detailed results of models' comparison. Legend: lang averaged by all languages, ds averaged by dataset, ar Arabic, bg Bulgarian, bs Bosnian, cs Czech, de German, en English, Spanish, fa Persian, fr- French, he Hebrew, hi Hindi, hr Croatian, hu Hungarian, it Italian, ja Japanese, Iv Latvian, pl Polish, pt Portuguese, ru Russian, sk Slovak, sl Slovenian, sq Albanian, Serbian, Swedish, th Thai, ur Urdu, zh Chinese.
B.5 Transfer between data modalities
The last example compares the effectiveness transfer learning between data modalities. The results are presented in Figure 5 As expected, when models are tested against datasets from the same domain (news, social media, reviews), the average performance is much higher than on out-of-domain datasets. What is striking is the visible deterioration of the performance of models trained in the news domain. When knowledge transfer happens between social media and review domains, the average performance of models stays relatively the same.
22

---

0000000
01
10CMl media
Irain domain
Figure 5: Transfer learning between data modalities
C
Listings
The ease of loading and using more than 6M sentiment annotated texts is simple the listings above. All data will downloaded automatically; you must only appropriately filter it to your use case.
import datasets
mms _at_set datasets dataset ("Brand24/nm""
slavic mms datasddttttt.... filter (lambda row: row [" ["Genus"] "Slavic" and row ["Polar questions") "] == "interrogative word order")
Listing I: Loading and filtering sentiment examples for Slavie genus with polar questions formed using interrogative word order.
import datasets
ams_dataset datasets. load_ddttseeeeeeeeeeaaaaaaassssa ("Brand24/mms
afro_ace asiatic mms dataset filter (lambda row: row ["Family"] -- "Afro-
Asiatic" and row [ "Number of cases"] == "no morphological case making")
Listing 2: Loading and filtering sentiment examples for Afro-Asiatic Language Family with no morphological case-making.
import datasets
mms dat_set datasets load fadd_ddaasettt ("Brand24/nms"
www.onllsscillloooooooo....lllls pl and row [ doomain' '] "social _maiiaa_mmdia)))
mms aaas_dataset... filter (lambda row: row [ language filter
Listing 3: Loading and filtering sentiment examples for specific language and domain
import datasets
CLEANLAB THRESHOLD = 0.7
mms_ dataset datasets load_ (BBrand24/mms afro_aelatic mms dataset filter (lambda row: row
cleanlab_self cta.aaieeeecccccdeceeeccceeee >= CLEANLAB (
Listing 4: Loading data based on cleanlab data quality score
23

---

D Error analysis
We present the error analysis of sentiment model results. The Tables 6 and could be little bit small, so we advise checking the editable version of results with the option to check ours or create your pivot tables of the results.
couple of patterns could be spotted Table 6
Japanese neutral class is consistently low performing across all models, not exceeding even 50% fl score. Interestingly, even well-resourced languages like English still have problems with the neutral class. only one English model FT XLMMR achieved 0.7 Fl score.
The positive class is the easiest class to learn for the models, but the performance of neutrals and negatives varies very much, and it always needs to be verified, and potentially there could a need to annotate more data for these classes.
Table 6: F1 scores across all languages and sentiment classes. Legend: ar Arabic, bg Bulgarian, bs Bosnian, cs Czech, de German, en English, es Spanish, fa Persian, fr- French, he Hebrew, hi Hindi, hr Croatian, hu Hungarian, it Italian, ja Japanese, lv Latvian, pl Polish, Portuguese, ru Russian, sk Slovak, Slovenian, sq Albanian, sr Serbian, sv Swedish, Thai, Urdu, zh- Chinese.
couple of patterns could be spotted in Table
The first interesting pattern is hard-to-learn neutral class for 'poems'. It could be related to a tiny dataset, only with approximately 1000 English examples (693 positive, 193 negative, and 166 neutral examples).
The review texts are easy to train for positive and negative; however, they are tough to work decently for neutral class. The reason for it could be mixing positive and negative sentiment in many neutral annotated reviews, mentioning both advantages and disadvantages of products services.
News domain shows interestingly high performance for the neutral class; however, could be related to skewed distribution in this domain for the neutral class. We have neutral as many positive and negative examples summed together.
There many other angles how can benchmarking results. We advise you check the editable results and create your pivot tables for your own purposes and needs.
sharing ing

---

Table 7: Fl scores across domains. Legend: ar Arabic, bg Bulgarian, bs Bosnian, Czech, de- German, en English, es Spanish, fa Persian, fr French, he Hebrew, hi Hindi, Croatian, hu Hungarian, it Italian, ja Japanese, Iv Latvian, pl Polish, pt Portuguese, ru Russian, sk Slovak, sl Slovenian, sq Albanian, sr Serbian, sv Swedish, th Thai, ur Urdu, zh Chinese
Chuts
niked
Dews
poens
revieess
social melia
flaaeu
[1_scu
[]_ace
[1.ps
[1_ace
fl_acx
DisiiBBET
0.72
0.34
Lanse
0,65
0.50
0.55
0.80
0.71
00:3
0.76
0.53
0.90
1.00
0.57
0.96
0.50
0,81
0,26
0,66
0.9
1.55
067
0.30
1860
0.06
0.78
0.63
0183
100
0.57
0.94
02.00
11.06
0.34
046
0:21
MINNI
0.8
0.34
0.51
0.80
0.45
0716
niSSE-dit
0.23
0.41
0.20
0.53
0,75
ET XLMER
0.68
6.7
1.53
0.76
0.77
11.08
0.66
0.56
0.70
0..33
13200
1.00
0.87
0.67
11.00
3.50
0.72
0,69
XLM-R-da
0.70
0.77
0.70
0.77
0.71
0.51
0.82
0.74
01.00
11.00
1.00
0.96
0,47
0.78
0.6
0.74
0.20
0.6
0,73
0.59
0.75
0.45
0.00
0.96
0.50
0154
0.35
0,21
HB. HBBKTT
0.64
0.36
0.68
0.71
0.65
1..0
0.55
0.75
0.59
0.73
0.00
0.5
0.94
0.34
0151
0.35
065
663
MPNo
0.66
0.67
0.57
0.76
0.20
0.57
0.74
0.7
1133
1.000
1.00
0.35
0
1157
0135
066
0.7
0.63
0.56
XLMRR
0.24
0.86
0.57
066
0,82
0.59
0.57
0.55
0.92
0.00
0.50
0.96
0.47
0.54
073
0A5
EIB, XLM--R-do
0,63
0.61
0.5
0.70
0,73
0.68
0.5
0,76
0.52
0.90
0.00
1.00
0.05
0.43
0.82
0.11
067
0
DistUBERT
0.58
0.61
0.38
HI LIBERT
0.83
0.32
043
0.6
0.59
0.55
0,59
0.75
0.37
1.00
0.00
067
0.9
0.21
0.79
062
969
086
TJ. .MENNE
0.65
0.55
0.6
076
6.24
0.00
0.0
0.62
0.43
0.73
800
1.0
0.66
0.36
088
066
0.34
061
0.69
0.79
067
0.29
052
XLME-
0.54
0.66
0.52
0.8
0.71
0.54
0.74
0.82
0.50
0
0.00
0.50
0.96
0.41
0.75
966
0
HL. NLM RT.M-RRddst
0.74
0.51
0.78
0.80
0.63
0.65
0.71
0.70
0.57
0.50
0000
1.00
0.56
0.24
1.77
0.75
963
0,7
25


Reviews:
[{"review_id": "H96UvUwSt2", "reviewer": "Reviewer_5m3s", "summary": "", "questions": "", "limitations": "Yes, the authors have addressed the limitations for this work in sufficient detail. ", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Multilingual SA benchmark, multifaceted too.\n- The benchmarking/evaluation is sufficiently exhaustive. \n- Quality checks seem stringent, leading to only 79 datasets being considered out of 350. \n\nOverall, significant contributions from the paper in terms of dataset and benchmarking.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "I083uUjMN2", "reviewer": "Reviewer_HqfF", "summary": "", "questions": "", "limitations": "The contributions of this paper are quite weak. The authors mainly present basic statistics of the collected datasets and the main results of some selected models on this benchmark. The readers cannot obtain any useful insights from this paper.\n\nAlternatively, a more detailed analysis of the constructed benchmark should be conducted. For example, why the filtered dataset has higher quality? What are the typical expressions in terms of different sentiment polarities in each language? Similarly, more analysis around those adopted models should be conducted.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The investigated problem is interesting where multilingual sentiment classification datasets can be a good lens to analyze the cultural difference in expressing sentiments across different countries.\n* Multiple models are taken to evaluate against this newly collected benchmark.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "I9SAIGOH0h", "reviewer": "Reviewer_98Yo", "summary": "", "questions": "", "limitations": "Yes, the authors have discussed the limitations of their work in 3 fold: \n1 - The collection of the dataset is highly biased toward English. \n2 - There exists a significant variance in sample quality across all datasets and all languages. \n3 - The authors tried enforcing strict quality criteria and rejected many unqualified datasets, but they were unable to unify the annotation guidelines. Understandably, that is a challenging task. ", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1 - The paper presents a collection of a large multilingual dataset for sentiment analysis classification models \n2 - The paper has a strong and convincing motivation. \n3 - The paper addresses the limitations and ethical limitations. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "aqXmtaxJP8", "reviewer": "Reviewer_NixT", "summary": "", "questions": "", "limitations": "Please read my comment in correctness.  ", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The dataset is a valuable resource, which will facilitate research for resource-lean languages. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The submission presents interesting and extensive work. The authors improved many aspects of their paper based on reviewer comments and have put in a lot of additional work in making it clearer and easier to understand. The fact that the proposed datasets are already up and ready for download via `datasets` makes them user-friendly and a clear contribution to the community.
Decision: Accept (Poster)