SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By far, the most popular pre-trained multilingual language model is Multilingual BERT (mBERT) [26]. It has been used in several cross-lingual studies, for instance, in zero-shot knowledge transfer between Slovenian and Croatian languages [62], exploring code-switching on Spanglish and Hinglish [61], and building hierarchical architecture for zero-shot setting [74]. The properties and limitations of mBERT have been extensively studied. One of the studies showed that mBERT is not learning a joint representation of languages. Rather, it partitions the representation space between languages [79]. The authors used the Projection Weighted Canonical Correlation Analysis (PWCCA) to analyze how translations of the same sentence are represented in mBERT layers. The correlations were stable in pre-trained and fine-tuned models, with the effect being more pronounced in deeper layers of the model. The hierarchical structure induced by correlations was similar to the structure produced by genealogical linguistics.
Another interesting finding was that mBERT encodes language-specific information within the parameter space, and this language component is not removed by fine-tuning [44]. The language-specific component of mBERT can be removed by estimating the centroid of the language (the mean of mBERT embeddings of a given language vocabulary) and subtracting this centroid from representations produced in the language. The existence of the language component has been proven in multiple tasks: language classification, language similarity, sentence retrieval, word alignment, and machine translation.
Several works tested the cross-lingual abilities of mBERT as compared to monolingual models, finding its performance on low-resource languages inferior to monolingual models [91]. One experiment used a bilingual version of mBERT and trained it in multiple configurations to test the influence of the linguistic relationship between the source and the target language, the network architecture, and the input and learning objective [37]. The authors found that structural similarity and depth of a model are the most significant factors behind mBERT's cross-lingual performance. On the other hand, multi-head attention was not particularly important. Regarding the low-resource languages, [91] compared mBERT with baseline models on tasks of named entity recognition, universal part-of-speech tagging, and universal dependency parsing. The authors focused on the comparison between low- and high-resource languages (determined by the size of the Wikipedia dump in each language). mBERT was found to work well on high-resource languages but under-performed on low-resource
languages. Finally, Liu et al. [46] analyzed the relationship between the contextual aspect of mBERT and the training dataset size in the context of multilingual datasets. The authors compared contextual mBERT embeddings with non-contextual models of Word2Vec and GloVe. As it turns out, mBERT outperforms non-contextual models only when large datasets are available for training or fine-tuning. In the low data regime, as well as when the context window is short (limited input), multilingual mBERT is surpassed by static token embeddings.
5.2 Multilingual sentiment classification
Several surveys present an overview of traditional sentiment classification methods (lexicon-based approaches and shallow models with lexical features engineering) [25, 70]. Recently, deep-learning approaches became more prevalent. Attia et al. [6] use convolutional neural networks on word-level embeddings of texts in English, German and Arabic. The approach is computationally expensive as it requires separate embedding dictionaries for each language. An alternative approach is to use character-level embeddings. Wehrmann et al. [90] trained such a model for binary sentiment classification of English, German, Portuguese, and Spanish tweets. A similar model was used in [4], but the authors applied multi-stage pre-processing, including dictionary checks, Soundex algorithm, and lemmatization. Other examples of deep neural models for multilingual sentiment classification include recurrent neural networks supplied with machine translation. Can et al. [16] trained a model on English reviews and evaluated it on machine-translated reviews in Russian, Spanish, Turkish, and Dutch using the Google Translation API and pre-trained GloVe embeddings for English. A similar approach is presented in [38], where the authors used LASER sentence embeddings to train a sentiment classifier on Polish reviews and used this classifier to score reviews translated into other languages.
5.3 Multilingual sentiment datasets
Multilingual sentiment datasets are crucial resources for training sentiment analysis models that can operate across various languages. Unfortunately, very few multilingual datasets contain sentiment polarity annotations. The Multilingual Amazon Reviews Corpus [57] is a noteworthy example of such a dataset. It provides customer review data in English, Japanese, German, French, Spanish, and Chinese, annotated for sentiment. Another significant resource is the Multilingual Sentiment Lexicons developed by Chen and Skiena [18]), which provides sentiment lexicons for 136 languages. Being a lexicon, it cannot be used to train sentiment models directly, but it can be used for weak supervision. The NTCIR corpus contains information on sentiment polarity for news related to sport and politics for articles in English, Chinese, and Japanese [76]. The JMTC dataset [52](Jigsaw Multilingual Toxic Comment Classification) dataset contains comments from Wikipedia's talk page edits and is available in multiple languages. It is annotated for toxic vs. non-toxic sentiment, which cannot be aligned with our 3-class annotation schema. An interesting resource is Task 4 from SemEval providing English, Spanish, Dutch, Arabic, Russian, and Turkish tweets and SMS messages annotated for 3-class sentiment [54, 67]. The most similar dataset to ours is XED compiled by Ohman et al. [59]. However, this dataset contains original annotations only for English and Finnish, and the remaining 30 languages are "projected" (i.e., annotated samples in English and Finnish are translated into multiple languages).
The main focus of the paper is the introduction of a massively multilingual large collection of sentiment datasets and the extensive benchmark of model training and validation scenarios. During the construction of the benchmark, we have gathered valuable experiences that we want to share with the community. Our most important observation is that a single multilingual sentiment classifier can perform approximately equally well for all languages. A small group of pre-trained models (XML-R, LaBSE, MPNet) under fine-tuning scenario produces the best results relative to other models. Obviously, the absolute  $F_{1}$  score across languages differs significantly (due to data scarcity, data quality, or difficulty of samples), but these selected models always produce top classifiers. Secondly, all models perform better under the fine-tuning scenario, but the performance gain varies from model to model. When evaluated on the test dataset, models gained between 4 pp. (mUSE-dist) and 9 pp. (mBERT, DistilmBERT). When tested on the internal datasets,  $F_{1}$  gains varied between 0 pp (mUSE-
dist) and 20 pp. (DistilmBERT), with mT5 and XML-R improving by 17 pp. and 15 pp., respectively. In general, the largest gains can be observed for models trained with the masked language modeling technique (XML-R, mBERT). We also observe that models which produce lower-quality sentence embeddings gain more from fine-tuning.
We also find that bigger models tend to achieve better performance in all data modalities and training scenarios. Of course, there are counterexamples, e.g., mUSE-dist is smaller than mBERT but achieves better performance in the HL scenario for all datasets. This indicates that the size of the model is an important factor for determining its performance, but other factors, like the domain and the type of pretraining task, may also affect the results. Moreover, we observe that the correlation between model size and model performance is weaker after fine-tuning. This means that one may often find a competitive model with similar performance to the state-of-the-art model but significantly smaller and faster for the production environment.
Our final remark is a warning: in our experiments, we have encountered a significant variability in performance conditioned on a particular data split. In one of the cross-validation fine-tuning experiments, we observed a data fold that produced consistently worse results (up to 4 pp.) for all models. This fold was produced by the same random seed and (most probably) consisted of samples with increased difficulty. Since our experiments involved multiple models, multiple training scenarios, multiple datasets, and multiple languages, we had to rely on data subsampling. Conducting experiments on a fixed seed for sample selection could lead to a similar data fold and, consequently, to biased experimental results.
Despite the fact that our collection is the largest public collection of multilingual sentiment datasets, it still covers only 27 languages. The collection of datasets is highly biased towards the Indo-European family of languages, English in particular. We attribute this bias to the general culture of scientific publishing and its enforcement of English as the primary carrier of scientific discovery. Our work's main potential negative social impact is that the models developed and trained using the provided datasets may still exhibit better performance for the major languages. This could further perpetuate the existing language disparities and inequality in sentiment analysis capabilities across different languages. Addressing this limitation and working towards more equitable representation and performance across languages is crucial to avoid reinforcing language biases and the potential marginalization of underrepresented languages. The ethical implications of such disparities should be thoroughly discussed and considered.
An important limitation of our dataset collection is a significant variance in sample quality across all datasets and all languages. Figure 2 presents the distribution of self-confidence label-quality score for each data point computed by the cleanlab [58]. The distribution of quality is skewed in favor of popular languages, with low-resource languages suffering from data quality issues. A related limitation is caused by an unequal distribution of data modalities across languages. For instance, our benchmark clearly shows that all models universally underperform when tested on Portuguese datasets. This is the direct result of the fact that data points for Portuguese almost exclusively represent the domain of social media. As a consequence, some combinations of filtering facets in our dataset collection produce very little data (i.e., asking for social media data in the Germanic genus of Indo-European languages will produce a significantly larger dataset than asking for news data representing Afro-Asiatic languages).
Finally, we acknowledge the lack of internal coherence of annotation protocols between datasets and languages. We have enforced strict quality criteria and rejected all datasets published without the annotation protocol, but we were unable, for obvious reasons, to unify annotation guidelines. The annotation of sentiment expressions and the assignment of sentiment labels are heavily subjective and, at the same time, influenced by cultural and linguistic features. Unfortunately, it is possible that semantically similar utterances will be assigned conflicting labels if they come from different datasets or modalities.
The work presents results achieved in projects funded by (1) European Regional Development Fund (ERDF) in RPO WD 2014-2020 (project no. RPDS.01.02.01-02-0065/20); (2) the Polish Ministry of Education and Science, CLARIN-PL; (3) the ERDF in the 2014-2020 Smart Growth Operational Programme, CLARIN – Common Language Resources and Technology Infrastructure; (4) project CLARIN-Q (agreement no. 2022/WK/09); (5) the Department of Artificial Intelligence at Wroclaw University of Science and Technology. Mikołaj Morzy is supported by (6) EEA Financial Mechanism 2014-2021 Project 2019/35/J/HS6/03498.
[93] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.41. URL https://aclanthology.org/2021.naacl-main.41.
[94] Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5370–5378. International Joint Conferences on Artificial Intelligence Organization, 7 2019. doi: 10.24963/ijcai.2019/746. URL https://doi.org/10.24963/ijcai.2019/746.
[95] Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. Multilingual universal sentence encoder for semantic retrieval. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 87–94, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-demos.12. URL https://aclanthology.org/2020.acl-demos.12.
[96] Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. The united nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3530-3534, Portoorž, Slovenia, May 2016. European Language Resources Association (ELRA). URL https://www.aclweb.org/anthology/L16-1561.
[97] Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora. In Proceedings of the 10th Workshop on Building and Using Comparable Corpora, pages 60-67, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-2512. URL https://www.aclweb.org/anthology/W17-2512.
We present detailed lists of datasets included in our research in Tables 3 and 4. They include language, category, dataset size, class balance, and basic dataset characteristics.
Table 3: List of all monolingual datasets used in experiments. Category (Cat.): R - Reviews, SM - Social Media, C - Chats, N - News, P - Poems, M - Mixed. HL - human labeled, #Words and #Chars are mean values
Paper	Lang	Cat.	HL	Samples	NEG/NEU/POS	#Words	#Char.
Al Omari et al. [1]	ar	R	No	3096	13.0/10.2/76.8	9	51
Elnagar et al. [29]	ar	R	No	400101	13.0/19.9/67.1	22	127
Aly and Atiya [2]	ar	R	No	6250	11.6/17.9/70.5	65	343
Elnagar and Einea [28]	ar	R	No	504007	15.4/21.0/63.6	77	424
Baly et al. [7]	ar	SM	Yes	2809	47.2/23.9/29.0	22	130
Nabil et al. [53]	ar	SM	Yes	3224	50.9/25.0/24.1	16	94
Salameh et al. [71]	ar	SM	Yes	1199	48.0/10.5/41.5	11	51
Salameh et al. [71]	ar	SM	Yes	1998	67.5/10.1/22.4	20	107
Habernal et al. [32]	cs	R	No	91140	32.4/33.7/33.9	50	311
Habernal et al. [32]	cs	R	No	92758	7.9/23.4/68.7	20	131
Habernal et al. [32]	cs	SM	Yes	9752	20.4/53.1/26.5	10	59
Habernal et al. [32]	cs	SM	Yes	2637	30.8/60.6/8.6	33	170
Cieliebak et al. [19]	de	SM	Yes	9948	16.3/59.2/24.6	11	86
Schabus and Skowron [75]	de	SM	Yes	3598	47.3/51.5/1.2	33	237
Chapuis et al. [17]	en	C	Yes	12138	31.8/46.5/21.7	12	48
Chapuis et al. [17]	en	C	Yes	4643	22.3/48.9/28.8	15	71
Malo et al. [48]	en	N	Yes	3448	12.2/62.1/25.7	22	124
Bastan et al. [10]	en	N	Yes	5333	11.6/37.3/51.0	388	2129
Hutto and Gilbert [35]	en	N	No	5190	29.3/52.9/17.8	17	104
Sheng and Uthus [78]	en	P	Yes	1052	18.3/15.8/65.9	7	37
Hutto and Gilbert [35]	en	R	No	3708	34.2/19.5/46.3	16	87
Hutto and Gilbert [35]	en	R	No	10605	49.6/1.5/48.9	19	111
Ni et al. [56]	en	R	No	1883238	8.3/8.0/83.7	70	382
Sanders [72]	en	SM	Yes	3424	16.7/68.1/15.2	14	97
Thelwall et al. [84]	en	SM	Yes	11759	28.0/34.0/38.0	26	147
Inc. [36]	en	SM	Yes	14427	63.0/21.2/15.8	17	104
Hutto and Gilbert [35]	en	SM	No	4200	26.9/17.0/56.1	13	79
Keith Norambuena et al. [39]	es	R	Yes	382	45.0/27.2/27.8	165	1033
Cruz et al. [23]	es	R	No	3871	32.9/32.3/34.9	511	3000
Hosseini et al. [33]	fa	R	Yes	13525	12.0/37.5/50.5	21	104
Amram et al. [3]	he	SM	Yes	8619	26.5/2.8/70.8	22	110
Pelicon et al. [63]	hr	N	Yes	2025	22.5/61.4/16.0	161	1021
Barbieri et al. [8]	it	SM	Yes	8926	36.7/41.7/21.6	14	101
Sprugnoli [82]	it	SM	Yes	3139	24.4/14.9/60.6	17	106
Sprogis and Rikters [81]	lv	SM	Yes	5790	23.8/45.2/31.0	20	138
Rybak et al. [69]	pl	R	No	10074	30.8/13.2/56.0	80	494
Kocoń et al. [41]	pl	R	Yes	57038	42.4/26.8/30.8	30	175
Sobkowicz and Sobkowicz [80]	pl	SM	Yes	645	50.7/47.3/2.0	33	230
Brum and Volpe Nunes [13]	pt	SM	Yes	10109	28.8/25.1/46.1	12	74
Rogers et al. [65]	ru	SM	Yes	23226	16.8/54.6/28.6	12	79
Bučar et al. [14]	sl	N	Yes	10417	32.0/52.0/16.0	309	2017
Batanović et al. [11]	sr	R	No	4724	17.8/43.7/38.5	498	3097
Batanović et al. [12]	sr	R	Yes	3948	30.3/18.1/51.5	18	105
Thongthanomkul et al. [86]	th	R	No	46193	5.4/30.5/64.1	29	544
Suriyawongkul et al. [83]	th	SM	Yes	26126	26.1/55.6/18.3	6	90
Sharf and Rahman [77]	ur	M	Yes	19660	26.7/43.6/29.7	13	69
Lin et al. [45]	zh	R	No	125725	28.6/21.9/49.5	51	128
Table 4: List of all multilingual datasets used in experiments. Category (Cat.): R - Reviews, SM - Social Media, C - Chats, N - News, P - Poems, M - Mixed. HL - human labeled
Paper	Cat.	Lang	HL	Samples	(NEG/NEU/POS)	#Words	#Char.
Narr et al. [55]	SM	de	Yes	953	10.0/75.1/14.9	12	80
de	Yes	1781	16.9/63.3/19.8	13	81
en	Yes	7073	17.4/60.0/22.6	14	78
fr	Yes	685	23.4/53.4/23.2	14	82
fr	Yes	1786	25.0/54.3/20.8	15	83
pt	Yes	759	28.1/33.2/38.7	14	78
pt	Yes	1769	30.7/33.9/35.4	14	78
Keung et al. [40]	R	de	No	209073	40.1/20.0/39.9	33	208
en	No	209393	40.0/20.0/40.0	34	179
es	No	208127	40.2/20.0/39.8	27	152
fr	No	208160	40.2/20.1/39.7	28	160
ja	No	209780	40.0/20.0/40.0	2	101
zh	No	205977	39.8/20.1/40.1	1	50
Rosenthal et al. [66]	M	ar	Yes	9391	35.5/40.6/23.9	14	105
en	Yes	65071	19.1/45.7/35.2	18	111
Patwa et al. [60]	SM	es	Yes	14920	16.8/33.1/50.0	16	86
hi	Yes	16999	29.4/37.6/33.0	27	128
Mozetič et al. [51]	SM	bg	Yes	62150	22.6/45.9/31.5	12	85
bs	Yes	36183	33.4/30.5/36.1	12	75
de	Yes	90534	19.7/52.8/27.4	12	94
en	Yes	85784	26.8/44.1/29.1	12	77
es	Yes	191412	11.8/37.9/50.3	14	92
hr	Yes	75569	25.7/23.9/50.4	12	91
hu	Yes	56682	15.9/31.0/53.1	11	83
pl	Yes	168931	30.0/26.1/43.9	11	82
pt	Yes	145197	37.2/35.0/27.8	10	61
ru	Yes	87704	32.0/40.1/27.8	10	67
sk	Yes	56623	25.6/22.5/51.9	13	97
sl	Yes	103126	29.9/43.3/26.8	13	91
sq	Yes	44284	15.7/33.1/51.1	13	90
sr	Yes	67696	34.8/42.8/22.4	13	81
sv	Yes	41346	40.3/31.2/28.5	14	94
B.1 Data quality
In this section, we expand upon the results of experiments included in the benchmark. We begin with the presentation of the distribution of data point quality across all datasets. The data point quality has been computed using the ccleanlab library self-confidence label-quality score [58]. Figure 2 presents the cumulative distribution of data quality score. As we can see, the data quality still is a pressing issue, despite our efforts to publish a well-curated collection. Over  $40\%$  of all data points have quality 0.6 or worse. As we have mentioned, low-quality data points are not distributed uniformly across all languages and datasets, but the impact of low-quality data is most pronounced for low-resource languages.
B.2 Pairwise ranked comparison of models
Of course, everyone is most interested in finding the answer to the following question: "Which model should I use?" The answer, as usual, is: "It depends". Figure 3 presents the results of the Nemenyi pairwise rank comparison test. In neither of the considered training scenarios (HL, HB, TF), a single model is statistically better than others. However, these results allow us to draw partial conclusions. For instance, the MPNet performs best in the HL scenario, and the same model is not significantly worse than XML-M, which is the best model in HB and TF scenarios. We can also notice that mBERT-based models (mBERT and DistillmBERT) proved to be the worst language models for our
Figure 2: Cumulative distribution of sample quality across all datasets
(a) HL scenario
(b) HB scenario
(c) FT scenario
Figure 3: Nemenyi diagrams of model ranks according to the F1-score on each dataset
tasks. Three models stand out as the most promising choices: XLM-R, LaBSE, and MPNet. They achieve comparable performance in all scenarios and test cases. Furthermore, they are better than other models in almost all test cases.
Table 5: Aggregated  $F_{1}$  results of models. The best results for each test set are highlighted. W - whole test, A - avg. by dataset, I - internal
XLM-R	LaBSE	MPNet	XLM-R-dist	mT5	mBERT	DistilmbERT	mUSE-dist	LASER	mUSE-trans.	mUSE-cnn
HL scenario
W	62	62	63	60	59	56	55	59	55	55	54
A	51	54	55	51	49	45	43	50	47	47	45
I	55	61	61	56	50	43	38	60	50	49	50
HB scenario
W	66	62	63	62	65	60	59	62	-	-	-
A	57	55	56	54	56	49	48	54	-	-	-
I	64	63	64	63	63	54	48	64	-	-	-
FT scenario
W	68	68	67	67	66	65	64	63	-	-	-
A	61	62	62	62	60	56	56	56	-	-	-
I	70	69	65	67	67	57	58	60	-	-	-
B.3 Detailed results for each model
Table 5 shows the  $F_{1}$  scores of all models aggregated by datasets. The models improve with fine-tuning (up to 0.7  $F_{1}$ ) as compared to linear (0.61) or BiLSTM (0.64) layers operating on embeddings produced by models. The performance gains are more pronounced for models trained with masked language modeling objectives (mBERT, XML-R) than for models trained with sentence classification or sentence similarity objectives (LaBSE). Fine-tuning reduces inequalities in the results between models (0.55 vs. 0.43 for best and worst models in the HL scenario). The additional BiLSTM layer on top of transformer-based token embeddings has the capacity to improve the results compared to the linear classification layer model. The differences are most clear in the case of the results for our internal dataset, where the result improved even by 13 pp. for the mT5 model.
B.4 Detailed results for each language
We assessed the performance of each model in each experimental scenario per language. The texts were sub-sampled with stratification by language and class label so that language distribution in the test dataset follows the distribution in the whole dataset. We also include the total macro  $F_{1}$  score value in column "all". Results are presented in Figure 4. Those results confirm our previous findings regarding the advantage of XLM-R, LaBSE, and MPNet models. They outperform other models in most languages, with no clear indication of superiority among them.
Figure 4: Detailed results of models' comparison. Legend: lang - averaged by all languages, ds - averaged by dataset, ar - Arabic, bg - Bulgarian, bs - Bosnian, cs - Czech, de - German, en - English, es - Spanish, fa - Persian, fr - French, he - Hebrew, hi - Hindi, hr - Croatian, hu - Hungarian, it - Italian, ja - Japanese, lv - Latvian, pl - Polish, pt - Portuguese, ru - Russian, sk - Slovak, sl - Slovenian, sq - Albanian, sr - Serbian, sv - Swedish, th - Thai, ur - Urdu, zh - Chinese.
B.5 Transfer between data modalities
The last example compares the effectiveness of transfer learning between data modalities. The results are presented in Figure 5. As expected, when models are tested against datasets from the same domain (news, social media, reviews), the average performance is much higher than on out-of-domain datasets. What is striking is the visible deterioration of the performance of models trained in the news domain. When knowledge transfer happens between social media and review domains, the average performance of models stays relatively the same.
Figure 5: Transfer learning between data modalities
C Listings