SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Łukasz Augustyniak
WUST (Wroclaw University of Science and Technology)
lukasz.augustyniak@pwr.edu.pl
Szymon Wozniak
Brand24 AI
Marcin Gruza
Brand24 AI, WUST
Piotr Gramacki
Krzysztof Rajda
Mikołaj Morzy
Tomasz Kajdanowicz
WUST
Despite impressive advancements in multilingual corpora collection and model training, developing large-scale deployments of multilingual models still presents a significant challenge. This is particularly true for language tasks that are culture-dependent. One such example is the area of multilingual sentiment analysis, where affective markers can be subtle and deeply ensconced in culture. This work presents the most extensive open massively multilingual corpus of datasets for training sentiment models. The corpus consists of 79 manually selected datasets from over 350 datasets reported in the scientific literature based on strict quality criteria. The corpus covers 27 languages representing 6 language families. Datasets can be queried using several linguistic and functional features. In addition, we present a multi-faceted sentiment classification benchmark summarizing hundreds of experiments conducted on different base models, training objectives, dataset collections, and fine-tuning strategies.
Consider a hotel booking service that allows its customers to post reviews. You have found just the perfect accommodation to stay for a couple of days with your family, but you browse through the reviews section of the website to check the experiences of former guests. Suddenly, you encounter a review in Polish: "hotel jak hotel, mogło być gorzej." This review has the following sentiment scores<sup>1</sup>:  $s_{neg} = 0.44$ ,  $s_{neu} = 0.44$ ,  $s_{pos} = 0.12$ . Intrigued by the ambiguity of scores, you translate the review into English: "hotel like a hotel, all in all, it could have been worse," which is scored as  $s_{neg} = 0.80$ ,  $s_{neu} = 0.16$ ,  $s_{pos} = 0.04$ . Apparently, the stereotypically pessimistic Polish outlook on life gets lost in translation. The next review is written in Czech: "ok, ale nic zajímavěho" with scores  $s_{neg} = 0.32$ ,  $s_{neu} = 0.54$ ,  $s_{pos} = 0.14$ . After translating into English ("ok, but nothing interesting") the sentiment classification model scores the review as negative ( $s_{neg} = 0.50$ ,  $s_{neu} = 0.37$ ,  $s_{pos} = 0.13$ ). After your stay, you decide to add a very positive review of the hotel ("it was a killer place to stay",  $s_{neg} = 0.03$ ,  $s_{neu} = 0.05$ ,  $s_{pos} = 0.92$ ). You would be very surprised to learn that the Czechs would be quite confused about your opinion ("to bye lo vrazedné
Sentiment scores in this paragraph are produced by the multilingual cardiffnlp/twitter-xlm-roberta-base-sentiment model [9]
mistro  $k$  pobytu",  $s_{\text{neg}} = 0.51$ ,  $s_{\text{neu}} = 0.09$ ,  $s_{\text{pos}} = 0.39$ ), while the Poles would stay away from the hotel at all costs ("to byto zapójcze.), "to be yo to zabojece.), "to be yo to zabojece na pobyt",  $s_{\text{neg}} = 0.78$ ,  $s_{\text{neu}} = 0.09$ ,  $s_{\text{pos}} = 0.13$ ).
multilingual services become ubiquitous in the modern global economy. As more websites begin to offer automatic translation of content, users do not bother to express themselves in the lingua franca of the Web, writing instead in their native languages. Despite impressive advancements in automatic translation, many NLP tasks are still difficult in the multilingual setting. And sentiment classification is among the most challenging. The expression of sentiment is highly culture-dependent [31]. The emotional valence of individual words, the presence of specific phrasemes, and the expectations around sentiment values make sentiment classification across languages a demanding task.
Models performing sentiment classifications have to cope with two independent sources of variance in the input data: cultural expressions of sentiment and errors in automatic translations. In addition, the productization of sentiment classification leads to several engineering choices which influence the efficiency of the model:
- single multilingual model vs. dedicated monolingual models: deploying a single model in a production environment is much easier than orchestrating an ensemble of models,
- training vs. fine-tuning: sentiment classification model can be trained from scratch, both in the multi- and monolingual regimes, and the alternative is the fine-tuning of a general-purpose pre-trained language model with sentiment data,
- transfer learning between domains: a sentiment classification model trained for a specific domain (e.g., book reviews) can be transferred to another domain (e.g., hotel reviews) under the assumption that sentiment expressions in a given language remain independent of the subject of sentiment,
- transfer learning between languages: one may hypothesize that related languages can utilize similar sentiment expression mechanisms regarding grammar, punctuation, and vocabulary. In theory, it is possible to use training data in language  $L_{1}$ , fine-tune a multilingual model with this data, and as a result of fine-tuning, improve the performance of the model on language  $L_{2}$ , provided languages  $L_{1}$  and  $L_{2}$  are sufficiently similar.
Working with multilingual datasets and models opens another opportunity. Most NLP research focuses on 20 major languages. Many languages native to significant human populations are not adequately studied. Although the term low-resource language is not precisely defined and can be understood in terms of computerization, privilege, the abundance of resources, or density [47], the existence of a large chasm between languages with respect to linguistic resources is apparent. Our work aims at supporting low-resource languages in performant sentiment classification. Finally, there is a lively debate in the scientific community about the inherent ability of neural models to handle general linguistic phenomena. This paper tries to answer this question in the domain of sentiment classification.
Contribution presented in this paper is threefold:
- multilingual corpus: we present the largest multilingual collection of sentiment classification datasets consisting of 79 high-quality datasets covering 27 languages. The collection of the corpus and detailed descriptions of collected datasets are presented in Section 3.
- multi-faceted benchmark: the dataset is supplemented with a benchmark containing detailed run statistics of hundreds of experiments representing different training and testing scenarios. All details relevant to the benchmark are presented in Section 4.
- library for dataset access: all datasets in the multilingual corpus are publicly available via a library compatible with the HuggingFace library, along with the ability to filter and select datasets, verify their licenses, etc $^2$ .
2 Linguistic typology
The similarity of languages and the main aspects of their differences is the field of study of language typology. The differences between languages can be phonological (differences in sounds used
$^{2}$ https://huggingface.co/datasets/Brand24/mms
by languages), syntactic (differences in language structures), lexical (differences in vocabulary), and theoretical (differences characterized as general properties of languages). Linguistic typology analyzes the current state of languages and is often contrasted with genealogical linguistics. The latter is concerned with historical relationships between languages established via historical records or with the help of comparative linguistics. The main focus of genealogical linguistics resolves around language families and language genera. The term language family refers to a group of languages sharing pedigree from a common ancestral language (the proto-language). As of today, linguists define over 7000 languages categorized into 150 families [15]. The largest families of languages include Indo-European, Sino-Tibetan, Turkic, Afro-Asiatic, Nilo-Saharan, Niger-Congo, and Eskimo-Aleut [24]. Main families are further divided into branches called genera. Examples of genera within the Indo-European family of languages include Slavic, Romance, Germanic, and Indic. This division of language families into genera closely follows the genetic family of humankind as attested by DNA similarity [43]. Some language families do not produce distinctive genera but form dialect continua defined by mutual intelligibility. Finally, an important concept is that of a sprachbund, a geographical area occupied by languages sharing linguistic features. The linguistic similarities within a sprachbund result from cultural exchange and geographical contact rather than by chance or common origin, as described by Thomason and Kaufman [85].
Languages can be described using hundreds of linguistic features. World Atlas of Language Structures [27] lists almost 200 different features. Since our work focuses on sentiment classification, we select 10 features that seem to be the most relevant to the task of sentiment expression. These features are:
1. definite article: the morpheme associated with nouns and used to code the uniqueness or definiteness of a concept; almost half of the languages do not use the definite article.
2. indefinite article: the morpheme used together with nouns to signal that the related concept is unknown to the hearer; half of the languages do not use the indefinite article, some languages use a separate article, and some use the numeral "one" as the indefinite article.
3. number of cases: morphological cases are a common way to express various relationships between words; human languages vary greatly in the number of cases used by a language.
4. order of subject, verb, and object: some languages have the strict ordering of words, while languages that convey semantics through inflection may be much looser with the ordering; half of all languages use the SOV (subject-object-verb) ordering, one third uses the SVO (subject-verb-object) ordering, and a small fraction of languages use the remaining VSO, VOS, OVS, and OSV orderings. Interestingly, around  $13\%$  of world languages do not have any fixed word order.
5. negative morphemes: negative morpheme is used to signal clausal negation in declarative sentences; this is usually achieved using a negative affix or a negative particle.
6. polar questions: a polar question is a question with only yes/no answers; these questions can be built using question particles, interrogative morphology, or intonation only.
7. position of the negative morpheme: languages differ by the position of the negative morpheme in relation to subjects and objects, with many variants such as SNegVO, NegSVO, SVNegO, obligatory and optional double negations, etc.
8. prefixing vs. suffixing: languages differ significantly in their use of prefixes versus suffixes in inflectional morphology.
9. coding of nominal plurals: two major types of plural coding are present in languages, either by changing the morphological form of the noun or by using a plurality indicator morpheme somewhere in the noun phrase.
10. grammatical genders: there is significant variability among languages with respect to the number of grammatical genders, some languages do not use the concept at all, and some languages may have 5 or more grammatical genders.
All language features mentioned above are available as filtering features in our library. Thus, when training a sentiment classifier using our dataset, one may download different facets of the collection. For instance, one can download all datasets in Slavic languages in which polar questions are formed using the interrogative word order (Listing 1) or download all datasets from the Afro-Asiatic language family with no morphological case-making (Listing 2).
3.1 Quality criteria
The initial pool of sentiment datasets has been gathered using extensive search and consisted of 345 datasets found through Google Scholar, GitHub repositories, and the HuggingFace datasets library. This initial pool of datasets has been manually filtered based on the following set of quality assurance criteria:
1. strong annotations: we have rejected datasets containing weak annotations (e.g., datasets with labels based on emoji occurrence or generated automatically through classification by machine learning models) due to an extensive amount of noise [58].
2. well-defined annotation protocol: we have rejected datasets without sufficient information about the annotation protocol (e.g., whether annotation was manual or automatic, number of annotators) to avoid merging datasets with contradicting annotation instructions.
3. numerical ratings: we have accepted datasets with numerical ratings, mapping Likert-type 5-point scales into three class sentiment labels as follows: ratings 1 and 2 were mapped to negative, rating 3 was mapped to neutral, and ratings 4 and 5 were mapped to positive.
4. three classes only: we have rejected datasets annotated with binary sentiment labels as their performance in three class settings was unsatisfactory.
5. monolingual datasets: when a dataset contained samples in multiple languages, we opted to divide it into independent datasets in constituent languages.
3.2 Pre-processing of datasets
Despite quality assurance criteria described in Section 3.1, the datasets still contained conflicting entries, i.e., duplicated records with different sentiment labels. We have cross-referenced all datasets to identify conflicts and have made the data coherent using majority-label voting. Finally, labeling and rating schemes of all datasets have been mapped to a 3-class scheme with negative, neutral, and positive labels only. For datasets with emotional annotations, we mapped positive emotions (joy, happiness) into positive sentiment and negative emotions (fear, sadness, disgust, anger) into negative sentiment. Texts with ambiguous emotions like anticipation and surprise were discarded. This preprocessing pipeline resulted in 79 datasets containing 6164762 text samples. Most of the datasets are in English (2330486 samples across 17 datasets), Arabic (932075 samples across 9 datasets), and Spanish (418712 samples across 5 datasets). The datasets represent four different domains: social media (44 datasets), reviews (24 datasets), news (5 datasets), and others (6 datasets). In addition, all datasets were processed by the cleanlab library to produce a self-confidence label-quality score for each data point.
Exhaustive testing of all configurations of the benchmark is unfeasible, but the total lack of baseline is unacceptable, either. We have decided to test the benchmark dataset by manually compiling a strong baseline. The main rationale behind this effort was the lack of coherence in annotation guidelines between considered datasets. Our baseline is built using strict annotation guidelines constructed iteratively over annotation batches, resulting in highly coherent annotations. The baseline dataset consists of 3911 short text samples (trimmed to 350 characters) in Polish and English, annotated independently by 3 annotators fluent in these languages. Baseline texts cover multiple domains, such as social media, news sites, blogs, and Internet forums. For each instance, the majority label assigned by the annotators has been stored. The inter-rater agreement is  $\kappa = 0.665$  (average Cohen's kappa between three pairs of annotators) and  $\alpha = 0.666$  (Krippendorff's alpha).
One may think of the baseline dataset as representative of current sentiment model training, where researchers have to build domain-aligned corpora of various languages and annotate them. The comparison of results between the benchmark and the baseline datasets is a proxy of the performance trade-offs should one decide to use our benchmark dataset to train domain and language-specific sentiment classifiers.
Table 1: Summary of the corpus. Categories: N - news, R - reviews, SM - social media, O - other
#datasets	category	#samples	mean
N	R	SM	O	NEG	NEU	POS	#words	#chars
English	17	3	4	6	4	304,939	290,823	1,734,724	62	339
Arabic	9	0	4	4	1	138,899	192,774	600,402	52	289
Spanish	5	0	3	2	0	108,733	122,493	187,486	26	150
Chinese	2	0	2	0	0	117,967	69,016	144,719	60	80
German	6	0	1	5	0	104,667	100,071	111,149	26	171
Polish	4	0	2	2	0	77,422	62,074	97,192	19	123
French	3	0	1	2	0	84,187	43,245	83,199	28	159
Japanese	1	0	1	0	0	83,982	41,979	83,819	61	101
Czech	4	0	2	2	0	39,674	59,200	97,413	34	212
Portuguese	4	0	0	4	0	56,827	55,165	45,842	11	63
Slovenian	2	1	0	1	0	33,694	50,553	29,296	41	269
Russian	2	0	0	2	0	31,770	48,106	31,054	11	70
Croatian	2	1	0	1	0	19,757	19,470	38,367	17	116
Serbian	3	0	2	1	0	25,089	32,283	18,996	44	269
Thai	2	0	1	1	0	9,326	28,616	34,377	22	381
Bulgarian	1	0	0	1	0	13,930	28,657	19,563	12	86
Hungarian	1	0	0	1	0	8,974	17,621	30,087	11	83
Slovak	1	0	0	1	0	14,431	12,842	29,350	13	98
Albanian	1	0	0	1	0	6,889	14,757	22,638	13	91
Swedish	1	0	0	1	0	16,266	13,342	11,738	14	94
Bosnian	1	0	0	1	0	11,974	11,145	13,064	12	76
Urdu	1	0	0	0	1	5,239	8,585	5,836	13	69
Hindi	1	0	0	1	0	4,992	6,392	5,615	26	128
Persian	1	0	1	0	0	1,602	5,091	6,832	21	104
Italian	2	0	0	2	0	4,043	4,193	3,829	16	103
Hebrew	1	0	0	1	0	2,279	243	6,097	22	110
Latvian	1	0	0	1	0	1,378	2,618	1,794	20	138
4 Multi-faceted benchmark
As we have explained in Section 1, the deployment of a multilingual sentiment classifier can be evaluated using several criteria leading to different architecture choices. Thus, we do not publish a single benchmark, but we aggregate the results along several dimensions. The benchmark available at https://huggingface.co/spaces/Brand24/mms_benchmark, allows to compare models according to:
number of languages: multilingual models with monolingual models,
- training procedure: models trained from scratch vs. fine-tuning,
- domain language: language to which a model is applied,
- data modality: news, reviews, social media,
- knowledge transfer: transfer between languages, transfer between domains.
Table 2 presents the list of models included in the benchmark. For each model, we include the number of parameters and languages used in pre-training and the base model. The results presented in the benchmark reflect three possible scenarios of model deployment. A pre-trained model can be used to generate text representation only. In this scenario, denoted head-linear (HL), a model serves as a feature extractor followed by a small linear classification head. In the second scenario, the linear classification head is replaced by a BiLSTM classifier operating on features extracted by the pre-trained model. We refer to this scenario as head-bilstm (HB). Finally, each pre-trained transformer-based model (with the exception of mUSE-transformer) has been fine-tuned to the sentiment classification task. We refer to this scenario as fine-tuning (FT).
Table 2: Models included in the benchmark
Model	#params	#langs	base	reference
mT5	277M	101	T5	Xue et al. [93]
LASER	52M	93	BiLSTM	Artetxe and Schwenk [5]
mBERT	177M	104	BERT	Devlin et al. [26]
MPNet	278M	53	XLM-R	Reimers and Gurevych [64]
XLM-R-dist	278M	53	XLM-R	Reimers and Gurevych [64]
XLM-R	278M	100	XLM-R	Conneau et al. [22]
LaBSE	470M	109	BERT	Feng et al. [30]
DistilmBERT	134M	104	BERT	Sanh et al. [73]
mUSE-dist	134M	53	DistilmBERT	Reimers and Gurevych [64]
mUSE-transformer	85M	16	transformer	Yang et al. [95]
mUSE-cnn	68M	16	CNN	Yang et al. [95]
Fine-tuning - F1 score per language
Figure 1: Detailed comparison of models. Legend: lang - averaged by all languages, ds - averaged by dataset, ar - Arabic, bg - Bulgarian, bs - Bosnian, cs - Czech, de - German, en - English, es - Spanish, fa - Persian, fr - French, he - Hebrew, hi - Hindi, hr - Croatian, hu - Hungarian, it - Italian, ja - Japanese, lv - Latvian, pl - Polish, pt - Portuguese, ru - Russian, sk - Slovak, sl - Slovenian, sq - Albanian, sr - Serbian, sv - Swedish, th - Thai, ur - Urdu, zh - Chinese.
The hyperparameters of models included in the benchmark are as follows. The hidden size of the model was set to the embedding size of each model when used in HL and FT scenarios. The hidden size of the HB scenario was set to  $h = 32$ . The learning rate for the HL scenario was  $\eta = 1 \times 10^{-3}$ , fine-tuning used  $\eta = 1 \times 10^{-5}$ , and HB scenario used  $\eta = 5 \times 10^{-3}$ . The batch size in HL and HB scenarios was  $b = 200$  and  $b = 6$  for fine-tuning. Dropout for HB was  $d = 0.5$  and  $d = 0.2$  for other scenarios. Training took 5 epochs for fine-tuning and 2 epochs for HL and HB scenarios (beyond 2 epochs most models started to overfit). The models are evaluated using the traditional  $F_{1}$  score computed on three levels: the entire dataset, averaged over all datasets, and the internal dataset.
We performed our experiments using Python 3.9 and PyTorch (1.8.1) (and Tensorflow (2.3.0) for original mUSE). Our experimental setup consists of Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz and Nvidia Tesla V100 16GB.
Here we present one result available through the benchmarks. Due to space constraints, we move the presentation of other case studies to Appendix B. Figure 1 contains  $F_{1}$  scores of all models fine-tuned on all datasets available for a given language. Interestingly, we see little variance in performance between models for high-resource languages and significant deterioration of performance for low-resource languages. An outlier is the aforementioned Portuguese, where the performance is caused by the lack of data points representing news and reviews. Figure 1 directly shows the usefulness of our benchmark. When designing sentiment classification solutions in Spanish, the choice of model is secondary (models' performances are very similar), and other model features can be considered (such as ease of deployment, cost of inference, and memory requirements). However, if sentiment classification is to be applied to Latvian, the performance difference between models can be as high as 32 pp depending on the choice of the model.
5.1 Multilingual text representations
One of the foundational discoveries in multilingual presentation learning was the fact that latent vector spaces seemed to encode very similar word relationships across a wide spectrum of languages. Needless to say, first monolingual static word embeddings were quickly followed by multilingual word embeddings which provided the basis for multilingual text representations [68]. A popular solution was to use pre-trained monolingual embeddings, such as word2vec [49] and align them via linear transformations using parallel multilingual dictionaries [50]. Another approach advocated for joint learning of multilingual word embeddings on pseudo-bilingual datasets, where tokens in one language would be randomly translated to another language [92].
Static multilingual embeddings were superseded by contextual embeddings produced by language models such as BiLSTM [5] and various Transformers [30, 22, 26, 93, 95]. In the case of these more complex architectures, the multilingual capabilities of language models resulted from specific objective functions used during training. These objectives "pushed" the models toward universal multilingual representations by forcing the models to perform machine translation [5], translation language modeling (TLM) [22, 20], or translation ranking [30, 94].
The evaluation of the quality of multilingual text representation is not trivial. Cross-lingual and multilingual tasks are actively developed to foster the development of multilingual models. Examples of such cross-lingual and multilingual tasks include cross-lingual natural language inference [21], question answering [42], named entity recognition [87, 88] or parallel text extraction [97, 96]. For example, results of comparison of LASER, mBERT, and XLM in tasks of named entity recognition and part-of-speech tagging in zero-shot settings suggest that LASER outperforms the latter methods in the case of knowledge transfer [89]. Another important benchmark is XTREME [34], designed for testing the abilities of cross-lingual transfer across 40 languages and 9 tasks. Despite its massive character, XTREME lacks benchmarking task of sentiment analysis. Also, only mBERT, XLM, XLM-R, and MMTE are used as baseline models.