BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages
Authors: Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nirmal Surange, Daniela Teodorescu, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino D. M. A. Ali, Ilseyar Alimova, Vladimir Araujo, Nikolay Babakov, Naomi Baes, Ana-Maria Bucur, Andiswa Bukula, Guanqun Cao, Rodrigo Tufiño, Rendi Chevi, Chiamaka Ijeoma Chukwuneke, Alexandra Ciobotaru, Daryna Dementieva, Murja Sani Gadanya, Robert Geislinger, Bela Gipp, Oumaima Hourrane, Oana Ignat, Falalu Ibrahim Lawan, Rooweither Mabuya, Rahmad Mahendra, Vukosi Marivate, Andrew Piper, Alexander Panchenko, Charles Henrique Porto Ferreira, Vitaly Protasov, Samuel Rutunda, Manish Shrivastava, Aura Cristina Udrea, Lilian Diana Awuor Wanzare, Sophie Wu, Florian Valentin Wunderlich, Hanif Muhammad Zhafran, Tianhui Zhang, Yi Zhou, Saif M. Mohammad
Accepted at: acl25

BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages

Shamsuddeen Hassan Muhammad1,2*, Nedjma Ousidhoum3*, Idris Abdulmumin4, Jan Philip Wahle5, Terry Ruas5, Meriem Beloucif6, Christine de Kock7, Nirmal Surange8, Daniela Teodorescu9, Ibrahim Said Ahmad10, David Ifeoluwa Adelani11,12,13, Alham Fikri Aji14, Felermino D. M. A. Ali15, Ilseyar Alimova31, Vladimir Araujo16, Nikolay Babakov17, Naomi Baes7, Ana-Maria Bucur18,19, Andiswa Bukula20, Guanqun Cao21, Rodrigo Tufiño22, Rendi Chevi14, Chiamaka Ijeoma Chukwuneke23, Alexandra Ciobotaru18, Daryna Dementieva24, Murja Sani Gadanya2, Robert Geislinger25, Bela Gipp5, Oumaima Hourrane26, Oana Ignat27, Falalu Ibrahim Lawan28, Roowether Mabuya20, Rahmad Mahendra29, Vukosi Marivate4,30, Alexander Panchenko31,32, Andrew Piper12, Charles Henrique Porto Ferreira33, Vitaly Protasov32, Samuel Rutunda34, Manish Shrivastava8, Aura Cristina Udrea35, Lilian Diana Awuor Wanzare36, Sophie Wu12, Florian Valentin Wunderlich5, Hanif Muhammad Zhafran37, Tianhui Zhang38, Yi Zhou3, Saif M. Mohammad39

1Imperial College London, 2Bayero University Kano, 3Cardiff University,
4Data Science for Social Impact, University of Pretoria, 5University of Göttingen, 6Uppsala University,
7University of Melbourne, 8IIIT Hyderabad, 9University of Alberta, 10Northeastern University, 11MILA, 12McGill University,
13Canada CIFAR AI Chair, 14MBZUAI, 15LIACC, FEUP, University of Porto, 16Saiplane AI,
17University of Santiago de Compostela, 18University of Bucharest, 19Universitat Politècnica de València, 20SADiLaR,
21University of York, 22Universidad Politécnica Salesiana, 23Lancaster University, 24Technical University of Munich,
25Hamburg University, 26Al Akhawayn University, 27Santa Clara University, 28Kaduna State University, 29Universitas Indonesia,
30Lelapa AI, 31Skoltech, 32AIRI, 33Centro Universitário FEI, 34Digital Umuganda,
35National University of Science and Technology Politehnica Bucharest, 36Maseno University, 37Institut Teknologi Bandung,
38University of Liverpool, 39National Research Council Canada

Contact: s.mohammad@imperial.ac.uk, OusidhoumN@cardiff.ac.uk

Abstract

People worldwide use language in subtle and complex ways to express emotions. Although emotion recognition—an umbrella term for several NLP tasks—impacts various applications within NLP and beyond, most work in this area has focused on high-resource languages. This has led to significant disparities in research efforts and proposed solutions, particularly for under-resourced languages, which often lack high-quality annotated datasets. In this paper, we present BRIGHTER—a collection of multi-labeled, emotion-annotated datasets in 28 different languages and across several domains. BRIGHTER primarily covers low-resource languages from Africa, Asia, Eastern Europe, and Latin America, with instances labeled by fluent speakers. We highlight the challenges related to the data collection and annotation processes, and then report experimental results for monolingual and crosslingual multi-label emotion identification, as well as emotion intensity recognition. We analyse the variability in performance across languages and text domains, both with and without the use of LLMs, and show that the BRIGHTER datasets represent a meaningful step towards addressing the gap in text-based emotion recognition.

![Languages included in BRIGHTER and their language families.](page_1012_1342_377_246.png)

Figure 1: Languages included in BRIGHTER and their language families.
Figure 2: Examples from the BRIGHTER dataset collection in 6 different languages with their translations and intensity levels. Note that the instances can have one or more labels (e.g., disgust and surprise as shown in the figure).

1 Introduction

While emotions are expressed and managed daily, they are complex, nuanced, and sometimes hard to articulate and interpret. That is, people use language in subtle and complex ways to express emotions across languages and cultures (Wiebe et al., 2005; Mohammad and Kiritchenko, 2018; Mohammad et al., 2018a) and perceive them subjectively, even within the same culture or social group. Emotion recognition is at the core of several NLP applications in healthcare, dialogue systems, computational social science, digital humanities, narrative analysis, and many others (Mohammad et al., 2018b; Saffar et al., 2023). It is an umbrella term for multiple NLP tasks, such as detecting the possible emotions of the speaker, identifying what emotion a piece of text is conveying, and detecting the emotions evoked in a reader (Mohammad, 2022). In this paper, we use emotion recognition to refer to perceived emotions, i.e., what emotion most people think the speaker might have felt given a sentence or a short text snippet uttered by them.

Most work on emotion recognition has focused on high-resource languages such as English, Spanish, German, and Arabic (Strapparava and Mihalcea, 2007; Seyeditabari et al., 2018; Chatterjee et al., 2019; Kumar et al., 2022). This is partly due to the unavailability of datasets in under-served languages, which has led to a major research gap in the area, which is particularly noticeable in low-resource languages. That is, despite the linguistic diversity present in different parts of the world, such as Africa and Asia, which are home to more than 4,000 languages1, few emotion recognition resources are available in these languages. To bridge this gap, we introduce BRIGHTER–a collection of manually annotated emotion datasets for 28 languages containing nearly 100,000 instances from diverse data sources: speeches, social media, news, literature, and reviews. The languages belong to 7 language families (see Figure 1) and are predominantly low-resource, mainly spoken in Africa, Asia, Eastern Europe, Latin America, along with mid- to high-resource languages such as English. Each instance in BRIGHTER is curated and annotated by fluent speakers based on six emotion classes: joy, sadness, anger, fear, surprise, disgust, and none for neutral. The instances are multi-labeled and include 4 levels of intensity that vary from 0 to 3 (examples in Figure 2). We describe the collection, annotation, and quality control steps used to construct BRIGHTER. We then test various baseline experiments and observe that LLMs still struggle with recognising perceived emotions in

1https://www.ethnologue.com/insights/how-many-languages
text. We further report on the observed discrepancies across languages such as the fact that, for low-resource languages, LLMs perform significantly better when prompted in English. We make our datasets public2, which presents an important step towards work on emotion recognition and related tasks as we involve local communities in the collection and annotation. Our insights into language-specific characteristics of emotions in text, nuances, and challenges may enable the creation of more inclusive digital tools.

2 The BRIGHTER Dataset Collection

As our BRIGHTER collection includes datasets in 28 different languages, curated and annotated by fluent speakers, we use several data sources, collection, and annotation strategies depending on 1) the availability of the textual data potentially rich in emotions and 2) access to annotators. We detail the choices made when selecting and balancing sources, annotating the instances, and controlling for data quality in the following section.

2.1 Data Sources

Selecting appropriate data can be challenging when resources are scarce. Therefore, we typically combine multiple sources, as shown in Table 1. Below, we outline the main textual domain inputs used to construct BRIGHTER.

Social media posts  We use social media data collected from various platforms, including Reddit (e.g., eng, deu), YouTube (e.g., esp, ind, jav, sun), Twitter (e.g., hau, ukr), and Weibo (e.g., chn). For some languages, we re-annotate existing sentiment datasets for emotions (e.g., the sentiment analysis benchmark AfriSenti (Muhammad et al., 2023a) for ary, hau, kin; the Twitter dataset by Bobrovnyk (2019) for ukr; the RED–v2 dataset (Ciobotaru et al., 2022) for ron).

Personal narratives, talks, and speeches  Anonymised sentences from personal diary posts are ideal for extracting sentences where the speaker is centering their own emotions as opposed to the emotions of someone else. Hence, we use these in eng, deu, and ptbr, mainly from subreddits such as, e.g., IAmI.

Similarly, the afr dataset includes sentences from speeches and talks which constitute a good source for potentially emotive text.

Literary texts  We manually translated the novel “La Grande Maison” (The Big House) by the Algerian author Mohammed Dib 3 from French to Algerian Arabic and further post-processed the translation to generate sentences to be annotated by native speakers. Note that the translator is bilingual and a native Algerian Arabic speaker. Such a source is typically rich in emotions as it includes interactions between various characters. Moreover, Algerian Arabic is mainly spoken due to the Arabic diglossia, which makes this resource valuable since it highly differs from social media datasets in arq.

News data  Although we prefer emotionally rich social media data from different platforms, such data is not always available. Therefore, when data sources are limited, to collect a larger number of instances, we annotate news data and headlines in some African languages (e.g., yor, hau, and vvm).

Human-written and machine generated data  We create a dataset from scratch for Hindi (hin) and Marathi (mar). We ask annotators to generate emotive sentences on a given topic (e.g., family). In addition, we automatically translate a small section of the Hindi dataset to Marathi, and native speakers manually fix the translation errors. Finally, we augment both datasets with a few hundred quality-approved instances generated by ChatGPT.

2.2 Pre-processing and Quality Control

Prior to annotation, we preprocess the data by removing duplicates, invisible characters, garbled encoding, and incorrectly rendered emoticons. We anonymise all texts and exclude content with excessive expletives or dehumanising language.

2.3 Annotating BRIGHTER

As a text snippet can elicit multiple emotions simultaneously, we ask the annotators to select all the emotions that apply to a given text rather than choosing a single dominant emotion class. The set of labels includes six categories of perceived emotions: anger, sadness, fear, disgust, joy, surprise, and is considered neutral if no emotion is picked. The annotators further rate the selected emotion(s) on a four-point intensity scale: 0 (no emotion), 1

2The datasets are available at https://brighter-dataset.github.io. Note that they were used in SemEval-2025 Task 11, which attracted over 700 participants (Muhammad et al., 2025).
3https://en.wikipedia.org/wiki/La_Grande_Maison
<table>
  <tr>
    <th>Language</th>
    <th>Data source(s)</th>
    <th>#Annotators (total)</th>
    <th>#Ann. / sample</th>
    <th>Train</th>
    <th>Dev</th>
    <th>Test</th>
    <th>Total</th>
  </tr>
  <tr>
    <td>Afrikaans (<b>afr</b>)</td>
    <td>Speeches</td>
    <td>3</td>
    <td>3</td>
    <td>1,325</td>
    <td>115</td>
    <td>1,247</td>
    <td>2,687</td>
  </tr>
  <tr>
    <td>Algerian Arabic (<b>arq</b>)</td>
    <td>Literature</td>
    <td>10</td>
    <td>4 to 9</td>
    <td>1,686</td>
    <td>182</td>
    <td>1,674</td>
    <td>3,542</td>
  </tr>
  <tr>
    <td>Moroccan Arabic (<b>ary</b>)</td>
    <td>News, social media</td>
    <td>3</td>
    <td>3</td>
    <td>1,813</td>
    <td>300</td>
    <td>931</td>
    <td>3,044</td>
  </tr>
  <tr>
    <td>Chinese (<b>chn</b>)</td>
    <td>Social media</td>
    <td>7</td>
    <td>5</td>
    <td>3,316</td>
    <td>250</td>
    <td>3,345</td>
    <td>6,911</td>
  </tr>
  <tr>
    <td>German (<b>deu</b>)</td>
    <td>Social media</td>
    <td>10</td>
    <td>7</td>
    <td>3,700</td>
    <td>294</td>
    <td>3,690</td>
    <td>7,684</td>
  </tr>
  <tr>
    <td>English (<b>eng</b>)</td>
    <td>Social media</td>
    <td>122</td>
    <td>5 to 30</td>
    <td>4,574</td>
    <td>189</td>
    <td>4,509</td>
    <td>9,272</td>
  </tr>
  <tr>
    <td>Latin American Spanish (<b>esp</b>)</td>
    <td>Social media</td>
    <td>12</td>
    <td>5</td>
    <td>2,835</td>
    <td>256</td>
    <td>2,340</td>
    <td>5,431</td>
  </tr>
  <tr>
    <td>Hausa (<b>hau</b>)</td>
    <td>News, social media</td>
    <td>5</td>
    <td>5</td>
    <td>2,656</td>
    <td>440</td>
    <td>1,352</td>
    <td>4,448</td>
  </tr>
  <tr>
    <td>Hindi (<b>hin</b>)</td>
    <td>Created</td>
    <td>5</td>
    <td>4 to 5</td>
    <td>2,841</td>
    <td>108</td>
    <td>1,070</td>
    <td>4,019</td>
  </tr>
  <tr>
    <td>Igbo (<b>ibo</b>)</td>
    <td>News, social media</td>
    <td>3</td>
    <td>3</td>
    <td>2,988</td>
    <td>497</td>
    <td>1,502</td>
    <td>4,987</td>
  </tr>
  <tr>
    <td>Indonesian (<b>ind</b>)</td>
    <td>Social media</td>
    <td>16</td>
    <td>3</td>
    <td>–</td>
    <td>247</td>
    <td>1,409</td>
    <td>1,656</td>
  </tr>
  <tr>
    <td>Javanese (<b>jav</b>)</td>
    <td>Social media</td>
    <td>13</td>
    <td>3</td>
    <td>–</td>
    <td>250</td>
    <td>1,395</td>
    <td>1,645</td>
  </tr>
  <tr>
    <td>Kinyarwanda (<b>kin</b>)</td>
    <td>News, social media</td>
    <td>3</td>
    <td>3</td>
    <td>5,350</td>
    <td>426</td>
    <td>1,298</td>
    <td>4,299</td>
  </tr>
  <tr>
    <td>Marathi (<b>mar</b>)</td>
    <td>Created</td>
    <td>4</td>
    <td>4</td>
    <td>2,590</td>
    <td>108</td>
    <td>1,103</td>
    <td>3,864</td>
  </tr>
  <tr>
    <td>Nigerian-Pidgin (<b>pcm</b>)</td>
    <td>News, social media</td>
    <td>3</td>
    <td>3</td>
    <td>1,553</td>
    <td>888</td>
    <td>2,691</td>
    <td>8,929</td>
  </tr>
  <tr>
    <td>Portuguese (Brazilian; <b>ptbr</b>)</td>
    <td>Social media</td>
    <td>5</td>
    <td>5</td>
    <td>2,318</td>
    <td>228</td>
    <td>2,580</td>
    <td>5,398</td>
  </tr>
  <tr>
    <td>Portuguese (Mozambican; <b>ptmz</b>)</td>
    <td>News, social media</td>
    <td>3</td>
    <td>3</td>
    <td>2,995</td>
    <td>258</td>
    <td>780</td>
    <td>2,591</td>
  </tr>
  <tr>
    <td>Romanian (<b>ron</b>)</td>
    <td>Social media</td>
    <td>8</td>
    <td>3 to 8</td>
    <td>1,352</td>
    <td>123</td>
    <td>1,893</td>
    <td>4,536</td>
  </tr>
  <tr>
    <td>Russian (<b>rus</b>)</td>
    <td>Social media</td>
    <td>10</td>
    <td>3 to 10</td>
    <td>3,443</td>
    <td>225</td>
    <td>1,127</td>
    <td>4,347</td>
  </tr>
  <tr>
    <td>Sundanese (<b>sun</b>)</td>
    <td>Social media</td>
    <td>16</td>
    <td>3</td>
    <td>1,495</td>
    <td>292</td>
    <td>1,351</td>
    <td>2,995</td>
  </tr>
  <tr>
    <td>Swahili (<b>swa</b>)</td>
    <td>News, social media</td>
    <td>3</td>
    <td>3</td>
    <td>1,000</td>
    <td>573</td>
    <td>1,727</td>
    <td>5,743</td>
  </tr>
  <tr>
    <td>Swedish (<b>swe</b>)</td>
    <td>Social media</td>
    <td>3</td>
    <td>3</td>
    <td>2,527</td>
    <td>253</td>
    <td>1,514</td>
    <td>3,262</td>
  </tr>
  <tr>
    <td>Tatar (<b>tat</b>)</td>
    <td>Social media</td>
    <td>3</td>
    <td>2</td>
    <td>1,558</td>
    <td>200</td>
    <td>1,000</td>
    <td>2,200</td>
  </tr>
  <tr>
    <td>Ukrainian (<b>ukr</b>)</td>
    <td>Social media</td>
    <td>106</td>
    <td>5</td>
    <td>3,133</td>
    <td>255</td>
    <td>2,278</td>
    <td>5,060</td>
  </tr>
  <tr>
    <td>Emakhuwa (<b>vvm</b>)</td>
    <td>News, social media</td>
    <td>3</td>
    <td>3</td>
    <td>1,558</td>
    <td>259</td>
    <td>781</td>
    <td>2,598</td>
  </tr>
  <tr>
    <td>isiXhosa (<b>xho</b>)</td>
    <td>News, social media</td>
    <td>3</td>
    <td>3</td>
    <td>–</td>
    <td>745</td>
    <td>1,744</td>
    <td>2,489</td>
  </tr>
  <tr>
    <td>Yoruba (<b>yor</b>)</td>
    <td>News</td>
    <td>3</td>
    <td>3</td>
    <td>3,133</td>
    <td>520</td>
    <td>1,572</td>
    <td>5,225</td>
  </tr>
  <tr>
    <td>isiZulu (<b>zul</b>)</td>
    <td>News, social media</td>
    <td>3</td>
    <td>3</td>
    <td>–</td>
    <td>940</td>
    <td>2,202</td>
    <td>3,142</td>
  </tr>
</table>

Table 1: Data sources, number of annotators, and data split sizes for the BRIGHTER datasets, sorted alphabetically by language code. Datasets without training splits (–) were used exclusively for testing (see Section 3).

(low intensity), 2 (moderate intensity level), and 3 (high intensity). We provide the definitions of the categories and annotation guide in Appendix D.

We use Amazon Mechanical Turk to annotate the English dataset, and Toloka\footnote{https://toloka.ai} to label the Russian, Ukrainian, and Tatar instances. However, as traditional crowdsourcing platforms do not have a large pool of annotators who speak various low-resource languages, we directly recruit fluent speakers to annotate the data and use the academic version of LabelStudio (Tkachenko et al., 2020-2025) and Potato (Pei et al., 2022) to set up our annotation platform.

\subsection*{2.4 Annotators’ Reliability}

While both inter-annotator agreement (IAA) and reliability scores evaluate annotation quality, they capture different aspects. IAA evaluates the extent to which annotators agree with one another, whereas reliability scores measure the consistency of aggregated labels across repeated annotation trials (Kiritchenko and Mohammad, 2016). Consequently, reliability scores tend to increase with a larger number of annotations, while IAA scores do not depend on the number of annotations per instance.

We report the annotation reliability using Split-Half Class Match Percentage (SHCMP; Mohammad, 2024). SHCMP extends the concept of Split-Half Reliability (SHR), traditionally applied to continuous scores (Kiritchenko and Mohammad, 2016), to discrete categories, such as our emotion intensity labels. SHCMP measures the extent to which \( n \) bins (i.e., random subsets) classify items consistently. The dataset is randomly split into \( n \) bins (corresponding to halves when \( n = 2 \)) 1,000 times, and the proportion of instances receiving the same class label across bins is averaged to return the final SHCMP score. A higher SHCMP indicates greater reliability, meaning that repeated annotations would likely result in similar class labels. Additional details are provided in Appendix D. Figure 3 shows a heatmap of SHCMP scores for the BRIGHTER datasets. Overall, the SHCMP scores are high (greater than 60% for \( n = 2 \)), indi-
Figure 3: Split-Half Class Match Percentage, SHCMP (%) values for the BRIGHTER datasets across varying numbers of bins (2 to 10). Higher values indicate better reliability scores. Note that ptmz and vvm have the same score as vvm instances were translated from ptmz and the translation was verified.

cating that our annotations are reliable.

2.5 Determining the Final Labels
We expected a level of disagreement as emotions are complex, subtle, and perceived differently even from people within the same culture. In addition, text-based communication is limited as it lacks cues such as tone, relevant context, and information about the speaker. Our approach for aggregating the per-annotator emotion and intensity labels is detailed below. We also publicly share the individual (non-aggregated) annotations, recognising that annotator disagreement can provide useful signals in itself (Plank, 2022).

Aggregating the emotion labels The final emotion labels are determined based on the emotions and associated intensity values selected by the annotators. That is, the given emotion is considered present if:
1. At least two annotators select a label with an intensity value of 1, 2, or 3 (low, medium, or high, respectively).
2. The average score exceeds a predefined threshold \( T \). We set \( T \) to 0.5.

Aggregating the intensity labels Once the labels for perceived emotions are assigned, we determine the final intensity score for each instance by averaging the selected intensity scores and rounding them up to the nearest integer. We assign intensity scores only for datasets in which the majority of instances are annotated by \( \geq 5 \) annotators, to ensure robustness. Therefore, BRIGHTER includes emotion labels for 28 languages and intensity labels for 10 languages.

2.6 Final Data Statistics
Figure 4 shows the distribution of the annotated emotions in the BRIGHTER datasets. The neutral class contains instances that do not belong to any of the six predefined categories (i.e., anger, disgust, fear, sadness, joy, and surprise). Although most languages include all six categories, the English dataset does not include disgust, and the Afrikaans one does not include surprise due to an insufficient class representation. Furthermore, class distributions show substantial variation as we chose various data sources as shown in Table 1.

3 Experiments
3.1 Setup
We report the data split sizes in Table 1. The test sets are relatively large, ranging from approximately 1,000 to nearly 3,000 instances. Datasets without training data are excluded from training and are used solely for testing in cross-lingual settings.

For our baseline experiments, we evaluate multi-label emotion classification and emotion intensity prediction using both Multilingual Language Models (MLMs) and Large Language Models (LLMs).

Multi-label emotion classification in few-shot settings We report emotion classification performance using five LLMs–Qwen2.5–72B (Yang et al., 2024), Dolly-v2-12B (Conover et al., 2023), LLaMA-3.3–70B (Touvron et al., 2023), Mixtral-8x7B (Jiang et al., 2024), and DeepSeek-R1–70B (DeepSeek-AI et al., 2025). We prompt the LLMs using Chain-of-Thought (CoT) reasoning to predict the presence of each emotion from the predefined set. We set the number of few-shot examples to 8 and consider only the first generated answer (i.e., top-1). We
Figure 4: Emotion label distribution across the BRIGHTER datasets. Each bar represents the number of labeled instances per emotion (i.e., anger, disgust, fear, joy, sadness, surprise, and neutral) and its percentage.

report macro F1 scores across 28 languages. In Appendix D, we also provide monolingual classification results for the 24 languages with training data (see Table 5).

Multi-label emotion classification in crosslingual settings  We report the macro F-score results for systems trained without using any data in the 28 target languages when testing on each. Hence, we train MLMs on all languages in one family (see Figure 1) except for one held-out target language, which we test on and report the results for each test set. For families with only one language (i.e., Sino-Tibetan, Creole, and Turkic), we train on Slavic languages (rus and ukr) and test on tat; two Niger-Congo languages (swa and yor) and test on pcm; and on rus and test on chn.

Emotion intensity prediction  We report Pearson correlation scores for systems trained on the intensity-labeled training sets in 10 languages.

3.2 Experimental Results

Table 2 reports the results of few-shot and crosslingual experiments for multi-label emotion classification and Table 3 reports those for emotion intensity classification. Our results corroborate how challenging emotion classification is for LLMs, even for high-resource languages such as eng and deu. The performance is worse for low-resource languages, for which DoLLy-v2-12B performs the worst, and Qwen2.5–72B performs the best on average.

We observe the largest performance for yor with a maximum of 27.44. hin, mar, and tat have the best performance among all languages, which is unsurprising since the tat dataset is single-labeled, and close to 70% and 80% of the test data for mar and hin respectively are single-labeled.

Multi-label emotion recognition results  The crosslingual experiments demonstrate that model performance depends on both the languages used for transfer learning and those included in the pre-training of the models. For instance, in some cases, training on languages from the same family improves performance and even surpasses few-shot settings, e.g., swe benefits when RemBERT is fine-tuned on other Germanic languages. However, all Niger-Congo languages, particularly vmw, benefit the least from crosslingual transfer across all models, with RemBERT performing the worst. This is largely due to the severe under-resourcedness of these languages, even when data is combined. Notably, XLM-R performs exceptionally well on languages such as deu, chn, hin, and ptbr, but struggles significantly with others (e.g., swe, ptmz). In contrast, mDeBERTa yields the most consistent results across most languages, even though it shows low performance on ibo, vmw, and yor, which are not part of the CC-100 corpus (Conneau
<table>
  <tr>
    <th rowspan="2">Lang.</th>
    <th colspan="5">Few-Shot Multi-Label Classification</th>
    <th colspan="5">Crosslingual Multi-Label Classification</th>
  </tr>
  <tr>
    <th>Qwen2.5-72B</th>
    <th>Dolly-v2-12B</th>
    <th>Llama-3.3-70B</th>
    <th>Mixtral-8x7B</th>
    <th>DeepSeek-R1-70B</th>
    <th>LabSE</th>
    <th>RemBERT</th>
    <th>XLM-R</th>
    <th>mBERT</th>
    <th>mDeBERTa</th>
  </tr>
  <tr>
    <td>afr</td>
    <td>60.18</td>
    <td>23.58</td>
    <td><b>61.28</b></td>
    <td>53.69</td>
    <td>43.66</td>
    <td>35.12</td>
    <td>35.04</td>
    <td><b>41.66</b></td>
    <td>16.95</td>
    <td>33.25</td>
  </tr>
  <tr>
    <td>arq</td>
    <td>37.78</td>
    <td>38.59</td>
    <td><b>55.75</b></td>
    <td>45.29</td>
    <td>50.87</td>
    <td><b>35.93</b></td>
    <td>33.78</td>
    <td>35.87</td>
    <td>31.38</td>
    <td>35.92</td>
  </tr>
  <tr>
    <td>ary</td>
    <td><b>52.76</b></td>
    <td>24.27</td>
    <td>44.96</td>
    <td>35.07</td>
    <td>47.21</td>
    <td><b>42.83</b></td>
    <td>35.46</td>
    <td>33.88</td>
    <td>24.83</td>
    <td>36.28</td>
  </tr>
  <tr>
    <td>chn</td>
    <td><b>55.23</b></td>
    <td>27.52</td>
    <td>53.36</td>
    <td>44.91</td>
    <td>53.45</td>
    <td>45.28</td>
    <td>24.56</td>
    <td><b>53.84</b></td>
    <td>21.61</td>
    <td>42.41</td>
  </tr>
  <tr>
    <td>deu</td>
    <td><b>59.17</b></td>
    <td>26.86</td>
    <td>56.99</td>
    <td>51.20</td>
    <td>54.26</td>
    <td>42.45</td>
    <td>46.84</td>
    <td><b>47.26</b></td>
    <td>28.60</td>
    <td>42.61</td>
  </tr>
  <tr>
    <td>eng</td>
    <td>55.72</td>
    <td>42.60</td>
    <td><b>65.58</b></td>
    <td>58.12</td>
    <td>56.99</td>
    <td>36.71</td>
    <td>37.54</td>
    <td><b>37.60</b></td>
    <td>18.80</td>
    <td>35.30</td>
  </tr>
  <tr>
    <td>esp</td>
    <td>72.33</td>
    <td>36.41</td>
    <td>61.27</td>
    <td>65.72</td>
    <td><b>73.29</b></td>
    <td>54.56</td>
    <td><b>57.37</b></td>
    <td>44.52</td>
    <td>30.09</td>
    <td>37.09</td>
  </tr>
  <tr>
    <td>hau</td>
    <td>43.79</td>
    <td>29.43</td>
    <td>50.91</td>
    <td>40.40</td>
    <td><b>51.91</b></td>
    <td><b>38.46</b></td>
    <td>31.98</td>
    <td>16.69</td>
    <td>15.59</td>
    <td>32.80</td>
  </tr>
  <tr>
    <td>hin</td>
    <td><b>79.73</b></td>
    <td>27.59</td>
    <td>60.59</td>
    <td>62.19</td>
    <td>76.91</td>
    <td>69.78</td>
    <td>13.75</td>
    <td><b>69.96</b></td>
    <td>36.94</td>
    <td>57.74</td>
  </tr>
  <tr>
    <td>ibo</td>
    <td><b>37.40</b></td>
    <td>24.31</td>
    <td>33.18</td>
    <td>31.90</td>
    <td>32.85</td>
    <td><b>18.13</b></td>
    <td>7.49</td>
    <td>10.42</td>
    <td>9.94</td>
    <td>9.52</td>
  </tr>
  <tr>
    <td>ind</td>
    <td><b>57.29</b></td>
    <td>36.61</td>
    <td>39.20</td>
    <td>54.37</td>
    <td>49.51</td>
    <td><b>47.50</b></td>
    <td>37.64</td>
    <td>25.39</td>
    <td>26.87</td>
    <td>35.68</td>
  </tr>
  <tr>
    <td>jav</td>
    <td><b>50.47</b></td>
    <td>36.18</td>
    <td>41.88</td>
    <td>48.37</td>
    <td>43.05</td>
    <td>46.24</td>
    <td><b>46.38</b></td>
    <td>20.39</td>
    <td>26.16</td>
    <td>35.34</td>
  </tr>
  <tr>
    <td>kin</td>
    <td>31.96</td>
    <td>19.73</td>
    <td><b>34.36</b></td>
    <td>26.35</td>
    <td>32.52</td>
    <td><b>30.35</b></td>
    <td>18.38</td>
    <td>13.12</td>
    <td>20.90</td>
    <td>17.30</td>
  </tr>
  <tr>
    <td>mar</td>
    <td>74.58</td>
    <td>25.69</td>
    <td>67.40</td>
    <td>50.36</td>
    <td><b>76.68</b></td>
    <td>74.65</td>
    <td><b>77.24</b></td>
    <td>76.21</td>
    <td>42.32</td>
    <td>54.05</td>
  </tr>
  <tr>
    <td>pcm</td>
    <td>38.66</td>
    <td>34.41</td>
    <td><b>48.67</b></td>
    <td>45.61</td>
    <td>45.00</td>
    <td><b>33.29</b></td>
    <td>1.01</td>
    <td>21.08</td>
    <td>22.55</td>
    <td>25.39</td>
  </tr>
  <tr>
    <td>ptbr</td>
    <td><b>51.60</b></td>
    <td>25.90</td>
    <td>45.03</td>
    <td>41.64</td>
    <td>51.49</td>
    <td>41.51</td>
    <td>41.84</td>
    <td><b>43.09</b></td>
    <td>23.86</td>
    <td>34.42</td>
  </tr>
  <tr>
    <td>ptmz</td>
    <td><b>40.44</b></td>
    <td>16.70</td>
    <td>34.06</td>
    <td>36.52</td>
    <td>39.58</td>
    <td><b>31.44</b></td>
    <td>29.67</td>
    <td>7.30</td>
    <td>13.54</td>
    <td>24.46</td>
  </tr>
  <tr>
    <td>ron</td>
    <td>68.18</td>
    <td>43.58</td>
    <td><b>71.28</b></td>
    <td>68.51</td>
    <td>65.02</td>
    <td>69.79</td>
    <td><b>76.23</b></td>
    <td>65.21</td>
    <td>61.50</td>
    <td>60.60</td>
  </tr>
  <tr>
    <td>rus</td>
    <td>73.08</td>
    <td>29.72</td>
    <td>62.61</td>
    <td>61.72</td>
    <td><b>76.97</b></td>
    <td>61.32</td>
    <td><b>70.43</b></td>
    <td>21.14</td>
    <td>37.15</td>
    <td>29.70</td>
  </tr>
  <tr>
    <td>sun</td>
    <td>42.67</td>
    <td>32.20</td>
    <td><b>46.33</b></td>
    <td>42.10</td>
    <td>44.61</td>
    <td><b>34.79</b></td>
    <td>19.43</td>
    <td>25.92</td>
    <td>25.29</td>
    <td>27.31</td>
  </tr>
  <tr>
    <td>swa</td>
    <td>27.36</td>
    <td>17.63</td>
    <td>29.47</td>
    <td>26.51</td>
    <td><b>33.27</b></td>
    <td>21.66</td>
    <td><b>18.99</b></td>
    <td>16.94</td>
    <td>18.61</td>
    <td>14.94</td>
  </tr>
  <tr>
    <td>swe</td>
    <td>48.89</td>
    <td>21.79</td>
    <td><b>50.26</b></td>
    <td>48.61</td>
    <td>44.60</td>
    <td>44.24</td>
    <td><b>51.18</b></td>
    <td>10.08</td>
    <td>28.86</td>
    <td>43.28</td>
  </tr>
  <tr>
    <td>tat</td>
    <td>51.58</td>
    <td>25.12</td>
    <td>49.84</td>
    <td>39.44</td>
    <td><b>53.86</b></td>
    <td><b>60.66</b></td>
    <td>44.54</td>
    <td>39.58</td>
    <td>35.81</td>
    <td>47.72</td>
  </tr>
  <tr>
    <td>ukr</td>
    <td><b>54.76</b></td>
    <td>17.16</td>
    <td>42.34</td>
    <td>40.15</td>
    <td>51.19</td>
    <td>44.37</td>
    <td><b>49.56</b></td>
    <td>34.06</td>
    <td>25.69</td>
    <td>35.12</td>
  </tr>
  <tr>
    <td>vmw</td>
    <td><b>20.41</b></td>
    <td>16.03</td>
    <td>18.96</td>
    <td>19.00</td>
    <td>19.09</td>
    <td>9.65</td>
    <td>5.22</td>
    <td><b>12.66</b></td>
    <td>12.11</td>
    <td>11.74</td>
  </tr>
  <tr>
    <td>xho</td>
    <td>29.56</td>
    <td>24.12</td>
    <td>30.79</td>
    <td>22.92</td>
    <td>29.08</td>
    <td>31.39</td>
    <td>12.73</td>
    <td>11.48</td>
    <td>17.08</td>
    <td>22.86</td>
  </tr>
  <tr>
    <td>yor</td>
    <td>24.99</td>
    <td>16.00</td>
    <td>23.70</td>
    <td>19.67</td>
    <td><b>27.44</b></td>
    <td><b>11.64</b></td>
    <td>5.33</td>
    <td>6.64</td>
    <td>9.62</td>
    <td>10.03</td>
  </tr>
  <tr>
    <td>zul</td>
    <td>22.03</td>
    <td>14.72</td>
    <td>21.48</td>
    <td>20.38</td>
    <td>20.38</td>
    <td>18.16</td>
    <td>15.26</td>
    <td>10.92</td>
    <td>13.04</td>
    <td>13.87</td>
  </tr>
  <tr>
    <th>AVG</th>
    <th><b>49.71</b></th>
    <th>26.88</th>
    <th>47.12</th>
    <th>43.56</th>
    <th>49.21</th>
    <th><b>40.50</b></th>
    <th>33.63</th>
    <th>30.61</th>
    <th>24.16</th>
    <th>32.38</th>
  </tr>
</table>

Table 2: Average F1-Macro for multi-label emotion classification. In the few-shot setting, we predict the emotion class on test set in 28 languages. In the crosslingual setting, we train on all languages within a language family except the target language, and evaluate on the test set of the target language. The best performance scores in few-shot and crosslingual settings are highlighted in blue and orange, respectively.

et al., 2020) used in its training. While mDeBERTa was also not trained on arq, the inclusion of Modern Standard Arabic (MSA) in its pretraining data might have positively influenced its performance.

Overall, our results indicate that multilingual models transfer more effectively to languages seen during pretraining, while often producing random or unreliable outputs for languages absent from their training data.

Emotion intensity prediction For intensity detection, a more challenging task, Dolly-v2-12B performs the worst, whereas DeepSeek-R1-70B shows promising results, outperforming other models in most languages. Llama-3.3-70B and Qwen2.5-72B achieve the highest scores in English. Interestingly, MLMs tend to perform better on high-resource languages–RemBERT, in particular, achieves strong results for deu, eng, esp, and rus, with chn being the only exception. In contrast, for primarily spoken, low-resource vernaculars (e.g., arq), LLMs demonstrate striking improvements – DeepSeek-R1-70B, for instance, achieves improvements exceeding 36 points.

4 Analysis

The results in Figure 5a suggest that LLM performance is highly dependent on the prompt wording when asking for the presence of emotion on the English test set using different paraphrases of the same text. Further, Figure 5b shows that, when testing the effect of n-shot settings on the English test set, we observe a significant improvement in performance with more shots, with Mixtral-8x7B and Llama-3.3-70B outperforming other models. However, the scores tend to reach a plateau at 4 shots for all LLMs except for Qwen2.5-72B, which suggests that 4 to 8 shots may be sufficient to obtain stable results. In addition, when testing how likely we can get the correct answer when prompting LLMs to generate tokens based on a selection of k generations, the results shown in Figure 5c suggest that increasing the value of k results
<table>
  <tr>
    <th rowspan="2">Lang.</th>
    <th colspan="5">Multilingual Language Models (MLMs)</th>
    <th colspan="6">Large Language Models (LLMs)</th>
  </tr>
  <tr>
    <th>LaBSE</th>
    <th>RemBERT</th>
    <th>XLM-R</th>
    <th>mBERT</th>
    <th>mDeBERTa</th>
    <th>Qwen2.5-72B</th>
    <th>Dolly-v2-12B</th>
    <th>Llama-3.3-70B</th>
    <th>Mixtral-8x7B</th>
    <th>DeepSeek-R1-70B</th>
  </tr>
  <tr>
    <td>arq</td>
    <td>1.42</td>
    <td><b>1.64</b></td>
    <td>0.89</td>
    <td>1.10</td>
    <td>0.47</td>
    <td>29.54</td>
    <td>3.80</td>
    <td>36.29</td>
    <td>31.05</td>
    <td><b>36.37</b></td>
  </tr>
  <tr>
    <td>chn</td>
    <td>23.37</td>
    <td><b>40.53</b></td>
    <td>36.92</td>
    <td>21.96</td>
    <td>23.25</td>
    <td>46.17</td>
    <td>8.11</td>
    <td><b>51.86</b></td>
    <td>46.52</td>
    <td>48.57</td>
  </tr>
  <tr>
    <td>deu</td>
    <td>28.93</td>
    <td><b>56.21</b></td>
    <td>38.30</td>
    <td>17.35</td>
    <td>18.14</td>
    <td>43.30</td>
    <td>7.43</td>
    <td>53.46</td>
    <td>47.60</td>
    <td><b>54.78</b></td>
  </tr>
  <tr>
    <td>eng</td>
    <td>35.34</td>
    <td><b>64.15</b></td>
    <td>37.36</td>
    <td>25.74</td>
    <td>8.85</td>
    <td><b>55.99</b></td>
    <td>13.35</td>
    <td>44.14</td>
    <td>55.26</td>
    <td>48.08</td>
  </tr>
  <tr>
    <td>esp</td>
    <td>56.89</td>
    <td><b>72.59</b></td>
    <td>55.72</td>
    <td>27.94</td>
    <td>29.18</td>
    <td>51.11</td>
    <td>10.49</td>
    <td>51.64</td>
    <td>55.54</td>
    <td><b>60.74</b></td>
  </tr>
  <tr>
    <td>hau</td>
    <td>26.13</td>
    <td><b>27.03</b></td>
    <td>24.68</td>
    <td>2.79</td>
    <td>0.00</td>
    <td>27.00</td>
    <td>6.43</td>
    <td><b>39.16</b></td>
    <td>25.84</td>
    <td>38.85</td>
  </tr>
  <tr>
    <td>ptbr</td>
    <td>20.62</td>
    <td><b>29.74</b></td>
    <td>18.24</td>
    <td>8.36</td>
    <td>1.32</td>
    <td>38.20</td>
    <td>9.02</td>
    <td>40.90</td>
    <td>39.17</td>
    <td><b>46.72</b></td>
  </tr>
  <tr>
    <td>ron</td>
    <td>35.57</td>
    <td><b>55.66</b></td>
    <td>37.77</td>
    <td>21.99</td>
    <td>4.63</td>
    <td>55.48</td>
    <td>12.62</td>
    <td>45.87</td>
    <td>57.07</td>
    <td><b>57.69</b></td>
  </tr>
  <tr>
    <td>rus</td>
    <td>68.43</td>
    <td><b>87.66</b></td>
    <td>68.96</td>
    <td>37.63</td>
    <td>5.03</td>
    <td>58.25</td>
    <td>13.96</td>
    <td>57.56</td>
    <td>56.01</td>
    <td><b>62.28</b></td>
  </tr>
  <tr>
    <td>ukr</td>
    <td>13.75</td>
    <td><b>39.94</b></td>
    <td>36.16</td>
    <td>4.32</td>
    <td>3.51</td>
    <td>37.74</td>
    <td>6.04</td>
    <td>36.99</td>
    <td>38.74</td>
    <td><b>43.54</b></td>
  </tr>
  <tr>
    <th>Avg</th>
    <th>30.54</th>
    <th><b>46.61</b></th>
    <th>35.25</th>
    <th>16.35</th>
    <th>9.97</th>
    <th>43.03</th>
    <th>8.74</th>
    <th>45.78</th>
    <th>43.97</th>
    <th><b>48.88</b></th>
  </tr>
</table>

Table 3: Pearson correlation scores for intensity classification using MLMs and LLMs. The best performance scores are highlighted in blue and orange, respectively.

(a) Performance of different LLMs across three prompt paraphrases on the English test set. Different prompts impact model performance.

(b) Few-shot performance of LLMs on the English test set. Performance improves with more shots.

(c) Pass@k performance of different LLMs on the English test set. Higher k values increase the likelihood of retrieving the correct answer.

Figure 5: Ablation studies on the effect of prompt wording variation, few-shot examples, and pass@k predictions conducted on the English test set.

consistently in better performance, particularly when using DeepSeekR1-70B, which achieves an F-score > 90 when \( k = 8 \) whereas Mixtral-8x7B shows a smaller change in performance followed by Llama-3.3-70B and Qwen2.5-72B. The ranking of the models for \( k = 8 \) remains consistent with the one achieved for \( k = 1 \).

When comparing the performance of models prompted in English versus the target language, Figure 6 shows that LLMs generally perform better with English prompts except for arq, where Qwen2.5-72B achieves better results when prompted in Modern Standard Arabic (MSA). The improvement from using English prompts is particularly evident in low-resource languages (e.g., hau, mar, vmw), where models like Dolly-v2-12B and Llama-3.3-70B perform poorly with prompts in the target language.

![Comparing models' performance across languages when prompted in English (orange) vs. when prompted in the target language (blue). LLMs perform better when prompted in English.](page_1012_1342_420_370.png)

Figure 6: Comparing models’ performance across languages when prompted in English (orange) vs. when prompted in the target language (blue). LLMs perform better when prompted in English.
5 Related Work

Appraisal theories of emotion propose that emotions arise from our evaluation of events based on personal experiences, leading to different emotional responses among individuals (Arnold, 1960; Frijda, 1986; Lazarus, 1991; Scherer, 2009; Ellsworth, 2013; Moors et al., 2013; Roseman, 2013; Ortony et al., 2022). The theory of constructed emotion claims that emotions are not hard-wired or universal, but rather conceptual constructs formed by the brain (Barrett, 2016, 2017).

Early work in NLP primarily focused on sentiment analysis—identifying whether a text conveys positive, negative, or neutral valence (Mohammad, 2016; Mohammad et al., 2023b). More recent research has shifted toward a broader goal: detecting specific emotions in text, such as anger, fear, joy, and sadness. This shift aligns with discrete models of emotion, including Paul Ekman’s six basic emotions (Ekman, 1992) and Plutchik’s Wheel of Emotions (Plutchik, 1980), which includes anger, disgust, fear, happiness, sadness, surprise, anticipation, and trust.

Several initiatives have created emotion classification datasets for languages other than English such as Italian (Bianchi et al., 2021), Romanian (Ciobotaru et al., 2022), Indonesian (Saputri et al., 2018), and Bengali (Iqbal et al., 2022). However, the field remains predominantly Western-centric. Although multilingual datasets such as XED (Öhman et al., 2020) and XLM-EMO (Bianchi et al., 2022) exist, the latter’s reliance on translated data for over ten languages may not adequately reflect cultural nuances in emotional expression. Emotions are culture-sensitive and highly contextual, shaped by different norms and values (Hershcovitch et al., 2022; Havaldar et al., 2023; Mohamed et al., 2024; Plaza-del Arco et al., 2024).

Furthermore, although emotions can co-occur (Vishnubhotla et al., 2024), most existing datasets assume a single-label classification framework. While GoEmotions (Demszky et al., 2020) addresses multi-label emotion classification, to our knowledge, no multilingual resources capture simultaneous emotions and intensity across languages. This work aims to advance the field by introducing emotion-labeled data for 28 languages. Given the lack of consensus around what constitutes a low-resource language, approximately 15 to 17 among these could reasonably be considered as such.

6 Conclusion

We presented BRIGHTER, a collection of emotion recognition datasets in 28 languages spoken across various continents. The instances in BRIGHTER are multi-labeled, collected, and annotated by fluent speakers, with 10 datasets annotated for emotion intensity. When testing LLMs on our dataset collection, the results show that they still struggle with predicting perceived emotions and their intensity levels, especially for under-resourced languages. Further, our results show that LLM performance is highly dependent on the wording of the prompt, its language, and the number of shots in few-shot settings. We publicly release BRIGHTER, our annotation guidelines, and individual labels to the research community.

Limitations

Emotions are subjective, subtle, expressed, and perceived differently. We do not claim that BRIGHTER covers the true emotions of the speakers, is fully representative of the language use of the 28 languages, or covers all possible emotions. We discuss this extensively in the Ethics Section.

We are aware of the limited data sources in some low-resource languages. Therefore, our datasets cannot be used for tasks that require a large amount of data from a given language. However, they remain a good starting point for research in the area.

Ethical Considerations

Emotion perception and expression are inherently subjective and nuanced, as they are closely tied to a myriad of factors (e.g., cultural background, social group, personal experiences, and social context). As such, it is impossible to determine with absolute certainty how someone is feeling based solely on short text snippets. Therefore, we explicitly state that our datasets focus on perceived emotions—that is, the emotions most people believe the speaker may have felt. Accordingly, we do not claim to annotate the true emotion of the speaker, as this cannot be definitively inferred from short texts alone. We recognise the importance of this distinction, as perceived emotions may differ from actual emotions.

We also acknowledge potential biases in our data. Text-based communication inherently carries biases, and our data sources may reflect such tendencies. Similarly, annotators may come with
their own subtle, internalised biases. Moreover, although many of our datasets focus on low-resource languages, we do not claim they fully capture the usage of these languages. While we took care to exclude inappropriate content, some instances may have been inadvertently overlooked.

We strongly encourage careful ethical reflection before using our datasets. Use of the data for commercial purposes or by state actors in high-risk applications is strictly prohibited unless explicitly approved by the dataset creators. Systems developed using our datasets may not be reliable at the individual instance level and are sensitive to domain shifts. They should not be used to make critical decisions about individuals, such as in health-related applications, without appropriate expert oversight. See Mohammad (2022, 2023) for a comprehensive discussion on these issues.

Finally, all annotators involved in the study were compensated at rates exceeding the local minimum wage.

Acknowledgments

Shamsuddeen Muhammad acknowledges the support of Google DeepMind and Lacuna Fund, an initiative co-founded by The Rockefeller Foundation, Google.org, and Canada’s International Development Research Centre. The views expressed herein do not necessarily represent those of Lacuna Fund, its Steering Committee, its funders, or Meridian Institute.
Nedjma Ousidhoum would like to thank Abderrahmane Samir Lazouni, Lyes Taher Khalfi, Manel Amarouche, Narimane Zahra Boumezrag, Noufel Bouslama, Abderraouf Ousidhoum, Sarah Arab, Wassil Adel Merzouk, Yanis Rabai, and another annotator who would like to stay anonymous for their work and insightful comments.
Idris Abdulmunin gratefully acknowledges the ABSA UP Chair of Data Science for funding his post-doctoral research and providing compute resources.
Jan Philip Wahle, and Terry Ruas, Bela Gipp, Florian Wunderlich were partially supported by the Lower Saxony Ministry of Science and Culture and the VW Foundation.
Meriem Beloucif acknowledges Nationella Språkbanken (the Swedish National Language Bank) and Swe-CLARIN – jointly funded by the Swedish Research Council (2018–2024; dnr 2017-00626) and its 10 partner institutions for funding the Swedish annotations.
Rahmad Mahendra acknowledges the funding by Hibah Riset Internal Faculty of Computer Science, Universitas Indonesia under contract number: NKB-13/UN2.F11.D/HKP.05.00/2024 for supporting the annotation of the Indonesian, Javanese, and Sundanese datasets.
Alexander Panchenko would like to thank Nikolay Ivanov, Artem Vazhentsev, Mikhail Salnikov, Maria Marina, Vitaliy Protasov, Sergey Pletnev, Daniil Moskovskiy, Vasilii Konovalov, Elisey Rykov, and Dmitry Iarosh for their help with the annotation for Russian. Preparation of Tatar data was funded by AIRI and completed by Dina Abdullina, Marat Shaehov, and Ilseyar Alimova.
Yi Zhou would like to thank Gaifan Zhang, Bing Xiao, and Rui Qin for their help with the annotations and for providing feedback.
Daryna Dementieva and Nikolay Babakov would like to acknowledge Toloka.ai for the research annotation grant. Daryna Dementieva would like as well to acknowledge Alexander Fraser, the head of Data Statistics and Analytics Group at Technical University of Munich, for the support.

References

Felermino Ali, Henrique Lopes Cardoso, and Rui Sousa Silva. 2024. Building resources for emakhuwa: machine translation and news classification benchmarks. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

Magda B Arnold. 1960. Emotion and personality. vol. i. psychological aspects.

Etienne Barnard, Marelle H Davel, Charl van Heerden, Febe De Wet, and Jaco Badenhorst. 2014. The nchlt speech corpus of the south african languages. Workshop Spoken Language Technologies for Under-resourced Languages (SLTU).

L.F. Barrett. 2017. How Emotions are Made: The Secret Life of the Brain. Expert Thinking Series. Macmillan.

Lisa Feldman Barrett. 2016. The theory of constructed emotion: an active inference account of interoception and categorization. Social Cognitive and Affective Neuroscience, 12(1):1–23.

Federico Bianchi, Debora Nozza, and Dirk Hovy. 2021. FEEL-IT: Emotion and sentiment classification for the Italian language. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 76–83, Online. Association for Computational Linguistics.
Federico Bianchi, Debora Nozza, and Dirk Hovy. 2022. Xlm-emo: Multilingual emotion prediction in social media text. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 195–203.

Kateryna Bobrovyk. 2019. Automated building and analysis of ukrainian twitter corpus for toxic text detection. In COLINS 2019. Volume II: Workshop.

Ankush Chatterjee, Kedhar Nath Narahari, Meghna Joshi, and Puneet Agrawal. 2019. SemEval-2019 task 3: EmoContext contextual emotion detection in text. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 39–48, Minneapolis, Minnesota, USA. Association for Computational Linguistics.

Alexandra Ciobotaru, Mihai Vlad Constantinescu, Liviu P. Dinu, and Stefan Dumitrescu. 2022. RED v2: Enhancing RED dataset for multi-label emotion detection. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1392–1399, Marseille, France. European Language Resources Association.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhichong Shao, Zhihuo Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, Online. Association for Computational Linguistics.

Paul Ekman. 1992. Are there basic emotions?

PC Ellsworth. 2013. Appraisal theory: old and new questions. emot. rev. 5, 125–131.

Nico H Frijda. 1986. The emotions. Studies in Emotion and Social Interaction.

Shreya Havaldar, Bhumika Singhal, Sunny Rai, Langchen Liu, Sharath Chandra Guntuku, and Lyle Ungar. 2023. Multilingual language models are not multicultural: A case study in emotion. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 202–214, Toronto, Canada. Association for Computational Linguistics.

Daniel Hershcovitch, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fiore, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6997–7013, Dublin, Ireland. Association for Computational Linguistics.

MD Asif Iqbal, Avishek Das, Omar Sharif, Mohammed Moshiul Hoque, and Iqbal H Sarker. 2022. Bemoc: A corpus for identifying emotion in bengali texts. SN Computer Science, 3(2):135.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088.

Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of The 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), San Diego, California.

Irina Krylova, Boris Orekhov, Ekaterina Stepanova, and Lyudmila Zaydelman. 2016. Languages of russia: Using social networks to collect texts. Information Retrieval: 9th Russian Summer School, RuSSIR 2015, Saint Petersburg, Russia, August 24-28, 2015, Revised Selected Papers 9, pages 179–185.

Shivani Kumar, Anubhav Shrimal, Md Shad Akhtar, and Tanmoy Chakraborty. 2022. Discovering emotion and reasoning its flip in multi-party conversations using masked memory network and transformer. Knowledge-Based Systems, 240:108112.

Richard S Lazarus. 1991. Emotion and adaptation, volume 557. Oxford University Press.

Youssef Mohamed, Runjia Li, Ibrahim Said Ahmad, Kilchbek Haydarov, Philip Torr, Kenneth Church, and Mohamed Elhoseiny. 2024. No culture left behind: ArtELingo-28, a benchmark of WikiArt with captions in 28 languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20939–20962, Miami, Florida, USA. Association for Computational Linguistics.
Saif Mohammad. 2023. Best practices in the creation and use of emotion lexicons. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1825–1836, Dubrovnik, Croatia. Association for Computational Linguistics.

Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018a. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1–17.

Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018b. SemEval-2018 task 1: Affect in tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana. Association for Computational Linguistics.

Saif Mohammad and Svetlana Kiritchenko. 2018. Understanding emotions: A dataset of tweets to study interactions between affect categories. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Saif M. Mohammad. 2016. 9 - sentiment analysis: Detecting valence, emotions, and other affectual states from text. In Herbert L. Meiselman, editor, Emotion Measurement, pages 201–237. Woodhead Publishing.

Saif M. Mohammad. 2022. Ethics sheet for automatic emotion recognition and sentiment analysis. Preprint, arXiv:2109.08256.

Saif M. Mohammad. 2024. WorryWords: Norms of anxiety association for over 44k English words. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16261–16278, Miami, Florida, USA. Association for Computational Linguistics.

Agnes Moors, Phoebe C Ellsworth, Klaus R Scherer, and Nico H Frijda. 2013. Appraisal theories of emotion: State of the art and future development. Emotion review, 5(2):119–124.

Shamsuddeen Muhammad, Idris Abdulmumin, Abinew Ayele, Nedjma Ousidhoum, David Adelani, Seid Yimam, Ibrahim Ahmad, Meriem Beloucif, Saif Mohammad, Sebastian Ruder, Ouaimma Hourrane, Ali-pio Jorge, Pavel Brazdil, Felermino Ali, Davis David, Salomey Osei, Bello Shehu-Bello, Falalu Lawan, Tajudeen Gwadabe, and 8 others. 2023a. AfriSenti: A Twitter sentiment analysis benchmark for African languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13968–13981, Singapore. Association for Computational Linguistics.

Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Seid Muhie Yimam, David Ifeoluwa Adelani, Ibrahim Sa'id Ahmad, Nedjma Ousidhoum, Abinew Ayele, Saif M. Mohammad, Meriem Beloucif, and Sebastian Ruder. 2023b. SemEval-2023 task 12: Sentiment analysis for african languages (AfriSenti-SemEval). In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023). Association for Computational Linguistics.

Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Seid Muhie Yimam, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine De Kock, Tadesse Destaw Belay, Ibrahim Said Ahmad, Nirmal Surange, Daniela Teodorescu, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino Ali, Vladimir Araujo, Abinew Ali Ayele, Oana Ignat, Alexander Panchenko, and 2 others. 2025. SemEval-2025 task 11: Bridging the gap in text-based emotion detection. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Vienna, Austria. Association for Computational Linguistics.

Emily Öhman, Marc Päimies, Kaisla Kajava, and Jörg Tiedemann. 2020. Xed: A multilingual dataset for sentiment analysis and emotion detection. arXiv preprint arXiv:2011.01612.

Andrew Ortony, Gerald L Clore, and Allan Collins. 2022. The cognitive structure of emotions. Cambridge university press.

Jessica Ouyang and Kathleen McKeown. 2015. Modeling reportable events as turning points in narrative. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2149–2158.

Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. 2022. Potato: The portable text annotation tool. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.

Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682.

Flor Miriam Plaza-del Arco, Alba A. Cercas Curry, Amanda Cercas Curry, and Dirk Hovy. 2024. Emotion analysis in NLP: Trends, gaps and roadmap for future directions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 5696–5710, Torino, Italia. ELRA and ICCL.

Robert Plutchik. 1980. Chapter 1 - a general psychoevolutionary theory of emotion. In Robert Plutchik and Henry Kellerman, editors, Theories of Emotion, pages 3–33. Academic Press.

Ira J Roseman. 2013. Appraisal in the emotion system: Coherence in strategies for coping. Emotion Review, 5(2):141–149.
Alieh Hajizadeh Saffar, Tiffany Katharine Mann, and Bahadorreza Oofghi. 2023. Textual emotion detection in health: Advances and applications. Journal of Biomedical Informatics, 137:104258.

Mei Silviana Saputri, Rahmad Mahendra, and Mirna Adriani. 2018. Emotion classification on indonesian twitter dataset. In 2018 International Conference on Asian Language Processing (IALP), pages 90–95.

Klaus R Scherer. 2009. The dynamic architecture of emotion: Evidence for the component process model. Cognition and emotion, 23(7):1307–1351.

Armin Seyeditbari, Narges Tabari, and Wlodek Zadrożny. 2018. Emotion detection in text: a review. arXiv preprint arXiv:1806.00674.

Språkbanken Text. 2024. Svensk absabank.

Carlo Strapparava and Rada Mihalcea. 2007. SemEval-2007 task 14: Affective text. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 70–74, Prague, Czech Republic. Association for Computational Linguistics.

Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2025. Label Studio: Data labeling software. Open source software available from https://github.com/HumanSignal/label-studio.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Krishnapriya Vishnubhotla, Daniela Teodorescu, Mallory J Feldman, Kristen Lindquist, and Saif M. Mohammad. 2024. Emotion granularity from text: An aggregate-level indicator of mental health. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19168–19185, Miami, Florida, USA. Association for Computational Linguistics.

Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39:165–210.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2-5 technical report. arXiv preprint arXiv:2412.15115.

Yuan Zhuang, Tianyu Jiang, and Ellen Riloff. 2024. My heart skipped a beat! recognizing expressions of embodied emotion in natural language. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3525–3537.
A  PLMs and LLMs Used

A.1  PLMs
1. https://huggingface.co/google-bert/bert-base-multilingual-cased
2. https://huggingface.co/FacebookAI/xlm-roberta-large
3. https://huggingface.co/microsoft/mdeberta-v3-base
4. https://huggingface.co/sentence-transformers/LaBSE
5. https://huggingface.co/microsoft/infoxlm-large
6. https://huggingface.co/google/rembert

A.2  LLMs
1. https://huggingface.co/databricks/dolly-v2-12b
2. https://huggingface.co/meta-llama/Meta-Llama-3.3-70B
3. https://huggingface.co/Qwen/Qwen2.5-72B-Instruct
4. https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
5. https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B

B  Data sources

• afr: Speeches from Barnard et al. (2014).

• arq: Manually translated novel (La Grande Maison by the Algerian author Mohammed Dib).

• ary, hau, ibo, kin, pcm, swa, xho, zul: Afrisenti Muhammad et al. (2023a) and BBC news headlines.

• chn: Weibo dataset https://github.com/aoguai/WeiboHotListDataSet?tab=readme-ov-file.

• deu: Anonymised Reddit data from nine German-language subreddits: de, einfach_posten, FragReddit, beziehungen, schwanger, de_IAmA, germany, depression_de, Lagerfeuer.

• eng: Personal narratives from the AskReddit subreddit collected by Ouyang and McKeeown (2015) and instances from Zhuang et al. (2024).

• esp: YouTube comments from Latin American (i.e., Ecuadorian, Colombian, and Mexican) channels across three genres: News/Politics, Entertainment, Education.

• hin, mar: Newly created emotion dataset. Most instances were manually drafted, while some were generated using ChatGPT.

• ind, jav, sun: YouTube comments from Indonesian videos.

• ron: Data from the subreddit r/Romania, YouTube, and tweets from Ciobotaru et al. (2022).

• rus: Russian Twitter corpus https://study.mokoron.com.

• swe: Sentiment dataset from the Swedish data bank (Språkbanken Text, 2024).

• tat: Instances from Krylova et al. (2016).

• vmw, ptmz: News headlines from Ali et al. (2024).

• yor: News data from BBC Yorùbá and Alaroye. https://alaroye.org/.

C  Annotation

C.1  Annotation Guidelines and Definitions
This is a guide for annotating text for emotion classification. The purpose of this study is to analyze the emotions expressed in a text. It is important to note that emotions can often be inferred even if they are not explicitly stated.

Task  The task involves classifying text into pre-defined emotion categories. The annotated dataset will be used for training emotion classification models and studying how emotions are conveyed through language.

Emotion Categories  We categorize emotions into the following seven classes:
Joy
• Definition: Expressions of happiness, pleasure, or contentment.
• Example: "I just passed my exams!"

Sadness

• Definition: Expressions of unhappiness, sorrow, or disappointment.

• Example: "I miss my family so much. It's been a tough year."

Anger

• Definition: Expressions of frustration, irritation, or rage.

• Example: "Why is the internet so slow today?!"

Fear

• Definition: Expressions of anxiety, apprehension, or dread.

• Example: "There's a huge storm coming our way. I hope everyone stays safe."

Surprise

• Definition: Expressions of astonishment or unexpected events.

• Example: "I can't believe he just proposed to me!"

Disgust

• Definition: A reaction to something offensive or unpleasant.

• Examples: "That video was sickening to watch."

Neutral

• Definition: Texts that do not express any of the above emotions.

• Example: "The weather today is sunny with a chance of rain."

Note: Factual statements can indicate an emotional state without explicitly stating it. For example:

• "An earthquake today killed hundreds of people in my home town."

Surprise differs from joy in that it represents an unexpected event, which may or may not be associated with happiness.

Emotion Description Categories   The following list provides a broader categorization of emotions by including synonyms and related emotional states.

Anger

• Includes: irritated, annoyed, aggravated, indignant, resentful, offended, exasperated, livid, irate, etc.

Sadness

• Includes: melancholic, despondent, gloomy, heartbroken, longing, mourning, dejected, downcast, disheartened, dismayed, etc.

Fear

• Includes: frightened, alarmed, apprehensive, intimidated, panicky, wary, dreadful, shaken, etc.

Happiness

• Includes: joyful, elated, content, cheerful, blissful, delighted, gleeful, satisfied, ecstatic, upbeat, pleased, etc.

Surprise

• Includes: taken aback, bewildered, astonished, amazed, startled, stunned, shocked, dumbstruck, confounded, stupefied, etc.

Joy

• Includes: happiness, delight, elation, pleasure, excitement, cheerfulness, bliss, euphoria, contentment, jubilation.

C.1.1 Emotion Intensity
After selecting the emotion category, annotators were further asked to select the intensity label, which could be: 0: No Emotion, 1 - Slight Emotion, 2: Moderate Emotion and 3: High Emotion. The following examples illustrate different levels of emotion intensity.

Anger

• No Anger: "I walked through the empty streets, the quiet hum of the city like a distant whisper."

• Slight Anger: "The buzz of voices around me blended into a monotonous drone, failing to distract from the pang of annoyance at the delay."
• High Anger: "When his friend’s brother knocked on the door, he was greeted with a shotgun blast through the door, which left him dead at the doorstep."

C.2 Pilot Annotation
We run a pilot annotation on different languages to further refine our guidelines. This has mainly led to clarifications related to the labeling process. For instance, the annotators were reminded that they should select all the labels that apply for a given text snippet, and that one label can encompass more than one specific emotion (e.g., in arq, we explained that a complex perceived emotion such as bitterness or jealousy might involve both anger and sadness).

C.3 Formula for Determining Final Labels
Aggregating emotion labels  Aggregating emotion labels can be formally expressed as:

\[
L_{\text{final}} = \begin{cases}
1, & \text{if Count(1,2,3) } \geq 2 \text{ and AvgScore} > T, \\
0, & \text{otherwise.}
\end{cases}
\]
Count(1,2,3) = \( \sum_{i=1}^N \mathbb{I}(A_i \in \{1,2,3\}) \), \( \text{AvgScore} = \frac{1}{N} \sum_{i=1}^N A_i \)

Where:
• \( A_i \) is the rating provided by annotator \( i \).
• \( N \) is the total number of annotators.
• \( \mathbb{I}(A_i \in \{1,2,3\}) \) Membership function that returns 1 if \( A_i \in \{1,2,3\} \), and 0 otherwise.
• \( T \) is the threshold for the average score, which we set as \( T = 0.5 \)

Aggregating intensity  Aggregating intensity can be formally expressed as:

\[
\text{AvgScore} = \frac{\sum_{i=1}^N A_i}{N},
\]
\[
L_{\text{final}} = \begin{cases}
0, & \text{if } 0 \leq \text{AvgScore} < 1, \\
1, & \text{if } 1 \leq \text{AvgScore} < 2, \\
2, & \text{if } 2 \leq \text{AvgScore} < 3, \\
3, & \text{if } \text{AvgScore} = 3.
\end{cases}
\]

Where:
• \( A_i \) is the intensity score provided by annotator \( i \), where \( A_i \in \{0,1,2,3\} \).
• \( N \) is the total number of annotators.

D  SCHMP Calculation
The computation of SHCMP involves the following steps:

1. Random Splitting with Tie-Breaking  The dataset of \( N \) annotated items is randomly divided into two equal subsets, \( A_1 \) and \( A_2 \). For datasets with an odd number of annotations, probabilistic tie-breaking is applied to ensure balanced splits.

2. Class Assignment  For each item \( x_i \ (i = 1, 2, \ldots, N) \):
   • Assign \( x_i \) a score based on its annotations in \( A_1 \) and \( A_2 \).
   • Let \( C_1(x_i) \) and \( C_2(x_i) \) denote the class of \( x_i \) derived from \( A_1 \) and \( A_2 \), respectively.

3. Class Binning  To manage continuous scores, divide the range of possible scores \([ -3, 3 ]\) into equal-sized bins, where the bin size \( b \) is determined as:
\[
b = \frac{6}{\# \text{Bins}}
\]
Scores from \( A_1 \) and \( A_2 \) are then assigned to their respective bins, denoted as \( c_1 \) and \( c_2 \).

4. Match Calculation  Define a match indicator \( M(x_i) \) to evaluate consistency for each item:
\[
M(x_i) = \begin{cases}
1, & \text{if } |c_1 - c_2| < 1, \\
0, & \text{otherwise.}
\end{cases}
\]
This ensures that items are considered consistent if their scores fall into the same bin or adjacent bins.

5. Proportion of Matches  Compute the total number of matches, \( N_{\text{match}} \), across all items:
\[
N_{\text{match}} = \sum_{i=1}^N M(x_i).
\]

6. SHCMP Computation  The SHCMP score is calculated as the proportion of matches, expressed as a percentage:
\[
\text{SHCMP (\%)} = \frac{N_{\text{match}}}{N} \times 100.
\]
7. Averaging   We repeat the process \( k \) times with different random splits and compute the average SHCMP score:

\[
\mathrm{SHCMP}_{\text{final}} = \frac{1}{k} \sum_{j=1}^{k} \mathrm{SHCMP}_j,
\]

where \( \mathrm{SHCMP}_j \) is the SHCMP score from the \( j \)-th split.

E  Experimental Settings

For LLMs, we used the default parameters from HuggingFace except for temperature which we set to 0 for deterministic output and top-k is set to 1. Only for the top-k ablations in which top-k > 1 in Figure 5c, we set temperature to 0.7. We ask all LLMs to perform CoT. We trained on the train set for 2 epochs with a learning rate of 1e-5 and evaluated on test set. For MLMs experiments, we trained on the training set for 2 epochs with a learning rate of 1e-5 and evaluated on the test set.
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="3">Train Set (%)</th>
    <th colspan="3">Development Set (%)</th>
    <th colspan="3">Test Set (%)</th>
  </tr>
  <tr>
    <th>Single</th>
    <th>Multi</th>
    <th>Neutral</th>
    <th>Single</th>
    <th>Multi</th>
    <th>Neutral</th>
    <th>Single</th>
    <th>Multi</th>
    <th>Neutral</th>
  </tr>
  <tr>
    <td>chn</td>
    <td>54.00</td>
    <td>23.74</td>
    <td>22.26</td>
    <td>53.60</td>
    <td>23.58</td>
    <td>22.82</td>
    <td>53.90</td>
    <td>24.30</td>
    <td>21.80</td>
  </tr>
  <tr>
    <td>sun</td>
    <td>58.94</td>
    <td>36.18</td>
    <td>4.88</td>
    <td>59.09</td>
    <td>36.26</td>
    <td>4.65</td>
    <td>59.40</td>
    <td>36.07</td>
    <td>4.54</td>
  </tr>
  <tr>
    <td>afr</td>
    <td>47.79</td>
    <td>6.69</td>
    <td>45.52</td>
    <td>56.14</td>
    <td>7.86</td>
    <td>36.01</td>
    <td>37.39</td>
    <td>10.35</td>
    <td>52.26</td>
  </tr>
  <tr>
    <td>swe</td>
    <td>43.16</td>
    <td>16.60</td>
    <td>40.24</td>
    <td>46.30</td>
    <td>20.37</td>
    <td>33.33</td>
    <td>42.76</td>
    <td>18.81</td>
    <td>38.43</td>
  </tr>
  <tr>
    <td>swa</td>
    <td>41.67</td>
    <td>3.33</td>
    <td>55.00</td>
    <td>45.78</td>
    <td>3.56</td>
    <td>50.66</td>
    <td>46.26</td>
    <td>3.81</td>
    <td>49.93</td>
  </tr>
  <tr>
    <td>esp</td>
    <td>61.02</td>
    <td>38.98</td>
    <td>0.00</td>
    <td>65.22</td>
    <td>34.78</td>
    <td>0.00</td>
    <td>65.14</td>
    <td>34.86</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>arq</td>
    <td>28.53</td>
    <td>50.05</td>
    <td>9.42</td>
    <td>28.57</td>
    <td>50.00</td>
    <td>10.71</td>
    <td>27.95</td>
    <td>44.76</td>
    <td>8.35</td>
  </tr>
  <tr>
    <td>ptbr</td>
    <td>52.11</td>
    <td>13.80</td>
    <td>34.09</td>
    <td>61.06</td>
    <td>11.82</td>
    <td>27.12</td>
    <td>52.68</td>
    <td>13.59</td>
    <td>33.73</td>
  </tr>
  <tr>
    <td>ptmz</td>
    <td>52.00</td>
    <td>0.44</td>
    <td>47.56</td>
    <td>50.92</td>
    <td>0.37</td>
    <td>48.71</td>
    <td>53.03</td>
    <td>0.51</td>
    <td>46.45</td>
  </tr>
  <tr>
    <td>ukr</td>
    <td>44.77</td>
    <td>2.24</td>
    <td>52.99</td>
    <td>47.24</td>
    <td>2.36</td>
    <td>50.39</td>
    <td>45.23</td>
    <td>1.79</td>
    <td>52.98</td>
  </tr>
  <tr>
    <td>mar</td>
    <td>67.69</td>
    <td>8.56</td>
    <td>23.75</td>
    <td>68.57</td>
    <td>7.62</td>
    <td>23.81</td>
    <td>68.94</td>
    <td>9.33</td>
    <td>21.73</td>
  </tr>
  <tr>
    <td>rus</td>
    <td>64.63</td>
    <td>11.08</td>
    <td>24.29</td>
    <td>66.35</td>
    <td>12.23</td>
    <td>21.42</td>
    <td>66.91</td>
    <td>12.89</td>
    <td>20.20</td>
  </tr>
  <tr>
    <td>ibo</td>
    <td>72.44</td>
    <td>3.63</td>
    <td>23.93</td>
    <td>61.12</td>
    <td>10.91</td>
    <td>27.97</td>
    <td>73.61</td>
    <td>3.97</td>
    <td>22.42</td>
  </tr>
  <tr>
    <td>amh</td>
    <td>50.82</td>
    <td>27.68</td>
    <td>21.50</td>
    <td>56.13</td>
    <td>30.31</td>
    <td>16.56</td>
    <td>48.50</td>
    <td>24.67</td>
    <td>26.83</td>
  </tr>
  <tr>
    <td>deu</td>
    <td>41.78</td>
    <td>34.05</td>
    <td>24.17</td>
    <td>41.84</td>
    <td>35.19</td>
    <td>22.97</td>
    <td>41.23</td>
    <td>32.10</td>
    <td>26.66</td>
  </tr>
  <tr>
    <td>vmw</td>
    <td>52.80</td>
    <td>0.45</td>
    <td>46.75</td>
    <td>53.49</td>
    <td>0.39</td>
    <td>46.12</td>
    <td>53.46</td>
    <td>0.52</td>
    <td>46.32</td>
  </tr>
  <tr>
    <td>pcm</td>
    <td>55.00</td>
    <td>40.46</td>
    <td>4.54</td>
    <td>50.00</td>
    <td>36.63</td>
    <td>4.37</td>
    <td>51.57</td>
    <td>38.08</td>
    <td>4.35</td>
  </tr>
  <tr>
    <td>eng</td>
    <td>38.64</td>
    <td>47.02</td>
    <td>14.34</td>
    <td>34.07</td>
    <td>42.22</td>
    <td>9.70</td>
    <td>38.58</td>
    <td>48.76</td>
    <td>10.34</td>
  </tr>
  <tr>
    <td>hin</td>
    <td>66.35</td>
    <td>10.80</td>
    <td>22.85</td>
    <td>60.40</td>
    <td>7.92</td>
    <td>31.68</td>
    <td>77.31</td>
    <td>5.66</td>
    <td>13.92</td>
  </tr>
  <tr>
    <td>tat</td>
    <td>81.48</td>
    <td>0.00</td>
    <td>18.52</td>
    <td>84.00</td>
    <td>0.00</td>
    <td>16.00</td>
    <td>85.71</td>
    <td>0.00</td>
    <td>14.29</td>
  </tr>
</table>

Table 4: Percentage distribution of SingleLabel, MultiLabel, and NeutralLabel for the Train, Development, and Test Sets.
<table>
  <tr>
    <th rowspan="2">Lang.</th>
    <th colspan="5">Monolingual Multi-Label Classification</th>
  </tr>
  <tr>
    <th>LaBSE</th>
    <th>RemBERT</th>
    <th>XLM-R</th>
    <th>mBERT</th>
    <th>mDeBERTa</th>
  </tr>
  <tr><td>afr</td><td>30.76</td><td><b>37.14</b></td><td>10.82</td><td>25.87</td><td>16.66</td></tr>
  <tr><td>arq</td><td><b>45.46</b></td><td>41.41</td><td>31.98</td><td>41.75</td><td>29.68</td></tr>
  <tr><td>ary</td><td>45.81</td><td><b>47.16</b></td><td>40.66</td><td>36.87</td><td>38.00</td></tr>
  <tr><td>chn</td><td>53.47</td><td>53.08</td><td><b>58.48</b></td><td>49.61</td><td>44.47</td></tr>
  <tr><td>deu</td><td>55.02</td><td><b>64.23</b></td><td>55.37</td><td>46.78</td><td>44.09</td></tr>
  <tr><td>eng</td><td>64.24</td><td><b>70.83</b></td><td>67.30</td><td>58.26</td><td>58.94</td></tr>
  <tr><td>esp</td><td>72.88</td><td><b>77.44</b></td><td>29.85</td><td>54.41</td><td>60.17</td></tr>
  <tr><td>hau</td><td>58.49</td><td><b>59.55</b></td><td>36.95</td><td>47.33</td><td>48.59</td></tr>
  <tr><td>hin</td><td>75.25</td><td><b>85.51</b></td><td>33.71</td><td>54.11</td><td>54.34</td></tr>
  <tr><td>ibo</td><td>45.90</td><td><b>47.90</b></td><td>18.36</td><td>37.23</td><td>31.92</td></tr>
  <tr><td>ind</td><td>--</td><td>--</td><td>--</td><td>--</td><td>--</td></tr>
  <tr><td>jav</td><td>--</td><td>--</td><td>--</td><td>--</td><td>--</td></tr>
  <tr><td>kin</td><td><b>50.64</b></td><td>46.29</td><td>32.93</td><td>35.61</td><td>38.00</td></tr>
  <tr><td>mar</td><td>80.76</td><td><b>82.20</b></td><td>78.95</td><td>60.01</td><td>66.01</td></tr>
  <tr><td>pcm</td><td>51.30</td><td><b>55.50</b></td><td>52.03</td><td>48.42</td><td>46.21</td></tr>
  <tr><td>ptbr</td><td><b>42.60</b></td><td>42.57</td><td>15.40</td><td>32.05</td><td>24.08</td></tr>
  <tr><td>ptmz</td><td>36.95</td><td><b>45.91</b></td><td>30.72</td><td>14.81</td><td>21.89</td></tr>
  <tr><td>ron</td><td>69.79</td><td><b>76.23</b></td><td>65.21</td><td>61.50</td><td>60.60</td></tr>
  <tr><td>rus</td><td>75.62</td><td><b>83.77</b></td><td>78.76</td><td>61.81</td><td>54.79</td></tr>
  <tr><td>sun</td><td>36.93</td><td><b>37.31</b></td><td>19.66</td><td>27.88</td><td>21.65</td></tr>
  <tr><td>swa</td><td><b>27.53</b></td><td>22.65</td><td>22.71</td><td>22.99</td><td>22.84</td></tr>
  <tr><td>swe</td><td>49.23</td><td><b>51.98</b></td><td>34.63</td><td>44.24</td><td>40.90</td></tr>
  <tr><td>tat</td><td><b>57.71</b></td><td>53.94</td><td>26.48</td><td>43.49</td><td>35.02</td></tr>
  <tr><td>ukr</td><td>50.07</td><td><b>53.45</b></td><td>17.77</td><td>31.74</td><td>28.55</td></tr>
  <tr><td>vmy</td><td><b>21.13</b></td><td>12.14</td><td>9.92</td><td>10.28</td><td>11.13</td></tr>
  <tr><td>xho</td><td>--</td><td>--</td><td>--</td><td>--</td><td>--</td></tr>
  <tr><td>yor</td><td><b>32.55</b></td><td>9.22</td><td>11.94</td><td>21.03</td><td>17.88</td></tr>
  <tr><td>zul</td><td>--</td><td>--</td><td>--</td><td>--</td><td>--</td></tr>
</table>

Table 5: **Average F1-Macro for monolingual multi-label emotion classification**. Each model is trained and evaluated within the same language. The best results are highlighted in blue.
<table>
  <tr>
    <th>Prompt Version</th>
    <th>Prompt Text</th>
  </tr>
  <tr>
    <td>Prompt v1</td>
    <td>Evaluate whether the following text conveys the emotion of {{EMOTION}}.<br>Think step by step before you answer.<br>Finish your response with 'Therefore, my answer is ' followed by 'yes' or 'no':<br><br>{{INPUT}}</td>
  </tr>
  <tr>
    <td>Prompt v2</td>
    <td>Analyze the text below for the presence of {{EMOTION}}.<br>Explain your reasoning briefly and conclude with 'Answer:' followed by either 'yes' or 'no'.<br><br>{{INPUT}}</td>
  </tr>
  <tr>
    <td>Prompt v3</td>
    <td>Examine the following text to determine whether {{EMOTION}} is present.<br>Provide a concise explanation for your assessment and end with 'Answer:' followed by either 'yes' or 'no'.<br><br>{{INPUT}}</td>
  </tr>
</table>

Table 6: The prompt variants used in the monolingual emotion recognition ablation study.
Track A: Example Few-Shot Prompt

### Task: ###
Analyze the text below for the presence of anger.
Explain your reasoning briefly and conclude with ’Answer:’ followed by either ’yes’ or ’no’.

### Examples: ###
Example 1:
Input: ””When I answered the phone, my heart beat extremely fast... I was very nervous!””
Answer: no

Example 2:
Input: ””I’ll never forget how businesslike and calm the Israeli guy was.””
Answer: no

Example 3:
Input: ””I wake up, my eyes fluttering open to a shield of darkness.””
Answer: no

Example 4:
Input: ””I lay in a large bed, the sheets and quilt pulled up to my chin, and the curtains were drawn to keep out the light.””
Answer: no

Example 5:
Input: ””Either way that idiot is gone.””
Answer: yes

Example 6:
Input: ””Seriously... did I really just shut my finger in the car door.””
Answer: yes

Example 7:
Input: ””I was really uncomfortable because I was sitting behind my dad and there isn’t enough room for my legs.””
Answer: yes

Example 8:
Input: ””He damn disturb plz, cover my head with a shirt that a customer which have body odour just tried on!!!”
Answer: yes

### Your Turn: ###
Input: ””/ o So today I went in for a new exam with Dr. Polvi today, I had to file new paperwork for the automobile accident case which is being done differently than the scoliosis stuff. So he comes in and starts talking about insurance stuff and how this looks bad since I was getting treatment on my neck and stuff already blah blah.””
Track B: Example Few-Shot Prompt

### Task: ###
In this task, you will assess the level of anger in a given text (0 = none, 1 = low, 2 = medium, 3 = high). Summarize your reasoning and conclude with 'Answer:' followed by the correct number.

### Examples: ###
Example 1:
Input: '"I try extremely hard to keep my details hidden. It was nice to know that what I had given people to know was pleasant, but I couldn’t deny the knot that was still in my stomach."'
Answer: 0

Example 2:
Input: '"I knew we were almost there when my midwife’s voice got more excited and Joey leaned in real close and said into my ear, “Don’t stop pushing!”"'
Answer: 0

Example 3:
Input: '"One ended up going to prison."'
Answer: 1

Example 4:
Input: '"Not to mention noisy."'
Answer: 1

Example 5:
Input: '"but Urban Dictionary confirmed Spook is indeed a racial slur."'
Answer: 2

Example 6:
Input: '"And..at his funeral, they fired him!"'
Answer: 2

Example 7:
Input: '"I ended up metaphorically throwing my hands in the air in disgust and just cancelling my account altogether."'
Answer: 3

Example 8:
Input: '"He would manipulate me into it and I was extremely upset."'
Answer: 3

### Your Turn: ###
Input: '"So today I went in for a new exam with Dr. Polvi today, I had to file new paperwork for the automobile accident case which is being done differently than the scoliosis stuff. So he comes in and starts talking about insurance stuff and how this looks bad since I was getting treatment on my neck and stuff already blah blah."'

Figure 8: Example of the few-shot prompt template for assessing anger in Track B.

Reviewer paper summaries: ## paper summary
This paper addresses the scarcity of high-quality emotion recognition datasets for low-resource languages. It introduces BRIGHTER, a multilingual, multi-labeled dataset covering 28 languages, including many under-resourced ones in Africa, Asia, and Latin America. By detailing data collection, annotation processes, and reliability measures, the authors highlight the complexity of multilingual emotion detection. Experiments using multilingual transformers and large language models reveal significant performance gaps, especially for low-resource languages. The work underscores both the importance of dedicated datasets and the need for improved modeling strategies in cross-lingual emotion recognition.

The paper presents BRIGHTER, a collection of manually annotated, multi-labeled emotion datasets covering 28 languages, with a particular focus on low-resource languages from regions such as Africa, Asia, Eastern Europe, and Latin America (of the 28 languages, 15 to 17 languages could be classified low-resource.). They underline that most existing multilingual resources for emotion classification rely on translated data that risk losing cultural nuances critical for understanding emotion. BRIGHTER overcomes this by using original text data and involving fluent native speakers in the annotation process to better capture cultural specificities.

The dataset comprises nearly 100,000 instances in total, from diverse domains including social media, news articles, literature, speeches and reviews. Each text instance is annotated by fluent speakers for six emotion categories (joy, sadness, anger, fear, surprise, disgust) or and None, together with an additional intensity level ranging from 0 to 3.

The paper details the data collection, annotation, and quality control processes. The authors then present monolingual and cross-lingual experiments for multi-label emotion recognition, as well as intensity-level emotion identification. They use both supervised (either monolingual or cross-lingual) models as well as few-shot LLM-prompting methods and some additional analysis, such as the dependence of prompt formatting and number of few-shot examples.

Their findings reveal that large language models (LLMs) struggle with recognizing emotions in text, especially for low-resource languages, although performance improves when prompts are given in English. By making the BRIGHTER dataset publicly available, the authors aim to support future research in emotion recognition and reduce disparities for low-resource languages.

This paper constructed BRIGHTER, a collection of emotion recognition datasets in 28 languages. The dataset has been released in huggingface: https://huggingface.co/datasets/brighter-dataset/BRIGHTER-emotion-intensities/tree/main

The samples in BRIGHTER are multi-labeled, collected and annotated by fluent speakers, with 10 datasets annotated for emotion intensities which are helpful to predict and evaluate the "density" of emotions.
The datasets for under-resourced languages are helpful for global researchers.

Experiments show that LLMs' scores are highly dependent on the wording of the prompts, the specfic language and the number of shots in few-shot settings.

This paper is open in arxiv and hf: https://huggingface.co/papers/2502.11926
Summary of strengths: ## Summary Of Strengths
1. Multi-label Annotation: By allowing multiple concurrent emotions to be labeled for each instance, the dataset captures the nuanced and often overlapping nature of emotional expression in text.
2. Broad Language Family Coverage: Spanning 28 languages across various language families, this dataset encourages more inclusive NLP research and facilitates cross-lingual comparisons.
3. Large-Scale Low-Resource Data: By focusing on underrepresented languages, the paper significantly expands available resources where they are most needed, bridging a critical gap in multilingual emotion analysis.
4. Native Speaker Annotations: Leveraging fluent or native speakers ensures higher cultural and linguistic authenticity, thus enhancing the reliability and quality of the annotated data.

* Overall, the paper is engaging, well-structured, and clearly written.
* It introduces a valuable resource for emotion classification, covering both low-resource and middle- to high-resource languages.
* The paper reflects a large-scale, carefully designed annotation effort, ensuring high-quality data.
* The dataset features multi-label emotion annotations and includes intensity scores (for 10 languages), enabling a broader range of research in emotion analysis.
* It covers diverse domains, enhancing the dataset's applicability across various contexts.
* The paper includes a comprehensive experimental evaluation, with both monolingual and cross-lingual supervised experiments for emotion classification and intensity-level prediction, as well as few-shot LLM results.
* It offers valuable insights into the limitations of large language models (LLMs) in recognizing emotions, particularly in low-resource language contexts.

1. a good dataset and published to the public with 28 languages, in which 10 datasets are also annotated with emotion intensities.
2. detailed results on sentiment analysis classifiers trained with LLMs (such as qwen2.5-72b and llama-70b distilled by deepseek-r1) using these 28 language included dataset.
Summary of weaknesses: ## Summary Of Weakness
1. Limited Domain and Context Diversity：
Despite including multiple text sources (social media, news, etc.), it remains unclear whether certain genres or registers are over- or underrepresented. More explicit statements on domain coverage or additional sampling strategies would help readers gauge the dataset’s scope and limitations.
2. Insufficient Cross-Referencing to Related Work：
While the paper references established multilingual benchmarks, it could compare its approach more thoroughly to recent emotion recognition datasets (especially for lower-resource languages) to highlight unique contributions or methodological novelties.

Overall, I find the content of the paper engaging, and I believe the presented resource is a highly valuable contribution. However, my main concern lies in the presentation of the experimental results, which I feel lacks clarity (please see the comments below for further details). Aside from this, I have three general points to raise:

1) Lack of Comparison with Other Datasets: The paper would benefit from a more detailed comparison with existing emotion classification datasets for low-resource or in general multilingual datasets. Also, maybe illustrate the problem of translated emotion datasets a bit more. Both would help to better highlight the unique contribution and strengths of BRIGHTER. 

2) The intensity scores are available for only 10 languages, which could limit the applicability of these scores for a wider range of languages in the dataset.

3) Separation of Cross-Lingual and Monolingual Results: In the main paper, the cross-lingual supervised experiments are presented, while the monolingual results are included in the appendix. Could you explain this choice? I believe a comparison between the two would be valuable and is currently missing from the paper.

1. the scalability of from 28 languages to 100+ languages (such as the cc-100 dataset from meta) are preferred to be discussed.
Comments, suggestions and typos: 1. Consider expanding or rebalancing sources to reduce potential over- or underrepresentation of certain genres and improve overall domain coverage.
2. Expand References

Please put some work into refining the presentation of results! I think that you really did a lot of nice experiments but there are several problems and missing/confusing things, which makes it hard to actually be able to notice that! Here are my comments:

* 074-076: "based on six emotion classes: joy, sadness, anger, fear, surprise, disgust, and none." --> So 7 classes in total. Maybe rephrase this.
* 179-181: Why did you use this novel in particular and only for Algerian Arabic? As you said, this is a mainly spoken language, so there are no novels originally written? This was not clear to me.
* General question: I like the diversity of sources on the one hand, but on the other it makes me wonder if this disparity (some languages only  have social media data, others only news or literature) could play a large role for explaining the differences in model performance?
* 222: typo: I think you need to refer to Appendix A here.
* 288-289: What is meant by "average score" here? The average intensity score? Or the average of assigned emotion label or not? Please be more specific here.
* 332-334: You briefly mention the monolingual results here. However, I believe they would be more appropriately placed alongside the cross-lingual setting, rather than within the few-shot experiments, as they are more closely related to the former (i.e. supervised training, same models). That said, I would appreciate it if the monolingual results were presented more prominently (see below), as I was confused and thought they were completely missing.
* 339: You should shortly name and cite the models you use here (LaBSE, RemBERT, XLM-r, ...).
* 347-349 (Section Emotion Intensity Prediction): This "explanation" is way to short in my opinion. Explain: Do you use prompting or supervised approaches? Regression or classification lassification?
* 361-362: "We observe the largest performance for yor with a maximum of 27.44." I don't understand this. This is not the largest performance but a rather low one. Did you mean weakest? However, there are worse results. Why the particular focus on this number?
* 361-366: This discussion is about the cross-lingual results, correct? Make this more clear, what table/setting you are talking about (also at other paragraphs).
* 371-374: Here you talk about "boosting performance" through cross-lingual training. I find it a bit weird that you compare the cross-lingual performance to the LLM few-shot results and not to the monolingual training? Especially when calling it "boosting".
* Table 2 and general question concerning all the results: You prompt for one emotion (you say "we consider the first answer generated by the LLMs (i.e., top–1)"), correct? So single-label, but your data is multi-labeled? How are the metrics calculated then? The appendix hints at that you prompted for each emotion individually, which would (kind of) explain the set-up. However, this needs to be stated clearly and the metric must be explained.
* Table 2 caption: "The best performance scores in monolingual and zero-shot settings are highlighted in blue and orange, respectively." Here you a) mean "cross-lingual" instead of "monolingual", I suppose and b) you mean "orange and blue" instead of "blue and orange" (or change the order of "cross-lingual" and "zero-shot").
* 392-402: Again, the intensity prediction experiments are not explained enough (prompting, or supervised, mono- or cross-lingual, classification or regression?). Also, you should refer to Table 3 here so help the reader find the results you are talking about.
* Section 5: It would help to quickly name the analyses you are conducting, instead of directly referring to a figure without context for the reader. Describe what modifications are tested in the figure first. So quickly name that you experiment with different prompt formulations, the number of few-shot examples, the number of k for top-k, and the language of the prompt (English vs. target language) before diving into the results.
* 420: "increasing the value of k results consistently in better performance" This is not very surprising, or am I missing something? If you allow more guesses the probability of getting one of them correct rises?
* Table 3: You switched the columns here, compared to the previous results table (Table 2). Similarly for the names of the columns ("Multilingual Language Models (MLMs)" vs. "Crosslingual Multi-Label Classification" and "Large Language Models (LLMs)" vs "Few-Shot Multi-Label Classification". This is confusing, make the order and naming more consistent.
* Figure 5, Section 5: I saw that you have the different prompt paraphrases for this experiment in the appendix. But please refer to it, otherwise the reader does not find it.
* Appendix, 977- 978: "We ask all LLMs to perform CoT." How? Refer to your prompts!
* Appendix, 978- 980: "We trained on the train set for 2 epochs with a learning rate of 1e-5" Which experiments are you talking about here? Mono- or cross-lingual? Both? It's confusing that this sentence comes directly after CoT, so the LLM-experiments.
* Appendix, 978- 980: typo: double "and"

1. better use the full name of "DeepSeek-R1-Distill-Llama-70B" instead of "DeepSeek-R1-70B" considering it is basing on llama-70b.
2. emotion driven long-context generation will possibly be an important application and prefer to learn related discussions of this direction.
Overall assessment scores: 3.5

3.5

3