UWB at WASSA-2024 Shared Task 2: Cross-lingual Emotion Detection (2024)

Abstract: This paper presents our system built for the WASSA-2024 Cross-lingual Emotion Detection Shared Task. The task consists of two subtasks: first, to assess an emotion label from six possible classes for a given tweet in one of five languages, and second, to predict words triggering the detected emotions in binary and numerical formats. Our proposed approach revolves around fine-tuning quantized large language models, specifically Orca 2, with low-rank adapters (LoRA) and multilingual Transformer-based models, such as XLM-R and mT5. We enhance performance through machine translation for both subtasks and trigger word switching for the second subtask. The system achieves excellent performance, ranking 1st in numerical trigger words detection, 3rd in binary trigger words detection, and 7th in emotion detection.

heading: 1 Introduction
text: Analyzing emotions in text, including emotion detection and other related tasks, is a well-studied area in natural language processing (NLP). This field has been extensively explored in various SemEval (Mohammad et al., 2018; Chatterjee et al., 2019) and WASSA (Klinger et al., 2018) competitions. The goal of WASSA-2024 Shared Task 2 (Maladry et al., 2024) is to predict specific emotions and identify the words that trigger these emotions. Additionally, this study investigates how emotional information can be transferred across five languages. While the training data is provided in English, the evaluation data includes English, Dutch, Russian, Spanish, and French.
Specifically, the task consists of two subtasks: 1) Cross-lingual emotion detection: Predicting emotion from six possible classes (Love, Joy, Fear, Anger, Sadness, Neutral) in five target languages.
2) Classifying words that express emotions: Identifying words that trigger emotions, with the output
format being either binary (assigning a binary value to each token in the text) or numeric (assigning a numeric value to each token in the text).
Figure 1 shows an example from the dataset for both subtasks. For detailed label descriptions, see the annotation guidelines (Singh, Pranaydeep and Maladry, Aaron and Lefever, Els, 2023).
The first subtask can be considered a sentencelevel classification task, while the second can be treated as a token-level classification task. Both subtasks can also be viewed as text-generation tasks (Raffel et al., 2023). Learning the representation of a given text is essential for solving these problems. Neural networks, including convolutional neural networks (CNNs) (Kim, 2014) and recurrent neural networks (RNNs) (Zaremba et al., 2015), alongside their variations, have been employed for this purpose. However, while these models are effective, they primarily rely on static word embeddings to capture semantic meanings. Consequently, they may struggle with more complex linguistic features such as anaphora and longterm dependencies.
Recent NLP research has shifted towards Transformer-based pre-trained language models (PLMs), such as BERT (Devlin et al., 2019) and T5 (Raffel et al., 2023). These models undergo initial pre-training on extensive datasets to grasp language representation intricacies. Subsequently, they can be fine-tuned on labelled data, capitalizing on the knowledge acquired during pre-training. For cross-lingual tasks, multilingual pre-trained models such as mT5 (Xue et al., 2021) and XLMRoBERTa (XLM-R) (Conneau et al., 2020) have emerged as standard choices (Hu et al., 2020).
483
Recently, open-source large language models (LLMs), such as LLaMA 2 (Touvron et al., 2023) and Orca 2 (Mitra et al., 2023), have made significant progress across various NLP tasks. These models show remarkable performance in many zeroand few-shot tasks. Nevertheless, they are primarily pre-trained on English, which often necessitates additional fine-tuning to optimize their performance for other languages (Zhang et al., 2023). However, fine-tuning LLMs on non-specialized consumer GPUs presents challenges due to their large number of parameters. Techniques like QLoRA (Dettmers et al., 2023) address this issue by employing a quantized 4-bit frozen backbone LLM with a small set of learnable LoRA weights (Hu et al., 2021).
This paper proposes improving cross-lingual emotion detection by combining a quantized Orca 2 LLM, fine-tuned with LoRA, and machine translation. Additionally, we leverage fine-tuned Transformer-based multilingual language models, such as XLM-R and mT5, for trigger word detection. By incorporating alignment-free translation and trigger word switching, we aim to enhance performance further.1

heading: 2 System Description
text: We conduct experiments using the dataset provided by Maladry et al. (2024), which contains tweets in five different languages.

heading: 2.1 Problem Formulation
text: For both subtasks, the input is a sentence x = {xi}Li=1 containing L tokens. We denote the parameters of the models as Θ, which includes taskspecific parameters W and b. Given the sentencelabel pairs (xS , yS) in the source language S, the aim is to predict labels yT for the sentence xT in the target language T .
We formulate the emotion detection subtask as a text-generation problem, which can be modelled, for example, with decoder-only Transformer-based models. The decoder-only model calculates the probability of generating the next token yt at each step t based on previous outputs y1, . . . , yt−1 as
PΘ(yt|y1, . . . , yt−1) = Dec(y1, . . . , yt−1), (1)
where Dec is the decoder function. 1The code is publicly available at https://github.com/ biba10/UWBWASSA2024SharedTask2.
We formulate the second subtask as a token-level classification problem. Given the feature representations h = {h}Li=1 for each token in the sentence obtained by the model, we apply a linear layer on top to get the label distribution for each token xi as
PΘ(yi|xi) = softmax(Whi + b), (2)
where y ∈ Y = {0, 1}. We select the class with the highest probability. To obtain the numerical values, we extract the logits for class 1, which represents “is a trigger word” class, and apply the softmax function to these logits to get numbers between 0 and 1. We consider only the first token of each word for both binary and numerical values.

heading: 2.2 Label Projection
text: Following related work in cross-lingual classification (Hassan et al., 2022; Zhang et al., 2021), we translate the English training set into all nonEnglish target languages using the Google API2. This approach significantly expands the training data. The process for the emotion detection task is straightforward: we translate the data and retain the original labels.
Creating translated data for the second subtask is more challenging because the translated text may not perfectly align with the original English text, resulting in different token counts between the two. Previous approaches (Mayhew et al., 2017; Fei et al., 2020) often rely on word alignment tools like fastAlign (Dyer et al., 2013) to map tokenlabel pairs from the source sentence to the targeted one. We adopt an alignment-free label projection proposed by Zhang et al. (2021) to create pseudolabelled data in the target languages.
First, we mark each trigger word in a sentence with predefined special symbols, such as “[]” or “{}”, before translating it. We use distinct symbols for a sentence containing multiple trigger words. After translation, we extract the trigger words from the translated sentence using the special symbols. The translation system may occasionally overlook these special symbols, leading to their omission. In such cases, we discard the sentences. This process, illustrated in Figure 2, yields pseudo-labelled sentences in the target language.
Furthermore, we create additional datasets for the second subtask by combining data from different languages, as shown in the lower part of Figure 2. Given an English source sentence xS and its
2https://translate.google.com/
translation xT , we switch the trigger words to construct two new bilingual sentences: the first, xSt, originates from xS with trigger words expressed in the target language; the second, xTs, originates from xT with trigger words in the source language.
We denote the original English dataset as DS , the translated dataset into all four non-English languages as DT , the English source dataset with trigger words in other languages as DSt, and the translated dataset with trigger words in English as DTs.

heading: 2.3 LoRA
text: Fine-tuning LLMs like Orca 2 requires significant computational resources due to the model’s extensive parameter count. To address this challenge, Dettmers et al. (2023) propose to use low-rank adapters (LoRA) (Hu et al., 2021) on top of a quantized backbone model. This method employs a small set of trainable parameters called adapters while keeping the original model frozen, thus reducing memory requirements.
For a pre-trained weight matrix W0, LoRA represents it with a low-rank decomposition as
W0 +∆W = W0 +BA, (3)
where B and A are matrices with much lower dimensions than W0. During fine-tuning, W0 is frozen while the weights of A and B matrices are updated. Figure 3 shows the concept of LoRA.

heading: 2.4 Models
text: We fine-tune the large versions of XLM-R (Conneau et al., 2020) and mT5 (Xue et al., 2021), as
well as the 13B version of Orca 2 (Mitra et al., 2023), using the HuggingFace Transformers library3 (Wolf et al., 2020). The Orca 2 model is used for the first subtask, while the XLM-R model and the encoder part of the mT5 model are employed for the second subtask.

heading: 2.5 Experimental Setup
text: For the initial hyperparameter selection, we sample 10% of the English training dataset as validation data for all experiments, following other crosslingual work (Hu et al., 2019; Zhang et al., 2021) that simulate true unsupervised settings. This 10% is also excluded from the translation process that creates additional datasets. We use the original validation data in five languages to further reduce the number of models.
For both subtasks, we always fine-tune the models on the original English dataset DS (excluding the 10% of data used for validation) and explore the impact of incorporating additional data. Specifically, for the first subtask, we experiment with adding the translated dataset DT . For the second subtask, we explore the following options: adding the translated dataset DT , datasets with switched trigger words (DSt and DTs), or both.
Figure 4 shows the prompt used for fine-tuning the Orca 2 model, where we train the model to output the specific emotion class in a textual format.

heading: 2.5.1 Hyperparameters
text: We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with a batch size 16 for all models.
3https://github.com/huggingface/transformers
For Orca 2, we use QLoRA for fine-tuning with 4- bit quantization, setting LoRA r = 64 and α = 16, a learning rate 2e-4 with linear decay, and applying LoRA adapters on all linear Transformer block layers. We fine-tune the model for up to 5 epochs. For other models, we fine-tune them for up to 30 epochs and search for the learning rate from {2e-6, 2e-5, 5e-5, 2e-4} using a constant scheduler. All experiments are conducted on an NVIDIA A40 with 48 GB GPU memory.

heading: 2.5.2 Evaluation Metrics
text: The main metric for the emotion detection subtask is the macro-averaged F1 score. The primary metric for the binary trigger word detection subtask is the token-level F1 score, calculated on a token level and averaged across instances. A new metric called accumulated precise importance attribution is used for the numerical trigger word detection subtasks. After normalization (ignoring negative values and ensuring the attributions for each sentence add up to 1), this metric sums up the attributions for each trigger word (i.e., the tokens with a label 1).

heading: 3 Results
text: This section presents the results.

heading: 3.1 Emotion Detection
text: Table 1 shows the results of the emotion detection subtask. The results indicate that the Orca 2 largely benefits from the additional data translated into target languages, improving the results by over 3% on test data and by more than 7% on the validation set. The best model achieves a test score of 59.10, ranking seventh in the competition.
Figure 5 and Figure 6 show the confusion matrices for the Orca 2 model fine-tuned on English data only and both English and translated data, respectively. The model fine-tuned on translated data demonstrates significantly better performance
for the Fear (0.52 vs 0.29) and Joy (0.66 vs 0.55) classes while maintaining similar performance for other classes compared to the model fine-tuned only on English data.
Misclassified labels tend to cluster by sentiment. For instance, the Love label is frequently misclassified as Joy. The Neutral, Anger, and Joy classes appear to be the easiest to predict, likely due to their higher representation in the training and test data, as shown in Table 2.

heading: 3.2 Trigger Words Detection
text: Table 3 presents the results of the trigger words detection subtask, evaluating the performance for binary and numerical outputs using various training combinations. For binary classification, XLM-R achieves the highest test score (59.19) when trained on the full combination of datasets (DS + DT + DSt+DTs), ranking third in the competition. Conversely, mT5 excels in numerical value prediction with a top test score of 70.52 under the same training conditions, securing first place. In most cases, the mT5 model’s overall performance on numerical triggers is substantially worse than the performance of XLM-R. However, the results improve significantly when all dataset combinations are used.
Determining the best dataset combination and model is challenging due to the similar results achieved across many cases. Variability from random seeds during fine-tuning can obscure slight performance differences, which may not truly indicate superiority but rather random fluctuations. Nonetheless, our best test set results are obtained by training on a combination of the original, translated, and trigger words switched datasets. Combining all datasets for numerical trigger word detection using the mT5 model significantly improves the second-best settings by 8%.

heading: 3.3 Discussion
text: Overall, the Orca 2 benefits more from the translated data than the XLM-R and mT5 models, likely because the Orca 2 is pre-trained mostly on English data. In contrast, the other two models are multilingual. The translated data adds more knowledge to a model pre-trained mainly on English data than those already exposed to multiple languages. In
addition, trigger word detection tasks may be more prone to translation errors, which could diminish the benefits of translated data for these tasks. However, the mT5 model shows marginal improvements from the translated and trigger words switched datasets for the numerical trigger words detection, improving the results by 8% and achieving the best result overall in the competition.

heading: 4 Conclusion
text: This paper describes our system for the WASSA2024 Cross-lingual Emotion Detection Shared Task. We propose fine-tuning a quantized large language model with low-rank adapters combined with machine translation for the emotion detection subtask and fine-tuning multilingual Transformer-based models enhanced with machine translation and trigger word switching for the trigger words detection subtask. We show that additional translated data improves the performance. Our system achieves excellent results and ranks first in numerical trigger word detection, third in binary trigger word detection, and seventh in emotion detection.

heading: Acknowledgements
text: This work has been partly supported by the OP JAC project DigiTech no. CZ.02.01.01/00/23_021/0008402 and by the Grant No. SGS-2022-016 Advanced methods of data processing and analysis. Computational resources were provided by the e-INFRA CZ project (ID:90254), supported by the Ministry of Education, Youth and Sports of the Czech Republic.
Limitations
The method relies on machine translation, thus its effectiveness depends on translation quality. In addition, we solve each subtask independently. Future research could address solving both subtasks simultaneously, potentially leading to a more robust model that better explains its decisions.
Ethics Statement
We ensure fair and honest analysis while conducting our work ethically and without harming anybody. However, we acknowledge that the language models used in this paper may introduce unintended biases related to race or gender due to pre-training on large corpora.

heading: Future Work
text: Future research could address solving both subtasks simultaneously, potentially leading to a more robust model that better explains its decisions