Enhancing Cross-Lingual Emotion Detection with Data Augmentation and Token-Label Mapping (2024)

Abstract: Cross-lingual emotion detection faces challenges such as imbalanced label distribution, data scarcity, cultural and linguistic differences, figurative language, and the opaqueness of pretrained language models. This paper presents our approach to the EXALT shared task at WASSA 2024, focusing on emotion transferability across languages and trigger word identification. We employ data augmentation techniques, including back-translation and synonym replacement, to address data scarcity and imbalance issues in the emotion detection subtask. For the emotion trigger identification subtask, we utilize token and label mapping to capture emotional information at the subword level. Our system achieves competitive performance, ranking 13th, 1st, and 2nd in the Emotion Detection, Binary Trigger Word Detection, and Numerical Trigger Word Detection tasks.

heading: 1 Introduction
text: Emotion detection in text has attracted significant attention in recent years due to its diverse applications (Nandwani and Verma, 2021). The growing presence of multilingual and code-mixed content on social media has emphasized the need for crosslingual emotion detection systems (Balahur and Turchi, 2014; Dashtipour et al., 2016).
However, developing accurate and interpretable cross-lingual emotion detection models presents several challenges, including the limited availability of labeled data in many languages (Xue et al., 2020), cultural and linguistic variations in emotion expression (Hareli et al., 2015), the use of figurative language and sarcasm in social media (Bouazizi and Ohtsuki, 2019; Reyes et al., 2012), and the lack of transparency in large pre-trained language models (PLMs) (Hase and Bansal, 2020; Feder et al., 2021).
To tackle these challenges and foster research on interpretable cross-lingual emotion detection,
the EXALT shared task1 was organized at WASSA 2024. EXALT focuses on the transferability of emotion information across languages and the identification of emotion triggers, encouraging the development of interpretable and explainable emotion detection systems.
This paper presents our approach to the EXALT shared task. The XLM-RoBERTa-XL (Goyal et al., 2021) model with encoder-only architecture is selected to better capture the emotion information contained in the context. For the emotion detection sub-task, we employ data augmentation techniques, such as back-translation and synonym replacement, to address the imbalanced and insufficient training data. For the emotion trigger identification subtask, we utilize token and label mapping to align subword-level predictions with word-level labels.

heading: 2 Related Work

heading: 2.1 Emotion Detection
text: Emotion detection aims to identify the emotional states expressed in text, such as joy, sadness, anger, and fear (Acheampong et al., 2020). Compared to sentiment analysis, which focuses on the overall polarity of text (positive, negative, or neutral), emotion detection provides a more fine-grained understanding of the affective information conveyed in text.
Traditional approaches to emotion detection relied on rule-based methods and machine learning algorithms, such as Naïve Bayes, Support Vector Machines (SVM), and decision trees (Alswaidan and Menai, 2020).Recently, deep learning models, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformers, have achieved state-of-the-art performance on emotion detection tasks by learning rich feature representations from text (Birjali et al., 2021). However, the success of these models relies
1https://lt3.ugent.be/exalt/
528
heavily on large annotated datasets, which are timeconsuming and expensive to collect, especially for low-resource languages and domains. Researchers are exploring techniques such as transfer learning (Hazarika et al., 2021), few-shot learning, and data augmentation to address this challenge and improve emotion detection performance in low-resource settings.

heading: 2.2 Data Augmentation
text: Data augmentation techniques have been widely adopted to address the issue of data scarcity in various natural language processing tasks, aiming to generate synthetic training data and improve model performance (Pellicer et al., 2023). These methods operate at different levels of granularity, such as word-level, phrase-level, and documentlevel.
At the word level, techniques like EDA (Wei and Zou, 2019) generate new examples by manipulating words or embeddings. Phrase-level augmentation, e.g., PPDB Augmenter (Ganitkevitch et al., 2013; Pavlick et al., 2015), uses paraphrase databases to replace phrases with semantic equivalents, adding linguistic diversity. Document-level techniques include back-translation (Mallinson et al., 2017), paraphrasing with models like T5 (Raffel et al., 2020) and BART (Lewis et al., 2019; Dopierre et al., 2021), and text generation using language models such as Llama (Touvron et al., 2023) and GPT-4 (Achiam et al., 2023).

heading: 2.3 Cross-lingual Transfer Learning
text: Cross-lingual transfer learning has emerged as a promising approach to overcome the data scarcity issue in low-resource languages. Methods such as machine translation (Demirtas and Pechenizkiy, 2013) and multilingual PLMs (Lewis et al., 2019; Xue et al., 2020) have been used to transfer knowledge from high-resource to low-resource languages.
However, these methods often fail to capture language-specific nuances in emotion expression and may suffer from translation errors. Adversarial training (Chen et al., 2018) and contrastive learning (Lin et al., 2023) have been proposed to learn language-invariant representations, but they often require parallel data or emotion lexicons.

heading: 2.4 Interpretability in Natural Language Processing
text: Interpretability has gained significant attention in natural language processing, particularly with the increasing complexity of deep learning models (Danilevsky et al., 2020). Various approaches have been proposed to provide explanations for model predictions, such as attention mechanisms (Bibal et al., 2022), saliency maps (Wallace et al., 2020), and post-hoc explanations (Madsen et al., 2022).
However, most existing methods focus on providing explanations at the input feature level and may not offer fine-grained interpretability at the token level. In the context of emotion detection, identifying the specific words or phrases that trigger the predicted emotions is crucial for understanding the model’s behavior.

heading: 3 Methods
text: This section illustrates our approach and main work, as shown in Figure 1. We focus on describing our data augmentation and token and label mapping methods. The details of the shared task can be found in this work (Maladry et al., 2024).
Briefly, Task 1 is sequence-classification task, where there are 6 possible emotion classes in the datasets. Task 2 and 3 are token-classification tasks that are focused on explaining which words are used to express the emotion. The training dataset contains only English data, while the test dataset contains five languages, i.e. Dutch, Russian, Spanish, English, and French.

heading: 3.1 Data Augmentation
text: The dataset for Task 1 includes 5,000 training samples, 500 development samples, and 2,500 test samples. For Tasks 2 and 3, the dataset consists of 3,000 training samples, 300 development samples, and 832 test samples.
The original data presents the characteristics of imbalanced label distribution, as shown in Table 1. For example, the proportion of Fear class is less than 3%. In order to mitigate this characteristic and enhance the diversity of the data to improve the generalisation of the model, we apply two data augmentation methods as detailed as follows. Other data augmentation methods would break sentence syntax, such as random deletion, or change emotion information, such as text generation with PLMs, and are therefore not considered in this work.
Back-translation: The original training samples are translated into other languages, in this case Dutch, French, Spanish and Russian, and then back into English to generate new instances. We adopted a stratified sampling approach to mitigate the imbalance in label distribution. Notably, back-translation may alter the position of trigger words, causing misalignment with their corresponding labels.
Synonym replacement: Words are randomly selected in the original samples and then replaced with their synonyms. In this paper, we implemented synonym replacement using the NLTK WordNet corpus (Loper and Bird, 2002). Notably, synonym replacement may alter the number of words, which lead to misalignment of the augmented data with the original labels.
We use DeepL to complete the back-translation. In Task 1, 2,068 new instances are added via backtranslation, while 4,995 instances are added via synonym replacement. The details are shown in Table 1. In Task 2 and 3, 2,291 instances are added through synonym replacement.

heading: 3.2 Token and Label Mapping
text: The mapping between each subword and its corresponding input word is recorded when the tokenizer processes the input data. The label of the subword is assigned to be the same as the label of its corresponding word. During prediction, the label of each word is set to the maximum value among the predictions of all its corresponding subwords. Fig-
ure 2 illustrates the configuration of token and label mapping.
By applying labels at the subword level, the emotional information of the text is captured more finegrained. During the prediction phase, taking the maximum value of the subword predictions as the label of the corresponding word effectively integrates the emotional information at the subword level. Moreover, this method is easy to implement and compatible with various PLMs.
For Task 3, we map the output of the classifier layer of the fine-tuned language model to wordlevel numerical results. For Task 2, we compute the softmax of the classifier layer output, then map it into word-level numerical probabilities and finally convert it to a binary result using a threshold.
We have shared the main scripts of this paper on Github2 for other researchers.

heading: 4 Experiments
text: We conducted experiments on NPU training machines, equipped with 8 Ascend 910B 64G NPUs, to compare the performance of multiple PLMs and investigate the impact of hyperparameter settings, data augmentation strategies on cross-lingual emotion detection.

heading: 4.1 Comparison of Pre-trained Language Models
text: Although many current PLMs adopt the decoderonly architecture, this architecture has limited support for natural language understanding tasks. In our tasks, we consider the information in the subsequent text to be equally important as the preceding text. Therefore, we compared multiple encoderonly models pre-trained on Dutch, Russian, Spanish, English, and French corpora to better understand the contextual content and cross-lingual emotion information.
We fine-tuned these models on the original data without data augmentation and tried different parameter freezing strategies. Figure 3 demonstrates that that larger models achieved better cross-lingual
2https://github.com/QQJellyy/ CTcloud-EXALT-WASSA2024
emotion detection ability. In the meanwhile, Figure 4 suggests that the PLMs lack emotion detection capabilities, as the results by training only on the classification layer are significantly lower than the other training strategies.
Training on the English dataset significantly improves emotion detection abilities on five languages, indicating that emotion information has been effectively transferred across languages. We explain this phenomenon with a task analogy theory, e.g., ’Queen = King + (Woman-Man)’ (Ethayarajh et al., 2018), where cross-lingual capabilities are inherent in PLMs, and emotion detection capability is attained through fine-tuning. The finetuned models integrate these capabilities and exhibit cross-lingual emotion detection abilities.

heading: 4.2 Hyperparameter Optimisation
text: We conduct hyperparameter optimisation on XLMRoBERTa-large and employ an orthogonal ap-
proach to optimize each hyperparameter. As shown in Table 2, a learning rate of 4e-5 achieves the best classification results for Task 1. Learning rates that are too large or too small can be less effective. Similarly, we find that a batch size of 64 yields the best detection results.
For Task 1, we train the XLM-RoBERTa-XL model on the augmented data using the optimal hyperparameters described above. For Tasks 2 and 3, we use the same hyperparameters and further train the model saved from Task 1. When making predictions, we use the default value of 0.1 as the threshold for identifying trigger words.

heading: 4.3 Data Augmentation
text: Table 3 demonstrates that both synonym replacement and back-translation improve the model’s performance. Synonym replacement enriches the training corpus, while back-translation implicitly introduces information about the target test language without altering the language of the training data.

heading: 5 Conclusion
text: In this paper, we address the label imbalance issue and enhanced data diversity in the training data by employing data augmentation techniques, including back-translation and synonym replacement. The augmented data is used to fine-tune the XLMRoBERTa-XL model, achieving competitive results in all three tasks: 13th, 1st, and 2nd places in Task 1, Task 2, and Task 3, respectively. These results demonstrate the effectiveness of our methods for the transferability of emotion information across languages and the identification of emotion triggers.
The reason why we choose XLM-RoBERTa-XL model is that the encoder architecture is able to capture the context of the data, making it wellsuited for token-level tasks. Furthermore, we select PLMs that have been trained on five languages to ensure that the models have the transferability of emotion information across languages, potentially contributing to the improved performance of our proposed system.

heading: 6 Limitations
text: In this paper, we assume that different models behave similarly, e.g., the optimal hyperparameters of XLM-RoBERTa-XL and XLM-RoBERTa-large are similar. However, this may not be the case in practice.
We use the default value of 0.1 as the threshold for identifying the trigger word in Task 2 and apply two data augmentation techniques, i.e. backtranslation and synonym replacement. In the future, we can explore the impact of different classification threshold on model performance and try other data augmentation methods. Moreover, we can also go deep into model enhancing techniques on cross-lingual emotion detection task, such as task analogy and model fusion.

heading: Future Work
text: In the future, we can explore the impact of different classification threshold on model performance and try other data augmentation methods. Moreover, we can also go deep into model enhancing techniques on cross-lingual emotion detection task, such as task analogy and model fusion