Knowledge Distillation from Monolingual to Multilingual Models for Intelligent and Interpretable Multilingual Emotion Detection (2024)

Abstract: Emotion detection from text is a crucial task in understanding natural language with wideranging applications. Existing approaches for multilingual emotion detection from text face challenges with data scarcity across many languages and a lack of interpretability. We propose a novel method that leverages both monolingual and multilingual pre-trained language models to improve performance and interpretability. Our approach involves 1) training a high-performing English monolingual model in parallel with a multilingual model and 2) using knowledge distillation to transfer the emotion detection capabilities from the monolingual teacher to the multilingual student model. Experiments on a multilingual dataset demonstrate significant performance gains for refined multilingual models like XLMRoBERTa and E5 after distillation. Furthermore, our approach enhances interpretability by enabling better identification of emotiontrigger words. Our work presents a promising direction for building accurate, robust and explainable multilingual emotion detection systems.

heading: 1 Introduction
text: Emotion detection, a sub-category of sentiment analysis, is the process of computationally identifying, extracting and categorising the emotion expressed in text. This granular analysis of affective states, including joy, sadness, fear, and anger, represents a crucial task in natural language understanding (NLU) that has garnered substantial research attention for several decades due to its wide range of applications. The growth of social media platforms, such as Facebook and Twitter, has led to an increasing trend of individuals sharing their emotions, thoughts and experiences through short snippets of text in posts, tweets, comments
†Corresponding author.
and captions. Consequently, a vast volume of usergenerated data enriched with emotional content has been generated, highlighting the immense value of automating the detection and analysis of underlying emotions.
Despite the significant progress made in emotion detection from text, there remain two key challenges that hinder the widespread applicability and trustworthiness of these systems: 1) the majority of existing work focused on developing models for high-resource languages like English, where large task-specific datasets for emotion detection, such as CANCEREMO (Sosea and Caragea, 2020) and EmoNet (Abdul-Mageed and Ungar, 2017), are readily available for training and fine-tuning. On the other hand, there is not enough training data for many minority languages, such as French and Dutch. This data scarcity poses a significant obstacle in building accurate and robust emotion detection models that can cater to the linguistic and cultural diversity present across different communities; 2) while current models excel at overall emotion classification, they often lack the ability to provide explanations or insights into the specific linguistic cues that triggered the detected emotions. Many previous studies have primarily concentrated on maximising the overall accuracy metrics (Wang et al., 2021; Wang and Gan, 2023), overlooking the importance of interpretability and rationale extraction.
Recent advances in multilingual pre-trained language models (multilingual PLM) present a promising direction. These models, such as mBERT (Devlin et al., 2019), XLM-RoBERTa (Conneau et al., 2020), and mDeberta (He et al., 2021), are pretrained on large volumes of unlabeled text from multiple languages in an unsupervised manner, allowing them to capture rich cross-lingual representations that can be effectively transferred to downstream tasks like emotion detection. They play a crucial role in enabling cross-lingual trans-
1 470
fer and facilitating joint training across different languages (Ruder et al., 2019). Furthermore, the transformer-based architecture with self-attention mechanisms can potentially provide interpretability benefits (Chefer et al., 2021), allowing us to analyse the important trigger words that contribute to the predictions. However, the task-specific performance of a fine-tuned multilingual PLM on the multilingual data may not be as comparable to that of separately trained monolingual language models evaluated on the data from their respective languages (Lothritz et al., 2021; Wu and Dredze, 2020). To address the above-mentioned issue, in this work, we propose a novel approach that combines the strengths of both monolingual and multilingual models for improved emotion detection performance and interpretability across diverse languages.

heading: 2 Methodology

heading: 2.1 Parallel Model Adaptation for Emotion Detection
text: Our proposed approach involves two separate training pipelines leveraging both monolingual and multilingual PLMs. In the first pipeline, we finetune a state-of-the-art English monolingual model (such as RoBERTa (Liu et al., 2019)), denoted as θen, on the provided English emotion detection training set, enabling the model to capture the linguistic meanings and emotion cues that are specific to the English language; in parallel, we fine-tune a multilingual PLM like XLM-RoBERTa (Conneau et al., 2020), denoted as θmul, on the same English set. The multilingual PLM, pre-trained on a large corpus of data from various languages, can leverage its cross-lingual representations to learn task-specific patterns from the data in a given language (Liu et al., 2020). By training these two models separately on the same English dataset, we can obtain a high-performance English monolingual model tailored for emotion detection, as well as a multilingual model that has adapted its cross-lingual representations to the task of emotion detection while still retaining its ability to generalise across languages.

heading: 2.2 Knowledge Distillation from the Monolingual Model
text: To further refine and improve the performance of the multilingual model, we propose a knowledge distillation strategy that utilises the high-
performing monolingual English model as a teacher. The refinement scheme for the fine-tuned multilingual model is shown in Figure 1. Since the unlabelled data from development set is in various languages, we first translate all the non-English instances into English using neural machine translation systems based on Marian (Junczys-Dowmunt et al., 2018), i.e. T : X → X ′, where X = {xi, x2, ..., x|X|}, standing for the original development set that contains multilingual texts and X ′ = {x′i, x′2, ..., x′|X′|}, representing the translated English version. We then obtain predictions from both the monolingual model on the translated data X ′, and the multilingual model on the original data X .
To transfer the ability of the monolingual model to the multilingual model, we compute the Kullback-Leibler (KL) divergence, a nonsymmetric loss function, as the consistency loss between their output distribution P and Q on each instance. Importantly, we focus on minimising the consistency loss only when the quality of translated data in X ′ is suspected to be of good quality, which ensures that we prioritise knowledge transfer from the teacher model on instances where the translation is sufficiently reliable. To achieve this, we introduce a normalised weight w̄i for each translated instance x′i and compute the loss with these weights included, i.e.
L = w̄ ∗KL(P∥Q)
=
|X|∑
i=1
w̄i
  k∑
j=1
p(yj |x′i, θen) log p(yj |x′i, θen) p(yj |xi, θxlm)
 
(1)
where p(yj |x′i, θen) and p(yj |xi, θxlm) represent the output probabilities for the j-th category of the monolingual model on the i-th translated English instance and the multilingual model on the i-th original instance, respectively. k is the number of categories for the emotion detection task.

heading: 2.3 Translation Quality Weighting
text: To compute the weight, which reflects the suspected translation quality, we first obtain the predictions of the same multilingual model on both the original data xi and translated data x′i. We then calculate the disagreement between these two prediction using the mean squared error (MSE), a
2
symmetric function, i.e.
wi =
k∑
j=1
[ p2 ( yj |x′i, θxlm ) − p2 (yj |xi, θxlm) ]
(2) A high MSE value between the predictions of the multilingual on x and x′ suggests that the translation quality is likely poor, as the understanding of the model for the original instance and translated instance significantly disagrees, indicating the potential translation flaws or errors, conversely, a low MSE value illustrates the consistency between the original and translated versions, implying a higher translation quality. Therefore, when the disagreement is high, such a translated sample should contribute less to the final loss. To account for this, we normalise the weight as follows:
w̄i = 1− wi −min(w)
max(w)−min(w) (3)

heading: 3 Experimental Results

heading: 3.1 Datasets and Shared-task
text: We utilised the dataset on explainable crosslingual emotion detection in tweets (EXALT)1. This dataset includes annotations for both word triggers and the overall expressed emotion for each instance. The training set is available in English only, while the development set and test set involve five different languages: English, French, Dutch, Russian and Spanish. More details about the shared task can be found in (Maladry et al., 2024).
1https://lt3.ugent.be/exalt/

heading: 3.2 Baseline
text: We conducted experiments involving generative models, discriminative models and feature extractors. For the generative models, we employed BLOOM (Le Scao et al., 2023) and BLOOMZ (Muennighoff et al., 2023) with 7.1 billion parameters. Both BLOOM and BLOOMZ are pre-trained on multilingual corpora, and BLOOMZ was further fine-tuned using prompted multitask learning.
For discriminative models, we utilised RoBERTa with the translated samples and XLM-RoBERTa. Additionally, we considered two recent feature exactors, BGE-M3 (Chen et al., 2024), and E5 (Wang et al., 2024), which were pre-trained on the massive multilingual text-pair to extract the cross-lingual features and construct the text embedding for multiple languages. To perform the classification task, we added a fully connected layer as the classifier and applied the average pooling to the text embedding to generate the predictions. The implementation details can be found in the Appendix .1.

heading: 3.3 Main Results
text: We reported the main results of the emotion detection task in Table 1. Based on the overall result, we can see that despite having a larger number of parameters, which generally indicates greater model capacity, generative models such as BLOOM and BLOOMZ did not perform as well as the large discriminative models. This implies that larger models do not necessarily lead to better performance in tasks that require detailed understanding and classification of emotions.
Another noteworthy observation is the signifi-
3
cant performance improvements achieved by the multilingual PLMs after knowledge distillation from the monolingual RoBERTa model. On the development set, all evaluated multilingual models, including XLM-RoBERTa, E5, and BGE-M3, showed significant gains, with an average improvement of 4.87% in terms of the F1 score. The consistent improvements across all multilingual models suggest that the knowledge distillation strategy was effective in transferring the specialised emotion detection capabilities of the monolingual model to the multilingual model. However, the results on the test set were more varied. While XLM-RoBERTa and E5 still demonstrated obvious improvements, the BGE-M3 model only showed a minor increase of 0.24% in performance. This suggests that while the multi-functional pre-training strategy enables BGE-M3 to handle inputs of different granularities, it may have resulted in representations that are less aligned with specific emotion detection tasks and hinder the ability to deal with a large proportion of instances with linguistic phenomena in the test set.

heading: 3.4 Explainability of Transformers
text: To gain insights into the explainability of our models and their ability to identify emotion triggers, we evaluated their zero-shot performance on the binary trigger detection task. Without any explicit training on the fine-tuned emotion detection models, we directly reported the token-level F1 score and mean average precision (MAP) on this task, as presented in Table 2.
Notably, we observed that the trigger detection

heading: R. + XLM-R. (↑) 33.73 25.46 31.25 24.34

heading: R. + E5 (↑) 33.85 25.02 31.61 24.64

heading: R. + BGE-M3 (↓) 32.97 24.49 29.35 23.03
text: performances of each model on the development and test sets were basically consistent with their emotion detection performance. Both the refined XLM-RoBERTa and E5 models showed better trigger detection capabilities compared to their original versions. However, the performance of the BGEM3 model on trigger detection became slightly worse than its original version, which can potentially account for the relatively poor performance gains observed on the emotion detection task on the test set.

heading: 4 Conclusion and Future Work
text: In this work, we proposed a novel approach that combines the strength of monolingual and multilingual PLM for improved emotion detection performance. Our method involves training a highperforming English monolingual model in parallel with a multilingual model on the same English emotion detection training set. We then employ a knowledge distillation strategy to transfer the
4
specialised emotion detection capabilities from the monolingual teacher model to refine the multilingual student model. Future work could explore more sophisticated knowledge distillation techniques, as well as employ more accurate and effective translation methods (Na et al., 2024).
Limitations
There are several potential limitations in our work: 1) while a weighting scheme is proposed to account for the translation errors, the quality of the translation system can still significantly impact the knowledge distillation process; 2) the computational complexity involved in training multiple models and performing additional inference steps for weighting and distillation may pose practical limitations.

heading: Acknowledgements
text: We would like to acknowledge the financial support provided by the Postgraduate Research Scholarship (PGRS) (contract number PGRS-20-06-013) at Xi’an Jiaotong-Liverpool University. Additionally, this research has received partial funding from the Jiangsu Science and Technology Programme (contract number BK20221260) and the Research Development Fund (contract number RDF-22-01132) at Xi’an Jiaotong-Liverpool University.

heading: Future Work
text: Future work could explore more sophisticated knowledge distillation techniques, as well as employ more accurate and effective translation methods (Na et al. , 2024). Limitations
There are several potential limitations in our work: 1) while a weighting scheme is proposed to account for the translation errors, the quality of the translation system can still significantly impact the knowledge distillation process; 2) the computational complexity involved in training multiple models and performing additional inference steps for weighting and distillation may pose practical limitations