HITSZ-HLT at WASSA-2024 Shared Task 2: Language-agnostic Multi-task Learning for Explainability of Cross-lingual Emotion Detection (2024)

Abstract: This paper describes the system developed by the HITSZ-HLT team for WASSA-2024 Shared Task 2, which addresses two closely linked subtasks: Cross-lingual Emotion Detection and Binary Trigger Word Detection in tweets. The main goal of Shared Task 2 is to simultaneously identify the emotions expressed and detect the trigger words across multiple languages. To achieve this, we introduce a Language-agnostic Multi Task Learning (LaMTL) framework that integrates emotion prediction and emotion trigger word detection tasks. By fostering synergistic interactions between task-specific and task-agnostic representations, the LaMTL aims to mutually enhance emotional cues, ultimately improving the performance of both tasks. Additionally, we leverage large-scale language models to translate the training dataset into multiple languages, thereby fostering the formation of language-agnostic representations within the model, significantly enhancing the model’s ability to transfer and perform well across multilingual data. Experimental results demonstrate the effectiveness of our framework across both tasks, with a particular highlight on its success in achieving second place in sub-task 2.

heading: 1 Introduction
text: Sentiment Analysis is an important task in Natural Language Processing (NLP), aiming to identify and assess the sentiment polarity in texts (Cambria, 2016). With the rapid development of social media and the Internet, sentiment analysis has become increasingly important in various fields such as customer service (Zvarevashe and Olugbara, 2018) and finance (Xing et al., 2020). Despite notable strides in sentiment analysis research (Jiang et al., 2023; Tu et al., 2023; Zhang et al., 2023; Hartmann et al., 2023; Zhong et al., 2023), challenges persist, particularly concerning foreign language texts where annotated data may be scarce.
*Corresponding author.
Cross-lingual Sentiment Analysis (CLSA) (Liu, 2012) mitigates these challenges by utilizing resources from one or more source languages to assist in sentiment analysis for low-resource languages (Esuli et al., 2020). This approach addresses the lack of annotated corpora for many non-English languages, making it a crucial research area in NLP. The fundamental strategy entails the transfer and adaptation of knowledge across various linguistic contexts (Zhao et al., 2024). Building on the foundational principles of CLSA, our study further explores how these methodologies can be practically implemented to enhance model performance across diverse linguistic settings.
The main challenge in Shared Task 2 (Maladry et al., 2024) involves two key aspects: (1) Enhancing the model’s capability to transfer knowledge to languages not present in the training data. (2) Developing strategies to effectively utilize complementarities between dual tasks given the limited availability of annotated data. To address the aforementioned challenges, we have developed a Language-agnostic Multi Task Learning (LaMTL) framework that effectively navigates cross-lingual obstacles while simultaneously bridging shared emotional cues across dual tasks.
Specifically, we utilize ChatGPT1 as a sophisticated tool for translation. By refining prompts, we translate the original tweets into Dutch, Russian, Spanish, and French, striving to maintain the original style as accurately as possible. By aligning the representations of identical tweets across different languages (Feng et al., 2022), we aim to develop a language-agnostic representation. Due to the complementary relationship between emotions and trigger words within tweets, we have designed a novel multi-task framework that includes both task-agnostic and task-specific encoders. The task-agnostic encoder captures task-invariant fea-
1https://chat.openai.com/
476
tures, facilitating the identification of underlying commonalities and related characteristics across tasks, while the task-specific encoder learns features unique to each task. To ensure comprehensive training, we incorporated multiple loss functions, including reconstruction loss and task prediction loss. This approach aims to improve the accuracy and robustness of multi-task learning by leveraging both shared and task-specific features, ultimately enhancing the framework’s performance on various tasks.
Additionally, we conducted a rigorous evaluation of our approach utilizing the EXALT dataset2, which substantiated its effectiveness. This comprehensive validation process led to our achievement of an esteemed second place in a highly competitive arena, as documented on the official leaderboard3.

heading: 2 Related Evaluation Tasks
text: In recent years, CLSA has gained widespread attention. In 2013, NLP&CC organized a crosslanguage evaluation by releasing annotated English data from Amazon user reviews and unannotated Chinese reviews. This initiative facilitated the development of methods for cross-lingual sentiment analysis. SemEval-2017 Task 4 (Rosenthal et al., 2017) focused on multilingual sentiment analysis of Twitter posts. This task utilized product ratings from platforms such as Amazon, TripAdvisor, and Yelp, and included five subtasks for both Arabic and English. SemEval-2020 Task 9 (Patwa et al., 2020) concentrated on sentiment analysis of codemixed tweets in Hinglish and Spanglish, providing annotated corpora and attracting 89 submissions. The top models achieved F1 scores of 75.0% for Hinglish and 80.6% for Spanglish. SemEval-2022 Task 10 (Barnes et al., 2022) introduced the first shared task on Structured Sentiment Analysis. Participants were required to predict sentiment graphs composed of a holder, target, expression, and polarity across seven datasets in five languages.

heading: 3 Methodology
text: In this section, we offer a comprehensive introduction to each component of the proposed LaMTL framework, illustrated in Fig. 1.
2https://huggingface.co/datasets/pranaydeeps/EXALT-v1 3https://lt3.ugent.be/exalt

heading: 3.1 Feature Extraction
text: We first employ ChatGPT to translate each English tweet xei from the training dataset D into x ψ i , where ψ ∈ {d, r, s, f} corresponds to Dutch, Russian, Spanish, and French, respectively. Subsequently, we utilize a multilingual pretrained model as the foundational encoder to extract feature representations from the tweets across various languages.
Specifically, for a tweet xi = {s1, s2, . . . , sNi}, where Ni denotes the number of words of xi, the corresponding sequence of tokens resulting from the application of subword tokenization techniques such as WordPiece and Byte Pair Encoding (BPE) is denoted by {w1, w2, . . . , wN̂i}. N̂i signifies the number of tokens corresponding to xi. The output of the last layer is denoted as he ∈ RN̂i×dh and hψ ∈ RN̂ψi ×dh . For each word sj , its representation hj ∈ Rdh is obtained by averaging the representations of its corresponding tokens.

heading: 3.2 Cross-Lingual Semantic Alignment
text: The Cross-Lingual Semantic Alignment (SA) Module aims to align semantic representations across language barriers. To achieve this, we employ the Mean Squared Error (MSE) as a reconstruction loss function. This function aims to minimize the semantic distance between translated non-English tweets and their English counterparts. It promotes the convergence of the feature vectors hψ of the translated tweets toward the feature vectors he of the original English tweets, ensuring consistent semantic representation across languages. The reconstruction loss Lrec can be represented as,
Lrec = ∑
ψ
MSE(ĥe, ĥψ), (1)
where ĥe and ĥψ denote the [CLS] representation or the average of all tokens for he and hψ.

heading: 3.3 Multi-Task Fusion
text: To effectively encapsulate the pertinent emotional cues in diverse tasks, we developed the Multi-Task Fusion (MTF) Module. This module integrates a task-invariant encoder alongside two task-specific encoders, each comprising a stacked structure of L transformer encoder layers. In MTF, each tweet xi is processed by three distinct encoders. The first encoder, Es, is task-invariant and designed to learn a generalized representation across multiple tasks by positioning the learned features within a common subspace. The other two encoders, Ee and Et,
are task-specific. Ee is dedicated to emotion detection, focusing on the subtle emotional nuances embedded within the tweets. In contrast, Et specializes in emotion trigger detection, identifying key triggers that indicate the presence or absence of specific emotional conditions. The process can be represented as:
hs = Es(h e,Θs), (2)
hη = Eη(h e,Θη), η ∈ {e, t}. (3)
The encoder Es shares parameters Θs across the two tasks, while Eη utilizes distinct parameters Θη for each task.
Finally, we concatenate the task-invariant representations hs with the task-specific representation hη and use a Multi-Layer Perceptron (MLP) with softmax for classification. Formally,
Hη = hs ⊕ hη, (4) Pη = softmax(MLPη(Hη)), (5)
ŷη = argmax(Pη), (6)
where ⊕ denotes the concatenation operation. Notably, we utilize ŷη as the prediction results for the task η.

heading: 3.4 Model Training
text: We utilize cross-entropy loss for the classification of the Cross-lingual Emotion Detection and Binary Trigger Word Detection tasks, denoted as Le and Lt, respectively. The computation process can be described as follows:
Lη = − 1
Nη
Nη∑
j=1
Cη∑
k=1
yη,[j][k] log(Pη,[j][k]), (7)
where N denotes the number of samples, C represents the number of classes for task η, Pη,[j][k] denotes the probability distribution for instance j over class k, and yη,[j][k] is a binary indicator that equals 1 if class k is the correct classification for instance j in task η, and 0 otherwise. Specifically, C is set to 6 for task η = e and to 2 for task η = t. For samples in the dataset containing two types of labels, we compute Le and Lt. For samples with only one type of label, we compute the loss specific to the corresponding task. To facilitate bettermixed learning, we apply a shuffling strategy to the dataset.
By combining the reconstruction loss and taskspecific loss, our final loss function can be expressed as,
L = Le + Lt + λrecLrec + λ||Θ||22, (8)
where Le and Lt denote the classification loss for Cross-lingual Emotion Detection and Binary Trigger Word Detection, while λ represents the L2 regularization weight, and Θ signifies the set of all trainable parameters.

heading: 4 Experiments

heading: 4.1 Baselines
text: To demonstrate the efficacy of our approach, we fine-tuned multilingual pre-trained models using the official codebase4, including MultilingualBERT (Devlin et al., 2019), LaBSE (Feng et al., 2022), and Multilingual-E5-Large (Wang et al., 2024). Furthermore, due to the robust multilingual capabilities of LLM, we conducted experiments using various configurations. Specifically,
4https://github.com/pranaydeeps/WASSA24_EXALT
we fine-tuned LLaMA-2 (Touvron et al., 2023) in conjunction with LoRAMoE (Dou et al., 2024) and LLaMA-3 (AI@Meta, 2024) with LoRA (Hu et al., 2022). Additionally, we performed zero-shot emotion detection experiments on GPT-4 (Achiam et al., 2023), and the designed prompt template can be found in Appendix A.
In ablation studies, ‘w/o SA’ denotes the removal of the SA module, and ‘w/o MTF’ indicates the removal of the MTF module.

heading: 4.2 Experimental Settings
text: In our experimental settings, we utilize a learning rate of 1e-4 with the AdamW optimizer to optimize the model parameters. We configured gradient accumulation to 4 and batch size to 8. In this study, we employ XLM-RoBERTa-Large (Conneau et al., 2020) as the backbone model. We configured the encoder in the MTF module as a single-layer transformer encoder. Additionally, the λrec parameter was strategically set to 0.05. The word embedding dimension dh is 1024, and the maximum sequence length is 512. All experiments were conducted on a single RTX 4090 GPU, using BF16 precision to optimize both speed and computational efficiency.

heading: 4.3 Evaluation Metrics
text: We use the official metrics for evaluation. For the Cross-lingual Emotion Detection task, we use the Macro-averaged F1 score as the evaluation metric. For the Binary Trigger Word Detection task, we utilize the Token F1 score as the evaluation metric.

heading: 4.4 Experimental Results and Analysis
text: Comparative Results: Table 1 presents a comparative analysis of our LaMTL model against various cross-lingual baseline models and LLMs. Our LaMTL model consistently outperforms the baselines across both sub-tasks, demonstrating superior performance. In addition, LLMs also exhibit remarkable performance and will be a primary focus of our future research. Ablation Studies: We conducted ablation experiments for our framework. According to the results shown in Table 2, the LaMTL model achieved F1 scores of 56.88% on the Emotion task and 60.95% on the Binary Triggers task. Removing the Crosslingual Semantic Alignment (SA) module resulted in F1 score decreases of 2.03 and 2.48 percentage points for the Emotion and Binary Triggers tasks, respectively, indicating the importance of cross-linguistic feature semantic alignment, especially for the Binary Triggers Word Detection task. Similarly, removing the Multi-Task Fusion (MTF) module led to F1 score decreases of 1.98 and 1.11 percentage points for the Emotion and Binary Triggers tasks, respectively. Leaderboard Results: Table 3 presents the performance of the top five teams in Binary Trigger Word Detection task. Our method achieves second place on the leaderboard.

heading: 5 Conclusions
text: In this paper, we propose a language-agnostic multitask learning approach to address the challenge of interpretability in cross-lingual sentiment analysis.
Firstly, we designed a reconstruction loss to mitigate cross-lingual discrepancies. Secondly, we implemented a multi-task learning framework to share sentiment cues between two tasks, thereby enhancing performance in both tasks. Through these methods, our model effectively enhances crosslingual capabilities and facilitates the sharing of emotional cues between multiple tasks, thereby achieving competitive performance.

heading: 6 Limitations
text: Although our LaMTL framework has demonstrated significant efficacy in cross-lingual sentiment detection and binary trigger word detection, there are several limitations that need to be addressed in future work. The use of large-scale language models for translation introduces potential biases and inaccuracies, especially for less common or informal texts in tweets. These translation inconsistencies can affect the quality of language-agnostic representations. While our framework has achieved commendable results, real-world applications might present additional challenges, such as subtle nuances specific to certain domains and the evolving use of language, which were not extensively explored in this study. Addressing these limitations in future research is crucial for enhancing the applicability and performance of cross-lingual sentiment analysis models.

heading: Acknowledgments
text: This work was partially supported by the National Natural Science Foundation of China (62176076), the Natural Science Foundation of Guangdong (2023A1515012922), the Shenzhen Foundational Research Funding (JCYJ20220818102415032), and the Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies (2022B1212010005).

heading: A Prompt Template for Emotion Detection
text: To employ GPT-4 for Cross-lingual Emotion Detection tasks, we have designed an effective prompt template, as illustrated in Figure 2.

heading: Future Work
text: Although our LaMTL framework has demonstrated significant efficacy in cross-lingual sentiment detection and binary trigger word detection, there are several limitations that need to be addressed in future work. The use of large-scale language models for translation introduces potential biases and inaccuracies, especially for less common or informal texts in tweets. These translation inconsistencies can affect the quality of language-agnostic representations. While our framework has achieved commendable results, real-world applications might present additional challenges, such as subtle nuances specific to certain domains and the evolving use of language, which were not extensively explored in this study. Addressing these limitations in future research is crucial for enhancing the applicability and performance of cross-lingual sentiment analysis models