Effectiveness of Scalable Monolingual Data and Trigger Words Prompting on Cross-Lingual Emotion Detection Task (2024)

Abstract: This paper introduces our submitted systems for WASSA 2024 Shared Task 2: CrossLingual Emotion Detection. We implemented a BERT-based classifier and an in-context learning-based system. Our best-performing model, using English Chain of Thought prompts with trigger words, reached 3rd overall with an F1 score of 0.6015. Following the motivation of the shared task, we further analyzed the scalability and transferability of the monolingual English dataset on cross-lingual tasks. Our analysis demonstrates the importance of data quality over quantity. We also found that augmented multilingual data does not necessarily perform better than English monolingual data in cross-lingual tasks. We open-sourced the augmented data and source code of our system for future research.1

heading: 1 Introduction
text: Recognizing the affect of tweets presents a crucial challenge as they encapsulate the semantic and emotional dialogue of individuals spanning diverse cultures and languages through a short, informal, and noisy medium. The unique constraint of limited length allows users to communicate through abbreviations, slang, and emojis. Such lexical idiosyncrasies, compounded by fragmented language and cultural references, further exacerbate the challenge for traditional natural language processing (NLP) models to interpret emotional content in tweets.
Previous studies have explored various approaches to sentiment and emotion classification (Mohammad, 2016). The SemEval-2018 Task 1 (Mohammad et al., 2018) presented an array of tasks for recognizing the affect of tweets, focusing on building monolingual English, Arabic, and Spanish systems. The vast majority of prior research on affect recognition models has been con-
1∗ Equal contribution.
ducted on monolingual English data, but an increasing number of studies are now focused on the use of cross-lingual models to improve emotion classification (De Bruyne, 2023). WASSA 2024 Shared Task 2 follows the trend of multilingual emotion classification with an emphasis on model explainability and interpretability (Maladry et al., 2024). With the advent of large language models (LLM) accessible through closed API calls, explaining the rationale of LLM prediction is increasingly important.
In our participation in the WASSA 2024 Shared Task 2, we present key observations in data selection and monolingual vs. multilingual approaches through our comparison of our BERT-based models to LLM prompting methods. Our findings indicate that pre-training with in-domain data yields better performance than within-task data. Additionally, monolingual data (back-translated) outperforms multilingual data. We also introduce our balanced dataset for fine-tuning and make it opensource for future research2.

heading: 2 System Overview

heading: 2.1 BERT-based classifier
text: We summarize our system in Figure 1, and detailed hyper-parameters can be found in Table 5. The system consists of 1) data augmentation, 2) preprocessing, 3) creating our Treehouse LM by continued pre-training the pre-trained TwHIN-BERTLarge (Zhang et al., 2022) language model with augmented data, and 4) fine-tuning the Treehouse LM with in-domain tweets for the downstream task.

heading: 2.1.1 Data Augmentation
text: We observed a significant label imbalance for the provided official English training set (Official Train), as seen in Figure 3. To boost the robustness of our system, we used the English SemEval 2018
2https://github.com/freddy5566/cross-lingual-emotiondetection
511
Task1 dataset (Mohammad et al., 2018), and Sentiment140 (Go et al., 2009) to create an augmented training set for both fine-tuning and continued pretraining.
To build a balanced train set (Balanced Train) for fine-tuning, we filtered the English SemEval 2018 Task 1 dataset for entries that were labeled with one of the five underrepresented labels: Joy, Anger, Love, Sadness, or Fear. Then, we randomly selected the samples, reaching 6, 970 entries.
To build the augmented train set for continued pre-training (Back Translated Pre-train), we started with translating Arabic and Spanish samples in the SemEval 2018 Task 1 dataset into English. Combined with the Official Train dataset, the dataset reached a size of 27, 458. It was further backtranslated with four target languages: Spanish, Russian, French, and Dutch, yielding a set of 132, 290 entries after removing instances from Official Train to prevent data contamination. All translations are conducted through Google Translation API.
Furthermore, we combined our Back Translated Pre-train dataset with the Sentiment140 dataset3, a tweet dataset with 1.6 million instances. We experimented with the various combinations of the two datasets to show the scalability and transferability of the monolingual English dataset in cross-lingual emotion recognition. Table 1 summarizes the data used in the system.
To further compare the effect of language variety
3https://www.kaggle.com/discussions/productfeedback/176309

heading: Data # of Sentences
text: in training data, we also created a multilingual training set for fine-tuning and continued pre-training. Details are provided in the Appendix A.2. The multilingual dataset was used only for paper-writing purposes and was thus not used during the competition.

heading: 2.1.2 Pre-processing
text: Through experimentation, we found that certain pre-processing methods, such as lowercasing, autocorrection, and slang replacement, degraded performance. The impact of removing hashtags, numbers, and punctuation varied, as these elements can both convey important emotional context and introduce noise.
The optimal pre-processing methods we identified were: converting Unicode symbols to ASCII, removing mentions and links, eliminating repetitive characters, and converting over-segmented character sequences back into words. These choices
2
indicate that TwHIN-BERT, trained on extensive Twitter data, effectively handles the inherent noise in tweets and that excessive pre-processing can overcorrect testing data and degrade its ability to capture semantic representations.

heading: 2.1.3 Continued pre-training
text: Following the findings in Gururangan et al. (2020) of domain and task-adaptive continued pre-training, we continue to pre-train BERT-based LMs (Devlin et al., 2019; Liu et al., 2019; Nguyen et al., 2020; Conneau et al., 2020; Zhang et al., 2022) to further adapt the general Twitter domain to the specific Twitter domain of EXALT data with the augmented data introduced in Section 2.1.1.

heading: 2.1.4 Fine-tuning
text: We fine-tuned the continued pre-trained LM for downstream emotion classification with the Balanced Train data. We add one linear classifier on top of the pre-trained LM that takes the CLS token x as the input and applies a linear projection T : Rd → Rn, where d is the dimension of CLS token embedding and n is the number of classes. The final distribution can be obtained by applying the softmax function σ(T (x)).

heading: 2.2 In-context Learner
text: Brown et al., 2020 found that large language models can learn the contextual information from raw text, indicating that they can be used for downstream tasks without fine-tuning on additional labeled data, a technique called in-context learning (ICL). In this section, we describe our method for building an ICL-based system.

heading: 2.2.1 Chain of Thought with trigger words
text: Chain-of-Thought (CoT) is a method that introduces intermediate steps that decompose the reasoning process to the LLMs, enabling them to significantly improve their performance (Wei et al., 2022). To ensure the explainability of affect recognition, we designed English CoT prompts (Eng-
CoT) that explain the classification process by identifying the trigger word. As shown in Figure 1, the given prompt points out that “hate” is the trigger word for the tweet “I hate it.” We provided an explanation for the Neutral label where no clear trigger word exists based on the annotation guidelines (Singh, Pranaydeep and Maladry, Aaron and Lefever, Els, 2023). Example trigger words and tweets are selected from the trigger word detection dataset from subtask 2 of EXALT Shared Task.
Our system uses GPT4o, which performed best in comparison to other models. See A.6.2 for the details. Similar to the BERT-based classifier, we further analyzed system performance with multilingual CoT prompts (MulCoT). The experiment details and full prompts used in our system are provided in Appendix A.6.

heading: 3 Results
text: Table 2 summarizes our system results. The overall best-performing model was GPT4o with English Chain of Thought instructions (EngCoT). This system also reached 3rd place overall in the leaderboard. However, the best recall was achieved through multilingual CoT (MulCoT).
Regarding BERT-based classifiers, continued pre-training TwHIN-BERT with 130k backtranslated data outperformed 1M and 1.7M models. A detailed ablation study can be found in Appendix A.3.

heading: 4 Discussion

heading: 4.1 Scalability of continued pre-training
text: We examine whether data quantity or quality is more important in pre-training. As suggested in (Sun et al., 2019), continued pre-training with indomain or within-task data can provide a certain level of improvement to the downstream task. We used Back Translated Pre-train as the in-domain data because of the similarity of data distribution with fine-tuning data. We used Sentiment140 as
3
the within-task pre-training data because it fits in the sentiment classification task to conduct this examination.
As we can see in Figure 2, all models with continued pre-training perform better than those with no pre-training. Among them, pre-training with in-domain data achieved the best performance. On the other hand, the performance of models (1M and 1.7M) trained with within-task data slightly trails that of the model (130k) trained with in-domain data despite the huge difference in the amount of training data in the pre-training stage. This result suggests that the similarity of data distribution between pre-training and fine-tuning is key to yielding the best performance.

heading: 4.2 Multilingual vs. Monolingual

heading: 4.2.1 BERT-based classifier
text: To investigate the effectiveness of multilingual data, we compared the multilingual model (please refer to Appendix A.2 for a detailed description) and back-translated English-only monolingual model.
The results, shown in Table 3, show our English back-translated model “Balanced Train” performing better than the multilingual model “Subsampled” under the same amount of training data. However, the multilingual model “All” outperforms “Balanced Train”. We hypothesize that performance gain in “All” comes from the amount of training data. Furthermore, our experiments demonstrate that, unlike monolingual continued pre-training, multilingual continued pre-training can be harmful. The root causes still need further study.

heading: 4.2.2 Prompting-based methods
text: Similar to our findings with the BERT-based classifier, the use of multilingual CoT did not outperform monolingual English CoT prompts. Further analysis of the output labels from EngCoT and MulCoT showed that both methods improve the F1 scores by enhancing Recall at the expense of Precision (Table 11 and 12). Compared to EngCoT, MulCoT resulted in a notable drop in Precision. This trend is consistent, as on average, over five runs, EngCoT achieved results of 0.5923, 0.5972, and 0.5997 in F1, Recall, and Precision, respectively. In comparison, MulCoT resulted in 0.5893, 0.6046, and 0.5863. The reason for such difference remains unanswered, but we present our additional experiments on prompts in Section A.6.3 for future research.

heading: 5 Conclusion
text: This paper describes our proposed systems for Shared Task 2 of WASSA 2024. Through the analysis of BERT-based classifiers and in-context learning-based systems, we highlight the importance of high-quality data and the effectiveness of CoT using trigger words. In our continued pre-training experiments, we discovered that aligning the data distribution between continued pretraining data and fine-tuning data is crucial. Without this alignment, the size of the dataset does not significantly impact information transferring into cross-lingual settings. While it may seem intuitive to use multilingual data for cross-lingual tasks, our findings revealed that this approach did not enhance performance in both systems. Further research is needed to understand the underlying mechanisms of in-context learning and its impact on performance.
4

heading: 6 Limitations
text: The main limitations of our work relate to these points: a) Our augmented data is highly dependent on the quality of Google Translation API. Furthermore, it is not a deterministic output; b) We did not perform an extensive hyper-parameter search in continued pre-training, which might improve classifiers’ performance. c) We presented outputs from closed-source models, where access is limited through paywall APIs. d) The outputs from GPT-4 are not deterministic and are vulnerable to changes in prompts. e) Considering the similarity between sentiment classification and emotion detection, we treated the sentiment 140 data as within-task data. However, those two tasks are related, not identical; therefore, results could be affected by such differences.

heading: A Appendix

heading: A.1 Data
text: Figure 3 shows the label distribution in Official and Balanced Train.

heading: A.2 Multilingual Training Data
text: This section explains the process of building multilingual training data for fine-tuning and continued pre-training. Likewise, we translated the Official Train into four other languages and combined it with the Official Train, resulting in 25, 000 entries for the multilingual training set (Forward Translated Train). To have a fair comparison with Balanced Train, we also subsample 6,923 entries with a similar label distribution to Balanced Translated Train.
For the multilingual continued pre-training dataset (Forward Translated Pre-train), we combined 27, 458 English data, created by translating Arabic and Spanish SemEval data in English, and translated it into French, Dutch, Spanish, and Russian. After removing identical entries, the Forward Translated Pre-train contains 134, 745 entries. Table 4 summarizes the data.

heading: A.3 Classifiers
text: In this section, we conducted a detailed ablation study on BERT-based classifiers. We examined five strong pre-trained LMs (Devlin et al., 2019; Liu et al., 2019; Nguyen et al., 2020; Conneau et al., 2020; Zhang et al., 2022) with different conditions. The detailed hyper-parameters are summarized in Table 5.

heading: Hyper-parameter Pre-training Fine-tuning
text: A.3.1 Which pre-trained LM as the seed model?
Our results are summarized in Table 6. We found TwHIN-BERT performed best regardless of whether it was pre-processed or not and whether the test set was translated into English. Therefore, we use TwHIN-BERT as our seed model in the rest of the experiments.

heading: A.3.2 Pre-processing
text: Surprisingly, models without pre-processing generally perform better than those with pre-processing. We hypothesize that this is because TwHIN-BERT is capable of capturing and representing the deep lexical and semantic properties of tweets, having been trained on complete Twitter data. Although noisy, various lexical features of tweets are strong indicators of emotional context, unique tokens such as emoticons and emojis can be challenging for the model to tokenize and represent accurately. Therefore, a promising area for future research is to explore methods for converting these tokens into meaningful words or alternative representations.

heading: A.4 English dev/test
text: Another obvious trend is translating dev and test sets into English, dramatically improving performance. Despite some pre-trained models being trained in the multilingual datasets, the English-translated pairs still perform better. For example, XLM-Roberta with and without
7
English translation, (0.3200/0.3268/0.3257) vs. (0.2311/0.2450/0.3000) for F1, Recall, and Precision, respectively. This is very intuitive because the training tweets are all in English. Therefore, models can only perform well on English tweet emotion classification.

heading: A.5 Data-augmentation
text: Our results on data augmentation are shown in Table 7. As we can see, it achieved performance gain in almost every pre-trained model. Notably, this data augmentation only works well on BERT and TwHIN-BERT. It is harmful for BERTweet and Roberta pairs and almost zero-gain for XLMRoberta.
A.6 In-Context Learning

heading: A.6.1 Experiment Detail
text: We tested multiple OpenAI’s GPT models using their APIs. Our system uses the latest model, GPT4o, which uses a different tokenizer that achieves better multilingual performance. Although ensuring deterministic output is challenging due to the GPU-based calculations of LLMs, we minimized the variables by setting the temperature, frequency penalty, and presence penalty as zero.
Furthermore, we penalized undesirable output tokens and boosted the probability of desired labels. By setting log probability and max token number as one, we ensured our models to return label only.

heading: A.6.2 Model Selection
text: We tested multiple GPT base models to find the best-performing one. On the dev dataset, Zeroshot GPT-4o outperformed both Zeroshot GPT-4 and Zeroshot GPT-3.5, achieving an F1 score of 0.5645, compared to 0.5616 and 0.4847, respectively. Therefore, we selected GPT-4o as our main model. Table 8 summarizes the results of various models and prompting methods.
A.6.3 Vulnerability of prompting It is noteworthy that the effect of our Trigger Word CoT differs by model type. In Table 8, both GPT4o and GPT-3.5 show a similar trend of increasing F1 scores with EngCoT and MulCoT. Compared to Zeroshot, EngCoT improved by approximately 0.2 to 0.4, and MulCoT gained roughly 0.5 to 0.7. However, as shown in Table 2, such gains were not transferable to the test data. Additionally, GPT4 suffered from additional CoT steps, with the F1 score decreasing by approximately 0.5. This
8
demonstrates that the effect of CoT and prompting methods depends on the model type, as well as the content and distribution of the dataset.
Additionally, we conducted a simple ablation study on the number of CoT examples using GPT3.5, which benefited the most from CoT with the dev data. Following the annotation guidelines, the current CoT steps consist of 8 examples: 5 emotion categories and 3 cases of the Neutral label. When we reduced the number of cases to 6 by removing two random Neutral examples, the F1 score dropped from 0.5305 to 0.4434. Adding two additional emotion-label examples decreased the F1 score to 0.5062.
This ablation study suggests that the number of
CoT examples affects the results. However, as mentioned earlier, our findings are specific to the model types and dataset, making it difficult to generalize that 8 steps are the best. Furthermore, there is no strong evidence that our instructions significantly impacted the results, as several studies suggest that the quality and content of instructions do not matter much in ICL (Min et al., 2022; Webson and Pavlick, 2022; Wang et al., 2023).

heading: A.6.4 Error Analysis
text: This section provides a label-wise analysis of the test data. Due to time and financial constraints, we were not able to perform a detailed ablation study and experiments on the test data.
As mentioned in Section 4.2.2, the majority of changes in F1 occur in the Fear, Love, and Neutral labels. Table 11 and Table 12 show that CoT improves the F1 scores of these labels by increasing Recall at the expense of Precision. Given that CoT improves Recall, the performance gain on Fear, the most underrepresented label in the dataset, seems intuitive. Similarly, Love, the second most scarce label, potentially benefits from the robustness provided by CoT. The changes in the Neutral label suggest that the model might be following the reasoning steps detailed in our CoT examples, where we emphasize the lack of keywords in the Neutral examples. However, there is no clear evidence supporting this hypothesis. Therefore, further research
9
is needed to understand the effect of CoT reasoning steps.

heading: A.6.5 Prompts
text: This section provides a list of prompts used in the ICL experiments and descriptions.
CoT examples are made up of 8 examples: 5 Emotion labels and 3 different cases of Neutral labels. For non-Neutral labels, we randomly selected tweets with token sizes between 5 and 10. We excluded lengthy tweets ( > 10) as it increases the cost and context window of prompts. On the other hand, shorter tweets ( < 5) do not provide enough context information. Therefore, we randomly selected from the given range of tweets. We also included tweets with emojis in the CoT examples as emojis also serve as trigger words in the given dataset. Similarly, we randomly selected from three different scenarios of Neutral labels: bot-like, opinionated, and fact-stating tweets.
10
11
12

heading: Future Work
text: Therefore, a promising area for future research is to explore methods for converting these tokens into meaningful words or alternative representations