## 3 Methodology
### Data Preprocessing  
Regarding the GoEmotions dataset, we used the Ekman level of emotion classification proposed by the authors of the original paper, excluding the neutral category. This resulted in a dataset of 38,237 messages [37]. The neutral class was not considered for this paper because, as noted in the source paper, it did not include messages with no emotion or with all emotions present, but did include messages that were difficult to classify as containing a specific emotion.  
For the XED dataset, the emotions considered were those that are part of the Ekman emotion theory, as this was the classification also used in the GoEmotions dataset: anger, disgust, fear, joy, sadness, and surprise [36; 47]. The emotion labels 'anticipation', 'trust', and 'neutral' were excluded.  
For this dataset, the granularity of the annotations is at the sentence level; however, the shortest movie subtitle present is \"!\" and the longest subtitle includes three sentences [36].  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Study** & **Source** & \
\begin{tabular}{c} **Number of** \\ **Emotions** \\ \end{tabular} & **Multi** & **Macro F1** & **Accuracy** & **Is It Balanced?** \\ \hline [PERSON] and [PERSON] [43] & Twitter & 8 & No & N/A & 95.68\% & No \\ \hline [PERSON] and [PERSON] [43] & Twitter & 24 & No & 87.47\% & N/A & No \\ \hline [PERSON] et al. [44] & Books, etc. & 8 + neu & No & 60.4\% \(\mu\) & N/A &
\begin{tabular}{c} Yes \\ Only after dropping disgust and surprise. \\ \end{tabular} \\ \hline [PERSON] et al. [37] & Reddit & 27 & Yes & 46\% & N/A & No \\ \hline [PERSON] et al. [37] & Reddit & 6 & No & 64\% & N/A & No \\ \hline XED [36] & Movie Subtitles & 8 + neu & Yes & 53.6\% & 54.4\% & Yes \\ \hline \hline \end{tabular}
\end{table}
Table 1: Metrics used in emotion-annotated datasets. The symbol \(\mu\) indicates that the micro F1-score was used, instead of the macro F1-score and N/A indicates that the information is not available ([PERSON] et al., 2020) [36].  
Due to this discrepancy, we considered the level of granularity to be at the message level. From this dataset, only the English messages were considered, which resulted in the use of 13,682 English messages (see Table 2).  
A frequent problem with data is that they can be skewed, with the majority of examples pertaining to one class [23; 48; 49]. When this occurs, the dataset is imbalanced [48; 49]. A balanced dataset has a similar number of cases in each class [49]. It is important to determine whether a dataset is balanced because predictive models developed using imbalanced datasets tend to perform better for the largest class [49].  
Table 2 shows that, except for the emotion label 'anger', the XED dataset presents more balanced classes compared to the GoEmotions dataset. The GoEmotions dataset exhibits class imbalance; the number of messages labeled as 'joy' in GoEmotions is more than twenty times greater than that of message labeled as 'fear'. This imbalance may stem from using [PERSON]'s emotion classification, where 'joy' is the only positive emotion included [37]. Considering this, we created two datasets: GoEmotions Aggregated and XED Aggregated. These datasets consolidate the emotions into three categories:'surprise', 'joy', and 'negative emotions'. The 'negative emotions' category combines the emotions 'anger', 'disgust', 'fear', and'sadness'. Table 3 provides a detailed description of these datasets. The objective of aggregating the negative emotions into one category is to see if there are changes in the F1-scores when classes are more balanced or when negative emotions are part of the same label.  
Both datasets underwent identical preprocessing to standardize the text data for consistent analysis. Using the SpaCy library [50], stopwords and non-alphanumeric characters were removed from the messages, and the text was lowercased. The idea behind removing stopwords is that they are frequently used words that might not convey information and may reduce accuracy and performance if they are not eliminated [51].