## 3 Methodology
### Application of Lexicons  
Each compiled lexicon was applied to both the GoEmotions and XED datasets, as well as their aggregated versions, to facilitate the evaluation of different F1-score variants: micro, macro, and weighted. The weighted F1-score is calculated by determining the F1-score for each class, defined as the harmonic mean of precision and recall, and then multiplying each result by a weight that considers the number of true instances or true positives (TPs) for each class, and then averaging the outcomes [52]. The weighted F1-score is used when there are imbalanced classes and shows a bias toward the majority class [7,52]. Conversely,  
Figure 1: Flow chart describing the process of distilling a lexicon from an annotated dataset. the macro F1-score treats all classes equally by assigning the same weight to each class, meaning that the minority class is as influential as the majority class [52]. Consequently, the model is penalized if it performs poorly on the minority class [52; 53]. The micro F1-score is calculated over the entire dataset; this means that it disregards class membership [54]. The process of applying the lexicon into a dataset is as follows:  
Input data: The distilled lexicon and the dataset. 1. Tokenization and Stopword Removal: Stopwords are removed, and each message from the dataset is tokenized. 2. Token Evaluation: Each message is scanned to find tokens present in the lexicon. 3. Labeling Emotions: If a token is not in the lexicon, the next token is evaluated. Each message is labeled with the emotions associated with the tokens present in both the text and the lexicon. If tokens are linked to different emotions, or if a single token is linked to multiple emotions, the message reflects all detected emotions. Labeling is achieved through a vector, where each component corresponds to one of the emotions (six for the full model, three for the aggregated model). 4. Matrix Update: The vectors are filled with binary values (0 or 1), indicating the absence or presence of each emotion
5. Comparison with Original Labels: After all tokens have been analyzed, the resulting NxM matrix is compared with the original annotated message labels. Output: the performance metrics, the graphics and, the dataset. This process is visually summarized in Figure 2.  
Figure 2: Flow chart describing the process of applying a distilled lexicon to an annotated dataset. Subsequently, confusion matrices were calculated for each emotion and threshold. A confusion matrix is a tool used to evaluate the performance of a classification task [7; 11; 55]. It is divided into four segments:  
1. Number or percentage of examples that are correctly classified as part of a class (true positives or TPs);
2. Number or percentage of examples that are correctly classified as not being part of a class (true negatives or TNs);
3. Number or percentage of examples that are incorrectly classified as being part of a class (false positives or FPs);
4. Number or percentage of examples that are incorrectly classified as not being part of a class (false negatives or FNs). Calculating the confusion matrices assesses the model's classification performance. At the same time, the values that form these matrices are the basis for calculating the F1 measures under varying conditions [38; 56]. The structure of a confusion matrix is shown in Figure 3.  
The comparison of the micro, macro, and weighted F1 metrics is conducted by analyzing the results of the distinct types of F1-scores found after applying the distilled lexicon achieved at various thresholds to emotion-annotated datasets. The specific metrics considered in this paper are recall, precision, and the micro, macro, and weighted F1-scores. These performance metrics are obtained from the data present in the confusion matrix [12]. The F1-score variants are then calculated with the Equations (3)-(6). Acceptable F1-score values vary depending on the domain, task, and specific dataset characteristics.