## 3 Methodology
### Lexicon Distillation Process  
This process extracts and categorizes unique tokens from the preprocessed messages. Instead of considering the frequency of each token, lexicon distillation classifies tokens based on their association with specific emotions. The process involves several necessary steps:  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Datasets**} & \multirow{2}{*}{**Origin**} & \multirow{2}{*}{
\begin{tabular}{c} **Number of** \\ **Messages** \\ \end{tabular} } & \multicolumn{6}{c}{**Emotions**} \\ \cline{3-8}  & & & **Anger** & **Disgust** & **Fear** & **Joy** & **Sadness** & **Surprise** \\ \hline GoEmotions & Reddit forum & 38,237 & 7021 & 1012 & 929 & 21,730 & 4030 & 6668 \\ \hline XED & Subtitles from movies & 13,682 & 3828 & 2317 & 2439 & 2832 & 2464 & 2442 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Emotion distribution of messages used from each dataset. This represents the quantity of messages per emotion that were used after removing the ‘neutral’ class in both datasets, and the ‘anticipation’ and ‘trust’ classes in the XED dataset.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Datasets**} & \multirow{2}{*}{**Origin**} & \multirow{2}{*}{\
\begin{tabular}{c} **Number of** \\ **Messages** \\ \end{tabular} } & \multicolumn{6}{c}{**Emotions**} \\ \cline{3-8}  & & & **Negative** & **Joy** & **Surprise** \\ \hline GoEmotions & & & & & \\ Aggregated & Reddit forum & 38,237 & 12,942 & 21,730 & 6668 \\ \hline XED Aggregated &
\begin{tabular}{c} Subtitles \\ from movies \\ \end{tabular} & 13,682 & 9425 & 2832 & 2442 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Emotion distribution of the messages used from both datasets after aggregating all messages that contain negative emotions into the ‘negative’ category.  
Input: Dataset  
1. Tokenization: Tokenize each message from the dataset into individual words.
2. Preprocessing: Remove tokens that are stopwords or non-alphanumeric and lowercase the text.
3. Calculate the token-emotion association and Threshold Setting: Choose a threshold value in [0, 1]. A threshold of 0.0 indicates that a token must appear in at least one message labeled with a particular emotion to be assigned that emotion in the lexicon. Any presence of the token in messages associated with an emotion is sufficient for labeling the token with it.
4. Token Analysis: Determine each token's association with a particular emotion based on the percentage of messages containing the token labeled with that emotion. Identify the messages containing the token. If the percentage of these messages meeting or exceeding the threshold is labeled with a particular emotion, assign that emotion to the token in the lexicon. Repeat this analysis for each token across all emotions. Include tokens that do not meet the threshold for any emotion in the lexicon with a zero value for each emotion.
5. Assign the token to emotion or discard it.
6. Lexicon compilation: Compile the evaluated tokens into a lexicon. Represent each token with a six-component vector, where each component corresponds to one of the six emotions. Association with the emotion is given in a Boolean manner with 0 meaning no association, and 1 meaning association.  
Output: Distilled Lexicon  
Figure 1 shows a flow chart that describes the process of distilling a lexicon from an annotated dataset.