## 4 Experimental Results
### Zero-Shot Transfer  
Our taxonomy (Table 1) includes all emotions used in the selected public benchmark datasets, except for \"realization\" in GoEmotions. As such, it is possible to directly evaluate a model trained on our data on these datasets without any adaptation by simply mapping labels (See Appendix A). To handle \"realization\", we simply merge it with \"surprise\" (based on the definition and mapping provided in Ref. [13]). This creates the dataset we refer to in this section as \"GoEmotions26\". This experimental setup is, conceptually, the inverse of the UnifyEmotions [11] where instead the labels of the target dataset are mapped into a common taxonomy and the classifiers are trained and evaluated on that. Such an evaluation makes the datasets and classifiers use a more coarse-grained taxonomy than originally intended.  
To compare the results, we train and evaluate models using 5-fold cross validation on Tales, ISEAR, and EMOINT. These models follow the same architecture described in Section 2.6 and are trained with the hyperparameters in Table 12. We also trained a model on GoEmotions, using the same model architecture, following the exact same procedure as the supervised model trained on EmoLit (Section 4.2) and the same hyperparameters (Table 9 with the model from the best epoch selected on the basis of the GoEmotions development set. Table 13 shows the results.  
\begin{table}
\begin{tabular}{l c c c} \hline
**Encoder Name** & **Encoder Architecture** & **Encoder Parameters** & **F1 (Macro)** \\ \hline RoBERTa-large & L = 24, H = 1024, A = 16 & 355 M & 0.59 \\ BERT-large & L = 24, H = 1024, A = 16 & 340 M & 0.59 \\ RoBERTa-base & L = 12, H = 768, A = 12 & 125 M & 0.59 \\ BERT-base & L = 12, H = 768, A = 12 & 110 M & 0.58 \\ DistilBERTa-base & L = 6, H = 768, A = 12 & 82 M & 0.58 \\ DistilBERT-base & L = 6, H = 768, A = 12 & 66 M & 0.56 \\ \hline \end{tabular}
\end{table}
Table 10: Comparison between different model sizes trained on our training set and evaluated on the gold set. The F1 score is similar for all RoBERTa model sizes, although DISTIL is slightly worse (0.59 vs. 0.58). BERT underperforms for RoBERTa except for LARGE.  
\begin{table}
\begin{tabular}{l c c c} \hline
**Dataset** & **Emotions** & **Model** & **F1 (Macro)** \\ \hline EmoLit (ours) & 38 & RoBERTa & 0.59 \\  & & BERT & 0.58 \\ \hline SemEval-2018 Task-1C & 11 & RoBERTa & 0.60 [30] \\ XED & 8 & BERT & 0.54 [26] \\ GoEmotions & 27 & BERT & 0.46 [13] \\ \hline \end{tabular}
\end{table}
Table 11: Comparison with the results reported in related work, using _BASE_ model sizes.  
\begin{table}
\begin{tabular}{l c c} \hline
**Hyperparameter** & **Value** \\ \hline Batch Size & 8 \\ Learning Rate & \(2\times 10^{-5}\) \\ Epochs & 3 \\ \hline \end{tabular}
\end{table}
Table 12: Hyperparameter values used for training the Cross-Validation models.  
In this experiment, training the model on soft labels outperformed hard labels. Both models trained on our dataset outperformed the model trained on GoEmotions on the other datasets, with the best being the model trained with soft labels, which attains an average score of 0.62 over the first 3 datasets, followed by the model trained on hard labels at 0.61 and lastly, the model trained on GoEmotions with 0.59. Unsurprisingly, training models on the same dataset (relative to evaluation) significantly outperform zero-shot transfer. The conclusion of this experiment is that models trained on our dataset learn about the emotions they are trained to distinguish and can be used in zero-shot emotion classification. Surprisingly, the model trained on GoEmotions (Reddit posts) did not outperform models trained on our dataset (literature) when evaluated on EMOINT (tweets) but it did match our results on Tales (literature).