## 2 Related Works
### Image Emotion Prediction/Annotation  
Understanding emotions requires effective models for their representation. Two primary types of models prevail categorical emotion states (CESs) and dimensional emotion space (DES). CES models employ discrete categories like amusement, contentment, anger, and sadness to classify emotions, offering a straightforward and easily interpretable framework. However, this simplicity comes at the cost of limited descriptive power, needing help capturing emotional experiences' nuances and complexities. In contrast, DES models provide greater flexibility and power in describing emotions, typically utilizing a 2D valence-arousal coordinate space or a 3D valence-arousal-dominance space. This approach represents a broader range of emotional states and their varying intensities.  
Due to its enhanced descriptive capabilities, we utilize the 3D DES model for predicting image emotions, capturing the core effect of valence (pleasantness-unpleasantness) and arousal (activation-deactivation). However, a comprehensive understanding of emotion representation necessitates a thorough review of both CES and DES models, exploring their strengths, limitations, and applications in diverse fields. This section delves into the intricacies of both models, providing a comparative analysis and highlighting their significance in emotional research and artificial intelligence. In Table 2, we summarized popular datasets based on their different emotion annotation methods.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Method** & **Paper** & **Architecture** & **Emotions** & **Database** \\ \hline HC features & [PERSON]; [PERSON] [16] & AUs & 7 & CK+ [35] \\  & [PERSON], et al. [15] & AUs & 23 & CK+ [35], CE [36], DISFA [37] \\  & [PERSON], et al. [17] & LBP & 6 & Private \\  & [PERSON], et al. [18] & LBP & 7 & CK+ [35] \\  & [PERSON]. [19] & HoG & 7 & CK+ [35] \\ \hline CNN & [PERSON], et al. [38] & CNN & 8 & MMI [39], FER2013 [40] \\  & [PERSON], [41] & CNN + LSTM & 6 & MMI [39] CASME II [42] \\  & [PERSON] et al. [26] & CNN + RNN & 7 & EmotiW [43] \\  & [PERSON], et al. [23] & CNN & 8, 50 (AU) & CK+ [35], NovaEmotions [23] \\  & [PERSON], et al. [27] & CNN + LSTM & 12 (AU) & BP4D [44] \\  & [PERSON], et al. [45] & I3D & 23 & CK+ [35], DISFA [37] \\  & [PERSON], et al. [28] & CNN + LSTM & 6 & CK+ [35], MMI [39] \\  & [PERSON], et al. (EmoFAN) [24] & CNN & 8 & AffectNet [46], SEWA [47], FER+ [48] \\  & [PERSON], et al. (ESR-9) [25] & CNN & 8 & AffectNet [46], FER+ [48] \\  & [PERSON], et al. (SCN) [49] & CNN & 8 & AffectNet [46], FER+ [48] \\ \hline Transformer & [PERSON], et al. (MVT) [33] & ViT-base & 8 & FER+ [48], AffectNet [46] \\  & [PERSON], et al. (VTFF) [34] & ViT + CNN & 8 & FER+ [48], AffectNet [46] \\ \hline \hline \end{tabular}
\end{table}
Table 1: Review of FER systems.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Datasets** & **Annotation Method** & **Description** & **Metrics** \\ \hline FER2013 [40], JAFFE [50], CK+ [35], EmotiW [43], MMI [39] & Single Label & Multi-class labeling with one label for each emotion & Accuracy, F1, ROC, Cross-Entropy \\ \hline BU-3 DFE [51], BP4D [44], DISFA [37] & EIL & Indicates the intensity of the emotion & L1, L2, Euclidean Distance \\ \hline FER+ [40] & MLL & Use one or more possible basic emotions with weights to estimate the emotion expression (weights could be 0) & ML-KNN, CLM, LIFT \\ \hline s-JAFFE [50], s-BU-3 DFE [51] & EDL & Appropriate proportion of the possible emotions to estimate the emotion expression & L1, L2, Euclidean Distance \\ \hline AffectNet [46], SEWA [47] & 2D emotion & Use 2 dimensions (valence or pleasant and arousal) to estimate the emotion expression & L1, L2, Euclidean Distance \\ \hline \hline \end{tabular}
\end{table}
Table 2: Review of emotion annotation methods and dataset.