# A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability
## 3 Datasets used in State-of-the-Art Techniques
### Emotion Categories and Annotations  
Techniques related to affective computing usually focus on some variation of the well-known set of [PERSON]'s [72] basic emotions, which are _Anger_, _Fear_, _Sadness_, _Enjoyment_, _Disgust_, and _Surprise_. The same applies to datasets related to emotion recognition and its parent fields of study, which usually add the _Neutral_ emotion. This set of categorical annotations is the case for the CAER dataset [17], in which videos and images are annotated using [PERSON]'s basic emotions. A limitation, however, in the format of annotation used in both CAER and CAER-S, is that a single annotation is given for the video or image. This means that in images with multiple people, only one label is given, and no bounding box or identifiable information is provided to identify on which person that annotation was made. This harms techniques such as CAER-Net [17], GLAMOR-Net [18], and other techniques trained on this dataset, because the authors usually use the first face detected in the image, as discussed in their research papers. EmotiRAM [19] employs a face selector algorithm that searches for the leading performer in the scene based on the assumption that the annotation would be from them, but is also limited to images with difficult scenarios. We show examples of images with more than one person framed in Figure 2.  
For GroupWalk [56], a completely different set of emotions is proposed. The authors proposed using a new set of seven categories that take into consideration the level of the perceived emotion, which can be _Happy_, _Sad_, or _Angry_, combined with the modifiers _Somewhat_ or _Extremely_, in addition to _Neutral_. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Approach** & **Available From** & **Dataset** \\ \hline [PERSON] et al. [62] & May 2019 & EMOTIC \\ [PERSON] et al. [63] & August 2019 & EMOTIC \\ [PERSON] et al. [56] & June 2020 & EMOTIC \\ [PERSON] et al. [64] & September 2021 & EMOTIC \\ [PERSON] et al. [65] & December 2021 & EMOTIC \\ [PERSON] et al. [66] & August 2022 & EMOTIC \\ [PERSON] et al. [67] & October 2022 & EMOTIC \\ [PERSON] et al. [17] & October 2019 & CAER \\ [PERSON] et al. [17] & October 2019 & CAER-S \\ [PERSON] et al. [68] & January 2021 & CAER-S \\ [PERSON] et al. [69] & May 2021 & CAER-S \\ [PERSON] et al. [70] & July 2021 & CAER-S \\ [PERSON] et al. [18] & December 2021 & CAER-S \\ [PERSON] et al. [18] & December 2021 & NCAER-S 1  \\ [PERSON] et al. [66] & August 2022 & CAER-S \\ [PERSON] et al. [20] & October 2022 & CAER-S \\ [PERSON] et al. [19] & October 2022 & CAER-S \\ [PERSON] et al. [71] & January 2023 & CAER-S \\ \hline \hline \end{tabular}
\end{table} 
Table 3: State-of-the-art techniques, published from May 2019 until January 2023, and the datasets they used for evaluation. In case of duplicates, such as [18], both datasets were used in the evaluation. The authors proposed a different approach for EMOTIC, by defining an extended list of 26 emotional categories also containing [PERSON]'s basic emotions, leading to 20 novel emotional states for comprehension, which was also employed in BoLD. To define these emotional categories, the authors proposed an approach based on word connections (affiliations and relevance of words) and interdependence of words (psychological and affective meaning) to form word groupings. However, the main difference is that neither EMOTIC or BoLD contain the _Neutral_ category, with the authors arguing that, generally, at least one category can be applicable, even with low intensity. However, the lack of a neutral annotation group could lead to neutral images being sampled into opposite groups and therefore harm the learning process of a network. For example, research on social cognition points out that humans perceive emotionally neutral faces depending on visible traits; positive traits are correlated with happiness, and traits involving dominance and threat are correlated with anger [73,74].