# A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability
## 4 Discussion
### Limitations  
**Specific tasks. ** Two of the datasets surveyed in this work (iMiGUE and GroupWalk) were proposed to solve specific tasks. How iMiGUE data was sampled places it close to the optimal scenario of extracting emotion from interviews. However, the simplified category representation used by the authors limits techniques related to the overall perception of emotion, narrowing them down to only \"positive\" or \"negative\" representations. The identity-free aspect of the dataset also negatively impacts how techniques could learn to represent these other emotion categories since, for example, the facial expression is hidden for protection. Therefore, the dataset limits the techniques to a specific task, which could not be expanded for similar applications such as _assessing comfort in job interviews_ or _evaluating the quality of customer care services_, due to these imposed restrictions. Other limitations are present in the GroupWalk dataset, related to sample quality and image resolution. Although the variation in camera placements is a feature that would add robustness to techniques, in some cases, the camera appears to be handheld. We can notice distortions in the videos, probably caused by stabilization software to remove motion blur. Besides the noise and low image quality in some samples, bounding boxes are also annotated directly to the image. Researchers and engineers have been working with data augmentation and proved that employing techniques such as random crops, rotations, and noise can improve the overall performance of the image. However, when these artifacts are written directly to the image, the accuracy of the models could be harmed. We still lack datasets for other specific tasks in the literature. For example, for in-the-wild scenarios, a viewpoint from a security camera would be ideal since it would allow the reuse of this footage for emotion recognition. A possibility would be to augment the data to be close to this scenario; however, this is difficult when using 2D data such as images and 2D body landmarks. **Labels. ** Another limitation in this field of study is related to the labeling of datasets. From this survey, we show that there is at least one significant problem related to the labeling of data that could be harmful to techniques. EMOTIC and BoLD, datasets that employ the annotation format of 26 categories, do not add a neutrality class but rather rely on categories such as _Disconnection_ or _Peace_ that could also be relatable to _Neutral_. However, two problems arise with this approach: first, as we discussed before, visible traits of the person sampled could lead to a correlation with both positive and negative emotions. Second, the high number of possible categories has a direct impact on annotators' agreement. On the other hand, the CAER and CAER-S datasets employ an annotation format of seven categories, one related to neutrality. However, their annotations are not as structured as in EMOTIC, in which one can access individual emotions and demographics for each person in the image. In multiple cases, more than one person is present in the scene, and it is difficult to correlate the annotated sample. _Lack of consistency in labeling._ We also show that there is little concordance among datasets related to which classes should be employed, as we show in Table 6. This is a problem because it limits the possible datasets that one may use regarding a specific application scenario. For example, for an application more closely related to iMiGUE or CAER-S, one may not be able to extract representations related to _Suffering_, which is present in EMOTIC or BoLD. We may argue that this could be achieved through transfer learning approaches such as fine-tuning the model to be able to generate predictions related to a dataset after learning representations from other datasets. Although there is no domain mismatch in this scenario, the data augmentations that are performed by techniques could be harmful to performance.