# A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability
## 1 Introduction
These restrictions motivated researchers to fuse facial expressions with other types of nonverbal cues, these approaches are what is known today as emotion recognition. Approaches such as the work by [PERSON] et al. [16], context-aware emotion recognition networks (CAER-Net) [17] and global-local attention for emotion recognition network (GLAMOR-Net) [18] proposed the extraction of background features referenced to a context, while approaches such as emotion recognition using adaptive multi-cues (EmotiRAM) [19] and the work by [PERSON] et al. [20] proposed, beyond context, to leverage body language. Therefore, combining multiple cues may lead to more robust techniques since, in case of the occlusion of a specific cue, the model could rely on other cues to make predictions. As the evaluations on the benchmarks evolve, there is a tendency to deploy these systems in real-life scenarios to contribute to society and generate results outside the academic scenario, in which we usually validate datasets. However, it is well-known that deep learning systems, such as those employed for emotion recognition techniques, depend highly on data availability and quality. Although there are several datasets for this task, each dataset has its particularities and limitations, which could severely harm the capacity of execution in real scenarios. Furthermore, there are limitations regarding sample bias, in which samples present in the dataset are not representative of the real world, and recall bias, which is caused by the way multiple annotators are usually handled in these scenarios. In this work, we survey datasets currently used for benchmarking techniques for emotion recognition. Specifically, we focus on vision-based datasets containing images or videos for evaluation. Therefore, we do not evaluate datasets focusing on speech tonality, for example. Some of the contributions of our work are:  
* We review how the behavioral psychology literature describes emotion perception in humans and correlate this with current datasets (Section 2);
* We survey and list the datasets currently employed for benchmarking in the state-of-the-art techniques (Section 3);
* We explore the difference in the annotations of these datasets and how they could impact training and evaluating techniques (Section 3.2);
* We discuss annotations described using continuous models, such as the valence, arousal and dominance (VAD) model, and how they can harm the ability of a model to understand emotion (Section 3.3);
* We investigate the presence of nonverbal cues beyond facial expression in the datasets of the state-of-the-art techniques and propose experiments regarding their representativeness, visibility, and data quality (Section 3.4);
* We discuss possible application scenarios for emotion recognition extracted from published works and how each dataset can impact positively and negatively according to its features (Section 4). This survey also differs from other published surveys in emotion recognition, such as [21; 22; 23; 24; 25], since we focus on listing, understanding, and discussing datasets and the permissions and limitations they bring to techniques, while surveys usually focus on listing techniques, their limitations, specifications, and results on benchmarks. To the best of the authors' knowledge, this work is the first survey that focuses on datasets instead of techniques.