# A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability
## 3 Datasets used in State-of-the-Art Techniques  
In this section, we will introduce the datasets currently employed as benchmarks for state-of-the-art techniques. Rather than providing an exhaustive overview of past datasets for emotion recognition, we focus on the recent efforts currently being employed and evaluated in the state-of-the-art techniques. We show samples from these datasets in Figure 1 and give an overview of these datasets in Table 2. As discussed previously, due to the importance of context and other nonverbal cues for perceiving emotion, we will focus our efforts on datasets that explore such features or have them available, discarding, for instance, datasets for facial expression recognition (FER) in this evaluation. Therefore, datasets such as AffectNet [41], the facial expression recognition 2013 (FER2013) dataset [42], and the extended Cohn-Kanade dataset (CK+) [43] will not be considered in this evaluation, neither will techniques that evaluate solely using these datasets, such as the technique proposed by [PERSON] et al. [44], the work by [PERSON] and [PERSON] [45], distract your attention (DAN) [46], exploiting emotional dependencies with graph convolutional networks for facial expression recognition (Emotion-GCN) [47], the model proposed by [PERSON] et al. [48], the model proposed by [PERSON] et al. [49], frame attention networks (FAN) [50], and many others that place themselves as benchmarks for FER. The Acted Facial Expressions in the Wild (AfeW) dataset [51] was proposed to tackle the limitation imposed by the lack of data from real-world scenarios. At the time of publication, datasets for this task were mainly composed of images recorded in laboratory scenarios with posed expressions. The authors argue that, ideally a dataset for this task would be recorded with spontaneous expressions and in real-world environments, which continues to be difficult today. Therefore, they propose using scenes extracted from movies, that contain environments close to the real world. We show samples of this dataset in Figure 1a. The authors extracted clips from 54 movies containing subtitles, which were later parsed for expression-related keywords that could assist the labeler. Two labelers annotated each sample with an expression. For each person present in the scene, the labels also contained information on the head pose, age of character, gender, and expression of the person. A static subset, SfeW, was also generated from this dataset. They generated 1426 video clips with sequence lengths from 300 to 5400 ms and with 330 subjects (performers). Later, the AfeW was extended to the AfeW-VA dataset [52], which labeled emotion using a continuous model. We will discuss these annotations in Section refsub:cont. Figure 1: Datasets currently used for the task of emotion recognition. Samples were extracted directly from the dataset, except for (**f**), which was extracted from their arXiv manuscript with a CC-BY-SA license since the download link for the dataset is currently offline. Although some datasets explicitly show faces, such as (**a,c**), others, such as (**b,d**) have samples with severe occlusion, given their focus on other nonverbal cues. The Emotions in Context (EMOTIC) dataset [53] is a benchmark widely employed in the current literature for emotion recognition. The objective behind the construction of EMOTIC was to gather images and subjects in their natural, unconstrained environments, including contexts. The authors based their collection on well-established datasets such as Microsoft's common objects in context (MSCOCO) dataset [57] and ADE20K [58], and expanded these samples with images captured using Google Images. The same authors later revisited the dataset in their 2019 work _Context based emotion recognition using emotic dataset_[16]. Please note that the latter version is discussed in this work. In addition, we show samples of this dataset in Figure 1b. EMOTIC is a dataset widely used today in the literature due to its proximity to an in-the-wild setting.