# A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability
## 1 Introduction  
The recognition of human behavior is a task that has been of increasing interest to our society in recent years due to its ability to enable a deeper understanding of users in the most diverse scenarios. By monitoring humans, machine intelligence systems can detect patterns in behavior and suggest changes that could improve interaction and quality of experience. One of the many sub-tasks involved in the recognition of behavior is called emotion recognition, which, from multiple definitions in both computer science and psychology research, can be defined as the process of identifying emotions based on nonverbal cues given by the person [1; 2; 3; 4; 5; 6; 7]. The nonverbal aspect is essential in many contexts where the user needs to interact with environments instead of systems, allowing for an unobtrusive experience and a perception of emotion that is not forced. Especially in the context of smart environments, which can be extended to smart cities, understanding their user is crucial. Given the focus of smart cities on building more attractive and lively spaces, physical sensors and internet-of-things frameworks are typically deployed to measure various aspects of a region, such as weather, pollution, and traffic. However, they need more focus on understanding how their citizens feel near specific areas in the city and how changes could impact their perception and feeling about those spaces [8]. There have been studies focused on understanding the impact of emotion recognition approaches for decision making in these scenarios, mainly to understand the correlation of green spaces, such as public parks and forests, on the urban scene. For instance, the work by [9] studied preferences of ornamental plant species for green spaces by leveraging emotion perception. In addition, the work by [10] used selfies to extract emotion, revealing that visitors from parks near the city center are more susceptible to displaying negative emotions. Other works have focused on different scenarios, such as [11], which investigated the relationship between perceived emotions on the university campus on variables such as environment and habitation availability, and [12], which investigated how sound pollution can affect perceived emotion. Therefore, by focusing on human factors, smart cities can engage with the citizens in a personalized way and have real-time, not modeled, feedback about their services or public spaces. Along the nonverbal axis, there are many cues that humans can naturally perceive from different points of view. For example, involuntary movements (facial expressions, body language, and eye gaze), voluntary movements (touching and avoidance of touching by the peer), or conversational cues (changes in subject, lies, superficial talk, and variations in speech tonality). Although each sub-group of nonverbal communication has significant research works that propose approaches for understanding them, in this work, we focus on those which can be understood through vision-based approaches and in the wild, discarding, for instance, datasets for emotion recognition based on speech tonality, since in these uncontrolled scenarios people will not always be in conversation with peers and a high-quality recording of the voice would be very difficult to achieve. The usage of other nonverbal cues is what currently separates the literature into two tasks: facial expression recognition (FER) and emotion recognition. The first task is motivated by the fact that facial expressions carry many discriminative features that we can correlate with emotion. Early approaches for FER focused on lab-controlled scenarios and posed expressions. Throughout the years, competitions such as Emotion Recognition in the Wild (EmotiW) [13] have been driving this task towards less restrictive approaches that could also be deployed in the wild. However, such techniques are still restricted to scenarios with a visible face and are harmed when the face is occluded or not directly aligned with the camera [14; 15].