Region-Based Emotion Recognition via Superpixel Feature Pooling

Authors: zhihang  ren
yifan  wang
tsung-wei  ke
yunhui  guo
stella x yu
david  whitney
Venue: thecvf_com_CVPR_2024_Workshop_POETS
Published: 2024-05-13 07:56:21

Abstract: Perceiving other people's emotional states is fundamentally important for successful social interactions. Traditional emotion recognition algorithms exclusively focus on facial expressions, ignoring the critical role of background context, which is now known to be necessary to accurately represent and understand the emotions of others. More recent studies have utilized different fusing techniques to combine facial and contextual information in visual scenes, but these approaches are limited to detection-based methods. In this study, we propose a new region-based emotion recognition pipeline via superpixel feature pooling that does not rely on detection. Our proposed pipeline consists of three types of blocks, including an initial over-segmentation block, the superpixel pooling block, and the emotion recognition block. On EMOTIC and VEATIC datasets, our proposed pipeline improves state-of-the-art performance by 68.57% and 11.79% respectively. We also achieve competitive performance on the CAER-S dataset.

Main paper: 
 introduction: 
 Recognizing human emotions is routine and necessary to successfully navigate social interactions on a daily basis. Nowadays, as robotic techniques grow fast, it is natural to make future intelligent machines socially aware in the human-populated world. Moreover, emotion recognition may help the autonomous driving system to anticipate pedestrians' or drivers' intentions and react properly. Therefore, understanding emotion perception mechanisms and designing automatic emotion recognition methods are essential for future robotics and autonomous driving developments.Over the past several years, the interest in utilizing deep learning models to automatically recognize emotional states has grown rapidly. Following a long tradition of research on emotion recognition in the fields of psychology, neuro-science, and vision science, previous computer vision research focused almost exclusively on facial expressions as the key information for emotion recognition. This is unsurprising, as facial expressions seem to be the most direct and inherent way for humans to understand the emotions of others. Consequently, many early datasets annotated only the emotional states of character faces or lab-controlled human interactions, treating them as if they are independent of the context [5, 11, 14, 24-26, 39, 40, 44, 46, 53, 56, 57, 66] . With this massive annotated data involving facial expressions, researchers primarily focused on the analysis of facial expression to predict emotions [6, [62] [63] [64] . Later studies also found extra information, such as shoulder location and body pose, could be utilized to infer emotional states [43, 54] . Overall, though, early datasets and recognition models focus strictly on character-specific information to infer emotional states.Although the character itself-including the facial expression-contains a great deal of information about its emotional state, many studies in Psychology have proven that context information is critically important for accurate emotion perception [3, 4, 10] . In many scenarios, scene context influences the perception of human emotion even though the facial expression is unchanged or very similar [1, 37, 52] . And, scene context can explain as much of the variance in human emotion perception as facial expression [9] . For these reasons, annotations of isolated faces may not accurately reflect true human emotion, and contextbased emotion datasets are necessary.In light of the importance of context in emotion recognition, several datasets that include contextual information have emerged recently [23, 28, 29, 51] . Those datasets not only contain characters but also large areas of the surrounding context. In turn, recent algorithms [17, 21, 30, 31, 41, 45, 65, 68] then focus on different feature extraction methods and fusing techniques for various types of information and visual features.To locate different types of visual information, such as the face, body, and background scene, previous methods rely heavily on object detection methods [49] . They often utilize rectangular bounding boxes to select or mask out certain image blocks (Figure 1 (b)). Then, visual feature encoding and fusing modules are utilized to represent the visual information of the whole scene.When humans try to perceive the emotions of other people, they rely on bottom-up and top-down visual processes, where fine-scale visual features and coarse-scale object and scene knowledge mutually facilitate each other [7, 38] . In this process, there are no bounding box structures; instead, there are fine or coarse scales of object and scene regions (superpixels). With this insight, researchers have started to utilize superpixels in a variety of computer vision models [2, 15, 16, 18, 22, 42, 48, 50, 55, 61, 69] .So far, superpixel-based models have been successful in understanding and grouping semantically similar image regions, achieving good performance on part parsing, saliency detection, and image segmentation tasks. As emotion recognition requires the understanding of characters' facial expressions, as well as their interactions with different regions of objects and background scenes, the superpixel approach could be useful for emotion recognition. But, to the best of our knowledge, no emotion recognition method has utilized superpixel-based methods.In this paper, we propose a new emotion recognition method that utilizes superpixel-level visual features. Our proposed method consists of three types of blocks: 1) an over-segmentation block to initialize the fine-grained segments and generate initial superpixel features; 2) the superpixel pooling block to learn the grouping policy and aggregate the current level finer-scale features to the next level coarser-scale features; and 3) the emotion recognition block for either emotion regression or classification tasks based on the final aggregated feature. We test our proposed method on three public context-aware emotion recognition datasets, EMOTIC [28] , CAER [29] , and VEATIC [51] . We achieve state-of-the-art performance on EMOTIC [28] and VEATIC [51] , with 68.57% improvement on EMOTIC and 11.79% improvement on VEATIC. We also achieve competitive results on CAER-S [29] . Moreover, we show that by using superpixels as feature extraction anchors, we can naturally obtain semantically similar superpixels for free with the learned grouping policy.In summary, our contribution of this work lies in three aspects: 1. We propose the first region-based emotion recognition method via superpixel feature pooling. 2. We achieve state-of-the-art emotion recognition performance on VEATIC and EMOTIC datasets with 68.57% improvement on EMOTIC and 11.79% improvement on VEATIC. 3. We show that the method can also provide us with clusters that contain semantically similar superpixels via the learned grouping policy. 
related work 2.1. context-aware emotion recognition: 
 When inferring emotion states, the context-aware emotion recognition methods do not only rely solely on the face or body information but also consider the emotion cues from scene context and background information. Traditional context-aware emotion recognition methods invariably extract multiple representations from various visual information sources and then apply feature fusion to make the final prediction [17, 21, 30, 31, 41, 45, 65, 68] . Object detection methods are widely utilized to identify the information sources, marking them with rectangular bounding boxes. For example, the model released along with the EMOTIC dataset [28] fused the body region feature and the whole image as the context feature via a Convolutional Neural Network (CNN). In this study, we utilize superpixel as the feature anchor for subsequent feature fusing, which does not rely on object detection or bounding boxes.Vision Transformers (ViT) [13] have achieved amazing performance in image recognition. They treat images as sets of rectangular patch tokens and employ an attention mechanism in learning [60] . ViTs can be computationally expensive. To improve their efficiency, hierarchical transformers aim to reduce the number of tokens by spatial pooling [12, 20, 32, 34] . Other approaches directly prune tokens away according to their significance scores [8, 19, 35, 47, 67] . Our grouping procedure looks like the latter approach. However, we focus on grouping different visual regions for emotion recognition while those methods aim for efficiency. Additionally, we use superpixels as input units instead of square patches.Superpixels are sets of locally connected pixels that encapsulate coherent structures, such as colors [48] . Intuitively, superpixels have been utilized in various computer vision tasks that involve dense labeling, including part parsing [42] , saliency detection [50] , and image segmentation [15, 16, 18, 50, 55, 61] . Recent studies have replaced patches with superpixel tokens in ViT architectures to achieve semantic segmentation [22, 69] . In this study, we adopt superpixels as the visual feature extraction anchors for feature fusing in emotion recognition tasks. 
method: 
 Inspired by the human emotion recognition process, we propose the first region-based emotion recognition method via superpixel feature pooling. Our idea revolves around utilizing superpixels to enhance our understanding of characters' facial expressions, along with their interactions within various regions of objects and background scenes. Figure 2 illustrates an overview of the method. The image/frame is at first over-segmented to obtain the fine-grained segments.At each level, the finer-scale superpixels are grouped into coarser-scale superpixels via the superpixel pooling block.The corresponding superpixel features are aggregated according to the learned pooling policy, processed by the visual transformer block to learn better features, and then sent to the next level pooling and aggregation. At last, the emotion recognition block can take the final aggregated feature to complete the emotion recognition task. Now, we introduce each block respectively.Each time, we start with the finest-level pixel grouping, denoted as G 0 , i.e., the initial image region grouping. These groupings are based on low-level visual cues and designed to align with image contours. In this paper, we utilize SEEDS [59] to obtain the locally connected and color-wise coherent regions, i.e., the superpixels. Then, we progressively group these superpixels into coarser regions and fuse the corresponding superpixel features to get the final aggregated feature for emotion recognition.The initial pixel features of the input image, X cnn , are obtained via a convolutional neural network (CNN). These pixel features are then aggregated within each superpixel in G 0 to create the initial superpixel features, referred to as X G . The aggregation is achieved by averaging each pixel feature within a specific superpixel. After this, we append a class token X class , and positional encodings E pos into the initial features X G . We set E pos to align with the resolution of the CNN features X cnn and then average it within each superpixel. The resulting input segment features are defined asZ 0 = [X class ; X G ] + [0; E pos ]To form better features for the aggregated superpixel features at each grouping level, such that semantically similar superpixel features at each grouping level would be more similar in the feature space and vice versa, we apply two visual transformer (ViT) blocks before the superpixel pooling. Then, we pool the similar fine-scale regions into a coarser scale and move to the next level. Various pooling strategies can be applied here. In this paper, we adopt a graph pooling strategy [22] . The similarities between different superpixel features in neighbor levels are computed and utilized to quantify the soft assignment probability P l from a finer level l-1 to a coarser level l. Then, the next level of coarser groupings G l can be determined by the finer level grouping G l-1 and the soft assignment probability P l .G l = G l-1 × P l = G 0 l i=1 P i (1)At the final stage, we will have an aggregated feature that contains the combined visual information from separate visual regions. Then, we utilize a multilayer perceptron (MLP) to achieve the emotion recognition task. At last, we utilize either categorical emotion states or continuous emotion ratings to guide the training. We emphasize that no segmentation maps are utilized in the training. The grouping of superpixels is only trained to make good emotion recognition results, though the method naturally learned how to group superpixels efficiently. Aggregated Feature Emotion Recognition Figure 2. Overview of the method: In our proposed method, We start with over-segmented regions, i.e., the superpixels, and then gradually group similar superpixels, and aggregate features of corresponding superpixels. We utilize the final aggregated feature for emotion recognition. Along with the training, we also obtain clusters that contain semantically similar pixels, e.g., the red face region in G2. 
datasets and evaluation metrics: 
 Datasets: We conduct our experiments on three standard datasets for the context-aware emotion recognition task, namely EMOTIC [28] , CAER-S [29] , and VEATIC [51] . EMOTIC contains 23,571 images of 34,320 annotated subjects in uncontrolled environments. The annotation of these images contains the bounding boxes of the target subjects' body regions and 26 discrete emotion categories. CAER-S includes 70k static images extracted from video clips of 79 TV shows to predict emotional states. These images are annotated with 7 emotion categories: Anger, Disgust, Fear, Happy, Sad, Surprise, and Neutral. VEATIC has 124 video clips from Hollywood movies, documentaries, and home videos with continuous valence and arousal ratings of each frame via real-time annotation. Evaluation Metrics: Following [27, 41, 65] , we utilize the standard classification accuracy to evaluate performance on CAER-S. For VEATIC, the root mean square error (RMSE) is used. At last, we utilize the mean Average Precision (mAP) to evaluate the classification results on the EMOTIC.Given the fact that our model is tested on three datasets, we select several models with different structures tested on each of the corresponding datasets for comparison.For the EMOTIC dataset, we select seven distinct models for comparison. EMOT-Net [27] is a two-branch Convolutional Neural Network model, whose unique branches capture the body features and context features separately. GCN-CNN [68] is a Graph Convolutional Network trying to infer emotion relationships utilizing the affective graph constructed by context elements. CAER-Net [29] is a double-stream Convolutional Neural Network model with an adaptive fusion module focusing on inferring emotion by integrating context information with facial information. RRLA [30] proposed the Body-Object Attention module and Body Part Attention module to estimate the importance of body parts and background information. VRD [21] utilizes both the spatial and semantic features by attention mechanism to learn the impact of each part on emotion recognition. EmotiCon [41] takes advantage of visual attention and depth maps to obtain multi-modal information. And CCIM [65] utilizes causal inference for model training. For CAER-S Dataset, we have added two more models for comparison in addition to the ones mentioned above. SIB-Net [31] is inspired by the study of context-containing order, interaction, and bias relationships. GRERN [17] proposes a framework based on a Graph Convolutional Network to do emotion classification utilizing the region-wise semantic relationships. For VEATIC Dataset, We compare our model to VEATIC-NET [51] , which is a two-stream Video Transformer using the attention mechanism to learn the contextual relationships between frames. We reproduce the results on the corresponding datasets based on the details given by the models above.We conducted supervised training following the setup of DeiT [58] . The model is trained on 4 NVIDIA GeForce RTX 2080 Ti GPUs. For the hyperparameters, we have 4 levels in total for the superpixel pooling. There are 64, 32, 16, and 8 clusters respectively at each grouping level and the batch size of the data is 64. For the superpixel inputs, we utilize 196 pixels as default. We resize the input images into 224 × 224 and apply normalization to the images. Our model is trained using the AdamW optimizer [33] . For the learning rate schedule, we use a linear warmup of 5 epochs to reach a peak learning rate of 5.0 × 10 -4 from 1.0 × 10 -6 , followed by a cosine decay of 30 epochs to decay the final learning rate to minimum learning rate of 1.0 × 10 -5 . In Table 1 , we see that our proposed method significantly improves the recognition precision of most emotion categories. In particular, compared to EMOT-NET [27] , GCN-CNN [68] , CAER-NET [29] , RRLA [30] , VRD [21] , EmotiCon [41] , and CCIM [65] , our proposed method improve the mAP scores by 136.16%, 134.23%, 176.56%, 103.52%, 87.60%, 86.96%, and 68.57% respectively. For certain emotion categories, the emotion recognition average precision is even improved drastically compared to SOTA performances, such as Yearning (+349.34%), Sensitivity (+290.14%), Pain (+288.46%), Anger (+269.81%), Embarrassment (+262.16%), and Fatigue (+247.52%).Table 2 shows our proposed method performs competitively with recently released emotion recognition methods. It is worth noting that the CAER dataset utilized very few annotators (six) and has little control over the annotation quality compared to EMOTIC and VEATIC datasets. As annotation uncertainty and bias may result from the insufficiency of annotators [36, 51] , this may influence the interpretation of any model's performance to some extent. We also test our proposed method on a recently released dataset, VEATIC [51] . Our proposed method improves the RMSE of the overall rating by 11.79%. In terms of valence and arousal, our method improves 16.44% and 5.89% respectively.Although our supervised training process does not utilize segmentation maps as guidance, through the learned grouping policy, we show that the semantically similar finer-scale superpixels are pooled to form coarser-scale regions at the next level. As in Figure 3 (e), we can find groupings of facial regions and the object/scene regions which the character is interacting with. Compared to traditional detection-based methods, where the emotion recognition module passively encodes the visual information selected by bounding boxes, our proposed method proactively learns which superpixels to group and aggregate. It is clear to see which regions contribute similarly to the final emotion prediction. Thus, by utilizing superpixels to enhance our understanding of characters' facial expressions, we can achieve more accurate emotion recognition. 
conclusion: 
 In this paper, we proposed the first region-based emotion recognition method via superpixel feature pooling. It achieves state-of-the-art emotion recognition performance on VEATIC and EMOTIC datasets. It also achieves competitive results on CAER-S dataset. Moreover, the proposed method can also provide us with clusters that contain semantically similar superpixels via the learned grouping policy. 
 
 Appendix: