Two in One Go: Single-stage Emotion Recognition with Decoupled Subject-context Transformer

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 10:32:32

Abstract: Emotion recognition aims to discern the emotional state of subjects within an image, relying on subject-centric and contextual visual cues. Current approaches typically follow a two-stage pipeline: first localize subjects by off-the-shelf detectors, then perform emotion classification through the late fusion of subject and context features. However, the complicated paradigm suffers from disjoint training stages and limited interaction between fine-grained subject-context elements. To address the challenge, we present a single-stage emotion recognition approach, employing a Decoupled Subject-Context Transformer (DSCT), for simultaneous subject localization and emotion classification. Rather than compartmentalizing training stages, we jointly leverage box and emotion signals as supervision to enrich subject-centric feature learning. Furthermore, we introduce DSCT to facilitate interactions between fine-grained subject-context cues in a ``decouple-then-fuse'' manner. The decoupled query tokens—subject queries and context queries—gradually intertwine across layers within DSCT, during which spatial and semantic relations are exploited and aggregated. We evaluate our single-stage framework on two widely used context-aware emotion recognition datasets, CAER-S and EMOTIC. Our approach surpasses two-stage alternatives with fewer parameter numbers, achieving a 3.39% accuracy improvement and a 6.46% average precision gain on CAER-S and EMOTIC datasets, respectively.

Main paper: 
 related work: 
 Visual Emotion Recognition. The Visual Emotion Recognition (VER) task can be broadly categorized into two main paradigms. 1) Two-stage without fusion. Traditional methods focus on utilizing subject-centric regions while treating contextual areas as noise, as observed in various studies [8, 23, 39, 46, 71] . The pipeline includes subject detection and emotion classification. These studies primarily address challenges associated with label uncertainty [4, 5, 19, 30, 48, 51, 52, 62, 76, 77] , micro expressions [37, 67] , and disentangled representations [65, 75] . 2) Two-stage with late fusion. In recent years, the research has paid increasing attention to context-aware emotion recognition, which emphasizes the use of multiple contexts for more robust emotion classification [18, 20, 24, 34, 35, 49, 60, 61, 73] . In addition to two-stage components, the pipeline includes multi-branch and late fusion characteristics. Typically, a multiplestream architecture, followed by a fusion network, is employed to independently encode the subject and context information. Despite the effectiveness of these methods, they suffer from disjoint training stages and limited interaction between fine-grained subject-context elements. In contrast, we present a single-stage approach with an early fusion, employing a Decoupled Subject-Context Transformer, for simultaneous subject localization and emotion classification.End-to-End Object Detection. The end-to-end framework with vision Transformers stirs up wind in the object detection task. DETR [2] streamlines object detection into one step by a set-based loss and a transformer encoder-decoder architecture. The following works have attempted to eliminate the issue of slow convergence by designing architecture [9, 50] , query [28, 54, 81] , and bipartite matching [3, 21, 22, 68, 69] . The original DETR framework, along with its various adaptations, has not only brought forth a simple yet powerful end-to-end architecture for common object detection but has also been extended to other related tasks, including multipleobject tracking [66] , action detection [29] , human-object interaction [16, 17] , person search [1] , and instance segmentation [14, 53] .We propose the adaptation and modification for VER: First, since we suggest a single-stage framework, we adopt deformable DETR for simultaneous subject localization and emotion classification; Second, as generic objects exhibit distinct and localized characteristics, but contexts are essential and nuanced-related for VER, we introduce a decoupled subject-context transformer to capture contextual interaction. 
method 3.1 single-stage framework: 
 Current two-stage approaches for in-the-wild emotion recognition may suffer low efficiency from disjoint training stages and limited interaction between fine-grained subject-context elements.To address the limitation, we introduce a single-stage framework, employing a Decoupled Subject-Context Transformers with early fusion, for subject localization and emotion classification.Architecture. As shown in Fig. 3 , the system handles an entire image through a CNN backbone, an encoder with deformable transformers, and a decoder with novel Decoupled Subject-Context Transformers (DSCT). Given an image, we extract multi-scale features through the backbone, flatten them in spatial dimensions, and supplement position encoding and level embeddings. The encoder subsequently encodes the global and multi-scale features through six deformable transformers. After that, the decoder correlates the given learnable queries with encoded features with six DSCTs. Finally, the Feed-Forward Networks (FFNs) transform a set of 𝑁 subject queries into 𝑁 final predictions, including emotion classes and bounding boxes. We defer to the supplementary material the detailed definition of the architecture, which follows deformable DETR [81] . We jointly train classification and localization to enrich subject-centric feature learning. Furthermore, DSCTs facilitate fine-grained subject-context interactions by early fusion.Queries. Each learnable query is a concatenation of 256-dimension spatial and 256-dimension semantic embeddings. The spatial embedding is decoded into the 2-d normalized coordinate of the reference point and the semantic one into 1) the bounding box as relative offsets w.r.t. the reference point and 2) the corresponding emotion class of the subject. The semantic embeddings of queries adaptively integrate multi-scale image features by sampling locations around the reference points. We refer the reader to the supplementary material for detailed definitions, which follow deformable DETR [81] . We adopt a set of 𝑁 queries for prediction, where 𝑁 is typically larger than the average subject number per image in a dataset.Optimization. We use set-level prediction [2] that encapsulates several predictions or ground truth within a set. For clarity, we denote the 𝑖-th element of a set as (cls 𝑖 , box 𝑖 ), where cls 𝑖 represents the categorical emotion label and box 𝑖 ∈ [0, 1] 4 specifies the normalized center coordinate and box's height and width.During training, since the prediction number is larger than the actual number of subjects in an image, we first pad the set of ground truths with ∅ to ensure a consistent size. We employ the bipartite matching [2] that computes one-to-one associations between the set of predictions ŷ and the padded ground truths 𝑦:EQUATIONwhere σ represents the optimal assignment, 𝜎 ∈ 𝔖 𝑁 denotes a permutation of 𝑁 elements, L match 𝑦 𝑖 , ŷ𝜎 (𝑖 ) indicates a pair-wise matching cost between ground truth and a prediction with index 𝜎 (𝑖). L match encompasses a classification loss L cls and a box regression loss L box , expressed as:EQUATIONwhere 𝜃 cls , 𝜃 box ∈ R are hyperparameters. We efficiently compute the matching results using the Hungarian algorithm [2] .Given the optimal assignment σ, the training loss L is:EQUATIONwhere 𝜆 cls , 𝜆 box ∈ R are hyperparameters. For matching and training, we employ the focal loss [26] for L cls and set L box as the 𝑙 1 loss and generalized IoU loss [45] . During inference, we set the mean of the class output logit as the score of each prediction. For multi-label tasks, subject emotions are determined with a threshold 𝑡:EQUATIONwhere 𝑜 represents the index list of the emotion class. In the case of multi-class tasks, subject emotions are determined as:EQUATIONwhere 𝑜 corresponds to the index of the emotion class. Discussion. Current approaches of emotion recognition usually include two steps of detection and classification, which suffer from low efficiency from disjoint training stages. There, we pursue a single-stage framework to simultaneously recognize the subject's bounding box and emotion class. The deformable DETR pipeline, including the above-mentioned one-stage processing and joint classification and localization loss, aligns well with our demand.To facilitate interactions between fine-grained subject and contextual elements, we introduce a novel Decoupled Subject-Context Transformer (DSCT), which treats queries in a "decouple-then-fuse" manner and exploits spatial-semantic relational aggregation.Decouple then Fuse. Before the DSCT, the queries are decomposed into subject and context queries. As shown in Fig. 3 , we adopt 𝑁 subject queries and a large number of context queries as input queries of the decoder, where all queries have the same tensor size. In DSCT, both types of queries are correlated with multi-scale image features through the base deformable transformer, and then subject queries integrate context queries by spatial-sentimental relational aggregation before output. As illustrated in the left section of Fig. 4 , the reference point of the subject query primarily attends to the subject area to capture the subject's emotional signal, e.g., facial expression, while the reference points of the context queries are distributed across the entire image to pick up extensive and subtle contextual cues, e.g., body posture and gesture, surrounding agents, and scene attributes like grass and sky.Spatial-Semantic Relational Aggregation. As shown in the right part of Fig. 4 , the DSCT fuses context queries based on their spatial-semantic relationships w.r.t. the subject query.The DSCT first picks up the short-range contextual cues, such as objects in hands and close agents, based on the relative spatial distance between the subject and context queries. For each context query, the distance is calculated as the Euclidean distance between reference points of the context and subject queries. For the subject and context queries, we denote their coordinate vectors of the reference points as 𝑝 𝑛 𝑆 , where 𝑛 = 1, ..., 𝑁 and 𝑁 is the total number of the subject queries, and 𝑝 𝑚 𝐶 , where 𝑚 = 1, ..., 𝑀 and 𝑀 is the total number of the context queries. The relative spatial distance for a pair of subject and context query 𝑑 𝑛 𝑚 is computed as:EQUATIONThen we select 𝐾 𝑠𝑝 queries of the shortest spatial distance from total 𝑀 context queries for each subject query.As complementary, the long-range contextual signals, like scene attributes and distant people, are chosen via the semantic relevance of the subject and context queries. For each context query, the relevance is calculated as the similarity between semantic embeddings of context and subject queries. For the subject and context queries, we denote their semantic embeddings as 𝐸 𝑛 𝑆 , where 𝑛 = 1, ..., 𝑁 and 𝑁 is the total number of the subject queries, and 𝐸 𝑚 𝐶 , where 𝑚 = 1, ..., 𝑀 and 𝑀 is the total number of the context queries. The semantic relevance for a pair of subject and context query 𝑟 𝑛 𝑚 is calculated as:EQUATIONSince the semantic embeddings are processed with image features in the same architecture, we can measure their similarity without the transforming matrices in previous methods [24, 61] . Then we select 𝐾 𝑠𝑚 queries of the smallest semantic relevance from total 𝑀 context queries for each subject query.Finally, we adopt relevance re-weighting fusion to integrate the context queries into the subject query. We denote their semantic embeddings as 𝐸 𝑆 and 𝐸 𝑘 𝐶 , where 𝑘 = 1, ..., 𝐾 𝑠𝑚 + 𝐾 𝑠𝑝 . The attention weight 𝑤 𝑘 is computed by the dot product of 𝐸 𝑆 and 𝐸 𝑘 𝐶 . Then the softmax function makes the sum of attention weights to be 1. After that, the fused contextual subject query Ê𝑆 is defined as:EQUATIONDiscussion. In the object detection task, queries can capture effective information from distinctive and localized areas. While in  
experiments 4.1 implementations: 
 We set the number of queries 𝑁 to 4 and 9 for CAER-S [20] and EMOTIC [18] . We set 𝑁 to 4 and 9. To facilitate the training, we initialize the weights of architecture and borrow 300 context queries from Deformable DETR [81] , which was pre-trained on COCO [27] .Our batch size is 32, and we set hyperparameters 𝜃 box , 𝜆 box , 𝜃 cls , and 𝜆 cls to 5, 5, 2, and 5, respectively. For evaluation, we first use nonmax-suppression to remove the duplicate subjects and then select the subject that exhibits the highest bounding box overlap with the ground truth. The experiments were conducted using 8 GPUs of the NVIDIA Tesla A6000. We show details about architectural configurations, training strategies, and preprocessing steps, which follow those outlined in [81] , in the supplementary material.We conducted extensive experiments on two typical and popular context-aware emotion recognition datasets in real-world scenarios, namely the CAER-S [20] and EMOTIC [18] . The CAER-S dataset consists of 70,000 images, randomly divided into training (70%), validation (10%), and testing (20%) sets. Annotations include face bounding boxes and multi-class emotion labels. The dataset encompasses seven emotion categories: Surprise, Fear, Disgust, Happiness, Sadness, Anger, and Neutral. Performance on this dataset is measured using overall accuracy (acc) [20] .The EMOTIC dataset [18] contains a total number of 23,571 images and 34,320 annotated agents, which are randomly split into training (70%), validation (10%), and testing (20%) sets. Annotations include body and head bounding boxes, as well as multi-label emotion categories. EMOTIC encompasses 26 emotion categories: Affection, Anger, Annoyance, Anticipation, Aversion, Confidence, Disapproval, Disconnection, Disquietment, Doubt/Confusion, Embarrassment, Engagement, Esteem, Excitement, Fatigue, Fear, Happiness, Pain, Peace, Pleasure, Sadness, Sensitivity, Suffering, Surprise, Sympathy, Yearning. Performance on EMOTIC is evaluated based on the mean Average Precision (mAP) for all classes [18] .The performance of various methods on CAER-S and EMOTIC datasets is presented in Table 1 and Table 2 . To facilitate a fair comparison, we categorize the methods into two groups based on the number of parameters: similar-parameter measures and larger-parameter ones, using a threshold of 100 Million (M) parameters. For the methods without released code, we count their parameters through their backbone configuration in paper and present the details in the rightmost column. Subscripts in Emoti-Con [34] and HECO [61] correspond to specific context modalities as mentioned in their respective papers. The performance of the methods is sourced from their original papers or re-implemented results of other papers. Our proposal framework outperforms similarparameter methods by a notable margin, achieving a significant 3.39% improvement on CAER-S and an impressive EMOTIC. Notably, the proposal even surpasses larger-parameter approaches, underscoring its suitability for emotion recognition when compared to two-stage methods.We present the qualitative results in Fig. 5 and Fig. 6 , depicting the bounding boxes and emotion classes output by our proposal framework, alongside those produced by the EMO-Net [18] , a representative two-stage late-fusion method. To enhance the clarity of the proposal's output, we include visual indicators for subject queries' reference points (colored in red) and sampling locations. Outputs of different subjects are color-coded for differentiation. These visualizations illustrate that the proposal consistently yields high-quality results, showcasing superior classification accuracy when compared to EMO-Net. In EMOTIC, the EMO-Net might neglect subtle emotions like "Pain", or produce wrong even opposite emotions like "Disapproval". In CAER-S, when the facial expression is not distinguishing, the EMO-Net is confused with "Anger" and "Neutral", "Happy" and "Surprise", or "Surprise" and "Fear". Classification and localization. We conducted experiments to fine-tune 𝜃 cls and 𝜆 cls on the EMOTIC dataset [18] and keep 𝜃 box = 5 and 𝜆 box = 5. The results of these hyperparameter experiments are thoughtfully presented in Table 3 . Notably, the most compelling performance is achieved when 𝜃 cls : 𝜃 box is set to 2:5, and 𝜆 cls :𝜆 box is set to 5:5. We can see adding appropriate localization loss can boost classification performance and facilitate subject-centric feature learning. Besides, we noticed that 𝜃 cls has minimal impact on performance, whereas 𝜆 cls significantly influences the results.Feature map activation. We visualize feature map activation of methods of different paradigms. We select EMO-Net [18] representative two-stage method. For fairness, we re-implement EMO-Net using ResNet50 as the backbone to achieve a similar model complexity to the proposal (referred to as EMO-Net-R50). The selected feature maps originate from the final layer of the ResNet50 backbone. In Fig. 7 , a clear distinction emerges: EMO-Net-R50 exhibits a tendency to emphasize a few large regions, while the proposal consistently places importance on smaller, more intricate areas. This observation suggests that the proposal excels in the precise handling of fine-grained subject-context cues compared to conventional two-stage methods of coarse-grained cues.OursFeature sampling positions of queries We visualize the reference point and feature sampling positions of the normal subject queries and contextual subject queries of the DSCT in Fig. 8 . The reference points are drawn as big blue stars and the sampling positions as small grey circles. For the normal subject query, the sampling positions only rely on its own sampling points [81] . For the contextual subject query, the sampling positions rely on both the sampling points of itself and context queries. We can see the sampling positions of normal subject queries only cover the subject area while the contextual ones are densely distributed across the image. The quantitative result shows the contextual query of DSCT outperforms the normal one with a 0.71% precision improvement. The gain can be attributed to aggregating extensive contextual cues, which are essential for in-the-wild emotion recognition.Multiple subjects. We conduct an evaluation on images with varying subject numbers. We select EMO-Net-R50 and EMO-Net-R50-M, which further masks subjects in the context [20] , for comparison. Table 4 presents the performance on EMOTIC for images with different subject numbers. As the subject number in an image increases, the complexity of subject-context interaction also rises. Notably, the proposal maintains stable performance with increasing subject numbers, while EMO-Net-R50's performance deteriorates. This observation verifies our early fusion proposal can handle complex interactions better than late fusion two-stage methods. Subject Query Number. We conduct experiments to investigate the impact of query number setting. Table 5 presents the performance of different query numbers. Fig. 9 offers a visualization of bounding boxes (colorful rectangles), reference points (red circles), and sampling locations (colorful circles) corresponding to different subject query numbers. Table 5 shows the proposal achieves the best result on EMOTIC and CAER-S when the query number is 4 and 9 respectively. Notably, we observe a consistent performance stability trend as the query number increases, ranging from 1 to 6 on EMOTIC and from 1 to 10 on CAER-S. Fig. 9 also indicates that different subject queries attend to separate subject areas.Components of DSCT. We assess the impact of components of DSCT on the EMOTIC dataset [18] , and the results are displayed in Table 6 . The categories include "baseline" (only subject queries), "Decouple-fuse" (decouple queries and then fuse), "Spatial" (select 𝐾 𝑠𝑝 context queries with shortest spatial distance), and "Semantic" (select 𝐾 𝑠𝑒 context queries with semantic relevance). As we can see, the DSCT enhances performance by 0.71%, highlighting the importance of sufficient contextual interaction and fusion. Specifically, decoupling and fusing context queries improve performance by 0.18%, and selecting context queries based on spatial and semantic relation boosts the result by an extra 0.56% and 0.12%. The results demonstrate the effectiveness of each proposed component.Selection of Spatial and Semantic Relation. We conduct experiments with varying values of 𝐾 𝑠𝑝 and 𝐾 𝑠𝑒 on EMOTIC [18] to evaluate the sensitivity of Spatial and Semantic Relation parameters. As shown in Figure 11 , the optimal performance is achieved when 𝐾 𝑠𝑝 is set to 100. The best performance is 0.41% higher than the one when 𝐾 𝑠𝑝 is 300. This observation suggests that not all contextual information is valuable for effective emotion recognition. The selection of the 100 closest context queries w.r.t. the subject query also aligns with gradual interaction decay in [61] . Figure 10 shows spatial relational selection keeps short-range contextual cues such as objects in hands and close agents. Besides, the optimal performance is achieved when 𝐾 𝑠𝑒 is set to 50, which is 0.39% higher than the one when 𝐾 𝑠𝑒 is 300. This suggests that some contextual information is a disturbance. Figure 10 shows semantic relational selection preserves long-range contextual signals, like scene attributes and distant people. We can see two selections are complementary.Re-weighting Strategy. We conduct experiments of re-weighting fusion strategies for context queries on the EMOTIC dataset. The results, shown in Table 7 , indicate the performance under different re-weighting strategies: Semantic (semantic relevance weights based on semantic relation w.r.t the subject query), Equal (equal weights), Spatial (spatial distance weights based on spatial relation w.r.t the subject query), and Attentive (attentive weights learned from the subject query through a linear layer). The findings reveal that the best performance is achieved when employing semantic reweighting, which performs better than equal re-weighting by 0.53%. The observation aligns well with previous studies that contexts have variant contributions to emotion recognition [24, 61] Feature Extractor. To evaluate the influence of feature extractors, we conducted experiments with various backbone architectures for the proposal on both EMOTIC and CAER-S datasets. The results, as depicted in Table 9 , include performance metrics and corresponding parameter counts. Specifically, "R" refers to ResNet [12] , and "WR" designates Wide ResNet [64] . For ResNet50, we utilized a pre-trained backbone from deformable DETR as the initialization. The optimal performance on EMOTIC is attained when using ResNet-101 as the backbone, while on CAER-S, ResNet-50 yields the best results. Interestingly, it's evident that the relationship between performance and parameter counts is not linear on both datasets. Moreover, the performance on CAER-S seems to be significantly influenced by the choice of pre-trained initialization.DETR-like Architecture. The family of DETR-like architectures has gained significant momentum in the object detection task. To assess the impact of incorporating different DETR-like architectures into proposal, we conducted experiments on the CAER-S dataset by leveraging the detrex platform [44] . For equitable comparisons, all architectures utilize ResNet50 as the backbone. The results, summarized in with various DETR-like architectures. Notably, there is a performance gap of around 1% between DETR-based and deformable DETR-based architectures. However, performances remain comparable within DETR-based and deformable DETR-based architectures even though they adopt different techniques.Acc % Architecture Acc % DETR [2] 89.74 DINO [69] 89.27 Anchor-DETR [54] 90.42 Deformable DETR [81] 91.81 DAB-DETR [9] 87.49 DAB-D-DETR [9] 91.39 DN-DETR [21] 89.79 H-D-DETR [15] 91.65 Conditional-DETR [31] 90.07 DETA [40] 91.77 Table 10 : Ablation study on different DETR architectures. 
conclusion: 
 This paper introduces a single-stage visual emotion recognition framework with Decoupled Subject-Context Transformers (DSCT). The proposal predicts subjects' emotions and locations simultaneously and processes subject and context emotional cues by early fusion. We evaluate our single-stage framework on two widely used context-aware emotion recognition datasets, CAER-S and EMOTIC. Our approach surpasses two-stage alternatives with fewer parameter numbers, achieving a 3.39% accuracy improvement and a 6.46% average precision gain on CAER-S and EMOTIC datasets, respectively. We observe that the joint training of localization and classification can facilitate subject-centric feature learning. Besides, we find that early fusion improves handling the fine-grained subjectcontext interaction, e.g. multiple subjects in one scene. We also explore the spatial and semantic relationships between subject and contextual cues for more effective interaction and fusion.  
: 
 In this paper, we focus on the problem of inferring the emotion of one person in a real-world image. Concretely, given an inthe-wild image, we aim to identify the subject's apparent discrete emotion categories (e.g. happy, sad, fearful, or neutral). Existing methods typically involve two stages: subject detection and emotion classification. Conventional approaches primarily emphasize facial cues [6, 47, 56-58, 71, 78] , featuring a two-stage without fusion paradigm. As depicted in Fig. 1 (a), a standard off-the-shelf detector indicates a facial region, and a dedicated face encoder extracts facial features for subsequent classification into distinct emotional categories. Recent advances have increasingly recognized the importance of contextual cues in emotion recognition, like body language, scene semantics, and social interactions [18, 20, 24, 33, 34, 41, 60, 61] . This system is characterized as a two-stage with late fusion paradigm. As illustrated in Fig. 1 (b), it first identifies subjects and contexts within the image, processes them through independent encoders, and fuses the resulting features for emotion prediction.While effective, existing approaches are hindered by two primary limitations. Firstly, the disjointed learning processes of emotion classifiers and subject detectors in a two-stage paradigm often result in inefficient computational efficiency. Illustrated in Fig. 2 , existing HECO (1) Emoticon (1, 2) HECO (1, 2) HECO (1, 2, 3) Emoticon ( methods' effectiveness is limited with many parameters. Secondly, existing paradigms may exhibit a restricted capacity for subjectcontext fusion, thereby falling short in addressing real-world images that are susceptible to nuanced contextual influences [32, 55] . Shown in Fig. 1 , the first paradigm focuses solely on facial expressions, neglecting essential contextual cues, and the second paradigm's late fusion scheme misses fine-grained subject-context interaction, leading to sub-optimal emotion recognition.To alleviate the limitation, we introduce a single-stage framework, employing a Decoupled Subject-Context Transformer (DSCT) with early fusion, for simultaneous subject localization and emotion classification, characterized as a single-stage with early fusion paradigm. As illustrated in Fig. 1 (c), we adopt an encoder-decoder architecture with DSCT, where learnable queries are correlated with the global and multi-scale features for prediction. Rather than disjoint training stages, we jointly leverage box and emotion signals as supervision to enrich subject-centric feature learning, i.e., the framework is trained with a joint loss of classification and localization. Fig. 2 demonstrates that our method is effective and efficient, surpassing two-stage prior arts with fewer parameters.Furthermore, we introduce DSCT to facilitate interactions between fine-grained subjects and context in a decouple-then-fuse manner. As depicted in Fig. 1(c ), the queries are decomposed into subject and context queries to capture the subject's emotional signal, e.g., facial expression, and a wide range of contextual cues, e.g., body posture and gesture, agents, objects, and scene attributes. The decoupled query tokens-subject queries and context queries-gradually intertwine across layers within DSCT. For effective fusion, the spatial and semantic relations between context and subject information are exploited and aggregated. The spatial relation picks up contextual queries with short-range subject-context interaction, such as the subject between objects in hands and close agents. As complementary, the semantic relation chooses contextual queries with long-range subject-context interaction, like the subject between scene attributes and distant people. Fig. 1(c ) shows that the singlestage framework notices useful and subtle emotional cues between the subject and context, e.g. the kid is looking at the father's clothes.Extensive experiments are conducted on two standard contextaware emotion recognition benchmarks to validate the efficacy of our approach. The proposed framework attains impressive results, achieving 91.81% accuracy on the CAER-S dataset [20] and 37.81% mean average precision on EMOTIC [18] . In the case of similar parameter numbers, the proposal surpasses counterparts by a substantial margin of 3.39% accuracy and 6.46% average precision on CAER-S and EMOTIC respectively. Furthermore, we provide valuable insights by visualizing network output, feature map activation, and query selection, underscoring the proposal can discern useful and nuanced emotional cues of subject and context.The main contributions can be summarized as follows:• We present a novel single-stage framework for simultaneous subject localization and emotion classification to address the limitations of disjoint training stages. • To facilitate fine-grained interactions between subjects and context, we introduce a new decoupled subject-context transformer to decouple and fuse queries across layers. • The spatial and semantic relations are exploited and aggregated to capture the short-range and long-range subjectcontext interaction complementarily. • Extensive experiments and visualization on two standard datasets show that the single-stage framework outperforms two-stage alternatives by a significant margin and excels in capturing useful and nuanced emotional cues. 
 
 Appendix: 2024-04-13 12:43. Page 2 of 1-10.2024-04-13 12:43. Page 3 of 1-10.2024-04-13 12:43. Page 4 of 1-10.2024-04-13 12:43. Page 6 of 1-10.2024-04-13 12:43. Page 8 of 1-10.2024-04-13 12:43. Page 10 of 1-10. 


Decision: This manuscript received overall positive ratings. The reviewers kept the positive evaluation of the paper after the rebuttal.

The only negative point of the paper is that it is not multi-modal, since a single modality is used.

Reviews: review: paper_summary: paper_summary: This paper distinguishes itself from traditional two-stage methods by proposing a novel one-stage approach for emotion recognition in context. The authors introduce a single-stage framework that effectively reduces the number of model parameters while achieving superior performance.
strength_weakness: strength_weakness: This innovative approach integrates the tasks of subject localization and emotion classification into a unified process, addressing the inefficiencies of disjointed training stages found in previous two-stage models. The reduction in parameters not only enhances computational efficiency but also demonstrates significant improvements in accuracy and precision on benchmark datasets, underscoring the potential of the proposed method for real-world applications.
limitations: limitations: Although the motivation of the paper is strong, I still do not fully understand how the context query and subject query ensure the theoretical effectiveness claimed by the authors. For instance, if there are multiple subjects within a single image exhibiting conflicting emotional cues, it is unclear how the model can accurately resolve these conflicts and make correct judgments. The manuscript lacks detailed explanations and empirical evidence demonstrating how these queries interact and contribute to the overall performance improvements. Further clarification and more comprehensive experimental results are needed to validate the theoretical claims and to provide a clearer understanding of the mechanism behind the proposed approach, particularly in scenarios with conflicting emotional signals.
paper_summary: paper_summary: The paper proposes a single-stage emotion recognition transformer. In conjunction with emotion recognition, the proposed framework predicts the subject’s localisation (bounding box) with no additional stages. This is done through the simultaneous processing of the subject’s localisation and emotion recognition by employing a decoupled subject-context transformer. The method is evaluated on the CAER-S and EMOTIC datasets.
strength_weakness: strength_weakness: The paper is well written and clear. The method is novel for the emotion recognition task, with the minimal preprocessing requirement. The experiments are done on two challenging in-the-wild datasets. Considering that there are <40M training parameters, the results look impressive.
limitations: limitations: 1- One of the primary concerns is with consideration of spatial-semantic relationships with respect to the subject query. This is because the paper makes the naive assumption that the relative distance between subject and context is directly proportional to subject-context relevance. This is not always true as the images considered are the 2D ‘projection’ of the 3D perceived world. As a result of the assumption, the short-range contextual cues are naively picked as the few shortest spatial Euclidean distance from the image coordinates. However, the long-range contextual signals involve semantic relevance. 
2- The other concern/query is if the context queries are ‘borrowed’ from Deformable DETR[81] before the training, is the proposed method truly a single-stage then?
3- L154: How are learnable queries correlated with the global and multi-scale features. Is the conclusion derived from the qualitative results (for example, Figure 6, Figure 8, etc.)?
4- L399: How are reference of the context queries distributed? Uniformly across the image?  Is this what ‘Sampling Locations’ in the Supplementary results about?
5- How is $K_{sp} $ and $K_{sm}$ chosen?
6- Also, request the authors to see other existing methods (for example, [1]) which are related to the current work of jointly predicting emotion classes and face localisation.
[1] "Estimation of continuous valence and arousal levels from faces in naturalistic conditions", Antoine Toisoul, Jean Kossaifi, Adrian Bulat, Georgios Tzimiropoulos and Maja Pantic, published in Nature Machine Intelligence, January 2021
paper_summary: paper_summary: The authors propose a single-stage framework using a Decoupled Subject-Context Transformer (DSCT) that integrates subject localization and emotion classification simultaneously. This approach enriches subject-centric feature learning by using both box and emotion signals as supervision and enables detailed interaction between subject and context cues through a "decouple-then-fuse" mechanism. Extensive experiments on the CAER-S and EMOTIC datasets demonstrate the proposed method's effectiveness, achieving higher accuracy and precision compared to existing two-stage methods while using fewer parameters.
strength_weakness: strength_weakness: 1. The method seems reasonable and the results are good.

2. The idea of 'decouple-then-fuse' is interesting.
limitations: limitations: 1.As the author claimed that one of the limitations of the two-stage methods is that they ‘often result in inefficient computational efficiency’, which is caused by the disjointed learning processes of emotion classifiers and subject detectors. Is there any experimental result that can support that the proposed one-stage method is more efficient than the two-stage methods? I did not find corresponding results. Or giving some theoretical analysis is also good. 

2.There are some hyper parameters in the loss function. It would be better to introduce how the choose the value of them. 

3.I am wondering what the ‘decouple’ operation is to disentangle the context-subject content. All I can see is to use different queries to get them from a Transformer. Is this right?
paper_summary: paper_summary: The authors propose a DSCT to facilitate interactions between fine-grained subject and contextual elements. They think this two semantic has disjoint training and insufficient interaction. They jointly leverage box and emotion signals as supervision to enrich subject-centric feature learning.
strength_weakness: strength_weakness: 1. The proposed method has relatively few parameters, yet good performance.This reduces the complexity and inefficiency associated with the traditional two-stage pipeline.
2. DSCT enhances the interaction between subject and context cues by decoupling them into separate queries and then fusing them through multiple layers. This allows the model to capture fine-grained details from both the subject and the surrounding context.
limitations: limitations: 1. The author uses two different semantic queries for their tasks. How to decompose the queries into subject and context.
2. The paper primarily compares the proposed method with two groups based on the number of parameters: similar-parameter measures and larger-parameter ones, using a threshold of 100 Million (M) parameters. However, there is no related single-stage extensively compared or no intuitive contrast with other recent single-stage or end-to-end emotion recognition models to verify the effectiveness of the proposed single-stage query approach. Please add a comparison to 2023 related papers.
3. This paper proposes contextual semantics and queries within the image but does not have an intuitive consideration of how it handles long-range contextual signals, such as emotion variations or situational factors that might influence emotions.