Title: Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction Detection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Social interaction understanding

Nonverbal interaction detection
Abstract: Social interactions often emerge from subtle, fine-grained cues such as facial expressions, gaze, and gestures. However, existing methods for social interaction detection overlook such nuanced cues and primarily rely on holistic representations of individuals. Moreover, they directly detect social groups without explicitly modeling the underlying interactions between individuals. These drawbacks limit their ability to capture localized social signals and introduce ambiguity when group configurations should be inferred from social interactions grounded in nuanced cues. In this work, we propose a part-aware bottom-up group reasoning framework for fine-grained social interaction detection. The proposed method infers social groups and their interactions using body part cues and their interpersonal relations. Our model first detects individuals and enhances their features using part-aware cues, and then infers group configuration by associating individuals via similarity-based reasoning, which considers not only spatial relations but also subtle social cues that signal interactions, leading to more accurate group inference. Experiments on the NVI dataset demonstrate that our method outperforms prior methods, achieving the new state of the art, while additional results on the Café dataset further validate its generalizability to group activity understanding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies non-verbal interaction detection, which detects the interaction between a person and a group in the image. The proposed method is inspired by DETR, where a CNN backbone and a transformer encoder extract features, with two decoders for person detection and interaction recognition. To better recognize the fine-grained human parts for interaction recognition, this method also uses ViTPose to extract human keypoints, which are then used as L_{part} to regularize the attention maps. The proposed method achieves SOTA results on NVI-DET benchmark. Additional ablation analysis are conducted to validate the design of each proposed component.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The problem studied in this paper is important for some real world applications.\n* The proposed method achieved SOTA performance on benchmarks.\n* The ablation studies are generally comprehensive to validate design of each proposed component.\n* The paper is well-written and easy-to-follow. Every proposed component are explained clearly.\n\nWeaknesses:\n* The task studied in this paper, NVI-DET, is very similar to HOI-detection, except that latter detects <human, object, interaction> while this paper studies <human, group, interaction>. Although the authors discussed in this paper that the NVI-DET requires subtle human clues for interaction recognition, the same also applies to HOI-DET. To me, it is unclear how this task significantly differs from the existing HOI-DET task, which has been studied for several years in the previous literature.\n\n* The proposed achitecture is also very similar to a previous work on HOI-DET, i.e., GEN-VLKT [1]. Both models applies the idea of DETR, and adopt two decoders, where one decoder for instance detection while the other for interaction detection&recognition. Although the utilization of human pose seems to be a new contribution to the NVI-DET task, the same idea has been extensively utilized in HOI-DET tasks [2,3]. As such, I think the novelty of this paper is quite limited. The authors could further elaborate and discuss on the difference between this work and existing HOI literacture, like why the human information in HVI-DET is different from that in HOI-DET.\n\n* The proposed method can only handle closed-world settings, where interactions unseen during training and not recognizable during inference. However, this is already achievable in a previous work [1]. In the era of various pretrained and large models (e.g., CLIP, various MLLMs), it constitutes a deficiency of the model to be able to handle seen interactions only.\n\n\nReferences:\n\n[1] Liao et. al. \"GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection\". In CVPR 2022\n\n[2] Yu et. al. \"Exploring Pose-Aware Human-Object Interaction via Hybrid Learning\", In CVPR 2024\n\n[3] Wan et. al. \"Pose-aware Multi-level Feature Network for Human Object Interaction Detection\". In CVPR 2020\n\n\n Overall, I am looking forward to the rebuttals from the authors and would like to change my ratings after seeing the rebuttal.\n\nQUESTIONS:\nPlease refer to weakness section for my questions.\n\nLIMITATIONS:\nThe authors do discussed the limitations in the paper, which mainly focuses on the utilization of the external pose detection model. However, I think there are still other limitations of the proposed method and the studied task that are not discussed in the paper. For example\n\n* the ambiguility of the interactions, which may stem from the inherent difference between the image and text modalities.\n\n* the closed-world setting for interaction recognition, where interaction types unseen during training are not recognizable during inference. This can be somehow solved by combining a visual understanding model with modern LLMs, but can not be handled with the model proposed in this paper. \n\n* the privacy issues.  I image with sufficient human information should be given to the model for recognition, which requires the human information.\n\nI appreciate the efforts of the authors made in this work and understand that these points can not be handled with a single or several works. But discussing them in the paper could make it more coherent.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a part-aware bottom-up framework for fine-grained social interaction detection by modeling subtle cues (e.g., facial expressions, gaze) from body parts. Unlike existing holistic approaches, the proposed method enhances individual features with part-level details, then infers groups through similarity reasoning incorporating both spatial relations and social signals. Experiments on NVI dataset demonstrate state-of-the-art performance in detecting interaction-based groups\n\nSTRENGTHS AND WEAKNESSES:\nStrengths And Weaknesses:\n•\tStrengths: This paper addresses the problem of social interaction detection by focusing on fine-grained limb features. Its core strengths lie in a bottom-up interaction modeling approach, which progresses from individuals to pairwise relationships and then to groups, ensuring that group structures naturally emerge from interpersonal interactions (in contrast to traditional methods that directly detect group structures). Additionally, it employs part-aware feature enhancement by extracting local features from facial, arm, and leg regions based on pose estimation, accurately capturing fine-grained interaction semantics that are easily confused.\n•\tWeaknesses: \n(1)\tOne limitation lies in the clarity of Figure 1, particularly in the \"Bottom-up group reasoning\" section. The visual representation of the embedding process, especially the part involving pose-based interaction, is not clearly aligned with the textual description in the manuscript.  Improving the figure’s clarity and its correspondence with the written explanation would enhance the overall comprehensibility of the proposed approach.\n(2)\tThe transition introduced in line 107 (\"Unlike this approach...\") appears abrupt and lacks a clear logical bridge from the preceding discussion. While the paragraph describes the strengths of prior work such as [54], it does not provide a critical analysis of its limitations. A smoother transition that explicitly contrasts the proposed method with the weaknesses of existing approaches would strengthen the narrative flow and clarify the novelty of the current work.\n(3) In Figure 3(a), the illustrated example appears to involve both gaze and hug interactions simultaneously. It would be helpful if the authors could clarify how such cases are handled in the dataset annotation or during model inference. Specifically, how are multiple co-occurring interaction types defined, labeled, or distinguished—either in the data or by the algorithm?\n\nQUESTIONS:\n(1)\tThe potential issue is that the model does not clarify the expected range of interacting group members. If too many people are involved, pose estimation may suffer from occlusions, which could reduce the accuracy of the pose-guided mask.\n(2)\tIn some tasks related to group activity recognition (your Related Work), there are several classic datasets, such as the Volleyball [1] and Collective Activity Dataset [2]. These datasets focus more on group actions—can they be considered as representations of social interactions?\n[1] Mostafa S. Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori. 2016. A hierarchical deep temporal model for group activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1971–1980.\n[2] Wongun Choi, Khuram Shahid, and Silvio Savarese. 2009. What are they doing?: Collective activity classification using spatio-temporal relationship among people. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops. 1282–1289.\n\nLIMITATIONS:\n(1)\tIn the introduction, when social interaction is first mentioned, it is advisable to include a definition, as referenced in Marco Cristani et al. “Social interaction discovery by statistical analysis of F-formations. (BMVC 2011)”.\n(2) It would be beneficial if the authors could provide a comparison of the training and/or inference time between their proposed method and existing approaches. Highlighting the computational efficiency or runtime advantage would further demonstrate the practicality and scalability of the method in real-world applications.\n(3) In the supplementary material, the authors provide an analysis of the hyperparameters λₚ and λₐ. However, it would be helpful to also include a discussion of the other loss weights, such as λᵢ, λ_c, and λ_l. Including these would offer a more complete understanding of the model's sensitivity to different components of the overall loss function.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a bottom up reasoning framework for group behaviour reasoning. Particularly, the method starts from the body parts of the individuals in the images, and is composed of dedicated decoders for individuals and groups and an association module for group behaviour reasoning such as looking at each other or hand shaking. The proposed methodology is evaluated on one dataset.\n\nSTRENGTHS AND WEAKNESSES:\nThe reviewer appreciates the hard work of the authors. The strengths and weaknesses are summarised below. \n\nStrengths: \n- Group behaviour understanding is still a growing topic. The majority of work treats gaze communication behaviour and activity recognition separately. This paper offers a unified framework for reasoning. \n- The results demonstrate the potential of the approach. \n\nWeaknesses: \n- The technical novelty is limited. The proposed architecture is a very typical architecture, feature extraction, individual and group modelling, and association module. It is not clear the main contribution to the machine learning domain. Part-aware representation seem to be only contribution, which is incremental. \n- The proposed method focuses on images only, which has limited applications. \n- The proposed method is tested on one dataset only, which is substandard. The reviewer understands that the datasets for this problem might be limited. However, the authors could test this on gaze communication datasets such as GPStatic. Alternatively, there is a dataset for group activity recognition called JRDB dataset, https://jrdb.erc.monash.edu. \n- Following the previous comment, the experimental results are limited. For instance the authors could have implemented stronger baselines to compare their work, for instance, leaderboard methods on the JRDB dataset. \n- The paper could also benefit further analysis. For instance, one of the problems with the group activity datasets is the data imbalance problem. It would be good to see the results over different classes to see if the method is performing equally well and where are the limitations.\n\nQUESTIONS:\n- What is the contribution to the domain of machine learning? \n- What are the authors focusing on images, which is limited? Today LLMs are pretty good at reasoning about images, but they are failing on videos. How do the authors compare their method with LLM-based reasoning methods? \n- Can this method be applied to other datasets and downstream tasks?\n\nLIMITATIONS:\nLimitations are adequately discussed, except for the weaknesses outlined above. Additionally, visualisation results are helpful in both in the paper and the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}