Title: Learning Human-Object Interaction as Groups
Venue: NeurIPS 2025
Accepted: yes
Keywords: human object interaction

graph models

visual relationship understanding
Abstract: Human-Object Interaction Detection (HOI-DET) aims to localize human-object pairs and identify their interactive relationships. To aggregate contextual cues, existing methods typically propagate information across all detected entities via self‑attention mechanisms, or establish message passing between humans and objects with  bipartite graphs. However, they primarily focus on pairwise relationships, overlooking that interactions in real-world scenarios often emerge from collective behaviors ($\textit{i}.\textit{e}.$, multiple humans and objects engaging in joint activities). In light of this, we revisit relation modeling from a $\textit{group}$ view and propose GroupHOI, a framework that propagates contextual information in terms of $\textit{geometric proximity}$ and $\textit{semantic similarity}$. To exploit the geometric proximity, humans and objects are grouped into distinct clusters using a learnable proximity estimator based on spatial features derived from bounding boxes. In each group, a soft correspondence is computed via self-attention to aggregate and dispatch contextual cues. To incorporate the semantic similarity, we enhance the vanilla transformer-based interaction decoder with local contextual  cues from HO-pair features. Extensive experiments on HICO-DET and V-COCO benchmarks demonstrate the superiority of GroupHOI over the state-of-the-art methods. It also exhibits leading performance on the more challenging Nonverbal Interaction Detection (NVI-DET) task, which involves varied forms of higher-order interactions within groups.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GroupHOI for Human-Object Interaction (HOI) detection that models interactions not just as pairs but as groups, inspired by social grouping principles. The method forms geometric groups based on spatial proximity and semantic groups based on feature similarity, leveraging both for improved contextual reasoning. GroupHOI integrates these principles into a DETR-based architecture, enhancing interaction prediction by propagating information within learned groups. Experiments on the HICO-DET and V-COCO benchmarks show GroupHOI achieves state-of-the-art results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The joint geometric and semantic perspective to model complex relationships between entities and propagate contextual information across them is both innovative and interesting.\n2. The paper defines two visual principles for constructing geometric and semantic groups, and effectively integrates these principles into HOI detection.\n3. Overall writing of this paper is good and easy to understand and follow.\n\nWeaknesses:\n1. The authors did not provide evidence to demonstrate that the proposed *geometric and semantic perspective groups* can address the current challenges faced by existing HOI methods. It remains unclear whether these groups can cover all possible HOI scenarios, and whether the method is effective in handling cases that do not fit the situations described in the paper.\n2. The size of geometric and semantic groups are fixed. Since the content of each image varies, this fixed number of sizes conflicts with the assumptions made in the paper.\n3. Although the model achieves SOTA performance, the source of the improvement is unclear. As shown in Table 2(a), the baseline without GEO and SEM already outperforms CMMP and LOGICHOI. While it is undeniable that GroupHOI improves upon the baseline, Table 2(a) suggests that the performance gain of GroupHOI may stem more from other factors rather than the GEO and SEM modules.\n\nQUESTIONS:\nWhy does the GroupHOI set the size of the geometric and semantic groups to 4 and 2, while the example in Figure 4 uses a size of 3? Does this mean that group sizes of 4 and 2 may not be the most appropriate for some images?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for Human-Object Interaction detection that explicitly models interactive relationships within a group. To exploit these relationships, the authors propose constructing geometric and semantic groups based on geometric proximity and semantic similarity. A modified transformer architecture is also introduced to perform feature aggregation and distribution within each group. Experimental results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Considering the relationships within groups to promote HOI is an effective and intuitive idea.\n- The authors exploit the principle of proximity to form groups, which adheres to real-world human behavior and is conceptually sound.\n\nWeaknesses\n- The author proposes a learnable estimator to measure the proximity among bounding boxes. Does this module require supervision? Without supervision, how can the MLPs distinguish whether two entities belong to the same group? How much improvement can be achieved by using MLPs over conventional distance-based calculations for estimating group? Additionally, the MLPs only take distance and IoU as inputs, which may not be sufficient to determine whether two entities belong to the same group—for example, two people who are far apart in depth might still have overlapping bounding boxes. Therefore, should the input also incorporate image features?\n- The group sizes within the same image can vary significantly, but this paper does not account for the variation in group size. Simply setting the geometric and semantic group numbers to 4 and 2 does not reflect the actual variability in real-world scenarios.\n- There are previous works that have explored leveraging intra-group information to understand interactions. They also form groups based on proximity and similarity, and their structures are not strictly limited to fully-connected or bipartite graphs. Although these works do not specifically address the HOI problem, they share many similarities with this work in terms of human group construction, and thus, I believe they should be discussed as well.\n\n     [A] \"Reconstructing groups of people with hypergraph relational reasoning.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n     [B] \"Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n- In the Semantic Group, there may be significant differences between human and object features. When computing feature similarity, is it possible that only humans are grouped together while objects are excluded from the group?\n\nQUESTIONS:\n- How does the learnable estimator ensure the accuracy of group classification?\n- How does the semantic group account for category differences?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GroupHOI to enhance human-object interaction (HOI) understanding by modeling relationships from a group perspective rather than pairwise ones. GroupHOI is built upon two grouping principles:\n- Proximity principle: This principle suggests that individuals tend to form groups with those they are physically close to. In GroupHOI, a learnable proximity estimator is introduced to measure this. Contextual cues within these groups are aggregated and update the entity embeddings.\n- Semantic similarity principle: This principle suggests that interactions sharing similar visual or behavioral patterns tend to form semantic groups. GroupHOI implements this by forming clusters of interaction proposals (i.e., human-object pairs) based on cosine similarity of their feature embeddings. These groups are then used to provide semantic context to the interaction reasoning process, and local features within bounding boxes are also incorporated.\n\nThe framework adopts a DETR-based cascade architecture, in line with other recent state-of-the-art approaches.\n\nGroupHOI demonstrates performance improvements over existing methods on standard HOI benchmarks, including HICO-DET and V-COCO.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is clearly written and well-organized. The authors clearly articulate the motivation for introducing grouping in HOI tasks and provide clear explanations of their corresponding design choices. The overall presentation reflects strong experience in writing research papers.\n\n- The central idea of modeling relationships from a group perspective is technically sound. The proposed solution is coherent and conceptually aligned with the proposals.\n\n- The experimental results, including ablations, are comprehensive and provide convincing support for the proposed method. Overall, the paper slightly exceeds my acceptance threshold.\n\n**Weaknesses**\n\n- Overall, I think the selected datasets such as HICO-DET and V-COCO may not be ideal for showcasing the benefits of group-based relationship modeling. These datasets have been used extensively for years, and most recent works report only marginal improvements. It's unclear how much of the remaining error is attributable to the absence of group-level reasoning, as opposed to dataset-specific limitations such as annotation bias, limited scale, or inherent noise. Additionally, the majority of interactions in these datasets likely involve single human–single object pair, which may limit the opportunity for group modeling techniques to demonstrate clear advantages.\n\n- A more compelling direction could be for the authors to curate or design a new benchmark that emphasizes scenarios requiring group-level reasoning, for example, interactions involving multiple humans and objects with interdependent roles. Alternatively, the authors could consider filtering or reorganizing the existing datasets to introduce a new split of complex multi-entity interactions, going beyond the typical rare/non-rare splits.\n\nQUESTIONS:\n- From the examples shown in Figure 4, it appears that some nearest neighbors in the geometric grouping may actually refer to the same entity due to overlapping detection boxes (e.g., before non-maximum suppression is applied). In such cases, how often do self-matches occur when selecting the top K_g nearest entities? Do you feel necessary to address the self-matches?\n\n- Moreover, since object instances can vary significantly in category, size, and spatial layout, why use fixed neighbors? Would it be more flexible to allow the learnable proximity estimator to determine the appropriate number of neighbors dynamically, rather than using a fixed top-k scheme?\n\n- For the geometric group construction, how is the proximity score S_{ij} trained? What kind of supervision signal is used to guide the learning of this estimator? Is there a loss that explicitly supervises the group formation process, or is it trained in an end-to-end manner solely based on the final HOI loss in Eq 9?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for Human-Object Interaction detection that re-examines relation modeling from a group perspective.\n\nThe authors propose that current HOI methods neglect the inherent collective and non-pairwise social attributes within human-centric scenarios.\n\nTo address this, GroupHOI leverages two visual attraction principles: geometric proximity and semantic similarity. Geometric proximity is exploited by grouping humans and objects into distinct clusters using a learnable proximity estimator based on spatial features derived from bounding boxes. Semantic similarity is incorporated by enhancing a transformer-based interaction decoder with semantic-aware local cues derived from HO-pair features. \n\nThe paper presents extensive experiments on the HICO-DET and V-COCO benchmarks, claiming superior performance over state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. Presentation. The paper is generally well-written and easy to follow. The methodology is explained logically, and the figures, particularly Figure 1 illustrating relation modeling paradigms, are helpful in understanding the core ideas.\n\n\n2. Novel perspective on social attributes for HOI detection. The emphasis on learning social attributes of human interactions in groups is an interesting and valid perspective. It attempts to capture more complex, collective patterns in visual scenes, which aligns with how humans naturally categorize and understand groups.\n\n3. Structured Grouping Principles: The definition of two visual attraction principles – proximity and similarity – to construct groups is clear and easy to follow. It makes the contribution of the paper more understandable.\n\nCons:\n\n1. Originality for overlap with existing graph-based methods. The claim of novelty for the proposed solutions, specifically the geometric-aware local transformer and semantic-aware interaction decoder, needs further justification. As is shown in Section 3.2, these solutions bear significant resemblance to previous two-stage HOI detection methods that focus on graph modeling. The paper's distinction from existing graph-based methods like DRG (ECCV 2020) , SCG (ICCV 2021), SKGHOI (ICDMW 2023) utilizing spatial relationships and semantic similarities to build graphs for HOI classification, is not sufficiently highlighted or critically analyzed. The differences seem incremental rather than a significant departure from established graph-based approaches.\n\n\n2. Weak link of the solution to the motivation of \"social attributes in a group of people\". The connection between the proposed technical solutions (geometric-aware local transformer and semantic-aware interaction decoder) and the overarching idea of \"learning social attributes of human interactions in a group\" appears tenuous. While geometric proximity and semantic similarity can contribute to group formation, the paper doesn't deeply elaborate on how these mechanisms specifically capture or represent \"social attributes\" beyond basic visual grouping. The discussion remains at a low-level feature aggregation rather than a high-level social understanding.\n\n3. Claim of SOTA performance. The claim of sota) performance may not be entirely accurate. The paper appears to miss some recent works published in the past three years that might achieve higher performance, especially on the V-COCO dataset. For instance, VIL (ACMMM 2023) and CQL (CVPR 2023) and some more recent works obtain stronger performance than GroupHOI on V-COCO, which could challenge the paper's main empirical claim. A more comprehensive and up-to-date comparison is necessary.\n\n4. Reproducibility for insufficient implementation details. While Section 3.5 provides some implementation details, they may not be granular enough to ensure full reproducibility of the work. \n\n5. The additional modules may brought significant computational overhead. A experimental analysis on the cost is very important.\n\n6. In the last paragraph of introduction, the authors seem to confuse the results for V-COCO and HICO-Det.\n\nQUESTIONS:\nClarification on novelty vs. existing graph-based methods. Please provide a more detailed and concrete discussion distinguishing GroupHOI's geometric and semantic grouping mechanisms from prior graph-based HOI detection methods (e.g., DRG, SCG, and other works that use spatial/semantic relationships for graph construction). What are the fundamental differences in how groups are formed and how information is propagated that makes GroupHOI a novel approach rather than an incremental improvement?\n\nElaborating \"social attributes\". Could you expand on how the proposed geometric and semantic grouping truly captures \"inherent social attributes\" beyond basic visual co-occurrence or similarity? Are there specific architectural components or loss functions that encourage the model to learn more abstract social cues, or is \"social attributes\" primarily a high-level interpretation of visual grouping?\n\nComprehensive SOTA comparison. Please provide an updated comparison table that includes very recent state-of-the-art methods, especially those published in the last three years, to firmly establish GroupHOI's competitive performance. If the current SOTA claim is indeed challenged by other works, please re-evaluate and clarify the contribution in light of these stronger methods.\n\nMore details. Could the authors provide more fine-grained implementation details? This could include specific architectural diagrams for the geometric-aware local transformer and semantic-aware interaction decoder, exact configurations for the learnable proximity estimator and pooling operator, and any non-standard training tricks or hyperparameter tuning procedures.\n\nExperimental analysis on the cost and burden of the proposed method.\n\nLIMITATIONS:\nThe paper lacks discussion over the additional cost as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}