Learning Human-Object Interaction as Groups
Authors: Jiajun Hong
Jianan Wei
Wenguan Wang
Venue: NeurIPS 2025 poster ()
Keywords: human object interaction
graph models
visual relationship understanding
Primary area: applications
TL;DR: we  propose a  framework  propagating contextual information with geometric groups and semantic groups.

Abstract: Human-Object Interaction Detection (HOI-DET) aims to localize human-object pairs and identify their interactive relationships. To aggregate contextual cues, existing methods typically propagate information across all detected entities via self‑attention mechanisms, or establish message passing between humans and objects with  bipartite graphs. However, they primarily focus on pairwise relationships, overlooking that interactions in real-world scenarios often emerge from collective behaviors ($\textit{i}.\textit{e}.$, multiple humans and objects engaging in joint activities). In light of this, we revisit relation modeling from a $\textit{group}$ view and propose GroupHOI, a framework that propagates contextual information in terms of $\textit{geometric proximity}$ and $\textit{semantic similarity}$. To exploit the geometric proximity, humans and objects are grouped into distinct clusters using a learnable proximity estimator based on spatial features derived from bounding boxes. In each group, a soft correspondence is computed via self-attention to aggregate and dispatch contextual cues. To incorporate the semantic similarity, we enhance the vanilla transformer-based interaction decoder with local contextual  cues from HO-pair features. Extensive experiments on HICO-DET and V-COCO benchmarks demonstrate the superiority of GroupHOI over the state-of-the-art methods. It also exhibits leading performance on the more challenging Nonverbal Interaction Detection (NVI-DET) task, which involves varied forms of higher-order interactions within groups.

Learning Human-Object Interaction as Groups
Jiajun Hong", Jianan Wei", Wenguan Wang
Zhejiang University
https:///gt wtm. haam/Jaajjuffin jtn.jonnn///ooooorp
Abstract
Human- Objject Interaction Detection (HOI-DET) aims to localize human-object pairs and identify their interactive relationships. To aggregate contextual cues, existing methods typically propagate information across all detected entities via self-attention mechanisms, establish message passing between humans and ob- jects with bipartite graphs. However, they primarily focus on pairwise relationships, overlooking that interactions in real- world scenarios often emerge from collective behaviors (i.e., multiple humans and objects engaging in joint activities). In light of this, revisit relation modeling from group view and propose GroupHOI, framework that propagates contextual information in terms of geometric proximity and semanfic similarity. To exploit the geometric proximity, humans and objects are grouped into distinct clusters using a Icarnable proximity estimator based on spatial features derived from bounding boxes. In each group, soft correspon- dence is computed via self-attention to aggregate and dispatch contextual cues. To incorporate the semantic similarity, we enhance the vanilla transformer- based interaction decoder with local contextual cues from HO-par features. Extensive ex- periments on HICO-DET and V-COCO benchmarks demonstrate the superiority of Gr over the state-of-the-art methods. It also exhibits leading performance on the more challenging Nonverbal Interaction Detection (NVI-DET) task, which involves varied forms of higher-order interactions within groups
Introduction
No an island, entire of itself.
John Donne, Meditation XVII Devotions
Cllman-Obbeed Interaction Detection (HOI-DET), critical pillar in visual relationship understand- ing. identifies entities (i.e., humans and objects) as basic building blocks, and leverages relationships as the connective glue that weaves them into meaningful patterns. Early works [1, 2] typically recog- nize HO-pairs, which are cropped from natural images by manually obtained bounding boxes, as composites (i.e., visual phrases) in isolation. Recent efforts [3, 4] extend HOI reasoning callworldd scenarios involving mul tiple entities with complex relational structures. Building upon object detection frameworks, HOI-DET methods evolve alongside advancements detector architectures from Faster R-CNN [5] to DETR-like variants [6, 7J. As a semantic interpretation task, HOI-DET can also benefit from large visual-linguistic models (e.g.. CLIP [8] and BLIP [9]) pre-trained extensive image-text corpora. Despite architectural innovations and multi-modal knowledge transfer, the core challenge remains unchanged: hhow structure and reason about relationships among entities?
Current methods [3. 10, 11] primarily reason over global context via the self-attention mechanism (Fig. (a))), while another strand of research constrains information exchange within homogeneous entities (í.e., object, object-human) [11] via bipartite graph (Fig. 1(b)). Despite achieving impressive performance,
object-object) [12, 13] or heterogeneous neighborhoods (i.e., human-
The first two authors contribute equally to this work. Corresponding Author: Wenguan Wang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Complete Graph
(b) Bipartite Graph
(c) Geometric Graph (d) Semantic Graph
Figure I: Relation modeling paradigms. While existing methods primarily model relations using complete (a)
bipartite (b) graphs, our approach introduces geometric (c) and semantic (d) graphs, enabling more structured and context-aware relational reasoning.
and denote humans, objects, and interactions, respectively.
these methods remain confined to the modeling of predefined pairwise relations, leaving the inherent collective pattemss [14] (i.e., multiple entities engaging in joint activities) between entities unexplored.
Tackling this notable void brings us back to the classical clustering view for structuring and organizing visual entities by finding groups of similar ones, ie., detecting HOIs as groups. Tomasello's s theory of shared intentionality [15] posits that with joint goals ("we-mode") and role coordination, participants tend to draw close to each other and perform collaborative actions. Considering a busy street with numerous pedestrians (Fig. 1), even without their explicit interpersonal relationships or social affiliations, the individuals can be naturally categorized into distinct groups based on their spatial proximity (Fig. (c)). Moreover, the walkers pulling suitcases typically exhibit similar visual characteristics- warkkin forward with swinging arms and suitcases trailing behind them highlighting their shared behavior (Fig. (d)). Hence question becomes more fundamental from a group view: @ how a group formed? and how does a group function?
Driven by question 0 we construct social groups guided by the Gestalt grouping principles of Proximity and Similarity [16]: First, the geometric proximity principle, i.e., the tendency for individuals to form groups with those physically close to, is implemented by constructing geometric groups based on the central distance and Intersection over Union (IoU) of bounding boxes. Second, the semantic similarity principle, i.e., the tendency for individuals to affiliate with those sharing visual characteristics or behavioral patterns, is operationalized through building semantic groups of interaction proposals based on similarity of their feature embeddings. Notably, groups in social environments demonstrate complex and diverse compositions with overlapping memberships, which complicates quantification and modeling. To address this, we treat each proposal and its neighbors as distinct group, establishing a one-to-one correspondence between groups and proposals.
To address question 2, we propose GroupHOI, a HOI detector that enhances interaction reasoning by discovering inherent group patterns within visual scenes. Building on DETR [17], we model two distinct group structures to capture contextual dependencies: First, based on object detection outputs, we construct geometric groups for each entity and perform structure-aware reasoning via soft correspondence through self-attention, enabling fine-grained aggregation and targeted dissemination of local cues ($3.3). Second, construct semantic groups to extract localized interaction priors, which are then integrated into the interaction decoder via a pooling operator to enable context-award reasoning through a structured local-global processing scheme ($3.4). Visualizations in $4.5 provide transparency into how contextual dependencies are organized and leveraged.
To conclude, our contributions are: i) defining two visual attraction principles to construct groups; ii) proposing a one-stage framework that integrates these principles into HOI-DET; iil) endowing the HOI detector with enhanced interpretability via grouping mechanisms.
Our method is evaluated on two standard HOI-DET benchmarks: V-COCO [18] and HICO-DET [2], achieving impressive performance with 36.70 mAP on HICO-DET and 65.0 mAp on V-COCO. It surpasses the state-of-the-ee method (i.e., Pose-Aware [19]) by solid margins, i.e., +0.84 and +3.9 mAP. Moreover, it achieves leading performance on the more challenging Nonverbal Interaction Detection (NVI-DET) [20] task, which permits diverse forms of higher-order group interactions. On the NVI benchmark [20], it reaches 73.19 AR on val and 75.21 AR on test.
2 Related Work
Human-Object Interaction Detection. Current HOT detectors can categorized into two-stage and one-stage paradigms. Two-stage methods [1, 2, 21-28] typically employ off-the-shed detectors (e.g..

---

Faster R-CNN [5]) to locate humans and objects, then generate human-objeet object pairs for interaction recognition. In contrast, one-stage methods [29-39], inspired by DETR [6], reformulate the task as a set prediction problem, and perform end-to-end triplets detection without explicit pairing. Recent studies [3, 4, 40-44] show strong gains by transferring knowledge from models pre-trained on large- scale image-text corpora (e.g.. CLIP [8]. BLIP [9], and Stable Diffusion [45]) or large language models (e.g.. OPT [46].. Instead of focusing solely on architectural innovations or large-scale knowledge transfer, our approach highlights complementary perspective on relation modeling, emphasizing how to organize and structure relations among entities.
Early object detectors [47-49] mainly built upon R-CN. family to localize and recognize each object in isolation, without contextual information exchange. To tackle this issue, [50] proposes a cascaded multi-stage framework with group recursive learning, while [51] incorporates global context across local regions. Recently, transformer- based methods such as DETR [6] reformulate detection as a set prediction problem, using self- self-attention to aggregate global context and enhance relational reasoning. HOI-DET methods have undergone similar evolution. Early two-stage HOI-DET methods [23-25] confine information sharing to pre-defined human-object pairs and restrict interaction reasoning to isolated triplets, while DETR-bol -base HOI- DET methods [30-32] leverage self-attention mechanisms to enable more comprehensive relational reasoning. Graph- based approaches [11, 13, 52-56] construct fully connected or bipartite graphs to facilitate message passing between entities. Building this idea, several recent works [57. 58] further explore hypergraph representations, offering a more expressive way for high-order relations. However, most of them prioritize how to reason about relations across entities while neglecting how to structure the relations among them, thereby introducing noise from spurious correlations.
Relation Modeling for Detection Tasks.
Group Analysis for Vision Tasks. Groups, as the fundamental units of human society, have become a pivotal analysis tool to understand complex human-centric scenes, with applications in crowd analysis, pedestrian trajectory prediction, and collective activity recognition. In crowd analysis, group- based methods [59] treat groups as atomic units of the scene, while undvidual-grrup methods [60] seek to leverage collective human information rather than processing them in isolation. Recently, PANDA [61] dataset enriched the task by integrating global context, high-resolution localized details, and temporal activity patterns, providing rich and hierarchical annotations. For pedestrian trajectory prediction, a number of existing approaches [57, 62-64] aim to improve multi person tracking by incorporating group relationships. For instance, GroupNet [57] addresses the multi-agent trajectory prediction problem by leveraging multiscale hypergraphs to model both pairwise and group-level interactions more effectively. Collective activity recognition principally aims to capture the nuances of multi-human interactions in wide field-of-view scene, prompting many explorations of intra-group dynamics. For instance, ARG [65] tries to model both appearance and spatial dependencies between humans, and SAM [66] builds a dense relation graph and prunes it to a sparse one.
multi-person
Inspired by these endeavors, we revisit HOI-DET from a group perspective, explore the visual principles underlying group formation, and propose a dedicated framework designed to operationalize these principles for HOI-DET, while introducing little extra computational overhead.
3 Methodology
3.1 Preliminary
Prevailing HOI-DET methods [3, 4, 19, 67, 68] employ a standardized DETR-based [69] pipeline for instance detection. Given an input image with position embeddings P.. a visual encoder preceded by a CNN backbone is applied to extract features Ve. Then, an instance decoder Dine is employed to transform human Qn and object Qo queries into the outputs by retrieving the encoded features Ve:
(1) where Qh Q' are then processed into bounding boxes B₁, B, and object class labels L.. In the interaction branch, human Q' and object Q' outputs are exhaustively enumerated [33, 68] or directly associated [3, 67, 69] to form HO-pairs and interaction queries inns for interaction classification.
14, Q'1 = (Ve. Qi. Qo),
3.2 Rethinking the Relation Modeling in HOI-DET
Revisiting Transformer-based Relation Modeling. Building upon DETR [69], most transformer- based HOI-DET models [3, 10, 11] perform global relational modeling among all the entity candidates
3

---

Bounding |Boses
Geometiic Group Construction
Semartic Group Construction
Rg
CA
Human Hold Surfboard
CNN
9/2
9%
Positional Enooding
Instamne Decoder
Geometric Local Transformer
Semaaic-aawrr Interaction Decoder
FFN
2+
0,
0n
Qiee
Visual Encoder
Object
Quectes
Queries
Figure Overview of GroupHOI. GroupHOI comprises three key modules: visual encoder extracts features, followed by an instance decoder to locate numan-object pairs; ii) a geometric- aware local transformer aggregates and distributes contextual information within geometric groups; iii) a semantic-aware interaction decoder leverages local-global semantic dependencies for interaction prediction.
or their compositions via attention, which inherently establish cfully-connected graph for information exchange. Specifically, given an entity candidate of E Q. = Q, U the relational modeling is achieved by the transformer decoder through self-attention operations with all entity queries:
q
Σ EGEQ. Sof Softmax (q: .q))
where Softmax propagates contextual cues by establishing soft correspondences from a global view, of Q. denotes all the detected entities.
Revisiting Graph-based Relation Modeling. Another line of research [11-13, 32, 54] constructs graphs based on the detected entities to enable relational modeling. Each node of is connected to predefined set of neighboring nodes Nr and exchanges contextual information via message passing:
q of
LafeN
Otij MeasagePassing((" {f)),
(3)
where a is the activation function, is an adjacency weight between nodes, and MessagePassing denotes the message passing function. In terms of the set of neighboring nodes (i). many studies [12, 13] define it as the union of all other entities in the scene, resulting comprising both human and object nodes, while others [11, 13, 32, 54] restrict it to heterogeneous entities (i.e., human-object, hajent, therehy formulating a bipartite graph.
fully-connected graph
Though effective, these methods either treat all the entities as a single group or crudely partition them into two homogeneous groups. However, real-world scenarios demonstrate greater complexity in collective patterns: picture a cocktail party where cluster of guests holding wine glasses or bottles debating heatedly, while individuals seated on chairs around the venue are eating pizza. Given this divergence, we propose learning HOI as groups guided by the Gestalt grouping principles [16]: the geometric proximity principle and semantic similarity principle. The former is operationalized by learning entities as geometric groups, and the latter by learning interactions as semantic groups.
3.3 Learning Entities as Geometric Groups
According to Gestalt psychology [16], the proximity principle states that spatially adjacent elements are naturally perceived as coherent groups. This perceptual tendency has been applied as a strong inductive bias for structuring visual scenes [70]. Motivated by this principle, extend it to HOI-DET by constructing geometric groups that organize detected entities based on their spatial configurations.
Geometric Group Construction. We construct groups for each entity embedding of by selecting its K nearest neighbor entities based on the geometric distance of their corresponding bounding boxes B obtained by the instance decoder. However, simple geometric metrics often fall short in capturing the diverse sizes and spatial configurations of bounding boxcs. To address this, we introduce learnable estimator to measure geometric proximity among bounding boxes. First, formulate the spatial feature fG = [diss,,, IoU4,]] by concatenating two geometric properties: the Euclidean distance disi, between the centroids of two bounding boxes e; and ej (i.e., dist. =
V (Ac )² + (Ac),)² where Act, = = - c5 and Act = = - -c"), ) and ii) the IoU between two

---

bounding boxes B, and B,, i.e., IoU1, = [B, n B, U B,l. employed to compute the proximity score si from fl, and yields proximity score matrix. this matrix, lower scores indicate closer spatial proximity, For each entity, we select the K neighbors with the lowest scores to construct its geometric neighbor set N?.
Then, simple linear layer is
Geometric Context Aggregation and Dispatch. GroupHOI adopts geometric-aware local trans- former to capture the contextual cues from unordered entities by applying self-attention locally. First, to preserve the relative positional information within groups, introduce trainable, parameterized position encodings ji,j as follows:
Pij _ qÿ),
(4)
where d a two-layer MLP with ReLU nonlinearity. Then, we compute the dispatch matrix b.g for each entity:
J Softmax(77(hn (g.) d2(gf) + Po.j));
(5)
where 01 and Φ2 are linear projections to perform woint-wise feature transformation, and is MLP to produce the subtraction relation. Based on dispath matrix, GroupHOI adaptively aggregates the contextual cues within groups and utilizes them to update the entity embeddings:
q of C (Øs + Pi.)) gi
(6)
where Øy is a linear projection, and 0 denotes a feature alignment function. This process is applied among homogeneous entities, yielding Qn and Qu for human and object embeddings. respectively.
3.4 Learning Interactions as Semantic Groups
The concept of semantic grouping has long been recognized in cognitive science as fundamental mechanism by which humans perceive and organize the world. According to Gestalt principles [16], humans naturally group elements based on contextual similarity to reduce cognitive load and enable efficient reasoning. Inspired by this insight, we incorporate semantic-aware grouping into interaction reasoning and enhance the vanilla transformer decoder with semantic-aware local cues.
Semantic Group Construction. We first initialize the interaction queries Qint by computing the mean of Qn and Q.. Given the complex and diverse group structures with overlapping memberships in social environments, we avoid using traditional partitioning techniques like k-means clustering to construct groups. Instead, we formulate distinct groups N for each interaction query q Qine by assessing pairwise cosine similarity sim,, = (q 93)) [[]لا]]]) and selecting its top-K* most similar counterparts.
between it and other queries q3
Semantic Context Aggregation. Given each query of with its semantic group N3; we utilize max pooling aggregate semantic message from its group members gf:
m; = mamax (gf q D)),, of NE
where 24 represents a sequence of linear layers, batch normalization, and ReLU layers.
Local-Global Integration. As described in $3.2, the transformer based interaction decoder models global relations through self-attention but lacks explicit local relation modeling. To address this limi- tation, we construct a semantic-avee context aggregation mechanism into the naive decoder, which enhances its capacity for interaction reasoning. Specifically, prior to each decoder layer, the local semantic context is aggregated and incorporated into the original query via residual connection:
interaction decoder by integrating the proposed local semantic
g. = q. + 65 (m:))
(8)
where d; shares the same identity structure as σ1. q. is subsequently passed through the self-attention and cross-attention modules in the interaction decoder layer.
3.5 Implementation Details
Network Architecture, To ensure fair comparison with previous HOI-DET methods [3, 4, 67, 71]. we adopt ResNet-50 [72] as the visual backbone. Our transformer-bases architecture consists of a 6-layer encoder, a 3-layer instance decoder, and 3-layer interaction decoder. Following [3, 4], we initialize 64 learnable queries for human and object branches, and set the feature dimensions to
5

---

256 for human/object representations and 768 for interaction representations. We perform group construction independently at each layer of both the instance and interaction decoders, where the geometric and semantic group sizes are set to 4 and 2. We evaluate three variants of GroupHOI using different pre-trained VL.Ms: i) CLIP (ViT-B/16) [8], ii) CLIP (ViT-I/144, and iii) BLIP2 (ViT-L.) [9]. In each variant, we follow [3, 4] to initialize the classification heads using VLM text embeddings and integrate visual features from the VLM visual encoder into the interaction decoder.
Training Objectives. Following [3, 4], our model is optimized by the following loss:
Luo >a + XuCu NO.2 X2
where Cs denotes the box regression loss, Lu indicates the represent the cross-cntropy loss for object and interactions classification, respectively. The coefficient factors (As, Xu, X2, { are empirically set as {2.5, 1,1.1).
loss, C and L
Reproducibility. GroupHOI is implemented in PyTorch. The model is trained with a batchsize of 8 for 90 epochs on GeForce RTX 4090 GPUs.
4 Experiment
4.1 Experiments HOI-DET
Datasets. We conduct experiments on V-COCO [18] and HICO-DET [2]:
V-COCO is a specialized subset of MS-COCO [73], which comprises 10,346 images (5,400 for training and 4,946 for testing). The dataset annotates 263 unique human-object interactions, covering 29 action categories and 80 object categories.
HICO-DET comprises total of 47,776 images, with 38,118 designated for training and 9,658 for testing. It includes 80 object categories, consistent with those in V-COCO dataset, 117 action categories and 600 distinct HOI classes.
Evaluation Metries. We employ mAP as our evaluation metric. For V-COCO [18]. mAP is reported under two different scenarios: scenario 1 for all 29 action categories, and scenario 2 which excludes body motion categories. For HICO-DET [2], we evaluate GroupHOI in complete 600 HOI categories (Full), the 138 rare categories with fewer than 10 training instances (Rare), and remaining 462 categories (Non-Rare). For each object, mAP computed in the whole dataset (Default) and subset containing the object (Known Object).
Training. Our backbone is initialized with DETR [6] pre-trained MS-COCO [73]. The model is trained using Adam W optimizer [74] with a batchsize 8 for 90 epochs. The initial learning is set to 5e which reduces by factor of 10 every 30 epochs.
Testing. For fairness, GroupHOI operates without data augmentation. We retain the top- HOI candidates (K- = 100) followed by Non-Maximum Suppression for redundancy removal.
Quantitative Results on HICO-DET, As shown in Table 1, our model with a VIT-B/I6 CLIP [8] variant already outperforms all the competitors under the same setting by a substantial margin. In
particular, GroupHOI surpasses the previous state-of-the-art art Pose-Aware [19] by 0.84/2.38/0.40 mAP on the Full, Rare, and Ron-Rare settings. Notably, GroupHOI exhibits a comparable number of pa- rameters and FLOPs to HOICLIP [4] ($4.3), but delivers substantial improvements of 2.11/3.74/155 mAP on HICO-DET [2]. Moreover, leveraging more powerful VLMs, i.e., VIT-L/14 CLIP [8] and ViT-L BLIP2 19], GroupHOI boosts its performance and sustains state-of-the-an results. Both with ViT-L/14 CLIP, GroupHOI suppresses CMMP [68] by large margin with 1.32 mAP under Full, while it narrowly trails CMMP in Rare setting. Compared to UniHOI [40] which retrieves knowledge from large language models, GroupHOI with ViT/L BLIP2 still leads by 0.47/ 0.71/0.39 mAP.
Quantitative Results on V-COCO. For V-COCO, GroupHOI exhibits competitive performance but remains slightly inferior to STIP [33], VIL [75], and CQL [76]. However, on HOI-DET, our method surpasses them by solid margin. We attribute this to GroupHOl's improved capability in handling more complex interactions, as evidenced by: HOI-DET contains 600 HOI interactions (vs. (v3.293 in V-COCO), an average of 4 interactions (vs.3 in V-COCO) and entities per sample (vs.4 in V-COCO).

---

Table 1: Quantitative results HICO-DET [2] test and V-COCO [18] test. DF and KO denote the Default
and Known Object evaluation settings of HICO-DET. CL CLIP/B32, CLIP/L114 represent VTT-B/16, VIT-B/32 and VIT-L/I4 settings for CLIP respectively. See $4.1 for details.
HICO-DET (DF)
HICO-DET (KO)
V-COCO
Method
Config
Full
Rare
Non-Rare
Full
Rare
Non-Rare
APSI
AP
nale
QPIC
CYPR211
R50
29.07
21.85
31.23
31.68
24.14
33.93
58.8
61.0
QPIC[67 CVPR21]
R101
29.90
23.92
31.69
32.38 26.06
34.27
58.3
HOTR [69]
CVPR21J
R50
23.46
16.21
25.60
55.2
64.4
CDN 17
[curIPS21]
R50
31.44
27.39
32.64
34.09
29.63
35.42
61.2
63.8
LDN NeurlPS21]
R50
31.78
27.55
33.05
34.53 29.73
35.96
62.3
64.4
(17 NeurlPS21]
RI01
32.07
27.19
33.53
34.79 29.48
36.38
63.9
65.9
MSTR 777
CVPR22]
R50
31.17
25.31
32.92
34.02 28.83
35.57
62.0
STIP[33]
CVPR22]
R50
32,22
28.15
33.43
35.29
31.43
36.45
65.1
69.7
PVIC[71 [CCV23]
R50
34.69
32.14
35.45
38.14 35.38
38.97
62.8
67.8
Pose-Aware
CVPR24)
R50
35.86
32.48
36.86
39.48
36.10
40.49
61.1
66.6
DOQ
CVPR22]
R50+CLIP/B
33.28
29.19
34.50
63.5
GEN-VLKT(3
CVPR22]
R50+CLIP/B 16
33.75
29.25
35.10
37.80
34.76
38.71
62.4
64.4
ADA-CM[79)
[CCV23]
R50+CLIP/B 16
33.80
31.72
34.42
56.1
61.5
HOICLIP
RS0+CLIP/B32
34.59
31.12
35.74
37.61
34.47
38.54
63.5
64.8
VIL[75]
ACMMM25
R50+CLIP/BIE
34.21
30.58
35.30
37.67
34.88
38.50
65.3
[76[76]
CVPR23]
R50+CLIP/B16 16
35.36
32.97
36.07
66.4
69.2
LOGICHOI
NeurIPS23
R50+CLIP/B16 16
35.47
32.03
36.22
38.21 35.29
39.03
64.4
65.6
CMMP[68 [CCV24]
R50+CLIP/B16 16
33.24
32.26
33.53
61.2
HOIGen [68 ACCMMM2
R50+CLIP/B16 16
34.84
34.52
34.94
CEFA [80 ACMMM24
R50+CLIP/B16
35.00
32.30
35.81
38.23
35.62
39.02
63.5
GroupHOI (ours)
R50+CLIP/B 16
36.70
34.86
37.26
39.42 37.78
39.91
65.0
66.0
ADA-CM[ [CCV23]
R50+CLIP/L14
38.40
37.52
38.66
58.6
64.0
UniVRD[8] [CCV23]
R50+CLIP//14
37.41
28.90
39.95
CMMP(68)
RCCV24)
R50+CLIP/1.14
38.14
37.75
38.25
GroupHOT
(ours)
R50+CIIPP/11
39.46
37.10
40.16
41.58
39.42
42.40
66.4
67.3
UniHOl[ 40 NeurlPS23)
R50+BLIP2
40.06
39.91
40.11
42.20
42.60
42.08
65.6
68.3
GroupHOI (ours)
R50+BIIP22
40.53 40.62
40.50
42.70
42.92
42.64
66.4
67.8
4.2
Experiments NVI-DET
Task Formulation. Given an input image, it predicts a set of (individual, group, interaction) triplets, aiming localize each individual and identify the social group belongs while determining the category of its nonverbal interaction. Unlike HOI-DET [18] that focuses on recognizing actions between human-object pairs, NVI-DET models interactions in more flexible and generalized manner. Specifically, it accounts for interactions involving arbitrary numbers of humans, which makes the task significantly more challenging.
Dataset. NVI [20] densely labeling social groups in pictures, along with 22 atomic-level nonverbal behaviors (16 individual- and 6 group-wise) under five broad interaction types. It contains 13,711 images in total and splits them into 9,634, 1,418 and 2,659 for train, val and test.
Evaluation Metrics. Consistent with [20]. we utilize mean Recall @ K (mR@K K) for evaluation. Specifically, we report mR @25, mR @ 50, and mR @ 100 under different IoU thresholds for matching predictions with ground truth, along with their average (AR) to provide comprehensive evaluation.
Quantitative Results. Following [20], we compare GroupHOI with modified versions three Hate-of-the-we HOI-DET methods (i.e., m-QPIC [67], m-CDN [17], and m-GEN- [3]) and NVI-DEHR [20] on NVI [20]. As shown in Table 2, GroupHOI surpasses all other methods, reaching 73.19 and 75.21 AR on NVI val and test. Furthermore, we analyze performance across individual- wise and group-wis interactions. Table shows our model consistently outperforms others by significant margins in both sets. These results verify our model's effectiveness in identifying social group structures and capturing the collective behaviors within them.
4.3 Diagnostic Experiments
As shown in Table 4, we conduct set of ablation studies HICO-DET [2] for deeper analysis.

---

Table 2: NVI-DET results on NVT [20] val and test. See $4.2 for details.
val
test
Method
mR @5
m0
mR @ 100
AR
mR 25
mR @
mR@ 100
AR
CH-QPIC [67] CVPR211
56.89
69.52
78.36
68.26
59.44
71.46
80.07
70.32
m [I-CDN[11 [17] (Nur1PS21)
55.57
71.06
78.81
68.48
59.01
72.94
82.61
71.52
m--EN-VLLKE [3] [CVPR22)
50.59
70.87
80.08
67.18
56.68
74.32
84.18
71.72
NVI-DEHR ( [CCC244)
54.85
73.42
85.33
71.20
59.46
76.01
88.52
74.67
GroupHOI (ours)
55.67
76.73
87.16
73.19
62.67
76.57
86.42
75.21
Table 3: Individual- and group-wise interactions results on NVI [20] val. See $4.2 for details.
Method
individual
group mR 50
m5@25
mR 50
mR @ 100
AR
mR @5
mR@ 100
AR
m-QPICI67 [CVPR21]
52.23
66.09
75.98
64.77
69.18
78.62
84.85
77.55
m m-CDNII [17] (NurtPS211
50.67
68.23
76.74
65.21
68.66
78.60
84.34
77.20
[3 [CVPR22)
44.98
68.51
78.30
63.93
67.84
79.47
87.12
78.14
NVI-DEHR [CCCCC))
49.37
70.04
83.82
67.74
69.47
82.45
89.35
80.42
GroupHOI (ours)
49.60
74.03
85.63
69.75
71.87
83.92
91.24
82.34
Analysis of Key Modules. We analyze the effects of geometric and semantic groups in our framework. As shown in Table 4 (a), the results indicate that: First, both geometric and semantic relation learning contribute HOI prediction. Incorporating geometric and semantic groups individually yields mAP gains of 0.67/0.91/0.3 and 0.21/0.45/0.. under Doll/Rare/Non-Rate settings respectively, Second, the combination of them yields optimal performance, delivering 1.98/2.66/0.70 mAp improvements over the baseline, which highlights a more comprehensive relation modeling in HOI prediction.
Table 4: Ablation study of 10 on HICO-DET [2] test, where GEO and SEM represent geometric group and semantic group, Homo. and Hetero. mean homogeneous group and heterogeneous group ($4.3).
(a) Analysis of key modules
(b) Analysis of geometric group size
(c) Analysis of semantic group size
GEO SEM Full
Rare
Non-Rare
Group Size Full Rare Non-Rare
Group Size Full Rare Non-Rare
35.72 32.20
36.56
K =2 36.23 32.97
37.13
K' = 36.07 32.09
37.26
36.39 33.11
33.37
K = 36.34 33.87
37.32
K = =2 36.70 34,86
37.26
35.93 32.65
36.87
K
36.70 34.86
37.26
K =3 36.18 32.67
37.21
36.70 34.86
37.26
K 5 36.45 33.57
36.95
K"
35.92 32.06
37.07
(d) Analysis of fusion strategy
(c) Analysis of geometric group layer
(f) Analysis of semantic group layer
Model Full Rare Non-Rare
Group Layer Full Rare Non-Rare
Group Layer Full Rare Non-Rare
Baseline 35.72 32.20
36.56
1==1 36.70 34,86
37.26
1==1 36.13
37.49
+ Homo. 36.23 32.40
37.20
L =
36.47 31.96
37.81
L' ="=2 36.53 32.20
37.82
+
Hetero. 36.70 34.86
37.26
La = 3 35.64 31.20
37.00
L' == 3
36.70 34.86
37.26
L
36.17 32.72
37.20
L 4 36.13 31.53
37.51
Analysis of Group Size. We evaluate the impact of the geometric group size K and semantic group size K" in Table 4 (b) and (c). Our design performs independent group construction at each decoder layer, allowing entities and interactions to exchange information with different neighbors across layers. As shown in the table, the performance peaks at K = 4 and drops K increases, suggesting that oversized geometric groups introduce noise from less relevant neighbors. In contrast, the model performs best at K* 22, and both larger and smaller K* degrade performance, implying only highly relevant semantic cues are beneficial for interaction prediction. Notably, since groups are constructed independently per layer, the geometric and semantic group sizes across the three-layer decoder range from 4-10 and 2-4, respectively, under the optimal setup. This aligns with real-world scenarios: over 94% of samples in V-COCO [18] and 91% in HICO-DET [2] involve no more than 10 entities, and even large groups are dominated by a few key participants (only around 4) [82].
Homo. vs. Hetero. Geometric Group. We make a comparison between homogeneous and heteroge- neous paradigms for geometric groups in Table 4 (d). The heterogeneous paradigm models humans and objects as distinct node types with exclusive intra-class (human- human/object- message passing. contrast, the homogeneous paradigm treats all entities uniformly, allowing unrestricted cross-entity communication. The heterogeneous design achieves higher performance, surpassing the homogeneous counterpart by 0.47/2.46/0.06 mAP. This disparity suggests that integrating humans and objects into a group may dilute relevant context and hinder the specialized information exchange.
8

---

(a) buman work on computer human sit chair
(b) human ride elephant
(c) human kick ball
(d) human carry suitcase huran walk
(c) humun hold busehall but ball
(9 human work on computer bursan mouse
( human sit car
(h) human hold umbrella carry bag
on phone
① human sit chair human bold phone
ii human ride snowboard
(k) human hit ball human look at ball
(1 buman jump human look at ball
Figure 3: Visualization GroupHOI results on V-COCO [18] test. The first two rows represent samples where all interactions are successfully detected, while the third row corresponds to samples with missed interactions. Detected interactions are marked in Green, while missed interactions and objects are in Red.
Analysis of Group Layer. Table 4 (e) and (f) present the evaluation of geometric group layer L and semantic group layer I.a. We observe that increasing L does not lead consistent improvement. While the performance improves as I. increases from to 3, after which the gains plateau, indicating that excessive local context aggregation brings diminishing returns.
Comparison of Model Efficiency. Table 5 com- pares the model scalability in terms of the num- ber of Parameters (Params). Floating Point Opera- tions (FLOPs), and Frames Per Second (FPS). A1- though GroupHOI maintains a comparable param- eter count and experiences only marginal reduc- tion in inference speed compared to HOICLIP [4], it achieves significantly better performance, out-
Table 5: Comparison of model efficiency.
Method
Params|.
FL.PPs..
FPS
P31M [CPPR20)
194.9M
121.63G
15.28
GEN-VLKT[S [3]
[CVP22)
41.9M
60.04G
26.18
HOICLIP [4] [CPR223)
66.1M
104.68G
19.57
ViPLO [83] [CVPR23]
118.2M
41.35G
14.89
GroupHOl(ouu)
79.2M
83.67G
16.42
performing HOICLIP by 2.11 mAP on HICO-DET [2] (see Table 1). Compared to ViPLO [83],
GroupHOI achieves superior performance (+1.75 mAP) while requiring fewer parameters and less in- ference time, underscoring its efficiency. A detailed comparison is available in Table S2 in Appendix.
4.4 Qualitative Results
Fig. 3 illustrates the qualitative results of GroupHOI on V-COCO [18] test. As seen, it can robustly localize and predict interactions under challenging conditions. For instance, our method generalizes well to interactions with little training data like riding elephant in Fig. 3(b), consistent with its superior rare-set performance (Table 1). As shown in Fig. 3(c-h), GroupHOI also handles complex scenes with multiple humans and objects, benefiting from effective strap information propagation. The third row presents failure cases, mainly due to: severe occlusion causing missed detections, e.g., the failure in detecting human hold phone (Fig. 3()) and human ride skateboard (Fig. 3()). ii) Ambiguous interactions arising from insufficient spatiotemporal information, such as difficulty in recognizing gaze direction (Fig. 3(k)). iii) Lack of misclassification of looking at ball as kicking ball (Fig. 3()), which is caused by lacking knowledge of basketball, i.c., players typically do not use their feet to kick the ball during games. These
knowledge, exemplified by the

---

geometric group L.
geometric group L2
geomeiric group L3
semantic group L!
semantic group L2
semantic group 1.3
Figure 4: Visualization of groups on V-COCO [18] test. L1, 1.2 and I.3 represent the first, second and last layer of instance interaction decoder. The center of the geometric and semantic groups is mark in Red ($4.5).
limitations could potentially addressed by developing spatiotemporal reasoning modules based on video-level datasets [84] or integrating knowledge via pre-trained large language models [40].
4.5 Visualization of Groups
Fig. 4 shows examples of geometric and semantic groups across decoder layers V-COCO [18] test. the first row, two main geometric group patterns emerge: i) multiple proposals targeting to the same human/object, enabling feature completion; ii) adjacent distinct proposals that interact closely to show mutual influence. Moreover, groups may also include unannotated entities, revealing GroupHOl's ability to exploit implicit contextual relations. When grouping by semantic similarity (the second row of Fig. 4), the grouping patterns can also be observed, i.e., distinct individuals performing the same interaction. In summary, our method demonstrates the capability automatically uncover latent patterns in complex scenarios, thereby enhancing the model's holistic scene comprehension.
Discussion
Future Direction. Current HOI benchmarks involve limited entities within small fields of view, leading to global relational modeling in the mainstream HOI-DET [18] methods. However, when applied larger scenarios (e.g., gigapixel-level crowd images [61]), this paradigm leads high computational cost and information redundancy. A practical solution is to confine relational modeling local regions, making the principles proposed this paper highly applicable real-world settings.
Limitation. A limitation of our method is its focus on interaction reasoning without integrating the proposed mechanisms into the object detection branch. We think this direction intriguing, yet lies beyond the scope of the present study. Moreover, like other HOI-DET [18] models, GroupHOI is trained with limited label diversity, posing challenges for generalization to in-the-wild scenarios.
6 Conclusion
We present GoupHOI, framework builds upon the idea of reorganizing the unordered in the visual scenes by exploring their inherent grouping patterns. By revisiting HOI-DET [18] task from a group perspective, we formulate two visual attraction principles, i.e., geometric proximity and semantic similarity, to explain group dynamics. These principles are instantiated via two paradigms: i) learning entities geometric groups, and ii) learning interactions semantic groups. By introducing marginal computational overhead, GroupHOI advances state-of-the-an art methods by solid margins and sets new SOTAs for HOI-DET and NVI-DET [20] task, which verifies its superiority.
Acknowledgement. This work was supported by National Natural Science Foundation of China (No. 62372405), Fundamental Research Funds for the Central Universities (226-2025-00057), Zhejiang Provincial Natural Science Foundation China (No. LD25F020001), ), and CIE-Tenced Robotics Rhino-Bire Focused Research Program.
10

---

References
[1] Georgia Gkioxari. Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human-object interactions. CVPR. 2018. 1,2
[2] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In WACV, 2018. 1, 2, 6, 7, 8, 9, 22, 23
[3] Yue Liao, Aixi Zhang. Miao Lu. Yongliang Wang, Xiaobo and Si Liu. Gen-vikt: Simplity association and enhance interaction understanding for hoi detection. In CVPR, 2022. 1, 5, 6, 7, 9, 23
[4] Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. In CVPR, 2023. 1, 1,3,5, 5, 6, 7, 22, 23
[5] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-can: Towards real-time object detection with region proposal networks. In NeurlPS, 2015. 1,
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 3,
[7] Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In CVPR, 2023. 1
[8] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, 2021. 1,3, 6, 2,
[9] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In JCML, 2022. 1,
[10] Weiho Jiang, Weihong Ren, Jiandong Tian, Liangqiong Qu, Zhiyong Wang, and Honghai Liu. Exploring self-and cross-triplet correlations for human-object interaction detection. In AAAI, 2024. 1,3
[11] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interactions by graph parsing neural networks. In ECCV, 2018. 1, 4
[12] Oytun Ulutan, ASM lftckhar, and Bangalore S Manjunath. Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. In CVPR, 2020. 1,
[13] Hai Wang, Wei-shi Zheng, and Ling Yingbiao. Contextual heterogeneous graph network for human-object interaction detection. In ECCV, 2020. 3,4
[14] Ravin Alaci and Nicholas O Rule. Accuracy of perceiving social attributes. The social psychology of perceiving others accurately, 2016.
[15] Michael Tomasello. Why we cooperate. MIT press, 2009. 2
[16] Willis D Ellis. A source book of Gestalt psychology. Routledge, 2013. 2,4,
[17] Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li. Mining the benefits of two-stage and one-stage hoi detection. In NeurlPS, 2021. 2, 8, 23
[18] Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint arXiv:1505.044744 2015. 2. 6, 7, 10, 23,
[19] Eastman ZY Wu, Yali Li. Yuan Wang, and Shengjin Wang. Exploring pose-aware human- interaction via hybrid learning. In CVPR, 2024. 2, 3, 6,
[20] Jianan Wei, Tianfei Zhou, Yi Yang. and Wenguan Wang. Nonverbal interaction detection. ECCV. 2024.
2, 7, 7,8811 10
[21] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He. Pose-aware multi-level feature network for human object interaction detection. In ICCV, 2019. 2
[22] Chen Gao, Yuliang Zou, and Jia-Bin Huang. ican: Instance-centric attention network for human-object interaction detection. arXiv preprint arXiv: 1808. 10437, 2018. 23
[23] Yong- Lu Li. Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang. Yanfeng Wang, and Cewu Lu. Transferable interactiveness knowledge for hujan- interaction detection. In CVPR, 2019. 3
11

---

[24] Tanmay Gupta, Alexander Schwing, and Derek Hoiem. No-frills human-object interaction detection: Factorization, layout encodings, and training techniques. In ICCV, 2019.
[25] Yang Liu, Qingchao Chen, and Andrew Zisserman. Amplifying key eues for numan-object-ioe coiman--bccttinteraatioon detection, ECCV, 2020.
[26] Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon. Detecting human-object interactions with action co-occurrence priors. In ECCV, 2020.
[27] Tianfei Zhou, Wenguan Wang, Siyuan Qi, Haibin Ling, and Jianbing Shen. Cascaded human-object interaction recognition. In CVPR, 2020.
[28] Tianfei Zhou, Siyuan Qi. Wenguan Wang, Jianbing Shen, and Song-Chin Chen Zhu. Cascaded parsing of human-object interaction recognition. IEEE TPAMI, 2021. 2
[29] Hao-Shu Fang, Yichen Xie, Dian Shao, and Cewu Lu, Dirv: Dense interaction region voting for end-to-end human-object interaction detection. In AAAI, 2021. 3
[30] Xubin Zhong, Xian Qu, Changxing Ding, and Dacheng Tao. Glance and gaze: Inferring action-awan points for one-stage human-object interaction detection. In CVPR, 2021. 3
[31] Yue Liao, Si Liu, Fei Wang, Yanjie Chen. Chen Qian, and Jiashi Feng. Ppdm: Parallel point detection and matching for real-time buman-object interaction detection. In CVPR, 2020. 9, 23
[32] Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object interactions. In ICCV, 2021. 3, 4, 23
[33] Yong Zhang. Yingwei Pan, Ting Yao, Rui Huang. Tao Mei, and Chang Wen Chen. Exploring structure- aware transformer over interaction proposals human-object interaction detection. CVPR, 2022. 3, 7, 23
[34] Desen Zhou, Zhichao Liu, Jian Wang, Leshan Wang, Tao Hu, Errui Ding, and Jingdong Wang. Human- object interaction detection via disentangled transformer. In CVPR, 2022.
[35] Bumsoo Kim, Tacho Choi, Jaewoo Kang, and Hyunwoo J Kim. Uniondet: Union-level detector towards real-time human-object interaction detection. In ECCV, 2020.
[36] Tiancai Wang, Tong Yang. Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun. Learning human-object interaction detection using interaction points. In CVPR, 2020.
[37] Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, et al. End-to-end human object interaction detection with hoi transformer. In CVPR, 2021. 23
[38] Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural-logic homent interaction detection. In NeurIPS, 2024.
[39] Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Hydra-sgg: Hybrid relation assignment for one-stage scene graph generation. In /CLR, 2025. 3
[40] Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning foundation models. In NeurlPS, 2023. 3, 6, 7,
[41] Yang. Bingliang Li, Fengyu Yang, Ailing Zeng, Lei Zhang, and Ruimao Zhang, Boosting human-object interaction detection with text-to-image diffusion model. arXiv preprint arXiv:2305. 12252, 2023.
[42] Liulei Li, Wenguan Wang, and Yi Yang. Human-object interaction detection collaborated with large relation-driven diffusion models. In NewrlPS, 2024.
[43] Jinguo Luo, Weihong Ren, Weibo Jiang, Xi'ai Chen, Qiang Wang. Zhi Han, and Honghai Liu. Discovering syntactic interaction clues for human-object interaction detection. In CVPR, 2024.
[44] Guikun Chen. Jin Li, and Wenguan Wang. Scene graph generation with role-playing large language models. In NeurlPS, 2024. 3
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent difffusion models. In CVPR, 2022. 3
12

---

[46] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.001 01068, 2022. 3
[47] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR, 2014,
[48] Ross Girshick. Fast r-cnn. ICCV, 2015.
[49] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[50] Jianan Li, Xiaodan Liang. Jianshu Li, Yunchao Wei, Tingfa Xu, Jiashi Feng, and Shuicheng Yan. Multistage object detection with group recursive learning. IEEE TMM, 2017. 3
[51] Wonmin Byeon, Thomas M Breuel, Federico Raue, and Mareus Liwicki. Scene labeling with Istm recurrent neural networks. CVPR, 2015. 3
[52] Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Exploiting scene graphs for human-object interaction detection. In ICCV, 2021.
[53] Penghao Zhou and Mingmin Chi. Relation parsing neural network for human-object interaction detection. In ICCV, 2019.
[54] Chen Gao, Jiarui Xu, Yuliang Zou, and Jia Bin Huang. Drg: Dual relation graph for human-object interaction detection. In ECCV, 2020. 4, 23
[55] Lifeng Fan, Wenguan Wang, Siyuan Huang. Xinyu Tang, and Song-Chun Zhu. Understanding human gaze communication by spatio-temporal graph reasoning. In ICCV, 2019.
[56] Mu Chen, Liulei Li, Wenguan Wang, and Yi Yang. Difivsgg: Diffusion-driven online video scene graph generation. In CVPR, 2025. 3
[57] Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. In CVPR, 2022. 3
[58] Buzhen Huang, Jingyi Ju. Zhihao Li, and Yangang Wang. Reconstructing groups of people with hypergraph relational reasoning. In ICCV, 2023. 3
[59] Jing Shao, Chen Change Loy, and Xiaogang Wang. Scene-independent group profiling in crowd. In CVPR, 2014. 3
[60] Loris Bazzani, Marco Cristani, and Vittorio Murino. Decentralized particle filter for joint individual-group tracking. In CVPR, 2012. 3
[61] Xueyang Wang, Xiya Zhang, Yinheng Zhu, Yuchen Guo, Xiaoyun Yuan, Liuyu Xiang. Zerun Wang. Guiguang Ding, David Brady, Qionghai Dai, al. Panda: A gigapixel-leve numan-centric video dataset. In CVPR, 2020. 3, 10
[62] Stefano Pellegrini, Andreas Ess, and Lue Van Gool. Improving data association by joint modeling of pedestrian trajectories and groupings. In ECCV, 2010.
[63] Kota Yamaguchi, Alexander Berg, Luis E Ortiz, and Tamara Berg. Who are you with and where are you going? In CVPR, 2011.
[64] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In CVPR, 2018. 3
[65] Jianchao Wu, Limin Wang, Li Wang, Jie Guo, and Gangshan Wu. Learning actor relation graphs for group activity recognition. In CVPR, 2019. 3
[66] Rui Yan, Lingxi Xie, Jinhui Tang, Xiangbo Shu, and Qi Tian, Social adaptive module for weakly-supervised group activity recognition. In ECCV, 2020. 3
[67] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise human-object interaction detection with imdee-wide contextual information. In CVPR, 2021. 3, 5, 7,8, 23
[68] Ting Lei, Shaofeng Yin, Yuxin Peng, and Yang Liu. Exploring conditional multi modal prompts for shot hoi detection. In ECCV, 2024. 6,7
13

---

[69] Burssoo Kim. Junhyun Lee. Jaewoo Kan, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to-end human- object interaction detection with transformers. In CVPR, 2021. 7,
[70] Tanqiu Qiao, Qianhui Men, Frederick WB Li, Yoshiki Kubotani, Shigeo Morishima, and Huhert PH Shum. Geometric features informed multi- multi-person human-object interaction recognition in videos. In ECCV, 2022.
[71] Frederic Z. Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong. and Stephen Gould. Exploring predicate visual context in detecting human object interactions. /CCV, 2023. 5,7
[72] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 5
[73] Tsung-Y- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014. 6
[74] Πyy Loshchilov and Frank Hutter. arXiv:1711.0510/ 2017. 6
Decoupled
weight decay regularization.
arXiv preprint
[75] Shuman Fang. Shuai Liu, Jie Li, Guannan Jiang, Xianming Lin, and Rongrong Ji. Improving human-object interaction detection via virtual image learning. In ACM MM, 2023. 6,
[76] Chi Xie, Fangao Zeng, Yue Hu, Shuang Liang, and Yichen Wei. Category query learning for human-object interaction classification. In CVPR, 2023. 6,
[77] Bumsoo Kim, Jonghwan Mun, Kyoung-Woi Woon On, Minchul Shin, Junhyun Lee, and Eun-Sol Kim. Mstr: Multi-scale transformer for end-to-end human-object interaction detection. In CVPR, 2022."
[78] Xian Qu, Changxing Ding, Xingao Li, Xubin Zhong, and Dacheng Tao. Distillation using oracle queries for transformer-baset human-ohject interaction detection. In CVPR, 2022. 7
[79] Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng, and Yang Liu. Efficient adaptive human- object interaction detection with concept-guided memory. In ICCV, 2023.
[80] Lijun Zhang, Wei Suo, Peng Wang, and Yanning Zhang. A plug-and-play method for rare human-object interactions detection by bridging domain gap. In ACM MM, 2024.
[81] Long Zhao, Liangzhe Yuan, Boqing Gong, Yin Cui, Florian Schroff, Ming- Hsuan Yang, Hartwig Adam, and Ting Liu. Unified visual relationship detection with vision and language models. ICCV, 2023. 7
[82] Nicolas Fay, Simon Garrod, and Jcan Carletta. Group discussion intcractive dialogue or as serial monologue: The influence group size. Psychological science, 2000.
[83] Jeeseung Park. Jin- Woo Park, and Jong-Seok Lee. Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. CVPR, 2023. 9, 23
[84] Hema S Koppula and Ashutosh Saxena. Anticipating human activities using object affordances for reactive robotic response. IEEE TPAMI, 2015.
[85] Tal Ridnik, Emanuel Ben-Baruch. Nadav Zamir, Asaf Noy. Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor Asymmetric loss for multi-label classification. In ICCV, 2021. 22
[86] Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian. Reformulating hoi detection as adaptive set prediction. In CVPR, 2021. 23
[87] Yunyao Mao, Jiajun Deng, Wengang Zhou, Li, Yao Fang, and Houqiang Li. Clip4hoi: Towards adapting clip for practical zero-shot hoi detection, In NeurlPS, 2023. 23
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction make claims which accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The main limitations are discussed in $5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
There is no theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The implementation details have been provided in $3.5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

reproducible, they Answer: [Yes]
Justification:
Code is available at htt.:://ggthhuo.coo///jaaaa//ogp
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
wwsssooooiccyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
gguide//Codeuummi / wttpccccppppccccccccc wwwpc//ggg////eeeeeeeeeeeeeeeeee /oodeSubmeeeeeeee cerbbbissioooo aasionooliie
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setup, including data splits, training and testing detailed, provided in $4.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The prior work included into comparison does not provide error bar. could be too expensive and time-consuming to run the experiments on large-scale datasets for multiple times
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources are described in $4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http///eerri...eee dtl.oiiiee/hhhcccccielleleelee
Answer: [Yes]
Justification:
This work conforms the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The border impacts provided Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The proposed method uses pre- pre-trained models. This proposed methods is safe under the safeguards of adopted pre-trained models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: V-COCO dataset is released under the MIT license. HICO-DET dataset is released under the CCO: Public Domain license. The weight of DETR is released under Apache 2.0 license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We will release our code base with README file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: There is no research with human subjects in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is no research with human subjects in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLMs employed in this work are illustrated in $3.5.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: http///urrrrccccce//e /eeeurpppscc//ooffeeenc/e/////// /LLM) for what should or should not be described.
21

---

Summary of the Appendix
For a better understanding the main paper, we provide additional details in this supplementary material, which is organized as follows:
A provides more implementation details of GroupHOI.
B offers more qualitative results.
SC provides more experimental results.
SD presents the pseudo code of GroupHOI.
SE discusses the societal impact of this work.
A More Implementation Details
We follow HOICLIP [4] in converting HOI triplets and object labels into textual descriptions to generate CLIP [8] text embeddings. Specifically, each HOI triplet <human, verb, object> is converted into a sentence like "A photo of person [verb-ing] a/an [object]". For "no-interaction" instances, use "A photo of person and a/an [object]", and object labels are converted into "A photo of a/an [object]]. These textual descriptions are then processed by the pre-trained CLIP text encoder to obtain corresponding embeddings, which initialize the weights of the interaction classifier C and object classifier C During training, these classifiers are fine-tuned with a small learning rate to adapt to the specific dataset. We employ the pre-trained CLIP visual encoder to extract visual features Votip These features, along with our encoded features V., are independently processed by separate interaction decoders. The resulting outputs then fused to facilitate the final reasoning. We also introduce two key modifications based on HOICLIP [4]. First, to address the dimensional mismatch between positional embeddings (256) and the interaction decoder (768), we expand the positional embeddings by stacking them three times. Second, we replace the Focal Loss with Asymmetric Loss [85] to better handle the long-tail distribution of HOI categories.
B More Qualitative Results
We further provide qualitative examples of our approach in Fig. S1. These results highlight GroupHOI's robust performance in HOI detection across various scenes. Notably, our model effectively captures complex interaction patterns in scenarios involving group activities such as team sports (e.g., soccer or tennis). This capability enhances interaction prediction with mutual communication. We also present several failure examples of our model, primarily due to missed object detection, as seen in Fig. (1()). Additionally, our model encounters challenges when dealing with highly ambiguous relations. For instance, in Fig. S1(s), GroupHOI fails to detect the look at ball between the girl at the center and the ball, which is disrupted by her surrounding teammates.
C More Experiments
Ablative Experiments. We evaluate four strategies for measuring geometric proximity between entities, using intersection-over-uiioo (IoU), center distance (CD), and global image features (IF). As shown in Table $1, combining IoU and Table $1: Analysis of geometric proximity measurement HICO-DET [2] test(SC).
CD consistently yields the best performance across all splits, indicating that both spatial cues complement each other effectively. In contrast, adding global image features slightly degrades performance, suggesting that they are not essen- tial for proximity estimation.
Measurement
Full
Rare
Non-Rare
10U only
36.02
32.19
37.16
CD only
36.14
33.60
36,90
I0U CD
36.70
34.86
37.26
I0U CD +IF
36.21
33.45
36.97
Efficiency Comparison. Table S2 presents a comprehensive comparison of the model scalability (i.e., number of Parameters (Params), Floating Point Operations (FL.Oss) and Frames Per Second (FPS)) for various HOI detection methods. As shown, GroupHOI achieves significant performance improvements over previous models while maintaining comparable number of parameters. We also
22

---

Table S2: Comparison of efficiency and performancc HICO-DET [2] teat and V-COCO [18] teat.
Method Backbone
Params
FLOPs| FPSt
Default
AP81
AP82
Full
Rare
Non-Rare
Pole
rale
[22 [RMVCIS)
R50
14.84
10.45
16.15
45.3
DRG [54] [BCCV3))
R50-FPN
46.1
19.26
17.74
19.71
51.0
PPDM [3 [CVPR20]
HG104
194.9
21.73
13.78
24.10
SCG
[CCC22]
R50-FPN
53.9
31.33
24.72
33.31
54.2
60.9
HOTR
[CVPRI]]
R50
25.10
17.34
27.42
55.2
HOITrans 337
[CVP22]]
R50
41.4
23.46
16.91
25.41
52.9
AS-Net [86]
[CVPR]]]
R50
52.5
28.87
24.25
33.14
53.9
[61C [CVRR2I]
R50
41.9
29.07
21.85
31.23
58.8
61.0
C17-S(17 [Neu1PS211
R50
31.78
27.55
33.05
62.3
STIP [33 [CVPR2]]
R50
50.4
32.22
28.15
33.43
65.1
69.7
GEN-VLKT
[CVPR22]
R50
41.9
60.04
26.18
33.75
29.25
35.10
62.4
64.4
HOICLIP [4] [CVPR23]
R50
66.1
104.68
19.57
34.59
31.12
35.74
63.5
64.8
CLIP4HOI [NeurIP225]
R50
71.2
35.33
33,95
35.74
ViPLO[S3 [CVPR23]
VIT-B/32
118.2
34.95
33.83
35.28
60.9
66.6
GroupHO1 (ours)
R50
79.2
83.67
16.42
36.70
34.86
37.26
65.0
66.0
compare FLOPs and FPS with GEN-VLKT [3] and HOICLIP [4]. Despite marginal reduction in in- ference speed relative to HOICLIP, GroupHOI has lower FLOPs and yields solid mAP improvements of 2.11/..774/155 in HICO-DET [2], highlighting the effectiveness of our proposed framework.
D Pseudo Code
The pseudo code for semantic and geometric group are given in Algorithm and Algorithm
Algorithm 1: Pscudo-code for Geometric Group in a PyTorch-like style.
ha: output buman/object enbaddinga from the instance decoder.
pos enbed: position unbeddings for bejjat queries. coords: bounding boxes of hunans/objects. K_g: geonetric group
def Geonetric, Group(he, poe, poe_ enbed, coorde, K_g): Formulate the spatial feature
F_p Cat Cat([Square [Bquare_distance diseedddttaceeeeeee (coorde (:, None], (None, :)), [coordd, Mone], [None, Compute the proximity score
3 Linsar (F
topk neighbors
knn_idx (S, K_g)
Compute position encodinge pas_enc MLP (LP(pos, (poos_enbedd kathor(pos_enaan kmn_idx))
Formulate
and value
9, Linear(he), Linear (he, knn_idx)), Linear (Gather (ha, kdx))dx))
Compute diepatch matrix
G Sortmax k pos_enc)
Aggrogate
context
C_g Linear(G out he + C_g
poa_enc))
Algorithm 2: Pseudo-code for Semantic Group in Py PyTorch-like style.
interaction enbeddinge K_e: senantic group size.
interaction decoder.
def Semantic, Croupc (bs, pos_enbed, K_s):
Compute aimilarity score
3 CosineSinilarity iesiiiilarttylae (be [i, Kone], he[;, Nons])
Select the topk neighbors knn_idx (8, X_s)
Aggrogate senantic context
(MLP None], ha ha[:, None] -Gathar (ha, kmn_idx)), dim--1)) out ha + C_
23

---

E
Boarder Impact
This work advances the recognition of human-object interactions in complex scenarios, particularly in scenes where small groups of people naturally form, which is common occurrence in real-world set- tings. This capability holds significant promise for applications in collaborative robotics, autonomous systems, healthcare monitoring, among others. However, there are also potential downsides. Our method risks propagating irrelevant contextual information among entities that merely happen to be co-located but share no collective pattern, leading to "hallucinated" interaction predictions. Moreover, group-level clustering may inadvertently propagate systemic biases, particularly scenarios requiring differential treatment of individuals within clusters (e.g., unfair reward and punishment allocation in 10 crowd behavior analysis). Hence, it essential to rigorously consider legal regulations and integrate certain fairness constraints to avoid potential negative socictal impacts.
24

---

(a) human hold temnis ricket human hit ball
(b) human hold surfbourd human walk
(c) human hold baseball bat human hold bag
(d) human work on computer human hold mouse
(c) human hold cup hurnn hold phone
human sit chair (1
(g)
human hold bowl human walk
human snowboard snowboard bb)
human walk
human hold human
human throw frishee human look frisbee
(i snowboard snowboard human jump snowboard
human hold seball bat human stand
human catch frishce human jump
human look human human ride hoat
(n) human hold baseball bat human look ball
human bed human bed
(p) bumun kick ball human
(g) human ride elephunt
(r) humaa look at ball human kick ball
(s) humun look at ball human jump
(0 human work computer human sit chuir
(v) human hold knife human sit chair Figure S1: Visualization of GroupHOI results on V-COCO [18] test. Detected interactions are marked in Green, while missed interactions and objects are in Red.
(u) human hold human smile
(w) human skateboard skateboard human jump
(x) human hold knifé
human cat cake
25


Reviews:
[{"review_id": "1PNFv7Mlql", "reviewer": "Reviewer_oLM7", "summary": "This paper introduces a framework for Human-Object Interaction detection that re-examines relation modeling from a group perspective.\n\nThe authors propose that current HOI methods neglect the inherent collective and non-pairwise social attributes within human-centric scenarios.\n\nTo address this, GroupHOI leverages two visual attraction principles: geometric proximity and semantic similarity. Geometric proximity is exploited by grouping humans and objects into distinct clusters using a learnable proximity estimator based on spatial features derived from bounding boxes. Semantic similarity is incorporated by enhancing a transformer-based interaction decoder with semantic-aware local cues derived from HO-pair features. \n\nThe paper presents extensive experiments on the HICO-DET and V-COCO benchmarks, claiming superior performance over state-of-the-art methods.", "questions": "Clarification on novelty vs. existing graph-based methods. Please provide a more detailed and concrete discussion distinguishing GroupHOI's geometric and semantic grouping mechanisms from prior graph-based HOI detection methods (e.g., DRG, SCG, and other works that use spatial/semantic relationships for graph construction). What are the fundamental differences in how groups are formed and how information is propagated that makes GroupHOI a novel approach rather than an incremental improvement?\n\nElaborating \"social attributes\". Could you expand on how the proposed geometric and semantic grouping truly captures \"inherent social attributes\" beyond basic visual co-occurrence or similarity? Are there specific architectural components or loss functions that encourage the model to learn more abstract social cues, or is \"social attributes\" primarily a high-level interpretation of visual grouping?\n\nComprehensive SOTA comparison. Please provide an updated comparison table that includes very recent state-of-the-art methods, especially those published in the last three years, to firmly establish GroupHOI's competitive performance. If the current SOTA claim is indeed challenged by other works, please re-evaluate and clarify the contribution in light of these stronger methods.\n\nMore details. Could the authors provide more fine-grained implementation details? This could include specific architectural diagrams for the geometric-aware local transformer and semantic-aware interaction decoder, exact configurations for the learnable proximity estimator and pooling operator, and any non-standard training tricks or hyperparameter tuning procedures.\n\nExperimental analysis on the cost and burden of the proposed method.", "limitations": "The paper lacks discussion over the additional cost as a limitation.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n1. Presentation. The paper is generally well-written and easy to follow. The methodology is explained logically, and the figures, particularly Figure 1 illustrating relation modeling paradigms, are helpful in understanding the core ideas.\n\n\n2. Novel perspective on social attributes for HOI detection. The emphasis on learning social attributes of human interactions in groups is an interesting and valid perspective. It attempts to capture more complex, collective patterns in visual scenes, which aligns with how humans naturally categorize and understand groups.\n\n3. Structured Grouping Principles: The definition of two visual attraction principles – proximity and similarity – to construct groups is clear and easy to follow. It makes the contribution of the paper more understandable.\n\nCons:\n\n1. Originality for overlap with existing graph-based methods. The claim of novelty for the proposed solutions, specifically the geometric-aware local transformer and semantic-aware interaction decoder, needs further justification. As is shown in Section 3.2, these solutions bear significant resemblance to previous two-stage HOI detection methods that focus on graph modeling. The paper's distinction from existing graph-based methods like DRG (ECCV 2020) , SCG (ICCV 2021), SKGHOI (ICDMW 2023) utilizing spatial relationships and semantic similarities to build graphs for HOI classification, is not sufficiently highlighted or critically analyzed. The differences seem incremental rather than a significant departure from established graph-based approaches.\n\n\n2. Weak link of the solution to the motivation of \"social attributes in a group of people\". The connection between the proposed technical solutions (geometric-aware local transformer and semantic-aware interaction decoder) and the overarching idea of \"learning social attributes of human interactions in a group\" appears tenuous. While geometric proximity and semantic similarity can contribute to group formation, the paper doesn't deeply elaborate on how these mechanisms specifically capture or represent \"social attributes\" beyond basic visual grouping. The discussion remains at a low-level feature aggregation rather than a high-level social understanding.\n\n3. Claim of SOTA performance. The claim of sota) performance may not be entirely accurate. The paper appears to miss some recent works published in the past three years that might achieve higher performance, especially on the V-COCO dataset. For instance, VIL (ACMMM 2023) and CQL (CVPR 2023) and some more recent works obtain stronger performance than GroupHOI on V-COCO, which could challenge the paper's main empirical claim. A more comprehensive and up-to-date comparison is necessary.\n\n4. Reproducibility for insufficient implementation details. While Section 3.5 provides some implementation details, they may not be granular enough to ensure full reproducibility of the work. \n\n5. The additional modules may brought significant computational overhead. A experimental analysis on the cost is very important.\n\n6. In the last paragraph of introduction, the authors seem to confuse the results for V-COCO and HICO-Det.", "was_revised": true, "final_justification": "Rebuttal address my main concern, I will increase my score.", "rebuttal": "We thank reviewer oLM7 for the valuable time and constructive feedback. We provide point-to-point response below.\n\n***\n\n**Q1: The novelty needs further justification.**\n\n**A1**：We apologize for the confusion. In fact, our method is significantly different from previous two-stage graph-based HOI approaches in the following aspects:\n\n**1) Motivation**\n\nPrevious two-stage HOI detection methods, such as DRG [ref1] , SCG [ref2] and SKGHOI [ref3], primarily construct graphs to perform the interaction reasoning between humans and objects. In contrast, our method constructs groups to explore the **latent social attributes** based on two visual attraction principles, enhancing feature representation of each proposal by aggregating information from its neighbors.\n\n**2) Formulation**\n\nPrior work constructs a **complete bipartite graph** between objects and humans with size $N_h \\times N_o$, where $N_h$ and $N_o$ denote the number of object and human proposals, respectively. Our method constructs **localized geometric and semantic graphs** with sizes only  $(N_h + N_o) \\times K_g$ and $N_i \\times K_s$, respectively, with $N_h$, $N_o$, and $N_i$ denoting the number of human, object, and interaction proposals, and $K_g$, $K_s$ representing the geometric and semantic group sizes. Notably, $K_g \\ll N_h/N_o$ and $K_s \\ll N_i$, highlighting the efficiency of our method.\n\n**3) Implementation**\n\nPrior two-stage methods relied on Graph Neural Network (GNN) architectures, employing iterative message passing via message functions to perform interaction reasoning. Our method can be directly integrated into one-stage frameworks. It defines neighbor sets defined by geometric proximity and semantic similarity, and employs simple yet effective operators (i.e., self-attention and max-pooling) to enhance the feature representations of human/object and interaction proposals.\n\nIn summary, our method fundamentally differs from prior graph-based methods in motivation, formulation, and implementation, which underscores the novelty of our work.  We will revise the Sec. 2 of the manuscript to highlight our novelty.\n\n[ref1] DRG: Dual Relation Graph for Human-Object Interaction Detection. ECCV, 2020.\n\n[ref2] Spatially Conditioned Graphs for Detecting Human-Object Interactions. ICCV, 2021.\n\n[ref3] SKGHOI: Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection. ICDMW, 2023.\n\n\n**Q2: Weak link to the motivation of \"social attributes in a group of people\"**. \n\n**A2**：Thank you for pointing this out. We acknowledge the potential ambiguity in our use of the term \"social attributes\", which we intended to denote collective behaviors observed in real-world interactions (i.e., multiple humans engaging in joint activities).\nInspired by this, we approach relation modeling from a group perspective, conceptually grounded in the Gestalt principles of Proximity and Similarity [ref4], which describe the mechanisms underlying human visual organization. To operationalize these principles, we formulate **geometric proximity groups** and **semantic similarity groups**, and propose two modules (i.e., geometric-aware local transformer and semantic-aware interaction decoder) to integrate these groups into the HOI detection pipeline. To enhance clarity, the Abstract and Sec. 1 of the manuscript will be revised as follows:\n\nRevision for Abstract: \n\n> However, current HOI methods primarily focus on pairwise relationships, overlooking the social attributes in a group of humans (i.e., multiple humans engaging in joint activities).\n\nRevision for Sec. 1: \n\n> Despite achieving strong performance, these methods are still confined to predefined pairwise relations, leaving the inherent social attributes which exhibit collective patterns (i.e.,  multiple entities engaging in joint activities) unexplored. \n\n> we construct social groups guided by the Gestalt grouping principles of Proximity and Similarity: First, the geometric proximity principle, i.e., the tendency for individuals to form groups with those physically close to, is implemented by constructing geometric groups based on the central distance and Intersection over Union (IoU) of bounding boxes. Second, the semantic similarity principle, i.e., the tendency for individuals to affiliate with those sharing visual characteristics or behavioral patterns, is operationalized through building semantic groups of interaction proposals based on similarity of their feature embeddings. \n\n[ref4] Ellis, W. D. (2013). A source book of Gestalt psychology. Routledge.\n\n\n**Q3: More recent sota works for comparison**. \n\n**A3**：Thanks for your suggestion. Per your request, we compare our model with more recent works published in the past three years.\n\n| Method              | Full (DF) | Rare (DF) | Non-Rare(DF) | $AP^{S1}_{role}$ | $AP^{S2}_{role}$ |\n| ------------------- | --------- | --------- | ------------ | ---------------- | ---------------- |\n| VIL [ACMMM23]       | 34.21     | 30.58     | 35.30        | 65.3             | 67.7             |\n| CQL [CVPR23]        | 35.36     | 32.97     | 36.07        | 66.4             | 69.2             |\n| HOIGen [ACMMM24]    | 34.84     | 34.52     | 34.94        | -                | -                |\n| CEFA [ACMMM24]      | 35.00     | 32.30     | 35.81        | 63.5             | -                |\n| **GroupHOI (Ours)** | **36.70** | **34.86** | **37.26**    | **65.0**         | **66.0**         |\n\nFor V-COCO, GroupHOI exhibits competitive performance but remains inferior to STIP [ref5], VIL [ref6], and CQL [ref7]. However, on HOI-DET, our method surpasses these approaches by a solid margin, especially for rare interactions. We attribute this to the enhanced capabilities of GroupHOI in more complex interaction scenarios, as evidenced by: HOI-DET contains **600** HOI interactions (vs. **293** in V-COCO), an average of **4** interactions (vs. **3** in V-COCO) and **6** entities per sample (vs. **4** in V-COCO). Furthermore, we also provide additional validation on NVI dataset [ref8], which contains an average of **5**  social interactions involving a group of humans (from **2** to **12**). The experimental results on NVI val also support this claim:\n\n| Method              | mR@25     | mR@50     | mR@100    | AR        |\n| :------------------ | :-------- | :-------- | :-------- | :-------- |\n| m-QPIC              | 56.89     | 69.52     | 78.36     | 68.26     |\n| m-CDN               | 55.57     | 71.06     | 78.81     | 68.48     |\n| m-GEN-VLKT          | 50.59     | 70.87     | 80.08     | 67.18     |\n| NVI-DEHR [ECCV24]   | 54.85     | 73.42     | 85.33     | 71.20     |\n| **GroupHOI (Ours)** | **55.67** | **76.73** | **87.16** | **73.19** |\n\nThe  results and discussion on V-COCO will be updated in Sec. 4.2. We will also add Sec. 4.4 for the analysis of results on NVI dataset [ref8]  in the revised manuscript.\n\n[ref5] Exploring structure-aware transformer over interaction proposals for human-object interaction detection. CVPR, 2022.\n\n[ref6] Improving Human-Object Interaction Detection via Virtual Image Learning. ACMMM, 2023.\n\n[ref7] Category Query Learning for Human-Object Interaction Classification. CVPR, 2023.\n\n[ref8] Nonverbal Interaction Detection. ECCV, 2024.\n\n**Q4: More implementation details.** \n\n**A4**：Thanks for your suggestion. More implementation details and pseudo code can be found in Sec. A and Sec. D of the Supplementary Material, respectively. The Sec. A has been expanded to include additional details as follows:\n\n> To address the dimensional mismatch between positional embeddings (256) and the interaction decoder (768), we expand the positional embeddings by stacking them three times. During training, we replace the Focal Loss used in HOICLIP [ref9] with Asymmetric Loss [ref10] to better handle the long-tail distribution of HOI categories.\n\nFuthermore, as stated in L17 and L245, we commit to making our code publicly available upon acceptance.\n\n[ref9] Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. CVPR, 2023.\n\n[ref10] Asymmetric loss for multi-label classification. ICCV, 2021.\n\n**Q5: Model efficiency analysis.**\n\n**A5**：Following your suggestion, we conducted a detailed comparison of GroupHOI with other HOI methods in terms of Parameters (Params), Floating Point Operations (FLOPs), and Frames Per Second (FPS) as follows:\n\n| Method              | Params↓   | FLOPs↓     | FPS↑      | Full (DF) |\n| ------------------- | --------- | ---------- | --------- | --------- |\n| PPDM [CVPR20]       | 194.9M    | 121.63G    | 15.28     | 21.10     |\n| GEN-VLKT [CVPR22]   | 41.9M     | 60.04G     | 26.18     | 33.75     |\n| HOICLIP [CVPR23]    | 66.1M     | 104.68G    | 19.57     | 34.59     |\n| ViPLO [CVPR23]      | 118.2M    | 41.35G     | 14.89     | 34.95     |\n| **GroupHOI (Ours)** | **79.2M** | **83.67G** | **16.42** | **36.70** |\n\nwhere Full (DF) represents the results on Default evaluation setting of  HICO-DET. Compared to HOICLIP [ref11], which shares the same network architecture as GroupHOI except for the GEO and SEM modules, GroupHOI introduces a slight increase in parameter count and a marginal reduction in FPS, yet achieves a substantial performance improvement, surpassing HOICLIP by +2.11 mAP on the Full setting of HICO-DET. Moreover, when compared with ViPLO [ref12], a graph-based method, under the same conditions, GroupHOI delivers superior accuracy while using fewer parameters and achieving faster inference, highlighting the efficiency of our model design (see Q1 **Implementation**). We will supplement Sec. 4.3 with this analysis in the revised manuscript.​\n\n[ref11] Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. CVPR, 2023.\n\n[ref12] Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. CVPR, 2023.\n\n**Q6: The results for V-COCO and HICO-Det are confused**.\n\n**A6**：We apologize for this mistake. It has been corrected in the revision."}, {"review_id": "5o7ct4yFm6", "reviewer": "Reviewer_H8t4", "summary": "This paper introduces GroupHOI to enhance human-object interaction (HOI) understanding by modeling relationships from a group perspective rather than pairwise ones. GroupHOI is built upon two grouping principles:\n- Proximity principle: This principle suggests that individuals tend to form groups with those they are physically close to. In GroupHOI, a learnable proximity estimator is introduced to measure this. Contextual cues within these groups are aggregated and update the entity embeddings.\n- Semantic similarity principle: This principle suggests that interactions sharing similar visual or behavioral patterns tend to form semantic groups. GroupHOI implements this by forming clusters of interaction proposals (i.e., human-object pairs) based on cosine similarity of their feature embeddings. These groups are then used to provide semantic context to the interaction reasoning process, and local features within bounding boxes are also incorporated.\n\nThe framework adopts a DETR-based cascade architecture, in line with other recent state-of-the-art approaches.\n\nGroupHOI demonstrates performance improvements over existing methods on standard HOI benchmarks, including HICO-DET and V-COCO.", "questions": "- From the examples shown in Figure 4, it appears that some nearest neighbors in the geometric grouping may actually refer to the same entity due to overlapping detection boxes (e.g., before non-maximum suppression is applied). In such cases, how often do self-matches occur when selecting the top K_g nearest entities? Do you feel necessary to address the self-matches?\n\n- Moreover, since object instances can vary significantly in category, size, and spatial layout, why use fixed neighbors? Would it be more flexible to allow the learnable proximity estimator to determine the appropriate number of neighbors dynamically, rather than using a fixed top-k scheme?\n\n- For the geometric group construction, how is the proximity score S_{ij} trained? What kind of supervision signal is used to guide the learning of this estimator? Is there a loss that explicitly supervises the group formation process, or is it trained in an end-to-end manner solely based on the final HOI loss in Eq 9?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is clearly written and well-organized. The authors clearly articulate the motivation for introducing grouping in HOI tasks and provide clear explanations of their corresponding design choices. The overall presentation reflects strong experience in writing research papers.\n\n- The central idea of modeling relationships from a group perspective is technically sound. The proposed solution is coherent and conceptually aligned with the proposals.\n\n- The experimental results, including ablations, are comprehensive and provide convincing support for the proposed method. Overall, the paper slightly exceeds my acceptance threshold.\n\n**Weaknesses**\n\n- Overall, I think the selected datasets such as HICO-DET and V-COCO may not be ideal for showcasing the benefits of group-based relationship modeling. These datasets have been used extensively for years, and most recent works report only marginal improvements. It's unclear how much of the remaining error is attributable to the absence of group-level reasoning, as opposed to dataset-specific limitations such as annotation bias, limited scale, or inherent noise. Additionally, the majority of interactions in these datasets likely involve single human–single object pair, which may limit the opportunity for group modeling techniques to demonstrate clear advantages.\n\n- A more compelling direction could be for the authors to curate or design a new benchmark that emphasizes scenarios requiring group-level reasoning, for example, interactions involving multiple humans and objects with interdependent roles. Alternatively, the authors could consider filtering or reorganizing the existing datasets to introduce a new split of complex multi-entity interactions, going beyond the typical rare/non-rare splits.", "was_revised": true, "final_justification": "The authors have provided answers for my questions to clarify my misunderstandings.", "rebuttal": "We thank reviewer H8t4 for the valuable time and constructive feedback. We provide point-to-point response below.\n\n----\n\n**Q1: HOI-DET and V-COCO may not be ideal for group-based relationship modeling.**\n\n**A1**: Thanks for your thoughtful suggestion! Although HICO-DET and V-COCO contain abundant single human-object interaction instances (e.g., a photo of a person sitting on a chair), we find that there is still a significant proportion of samples that contain multiple human-object pairs via statistical analysis. Specifically, HICO-DET demonstrates an average of four interactions per image, while V-COCO contains three interactions per image. However, to address your concern, **we have supplemented our experiments with results on NVI datasets [ref1], which encompasses complex interactions involving multiple humans** (see Q2). Additionally, we strongly agree with your perspective and intend to develop a new visual relationship benchmark for group-based relationship modeling in our future work.\n\n[ref1] Nonverbal Interaction Detection. ECCV, 2024.\n\n\n**Q2: Design or adopt a benchmark for group-level reasoning.**\n\n**A2**: We appreciate the insight! We conduct additional experiments on NVI datasets [ref2], which encompasses complex interactions involving multiple humans. These experiments incorporate the Nonverbal Interaction Detection (NVI-DET) task, designed to localize individual entities and their corresponding social groups while identifying the category of its nonverbal interaction. The experimental results on NVI val are as follows:\n\n| Method              | mR@25     | mR@50     | mR@100    | AR        |\n| :------------------ | :-------- | :-------- | :-------- | :-------- |\n| m-QPIC              | 56.89     | 69.52     | 78.36     | 68.26     |\n| m-CDN               | 55.57     | 71.06     | 78.81     | 68.48     |\n| m-GEN-VLKT          | 50.59     | 70.87     | 80.08     | 67.18     |\n| NVI-DEHR [ECCV24]   | 54.85     | 73.42     | 85.33     | 71.20     |\n| **GroupHOI (Ours)** | **55.67** | **76.73** | **87.16** | **73.19** |\n\nThese results demonstrate that our method also generalizes well to more complex interaction scenarios. We will  add Sec. 4.4 for the analysis of results on NVI dataset [ref2]  in the revised manuscript.\n\n[ref2] Nonverbal Interaction Detection. ECCV, 2024.\n\n\n**Q3: How often do self-matches occur, and should they be addressed?**\n\n**A3**: As you noted, self-match can occur in both geometric and semantic groups (see Fig. 4). However, it is difficult to quantify the frequency of self-matches in the groups formulated by GroupHOI: with a fixed number of proposals (see L217) in our method, self-matches increase as detectable entities in an image decrease, and decrease as detectable entities increase. This scenario is not considered necessary to address in our method for the following reasons:\n\n- Detection boxes typically exhibit deviations from ground truth ones; therefore, the overlapping detection boxes corresponding to the same entity can mutually complement their information by self-matches.\n- For inputs with limited detectable entities (e.g., images only containing single human-object interaction), the groups formulated by GroupHOI are dominated by self-matches, which enables our method to maintain certain performance (albeit weaker) under such constraints.\n\n\n**Q4: Why adopt fixed neighbors？Is it possible to train an estimator for neighbor count assessment?**\n\n**A4**: My apologies for the misunderstanding. Both our geometry-aware local transformer and semantic-aware interaction decoder employ a three-layer Transformer decoder architecture. For each decoder layer, **we construct a new geometric/semantic group**. This strategy enables the total number of neighbors for each proposal (i.e., human, object, and interaction proposal) ranging from $K_g$ to $K_g\\times3$ (**4 to 12**) for the geometric group and from $K_s$ to $K_s\\times3$ (**2 to 6**) for the semantic group. Notably, over 96% of samples in V-COCO and over 94% in HICO-DET contain no more than 12 entities, so geometric group of 4 to 12 can cover the majority of scenarios. Meanwhile, the size of semantic group used in the paper also aligns with [ref3]'s finding, which underscores that the majority of the interactive dialogue involving 10 or more participants is produced by only the top 4-5 contributors. Hence, we deem it unnecessary to train an estimator for neighbor count assessment, as this could introduce instability or optimization challenges. The above discussion will be incorporated into Sec. 3.5 and 4.3. \n\n[ref3] Group discussion as interactive dialogue or as serial monologue: The influence of group size. Psychological science, 2000.\n\n**Q5: How is the proximity score trained and supervised?**\n\n**A5**: Our apologies for the lack of clarity. The linear layer, used to compute the proximity score by fusing the centroid distance and IoU between two bounding boxes, is not explicitly supervised but is trained in an end-to-end manner based on the final HOI loss. To clarify, the manuscript will be revised in Sec. 3.3 as follows:\n\n> A simple linear layer is then employed to compute the proximity score $s_{i,j}$ based on the central distance $dis_{i,j}$ and the Intersection-over-Union $IoU_{i,j}$ between bounding boxes, without requiring additional supervision. This results in a proximity score matrix, where lower scores indicate closer spatial proximity. For each entity, we select the $K_g$ neighbors  with the lowest scores to construct its geometric neighbor set $N^g_i$."}, {"review_id": "jiAp6MwOcx", "reviewer": "Reviewer_PSyr", "summary": "This paper proposes a method for Human-Object Interaction detection that explicitly models interactive relationships within a group. To exploit these relationships, the authors propose constructing geometric and semantic groups based on geometric proximity and semantic similarity. A modified transformer architecture is also introduced to perform feature aggregation and distribution within each group. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "- How does the learnable estimator ensure the accuracy of group classification?\n- How does the semantic group account for category differences?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Considering the relationships within groups to promote HOI is an effective and intuitive idea.\n- The authors exploit the principle of proximity to form groups, which adheres to real-world human behavior and is conceptually sound.\n\nWeaknesses\n- The author proposes a learnable estimator to measure the proximity among bounding boxes. Does this module require supervision? Without supervision, how can the MLPs distinguish whether two entities belong to the same group? How much improvement can be achieved by using MLPs over conventional distance-based calculations for estimating group? Additionally, the MLPs only take distance and IoU as inputs, which may not be sufficient to determine whether two entities belong to the same group—for example, two people who are far apart in depth might still have overlapping bounding boxes. Therefore, should the input also incorporate image features?\n- The group sizes within the same image can vary significantly, but this paper does not account for the variation in group size. Simply setting the geometric and semantic group numbers to 4 and 2 does not reflect the actual variability in real-world scenarios.\n- There are previous works that have explored leveraging intra-group information to understand interactions. They also form groups based on proximity and similarity, and their structures are not strictly limited to fully-connected or bipartite graphs. Although these works do not specifically address the HOI problem, they share many similarities with this work in terms of human group construction, and thus, I believe they should be discussed as well.\n\n     [A] \"Reconstructing groups of people with hypergraph relational reasoning.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n     [B] \"Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n- In the Semantic Group, there may be significant differences between human and object features. When computing feature similarity, is it possible that only humans are grouped together while objects are excluded from the group?", "was_revised": true, "final_justification": "All of my concerns have been addressed. I hope the additional experiments and discussions can be included in the final version of the paper.", "rebuttal": "We thank reviewer PSyr for the valuable time and constructive feedback. We provide point-to-point response below.\n\n***\n\n**Q1: Does the learnable proximity estimator require supervision? If not, how does it distinguish groups?**\n\n**A1**: Apologies for the confusion. The linear layer, used to compute the proximity score by fusing the centroid distance and IoU between two bounding boxes, is not explicitly supervised but is trained in an end-to-end manner based on the final HOI loss. The proximity score measures the spatial closeness between two detection boxes, where a smaller value indicates closer proximity. We construct the neighbor set by selecting the $K_g$ detection boxes with the smallest proximity scores. To clarify, the manuscript will be revised in Sec. 3.3 as follows:\n\n> A simple linear layer is then employed to compute the proximity score $s_{i,j}$ based on the central distance $dis_{i,j}$ and the Intersection-over-Union $IoU_{i,j}$ between bounding boxes, without requiring additional supervision. This results in a proximity score matrix, where lower scores indicate closer spatial proximity. For each entity, we select the $K_g$ neighbors  with the lowest scores to construct its geometric neighbor set $N^g_i$.\n\n**Q2: What is the performance gain of MLPs over distance-based methods?**\n\n**A2**: In response to your suggestion, we have added the ablation experiments, as:\n\n| Method              | Full (DF) | Rare (DF) | Non-Rare (DF) |\n| ------------------- | --------- | --------- | ------------- |\n| GroupHOI (IOU only) | 36.02     | 32.19     | 37.16         |\n| GroupHOI (CD only)  | 36.14     | 33.60     | 36.90         |\n| **GroupHOI (Ours)** | **36.70** | **34.86** | **37.26**     |\n\nwhere (IoU only) and (CD only) refer to using the IoU and center distance for group construction, respectively. Results demonstrate the linear fusion layer employed in GroupHOI provides a significant performance advantage.\n\n**Q3: Are distance and IoU sufficient for grouping, or should image features be included?**\n\n**A3**: Thank you for the insightful suggestion. We agree that incorporating depth maps [ref1] could be a promising direction to address the limitation you identified. However, it would introduce additional supervision and computational overhead. Moreover, it could lead to an unfair comparison with existing methods that rely solely on 2D visual cues.\n\nFollowing your suggestion, we also conducted additional experiments utilizing image features:\n\n| Method                    | Full (DF) | Rare (DF) | Non-Rare (DF) |\n| :------------------------ | :-------- | :-------- | :------------ |\n| GroupHOI (image features) | 36.21     | 33.45     | 36.97         |\n| **GroupHOI** **(Ours)**   | **36.70** | **34.86** | **37.26**     |\n\nFor GroupHOI (image features), we incorporate image features into the computation of the proximity score by concatenating them with two geometric metrics. However, its results showed no significant improvements. We hypothesize that the image features in the HOI detector are also insensitive to depth, as the labels do not provide such supervisory signals.\n\n[ref1] Improving visual relation detection using depth maps. ICPR, 2021.\n\n**Q4:  Fixed group size.**\n\n**A4**: We acknowledge the confusion and apologize. Both our geometry-aware local transformer and semantic-aware interaction decoder employ a three-layer Transformer decoder architecture. For each decoder layer, **we construct a new geometric/semantic group**. Through this strategy, each proposal (i.e., human, object, and interaction proposal) can aggregate contextual information from $K_g$ to $K_g\\times3$ (**4 to 12**) for the geometric group and from $K_s$ to $K_s\\times3$ (**2 to 6**) for the semantic group. Empirical statistics show that over 96% of samples in V-COCO and over 94%  in HICO-DET contain no more than 12 entities, ensuring that our geometric grouping strategy sufficiently covers the majority of scenarios. The semantic group design is motivated by [ref2], which observes that even in large-scale interactive settings, meaningful exchanges are often dominated by a small subset of participants (4–5 key contributors), aligning with our design. The above discussion will be incorporated into Sec. 3.5 and 4.3. \n\n[ref2] Group discussion as interactive dialogue or as serial monologue: The influence of group size. Psychological science, 2000.\n\n\n**Q5: Discussion about related prior group-based methods.**\n\n**A5**: Thank you for your valuable suggestion. We agree that these works share similarities with our work and will add a discussion of them in Sec. 2 as follows: \n\n> Recent studies in related domains have also explored group-wise relational modeling based on proximity and similarity. For example, [ref3] addresses 3D human mesh recovery in crowded scenes using a multiscale hypergraph to capture both individual and group-level relations. Similarly, [ref4] focuses on multi-agent trajectory prediction by learning multiscale hypergraphs to represent pairwise and group-wise interactions. These works  share the idea of modeling intra-group relations with spatial or semantic cues. However, they simply combine proximity and similarity information for relation reasoning (e.g., [ref3] introduces human spatial information through box coordinates). In contrast, our method explicitly constructs and learns relationships from both geometric and semantic perspectives through two distinct graphs: a geometric graph based on the spatial proximity of entities, and a semantic graph built on the semantic similarity of interactions. This design enables more fine-grained relation structuring tailored to HOI scenarios.\n\n[ref3] Reconstructing groups of people with hypergraph relational reasoning. ICCV, 2023.\n\n[ref4] Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. CVPR, 2022.\n\n\n**Q6: Semantic Group for human and object features.**\n\n**A6**: We apologize for the confusion. We construct geometric groups exclusively for human and object proposals (see L162-172), and semantic groups exclusively for interaction proposals (see L192-197) initialized by computing the mean of human and object proposals. Additionally, in Sec. 4.3, we empirically observed that distinguishing human and object proposals is necessary for constructing geometric groups (see Lines 278–281, Table 5c)."}, {"review_id": "Qz9Xt1x5Ta", "reviewer": "Reviewer_8R1p", "summary": "This paper introduces GroupHOI for Human-Object Interaction (HOI) detection that models interactions not just as pairs but as groups, inspired by social grouping principles. The method forms geometric groups based on spatial proximity and semantic groups based on feature similarity, leveraging both for improved contextual reasoning. GroupHOI integrates these principles into a DETR-based architecture, enhancing interaction prediction by propagating information within learned groups. Experiments on the HICO-DET and V-COCO benchmarks show GroupHOI achieves state-of-the-art results.", "questions": "Why does the GroupHOI set the size of the geometric and semantic groups to 4 and 2, while the example in Figure 4 uses a size of 3? Does this mean that group sizes of 4 and 2 may not be the most appropriate for some images?", "limitations": "See weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The joint geometric and semantic perspective to model complex relationships between entities and propagate contextual information across them is both innovative and interesting.\n2. The paper defines two visual principles for constructing geometric and semantic groups, and effectively integrates these principles into HOI detection.\n3. Overall writing of this paper is good and easy to understand and follow.\n\nWeaknesses:\n1. The authors did not provide evidence to demonstrate that the proposed *geometric and semantic perspective groups* can address the current challenges faced by existing HOI methods. It remains unclear whether these groups can cover all possible HOI scenarios, and whether the method is effective in handling cases that do not fit the situations described in the paper.\n2. The size of geometric and semantic groups are fixed. Since the content of each image varies, this fixed number of sizes conflicts with the assumptions made in the paper.\n3. Although the model achieves SOTA performance, the source of the improvement is unclear. As shown in Table 2(a), the baseline without GEO and SEM already outperforms CMMP and LOGICHOI. While it is undeniable that GroupHOI improves upon the baseline, Table 2(a) suggests that the performance gain of GroupHOI may stem more from other factors rather than the GEO and SEM modules.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns. However, since the proposed method GroupHOI is built upon a modified and stronger version of HOICLIP, the reported relative improvements appear somewhat overstated. As a result, I still consider this paper to be at the borderline for acceptance.", "rebuttal": "We thank reviewer 8R1p for the valuable time and constructive feedback. We provide point-to-point response below.\n\n***\n\n**Q1: Insufficient Evidence for Addressing HOI Challenges**\n\n**A1**: Respectively disagree. We would like to clarify that our work have clearly delineated the limitations of existing HOI methods and demonstrated how our proposed approach specifically addresses these challenges:\n\n- **Challenge**. We explicitly define the challenge addressed in this paper as **how to model the complex relationships between entities and propagate contextual information across them?** (see L32-33).\n\n- **Limitation**. ​​For this challenge, current methods remain confined to the modeling of predefined pairwise relations, leaving the inherent collective patterns (i.e., multiple entities engaging in joint activities) between entities unexplored (see L34-40).\n\n- **Motivation**. To tackle this limitation，we revist HOI detection from a group perspective, i.e., structuring and organizing visual entities by finding groups of similar ones, formulating the challenge as two foundamental issues: (1) **how is a group formed?** (2) **how does a group function?** (see L41-50).\n\n- **Technical Solution**. For group formulation, we propose two grouping strategies based on geometric proximity and semantic similarity principles (see L51-60). For group functioning, we introduce a new architecture for HOI detection involving these two grouping strategies (see L61-72). \n\n- **Experiment**. ​​The superior performance of GroupHOI on V-COCO and HICO-DET demonstrates its ability to address the core challenge targeted by this study (see L76-78). \n\nOverall, we argue that our geometric proximity and semantic similarity groups can effectively address the challenge raised by this study.\n\n**Q2:  Whether these groups can cover all possible HOI scenarios, and whether the method is effective in handling cases that do not fit the situations described in the paper.**\n\n**A2**: It seems a little unfair that requires our method to cover all possible HOI scenarios, as no existing dataset provides comprehensive coverage for such validation. However, in response to your concern, we introduce additional validation on NVI dataset [ref1] which incorporates complex interactions involving arbitrary numbers of humans ranging from 2 to 12. Following [ref1], we report mR@25, mR@50, and mR@100 under different IoU thresholds. The experimental results on NVI val are as follows:\n\n| Method              | mR@25     | mR@50     | mR@100    | AR        |\n| :------------------ | :-------- | :-------- | :-------- | :-------- |\n| m-QPIC              | 56.89     | 69.52     | 78.36     | 68.26     |\n| m-CDN               | 55.57     | 71.06     | 78.81     | 68.48     |\n| m-GEN-VLKT          | 50.59     | 70.87     | 80.08     | 67.18     |\n| NVI-DEHR [ECCV24]   | 54.85     | 73.42     | 85.33     | 71.20     |\n| **GroupHOI (Ours)** | **55.67** | **76.73** | **87.16** | **73.19** |\n\nAs seen, GroupHOI surpasses all these methods, reaching new sota for NVI-DET task. This result demonstrates that our model is applicable to more complex interaction scenarios.\n\n[ref1] Nonverbal Interaction Detection. ECCV 2024.\n\n\n**Q3: Why is the group size fixed?**\n\n**A3**: Sorry for this confusion. Both the geometry-aware local transformer and the semantic-aware interaction decoder adopt a three-layer Transformer decoder architecture. In each decoder layer, a distinct geometric or semantic grouping is established to model contextual relationships. This strategy enables the total number of neighbors for each proposal (i.e., human, object, and interaction proposal) ranging from $K_g$ to $K_g\\times3$ (**4 to 12**) for the geometric group and from $K_s$ to $K_s\\times3$ (**2 to 6**) for the semantic group. Notably, over 96% of samples in V-COCO and over 94% in HICO-DET contain no more than 12 entities, so geometric group of 4 to 12 can cover the majority of scenarios. Meanwhile, the size of semantic group aligns with [ref2]'s finding, which underscores that the majority of the interactive dialogue involving 10 or more participants is produced by only the top 4-5 contributors. The above discussion will be incorporated into Sec. 3.5 and 4.3. \n\n[ref2] Group discussion as interactive dialogue or as serial monologue: The influence of group size. Psychological science, 2000.\n\n**Q4: Source of baseline's performance gain**\n\n**A4**: We apologize for the omission of implementation details. Our baseline is built upon HOICLIP [ref3], with its performance primarily attributed to the following modifications:\n\n- **Training strategy**. We maintain the same training duration (​​90 epochs​​) as HOICLIP. However, our initial learning rate is set to $5\\times10^{-5}$​​, decaying by a factor of ​​10 after every 30 epochs​​. In contrast, HOICLIP uses an initial learning rate of $1\\times10^{-4}$​, reducing it by a factor of ​​10 only after 60 epochs​​.\n- **HOI/object classifier**. Unlike HOICLIP, we do not freeze the parameters of the HOI/object classifiers initialized from CLIP, allowing for further optimization during training.\n- **Interaction decoder**. Due to the dimension mismatch, HOICLIP omit positional embeddings (256) in the interaction decoder (768). Instead, we expand the positional embeddings to 768 by stacking them three times for the interaction decoder.\n- **HOI loss**. To better address the long-tail distribution issue, we replace the original Focal loss with Asymmetric Loss [ref4] for HOI loss computation. \n\nThe aforementioned information will be included in the Appendix Sec. A.\n\n[ref3] Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. CVPR 2023.\n\n[ref4] Asymmetric loss for multi-label classification. ICCV 2021.\n\n**Q5: Examples in Fig. 4.**\n\n**A5**: Apologies for the confusion. As stated in **Q3**, we construct a new geometric/semantic group at each interaction decoder layer, yielding geometric group sizes of **4-12** and semantic group sizes of **2-6**. Thus, to facilitate presentation, we select three entities from the group for visualization. We apologize for the ambiguity in our previous statement. To clarify, we will update the visualization in Fig. 4 to illustrate the evolving group composition at each layer in the revised manuscript."}]

Author rebuttal:

Meta-review: This paper proposes GroupHOI, a novel framework for Human-Object Interaction (HOI) detection that revisits relation modeling from a group perspective. Instead of focusing on pairwise relationships, it propagates contextual information based on two principles: geometric proximity and semantic similarity.

The key strength of this work is its innovative and intuitive approach to modeling HOI through group dynamics, which is technically sound and well-motivated. The proposed geometric and semantic grouping mechanisms are effectively integrated into a strong baseline, achieving state-of-the-art results on the complex HICO-DET and V-COCO benchmarks.

Initial weaknesses included questions about novelty compared to prior graph-based methods, the need for more SOTA comparisons, and justification for certain design choices like fixed group sizes. During the rebuttal, the authors provided thorough responses, including additional experiments, efficiency analysis, and clarifications that successfully addressed most of the reviewers' concerns. Although the reported contributions appear somewhat overstated as concerned by reviewer 8R1p, other reviewers raised their scores, acknowledging the authors' detailed feedback. 

After careful consideration, the AC recommends accepting this paper.
Decision: Accept (poster)