Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction Detection
Authors: Dongkeun Kim
Minsu Cho
Suha Kwak
Venue: NeurIPS 2025 poster ()
Keywords: Social interaction understanding
Nonverbal interaction detection
Primary area: applications

Abstract: Social interactions often emerge from subtle, fine-grained cues such as facial expressions, gaze, and gestures. However, existing methods for social interaction detection overlook such nuanced cues and primarily rely on holistic representations of individuals. Moreover, they directly detect social groups without explicitly modeling the underlying interactions between individuals. These drawbacks limit their ability to capture localized social signals and introduce ambiguity when group configurations should be inferred from social interactions grounded in nuanced cues. In this work, we propose a part-aware bottom-up group reasoning framework for fine-grained social interaction detection. The proposed method infers social groups and their interactions using body part cues and their interpersonal relations. Our model first detects individuals and enhances their features using part-aware cues, and then infers group configuration by associating individuals via similarity-based reasoning, which considers not only spatial relations but also subtle social cues that signal interactions, leading to more accurate group inference. Experiments on the NVI dataset demonstrate that our method outperforms prior methods, achieving the new state of the art, while additional results on the Café dataset further validate its generalizability to group activity understanding.

Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction Detection
Dongkeun Kim
Minsu Cho Suha Kwak
Pohang University of Science and Technology (POSTECH) (kdk1563, mscho, suha. kuhba.kwwk@@ppsteeeo ac
Abstract
Social interactions often emerge from subtle, fine-grained cues such as facial expressions, gaze, and gestures. However, existing methods for social interaction detection overlook such nuanced cues and primarily rely holistic representations individuals. Moreover, they directly detect social groups without explicitly modeling the underlying interactions between individuals. These drawbacks limit their ability to capture localized social signals and introduce ambiguity when group configurations should be inferred from social interactions grounded in nuanced cues. In this work, we propose a part-aware bottom-up group reasoning framework for fine-grained social interaction detection. The proposed method infers social groups and their interactions using body part features and their interpersonal relations. Our model first detects individuals and enhances their features using part-aware cues, and then infers group configuration by associating individuals via similarity-basee reasoning, which considers not only spatial relations but also subtle social that signal interactions, leading to more accurate group inference. Experiments on the NVI dataset demonstrate that our method outperforms prior methods, achieving the new state of the art, while additional results on the Café dataset further validate generalizability to group activity understanding.
Introduction
Understanding the intentions and behaviors of others is fundamental aspect of human perception and social intelligence. In our daily lives, we rely not only on what people say, but also on how they appear, move, and behave to make sense of others. Understanding such interactions enables broad applications, including surveillance, human-robot interaction, and sports analysis. Social interaction understanding encompasses wide range of tasks, including group activity recognition [13, 19, 23, 30, 54, 56]. pedestrian trajectory prediction [1, 2, 37, 52] and group activity detection [10, 11, 24, 45]. While these tasks have advanced the ability to model social interactions, most existing work primarily focuses on group behaviors inferred from coarse visual cues such as appearances, actions, or geometric configurations of group members. Consequently, they often overlook fine-grained social interactions, such as identifying whether two people are smiling at each other, engaging in mutual gaze, or performing gestures like pointing, that are essential for nuanced social perception. The ability to understand such fine-grande lineggrined social interactions is crucial for inferring intent, emotion, and social relationships, particularly when verbal communication limited or absent.
Recently, a new task of detecting multi-person interactions based on such fine-grained, nuanced, and ambiguous social cues has been introduced along with dedicated dataset, NVI [51]. The task, named nonverbal interaction detection (NVI-DET), is formulated to detect each individual, identify the social group they belong to, and classify their fine-grained social interaction, encapsulated <individual, group, interaction This formulation draws inspiration from human-object interaction (HOI) detection [31, 44, 58], but differs in that it targets human-human social interactions, including facial expression, gesture, posture, gaze, and touch, that are inherently more nuanced and
triplet
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

relational. The task poses unique challenges, requiring both accurate localization of individuals and the interpretation of subtle cues that define social groups and their interactions.
Previous approaches [31, 51] attempt to solve this task by utilizing transformer [5, 47] and hyper- graph [61] to capture high-order interactions among individuals and social groups. While these approaches are effective in capturing social interactions among individuals and groups, they have several drawbacks as follows. First, these methods directly detect social groups without explicitly modeling the underlying person-to-person relations. This design overlooks a fundamental principle of social interaction detection: both social interactions and social group composition should emerge from individual behaviors and interactions between individuals. Predicting a group without accounting for its members introduces ambiguity and uncertainty, especially when subtle interactions like gaze occur between individuals who are spatially distant. Second, most existing methods embed each person as holistic representation, neglecting body parts information that are essential for recognizing fine-graines social interactions. As result, they struggle to distinguish interactions seemingly alike but holding different semantics stemming from subtle visual cues, such as 'mutual gaze" versus 'gaze following' or "wave" versus 'point,' that require fine-grained social reasoning.
To address these limitations, we propose part-aware bottom-up group reasoning model for fine- grained social interaction detection. The core idea of our model is to infer group composition and interactions by reasoning from fine- finegrainedd body part cues to relations between individuals. Instead of directly detecting both individuals and social groups, we first detect individuals and construct group representations by aggregating individual-level information based on learned similarities. allowing social groups to naturally emerge from the interactions between individuals. Moreover, our model enriches individual embeddings by incorporating part-aware features, which are learned under the guidance of human pose estimation as privileged information [34] to attend specifically to different body parts such as face, arms, and legs. These representations encode localized body part cues such as facial expression, hand gesture, and posture, that are crucial for recognizing fine-grained social interactions. The proposed group reasoning framework reflects the compositional nature of social interactions and enables the model to detect fine-grained social interactions with improved performance.
We evaluated the proposed method on the [V] and Café [24], where it demonstrated substantial improvements over existing methods. In summary, our contribution is three-fold as follows:
We introduce part-aware representation learning that leverages pose-guided supervision as privileged information to capture fine-grained social cues for improving interaction recognition.
We propose bottom-up group reasoning framework that infers social groups based part-aware individual representations, rather than directly detect social groups. This design ensures that group composition naturally emerges from individuals.
The proposed method outperforms existing approaches on NVI and Café, demonstrating the effectiveness of incorporating body part representations and bottom-up group reasoning for fine-grained social interaction detection.
2
Related work
2.1 Social interaction understanding
Fine-grained social interaction recognition. Fine-grained social interaction recognition plays a central role in interpreting intent, behavior, and social dynamics [3]. Recent advances in computer vision have explored fine-grained social cues that are subtle yet essential for human communication. including gaze, facial expressions, and gestures. Gaze analysis has been a representative line of work in fine-grained social interaction understanding, which aims to localize where individuals direct their attention [6, 7, 15, 20, 43). Specifically, MTGS [15] introduces a unified dataset for multi perisopeson gaze following and social gaze prediction, while Tafasca et al. [43] extend the gaze following task by jointly predicting both the location and semantic label of the gaze target. Facial
expression [22, 50, 60), gesture [14, 38, 59] and posture [9] analysis form another line of research that targets specific types of fine-graine me-ggainedd social interactions. While these methods have shown success, they are typically developed on specialized datasets that focus specific signals.

---

Group activity understanding. Group activity recognition (GAR) has long been studied as representative group activity understanding task, aiming to classify collective activity exhibited by groups of people. Typical GAR approaches model spatio-temporal relations between individuals utilizing graph neural networks [54, 56, 57] or transformers [13, 23, 30). However, these methods are built on the assumption of a single-group setting, which restricts their applicability to more complex, real-world scenarios. Beyond recognizing a single group activity, some work tackles multi-group scenarios in the form of social activity recognition [11,45 56] or group activity detection [24], which aims to localize multiple groups and classify the activity performed by each group. While group activity detection shares similarity with NVI-DET in that it localizes multiple social groups in scene, most of these methods treat individuals as holistic units and focus on activity- classification, overlooking fine- fine-grained social cues.
Nonverbal interaction detection. Fine-grained social signals are a rich yet under-explored cue for understanding social interactions [48]. Wei et al. [51] introduce NVI-DET task, which seeks to detect grainnd social interactions by identifying triplets (individual wtw.paaaa.ggg.....e.tttttttti.aa interaction) in image. This formulation enables unified reasoning over individual and social interaction through visually grounded nonverbal cues. The accompanying NVI dataset includes annotations for five interaction categories: facial expression, gesture, posture, gaze, and touch. The proposed model, NVI-DEHR, leverages hypergraphs [61] to model high-order individual-6 cosuual-to-iidne costuual----niiiduuut group-to-group relations, improving its ability to recognize tin-grained social interactions. However, NVI-DEHR detects social groups directly without explicitly modeling inter- person relations a inter-person
individual and
limitation that becomes particularly problematic when detecting interactions like gaze, which often occur between spatially distant individuals. Moreover, it relies on holistic person representations, thereby overlooking body part-level cues that essential for distinguishing visually similar but semantically distinct interactions (e.g., wave vs. point, or mutual gaze vs. gaze following). Unlike this approach, our method adopts hierarchical reasoning strategy that first extracts part-aware individual representations and then infers social groups and their interactions based on inter-personal relations guided by ine-grained body part cues.
2.2 Human-object interaction detection
58], which predicts (human, object, interaction) triplets, is closely related to NVI-DET. Indeed, HOI-DET and NVI-DET share structural similarities in their problem formulation and modeling strategies, such as the use of set-based prediction and relational reasoning. However. HOI-DET primarily considers pairwise relations between a single human and an object whereas NVI-DET requires group- reasoning among multiple humans. Meanwhile, recent HOI-DET methods explore fine-grained reasoning to improve interaction understanding. For instance, Wan et al. [49] utilize an off-the- shelf pose estimator to zoom into relevant body parts. Wu et al. [53] incorporate pose cues to better capture spatial configuration between humans and objects, highlighting the benefit of human prior in interaction modeling. Lei et al. [28] leverages large language model (LLM) to reason over body-part contexts, enabling the model to associate specific interaction types with relevant body regions. However, unlike these methods, our model leverages human pose only for training, as privileged information [34]. This design allows the model to benefit from fine-grained supervision while maintaining efficient inference without requiring additional inputs.
Human-object interaction detection (HOI-DET) [4, 12, 16, 21, 25, 29, 31, 36, 44,
3 Proposed method
We propose a part-aware bottom-up group reasoning framework for fine-grained social interaction detection. The core idea of our framework lies in the bottom-up inference of group configurations and interactions, by leveraging pose-guided part-aware representations and modeling their interpersonal relations. This section describes the problem formulation of NVI-DET (Sec. 3.1), overall architecture of the proposed model (Sec. 3.2), training objectives (Sec. 3.3), and inference procedure (Sec. 3.4).
3.1 Problem formulation
The task of fine-grained social interaction detection [51] aims to recognize interactions of each individual with their respective social groups. Given an input image X E Hooo Xax3, the goal is to predict set of triplets y {(b., gi, b, E R g e IR c, E Ne e [I, where each
3

---

Positional encoding
+ bets
Predictions
Elerment wise addition
Cls. hcad
Group
Concatenation
Decoder
Loc. head
Triplet NMS
Individual Emb. Enhancer
Association module
Lpart
Backbone
Lassn
Lind
Transformer
Individual
Cls. head
Encoder
Decoder
Loc. head
Feature extraction
Bottom-up group reasoning
Similarity-based association
Figure I: Overall architecture of the proposed framework. Given an input image, our model extracts the visual features using the backbone and the transformer encoder. It then detects individuals and derives part-aware individual features through the individual embedding enhancer. Group queries attend both the encoded feature maps and part-aware individual embeddings infer social groups and interaction labels. Finally, triplets are obtained through the association module and NMS.
triplet (b,, gi; c:) consists of an individual bounding box b,, the corresponding group bounding box gi. and interaction probabilities c; over Nc predefined interaction classes. The interaction proba- bilities indicate the presence of each interaction for each detected group. Notably, this formulation allows single individual to participate in multiple concurrent social groups and interactions, similar to HOI-DET [16].
3.2 Model architecture
We introduce a part-aware bottom-up framework that detects social interactions by progressively capturing subtle body part cues and leveraging them to infer social groups and their fine-grained interactions. Fig. illustrates the overall pipeline of our model. Unlike prior methods that directly regress group bounding boxes and rely solely holistic human features, method first detects individuals and enriches their features with body part cues to ground ane-grained social cues, and then infers group association and interactions through part-aware reasoning.
Feature extractor. Given an input image X E R Hox Wox3, we extract feature map using a ResNet [17] backbone pretrained on ImageNet [8], followed by standard transformer encoder [5, 47] composed of L layers of multi-hear head self-attention and feed-forward network (FFN). To align the dimension of the feature map with the dimension of the transformer encoder, a linear projection applied before feeding the features into the encoder. The resulting encoded feature map F e R H x enriches the visual features for subsequent reasoning modules.
Individual decoder. Unlike prior work [51], which predicts both individual and group bounding boxes simultaneously, our individual decoder is dedicated solely detecting individuals in the image, We argue that directly predicting a group bounding box without modeling the interactions among individuals is often ambiguous and counterintuitive. For instance, social groups engaged in mutual gaze or pointing may be spatially distant, and naively predicting a bounding box around them can inadvertently include unrelated people. To address this, adopt sequential approach: detect individuals and subsequently infer groups based on their interactions. The individual decoder adopts the standard transformer decoder architecture of DETR [5]. It employs set of N, learnable queries Q E IR xD which attend to the encoded image features to produce individual embeddings E, R N D Each output embedding is then passed through
first
forwwrd network (FFN) to predict the corresponding individual bounding box coordinates b E N2 x4
Individual embedding enhancer. To address the fine-grained social interaction detection, which depends on subtle social cues from specific body parts such as the face, eyes, and hands, we introduce the individual embedding enhancer, which incorporates localized, body part-aware features into each

---

detected individual. Instead of relying solely on holistic representation, which can obscure subtle cues essential for distinguishing tineinrineed social interactions, we decompose their appearance into distinct body parts such as face, arms, hands, and legs. Given individual embedding E,, we generate P part-specific queries for each individual using a set learnable linear projections:
(1) where Wp E RDDXD is a learnable weight matrix for the p-th body part. The enhancer refines the part queries Qp via self-attention across part queries and cross-attention with the encoded feature map F, producing the part embedding Ep E x2NxXxDD We then concatenate the part embeddings Ep with the corresponding individual embeddings E, obtain part-aware individual features:
Qp = E [W1, W p...., Wpl E RNNIXxxD
E = Er, E....., E
E.) W fuse E RRNxX
where EX denotes the p-th body part embedding and W fuse E (B++1)DXD) is learnable projection matrix. The resulting part-aware individual embedding Σ captures both holistic features of the individual (e.g., appearance. position) and localized body cues (e.g.. facial expression, gaze, and gestures), which are critical for inferring group association and fine-grained social interaction classes.
Group decoder. To address the difficulty of directly regressing group bounding boxes without modeling interpersonal interactions, the group decoder performs group association and interaction recognition by adopting part-aware bottom-up group reasoning approach. Unlike a standard decoder that only attends visual features, our group decoder leverages a richer context by attending both the encoded image features F and the part-aware individual embeddings E enabling the model to extract group-level features by aggregating information from relevant individuals including their body parts and the feature map. Specifically, the group decoder utilizes set of NG learnable group queries QG NoNox x to produce group embeddings Eg. each of which is further decoded into two outputs: (1) predicted group bounding box coordinates g E R Nox4 and (2) multi-label classification scores over the predefined fine-gratn fine-grained social interaction classes e N.N.N
Similarity- -base association. To associate each detected individual with its corresponding social group, we adopt similarity-baset association approach. The individual embedding E, and the group embedding EG are separately projected and dot-producted to yield a similarity matrix:
S MLP(Ec) MLP(E,)² E IR Noxx
This matrix represents the affinity between each group and individual, enabling the model to identify the representative individual for each social groups based on learned similarities. For each predicted group, we select the individual with the highest similarity as the representative individual. Unlike prior work that directly predicts group bounding boxes spatial proposals using group queries [51], our method infers group configuration in a bottom-up manner driven by fine-grained social cues and interpersonal relations.
3.3 Training objective
Our model is trained with five losses: Lind for individual objectness, Les for multi-label interaction classification, Line for individual and group bounding box localization, Lpan for body part supervision, and Lassn for group association. All losses are computed between predictions and ground-truth instances matched via the Hungarian algorithm [27].
Standard NVI losses. Three among the five losses, namely Lind- Lets, and Licc are adopted from the previous work on NVI-DET [51]. Lind and Cets employ focal loss [32] and asymmetric loss (ASL) [42], respectively, to address the long-tailed and imbalanced nature of the labels. The
localization loss Live is computed as weighted sum of & and generalized loU loss [41].
Part loss. To supervise each part query focus on distinct body regions, adopt pose- pose-guided pseudo-supervision that guides the attention toward their corresponding areas. Fig. 2 illustrates the overall pipeline of this pose-guided binary mask generation. We used ViTPose [55] to ex- tract keypointss for each detected individual, other pose estimation models could also be used though; the pose estimation model is used only for training as privileged information [34, 46), and thus imposes no additional space-time complexity in testing. To generate supervision masks from keypoints, we define square window centered at each keypoint location. The size of the window is set proportional to the size of the corresponding individual box, calcu- lated as: 8; = a max(u;, h4), where (w;. h₂) are the width and height of the i-th in- dividual's bounding box. The binary mask MP is defined over the feature map such that:
5

---

Mº 11 if (u =x 51 and (0 y!' otherwise,
(4)
Image (X)
where (a. v() is the keypoint location for the p-th part of the i-th individual, and (u, 2) is spatial coordinate the feature map. A mean squared error (MSE) loss then computed between each part query's s attention map A& and its corresponding mask M,":
Pose
Estimator
Individual Emb. Enhancer
N
P 11 M
part N,P
(5)
where A" is the attention map of the p-th part query for the i-th instance, and MP" is the corresponding pseudo ground-truth mask. This supervision promotes localized attention over the human body, which is particularly bene- ficial for recognizing interaction types grounded specific body parts, including face, shoulders, elbows, wrists, hips, knees, and ankles.
Pose-guided mask (M)
Attention
part
map (A)
Figure 2: cration for part supervision.
Pose-guided binary mask gen-
Association loss. To address the ambiguity of directly regressing group bounding boxes, we associate individuals into group based on their similarity scores. We a binary cross-entropy (BCE) loss between the predicted and ground ground-truth similarity scores, computed only over matched groups and matched individuals. Let (i) be the predicted group matched the i-th ground-truie group, and let T(J) be the predicted individual matched to the j-th ground-trune individual. The association loss is defined as:
1
Cassn
Σ SEI
(a.() log Soggoooo (o()((())
a.())) S(ooo())))))))))) (6)
I
where az()) if the j-th individual belongs to the i-th group, I and I denote the set of groups and the set of individuals, respectively. This loss encourages the group embedding to attend to the individuals that belong to the corresponding group in the group decoder, by increasing the similarity between their embeddings.
Total loss. Our model is trained with five losses simultaneously in an end-to-end manner. Specifically, the total training objective is a linear combination of the five losses as follows:
L
Xinind + Aphate + Xn + App + An
3.4 Inference
At inference time, we associate the predicted individual and group embeddings to generate final predictions in the form of <individual, group, interaction triplets. To associate individuals with predicted group embeddings, we use the similarity matrix E NaxNN computed between their embeddings. For each predicted group i, we construct a triplet as: (by-, gs C.), where by- denotes the predicted bounding box coordinates of the individual with the highest similarity arg max, (j), g: represents the predicted group box coordinates, and c, is the predicted interaction logit. Finally, we apply triplet NMS to remove redundant predictions: triplets are suppressed if their individual and group bounding boxes, as well predicted interaction labels, significantly overlap with those of triplet with higher interaction scores.
4 Experiments
4.1 Experimental settings
Dataset. To verify the proposed method across diverse social scenarios, we evaluated on two benchmarks: NVI [51] and Café [24]. NVI contains 13,711 images, with 9,634 for training, 1,418 for validation, and 2,659 for test. It defines 22 atomic-level interaction classes grouped into categories:

---

Table
Comparison with the state-of-the-art methods on the NVI dataset.
val
test
Method
mR@25
mR@50
mR 100
AR
mR @5
mR @ 50
mR @ 100
AR
m-QPIC
56.89
69.52
78.36
68.26
59,44
71.46
80.07
70.32
m-CDD [58]
55.57
71.06
78.81
68.48
59.01
72.94
82.61
71.52
G-GNE--LKT [31]
50.59
70.87
80.08
67.18
56.68
74.32
84.18
71.72
NVI-DEHR [Si]
54.85
73.42
85.33
71.20
59.46
76.01
88.52
74.67
Ours
59.43
76.62
87.43
74.49
63.59
80.62
91.34
78.52
Table Comparison with the state-of-the-- methods on the Café dataset under the detection- setting. Scores are from [24].
Split by view
Split by place
Method
Group
Group
Outlier mIoU
Group mAP
Group
Outlier
mAP ..
mAP0.5
mAPo.5
mloU
Joint [10]
9.14
31.83
42.93
6.08
18.43
2.83
JRDB-base [11]
12.63
35.53
31.85
8.15
22.68
33.03
HGC [45]
6.77
31.08
57.65
4.27
24.97
57.70
Café-base [24]
14.36
37.52
63.70
8.29
28.72
59.60
Ours
18.23
46.88
67.62
10.65
39.03
63.60
facial expression, gesture, posture, gaze, and touch. These include both 16 individual-level and 6 group-level interactions, enabling the analysis of both fine-grained and group-level reasoning for fine-grained social interaction detection. Café is a large-scale multi-view, multi-person video benchmark for group activity detection. Each clip contains multiple co-occurring groups performing distinct actívities, allowing us to evaluate whether the proposed part-aware reasoning can generalize from fine-grained social interactions broader group activity understanding.
Evaluation metrics, Following NVI-DET protocol, we report mean Recall K (K = (25, 50, 100}), and their average (AR). Each recall is averaged over three IoU thresholds: 0.25, 0.5, and 0. 7.75. For Café, we use Group mAP at Group IoU thresholds of 0.5 and 1. 1.0, and Outlier mloU, which jointly capture accuracy in detecting multiple simultaneous group activities.
Hyperparameters. Our model initialized with the pretrained DETR ResNet-50. The feature dimension C' and the transformer dimension D) are set to 2048 and 256, respectively. The encoder consists of layers with attention heads, while the individual decoder, individual embedding enhancer, and group decoder comprise 3 layers with 8 attention heads. The number of individual queries, group queries, and parts are 24, 32, and 13, respectively. The NMS threshold is set to 0.5.
Training, We train our model for 90 epochs using the AdamW optimizer [35] with 31 = 0.9, 32 = 0.999, and 1e-8. The learning rate is set to 1e-4 initially and decayed le-5 after 60 epochs. We use a mini-batch size of 4. Loss coefficients are set to A; 1.0, X 2.0, \ 1.0, Ne = 2.5, AGIoU = 1.0, Ap = 10.0, and A, = 5.0. For part supervision, we use ViTPose [55] to extract 13 keypointss per person, excluding four facial keypoints (left-eye, right-eye, left-ear, right- ear) to avoid spatial overlap in pscudo-supervision masks. The window size proportion a is set to 0.2 relative the size of the individual box.
4.2 Quantitative analysis
We compare our method against three HOI-DET baselines, M-QPIC [44], m-CDN [58], and m- GEN-VLKT [31], adapted the NVI-DET setting, as well the current Kate-of-thhe-ar NVI-DET model, NVI-DEHR [51]. Table 1 presents performance comparisons on the NVI dataset [5 where our model outperforms all the others by substantial margins on both validation set and test set. Specifically, our method outperforms NVI-DEHR by 3.29 in AR on the validation set and 3.85 in AR on the test set. Notably, the gains are even more pronounced @ with improvements of 4.58 and 4.13 on the validation set and test set, respectively. It demonstrates the efficacy of our part-aware bottom-up group reasoning framework in both detecting social groups and inferring fine- -nainnd social interactions.

---

Table 3: Comparison with MLLMs.
Table 4: Impact of pose supervision.
Mothod
mk@225
m0
mK 100
AR
Setting
mR@25
mR@
mR 100
AR
Ours
63.59
80.62
91.34
78.52
Ours
59.43
76.62
87.43
74,49
LLaVA [33]
21.09
36.75
53.59
37.14
H.VA-LLRA [18]
17.40
32.12
51,93
33.81
VLM
55.18
76.94
86.96
73.02
Table 5: Impact of the proposed module.
Table 6: Impact of the loss functions.
Setting
25
mR@50
mk 100
AR
Setting
m5
mR@50
mR 100
AR
Ours
59,43
76.62
87.43
74.49
Ours
59,43
76.62
87.43
74.49
w/o enhancer
55.20
73.25
88.05
72.17
wo Lewon
30.38
47.20
64.49
48.32
w/o sim-assn
55.95
74.95
87.36
72.75
w/o Live
49.56
71.73
83.59
68.29
w/o both
56.29
70.52
85.77
70.86
w/o Epant
54.32
78.80
87.60
73.58
Table 7: Impact of the number of parts.
Table 8: Impact of the number of queries.
mR@25
mR
mR@
AR
N7
Ng
mR
mR
mR TO0
AR
57.01
75.29
87.01
73.11
24
58,
76.57
86.94
74.15
54.59
76.69
87.88
73.05
24
32
59.43
76.62
87.43
74.49
13 (Ours)
59,43
76.62
87.43
74.49
32
24
58.06
76.59
88.01
74.22
17 (All)
54.87
76.30
88.14
73.10
32
32
56.56
75.81
86.95
73.11
Table shows experiments on Café [24], a recent and challenging benchmark for group activity detection that emphasizes multi-group scenarios. Note that our method is not modified to perform temporal modeling instead we apply it as-is, in a frame-wise manner. Despite the lack of temporal modeling, our method outperformed prior methods in terms of both Group mAP and Outlier mloU, These results demonstrate the effectiveness of our method in the related task and further suggest. that part-aware representations and bottom-up group reasoning not only benefit fine-grained social interaction detection, but also contribute to group activity understanding tasks.
Table 3 presents comparison between our method and recent MLLM, LLaVA--.--vicuma [33], on the NVI test set. To support LLaVA, we provided ground-truth group bounding boxes and cropped the image accordingly before querying the model to identify fine-grained social interactions. Even under this favorable setup, LLaVA achieved only 37.14 AR, which is far lower than our method. Moreover, we fine-tuned LLaVA using Low-rank adaptation (LoRA) A) [18] on the NVI training set. We applied LoRA with rank 8 to the attention projection layers of LLaVA, enabling adaptation with a relatively small number of trainable parameters. We trained the LoRA adapter using Adam optimizer [26] with a learning rate of 1e 1e-4 for 15k steps. However, the result shows that LoRA fine-tuning does not yield additional gains. This suggests that even closed- world setting, naively fine-tuning LLaVA may not be sufficient, and additional task-specific prompt design, longer training steps, or full fine-tuning may be required.
4.3 In-depth analysis
We verify the effectiveness of model through in-depth analysis the NVI validation set.
Impact of the pose supervision. To validate the efficacy of our pose-guide gueded supervision, we compare it with variant using CLIP [40] to learn specific body-parts via text embeddings. To this end, text prompts are constructed in the form of "A photo of person [body part]". summarized Table 4, our method consistently outperformed this VI.M-guided variant, particularly in mR @25 and average recall (AR). It suggests that CLIP guidance less effective in providing fine-grained spatial cues than pose estimators. We attribute this performance gap to the relatively weak spatial reasoning capabilities of current VLMs, which are primarily trained via the image-text contrastive learning.
Impact of the proposed modules. To address the contribution of each proposed component, we evaluate three ablated variants of our model: (1) removing the individual embedding enhancer, resulting in individual embeddings without part-aware enrichment; (2) replacing the similarity-basee
association with the conventional guided embedding [31, 51]: [:: (3) removing both modules together. As shown in Table 5, removing either component leads to a noticeable drop in performance, and the degradation becomes more severe when both are excluded. It demonstrates the importance of both pose-guided part- representation and similarity- bassed bottom-up group reasoning approach.

---

Images
Individual decoder
Individual emb. enhancer
Group decoder
Prediction
Mutual- gaze Gaze- following
Hug
Handshake
(d)
Figure 3: Visualizations of the cross-attention map from the individual decoder, individual embedding enhancer, and group decoder. Blue and green bounding boxes indicate individuals and groups, respectively. Predicted interaction labels are shown on the right.
Impact of the loss function. To assess the impact of each loss, ablate Cpants Lussn and Lase for group localization, as described in Sec. 3.3, As shown in Table 6, removing Cassn leads to a drastic drop in performance, as the model can no longer associate individuals with their corresponding groups, underscoring the importance of similarity-based reasoning in our framework. Excluding Loe also causes a clear performance degradation, suggesting that accurate spatial localization of group boxes is helpful. Lastly, removing Cpat results in a slight performance drop, demonstrating the effectiveness of pose-guided supervision for detecting fine one-grained social cues.
Effect of the number of parts. Table 7 summarizes the effect of the number of parts P. We observe that using 13 parts, which excludes redundant facial keypoints, achieves the best overall performance. Using fewer parts leads to a drop in performance. Interestingly, using all 17 keypoints does not improve performance and even results in degradation. This highlights that not all keypoints are equally useful; using overlapping or redundant parts may confuse the model and hurt model effectiveness.
Effect of the number of queries. Table 8 shows the effects of the number individual queries Nr and group queries Ng. To ensure sufficient capacity for representing multiple individuals and interactions, we set both N2 and NG to be at least 24, based on the maximum number interactions observed in the NVI training set. The best performance is achieved with 24 individual queries and 32 group queries. Unlike previous methods utilizing guided embedding that require the number of individual and group queries to be identical, our bottom-up reasoning framework with similarity-basee association allows for a flexible number of queries for each component.
4.4 Qualitative results
Attention visualization. We visualize the cross-attention map of the last layer in the individual decoder, individual embedding enhancer, and group decoder for each predicted triplet in Fig 3. The individual decoder shows attention focused on spatial boundaries of the dedicated individual boxes. while the individual embedding enhancer attends to each body part of the individual; among these, we select one representative body part attention. The group decoder attends both the spatial boundaries and regions that are essential for inferring interactions. As shown in Fig. 3 (a) and (b), the group decoder attends the facial regions of the individuals belonging the social group, while in Fig. (d), it focuses on the hands to detect handshake interaction. In Fig. 3 (c), despite severe spatial overlap between individual boxes, our model successfully predicts hug interaction by leveraging part-aware features attending to the arms through the individual embedding enhancer.

---

Images
Gaze: gaze following
Gaze: gaze- foloowoolo
Expression: neutral
Gaze: mutual gaze
Images
Gaze: gaze-following
Touch: hug
Expression:
Gaze: gaze-following
ml
Images
Posture: bowing
Posture: bowing
Gaze: foze-following
Expression: smile
Images
Posture: arm-crossing
Expression: smile
Posture: arms-akimbo
Gaze: gaze- followolg
Images
Posture: slouching
Expression: smile
Gesture: wave
Touch: handshake
Figure 4: Qualitative results of our model the NVI test-set. The first column shows input images, and the remaining columns visualize the predicted NVI-DET triplets. Blue and green boxes denote individuals and groups, respectively. Predicted interaction labels presented below, where wrong predictions are highlighted in red.
Triplet visualization. Fig. shows qualitative results of our model on the NVI test set. For each image, select and visualize among top-10 10 predictions with the highest confidence scores. The results demonstrate that our model effectively localizes individuals, identifies their corresponding social groups, and recognizes fine-grained interactions through part-aware bottom-up reasoning. Even in the failure cases-for example, arms-akimbo in the fourth row and wave in the last row-the model attends relevant part cues, such as arms and hands, resulting in plausible predictions.
Conclusion
We have presented part-aware bottom-up group reasoning model for fine-grained social interaction detection. Our approach addresses key limitations of prior methods that rely on holistic representations of individuals and directly detect groups. By leveraging pose-guiee pose-guided supervision to enhance part- aware features and applying bottom-up group reasoning, our method effectively captures localized social signals that essential for recognizing subtle and nuanced social behaviors. Extensive experiments on NVI and Café validate the effectiveness of our method. We believe this framework offers promising direction for advancing social interaction understanding.
Limitation. Our model relies on the pretrained pose estimator extract part-aware representations, which depends on externl model and predefined keypoints. A valuable direction for the future would be to explore self-supervised or end-to-end approaches that can learn part-aware representations without rel relying pose estimator or predefined keypoints.
10

---

Acknowledgment
This work was supported by the NRF grant and the ITTP grant funded by Ministry of Science and ICT, Korea A2C3012728, RS-2022-11220290, RS-2022-11220966, RS-2022-12200559, RS-2019-1191996).
References
[1] Inhwan Bac, Jin-Hwi Park, and Hae-Gon Jeon. Learning pedestrian group representations for multi-modal trajectory prediction. In Proc. European Conference on Computer Vision (ECCV), pages 270-289. Springer, 2022.
[2] Jong Wook Bae, Jungho Kim, Junyong Yun, Changwon Kang, Jeongseon Choi, Chanhyeok Kim, Junho Lee, Jungwook Choi, and Jun Won Choi. Sit dataset: socially interactive pedestrian trajectory dataset for social navigation robots. Proc. Neural Information Processing Systems (NeurIPS), 36:24552 22425333-2,6,,,, 2023.
[3] Rebecca Bliege Bird, Elspeth Ready, and Eleanor A Power. The social significance of subtle signals. Nature human behaviour, 2(7):452-457, 2018.
[4] Yichao Cao, Qingfei Tang. Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models. Proc. Neural Information Processing Systems (NewrIPS), 36:739-751,
2023.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proc. European Conference on Computer Vision (ECCV), pages 213-229 Springer, 2020.
[6] Fei Chang, Jiabei Zeng, Qiaoyun Liu, and Shiguang Shan. Gaze pattern recognition in dyadic communication. In Proceedings of the 2023 Symposium on Eye Trucking Research and Applica- lions, pages 1-7, 2023.
[7] Yihua Cheng, Shiyao Huang. Fei Wang, Chen Qian, and Feng Lu. coarse-to-fine adaptive net- work for appearance-basee based gaze estimation. In Proc. AAAI Conference on Artificial Intelligence (AAAI), volume 34, pages 10623-10630, 2020.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
[9] Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. Revisiting skeleton-based action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2969-2978. 2022.
[10] Mahsa Ehsanpour, Alireza Abedin, Fatemeh Saleh, Javen Shi, Ian Reid, and Hamid Rezatofighi. Joint learning of social groups, individuals action and sub-group activities in videos. In Proc. European Conference on Computer Vision (ECCV), pages 177-195. Springer, 2020.
[11] Mahsa Ehsanpour, Fatemeh Saleh, Silvio Savarese, Ian Reid, and Hamid Rezatofighi. Jrdb-act: large-scale dataset for apatio-tempora action, social group and activity detection. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 20983-20992, 2022.
[12] Jiawei Gao, Ziqin Wang, Zeqi Xiao, Jingbo Wang, Tai Wang, Jinkun Cao, Xiaolin Hu, Si Liu, Jifeng Dai, and Jiangmiao Pang. Coohoi: Learning cooperative human-object interaction with manipulated object dynamics. Proc. Neural Information Processing Systems (NeurIPS), 37:79741-79763, 2024.
[13] Kirill Gavrilyuk, Ryan Sanford, Mehrsan Javan, and Cees GM Snoek. Actor-transsormers for group activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 839-848 2020.

---

[14] Weiyu Guo, Ying Sun, Yijie Xu, Ziyue Qiao, Yongkui Yang, and Hui Xiong. Spgesture: Source-free domain-adappe semg based gesture recognition with jaccard attentive spiking neural network. Proc. Neural Information Processing Systems (NeurIPS), 37, 2024.
[15]
Anshul Gupta, Samy Tafasca, Arya Farkhondeh, Pierre Vuillecard, and Jean-mare Odobez. Migs: A novel framework for multi-person temporal gaze following and social gaze prediction. Proc. Neural Information Processing Systems (NeurIPS), 37:1 15446-16733, 2024.
[16] Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. 150.04450 2015.
arXiv preprint
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[18] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Proc. International Conference on Learning Representations (ICLR), 1(2):, 2022.
[19] Mostafa S Ibrahim and Greg Mori. Hierarchical relational networks for group activity recogni- tion and retrieval. In Proc. European Conference on Computer Vision (ECCV), pages 721-736, 2018.
[20] Petr Kellnhofer, Adria Recasens, Simon Stent, Wojeiech Matusik, and Antonio Torralba. Gaze360: Physically unconstrained gaze estimation in the wild. In Proc. IEEE International Conference Computer Vision (ICCV), pages 6912-6922 2019.
[21] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End- to-end objent interaction detection with transformers. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 74-83, 2021.
[22] Daeha Kim and Byung Cheol Song. Optimal transport-based identity matching for identity- invariant facial expression recognition. Proc. Neural Information Processing Systems (NeurIPS), 35:18749-18762, 2022.
[23] Dongkeun Kim, Jinsung Lee, Minsu Cho, and Suha Kwak. Detector-free weakly supervised group activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 20083-20093, 2022.
[24] Dongkeun Kim, Youngkil Song, Minsu Cho, and Suha Kwak. Towards more practical group activity detection: A new benchmark and model. In Proc. European Conference on Computer Vision (ECCV), pages 240-258. Springer, 2024.
[25] Sanghyun Kim, Deunsol Jung, and Minsu Cho. Relational context learning for human-object interaction detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1925-29344 2023.
[26] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. International Conference on Learning Representations (ICLR), 2015.
[27] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logisties quarterly, 2(1-2):83-97, 1955.
[28] Ting Lei. Shaofeng Yin, and Yang Liu. Exploring the potential of large foundation models for open-vocabulary hoi detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16657-16667, 2024.
[29] Liulei Li, Wenguan Wang. and Yi Yang. Human-object interaction detection collaborated with large relation-driven diffusion models. Proc. Neural Information Processing Systems (NeurIPS), 2024.
[30] Shuaicheng Li, Qianggang Cao, Lingbo Liu, Kunlin Yang, Shinan Liu, Jun Hou, and Shuai Yi. Groupformer: Group activity recognition with clustered spatial-temporal transformer. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 13668-13677, 2021.
12

---

[31] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 20123-20132, 2022.
[32] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proc. IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
[33] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Proc. Neural Information Processing Systems (NeurIPS), 36:34892-34916, 2023.
[34] D. Lope B. Schölkopf, L.. Bottou, and V. Vapnik. Unifying distillation and privileged information. In Proc. International Conference on Learning Representations (ICLR), 2016.
[35] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05001, 2017.
[36] Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li. Clip4hoi: towards adapting clip for practical zero-shot hoi detection. Proc. Neural Information Processing Systems (NeurIPS), 004445955-5906 2023.
[37] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgenn: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 14424- 14432, 2020.
[38] Pradyumna Narayana, Ross Beveridge, and Bruce A Draper. Gesture recognition: Focus on the hands. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 5235-522 5244, 2018.
[39] A Paszke. Pytorch: An imperative style, high-performance deep learning library. In Proc. Neural Information Processing Systems (NeurIPS), 2019.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. International Conference on Machine Learning (ICML), pages 8748-8763. PMLR, 2021.
[41] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and loss for bounding box regression. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 658-666, 2019.
[42] Tal Ridník, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi -anni.-Manor. Asymmetric loss for multi-label classification. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 82-91, 2021.
[43] Samy Tafasca, Anshul Gupta, Victor Bros, and Jean-Mare Odobez. Toward semantic gaze target detection. Proc. Neural Information Processing Systems (NeurIPS), 37:121422-111 2024.
[44] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpie: Query-based pairwise human- object interaction detection with image-wide contextual information. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10410-10419, 2021.
[45] Masato Tamura, Rahul Vishwakarma, and Ravigopal Vennelakanti. Hunting group clues with transformers for social group activity recognition. In Proc. European Conference on Computer Vision (ECCV), 2022.
[46] Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: Similarity control and knowledge transfer. Journal of Machine Learning Research (JMLR), 2015.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Neural Information Processing Systems (NeurIPS), pages 5998-6008, 2017.
13

---

[48] Alessandro Vinciarelli, Hugues Salamin, and Maja Pantic. Social signal processing: Understand- ing social interactions through nonverbal behavior analysis. In 2009 IEEE computer society conference on computer vision and pattern recognition workshops, pages 42-49. IEFE, 2009.
[49] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He. Pose-aware multi-level feature network for human object interaction detection. In Proc. IEEE International Conference on Computer Vision (ICCV), pages
2019.
[50] Yan Wang, Yixuan Sun, Yiwen Huang, Zhongying Liu, Shuyong Gao, Wei Zhang, Weifeng Ge, and Wenqiang Zhang. Ferv39k: A large-scale multi-scene dataset for facial expression recognition in videos. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 20922-20931, 2022.
[51] Jianan Wei, Tianfei Zhou, Yi Yang, and Wenguan Wang. Nonverbal interaction detection. In Proc. European Conference on Computer Vision (ECCV), pages 277-295, 2024.
[52] Conghao Wong, Beihao Xia, Ziqian Zou, Yulong Wang, and Xinge You. Socialcircle: Learning the angle-based social interaction representation for pedestrian trajectory prediction. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 19005-19015, 2024.
[53] Eastman ZY Wu, Yali Li, Yuan Wang, and Shengjin Wang. Exploring pose-aware human-object interaction via hybrid learning. In Proc. IEEE Conference on Computer Vision and Pattern 2024.
Recognition (CVPR), pages
[54] Jianchao Wu, Limin Wang, Li Wang. Jie Guo, and Gangshan Wu. Learning actor relation graphs for group activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9964-9974, 2019.
[55] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Proc. Neural Information Processing Systems (NeurIPS), 55:38571-38584 2022.
[56] Rui Yan, Lingxi Xie, Jinhui Tang, Xiangbo Shu, and Qi Tian. Social adaptive module for wcakly-supervised group activity recognition. In Proc. European Conference Computer Vision (ECCV), pages 208-224, Springer, 2020.
[57] Hangjie Yuan and Dong Ni. Learning visual context for group activity recognition. In Proc. AAAI Conference on Artificial Intelligence (AAAI), volume 35, pages 3261-3269, 2021.
[58] Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li. Mining the benefits of two-stage and one-stage hoi detection. Proc. Neural Information Processing Systems (NeurIPS), 14:117299-17777 17220, 2021.
[59] Yifan Zhang, Congqi Cao, Jian Cheng, and Hanqing Lu. Egogesture: a new dataset and bench- mark for egocentric hand gesture recognition. IEEE Transactions on Multimedia, 20(5): 1038- 1050, 2018
[60] Guoying Zhao, Xiaohua Huang, Matti Taini, Stan Z Li, and Matti Pietikälnen. Facial expression recognition from near-infrared videos. Image and vision computing, 29(9):607-61,, 2011.
[61] Dengyong Zhou, Jiayuan Huang, and Bernhard Schölkopf. Learning with hypergraphs: Cluster- ing, classification, and embedding. Proc. Neural Information Processing Systems (NeurIPS).
19, 2006.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contribution, introducing a part-aware bottom-up group reasoning framework for fine- ane-grained social interaction detection, clearly presented in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations, the reliance on pose estimators, are discussed as limitations in Sec. 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describe the proposed framework and its implementation details in Sec. 3.2 and Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the code and the instructions in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss httppsccccpbbcccccceececccccclcc /wt.dee/ooodeeummiieeee /Cooeeeooiissseeeeee wwssssnoolicyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We describe implementation details and hyperparameters in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report error bars statistical significance metrics, due resource and time constraints.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computational resources in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /neuriss.....// http://tis..ppppiicc/////e/p/eea
Answer: [Yes]
Justification: We have carefully reviewed the NeuRIPS Code of Ethics and made every effort to ensure that our research follow its principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our method detects fine-grained social interactions an input image. While may contribute to future advances in social behavior understanding. we do foresee direct societal risks or harmful consequences from the proposed method in its current form.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
Our method does not release any model or dataset with high risk of misues.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use the NVI dataset [51] and the codebase of NVI-DEHR [51] and ViTPose [55], which is publicly release under MIT license and Apache-2.0 license. All assets are appropriately cited and follow the terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the code in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our method does involve LLMs as the core methods.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp///errpp/////ee cc/Conferences/20255 /LLM) for what should or should be described.
21

---

Appendix
This appendix provides contents omitted in the main paper due to the space limit. In Sec. A, we describe the detailed architecture of the proposed model, with particular focus on the group decoder. Sec. B presents further implementation and training details. Additional experimental results and in-depth analysis are presented in Sec. C. Qualitative comparisons with the previous state-of-the-art model, NVI-DEHR, are presented in Sec. D. More qualitative results are illustrated in Sec. E.
Details of the group decoder
We provide additional details of the group decoder to clarify the design and functionality, comple-
menting the description in Sec. 3.2. The group decoder plays central role in our framework, as performs part-aware bottom-up group reasoning to infer both social group configurations and their corresponding interactions. Fig. S1 illustrates detailed operations of the group decoder lay- ers. As described Sec. 3.2, the proposed group decoder attends to two information sources: (1) the part-aware individual embeddings E obtained through the individual decoder and individual enhancer, and (2) the encoded feature map F. Each group decoder layer begins with multi-head self-attention among set of learnable queries Qg. This is followed by multi-head cross-attention layer where the group queries attend to the part-aware individual embeddings. Through this attention layer, the group decoder learns to associate socially relevant individuals by capturing fine- ane-grained body part-aware cues in a bottom-up manner. Finally, as in prior work, the group queries attend to the encoded feature map capture appearance and localize the corresponding group regions. The resulting group embeddings are then used for three purposes: (1) interaction classification, (2) group bounding box regression, and (3) computing similarity scores for determining group membership. These predictions are made through separate feed-forward heads applied to each group query output.
B Experimental details
We provide additional implementation details complement those described in Sec. 4.1. Table S1 presents the specific hyperparameters used in our experiments. All experiments are conducted on four NVIDIA GeForee RTX 3090 GPUs. We implement our model using Py PyTorch [39] and utilize the official code repository of NVI [51], licensed under the MIT License.
XN
Multi-head self attention
Multi head cross aattentio
Multi-head cross-attention
Feed Forward Network
Add & LayerNorm
Add & LayerNorm
Add & LayerNorm
Add & LayerNorm
Group query (Q&)
Group embedding (E,)
Part-aware individual embedding (Ex)
Encoded feature
map (F)
Figure S1: Detailed architecture of the group decoder.
22

---

Table S1: Hyperparameters. We provide hyperparameters used during training.
Hyperparameters
Value
dimension in transformer # of individual queries Nj
256
24
# of group queries NG # of parts P
32
13
of encoder layers # of individual decoder layers # of individual embedding enhancer layers
3
3
# of group decoder layers
3
Box window size ratio a
0.2
A
1.0
Xe
2.0
1,0
dE
2.5
AGIoU
1.0
Ap
10.0
A
5.0
# of epochs batch size optimizer
90
16
Adam W
learning rate backbone learning rate learning rate drop epoch learning rate after drop
1e-4
1e-5
60
le-5
NMS IoU threshold 6
0.5
Table S2: Impact of the pose-guided mask error.
c
mR@25
mR @50
mR@ 100
AR
0.0
59.43
76.62
87.43
74.49
0.2
59.90
75.16
87.84
74.30
0.5
58.52
76.54
86.64
73.90
1.0
53.26
74.67
88.34
72.09
2.0
53.89
74.73
87.94
72.19
C
Additional experiments
Impact of the pose error. To verify the robustness of our method to pose errors, we perturb the outputs the off-the-shelf pose estimator. Specifically, for each keypoint, we apply random displacements Ar and Ay sampled from uniform distribution in the range [-e- s, e s]. where controls the magnitude of the perturbation and s the window size of the pose-guided mask. Table $2 summarizes the results across varying perturbation levels. We observe that even with substantial noise (e.g., e 2.0), the performance drop marginal. These results indicate that our pose- guided supervision is robust to keypoint localization errors, and does not require highly accurate keypoint estimation to remain effective.
Loss coefficients. We investigate the effect of loss coefficients Ap and Ag. which control the weights of the part loss Cpan and the association loss Lassn respectively. As shown in Table S3, removing the association loss (i.c., X = 0.0) leads to a dramatic performance drop, as the model is unable to associate individuals with their corresponding groups. While a small coefficient (A, 1.0) yields suboptimal performance, the model achieves robust performance when X₂ is in the range of 2.0 to 5.0. For Ap. our model achieves reasonably strong performance even with zero small loss weight, indicating that the model is not highly sensitive to this parameter. We hypothesize that this robustness arises from the model's ability to capture nuanced cues from individual embeddings through the part projection layer and individual enhancer, even with limited explicit supervision. Since our model performs well across wide range of Ap. from 0.0 10.0, we select the hyperparameter with the highest mR @25 for our final model and set Ap 10.0 and A 5.0.
23

---

Table S3: Impact of the loss coefficients.
Lasa
Lpart
m5@25
mR@50
mR@ 100
AR
0.0
10.0
30.38
47.20
64.49
48.32
1.0
10.0
40.18
57.12
75.53
57.61
2.0
10.0
57.35
78.31
88.96
74.88
5.0
0.0
54.32
78.80
87.60
73.58
5.0
2.0
59.10
75.09
88.06
74.08
5.0
5.0
54.48
75.15
87.37
72.33
5.0
10.0
59.43
76.62
87.43
74.49
Table S4:
Impact of other loss coefficients.
X
A
mR @2
mR @50
mR @ T00
AR
(1)
1.0
2.0
1.0
59.43
76.62
87,43
74.49
(2)
0.5
2.0
52.78
78.19
88.94
73.31
(3)
2.0
2.0
1.0
52.88
74.56
87,49
71.64
(4)
1.0
1.0
1.0
46.32
73.66
85.85
68.61
(5)
1.0
5.0
1.0
56.83
75.56
88.78
73.72
(6)
1.0
2.0
0.5
54.31
75.01
86.88
72.09
(7)
1.0
2.0
2.0
51.17
73.62
89.00
71.26
We further investigate the effect of the other loss coefficients A; de and X as suggested. Table S4 reports the performance under various combinations of these weights. First, for Xi comparing (1), (2), and (3), we find that increasing À; to 2.0 leads the model to over-emphasize person detection, resulting noticeable performance drop. Next, varying Ae comparison of (1), (4), and (5) shows that too small value degrades performance, with the best result at 2.0, while 5.0 causes slight drop. Finally, for \₁, comparing (1), (6), and (7) reveals that the performance remains relatively stable across its values, but λ = 1.0 leads to the best overall results. As stated in Section 4.2 and Table SI, we select configuration (1) as our final setting.
Triplet NMS IoU threshold. We investigate the effect of the triplet NMS threshold in post-processing summarized in Table S5. As explained in Sec. 3.4, triplet NMS suppresses redundant predictions by removing overlapping triplets that satisfy these conditions: (1) their individual bounding boxes have IoU above threshold 0, (2) their group bounding boxes also have IoU above 0, and (3) their predicted interaction class is the same. In each such group of duplicates, only the triplet with the highest confidence for the interaction class is retained. Without NMS. the performance drops approximately 10p in all AR. If IoU threshold 0 is set too low, NMS becomes overly aggressive and removes too many predictions; too high, duplicate predictions remain. Interestingly. we observe different trends for individual-wise interactions and group-wise interactions. For individual- interactions, using high IoU threshold leaves many duplicate predictions, which negatively impacts performance. In contrast, for group- wise interactions, using low IoU threshold aggressively removes valid predictions, resulting performance degradation. This suggests that individual-leve predictions benefit more from filtering redundant triplets, while group-level predictions are more sensitive to the loss of relevant instances. On the validation set, we find that 0.5 yields the best overall performance, and we use it as the default triplet NMS threshold in all experiments.
Class-wise recall. Table S6 compares our method and NVI-DEHR [51] in class-wise manner. Among the 22 interaction classes, the two most underrepresented categories, beckon and palmout, exhibit notably low performance for both methods due to their rarity. Nonetheless, our method achieves notably higher recall (66.67% and 22.22%) compared to NVI-DEHR (16.67% and 11.11%). suggesting improved robustness to data imbalance. We attribute this improvement to the use of part-aware representations, which allow the model to explicitly focus on the specific body parts. Unlike holistic representations that may overlook infrequent combinations of body parts, part-aware modeling enables better generalization to individual body parts and consequently leads to more reliable detection of fine-gran -raiggd interactions such as beckon and palmout, even under limited training examples for these classes.

---

Table S5: Effect of the triplet NMS threshold.
loU
individual
group
al1
@5
mR @5
mR@ @ 100
AR
mR 25
mR @5
mR @ 100
AR
AR
0.1
58.59
76.08
85,89
73.52
56.26
60.80
62.01
59.69
69.75
0.3
58.03
78.74
91.64
76.14
63.12
70.56
73.66
69.11
74.22
0.5
55.74
75.65
88.56
73.32
69.26
79.18
84.41
77.62
74.49
0.7
46.43
74.27
86.33
69.01
71.70
83.54
89.43
81.56
72.43
0.9
43.52
69.22
82.10
64.95
71.15
83.07
89.84
81.36
69.42
37.51
61.64
76.29
58.48
67.49
77.38
86.13
77.00
63.53
Table S6: Class-wise recall (%) comparison between Ours and NVI-DEHR.
Menhod
naucal
unser
s0il
sa0iic5s
loar
disgust
Rare
poin:
beckon
palm out
Ours
9X17
72.35
92.29
R2 35
76.06
66.92
$1.24
72.22
81.39
66.07
22.22
NVI-DEHR [S]]
92:75
72,85
94.57
82,50
78.35
74.42
70,49
64.14
73.15
16.67
11:11
Mestand
smmercssing
Slnccing
boning
mntuual gaae
gare-Inllowing
lng
Ini
Oura
88.19
87.40
88.3
87.79
$8.51
70.56
80.67
78.91
23.24
74.68
160.00
NVI-DAHR [SI]
70.79
85.1
55
20.34
64.78
69
78. 74.34
83.30
83.73
24.26
100.00
D Qualitative comparison with previous work
Fig, S2 presents comparison between the attention maps and predictions of our model and those of the previous state-of-the-aa NVI-DEHR [51]. Specifically, we visualize the attenntn maps from the last layer of the interaction decoder in NVI-DEHR and the group decoder in our model. We observe distinct characteristics in the attention maps of NVI-DEHR and our model. Since NVI- DEHR adopts a guided embedding method, its attention maps tend to focus solely on the individual bounding box area (shown blue), even for group-level interactions such as mutuull-aaze in (a) and 'handshake' in (b). In contrast, our model distributes attention not only within the individual but also across other relevant people with high similarity scores. In (a) and (b), our method shows strong attention regions such as the head or hands of the group members, demonstrating the benefits of part-aware group reasoning. Examples (c) and (e), which involve individual-level interactions such as 'wave', 'point', and 'surprise", demonstrate our model's ability to focus fine-grained body parts like the face, hands, elbows, and legs. This part-aware reasoning enables our model to detect subtle interactions that NVI-DEHR misses. In the failure case shown in (d), where both models make incorrect predictions, our model attends to hand and elbow part cues associated with interactions like "handshake" "estlnresulting in more plausible prediction. Finally, in (g). highlight limitation of direct group box prediction: since NVI-DEHR does not explicitly predict group membership. tends to attend to individuals who are spatially included in the predicted group box, if they not actual group members, leading to incorrect interaction predictions. In contrast, our model, while also attending to the seated person, distributes attention across relevant regions, such as the hands and faces of interacting individuals, and correctly identifies the "mutual-gaze" interaction.
E
Additional qualitative results
We visualize more qualitative results of our model on the NVI test set as shown Fig. S3. For each image, we select among top-10 predictions with the highest confidence scores. These results further demonstrate the model's ability to capture diverse fine-grained social interactions. Notably, the model successfully detects posture interactions such as 'bowing', 'arm-crossing' and 'arms- akimbo', which require attention to subtle body parts. For instance, in the last column of the third row (" ('slouching') and the fifth row ("arm-crossing') ), the model makes reasonable predictions based on the individuals' posture, even though these are not annotated the ground-truie This illustrates the model's sensitivity to subtle body cues even in ambiguous cases. Furthermore, it also accurately predicts gaze interactions such as ggaze-follwwe Our model also accurately detects various types of interactions, including touch interactions like "hug" and 'handshake'. as well as expressions such as 'smile', 'neutral', 'fear', and 'disgust', further highlighting its fine-grained perception capability.
and 'mutual-gaze', by attending to fine details.
25

---

Images
NVI-DEHR attention
NVI-DEHR prediction
Our attention
Our prediction
Mutual-gaze
Mutual-ga
(b)
Handshake
Handshake
(c)
Smile
Smile, Wave
(c])
folloooo,..,ng,
Gaze-following Handshake, Hit
Gaze-aversion
Neutral, Leg-crossing
Surprise, Point
foloowollo Hug
Gaze-following, Hug
(g)
Gaze-aversion
Mutual-gaze
Figure S2: Qualitative comparison of cross-attention maps and predictions between NVI-DEHR [51] and our model. In the attention maps, yellow indicates high attention and purple indicates low attention value. In the prediction result, blue and green bounding boxes denote individuals and groups, respectively. Incorrect predictions are highlighted in red.
26

---

Images
Posture: bowing
gaze-following
Expression: neutral
Posture: leg-
Images
Posture: arm-crossing
Posture: arms- akimbo
Gaze: gaze- gaze-following
Gaze: gaze- fallowoooo
Images
Expression:
Expression: fear
Gaze: gaze gazl-woloo
slouching
Images
gaze-aversion
Touch: handshake
Touch: handshake
Expression: smile
Images
Expression: disgust
Posture: arm-crossing
Posture: arm-crossing
Gaze: mutual-gazz
Images
gaze-following
Gaze: gaze-following
Touch:
Expression: smile
Images
Touch: hug
Expression: neutral
Gaze: aoze-following
Expression: smile
Figure S3: Additional visualizations of our model's 's predictions on the NVI test set. The first column shows the input image, while the remaining columns visualize the predicted triplets. Blue and green bounding boxes denote individual and groups, respectively. Predicted interaction classes are listed below, with incorrect predictions highlighted in red.
27


Reviews:
[{"review_id": "LVrIyl3P9R", "reviewer": "Reviewer_tcQL", "summary": "This paper presents a bottom up reasoning framework for group behaviour reasoning. Particularly, the method starts from the body parts of the individuals in the images, and is composed of dedicated decoders for individuals and groups and an association module for group behaviour reasoning such as looking at each other or hand shaking. The proposed methodology is evaluated on one dataset.", "questions": "- What is the contribution to the domain of machine learning? \n- What are the authors focusing on images, which is limited? Today LLMs are pretty good at reasoning about images, but they are failing on videos. How do the authors compare their method with LLM-based reasoning methods? \n- Can this method be applied to other datasets and downstream tasks?", "limitations": "Limitations are adequately discussed, except for the weaknesses outlined above. Additionally, visualisation results are helpful in both in the paper and the supplementary material.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The reviewer appreciates the hard work of the authors. The strengths and weaknesses are summarised below. \n\nStrengths: \n- Group behaviour understanding is still a growing topic. The majority of work treats gaze communication behaviour and activity recognition separately. This paper offers a unified framework for reasoning. \n- The results demonstrate the potential of the approach. \n\nWeaknesses: \n- The technical novelty is limited. The proposed architecture is a very typical architecture, feature extraction, individual and group modelling, and association module. It is not clear the main contribution to the machine learning domain. Part-aware representation seem to be only contribution, which is incremental. \n- The proposed method focuses on images only, which has limited applications. \n- The proposed method is tested on one dataset only, which is substandard. The reviewer understands that the datasets for this problem might be limited. However, the authors could test this on gaze communication datasets such as GPStatic. Alternatively, there is a dataset for group activity recognition called JRDB dataset, https://jrdb.erc.monash.edu. \n- Following the previous comment, the experimental results are limited. For instance the authors could have implemented stronger baselines to compare their work, for instance, leaderboard methods on the JRDB dataset. \n- The paper could also benefit further analysis. For instance, one of the problems with the group activity datasets is the data imbalance problem. It would be good to see the results over different classes to see if the method is performing equally well and where are the limitations.", "was_revised": true, "final_justification": "The reviewer thanks the authors for their hard work and their detailed response.\n\nWhile the reviewer acknowledges that this paper addresses an important topic in advancing our understanding of automatic social reasoning, they believe it is not a good fit for this venue, which primarily emphasizes contributions to machine learning methodology. Although the use of pose-guided approaches is relevant, it is not a novel concept and has been extensively explored in other contexts. Its application to this particular problem, while interesting, does not represent a sufficiently innovative contribution for NeurIPS.\n\nAdditionally, the paper's evaluation is limited to a single dataset with a narrow range of social interaction classes, all within the image domain. This further constrains its generalisability and impact.\n\nDuring the rebuttal, the authors provided additional results on the Café dataset. However, the rationale for the baseline modelling approach, as compared to temporal modelling, remains unclear. The reviewer is therefore uncertain about the final version of this paper.\n\nOverall, the paper is technically sound and well-written. However, given the considerations above, it does not meet the bar for acceptance at NeurIPS in the reviewer's opinion. Therefore, the reviewer maintains their decision.", "rebuttal": "## Contribution to the domain of machine learning (technical novelty)\n> The technical novelty is limited. The proposed architecture is a very typical architecture, feature extraction, individual and group modelling, and association module. It is not clear the main contribution to the machine learning domain. Part-aware representation seem to be only contribution, which is incremental.\n\nWe agree that our framework is based on a commonly used architecture. \nHowever, we argue that the significance of our work stems from how we leverage these standard components to address the unique challenges of fine-grained social interaction detection, which remains underexplored despite its importance. \nOur key contribution lies in introducing part-aware representations that are learned using human pose labels as privileged information [R1].\nIn other words, our model leverages human pose only for training, unlike HOI-DET methods [R2, 46] that explicitly use human pose as an additional input for both training and inference. \nThis design allows the model to benefit from fine-grained supervision while maintaining efficient inference without requiring additional inputs.\nTo the best of our knowledge, such an approach using pose-guided supervision has not been studied in the literature of fine-grained social interaction detection.\n\n## Why are focusing on images?\n\nWe focus on images in this work because existing research on interaction detection—such as NVI-DET and HOI-DET—is predominantly conducted in the image domain, with widely used benchmarks providing image-level annotations.\nWe however agree that extending this line of work to the video domain remains an important future direction and could further enhance the applicability of these methods.\n\n## Comparison with VLMs or LLMs\n> Today LLMs are pretty good at reasoning about images, but they are failing on videos. How do the authors compare their method with LLM-based reasoning methods?\n\nWe thank the reviewer for pointing out this interesting direction.\nRecent studies in HOI-DET [25,28] have indeed explored the use of VLMs or LLMs for interaction understanding.\nHowever, these models are rarely used for direct reasoning about visual inputs, and are instead often employed in an auxiliary manner. \nFor example, Lei et al. [25] leverage LLMs to describe the status of human body parts given an HOI label, and utilize VLMs to enhance interaction recognition. \nTo verify the effectiveness of VLMs and LLMs, we conducted two additional experiments. \nFirst, we employ CLIP [R3] to learn specific body-parts using text embeddings derived from body-part names. \nTo this end, text prompts are constructed in the form of \"A photo of a person [body part]\". \nAs summarized in Table R1, our method consistently outperformed this CLIP-guided variant, particularly in mR@25 and average recall (AR), suggesting that CLIP guidance is less effective in providing fine-grained spatial cues than pose estimators. \nWe attribute this performance gap to the relatively weak spatial reasoning capabilities of current VLMs, which are primarily trained via the image-text contrastive learning. \nSecond, we evaluated LLaVA-1.6-vicuna-7B [R4], a recent MLLMs, on the NVI test set.\nTo support LLaVA, we provided ground-truth group bounding boxes and cropped the image accordingly before querying the model to identify interactions. \nDespite this favorable setup, LLaVA achieved only 31.88 AR, which is far lower than our method (Table R2). \nWe believe this is because MLLMs are not trained specifically for fine-grained spatial reasoning and may not possess the ability to model group-level interactions. \nNevertheless, we agree that the zero-shot capability of such models offer promising directions for future research.\n\n**Table R1: Comparison with the pose supervision and VLM supervision.**\n| Setting               | mR@25 | mR@50 | mR@100 | AR    |\n|-----------------------|-------|-------|--------|-------|\n| **Pose supervision (Ours)** | **59.43** | 76.62 | **87.43** | **74.49** |\n| VLM supervision        | 55.18 | **76.94** | 86.96 | 73.02 |\n\n**Table R2: Comparison with LLMs.**\n| Method | mR@25 | mR@50 | mR@100 | AR    |\n|--------|-------|-------|--------|-------|\n| **Ours**   | **63.59** | **80.62** | **91.34**  | **78.52** |\n| LLaVA  |  13.44 | 28.98 | 53.21  |  31.88 |\n\n\n## Can this method be applied to other datasets and downstream tasks? (+stronger baselines)\nWe thank the reviewer for the constructive suggestions. \nWe initially considered GP-Static [R5] as suggested, but found that it is not publicly available. \nAs an alternative, we have conducted experiments on Caf\\'e [22], a more recent and challenging benchmark for group activity detection that emphasizes multi-group scenarios, similar to JRDB-Act [10]. \nAs shown in Table R3, our model outperforms stronger baselines such as Caf\\'e-base [22], JRDB-base [10], and HGC [39] in terms of both Group mAP and Outlier mIoU. \nThese results demonstrate the effectiveness of our method in the related task and further suggest that part-aware representations and bottom-up group reasoning not only benefit fine-grained social interaction detection, but also contribute to group activity understanding tasks.\n\n**Table R3: Comparison with the previous methods on Café detection-based setting.**\n| Method        | Group mAP$_{1.0}$ (view) | Group mAP$_{0.5}$ (view) | Outlier mIoU (view) | Group mAP$_{1.0}$ (place) | Group mAP$_{0.5}$ (place) | Outlier mIoU (place) |\n|---------------|----------------------|------------------------|----------------------|------------------------|------------------------|------------------------|\n| Joint [9]    | 9.14                 | 31.83                  | 42.93                | 6.08                   | 18.43                  | 2.83                   |\n| JRDB-base [10] | 12.63                | 35.53                  | 31.85                | 8.15                   | 22.68                  | 33.03                  |\n| HGC [39]       | 6.77                 | 31.08                  | 57.65                | 4.27                   | 24.97                  | 57.70                  |\n| Cafe-base [22] | 14.36              | 37.52                | 63.70              | 8.29                 | 28.72                | 59.60                |\n| **Ours**      | **18.23**            | **46.88**              | **67.62**            | **10.65**              | **39.03**              | **63.60**              |  \n\n## Analysis on the impact of data imbalance \n> One of the problems with the group activity datasets is the data imbalance problem. It would be good to see the results over different classes to see if the method is performing equally well and where are the limitations.\n\nWe appreciate the reviewer’s suggestion to further analyze the impact of data imbalance. \nTable R4 compares our method and NVI-DEHR [44] in a class-wise manner. \nAmong the 22 interaction classes, the two most underrepresented categories, *beckon* and *palmout*, exhibit notably low performance for both methods due to their rarity. \nNonetheless, our method achieves notably higher recall (66.67\\% and 22.22\\%) compared to NVI-DEHR (16.67\\% and 11.11\\%), suggesting improved robustness to data imbalance. \nWe attribute this improvement to the use of part-aware representations, which allow the model to explicitly focus on the specific body parts. Unlike holistic representations that may overlook infrequent combinations of body parts, part-aware modeling enables better generalization to individual body parts and consequently leads to more reliable detection of fine-grained interactions such as *beckon* and *palmout*, even under limited training examples for these classes.  \n\n**Table R4: Class-wise recall (%) comparison between Ours and NVI-DEHR.**\n| Method     | neutral | anger | smile | surprise | sadness | fear  | disgust | wave  | point | beckon | palmout |\n|------------|---------|-------|--------|----------|---------|-------|---------|--------|--------|--------|---------|\n| **Ours**     | **93.17** | 72.38 | 92.29  | 82.35    | 76.06   | 66.92 | **81.24** | **72.22** | **81.59** | **66.67** | **22.22** |\n| NVI-DEHR  | 92.75  | **72.83** | **94.57**  | **82.52**  | **78.35** | **74.42** | 70.49  | 64.14  | 73.18  | 16.67  | 11.11  |\n| Method     | arm-crossing | leg-crossing | slouching | arms-akimbo | bowing | gaze-aversion | mutual-gaze | gaze-following | hug   | handshake | hit      |\n| **Ours**     | **88.19**     | **87.40**      | **88.03**   | **87.79**     | **88.81** | **70.56**       | **80.67**     | 78.91           | 73.74 | **74.68**   | **100.00** |\n| NVI-DEHR  | 79.79       | 85.91        | 55.13     | 70.34       | 84.28  | 60.89         | 78.34       | **83.20**       | **83.73** | 74.28     | **100.00** |\n\n## Unclear output of the method\n> It is not clear, until the experimental results, the output of the method. In other words, what is the goal of social reasoning? However, if the problem definition can be introduced clearly in the beginning, it would be helpful for the readers. \n\nWe thank the reviewer for this helpful comment. \nTo improve clarity, we will include a figure in the introduction section that illustrates the target task and its expected output. \nIn addition, we will revise the description in Line 31–39 and Section 3.1 to make the task setup and the structure of the output more transparent and accessible to future readers. \n\n---\n[R1] Lopez-Paz et al., Unifying distillation and privileged information, ICLR 2016.\n\n[R2] Wan et al., Pose-aware Multi-level Feature Network for Human Object Interaction Detection, ICCV 2019.\n\n[R3] Radford et al., Learning Transferable Visual Models from Natural Language Supervision, ICML 2021.\n\n[R4] Liu et al., Visual Instruction Tuning, NeurIPS 2023. \n\n[R5] Chang et al., Gaze pattern recognition in dyadic communication, ETRA 2023."}, {"review_id": "m3AstC8F1L", "reviewer": "Reviewer_iWA3", "summary": "This paper proposes a part-aware bottom-up framework for fine-grained social interaction detection by modeling subtle cues (e.g., facial expressions, gaze) from body parts. Unlike existing holistic approaches, the proposed method enhances individual features with part-level details, then infers groups through similarity reasoning incorporating both spatial relations and social signals. Experiments on NVI dataset demonstrate state-of-the-art performance in detecting interaction-based groups", "questions": "(1)\tThe potential issue is that the model does not clarify the expected range of interacting group members. If too many people are involved, pose estimation may suffer from occlusions, which could reduce the accuracy of the pose-guided mask.\n(2)\tIn some tasks related to group activity recognition (your Related Work), there are several classic datasets, such as the Volleyball [1] and Collective Activity Dataset [2]. These datasets focus more on group actions—can they be considered as representations of social interactions?\n[1] Mostafa S. Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori. 2016. A hierarchical deep temporal model for group activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1971–1980.\n[2] Wongun Choi, Khuram Shahid, and Silvio Savarese. 2009. What are they doing?: Collective activity classification using spatio-temporal relationship among people. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops. 1282–1289.", "limitations": "(1)\tIn the introduction, when social interaction is first mentioned, it is advisable to include a definition, as referenced in Marco Cristani et al. “Social interaction discovery by statistical analysis of F-formations. (BMVC 2011)”.\n(2) It would be beneficial if the authors could provide a comparison of the training and/or inference time between their proposed method and existing approaches. Highlighting the computational efficiency or runtime advantage would further demonstrate the practicality and scalability of the method in real-world applications.\n(3) In the supplementary material, the authors provide an analysis of the hyperparameters λₚ and λₐ. However, it would be helpful to also include a discussion of the other loss weights, such as λᵢ, λ_c, and λ_l. Including these would offer a more complete understanding of the model's sensitivity to different components of the overall loss function.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths And Weaknesses:\n•\tStrengths: This paper addresses the problem of social interaction detection by focusing on fine-grained limb features. Its core strengths lie in a bottom-up interaction modeling approach, which progresses from individuals to pairwise relationships and then to groups, ensuring that group structures naturally emerge from interpersonal interactions (in contrast to traditional methods that directly detect group structures). Additionally, it employs part-aware feature enhancement by extracting local features from facial, arm, and leg regions based on pose estimation, accurately capturing fine-grained interaction semantics that are easily confused.\n•\tWeaknesses: \n(1)\tOne limitation lies in the clarity of Figure 1, particularly in the \"Bottom-up group reasoning\" section. The visual representation of the embedding process, especially the part involving pose-based interaction, is not clearly aligned with the textual description in the manuscript.  Improving the figure’s clarity and its correspondence with the written explanation would enhance the overall comprehensibility of the proposed approach.\n(2)\tThe transition introduced in line 107 (\"Unlike this approach...\") appears abrupt and lacks a clear logical bridge from the preceding discussion. While the paragraph describes the strengths of prior work such as [54], it does not provide a critical analysis of its limitations. A smoother transition that explicitly contrasts the proposed method with the weaknesses of existing approaches would strengthen the narrative flow and clarify the novelty of the current work.\n(3) In Figure 3(a), the illustrated example appears to involve both gaze and hug interactions simultaneously. It would be helpful if the authors could clarify how such cases are handled in the dataset annotation or during model inference. Specifically, how are multiple co-occurring interaction types defined, labeled, or distinguished—either in the data or by the algorithm?", "was_revised": true, "final_justification": "Thanks for authors’ response and I will maintain my orignal rating score.", "rebuttal": "## Clarify of Fig. 1\n> The visual representation of the embedding process, especially the part involving pose-based interaction, is not clearly aligned with the textual description in the manuscript.\n\nThank you for the constructive comment. As noted by the reviewer, the current figure incorrectly depicts the concatenation of the part-specific query $Q_P$ and the part embedding $E_P$, which may lead to confusion.\nIn accordance with the textual explanation in the manuscript, the part embedding $E_P$ should be concatenated with the corresponding individual embedding $E_I$—not with the part-specific query $Q_P$.\nWe will correct this misrepresentation in the revision to ensure consistency between the figure and text, thereby improving the overall clarity of the proposed bottom-up group reasoning framework.\n\n## Limitation of prior work\n> While the paragraph describes the strengths of prior work such as [44], it does not provide a critical analysis of its limitations.\n\nWe will revise the paragraph to more clearly contrast the proposed method with the limitations of prior work. \nSpecifically, we will clarify that prior work [44] detects social groups directly without explicitly modeling inter-person relations—a limitation that becomes particularly problematic when detecting interactions like gaze, which often occur between spatially distant individuals. \nWe will also point out that prior work relies on holistic person representations, thereby overlooking body part-level cues that are essential for distinguishing visually similar but semantically distinct interactions (e.g., *wave* vs. *point*, or *mutual-gaze* vs. *gaze-following*), as noted in the introduction (Line 40-52).\n\n## Handling co-occurring multiple interactions\n> It would be helpful if the authors could clarify how such cases are handled in the dataset annotation or during model inference. Specifically, how are multiple co-occurring interaction types defined, labeled, or distinguished—either in the data or by the algorithm?\n\nBoth the dataset annotation and our model are designed to handle the co-occurrence of multiple interactions.\nFollowing NVI-DEHR [44], we formulate interaction recognition as a multi-label classification task: for each detected group, the model outputs a probability vector over 22 interaction classes, indicating the presence of each interaction.  \nFor instance, in the fourth row of Figure S3 in the supplementary material, one group simultaneously exhibits both *gaze-aversion* and *handshake* interactions. \nMoreover, the dataset allows one person to belong to multiple groups simultaneously, allowing overlapping social groups. \n \n## Impact of accuracy of the pose-guided mask\n> The potential issue is that the model does not clarify the expected range of interacting group members. If too many people are involved, pose estimation may suffer from occlusions, which could reduce the accuracy of the pose-guided mask.\n\nWe agree that the accuracy of the pose-guided mask could be degraded when many people are present in the scene, due to occlusions. \nWe found that, in the training set, the average number of individuals involved in a group interaction is $2.02$, with some groups involving as many as $16$ people. \nHowever, our method is robust to such errors because the pose-guided mask provides soft supervision that guides the model to attend to the expected regions of each body part rather than enforcing exact localization. \nTo verify the robustness of our method to pose errors, we perturb the outputs of the off-the-shelf pose estimator (as ground-truth pose annotations are not available). \nSpecifically, for each keypoint, we apply random displacements $\\Delta x$ and $\\Delta y$ sampled from a uniform distribution in the range $[-\\epsilon \\cdot s, \\epsilon \\cdot s]$, where $\\epsilon$ controls the magnitude of the perturbation and $s$ is the window size of the pose-guided mask (Line 214). \nTable R5 summarizes the results across varying perturbation levels.\nWe observe that even with substantial noise, e.g., $\\epsilon=2.0$, the performance drop is minor. \nThese results indicate that our pose-guided supervision is robust to moderate keypoint localization errors, and does not require highly accurate keypoint estimation to remain effective.\n\n**Table R5: Impact of the pose-guided mask error.**\n| ε     | mR@25 | mR@50 | mR@100 | AR    |\n|--------|--------|--------|---------|--------|\n| 0.0    | 59.43  | 76.62  | 87.43   | **74.49**  |\n| 0.2    | 59.90  | 75.16  | 87.84   | 74.30  |\n| 0.5    | 58.52  | 76.54  | 86.64   | 73.90  |\n| 1.0    | 53.26  | 74.67  | 88.34   | 72.09  |\n| 2.0    | 53.89  | 74.73  | 87.94   | 72.19  |\n\n## Applicability to group activity understanding\n> In some tasks related to group activity recognition, there are several classic datasets, such as the Volleyball [R6] and Collective Activity Dataset [R7]. These datasets focus more on group actions—can they be considered as representations of social interactions?\n\nWe thank the reviewer for this insightful question. \nWe agree that group activity can be considered as a form of social interaction. \nHowever, most group activity benchmarks—such as Volleyball [R6], CAD [R7], or Café [22]—focus on coarse-grained interactions (e.g., *walking*, *queuing*, *fighting*) rather than the fine-grained social interaction (e.g., *gesture*, *gaze*, *facial expression*) that our work primarily targets.\nNevertheless, to further validate the applicability of our method beyond fine-grained interaction detection, we conducted additional experiments on Café [22], a recent benchmark that involves multiple group activities in complex scenes (Table R3).\nThe results demonstrate that the proposed part-aware representation and bottom-up group reasoning are beneficial even in understanding group activities, demonstrating the broader applicability of our method.\n\n\n**Table R3: Comparison with the previous methods on Café detection-based setting.**\n| Method        | Group mAP$_{1.0}$ (view) | Group mAP$_{0.5}$ (view) | Outlier mIoU (view) | Group mAP$_{1.0}$ (place) | Group mAP$_{0.5}$ (place) | Outlier mIoU (place) |\n|---------------|----------------------|------------------------|----------------------|------------------------|------------------------|------------------------|\n| Joint [9]    | 9.14                 | 31.83                  | 42.93                | 6.08                   | 18.43                  | 2.83                   |\n| JRDB-base [10] | 12.63                | 35.53                  | 31.85                | 8.15                   | 22.68                  | 33.03                  |\n| HGC [39]       | 6.77                 | 31.08                  | 57.65                | 4.27                   | 24.97                  | 57.70                  |\n| Cafe-base [22] | 14.36              | 37.52                | 63.70              | 8.29                 | 28.72                | 59.60                |\n| **Ours**      | **18.23**            | **46.88**              | **67.62**            | **10.65**              | **39.03**              | **63.60**              |  \n## Definition of the social interaction\nWe thank the reviewer for the helpful suggestion. \nWe will include a formal definition of social interaction and cite the paper [R8] when it is first introduced in Section 1 as recommended. \n## Comparison of the training and/or inference time\nWe thank the reviewer for useful comments.\nWe compare the inference time of our model and NVI-DEHR using $224 \\times 224$ resolution inputs. \nOur method achieves an inference speed of **26.62ms**, which is slightly faster than NVI-DEHR’s **27.07ms**. \n\n## Discussion of the other loss weights\n> It would be helpful to also include a discussion of the other loss weights, such as $\\lambda_\\text{i}$, $\\lambda_\\text{c}$, and $\\lambda_\\text{l}$.\n\nWe investigated the effect of the other loss coefficients $\\lambda_\\text{i}$, $\\lambda_\\text{c}$ and $\\lambda_\\text{l}$ as suggested. \nTable R6 reports the performance under various combinations of these weights.\nFirst, for $\\lambda_\\text{i}$, comparing (1), (2), and (3), we find that increasing $\\lambda_\\text{i}$ to $2.0$ leads the model to overly emphasize person detection, resulting in a noticeable performance drop. \nNext, varying $\\lambda_\\text{c}$, a comparison of (1), (4), and (5) shows that too small a value degrades performance, with the best result at $2.0$, while $5.0$ causes a slight drop. \nFinally, for $\\lambda_\\text{l}$, comparing (1), (6), and (7) reveals that the performance remains relatively stable across its values, but $\\lambda_\\text{l}=1.0$ leads to the best overall results. \nAs stated in Section 4.2 and Table S1, we select configuration (1) as our final setting.\n\n**Table R6: Impact of the other loss coefficients.**\n|      | $λ_i$  | $λ_c$  | $λ_l$  | mR@25     | mR@50     | mR@100    | AR      |\n|------|-----|-----|-----|-----------|-----------|-----------|---------|\n| (1)  | 1.0 | 2.0 | 1.0 | 59.43 | 76.62 | 87.43     | **74.49** |\n| (2)  | 0.5 | 2.0 | 1.0 | 52.78     | 78.19     | 88.94    | 73.31   |\n| (3)  | 2.0 | 2.0 | 1.0 | 52.88    | 74.56      | 87.49     | 71.64   |\n| (4)  | 1.0 | 1.0 | 1.0 | 46.32     | 73.66     | 85.85     | 68.61   |\n| (5)  | 1.0 | 5.0 | 1.0 | 56.83   | 75.56   | 88.78   | 73.72 |\n| (6)  | 1.0 | 2.0 | 0.5 | 54.31     | 75.01     | 86.88     | 72.09   |\n| (7)  | 1.0 | 2.0 | 2.0 | 51.17     | 73.62     | 89.00 | 71.26   |\n\n---\n[R6] Ibrahim et al., A hierarchical deep temporal model for group activity recognition, CVPR 2016. \n\n[R7] Choi et al., What are they doing?: Collective activity classification using spatio-temporal relationship among people, ICCVW 2009.\n\n[R8] Cristani et al., Social interaction discovery by statistical analysis of F-formations, BMVC 2011."}, {"review_id": "8OMc5vCyeO", "reviewer": "Reviewer_UmiK", "summary": "This paper studies non-verbal interaction detection, which detects the interaction between a person and a group in the image. The proposed method is inspired by DETR, where a CNN backbone and a transformer encoder extract features, with two decoders for person detection and interaction recognition. To better recognize the fine-grained human parts for interaction recognition, this method also uses ViTPose to extract human keypoints, which are then used as L_{part} to regularize the attention maps. The proposed method achieves SOTA results on NVI-DET benchmark. Additional ablation analysis are conducted to validate the design of each proposed component.", "questions": "Please refer to weakness section for my questions.", "limitations": "The authors do discussed the limitations in the paper, which mainly focuses on the utilization of the external pose detection model. However, I think there are still other limitations of the proposed method and the studied task that are not discussed in the paper. For example\n\n* the ambiguility of the interactions, which may stem from the inherent difference between the image and text modalities.\n\n* the closed-world setting for interaction recognition, where interaction types unseen during training are not recognizable during inference. This can be somehow solved by combining a visual understanding model with modern LLMs, but can not be handled with the model proposed in this paper. \n\n* the privacy issues.  I image with sufficient human information should be given to the model for recognition, which requires the human information.\n\nI appreciate the efforts of the authors made in this work and understand that these points can not be handled with a single or several works. But discussing them in the paper could make it more coherent.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The problem studied in this paper is important for some real world applications.\n* The proposed method achieved SOTA performance on benchmarks.\n* The ablation studies are generally comprehensive to validate design of each proposed component.\n* The paper is well-written and easy-to-follow. Every proposed component are explained clearly.\n\nWeaknesses:\n* The task studied in this paper, NVI-DET, is very similar to HOI-detection, except that latter detects <human, object, interaction> while this paper studies <human, group, interaction>. Although the authors discussed in this paper that the NVI-DET requires subtle human clues for interaction recognition, the same also applies to HOI-DET. To me, it is unclear how this task significantly differs from the existing HOI-DET task, which has been studied for several years in the previous literature.\n\n* The proposed achitecture is also very similar to a previous work on HOI-DET, i.e., GEN-VLKT [1]. Both models applies the idea of DETR, and adopt two decoders, where one decoder for instance detection while the other for interaction detection&recognition. Although the utilization of human pose seems to be a new contribution to the NVI-DET task, the same idea has been extensively utilized in HOI-DET tasks [2,3]. As such, I think the novelty of this paper is quite limited. The authors could further elaborate and discuss on the difference between this work and existing HOI literacture, like why the human information in HVI-DET is different from that in HOI-DET.\n\n* The proposed method can only handle closed-world settings, where interactions unseen during training and not recognizable during inference. However, this is already achievable in a previous work [1]. In the era of various pretrained and large models (e.g., CLIP, various MLLMs), it constitutes a deficiency of the model to be able to handle seen interactions only.\n\n\nReferences:\n\n[1] Liao et. al. \"GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection\". In CVPR 2022\n\n[2] Yu et. al. \"Exploring Pose-Aware Human-Object Interaction via Hybrid Learning\", In CVPR 2024\n\n[3] Wan et. al. \"Pose-aware Multi-level Feature Network for Human Object Interaction Detection\". In CVPR 2020\n\n\n Overall, I am looking forward to the rebuttals from the authors and would like to change my ratings after seeing the rebuttal.", "was_revised": true, "final_justification": "Thanks the authors for the detailed rebuttal. Although there are some weaknesses such as limited novelty as less comprehensive experiments, I think the authors addressed my questions during the rebuttal period, I thus raise my rating of this paper. \n\nHowever, I do agree with other reviewers that the evaluation on the Cafe dataset is somewhat wired. So I lowered my confidence of this paper.", "rebuttal": "## How NVI-DET differs from HOI-DET task?\n> The task studied in this paper, NVI-DET, is very similar to HOI-detection, except that latter detects <human, object, interaction> while this paper studies <human, group, interaction>. Although the authors discussed in this paper that the NVI-DET requires subtle human clues for interaction recognition, the same also applies to HOI-DET. To me, it is unclear how this task significantly differs from the existing HOI-DET task, which has been studied for several years in the previous literature.\n\nWe agree that NVI-DET shares structural similarities with human-object interaction detection (HOI-DET). \nHowever, Wei et al. [44] state that \n“Unlike HOI-DET that mainly focuses on recognizing actions between human-object pairs, NVI-DET aspires to comprehensively interpret the full spectrum of communicative nonverbal signals, whose patterns are generally more subtle, ambiguous, and involving multiple persons”.\nIn this regard, NVI-DET requires hierarchical reasoning over multiple individuals and their group-level interactions, making it fundamentally more complex than HOI-DET. \n\n## Limited novelty\n> The proposed achitecture is also very similar to a previous work on HOI-DET, i.e., GEN-VLKT [28]. Both models applies the idea of DETR, and adopt two decoders, where one decoder for instance detection while the other for interaction detection&recognition. Although the utilization of human pose seems to be a new contribution to the NVI-DET task, the same idea has been extensively utilized in HOI-DET tasks [R2, 46]. As such, I think the novelty of this paper is quite limited. The authors could further elaborate and discuss on the difference between this work and existing HOI literacture, like why the human information in HVI-DET is different from that in HOI-DET.\n\nAs pointed out by the reviewer, DETR-based architectures with separate decoders for instance and interaction prediction have been widely used in human interaction understanding [22, 28, 44].\nFor instance, GEN-VLKT [28] and NVI-DEHR [44] form interaction queries by fusing human-object embeddings. \nIn contrast, our method first detects individuals and then introduces separate group queries that attend to individual embeddings and dynamically aggregate relevant individuals into a group representation. \nThis design enables joint reasoning of group composition and interaction recognition, allowing social groups to emerge based on inter-personal relations rather than directly predicted from image features.\nFurthermore, we enhance individual representations through a part-aware representation learning module supervised by pose guidance, which enables more fine-grained reasoning over nonverbal cues.\nUnlike HOI-DET methods using human pose as an explicit input during both training and inference [R2, 46], our work leverages human pose only for training as a privileged information to learn part-aware representations. \nThis design improves both generalizability and efficiency, as the model does not depend on pose estimators at inference.\nWe appreciate the reviewer's suggestion and will cite Wan et al. [R2] in addition to the already-cited Wu et al. [46].\n\n## Limitation of the closed-world setting and the use of LLMs\n> The proposed method can only handle closed-world settings, where interactions are unseen during training and not recognizable during inference. However, this is already achievable in a previous work [28]. In the era of various pretrained and large models (e.g., CLIP, various MLLMs), it constitutes a deficiency of the model to be able to handle seen interactions only. This can be somehow solved by combining a visual understanding model with modern LLMs, but can not be handled with the model proposed in this paper.\n\nWe appreciate the reviewer’s insight on extending our work towards open-world settings and the potential of leveraging MLLMs. \nAs noted by the reviewer, VLMs and MLLMs have shown impressive performance across various tasks. \nHowever, for complex tasks such as NVI-DET and HOI-DET—which require pair-wise detection and interaction recognition—the performance of MLLMs without fine-tuning remains limited. \nTo empirically examine this, we utilized a recent MLLM, LLaVA-1.6-vicuna-7B [R4], on the NVI test set.\nWe first prompted the model to directly output <individual, group, interaction> triplets, but this did not yield meaningful results. \nTo support LLaVA, we provided ground-truth group bounding boxes and cropped the image accordingly before querying the model to identify fine-grained social interactions. \nEven under this favorable setup, LLaVA achieved only 31.88 AR, which is far lower than our method (Table R2). \nThis suggests that even in a closed-world setting, MLLMs struggle with fine-grained reasoning without task-specific tuning. \nWe acknowledge that our work is limited to the closed-world setting, and we agree that extending it to the open-world setting is an important future direction. \nWe chose to focus on the closed-world setting because we believe that fine-grained social interaction detection still remains challenging and underexplored even in this setup. \nIn the future work, we plan to explore incorporating VLMs to enable more generalizable social interaction understanding.\n\n**Table R2: Comparison with LLMs.**\n| Method | mR@25 | mR@50 | mR@100 | AR    |\n|--------|-------|-------|--------|-------|\n| **Ours**   | **63.59** | **80.62** | **91.34**  | **78.52** |\n| LLaVA  |  13.44 | 28.98 | 53.21  |  31.88 |\n\n## Ambiguity of the interactions\n> The ambiguity of the interactions, which may stem from the inherent difference between the image and text modalities.\n\nWe thank the reviewer for highlighting this important point. \nWe agree that ambiguity is inherent in social interactions.\nThis is also why NVI-DET adopted recall as the primary evaluation metric, since precision is less appropriate given the possibility of multiple plausible interactions that may not be exhaustively annotated in the dataset. \n\n## Privacy issue\n> Image with sufficient human information should be given to the model for recognition, which requires the human information.\n\nWe appreciate the reviewer’s concern regarding privacy. \nRecognizing fine-grained social interactions inevitably requires detailed human-centric visual cues, many of which can involve sensitive personal information.\nThe creators of the NVI dataset [44] have already addressed this concern explicitly in their paper. \nAs they stated:\n*“There is a risk that someone could use it for malicious purposes, e.g., widespread surveillance, invasion of privacy, and potential abuse of personal information. Therefore, we strongly advocate for the well-intended application of the proposed method, while simultaneously underscoring the importance of employing the dataset in a responsible and ethical manner”.*\nWe will include a discussion on the ethical considerations in the revised manuscript. \n\n---\n[R2] Wan et al., Pose-aware Multi-level Feature Network for Human Object Interaction Detection, ICCV 2019.\n\n[R4] Liu et al., Visual Instruction Tuning, NeurIPS 2023."}]

Author rebuttal:

Meta-review: This paper presents a part-aware bottom-up group reasoning framework that leverages body cues and social signals to more accurately detect fine-grained social interactions, achieving state-of-the-art results on the NVI dataset.

Summary of Strengths:
- The work tackles the important problem of fine-grained social interaction detection and has real-world applications.
- The proposed bottom-up reasoning approach, progressing from individuals to pairwise interactions to groups offers a natural way to infer group structures,
- The proposed approach achieves state-of-the-art results on the NVI dataset.
- The paper is well-written and easy to follow.

Summary of Weaknesses:
- The work mainly follows standard pipelines and has incremental novelty.
- Experimentation includes only one dataset and hence it is not clear whether the approach will generalize.
- The baselines could be stronger.

Overall, the paper proposes an effective method for modeling social interactions and will likely be useful to the community. Its incremental novelty and limited evaluation (dataset and baselines) were concerning to the reviewers, but authors added clarifications and new results during the rebuttals, that were found useful by the reviewers.
Decision: Accept (poster)