Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection
Authors: Chanhyeong Yang
Taehoon song
Jihwan Park
Hyunwoo J. Kim
Venue: NeurIPS 2025 poster ()
Keywords: Zero-shot HOI detection
Vision Language Model
Prompt Learning
Primary area: applications

Abstract: Zero-shot Human-Object Interaction detection aims to localize humans and objects in an image and recognize their interaction, even when specific verb-object pairs are unseen during training. Recent works have shown promising results using prompt learning with pretrained vision-language models such as CLIP, which align natural language prompts with visual features in a shared embedding space. However, existing approaches still fail to handle the *visual complexity of interaction*—including (1) *intra-class visual diversity*, where instances of the same verb appear in diverse poses and contexts, and (2) *inter-class visual entanglement*, where distinct verbs yield visually similar patterns. To address these challenges, we propose **VDRP**, a framework for *Visual Diversity and Region-aware Prompt learning*. First, we introduce a visual diversity-aware prompt learning strategy that injects group-wise visual variance into the context embedding. We further apply Gaussian perturbation to encourage the prompt to capture diverse visual variations of a verb. Second, we retrieve region-specific concepts from the human, object, and union regions. These are used to augment the diversity-aware prompt embeddings, yielding region-aware prompts that improve verb-level discrimination. Experiments on the HICO-DET benchmark demonstrate that our method achieves state-of-the-art performance under four zero-shot evaluation settings, effectively addressing both intra-class diversity and inter-class visual entanglement. Code is available at https://github.com/mlvlab/VDRP.

Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection
Chanhyeong Yang'
Taehoon Song? Jihwan Park' Hyunwoo Kim
Korea University
Korea Advanced Institute of Science and Technology
(0814gerrardso, jseven7071 ekorea. ac. kr (taehoons, hyunwoojkin//kaaist ac kr
Abstract
Zero-shot Numan-Obbeee Interaction detection aims localize humans and objects in an image and recognize their interaction, even when specific verb-object pairs are unseen during training. Recent works have shown promising results using prompt learning with pretrained vision- vision-language models such CLIP, which align natural language prompts with visual features However, existing approaches still fail to handle the visual complexity of inter- cection-includigg (1) intra-class visual diversity, where instances of the same verb appear in diverse poses and contexts, and (2) inter-class visual entanglement, where distinct verbs yield visually similar patterns. To address these challenges, we propose VDRP, framework for Visual Diversity and Region-awai aware Prompt learning. First, we introduce visual diversity-award aware prompt learning strategy that injects group-wise visual variance into the context embedding. We further apply Gaussian perturbation to encourage the prompts to capture diverse visual variations of verb. Second, retrieve region-specific concepts from the human, object, and union regions. These are used to augment the diversity-aware prompt embeddings, yielding region-aware prompts that enhance verb-level discrimination. Experiments on the HICO-DET benchmark demonstrate that our method achieves faaee-ofthhearr performance under four zero-shot evaluation settings, effectively addressing both intra-class diversity and inter-class visual entanglement. Code is available /nttps: thub. com/mlvlab//on
shared embedding space.
Introduction
aims to localize humans and objects in an image and recognize the interactions between them, serving cornerstone for fine-grained scene understanding. Unlike standard HOI detection, which assumes supervision over all interactions, zero-shot HOI detection must generalize to unseen combinations of verbs and objects. While recent advances in Vision-Language Models (VLMs) have significantly improved the zero-shot recognition of object and attribute classes [14 16 17 I8
Human- Human--bbecct Interaction (HOI) detection 12
19 20), zero-shot HOI detection remains fundamentally more challenging. As previous works 21 22 pointed out, the
difficulty stems not only from the compositional novelty of interactions but also from the visual complexity of interactions- where each verb class exhibits large intra-class visual diversity, and different verbs frequently produce visually similar patterns.
First, verb classes exhibit substantial intra-class diversity. As shown in Fig. (A), instances of the verb "holding a baseball glove" may appear in drastically different poses, scales, or scene contexts, yet must be classified under the same label. To quantify this, we compute diversity score using CLS features extracted from frozen CLIP visual encoder. For verbs, we crop the union region; for objects, the object bounding box is used. The average pairwise cosine similarity within each class is
*Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

[Diversity score comparison between object verb]
"Person holding baseball glove"
DOE
class
0.274
0.364 +0.060
Verb class
divessiy score
Objoct
Vorh
Intra-class diversity in the same interaction closs
[Inter-class visual overlap of verb classes in t-SNE]
Proeetype
coing
Beckig
"Person eating object."
"Person licking an object."
"Person sitting object."
collngg
boundryg
(B) Inter-closs visual entonglement between the different interaction classes
Figure 1: Analysis of the visual complexity in HOI detection. (A) Verb classes exhibit significant intra-class visual diversity, where instances the same verb (e.g., "holding baseball glove") appear under varied poses, viewpoints, and scene contexts. To quantify this, we crop the union region and extract the CLIP visual CLS feature. A diversity score is then computed as the expected cosine dissimilarity E5[1 cos(-)] across samples of the same class. Verb classes exhibit higher diversity (0.364 + 0.060) than object classes (0.274 + 0.048), highlighting the difficulty of representing verbs with a single static embedding. (B) Verb classification also suffers from inter-class visual entanglement, where semantically distinct verbs (e.g., "eating", "licking", "sitting at") yield visually similar patterns. To visualize this, randomly select five verb classes, extract their union-region CLS features, and project them to 2D using t-SNE. The resulting clusters show significant overlap, highlighting the need for region-awart prompts to improve verb separability in HOI detection.
measured, and the diversity score is defined as the expectation of c.s(.). This analysis shows that verbs have significantly higher intra-class diversity (0.364 + 0.060) compared to objects (0.274 0.048), indicating that single prompt embedding is likely insufficient to capture such variation.
Second, interactions often exhibit inter-class visual entunglement, where different verbs produce highly similar visual patterns. illustrated Fig- (B), semantically distinct verbs such as "eating", "licking", and "sitting at" frequently share similar human-object layouts. In such cases, accurate classification depends on regional differences- often localized the human or object region- global or union-level features may fail to capture. To analyze this phenomenon, we extract union- region CLS features from a frozen CLIP visual encoder across five verbs and project them into a 2D space using t-SNE. The resulting convex hulls show substantial overlap not only at the sample level but also across verb prototypes, revealing poor inter-class separability. This entanglement poses a major challenge for zero-shot HOI detection, where explicit class supervision is unavailable.
Recently, studies on zero-shot HOI detection have explored prompting strategies that leverage pretrained CLIP models 23 24
27 These methods map HOI triplets to textual descriptions and embed them in a shared lasion-language space. While effective for semantic aligmment, most approaches assume single static prompt per verb [3 10 26), making them inadequate for modeling the visual diversity within each class. Some incorporate spatial cues in the visual branch [24]. but leave the text prompts agnostic to region-specific semantics. Others rely on LLM- -e.--geerrttedd descriptions 25 focusing on semantic differences across verbs but overlooking intra-class variation.
25
26
To overcome these limitations, we propose new framework called VDRP (Visual Diversity and Region-aware Prompt learning) zero-shot HOT detection. Our method enhances prompt represen- tations in two complementary ways. First, we inject group-wise visual variance into the learnable context embeddings and apply Gaussian perturbation, allowing the prompts reflect intra-class appearance diversity and better adapt to varied visual realizations during training. Second, retrieve

---

region-specific concepts from the human, object, and union regions and use them to augment the prompt embeddings, producing region-aware prompts that improve verb discriminability.
Our contributions are summarized as follows:
We propose visual diversity-aware prompl learning that models intra-class variation by in- jecting group-wise variance into the context embedding and applying Gaussian perturbation, enabling the prompts to generalize across diverse appearances of the same verb.
We introduce a region-aware prompt augmentation that leverages region-specific concept retrieval from human, object, and union regions to enrich the prompts, enhancing discrim- inability among visually similar verbs.
We integrate the two modules into unified framework, VDRP, and demonstrate its effec- tiveness on the HICO-DET benchmark, achieving new state-of-the-art performance under multiple zero-shot evaluation settings.
2
Related works
HOI detection. Human-Objec Object Interaction (HOI) detection typically involves three sub-tasks:
object detection, human-object pairing, and interaction classification. Thanks to advances in large-
scale benchmarks [28] [9] 4 21 and transformer- architectures [30 9 31 a wide range of
approaches have been proposed. Broadly, HOI methods fall into one-stage
23
27
4 9
5 paradigms. One-stage methods jointly predict object locations and interactions, often using set prediction frameworks such DETR [0]. In contrast, two-stage methods decouple the task into object detection and interaction classification: a pre-trained detector localizes humans and objects, and dedicated module predicts the verb label for each human-object pair. The division of the HOI task in two-stage approaches allows efficient training (5 6]. and shows promising results. Our work falls in the two-stage paradigm.
and two-stage [10] 26 24
Zero-shot HOI detection. Zero-shot HOI detection aims to identify HOI triplets unseen during training, a task challenged by the long-tail distribution of compositional datasets. With the rise of pretrained on large- scale image-tex text pairs,
Vision-Language Models (VLMs) [15 32
33
14
34
many works leverage their generalization for HOI. Several methods 10 23 27 351 align HOI models
with CLIP's pretrained representations to enable effective transfer. Recent approaches [24 26 25
adopt prompt learning to adapt CLIP with few learnable parameters for fine-grained interaction understanding. However, most rely on single static prompt per verb [10 23 26], limiting their ability to capture intra-class visual diversity. Later works add spatial cues or -.Le-rgeeetted descriptions [24] 25), but still lack region-level adaptation or concept-level grounding. We address these gaps with visual diversity-aware prompt learning and region-aware prompt augmentation.
Prompt learning. Prompt learning is widely used strategy for adapting vision- models (VLMs) like CLIP to downstream tasks [36 37
38
39 40 E2] Early works such as CoOp [8]
and CoCoOp [19 optimize learnable image- conditiond context vectors, while MaPLe 38 jointly tunes visual and textual prompts to improve cross modal alignment. In zero-shot HOI detection, CMMP 24 and EZ-HOI 251 adopt multi-mode prompt learning, but do not account for visual diversity of verbs and use the same verb prompt across regions. This limits their ability to adapt to diverse verb appearances and handle visually similar interactions. Meanwhile, distribution--asse prompt learning (37 36 40 43 has shown that leveraging feature variance improves generalization. Inspired by this, we propose method that injects group-wiss visual variance- via modulation and perturbation-ing prompt embeddings, and augments them with region-specific concepts, enabling effective handling of visual diversity and improving verb discriminability in zero-shot HOI detection.
3 Methods
In this section, we present our framework, VDRP, designed to address two key challenges in zero-shot nuuann-Objes Interaction (HOI) detection: (1) intra-class visual diversity, where instances of the same verb exhibit wide range of visual appearances, and (2) inter-class visual entanglement, where semantically distinct verbs appear visually similar. To tackle these challenges, VDRP introduces two complementary components. The first, visual diversity-aware prompl leurning, addresses intra-class diversity by injecting group-wise visual variance into the context embeddings and applying variance- guided perturbation, resulting in visual diversity prompts that better capture verb-specific
3

---

Image
(A) Overall pipeline
Adapter f Imugee
T1
Logit,
eucoder
Rol Align
Tp
Logit
Logithai
Sparial Haad
Logit
Projatwm
Detector
b
De
Pg
(B) Visual diversity-awars
(C) Region-aware
prompt fearning
prompt augmentation
A photo person advucting an object.
Varanee
Concept retriever
ploto of person is ussembliig object.
embedding
Context
Concat
encoder
Text
Perturbation
E
Augmenation
photo person blocking on objecr.
...
Concept retriever
To
Xy
photo person is migcing object.
Concept rctrieeer
Verb prompt
Token Embedding
Figure 2: Overview of our VDRP framework. (A) We adopt two-stage HOI detection pipeline with frozen detector and CLIP image encoder to extract human (xp), object (x,). and union (xa) features. A spatial head further refines the union feature into for region-aware prompts via spatial encoding. (B) Visual diversity-awar prompts are generated by injecting group-wise variance and perturbation model intra-class variation. (C) Retrieved region concepts are then fused with these prompts to produce final region-aware prompts Th, To, and T, used for verb classification.
appearance variations. The second, region-aware prompt augmentation, enhances these prompts using region-specific concepts retrieved from the human, object, and union regions, yielding region prompts that improve inter-class discriminability. We begin by outlining the overall pipeline (Section 3.1).
followed by detailed descriptions of the two modules in Sections 3.2 and 3.3
3.1 Overall pipeline
As illustrated Fig. (A), our method follows a two-stage HOI detection framework 10 24
consisting of (1) human-object detection and (2) interaction classification. In the first stage, frozen object detector (DETR (30 applied the input image to identify object instances. Each detection yields triplet (s,, In b,), representing the confidence score, object class embedding, and bounding box of the n-th instance. In the second stage, we encode each instance into prior embedding Proj (Is; [; 6]) ) Ra which serves as guidance for task adaptation. Using this prior, we extract patch embeddings from frozen CLIP encoder augmented with lightweight adapter layers inserted into multiple transformer blocks. Fach adapter applies a bottleneck structure that projects the patch features, attends to the prior p. and reconstructs the output:
X = (aassAttt(Projaaoom (X.)..P)) P.P).
(1)
X X2 + Proj (X;),
where X; Nxd is the image feature map at layer i. The final image feature map X IR Nxd is obtained from the last transformer layer with projection. Given the image features and the boxes for the human (b,), object (b,), and union region (b,), we extract region features via RolAlign 44
X RolAlign(X... bn), X. RolAlign bo), X RolAlign (X, b.).
where Xh. Xo, Xú Rd denote the region features for the human, object, and union regions, respec- tively. To enhance the union-region representation with spatial priors, we apply spatial head that fuses Xj with human and object features, as well as their bounding boxes:
Xa = SpatialHead Xay X Xo: by b.) E Rd.
(3)
More details on SpatialHead( () are provided in the supplementary material. Next, each region feature-Xo Xh. Xo. and Xg is matched against set of verb prompts.

---

Region fenture
Concept pool C
oi
Group f(+)
Meon
Group-wisc
CLS
variance
Sparsemax()
Weighted
Concept weight
Group-wise variance estimation
Concept retriever
Variance Injection
Medulation
Perturbation
Concept
vector
Noise
sampling
Learnable
Concat
Encoder
embedding E
Text
Visual diversity- aware prorppt
Augmentation
P
Rogien-aware
Token embedding
prompt
Visual diversity-aware prompt learning
(B) Region-oware prompt augmentation
Figure 3: Detailed architecture of our methods. (A) To model intra-class variation, we compute verb-wise visual variance a² from union-region CLS features, average them over similar verbs to obtain group-wise variance &e and inject it into the shared context embedding E via an MLP. This is combined with the verb prompt P, and encoded by the CLIP text encoder to produce t", which further perturbed using Gaussian noise scaled by visual variance. (B) For inter-class discriminability. we retrieve region concepts from features X(-) using a Sparsemax over a concept pool Ciyy and add the result e" obtain the final region-aware prompt (()
As illustrated Fig (B) and (C), Each prompt i is constructed in two stages. Let T() = (...... (X) RdxV denote the region-aware prompts for (-) uh. We first generate
visual diversity-aware prompt t" via group-wise variance injection and Gaussian perturbation, then augment it with region-specific concepts retrieved based on region feature X(.)- This two-step design allows the final region-awan prompts reflect both the visual diversity of verbs and localized context. See Sections 3.2 and 3.3 for details. The logits for each region prompts is computed as:
Logit, X, Th. Logit, = x To Logit, x Tu:
(4)
where Logit,) Finall, the overall HOI classification logit obtained by averaging region-wise logits:
Logit (Logit, + Logit, + Logit,)
(5)
We train the model using focal loss 45 for multi-label verb classification.
3.2
Visual diversity-aware prompt learning
To address intra-class visual diversity in HOI detection, propose visual diversity-aware prompt learning method that incorporates visual variance into both the context modulation and prompt perturbation processes. Static prompts-optinn variation of instances within the same verb class. Inspired by recent findings [40] 37 36) showing that variance-aware representations improve generalization, explicitly encode class-level visual variance to guide prompt adaptation and inject noise that reflects the extent of such diversity.
single points points-struugle
struggle represent the wide
Group-wise variance estimation. As illustrated in Fig. (A), begin by extracting union-region features from the training set using a frozen CLIP image encoder. Specifically, we crop the union box of each human-object pair and extract the CLS token. For each verb let v denote the CLS feature of the j-th instance. compute the mean μ. and variance o² over N, samples. obtain a stable estimate for rare unseen verbs, construct group of verbs ((v) by selecting the
5

---

similar verbs based on cosine similarity between CLIP text embeddings. This grouping allows each verb to inherit variance statistics from its semantically similar neighbors. The group-wise variance is then computed as:
(
r'EG(o)
This group-wise variance serves as an inductive prior capturing the expected diversity of verb v.
Visual diversity- prompts. Then, we transform the group-wise variance into a modulation vector using lightweight MLP perform variance injection follows:
d = ML.(( R
This is added the shared context embedding E TR. Nax to produce verb-specific context:
E, E d at,
(8)
where a is small scaling factor for stability. Given a verb prompt sentence P,, (e.g., "A photo person is [t]+ing an object. ""), we tokenize it into token embedding P, and concatenate it with the modulated context form the final input for CLIP text encoder (--):
((É;; P.)) E
To further reflect visual variability, we perturb each prompt embedding using Gaussian noise scaled by the group-wing wise standard deviation. We normalize o. across dimensions and modulate it match the standard deviation of t", producing o.. Noise then sampled and applied element- wiie::
t" = + (c 8, (O,I.
(10)
where 3 is small scaling factor controlling the perturbation strength. This results in prompt embeddings that encode both the central semantics of the verb and its expected visual diversity. Finally, we collect all perturbed prompts across the V verbs to form the diversity-awand prompts T= -_....,, ( E RdxV, which serves as the base for region-aware prompts.
3.3 Region-aware prompt augmentation
To address inter-class visual entanglementtwheree semantically distinct verbs exhibit similar visual patterns-w augment prompts using region-specific concepts from the human, object, and union regions. While diversity-awart prompts reflect class-level variation, they cannot capture the region concepts needed to distinguish visually similar verbs. To bridge this gap, we introduce retrieval-based region-award prompt augmentation, which allows each prompt to specialize based on regions.
Region concept generation. To enrich prompt embeddings with localized semantics, we follow [46 47 20 and query LLMs (e.g., LLaMA-7B [48 and ChatGPT-4 [49]) generate K region-level concepts for each verb v and region type R. E [human, object, union). Each prompt follows the format: "For the verb [P.]. give K short visual concepts for the [R] region." The resulting concepts are encoded using the CLIP text encoder T(-) to form the concept pool:
C= = (
P.K.
u.k ())
R2
where (.) E uu denotes the region. For more detailed examples, please refer to Fig (A).
Region-aware prompts. As shown in Fig. (B), given a region feature X(-) Rdd and its corre- sponding concept pool Ci), compute cosine similarity between the feature and each concept:
(x())
m.ky
(11)
())
[x
To highlight the most informative concepts while ignoring irrelevant ones, we apply Sparsemax (50 to the similarity scores, which assigns exact zero weights to uninformative entries and retains only the most relevant concepts. Using the resulting scalar weight W we compute a region concept vector:
(.)
K
= Σ
o.k W E E
(12)
(

---

Table 1: Comparison under NF and RF settings. We report harmonic mean (HM) between Unseen
and Seen. Best in bold, second best underlined.
NF-UC
RF-UC
Method
Backbone
HM
Full
Unseen
Seen
HM
Full
Unseen
Seen
GEN-VLKT
23
Resmet50+ ViT-B
24.17
23.71
25.05
23.38
26.08
30.56
21.36
32.91
BoID 52
Resnet50
26.71
26.69
26.76
26.66
26.11
29.52
22.04
31.39
HOICLIP 27
Resnet50+ ViT-B
28.70
27.75
29.36
28.10
26.55
32.99
25.83
28.47
ADA-CM 10
Resnet50+ ViT-B
31.76
31.39
32.41
31.13
30.48
33.01
27.63
34.35
CLIP4HOI 26
Resnet50+ ViT-B
29.54
28.90
31.44
28.26
31.23
34.08
27.88
35.48
CMMP 24
Resnet50+ ViT-B
30.82
30.18
32.09
29.71
31.10
32.18
29.45
32.87
EZ-HOI
25
Resnet50+ ViT-B
31.76
31.17
33.66
30.55
31.18
33.13
29.02
34.15
Ours
Resnet50+ ViT-B
33.85
32.57
36.45
31.60
32.77
33.78
31.29
34.41
This concept vector is then used to augment the diversity-aware prompt t%, producing the final region-aware prompt follows:
6/ t" + c RR
(13)
where 7 is a scalar controlling the degree of augmentation. The final set of region-aware prompts
T = ...... ( used compute classification logits, as described in Eq. (4) This improves
discriminability of the model by aligning each prompt with region-specific semanties.
4 Experiments
4.1 Experimental settings
Datasets.
We conduct experiments on the HICO-DET benchmark for HOI detection. HICO-DET contains 80 object categories from the COCO dataset [51] and 117 actions, forming 600 HOI classes. includes 47,776 images, with 38,1 18 for training and 9,658 for testing.
Zero-shot setting on HICO-DET. Following prior works 3 2 1 23), we evaluate under four settings: Non-rare First Unseen Composition (NF-UC), Rare First (RF-UC), Unseen Object (UO), and Unseen Verb (UV). NF-UC and RF-UC define 120 unseen and 480 seen HOI triplets from 600 total, with unseen compositions drawn from head and tail categories, respectively. UO uses 68 object classes to construct 500 seen and 100 unseen triplets. UV withholds 20 out of 117 verb classes, yielding 516 seen and 84 unseen triplets.
Evaluation metric. Mean Average Precision (mAP) is used to evaluate the model for HOI detection. Specifically, sample is regarded as a true positive if two conditions are met: 1) the IoU of both human and object bounding boxes is larger than 0.5, and 2) the HOI triplet prediction is correct.
Implementation Details. We follow the standard training setup used in prior zero-shot two-stage HOI detection methods (24] 10 (25., where DETR is first fine-tuned on instance-level annotations from CL.P ViT-B/16 as the visual backbone. Additional implementation and training details are provided the supplementary material.
the HICO-DET training split. Unless otherwise noted,
4.2 Zero-shot HOI detection
We evaluate our method on four zero-shot HOI detection settings in HICO-DET: NF-UC, RF-UC, UO, and UV. Table. D 2 and 3 report mAP (Full/ Unseen / Seen), harmonic mean (HM), and trainable parameters (#TP), comparing with state-of-the-ae baselines. In NF-UC and RF-UC (Table. []. our method achieves the best scores across all metrics. In NF-UC, we obtain 36.45 (Unseen) and 33.85 (HM), surpassing CLIP4HOI by +5.01 (Unseen) and +4.31 (HM). In RF-UC, we outperform EZ-HOI by +2.27 (Unseen) and +1.59 (HM). Our method uses only 4.50M trainable parameters, much fewer than EZ-HOI (6.85M) and CLIP4HOI (56.7M), In UO (Table. (2) we achieve 36.13 (Unseen) and 34.41 (HM), outperforming CMMP and EZ-HOI by +1.97 and +2.27 HM, respectively. In UV (Table. 3). we again achieve the best scores: 26.69 (Unseen), 32.72 (Full), and 29.80 (HM), with a +1.59 gain on Unseen verbs over EZ-HOI. These results confirm the effectiveness of our visual diversity-awar aware prompts and region-aware prompts in the shot HOI detection.
7

---

Table 2: Comparison under the UO (Unseen Object) setting. We report harmonic mean (HM) between
Unseen and Seen. #TP: trainable parameters. Best in bold, second best underlined.
Method
Setting
Backbone
#TP
HM
Full
Unseen
Scen
FCL M
UO
Resnet50
17.65
19.87
15.54
20.74
ATL 2
UO
Resnet50
17.79
20.47
15.11
21.54
GEN-VLKT
UO
Resnet50
42.05M
20.11
25.63
15.01
28.92
HOICLIP 27
UO
Resnet50+ B
66.18M
20.32
28.53
16.30
30.99
CLIP4HO1 26
UO
Resnet50+ ViT-B
56.7M
31.98
32.58
31.79
32.73
CMMP 24
UO
Resnet50+ViT-a 2.30M
32.44
31.59
33.76
31.15
EZ-HOI 25
UO
Resnet50+Viil ViT-B
6.85M
32.14
32.27
33.28
32.06
Ours
UO
Resnet50+ViT--B
4.50M
34.41
33.39
36.13
32.84
Table 3: Comparison under the UV (Unseen Verb) setting. We report harmonic mean (HM) between Unseen and Seen. #TP: trainable parameters. Best in bold, second best underlined.
Method
Setting
Backbone
#TP
HM
Full
Unseen
Seen
GEN-VLKT 23
UV
Resnet50++
42.05M
24.35
28.74
20.96
30.23
EolD 52
UV
Resnet50
26.29
29.61
22.71
30.73
HOICLIP 27
UV
Resnet50+
66. 18M
27.72
31.09
24.30
32.19
CLIP4HO1 26
UV
Resnet50+viT-J
56.7M
28.35
30.42
26.02
31.14
CMMP 24
UV
Resnet50+ViT--
2.30M
29.23
31.84
26.23
32.75
EZ-HOI 25
Resnet50+
ViT-B
6.85M
29.09
32.32
25.10
33,49
Ours
UV Resnet50+ViT--B
4.50M
29.80
32.73
26.69
33.72
Table 4: Ablation results under four zero-shot settings. VDP: visual diversity-aware prompts, RAP: region-aware prompts, VDRP: full model with both.
NF-UC
Full
Unseen
Seen
RF-UC
Full
Unseen
Seen
BASE
28.99
31.68
28.32
BASE
29.80
25.64
30.84
VDP
30.17
32.19
29.66
+ VDP
31.95
29.16
32.65
RAP
30.43
34.93
29.30
+ RAP
32.43
26.46
33.93
VDRP
32.57
36.45
31.60
+ VDRP
33.78
31.29
34.41
UO
Full
Unseen
Seen
UV
Full
Unseen
Seen
BASE
28.92
28.60
30.50
BASE
29.72
22.41
30.91
VDP
31.49
33.29
31.13
VDP
31.53
23.78
32.79
RAP
31.85
33,90
31.44
-RAP
31.28
24.53
32.38
VDRP
33.39
36.13
32.84
+ VDRP
32.73
26.69
33.72
4.3
Ablation studies
Component analysis. To assess each component's contribution, conduct ablation study across four zero-shot HOI settings (Table(4). Starting from a static prompt baseline, we add two modules: (1) VDP models intra-class variation using group-wiss wise visual variance and Gaussian noise, and (2) RAP augments prompt embeddings with retrieved region-specific concepts. VDP improves generalization by capturing the variance of diverse verb appearances, while RAP enhances discrimination among visually similar verbs. The full model VDRP achieves the best performance across all settings, demonstrating the complementary benefits of both modules.
Group-wise modeling in VDP. To evaluate the impact of group-wise visual variance, we compare different configurations in the VDP module based on the number of verbs per group: (1) single global variance shared across all verbs (Global), (2) group-wise variance using clusters of verbs each (Group_95 and (3) clusters of 5 verbs each (Group_5) 5). As shown in Table 5] we observe that group-wise variance modeling generally improves over a global estimate, but performance varies depending the group size.
Impact of injection scale. We evaluate variance injection scale a E (0.01, 0.02, 0.10) to analyze its influence the delta scaling in Eq. Results across all settings summarized in Table 6 We

---

Table 5: Impact of group configuration in visual diversity-aware prompts (VDP). Best results bold.
Method
NF-UC
RF-UC
UO
Full
Unseen
Soen
Full
Unseen
Seen
Full
Unseon
Scen
Full
Unscen
Scen
Global
31.92
35.92
30.92
33.94
32.27
34.36
32.30
34.23
31.91
32.55
26.07
33.60
Group 3
32.43
35.89
31.57
33.15
29.57
34.04
32.70
34.76
32.29
32.75
26.25
33.81
Group 5
32.57
36.45
31.60
33.78
31.29
34.41
33.39
36.13
32.84
32.73
33.72
Table 6: Impact of injection scale on performance across zero-shot settings. Best results in bold.
NF-UC
RF-UC
UO
UV
Full
Unseen
Seen
Full
Unseen
Scen
Full
Unseen
Seen
Full
Unseen
Secn
0.01
32.03
35.29
31.22
33.51
30.58
34.24
32.81
35,52
32.27
32.48
25.97
33.54
0.10
32.02
35.80
31.07
33.16
29.86
33.98
32.81
35.11
32.35
32.54
24.73
33.81
0.02
32.57
36.45
31.60
33.78
31.29
34.41
33.39
36.13
32.84
32.73
26.69
33.72
Table 7: Effect of Gaussian perturbation across four zero-shot settings. Best results are in bold.
NF-UC
RF-UC
Method
HM
Full
Unseen
Seen
HM
Full
Unseen
Seen
W/O perturbation
33.57
32.49
35.68 36.45
31.69
31.76
33.12
29.83
33.95
W/ perturbation
33.85
32.57
31.60
32.77
33.78
31.29
34.41
UO
UV
Method
HM
Full
Unseen
Seen
HM
Full
Unseen
Seen
W/O perturbation
33.67
33.09
34.59
32.79
29.49
32.72
26.16
33.79
W/ perturbation
34.41
33.39
36.13
32.84
29.80
32.73
26.69
33.72
observe that 0.02 consistently achieves the best trade-off between unseen and performance.
This value was chosen to match the initialization scale of CLIP's context embeddings, ensuring stability as excessively large a causes over-perturbed context tokens and unstable training. Given that context tokens are highly sensitive to initialization, keeping the scale aligned promotes stable optimization while maintaining sufficient diversity.
Effect of Gaussian perturbation. We study the effect of Gaussian perturbation on prompt embeddings by varying the scale 3 E {0, 0.1). Table 7 summarizes the results across all zero-shot settings. Perturbation improves generalization in most unseen cases, confirming that stochastic sampling helps the model capture diverse visual realizations of each verb.
Retrieval strategy. We compare three retrieval methods for region concepts: Softmax, Top-3, and Sparsemax (Table (). Overall, Sparsemax performs most consistently, benefiting from its ability to suppress irrelevant concepts and emphasize informative ones.
Impact of augmentation scaling factor. We evaluate E {0.2, 0.5, 1.0] in Eq. 13 to study how strongly -eriovdderived cues contribute prompt adaptation. Results are reported in Table 9 A moderate scaling of = 0.2 provides the best overall balance, whereas excessive cue weighting (7 1.0) slightly degrades performance across multiple settings, likely due noisy region concepts.
4.4
Qualitative results
Fig. shows qualitative examples of region-specific concept generation and retrieval. Given verb prompt (e.g., "licking an object"), our method uses LLMs (e.g., LLaMA-7B (48) and GPT- 4 49 to generate K region concepts, capturing fine-gra tee-ggaaied concepts such as facial expression or visual patterns. While LLaMA-7B provides diverse concept descriptions, we observed noise and redundancy object-related concepts; therefore, GPT-4 was additionally used (see Supplementary for details). As illustrated, similar verbs like "licking" and "eating" are associated with distinct region-speciffe the discriminability.
the model better distinguish subtle visual differences, enhancing

---

Table 8: Comparison of retrieval strategies for region- aware prompts (RAP). Best results in bold.
NF-UC
RF-UC
UV
Method
Full
Unseen
Full
Unseen
Full
Unseen
Seen
Full
Unseen
Softmax(+)
32.01
35.93
31.03
33.32
30.37
34.06
32.91
35.83
32.32
32.57
25.45
33.74
Top-3
31.97
35.85
31,00
33.35
30.80
33.99
32.94
33.57
32.41
32.47
25.85
33.55
Sparsemax()
32.57
36.45
31.60
33.78
31.29
34.41
33.39
36.13
32.84
32.73
26.69
33.72
Table 9: Effect of augmentation scale y on performance across all settings. Best results in bold.
NF-UC
RF-UC
UO
UV
Full
Unseen
Seen
Full
Unseen
Seen
Full
Unseen
Seen
Full
Unseen
Seen
0.5
32.14
35.92
31.19
33.52
30.73
34,21
33.08
35.17
32.66
32.72
27.06
33.64
1.0
31.85
35.15
31.02
33.48
30.67
34,18
32.60
34.74
32.17
32.54
25.92
33.62
0.2
32.57
36.45
31.60
33.78
31.29
34.41
33.39
36.13
32.84
32.73
26.69
33.72
"Fur the "À photo of person licking object". give K short visual concepts for the houmantohjochunion region"
Retrieved concepts
Person tcegue toward the slighlly releced.
Penon's lips mouth are close the olpect Traces of szliva or spit visible.
LLM
A close up shones person tisting the oly with the lip their
Licking object.
Ch
'Tongue extended curled."
Humen
'Chooks are slighly puffed out.
Retrieved concepts
'Lips are parted and relaxed.
Person
their
Chooks putfed oach
'Objject
glussy surfacee.
Object
'O'jject's surface has a distinct flavor or sceet."
Penuu's month seer the object. visilly being comaumeed from the olject.
'Object''s shape or contour altered licking.
being touched vonuue tasting
Union
Eating an object
The pesse's closed is cssartation Bay wd. The wrinkhed gleasuro tlyy the obrect
person loans
surface
object is partially as licked by person."
rejien
(A) Generated region concepts
(B) Qualitative results of concepts retrieval
Figure 4: Qualitative examples of region concept generation and retrieval. (A) Given a verb prompt and region type, an LLM generates K region concepts per verb. (B) Retrieved concepts for "Licking" and "Eating" highlight subtle region concepts that help disambiguate visually similar interactions. Concepts are color-coded by region: blue (human), red (object), yellow (union).
5 Conclusion
We present VDRP, prompt learning framework for zero-shot HOI detection that addresses the visual complexity of interaction, including intra-class diversity and inter-class entanglement among verb classes. To this end, we propose visual diversity-aware prompt learning that injects group- wise visual variance into the prompt context and applies Gaussian perturbation, enabling prompt embeddings better reflect diverse visual appearances. We also introduce region- aware prompt augmentation, which augments diversity- aware prompts with region-specific concepts from the human, object, and union regions to improve verb discrininability. Extensive experiments on HICO- DET show that our method outperforms prior work across four zero-shot settings while maintaining high parameter efficiency. These results underscore the value combining distributional modeling and region-level augmentation for generalizable zero-shot HOI detection.
10

---

6 Acknowledgments
This work was partly supported by Korea Research Institute for defense Technology planning and advancement- Grant funded by Defense Acquisition Program Administration (DAPA) (KRIT-CT-23-- 021, 45%), by the Institute of Information & Communications Technology Planning & Evaluation (ITTP)-TTCC (Information Technology Research Center) grant funded by the Korea government (MSIT) (LITP-2025 -0---22--002-000------ 2024-00436877, 45%) and by the National Research Foundation of Korea (RS-2022-NR068758, 10%). We thank Jaewoon Byun for helpful discussion.
References
[1] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Detecting human-object interaction via fabricated compositional learning. In CVPR, 2021.
[2] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Affordance transfer learning for human-object interaction detection. In CVPR, 2021.
[3] Zhi Hou, Xiaojiang Peng. Yu Qiao, and Dacheng Tao. Visual compositional learning for human-object interaction detection. In ECCV, 2020.
[4] Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng. Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In CVPR, 2020.
[5] Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Efficient two-stage detection of human- object interactions with novel unary- mary-pairwise transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20104 2014220117 12, 2022.
[6] Frederic Z. Z.Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould. Exploring predicate visual context in detecting of human-object interactions. Proceedings of the IEEE/CVF international conference computer vision, pages 10411-10421, 2023.
[7] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise human- object interaction detection with image-wide contextual information. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10410-10419, 2021.
[8] Bumsoo Kim, Tacho Choi, Jaewoo Kang, and Hyunwoo J Kim. Uniondet: Union-level detector towards real-time human-object interaction detection. In European Conference on Computer Vision, pages 498-512 19. Springer, 2020.
[9] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to- end human- object interaction detection with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 74-83, 2021.
[10] Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng, and Yang Liu. Efficient adaptive human-object interaction detection with concept-guided memory. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6480-6490, 2023.
[11] Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models. Advances in Neural Information Processing Systems, 36:739-751, 2023.
[12] Guangzhi Wang, Yangyang Guo, Ziwei Xu, and Mohan Kankanhalli. Bilateral adaptation for human-object interaction detection with coclusion-robustnesss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27970-20 -79800, 2024.
[13] Jihwan Park. SeungJun Lee, Hwan Heo, Hyeong Kyu Choi, and Hyunwoo J Kim. Consistency learning via decoding path augmentation for transformers in human object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
pages
[14] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In JCML, 2021.
11

---

[15] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022.
[16] Sehyung Kim, Chanhyeong Yang, Jihwan Park, Tachoon Song, and Hyunwoo J Kim. Super- class guided transformer for zero-shot attribute classification. arXiv preprint arXiv:2501.0555 055288. 2025.
[17] Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding-box labels. ECCV, 2022.
[18] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9))2337-2333 2022.
[19] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for rision-languaa models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816-1682. 2022.
[20] Jooyeon Kim, Eulrang Cho, Sehyung Kim, and Hyunwoo J Kim. Retrieval-augmented open- vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17427-17436, 2024.
[21] Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard-hoi: Benchmarking few-shot visual reasoning for human-object interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19065, 2022.
[22] Jie Yang, Bingliang Li, Ailing Zeng, Lei Zhang, and Ruimao Zhang. Open-world human-object interaction detection via multi-modal prompts. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 16954-16966 2024.
[23] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vikt: Simplify association and enhance interaction understanding for hoi detection. In CVPR, 2022.
[24] Ting Lei, Shaofeng Yin, Yuxin Peng, and Yang Liu. Exploring conditional multi-modal prompts for zero-shot hoi detection. In European Conference on Computer Vision, pages 1-19. Springer, 2024.
[25] Qinqian Lei, Bo Wang, and Robby Tan. Ez-hoi: Vlm adaptation via guided prompt learning for zero-shot hoi detection. Advances in Neural Information Processing Systems, 37:55831 2024.
[26] Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li. Clip4hoi: Towards adapting clip for practical zero-shot hoi detection. Advances in Neural Information Processing Systems, 36:45895 45906, 2023.
[27] Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoielip: Efficient knowledge transfer for hoi detection with vision-language models. In CVPR, 2023.
[28] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Mattco Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 2020.
[29] Bingjie Xu, Yongkang Wong, Jumnan Li, Qi Zhao, and Mohan S Kankanhalli. Learning to detect human-object interactions with knowledge. In CVPR, 2019.
[30] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[31] Jongha Kim, Jihwan Park, Jinyoung Park, Jinyoung Kim, Sehyung Kim, and Hyunwoo J Kim. Groupwise query specialization and quality-aware multi-assignment for transformer-based visual relationship detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 28 1110-2816999 2024.
12

---

[32] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. 2. Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
[33] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
[34] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International conference on machine learning, pages 4904-4916. PMLR, 2021.
[35] Jinguo Luo, Weihong Ren, Weibo Jiang, Xi'at ai Chen, Qiang Wang, Zhi Han, and Honghai Liu. Discovering syntactic interaction clues for human-object interaction detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 28212-28222, 2024.
[36] Eulrang Cho, Jooyeon Kim, and Hyunwoo J Kim. Distribution-aware prompt tuning for vision- language models. In Proceedings of the IEEE/CVF international conference computer vision, pages 22004-22013. 2023.
[37] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5206-5215, 2022.
[38] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modua prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19113-19122, 2023.
[39] Wentao Bao, Lichang Chen, Heng Huang, and Yu Kong. Prompting language-infornng distribution for compositional zero-shot learning. In European Conference Computer Vision, pages 107-123. Springer, 2024.
[40] Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. Advances in Neural Information Processing Systems, 37:2001-2025, 2024.
[41] Jinyoung Park, Juyeon Ko, and Hyunwoo Kim. Prompt learning via meta-regullarization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26940-26950, 2024.
[42] Dongjun Lee, Seokwon Song, Jihee Suh, Joonmyeong Choi, Sanghyeok Lee, and Hyunwoo J Kim. Read-only prompt optimization for lision-languag few-shot learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1401-1411, 2023.
[43] Songlin Dong, Zhengdong Zhou, Chenhao Ding, Xinyuan Gao, Alex Kot, and Yihong Gong. Diversity covariance-ane aware prompt learning for lsioonlanguuae models. arXiv preprint arXiv:2503.01531, 2025.
[44] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[45] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[46] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim. and Percy Liang. Concept bottleneck models. In International conference machine learning, pages 1338-5348 PMLR, 2020.
[47] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 19187-19197,2099
13

---

[48] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13371, 2023.
[49] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (rXiv:2333308877: 2023.
[50] Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International conference on machine learning, pages 1614- 1623. PMLR, 2016.
[51] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[52] Mingrui Wu, Jiaxin Gu, Yunhang Shen, Mingbao Lin, Chao Chen, and Xiaoshuai Sun. End-to- end zero-shot hoi detection via vision and language knowledge distillation. In Proceedings of the AAAI Conference on artificial intelligence, volume 37, pages 2839-2846, 2023.
[53] Ila Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051011 2017.
[54] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[55] Frederic Z. Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object interactions. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 13319-13327, October 2021.
[56] Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint 1rX1::1505.0447,,00 2015.
[57] Hangjie Yuan, Shiwei Zhang, Xiang Wang. Samuel Albanie, Yining Pan, Tao Feng, Jianwen Jiang, Dong Ni, Yingya Zhang, and Deli Zhao. Rlipv2: Fast scaling of relational language- image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 21649-21661, 2023.
[58] Danyang Tu, Wei Sun, Guangtao Zhai, and Wei Shen. Agglomerative transformer for human- object interaction detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 21614-21624, 2023.
[59] Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural-logic human-object interaction detection. Advances Neural Information Processing Systems,
2023.
[60] Sedigheh Eslami and Gerard de Melo. Mitigate the gap: Improving cross-modal alignment in CLIP. In The Thirteenth International Conference on Learning Representations, 2025. URL
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state the main contributions and scope in the abstract and introduc-
tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our method as the Gaussian assumption in visual space and the reliance on -----reeertted concepts-in in the supplemental material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
Our work does not include any theoretical results or formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide implementation details and training configurations in the supple- mentary material to ensure reproducibility all main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will release code and data upon publication, along with instructions in the supplementary material to reproduce all main results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic/guiddes/oddeubbbmsss www.eeeeuuuuisssoooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppplgguuuiaaeeeae Stteeee//ooSSSSeissSioica www.ssssiicgcyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the training and evaluation protocols Section 4 and provide additional details on implementation and hyperparameters in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Our evaluation follows standard zero-shot HOI detection benchmarks with deterministic protocols, so statistical significance is not reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report training time and GPU specifications in the supplementary material to support reproducibility.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't iidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.. wttpppsss.cpppppppppcceeepeepeee i
Answer: [Yes]
Justification: We confirm that our research adheres to the NeurIPS Code of Ethics, including responsible data usage and transparency methodology and reporting.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We briefly discuss socictal benefits and risks the conclusion under "Societal Impact."
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work does not involve models or data with high risk of misuse, so this question is not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use publicly available datasets (HICO-DET) and pretrained models (e.g.. CLIP, DETR) that are properly cited in the paper, and their licenses and usage terms are fully respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We properly credit and cite all existing datasets, models (e.g.. CLIP, HICO- DET), and related assets used our work, and follow their license terms as described the paper and supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our study does not involve human subjects and therefore does not require IRB approval.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We use large language model generate region-level concepts (e.g., LLaMA- 7B) as part of our retrieval- based prompt augmentation module.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy htttg:///eaarrppcccc////ofeeeeee for what should or should not be described.
21

---

Appendix
A Implementation details
We conduct all experiments using PyTorch with mixed- -reedspre training. For the base model using CLIP ViT-B/16, we train on two NVIDIA GeForce RTX 3090 GPUs with a batch size of 8 for 12 epochs. We use the AdamW W [53 optimizer with initial learning rate of 1 x 10-3, decayed to 1 x 10-4 using a cosine scheduler and weight decay of 8. We report the best results after applying weight decay. The framework adopts CLIP ViT-B/16 as the visual encoder, where the adapter applies a bottleneck transformation from dup = 768 to ddown = 64, followed by projection into a shared embedding space of dimension d 512. Human-
detection is performed by frozen DETR model with Resnet50 [54], and detections with confidence below the threshold
0.2 are discarded. The context embedding consists of Nou. 24 learnable tokens, each initialized from Gaussian distribution with standard deviation 0.02. To incorporate group-wise variance, we inject a modulation 0.02) into the context embedding and apply Gaussian perturbation to the
vector (scaled by resulting prompt embedding with a noise scale 3 = 0.1. For region-aware prompt augmentation, we generate K 10 region concepts per verb and region type (human, object, union) using LLaMA- 7B 48 and ChatGPT-4 49). These are encoded by the frozen CLIP text encoder and aggregated via Sparsemax [50] form concept vector, which is added the prompt embedding with scaling factor = 0.2. We additionally conduct experiments with scaled-up version using CLIP ViT-L/14 as the visual encoder. In this setting. the adapter transforms dup = 1024 ddown 64, with the final embedding dimension set to d = 768. Due to memory constraints, these experiments are performed on two NVIDIA RTX 6000 Ada Generation GPUs with reduced batch size of 4 per GPU.
B Details of our architectures
Details of spatial head. To enhance the union-region representation with geometric priors, design spatial head that encodes the spatial relationship between the human and object bounding boxes and fuses it with the region features, inspired by prior works [55] 5 6 Given human box
bn = 1 V1: 22 M2] and an object box b, = [r', v1, 222, V2l. widths, and heights:
we compute the center coordinates,
I1
+ F2
51
1/2
x
+
12
V/
+ 1/2
C₁
C.
2
2
Wh
T1,
hn
U22 V1.
Wo = 2/2 211
ho = 3/2 3í
We then extract spatial features including normalized positions, relative box areas, aspect ratios, intersection-owe -overiooooooe--iiiioo (loU), and direction-awars relative distances:
Car Chzl
Coo Chyl
d
dy
wh +
hip +
These values are concatenated and passed through a small feedforward network to produce a spatial encoding vector Eg. To incorporate this geometric context, we apply multi-modal fusion module ÓMMP(-) that combines the human and object features [Xn; X.] with the spatial encoding:
fs = dMMF ([x;; X.), Es).
Finally, the spatial query fs is concatenated with the union feature xg. and passed through a projection layer to obtain the final spatially-enhanced representation:
Proj ([xa; fs]):
which is used for verb classification with visual diversity and region-aware prompts.
Details of sparsemax. To select region-level concepts relevant to each human-object interaction, we adopt the Sparsemax [50] activation function in our region-aware prompt augmentation module. Unlike Softmax, which produces dense probability distributions where all entries are non-zero, Sparsemax enables sparse selection by assigning exact zeros to irrelevant entries. This property is particularly useful our setting, as only a subset of the K generated region concepts are semantically aligned with the visual feature.
22

---

Formally, given a score vector E R representing the cosine similarity between a region feature and K region concept embeddings we define Sparsemax as projection onto the probability simplex:
K Σ k=1
Sparsemax()) == arg min
|a s||?, A T aEAR
Ok =
1, aK 2
0
(14)
This formulation yields closed-form solution that projects the input vector onto the probability simplex A resulting in a sparse vector a where low-scoring entries are assigned exact zeros. We apply Sparsemax to the similarity scores between each region feature and the corresponding concept pool, allowing the model to focus on the most informative region concepts while ignoring noisy or irrelevant ones.
C
Limitations
While our method demonstrates strong performance across multiple zero-shot HOI detection settings, several broader limitations remain. First, the region-aware prompts (RAP) module builds upon region-level concepts generated by large language models (LLMs). Although effective in capturing contextual semantics, LLM-based concepts may be noisy or misaligned with visual concepts due to inherent limitations in language-vision grounding. Improving robustness through confidence-aware filtering or vision-aligned concept refinement promising direction. Second, our framework builds on prompt learning, which presents structural challenges when applied to a large number of classes or diverse interaction types. Prompt representations can be sensitive to scaling strategies, initialization, and optimization dynamics, making them less stable in settings with limited data or rare class compositions. This reflects broader limitation of prompt- based models, where generalization can be affected by prompt granularity, representation collapse, or lack of compositional structure. Future directions may include more robust prompt initialization, adaptive scaling mechanisms, or compositional prompt construction to improve stability and generalization.
D Broader impacts
Our work builds upon vision-language models (VLMs) such as CLIP, which are trained on large-scale web-scraped image-text pairs. As result, our model may inadvertently inherit biases present in the pretraining data, such as cultural stereotypes or imbalanced representations across demographic groups. While method aims to improve generalization in zero-shot HOI detection, deployment in real-world applications should done with caution- caution-particularyy particularly in contexts involving surveillance, behavioral analysis, human activity interpretation. Furthermore, the retrieval-based region prompt augmentation using large language models (e.g., LLaMA-7B and ChatGPT-44, which may also reflect biases or hallucinated associations. Misinterpretation of human-object interactions in sensitive domains (e.g., law enforcement or healthcare) could lead to harmful outcomes such biases are not properly addressed. To mitigate potential misuse, we recommend restricting deployment controlled environments with human oversight, and further advocate for evaluating fairness metrics across subpopulations when applying the model downstream applications.
E Further experiments
V-COCO datasets. V-COCO [66] dataset has 80 object categories derived from COCO datasets [51] and includes 29 interactions, resulting 263 HOI compositions. The number of image samples is 10,39 (5,400/4,964 for train/test).
Fully supervised settings. We further evaluate our method under the fully supervised HOI detection setting on HICO-DET and V-COCO, as summarized in Table [0] Among two-stage methods, our model achieves the best performance on HICO-DET, with an mAP of 39.07. surpassing the prior state-of-the-ee art EZ-HOI [25]. While one-stage methods such as UniHOI and AGER benefit from end-to-end optimization and full supervision of all components, they typically rely on larger networks and longer training schedules. In contrast, our two-stage approach uses fewer learnable parameters and is designed primarily for zero-shot generalization, which likely limits the effectiveness of group wrse variance modeling and hinders the learning of cross cross-class structure. On
23

---

Table 10: Sate-oof-the-ar art comparison on HICO-DET and V-COCO under the fully-supervised setting.
Bold indicates the best best-performing method within each group (one-stage vs. two-stage).
Method
HICO-DET
V-COCO
Full
Rare
Nonrare
AP
AP
One-stage Methods
GEN-VLKT 23
33.75
29.25
35.10
62.4
64.5
HOICLIP 27
34.69
31.12
35.74
63.5
65.0
RLIPV2 [57]
35.38
29.61
37.11
65.9
68.0
AGER (58]
36.75
33.53
37.43
65.7
67.9
LogicHOI 59
35.47
32.03
36.64
64.6
65.6
UniHOI IMI
40.06
39.91
40.11
65.6
68.3
Two-stage Methods
UPT
32.62
28.62
33.81
59.0
ADA-CN 10
38.40
37.52
38.66
58.6
64.0
CLIP4HOI 266
35.33
33.95
35.75
66.3
CMMP 24
38.14
37.75
38.25
64.0
EZ-HOI 25
38.61
37.70
38.89
60.5
66.2
Ours
39.07
39.08
39.06
60.6
66.2
Table 11: Zero-shot HOI detection results under four splits-NF-UC, RF-UC, UO, and UV- the scaled CLIP (ViT-L). HM is harmonic mean (HM) between Unseen and Seen. Best results are shown in bold, and second best underlined.
NF-UC
RF-UC
Method
HM
Full
Unseen
Seen
HM
Full
Unseen
Seen
UniHOI
30.40
31.79
28.45
32.63
30.76
32.27
28.68
33.16
CMMP
34.50
35.13
33.52
35.53
36.69
37.13
35.98
37.42
EZ-HOI 25
35.38
34.84
36.33
34.47
35.73
36.73
34.24
37.35
Ours
36.83
36.46
37.48
36.21
37.58
38.13
36.72
38.48
UO
UV
Method
HM
Unseen
Seen
HM
Full
Unseen
Seen
UniHOI 11
25.17
31.56
19.72
34.76
30.50
34.68
26.05
36.78
CMMP 24
37.83
36.74
39.67
36.15
33.75
36.38
30.84
37.28
EZ-HOI 25
37.06
36.38
38.17
36.02
32.84
36.84
28.82
38.15
Ours
38.41
37.81
39.36
37.50
34.31
37.18
31.16
38.16
the V-COCO benchmark, our model achieves 60.6 and 66.2 AP under Scenario 1 and Scenario respectively- comparable other two-stage models. We attribute this the small number of verb classes (24 vs. 117 in HICO-DET) and limited dataset size, which reduce the effectiveness of group-wise variance modeling and cross-class structure learning. Overall, these results demonstrate that although our method is designed to address zero-shot HOI detection, it generalizes well to fully supervised settings, retaining strong performance Scaled-up CLIP (ViT-L) setting. To assess the effect of scaling the vision backbone, we evaluate our method using the CLIP ViT-L/14 encoder in place of the default ViT-B/16. As shown in Table our model consistently improves performance across all four zero-shot evaluation Splits-NF-UC RF-UC, UO, and UV. Notably, our method achieves the highest harmonie mean (HM) across all settings, reflecting balanced generalization to both seen and unseen verb compositions. While prior methods such CMMP 24 and EZ-HOI (25] exhibit competitive results on specific splits, their performance fluctuates across evaluation scenarios. In contrast, our scaled-up model maintains stable gains throughout, demonstrating that visual diversity-aware prompt learning and region-aware prompt augmentation remain effective even when paired with high-capacity vision-language encoders. These results highlight the scalability of our framework even when scaled to stronger backbones.
when supervision is abundant.
24

---

Table 12: Comparison of using both mean and variance (11, o²) vs. variance-only (a in the VDP across four zero-shot settings. Best in bold.
Setting
Stats
Full
Unseen
Seen
(n, o²)
32.03
35.75
31.10
NF-UC
o² only
32.57
36.45
31.60
(R, o²)
33.19
30.34
33.91
RF-UC
o² only 33.78
31.29
34.41
(R, o²) r² only 33.39
33.03
34.87
32.66
UO
36.13
32.84
(n, o²)
32.59
25.33
33.77
UV
o² only 32.73
26.69
33.72
Table 13: Ablation study comparing region branches (Human, Object and Union) and the full model (H+O+U) under different zero-shot settings. Best in bold.
NF-UC
RF-UC
Branch
HM
Full
Unseen
Seen
HM
Full
Unseen
Seen
Human
33.27
32.03
35.77
31.09
32.19
33.34
30.53
34.04
Object
33.48
32.17
36.16
31.17
31.90
33.32
29.92
34.17
Union
33.26
32.17
35.40
31.37
32.01
33.23
30.28
33.96
H+O+U
33.85
32.57
36.45
31.60
32.77
33.78
31.29
34.41
UO
UV
Branch
HM
Full
Unseen
Seen
HM
Full
Unseen
Seen
Human
34.08
32.90
36.11
32.26
29.35
32.66
25.97
33.75
Object
33.39
32.65
34.61
32.26
29.64
33.07
26.15
34.20
Union
33.39
32.70
34.52
32.34
29.55
32.60
26.36
33.62
H+O+U
34.41
33.39
36.13
32.84
29.80
32.73
26.69
33.72
F
More ablation studies
Effect of mean vs. variance in VDP. To evaluate the contribution of distributional statistics in our visual diversity-aware prompts (VDP) module, we compare two variants: one that uses only group-wise variance (a²), and another that combines both the group-wise mean (μ)) and variance (a²) through concatenation followed by projection. Table reports results across all four zero-shot settings. Overall, find that the variance-only variant consistently matches outperforms the mean-variance combination especnalllyy on the Unseen splits in all settings. For instance, NF-UC, using variance alone yields the highest unseen mAP of 36.45, compared to 35.75 75 with μ+o This suggests that variance alone serves as a stronger signal for modeling intra-class visual diversity, particularly under zero-shot generalization conditions. Prior studies such as Zhu et al. [40] also report that centtree prompt representations improve generalizability by capturing the dispersion of visual features without overfitting sample means. Our results align with this observation, indicating that the inclusion of the mean may introduce redundant or unstable signals signals-especially for low-shot noisy verb clusters, where the class prototype is poorly defined. An exception arises in the UV setting, where the combined variant slightly improves performance on seen metrics. Nonetheless, the overall trend supports the effectiveness and robustness of variance-only modeling for capturing distributional diversity in the prompt space.
Effect branches in RAP. We compare the individual contributions of the human, object, and union region branches under four zero-shot HOI detection settings. The results are summarized in Table [3] Across all settings, the full model consistently outperforms each individual branch, confirming the complementary nature of region-level concepts. Notably, the proposed combination achieves best unseen mAP in all settings, demonstrating the effectiveness region-aware prompts in enhancing discriminability. These findings highlight that human and object concepts capture distinct complementary contextual signals, while the union branch provides holistic spatial grounding that reinforces interaction understanding.
25

---

Table 14: Zero-shot verb variance robustness under the UV setting. We remove the top-3 semantic neighbors for each unseen verb before computing visual variance.
HM
Full
Unseen
Seen
W/O Top-3
29.48
32.64
26.21
33.69
W/ Top-3
29.80
32.73
26.69
33.72
Table 15: Sensitivity analysis of 7 Sparsemax under four zero-shot settings. Best results are in bold.
NF-UC
RF-UC
value
Full
Unseen
Seen
Full
Unscen
Scen
0.10
32.23
35.85
31.33
33.59
30.92
34.26
0.05
32.38
35.76
31.53
33.10
29.56
33.99
0.00
32.57
36.45
31.60
33.78
31.29
34.41
UO
UV
value
Full
Unseen
Soen
Full
Unseen
Scen
0.10
32.90
35.84
32.31
32.85
26.95
33.81
0.05
32.95
34.63
32.62
32.73
25.91
33.84
0.00
33.39
36.13
32.84
32.73
26.69
33.72
Table 16: Robustness of VDP under few-shot variance sampling (N, = 5). Best results are in bold.
NF-UC
Full
Unseen
Seen
RF-UC
Full
Unseen
Seen
32.23 All 32.57
Few-shot
35.90 36.45
31.31
Few-shot
33.40 33.78
30.59 31.29
34.10
31.60
34.41
UO
Full
Unseen
Seen
UV
Full
Unseen
Seen
Few-shot 32.55
34.75
32.11
Few-shot
32.89
25.88
34.04
All
33.39
36.13
32.84
32.73
26.69
33.72
G Robustness of visual diversity-aware prompt learning
Zero-shot verb variance robustness. To evaluate the robustness of group-wise variance estimation for unseen verbs, we conduct additional experiment under the UV setting. Specifically, we remove the top-3 semantic neighbors (based CLIP similarity) for each unseen verb from the training set before computing the group-wise visual variance. This setup simulates a scenario where certain verbs lack semantically related support, allowing to analyze the stability of variance estimation under reduced contextual guidance. As shown in Tab 14, the removal of neighboring classes results in only marginal performance drop in unseen verbs (-0.48 mAP). This indicates that the proposed grouping mechanism maintains its generalization ability even when semantic support is limited.
r-SSpasemax sensitivity. To investigate the effect of sparsity calibration in concept retrieval, we introduce r-Sparsemax(()). where all values below a threshold T (0.0, 0.05, 0.1} are zeroed post-Sparsemax(-). This modification controls the degree of sparsity in the region-concept selection process. The results in Tab [5 show that excessive pruning (7 = 0.05) slightly decreases overall performance, while higher sparsity (r 0.10) recovers seen-class precision the cost of unseen generalization. The default 0.0 provides the most balanced outcome, indicating that the original Sparsemax(-) formulation already achieves effective level of adaptive sparsity.
Robustness of visual variance under few-shot sampling. We further test the stability of visual variance estimation when fewer visual samples are available. For this experiment, group-wise covariance matrices are computed using only five randomly sampled features per verb (N, = 5), compared to using all available samples. As summarized in Tab the performance reduction across all settings minimal (typically less than 0.6 mAP), demonstrating that VDP remains robust even under limited sample diversity.
26

---

Before training
After training
CMMP
VDRP
0.5
0.366
0. 105 20.031
0. 4.4 1:0
937 133
0. 518 105
D. 550
090
Visual
Prompt
Visual
Prompt
Visual
Prompt
Figure 5: Pairwise inter-class distances between prompts and visual features. We report the average pairwise cosine distance (i.e., D Eeyy cos(2z, 2o)(2;,z)]] across verb classes, for both visual and prompt embeddings, before and after training. Visual features are extracted from union regions. Before training, we use the CLS token from the CLIP visual encoder applied to cropped union region images. After training, we follow the Rol-Align feature extraction pipeline consistent with the two-stage method (i.e., pooling patch embeddings within the union box). For each verb class, medoid is selected among all union features to represent its prototype. While prompt embeddings are initially collapsed with low diversity, VDRP maintains balanced and aligned distribution relative to visual features, unlike CMMP which over-separates prompts and disrupts cross-modal structure.
H Inter-class alignment of visual and prompt representations
To assess inter-class alignment between visual and prompt representations, we analyze the distri- butional structure of each modality after training. Specifically, we examine whether our visual diversity-aware prompt learning preserves class-level relationships that are consistent with those observed in the visual embedding space. We measure the pairwise cosine distance between class-level prototypes in both visual and prompt embedding spaces:
D = Ei cos(2,, 2;)1.
where 2; denotes the prototype embedding the i-th verb class. For the visual side, we extract features from the union region of each verb instance using the CLIP visual encoder. Before training, we use the CLS token of the encoder applied directly to the union-cropped image. After training, adopt region-specific pooling approach: patch embeddings within the union Rol-Align 44 aggregated to match the training architecture of our method. To mitigate over- wer-smoothing effects from averaging, we define the visual prototype for each class as the medoid-the instance whose embedding minimizes the average distance to others within the same class. For the prompt side, we collect the final trained verb embeddings for each class. Fig. 5 presents the average inter-class distance for both modalities across three conditions: (1) before training, (2) after training with CMMP 24 and (3) after training with our method (VDRP). We observe that CMMP significantly increases prompt diversity (0.937) while visual diversity remains moderate (0.420), indicating modality mismatch. In contrast, VDRP maintains comparable level of diversity in both modalities (prompt: 0.550, visual: 0.518), suggesting improved distributional alignment. This outcome implies that our prompt learning strategy though primarily guided by intra-class visual variance can indirectly induce structured inter-class layout without wer-ampliying semantic separation. We draw inspiration which emphasize distributional approaches to improving visual-promp alignment. Motivated by this perspective, we interpret the improved alignment as natural byproduct of our variance-aware design, even though cross-modal matching was not explicitly enforced.
from recent works (60) 36
R9
40
27

---

Discussion. This inter-class alignment result further substantiates the novelty of our visual diversity- aware design. Unlike prior distribution-baat based prompt learning methods such as ProDA [37 and DAPT [36,, which regularize textual embeddings, VDP explicitly grounds the prompt distribution in the visual modality by injecting group-wise variance extracted from union-region features. This design enables structured yet balanced prompt distributions that mirror the visual embedding space, achieving semantic coherence without explicit regularization losses. Consequently, VDP naturally promotes inter-class consistency and zero-shot generalization key objectives that motivated our dual-module framework.
I Qualitative Results
Human region concepts
Object region concepts
region concepts
Oloject
Retrieved concepts
Retrieved concepts
(0.38) Knees am beer, with Jeet Eatt tte grund.
(0 40) Mouth open and echiling forsefully
(0.31) Buttoeks are sightly nised off the ground
(0:30) Person is blowing air toward the olgect
[0.11] Auilas Danss, with feel peisting uoward las alject
(0.14) Nose wiikkled concetiation
(0.54) Sandle or soat is use.
(0 (2) Olty ect appoars to be lifted or moved by sir:
(0.1%) Ofjectt appaars a broycle.
[0 Oly Olgect' posilicn apreers unstable se shilting.
(0.16) Frams is good condition.
(0:21] D2 eet's shage form changss due to pressure.
551 Tacokpects being placced deiibeate dovervud eustios. the oe object
(C-58) The pamoels lips an patted they Mow.
Straddling an object.
12.151 po
Blowing an object.
[6:38] penas hivving vir the rijjct with wonth
(6.151 pessoss doedo ar sligatyy pabted
upright.
38) Hands positicces near
Lags espanded
padlling
Shouldars rilased and nlights lunched. Person grouping lig object
(3.21)
(0.22) Object
(3 451 Object has domls serodynemtic slape,
(0.58) Obgect distinctive tive shape design.
saddle for
(0 30) Fodd being insereed nemovved
[0 27)) I billnodd or contered within altact.
[0 26) Load socured with
(6.17) Loval
enkonari lucall..
clips.
(0.33) The abject is baing lowaned anoo veluca.
Riding an object.
(0.2)A A pessa is soaud tacokgost, with tictilgg entandodia
Loading an object.
(0:26] The pensers farm freoeed interily on the trak hend
Bock (aea
(0.24) The object being lifed laso position the persor's amn..
(0.17) passals an maae pooi Deoobjact
Human reglin concepts
Object regice concepts
Union region concepes
Human
box
Retrieved concepts
Retrieved concepts
(0.42) Wrisks ane flouel during the wwing
(8 99) Arrsa are excended and pullina on cars. (n (0.30) ags sm how and firt are planted fimmly the plaiform.
(U Hands are moving in cirouarr motion.
13) Hair ma be blown bock by the momonn
(I 14) Book
smagght and Shcing forwand
(B42) Olrectss wejast or momentam creace subtic are cano exteging nation (0.25) Switping modis cento station blke arvand
(0 67) Bowing mattoon craaie rippls waker (0 (0) Ulject postioond at an angle the water
(n 13) Ulject pornally suhmerged woner
(2:3) The tand
hand is prtznea the ote flmelty iniana
82 The
tucked to belance water reflective the undiing
Swinging an object.
(2335 ponce's cocaded is and objact paik
Rowing an object.
16 panson indiging dojee por
Mouth putteed they inholle
15) planed aahilty
poorkling
suggesinng
salisfaction
(6 Flagers graud wide, using man more leverage
(0-42) Liquid level in the object Jos. (0.20) Pesson's lps cuiled back in motion.
(b 64) Obsect's surfice wet or ditty from dragaing. (t 36) Uly ect partally or fully diagged away froiz original praition.
(0. Objoct is likel tovoade the parsoris moathh
(A Surfwce of objoct scaffied disturbeed
Drinking with an object.
(1-40) The door as drick Adbek vidit
(1.00 The parsodk haad totteed dightty they drink. binggt epping
Dragging an object
fex pall thcutted.
the oeject beawing jettig as persue palls alknng
181 The pecsens
in alon, shyslas
Figure 6: Region-aware concept retrieval results (1/2), Each row shows retrieved human, object, and union concepts. Concepts are color-coded by region: blue (human), red (object), yellow (union).
We present qualitative examples in Fig- 6 and 7 to illustrate how specifc concepts contribute to verb classification. Each example visualizes the top- wopgheighted concepts retrieved from three region branches-human, object, and union-borl based on parrsmmax-normallan concept weights. The full
28

---

Human region concapes
Object regice concepta
Union region concapte
Human box
Obhject box
Retrieved concepts
Retrieved concepts
toe slighlly
Person holding glass cap
cotending los omriet obrict.
Lineer kay stright for on greandl Back is losniigh shuilles sclasod.
angled object.
is slightly rused.
50) appears midet suspendedd Object's appcars vibrating trebiingg
body
celebration
approcketion
Objus''s pusition dhengasor shills.
(0 Olgect distinctive stape design.
(B:43) pensers amsic as vabble they tese the chject.
Kicking object.
(2.1) Tacobract tutna in ddagsnal dircctiss
(2:4) The pences lockuled back beflee raaling
Toasting object.
(8.14) the objacks sarface alono soitly the light ed
(8 29) The pensels fox in the light of object
lo amads
deruch
peroort glass
(0: 9) Shoulders see relveed and dighttyy bunchall
(0.59) Hands are ralsed and gesturina loward the object.
(0.7) Chin is siightly risaa, on dirceiigg sttention.
(0 35) Aras wrap around the object. enveloping it.
(0 24) Ches proseed spains the object, host beating
(0 34) Ensire body is lense with crastion.
egaind
Ofjecti in positicnaal near the persrn's hand.
Objects daspe crptesized by paneets
(3:22) Onject la penitional an sye level
Object's sanfacc warn touch.
(0:17 77] Person's hand is pooning towards the onject.
(0) Object curved se roundel slape.
00.44 Osject is partially abacured the personis badl..
(I Persnels thoee clase to obgoet's surbhon
(n The persokk ayu sre thod loueryy on objject
(3 53 The object andieata is the ponser' urra, vas cuit banck ding awaaty.
Directing object.
specome
Hugging object.
(1140 The parsatk body lingange ougges stpeccy aeporsace.
sease of warcih (a The peosezo lasd intiiniiaidddd scida objuets surface
lummnn region concepts
Object region comcupts
Union region concupis
Jummn box
Obhject box
Retrieved concepts
Retrieved concepts
Parson faat thei back
54) Weight distrbined beth Chee expands eahrly while obyect tihed
Eyclids sludddy closed reluced slightly
accommeecate the
(0.3 30y Pensrt's body prsdtioned sngle.
(0.31) Clothing accessocia ies around object well. covedironal match
29) Object appears be sofl cultioned sux
265 Peren's feo viubic reluved
(0 26) Ofject dispcil over penar'ss body
(u 200 weight evident
Lying object.
(2.21 pacpencei tho eduaud and scactfa.
(2:3) i is roina es aet cudlaaue aetiacs.
Tacobeact is cancred natcrnal
Wearing an object.
(8.44) The pesso werr the object arourd their aock (8.27) the objatt a norted the pescen' podict.
3) The abjor is mokked inn the porsae's shint
(0:22) Shonlder in temeel deperdlnng un
(8.3)) Eyes
(5:25) Arm ectraded at the athew
obrect and
(0
(D Fingers extended
cuppeil around food, if sorop.
object.
(0,20) Pesson reeching out with one or bott hands.
(0 20) Person holding food in their funnss.
(3333) Onjects movement
dynomem
[0. Food
opening. opening food,
energs
(0.27) sounces create
depth motion.
Penoars
objsct.
(0 Onject appeans to flying or Casstig
The passels palee Gidigg spwords realy
object with (0.24) paosers tmge aamad with fred (6.17) A place of Rodd is being plood on the eljec's oongas
Food being
Catching object.
Feeding an object.
(2.27) The sapended madiiz
racies (2.11) The ont rallin swadd the scnoek oustraceed hani.
(8.15) prooes hood postiiceed your the object's moatt
Figure 7: Region-aware concept retrieval results (2/2). Additional examples illustrating how different regions contribute enhancing discriminability.
concept pool contains K 10 entries per region type and verb class, and weights are displayed in parentheses. Due to sparsemax activation, concepts with low similarity scores are often assigned zero weight, resulting in more selective and interpretable retrieval. For instance, in kicking an object, the object region retrieves dynamic descriptions such as "(0.50) Object appears to be mid-air or suspended," while the human region emphasizes posture concepts like *(0.27) Buttocks are tense and slightly lifted." These complementary signals capture both motion and pose indicative of a kicking action. Similarly, in distinguishing hugging from toasting, human concepts like "(0.35) Arms wrap around the object" help identify close physical contact, whereas in toasting, object and union concepts highlight celebratory gestures such as "(0.90) Person's body language suggests celebration" and "(0.45) The person's ' smile is subtle as they toast the object." These examples show how our model selectively retrieves fine-grained region-specific concepts that support verb disambiguation subtle interaction contexts. Overall, these results show that our region-aware prompt augmentation (RAP) selects meaningful concepts from distinct regions, supporting both interpretability and fine-grained HOI discrimination.
29

---

"For the "A photo person [Bitin]] an object". give & sbort visual concepts [object] region"
LLaMA-7B
ChatGPTT-4
c] "Purson's munoles they lift
"The objeect is shiny metallic surface."
"Person's strains under the louil
"T object has solid, dense appearance."
<0
"Porant's quickens
<
angular and bexy."
:
c "Person's
ex "The object bulky and oversizedd"
effor or
Figure 8: Qualitative comparison concept generation for the object region. LLaMA-7B often yields human-centric concepts (left), while ChatGPT-4 provides more object-centered descriptions (right). This difference highlights the source of semantic noise in object-region concept retrieval.
Table 17: Comparison of performance when using object concepts from LLaMA-7B and ChatGPT4
across four zero-shot settings. Best results are in bold.
NF-UC
Full
Unseen
Scen
RF-UC
Full
Unseen
Seen
LLaMA-7B
32.18
36.16 36.45
31.18
LLaMA-7B
33.55
31.24 31.29
34.13
ChatGPT-4
32.57
31.60
ChatGPT-4
33.78
34.41
UO
Full
Unseen
Scen
UV
Full
Unseen
Seen
LLaMA-7B
32.79
35.04
32.34
LLaMA-7B
32.78
26.88
33.74
ChaiGPT-4
33.39
36.13
32.84
ChatGPT-4
32.73
26.69
33.72
J Analysis of object concept noise from LLaMA-7B
We further analyze the qualitative and quantitative effects of noisy object concepts generated by LLaMA-7B [48]. As discussed the main paper, the prompt template "a photo of person [verb]- semantic leakage from the human region, producing human-centric descriptions for the object region. Fig. 8 visualizes this issue for the verb lift, where LLaMA-7B frequently associates object concepts with human body parts or actions. In contrast, ChatGPT-4 49 produces more object-centered and visually grounded descriptions, effectively reducing such leakage. To quantitatively evaluate the impact of concept quality, we compare the model performance when object concepts are generated by LLaMA-7B versus ChatGPT-4. As shown in Table 17 GPT-4 generally achieves higher performance across all zero-shot settings, confirming that cleaner concept representations improve robustness without altering the model architecture.
ing the object" sometimes
30


Reviews:
[{"review_id": "wL2YV1DeuT", "reviewer": "Reviewer_FDPj", "summary": "This paper proposes VDRP, a novel framework for zero-shot Human-Object Interaction (HOI) detection. The method addresses two key challenges: (1) intra-class visual diversity via visual diversity-aware prompts (VDP) that inject group-wise variance and Gaussian noise into CLIP prompts, and (2) inter-class visual entanglement (distinct verbs appear similar) via region-aware prompts (RAP) that augment prompts with LLM-generated concepts retrieved from human/object/union regions. VDRP achieves SOTA results with only 4.5M trainable parameters.", "questions": "1) Zero-Shot Verb Variance Robustness\nSuggestion: Add experiments with intentionally isolated verbs (e.g., remove top-3 semantic neighbors from training). Report mAP on these verbs under UV setting.\n\n2) Sparsemax Failure Diagnosis\nSuggestion:\n(a) Quantify concept noise: Report % of shared concepts across verbs (e.g., \"table\" in verb pairs) for UO vs. UV.\n(b) Test adaptive thresholds: Replace Sparsemax with τ-Sparsemax (adjustable sparsity).\n\n3) Sensitivity Issues of Sparsemax:\nTable 6 shows that Softmax outperforms Sparsemax under the UO setting. The authors attribute this to “unreliable LLM concepts,” but no further analysis is provided:\n(1) Is this due to higher concept noise in object regions (e.g., “table” appears in both “sit at” and “work on”)?\n(2) Is the threshold selection sensitive? The authors did not attempt to adjust the sparsity level of Sparsemax.", "limitations": "1. **Computational costs**: No analysis of extra overhead from LLaMA queries/variance computation.  \n2. **Bias propagation**: LLM-generated concepts may encode societal biases (e.g., \"woman cooking\" for \"use stove\").  \n3. **Failure cases**: No discussion of scenarios where VDRP fails (e.g., heavy occlusion or rare verbs).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1) Novel technical contributions: The dual-module design (VDP+RAP) elegantly solves core HOI challenges. Variance injection and Sparsemax-based concept retrieval are innovative.  \n2) Rigorous evaluation: Extensive experiments across all four standard zero-shot HOI settings with SOTA results.  \n3) High efficiency: Achieve SOTA with lightweight model.  \n4) Well-structured analysis: Ablations (Tables 4-6) and visualizations (Figures 1-4) effectively validate design choices.  \n\n\n**Weaknesses**\n1) Fragile Zero-Shot Verb Generalization: Group-wise variance estimation (Eq 6) assumes training-set verbs adequately cover all semantic neighbors. For truly novel verbs with no similar training-set counterparts (e.g., \"teeter\"), variance degenerates to a global mean, failing to model diversity.\n2) Sparsemax Sensitivity Unexplored: While Sparsemax aids concept filtering, its threshold sensitivity harms robustness in object-centric settings (UO in Table 6). No analysis confirms if failure stems from noisy object concepts (e.g., \"table\" for both \"sit\" and \"work\") or improper sparsity calibration.\n3) CLIP Phrase Encoding Bias: The text encoder handles compound phrases (\"hold+baseball+glove\") less effectively than atomic concepts, introducing prompt bias. The fixed template design lacks ablation against alternative formulations (e.g., \"person hold glove\").", "was_revised": true, "final_justification": "The rebuttal partially clarifies certain issues, but my original evaluation persists after full consideration of all feedback.", "rebuttal": "We thank the reviewer for the constructive feedback. We are encouraged by the positive assessment of our technical contributions, evaluation rigor, and clarity. Below, we address the main concerns regarding zero-shot verb generalization, Sparsemax sensitivity, and phrase encoding bias.\n\n---\n\n### **1. Robustness to semantically isolated verbs**\n\nWe thank the reviewer for highlighting a potential limitation of our group-wise variance estimation when applied to **semantically isolated unseen verbs**.\nTo simulate this extreme case, we conduct an additional experiment under the UV setting, where we remove the top-3 semantic neighbors (based on CLIP text embedding similarity) for each unseen verb prior to computing group-wise visual variance. This setting emulates scenarios where unseen verbs lack nearby counterparts in the training set.\nAs shown below, the unseen performance drops only slightly (−0.67 Unseen mAP), indicating that **our method maintains generalization ability even under reduced semantic support**:\n\n|  | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| W/O $\\text{Top}-3$ | 29.48 | 32.64 | 26.21 | 33.69 |\n| W/ $\\text{Top}-3$ | 29.92 | 32.78 | 26.88 | 33.74 |\n\nAlso, our main paper also includes a related analysis using global variance estimation (Main paper: Tab.4), further supporting the robustness of our approach when class-specific support is limited.\n\n---\n\n### 2. Sparsemax failure diagnosis\n\nWe confirm that our method is **robust to concept overlap and sparsity thresholding**, as validated through additional quantitative and ablation analyses.\n\n### 2.1 Object concept overlap analysis\n\nTo evaluate the reviewer’s hypothesis on concept noise due to shared object semantics (e.g., “table” in both “sit” and “work”), we compute the percentage of exact string overlaps in object concepts across verb pairs. Using cosine similarity ≥ 0.90, we find:\n\n- **UO setting**: 2.98% shared concepts\n- **UV setting**: 2.94% shared concepts\n\nThese results suggest that **both settings exhibit similarly low overlap**, and the slightly lower UO performance appears to stem from stochastic variation rather than concept redundancy.\n\n### **2.3 Qualitative diagnosis of noisy object concepts**\n\nWhile concept overlap remains low, we observe that some object concepts contain **human-centric descriptions** due to the prompt template (“a photo of a person [verb]-ing **the object**”) and lack of explicit object grounding. For example, object concepts for \"lift\" include:\n\n> \"Person's legs are crossed or positioned comfortably.\", \"Person's feet are on the ground or resting on the object.\", \"Person's arms are folded or resting on their lap.\", etc.\n> \n\nThis suggests that **semantic leakage from the human region may introduce localized noise**, particularly in the UO setting. However, this appears to be a **minor stochastic effect** rather than a structural flaw, as overall performance remains stable.\n\n### 2.3 τ-Sparsemax analysis\n\nWe further test the effect of adaptive sparsity by applying τ-Sparsemax (τ ∈ {0.0, 0.05, 0.1}), where weights below τ are zeroed. As shown below, **our model remains robust across all τ values**, with only minor performance fluctuations.\n\n### **NF-UC**\n\n| τ value | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.00 (default) | 33.49 | 32.18 | 36.16 | 31.18 |\n| 0.05 | 33.51 | 32.38 | 35.76 | 31.53 |\n| 0.10 | 33.44 | 32.23 | 35.85 | 31.33 |\n\n### **RF-UC**\n\n| τ value | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.00 (default) | 32.62 | 33.55 | 31.24 | 34.13 |\n| 0.05 | 31.62 | 33.10 | 29.56 | 33.99 |\n| 0.10 | 32.50 | 33.59 | 30.92 | 34.26 |\n\n### **UO**\n\n| τ value | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.00 (default) | 33.64 | 32.79 | 35.04 | 32.34 |\n| 0.05 | 33.59 | 32.95 | 34.63 | 32.62 |\n| 0.10 | 33.98 | 32.90 | 35.84 | 32.31 |\n\n### **UV**\n\n| τ value | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.00 (default) | 29.92 | 32.78 | 26.88 | 33.74 |\n| 0.05 | 29.35 | 32.73 | 25.91 | 33.84 |\n| 0.10 | 29.99** | 32.85 | 26.95 | 33.81 |\n\nThese results demonstrate that **RAP is robust to sparsity thresholds**, and that performance variations are more likely caused by **concept generation noise**, not the retrieval strategy itself.\n\n---\n\n### 3. CLIP phrase encoding bias\n\nWe thank the reviewer for pointing out the potential bias in CLIP’s phrase encoding, especially for compound expressions like “holding+baseball+glove.”\n**While this is a known limitation of CLIP, our prompt template—“a photo of a person [verb]-ing the object”—is a standard formulation widely used in HOI literature.**\nTo assess its effect, we performed an ablation study using alternative templates (e.g., “person [verb] the object”). \n\nResults show minimal performance difference, indicating that **our method is robust to such phrasing variations.**\nSpecifically, we compare:\n\n- **BASE template**: \"A photo of a person is [verb]-ing the object.\"\n- **Simplified template**: \"Person [verb] the object.\"\n- **LLM-style template**: \"A example of the person [verb]-ing the object.\"\n\nResults are summarized below across all four zero-shot settings:\n\n### **NF-UC**\n\n| Template | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| a photo of a person [verb] + [ing] the object (default) | 33.49 | 32.18 | 36.16 | 31.18 |\n| person [verb] the object | 33.65 | 32.40 | 36.17 | 31.46 |\n| an example of the person [verb] + [ing] the object | 33.34 | 32.15 | 35.72 | 31.25 |\n\n### **RF-UC**\n\n| Template | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| a photo of a person [verb] + [ing] the object (default) | 32.62 | 33.55 | 31.24 | 34.13 |\n| person [verb] the object | 32.36 | 33.62 | 30.56 | 34.39 |\n| an example of the person [verb] + [ing] the object | 31.59 | 33.08 | 29.52 | 33.97 |\n\n### **UO**\n\n| Template | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| a photo of a person [verb] + [ing] the object (default) | 33.64 | 32.79 | 35.04 | 32.34 |\n| person [verb] the object | 33.82 | 32.99 | 35.21 | 32.54 |\n| an example of the person [verb] + [ing] the object | 33.57 | 32.83 | 34.77 | 32.45 |\n\n### **UV**\n\n| Template | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| a photo of a person [verb] + [ing] the object (default) | 29.92 | 32.78 | 26.88 | 33.74 |\n| person [verb] the object | 29.74 | 32.89 | 26.46 | 33.94 |\n| an example of the person [verb] + [ing] the object | 29.95 | 33.05 | 26.70 | 34.08 |\n\nTemplate variation shows minimal performance impact, indicating that our prompt learning framework is robust to sentence structure—even with CLIP’s known phrase encoding bias. This suggests that such bias does not significantly affect our method’s effectiveness.\n\n---\n\n### 5. Minor points\n\n- **Computational overhead**: All LLM concepts are generated offline, once per (verb, region) pair. The total size of the concept pool is fixed (10 concepts × 3 regions × 117 verbs). Variance computation is also performed once and cached. Thus, our method adds no runtime cost during inference.\n- **Bias propagation**: We acknowledge that LLM-generated concepts may contain bias (e.g., “woman cooking”). In practice, such cases were rare and did not result in consistent prediction shifts. Future work may incorporate human-in-the-loop filtering or bias-aware prompting.\n- **Failure cases**: We will add qualitative failure examples in the supplementary file, highlighting the generated noisy concepts in the final version.\n\n---\n\nWe again thank the reviewer for the encouraging evaluation and for the insightful suggestions on verb variance robustness and Sparsemax sensitivity. We hope our additional analyses and clarifications help address these concerns."}, {"review_id": "a6h8Tbzg0P", "reviewer": "Reviewer_cCr1", "summary": "This paper proposes VDRP, a framework specifically designed to enhance Zero-shot Human-Object Interaction (HOI) detection by addressing two primary challenges: intra-class visual diversity and inter-class visual entanglement. VDRP employs a visual diversity-aware prompt learning strategy by injecting group-wise visual variations into prompts and applying Gaussian perturbation to capture diverse verb visualizations. Additionally, it incorporates region-specific embeddings from human, object, and union regions to generate region-aware prompts with LLM, thus improving verb-level discrimination. Experimental evaluation on the HICO-DET benchmark shows that VDRP achieves SOTA results on the zero-shot settings of HICO-DET.", "questions": "In Line 150, the mean $\\mu_v$ and variance $\\sigma_v^2$ are computed over all $N_v$ samples, what if the $N_v$ is fixed to a small number, e.g., 5? Will the performance be affected significantly?\n\nAfter training, what is the distribution of the diversity-aware prompt embeddings $\\tilde{t}^v$? What is the different between the original token embeddings $\\bar{P}_v$?", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis paper explores the intra-class visual information for zero-shot HOI detection. The proposed visual diversity-aware prompt learning strategy facilitates the verb co-occurrence representation of the verb embeddings. The retrieval-based region prompt augmentation, an advanced LLM guided prompt refinement method, fuses the region-level embeddings while filtering out the irrelevant concepts in the verb embeddings. The proposed method achieves SOTA results with comparable efficiency to the previous methods on the zero-shot settings of HICO-DET.\n\nWeaknesses:\n\nBoth the VDP and RAP is carefully designed for specific purposes, however, the effectiveness of some important part of the introduced components are not sufficient. In Eq. (8), adjusting the hyper-parameter $\\alpha$ directly affects the contribution of the proposed group-wise variance, the impact of the variance is not clear. Similarly, the hyper-parameter $\\beta$ in Eq. (13). In addition, the contribution of the Gaussian perturbation (with or without) according to recent findings is not verified in this paper.", "was_revised": true, "final_justification": "Thank you for the authors' efforts and for providing detailed supplementary results. Based on the additional results, I observe that the choice of the variance injection parameter $\\alpha$ only marginally affects the outcomes. The selection of the concept scaling factor $\\beta$ does not have a significant impact on the results, except in the NF-UC setting. Additionally, the improvement in accuracy on unseen samples achieved by Gaussian perturbation aligns well with expectations.\n\nRegarding the zero-shot results with $N_v=5$, increasing the number of samples appears to have limited effect. Overall, these additional results have addressed my previous concerns. However, I would like to note that most of the observed improvements are not statistically significant. Therefore, I will maintain my original score.", "rebuttal": "We thank the reviewer for the thoughtful feedback and the recognition of our contributions to modeling intra-class visual diversity and inter-class entanglement in zero-shot HOI detection. Below, we address all the questions/concerns raised by the reviewer. \n\n---\n\n### 1. Analysis of hyperparameters (α, β, Gaussian perturbation)\n\nAs requested, we provide additional analyses of hyperparameters across four zero-shot settings:\n\n- **α** (Eq. 8): scaling factor for the delta vector derived from group-wise variance.\n- **β** (Eq. 13): scaling factor for region-aware concepts augmentation.\n- **Contribution for Gaussian perturbation**\n\n**Our analyses show that the proposed method is robust to the choice of hyperparameters in all settings and Gaussian perturbation added prompt embedding improves generalization capability in “unseen” classes.**\n\n### 1.1 Impact of variance injection scale (α)\n\nWe conduct ablations on α ∈ {0.02, 0.01, 0.1} to assess the effect of delta scaling in Eq. (8). The results are summarized below:\n\n### **NF-UC**\n\n| α | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.02 (default) | 33.49 | 32.18 | 36.16 | 31.18 |\n| 0.01 | 33.13 | 32.03 | 35.29 | 31.22 |\n| 0.1 | 33.27 | 32.02 | 35.80 | 31.07 |\n\n### **RF-UC**\n\n| α | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.02 (default) | 32.62 | 33.55 | 31.24 | 34.13 |\n| 0.01 | 32.31 | 33.51 | 30.58 | 34.24 |\n| 0.1 | 31.79 | 33.16 | 29.86 | 33.98 |\n\n### **UO**\n\n| α | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.02 (default) | 33.64 | 32.79 | 35.04 | 32.34 |\n| 0.01 | 33.82 | 32.81 | 35.52 | 32.27 |\n| 0.1 | 33.67 | 32.81 | 35.11 | 32.35 |\n\n### **UV**\n\n| α | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.02 (default) | 29.92 | 32.78 | 26.88 | 33.74 |\n| 0.01 | 29.27 | 32.48 | 25.97 | 33.54 |\n| 0.1 | 28.57 | 32.54 | 24.73 | 33.81 |\n\nWe find that α = 0.02 offers the best trade-off across most settings. This value (0.02) was chosen to match the initialization scale of CLIP’s context embeddings, which ensures stability as large delta scales can lead to over-perturbed context tokens and unstable training. Since context tokens are known to be sensitive to initialization, keeping the scale aligned enhances stability while capturing diversity.\n\n### 1.2 Concepts scaling factor (β)\n\nWe evaluate β ∈ {0.2, 0.5, 1.0} in Eq. (13) to study how strongly region-derived concepts should contribute to the prompt augmentation. \n\nThe results are summarized below:\n\n### **NF-UC**\n\n| β | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.2 (default) | 33.49 | 32.18 | 36.16 | 31.18 |\n| 0.5 | 33.39 | 32.14 | 35.92 | 31.19 |\n| 1.0 | 32.96 | 31.85 | 35.15 | 31.02 |\n\n### **RF-UC**\n\n| β | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.2 (default) | 32.62 | 33.55 | 31.24 | 34.13 |\n| 0.5 | 32.38 | 33.52 | 30.73 | 34.21 |\n| 1.0 | 32.33 | 33.48 | 30.67 | 34.18 |\n\n### **UO**\n\n| β | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.2 (default) | 33.64 | 32.79 | 35.04 | 32.34 |\n| 0.5 | 33.87 | 33.08 | 35.17 | 32.66 |\n| 1.0 | 33.41 | 32.60 | 34.74 | 32.17 |\n|  |  |  |  |  |\n\n### **UV**\n\n| β | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.2 (default) | 29.92 | 32.78 | 26.88 | 33.74 |\n| 0.5 | 29.99 | 32.72 | 27.06 | 33.64 |\n| 1.0 | 29.27 | 32.54 | 25.92 | 33.62 |\n\nModerate augmentation scaling (β = 0.2) achieves the best balance overall. Excessive weighting (β = 1.0) slightly degrades performance across multiple settings, due to overconfident region concepts.\n\n### 1.3 Effect of Gaussian perturbation (ε)\n\nWe evaluate the effect of adding stochasticity to prompt embeddings via ε ∈ {0, 0.1}. The results below demonstrate that the Gaussian perturbation consistently improves generalization capability in **“unseen” classes.**\n\n### **NF-UC**\n\n| Gaussian perturbation | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| W/O perturbation | 33.57 | 32.49 | 35.68 | 31.69 |\n| W/ perturbation (default) | 33.49 | 32.18 | **36.16** | 31.18 |\n\n### **RF-UC**\n\n| Gaussian perturbation | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| W/O perturbation | 31.76 | 33.12 | 29.83 | 33.95 |\n| W/ perturbation (default) | 32.62 | 33.55 | **31.24** | 34.13 |\n\n### **UO**\n\n| Gaussian perturbation | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| W/O perturbation | 33.67 | 33.09 | 34.59 | 32.79 |\n| W/ perturbation (default) | 33.64 | 32.79 | **35.04** | 32.34 |\n\n### **UV**\n\n| Gaussian perturbation | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| W/O perturbation | 29.49 | 32.72 | 26.16 | 33.79 |\n| W/ perturbation (default) | 29.92 | 32.78 | **26.88** | 33.74 |\n\nPerturbation improves generalization in most unseen settings, confirming that stochastic sampling helps model diverse visual realizations for each verb.\n\n---\n\n### 2. Robustness of visual variance under few-shot sampling\n\nAs requested, we provide additional analysis on the robustness of our model across four zero-shot settings. We estimate group-wise variance **using only 5 visual features per verb ($N_v = 5$)** and compare it to the default setting. Performance remains stable, indicating that our method generalizes well even under few-sample conditions.\n\n### **NF-UC**\n\n| Sampling | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| Few-shot ($N_v = 5$) | 33.45 | 32.23 | 35.90 | 31.31 |\n| All samples (default) | 33.49 | 32.18 | 36.16 | 31.18 |\n\n### **RF-UC**\n\n| Sampling | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| Few-shot ($N_v = 5$) | 32.25 | 33.40 | 30.59 | 34.10 |\n| All samples (default) | 32.62 | 33.55 | 31.24 | 34.13 |\n\n### **UO**\n\n| Sampling | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| Few-shot ($N_v = 5$) | 33.38 | 32.55 | 34.75 | 32.11 |\n| All samples (default) | 33.64 | 32.79 | 35.04 | 32.34 |\n\n### **UV**\n\n| Sampling | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| Few-shot ($N_v = 5$) | 29.40 | 32.89 | 25.88 | 34.04 |\n| All samples (default) | 29.92 | 32.78 | 26.88 | 33.74 |\n\nWe observe only minor drops (≤ 0.2 full mAP), confirming that our grouping strategy produces stable variance estimates even with few samples.\n\n---\n\n### 3. Distribution of diversity-aware prompt\n\n**As requested, we analyze the distribution of prompt embeddings after training using the inter-class cosine distance metric.**\nThis analysis was **conducted and reported in Section 6 of the Supplementary**, along with visualizations in Supplementary Fig.1.\nPrompt embeddings begin in a collapsed state (**0.105 ± 0.031**) and become significantly more diverse after training (**0.550 ± 0.103**), closely matching the visual space (**0.518 ± 0.105**).\nUnlike CMMP (**0.937 ± 0.133**), which over-separates classes, our method yields **structured and aligned distributions** that reflect visual diversity.\nThese results confirm that our prompts evolve into meaningful representations without needing additional alignment losses.\n\n---\n\nWe sincerely thank the reviewer for raising insightful questions regarding the effect of group-wise variance (Eq. 8, 13) and the role of Gaussian perturbation in our framework. We hope our additional analyses and clarifications help address these concerns."}, {"review_id": "IaGApeyAlT", "reviewer": "Reviewer_2XwT", "summary": "This paper presents VDRP (Visual Diversity and Region-aware Prompt learning), for zero-shot Human-Object Interaction detection. The work is motivated by two well-defined challenges inherent to the HOI task. The first is intra-class visual diversity, where a single verb class can be realized through a wide variety of visually distinct poses, scales, and contexts. The second challenge is inter-class visual entanglement, where semantically different verbs often produce visually similar human-object configurations, making them hard to disambiguate from global features alone.\n\nTo address these challenges, VDRP proposes two complementary modules that enhance the learnable textual prompts used for interaction classification within a frozen Vision-Language Model like CLIP.\n\n1. Visual Diversity-aware Prompts (VDP): To tackle intra-class diversity, this module learns a distribution of prompts for each verb rather than a single static embedding. It first estimates the visual feature variance for each verb class from the training data. To create more robust estimates, especially for rare verbs, it groups semantically similar verbs and averages their variances. This \"group-wise variance\" is then used in two ways: it modulates the learnable context of the prompt via a small MLP, and it scales the magnitude of Gaussian noise that is added to the final prompt embedding. This process yields prompts that are designed to cover the diverse visual manifold of each verb class. \n\n2. Region-aware Prompts (RAP): To resolve inter-class entanglement, this module augments the prompts with fine-grained, localized visual information. For each verb, it pre-generates a pool of descriptive textual \"concepts\" for the human, object, and union regions using an external Large Language Model (LLaMA-7B). During inference, the model dynamically retrieves the most relevant concepts from this pool by comparing them to the visual features extracted from the corresponding image regions (human, object, and union). These retrieved concepts are then added to the diversity-aware prompt, creating a final region-aware prompt that is sensitive to the subtle, localized cues necessary to distinguish between visually similar interactions.  \n\nThe paper integrates these two modules into a unified, two-stage HOI detection pipeline. Extensive experiments on the HICO-DET benchmark demonstrate that VDRP achieves new state-of-the-art performance across four standard zero-shot settings, while being more parameter-efficient than previous methods.", "questions": "1. The VDP module is conceptually very similar to prior work in prompt distribution learning, such as ProDA[1] and DAPT[2], which also model distributions over prompts to capture visual variance. Could the authors please elaborate on the specific novel contributions of VDP beyond the application of these principles to the HOI detection task?\n\n2. As the authors stated in the Supplementary material, the performance of the RAP module is implicitly dependent on the quality of the concepts generated by the external LLaMA-7B model. Has there been any analysis of the model's sensitivity to the quality of these concepts? For example, Table 6 shows that Softmax outperforms your chosen Sparsemax strategy in the UO setting, which seems to contradict the motivation of aggressively pruning irrelevant concepts. Could this difference suggest the concept pool is noisy, making Sparsemax's pruning harmful in some cases?\n\nOverall, I believe this paper offers valuable insights into the longstanding issues of intra-class visual diversity and inter-class visual entanglement in the HOI detection task. My score currently falls between 4 and 5; if the authors can provide reasonable comments addressing my concerns, I am willing to raise my score.\n\nReferences:\n\n[1] Prompt Distribution Learning\n\n[2] Distribution-Aware Prompt Tuning for Vision-Language Models", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Quality\n\n## Strengths\n\nThe paper is well-motivated, grounding its contributions in a clear and compelling analysis of the core challenges in zero-shot HOI detection. The empirical evaluation is comprehensive and methodologically sound. The model presented in the paper achieved State-of-the-Art performance in all zero-shot settings on the HICO-DET dataset, with thorough ablation studies to validate the contributions of the proposed components.\n\n## Weaknesses\n\nAs the author stated in the \"Limitations\" section in the Supplementary material, The RAP module's effectiveness is critically dependent on the quality and consistency of the concepts generated by the LLM. This introduces several potential points of failure and a lack of robustness that are not adequately addressed in the paper. LLMs are known to be susceptible to generating generic, biased, or factually incorrect information, and their outputs can vary based on the specific model version and prompting strategy. The paper provides no analysis of the quality of the generated concepts, nor does it describe any mechanism for filtering or verifying them. The performance of retrieval-augmented models is known to be sensitive to the quality of the retrieved information. Providing noisy or irrelevant context can actively harm model performance, a phenomenon well-documented in retrieval-augmented NLP. The RAP module is a form of retrieval-augmented generation, yet the paper does not evaluate its robustness to a noisy concept pool.\n\n# Clarity\n\nThe paper is well-written, clearly structured, and easy to follow. The authors have done an excellent job of motivating the problem and presenting their solution.\n\n# Significance\n\n## Strengths\n\nThe work makes a significant and timely contribution to the field. Zero-shot HOI detection is a critical and challenging task for advancing scene understanding, and this paper pushes the state-of-the-art forward meaningfully.\n\n## Weaknesses\n\nHowever, the problem of intra-class visual diversity and inter-class visual entanglement has been identified by some previous works[1][2], although this paper proposed a meaningful method to improve the performance on this task, it is not the first paper to contribute on these issues.\n\n# Originality\n\nThe originality of the paper is good, but it is the area with the most notable weakness. The novelty lies more in the successful synthesis and specific adaptation of existing ideas to the HOI domain, rather than the invention of fundamentally new techniques. \n\nThe core idea of the VDP module is learning a distribution over prompts to capture the visual variance of a class, which is a well-established method. Paper \"Prompt Distribution Learning\" from CVPR 2022, explicitly proposes learning a Gaussian distribution over prompt embeddings to handle intra-class visual diversity in few-shot classification, also, I suggest the authors to cite this paper. The VDP module's use of variance to modulate a context vector and to scale Gaussian noise applied to the final prompt embedding is a direct application of this established principle. The main novel element here appears to be the \"group-wise variance estimation\", which is a clever engineering solution to handle data sparsity for rare verbs, but the conceptual mechanism of learning a prompt distribution is inherited. However, I must emphasize that the VDP module is well tailored for the HOI detection task.\n\nReferences:\n\n[1] Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions\n\n[2] Open-World Human-Object Interaction Detection via Multi-modal Prompts", "was_revised": true, "final_justification": "I appreciate the authors’ rebuttal.\n\nI agree VDP is a well-designed module for the zero-shot HOI detection task. Although it builds upon existing methods such as ProDA and DAPT, the authors have demonstrated its effectiveness, particularly in zero-shot settings.\n\nRegarding the robustness of the RAP module, I generally agree with the authors’ explanation that concept generation noise leads to performance variations, as evidenced by the τ-Sparsemax experiments. However, considering that HICO-DET contains only 600 HOI classes, maybe the authors should explore leveraging more advanced LLMs for visual concept generation to further enhance performance. In 2025, it is time to put LLaMA-7B out to pasture.\n\nBased on the clarifications and the overall contribution, I will increase my score from 4 to 5.", "rebuttal": "We thank the reviewer for the constructive feedback. We appreciate the recognition of our motivation and its effectiveness in addressing intra-class diversity and inter-class entanglement in HOI detection. Below, we clarify the novelty of VDP and the robustness of RAP with supporting analyses.\n\n---\n\n### 1. On the novelty and contributions of VDP\n\nWe thank the reviewer for pointing out connections to prior distribution-based prompt learning methods such as ProDA and DAPT. \nWhile all three methods adopt a distributional approach, VDP introduces several key innovations that make it uniquely suited for zero-shot HOI detection.\n\n**Importantly, as noted by Reviewer FDPj, our design offers “novel technical contributions”, and both variance injection and Sparsemax-based concept retrieval are considered innovative**.\n\n### (1) Difference in distribution modeling strategy\n\nUnlike ProDA and DAPT, which model prompt distributions through non-parametric or regularization-based methods on text embeddings, **VDP directly leverages visual modality** to guide the prompt distribution.\nSpecifically, we extract **union-region visual features per verb** and **inject their group-wise variance** into the context vectors of the prompt learner. This method enables more faithful modeling of intra-class diversity in HOI representations.\n\n### (2) Zero-shot verb generalization\n\nBoth ProDA and DAPT focus on seen-class classification tasks.\nIn contrast, **VDP is explicitly designed for zero-shot HOI detection**, with variance modeling applied to unseen verb classes via **semantically grouped visual variance transfer**, as shown in **Table 5 (Group-5 > Global)**.\nThis structure-aware grouping stabilizes learning and improves generalization to rare or novel classes.\n\n### (3) Region-aware extension\n\nVDP is also integrated into a dual-module architecture (VDP + RAP), where **RAP introduces region-specific augmentation via Sparsemax-based concept selection** from human, object, and union regions.\nThis region-aware design is absent in prior works and further improves verb disambiguation by capturing localized concepts.\n\n### (4) Structured prompt distribution without explicit regularization\n\nUnlike methods such as CMMP that rely on explicit loss functions to shape the prompt space, **VDP naturally induces a structured and balanced distribution** through variance-guided training.\nBased on the analysis we conducted in **Supplementary Section 6 and Fig. 1**, prompt embeddings evolve from a collapsed space (**0.105 ± 0.031**) to a more expressive and semantically structured representation (**0.550 ± 0.103**), **closely reflecting the diversity of visual features (0.518 ± 0.105)**.\nIn contrast, CMMP over-expands the prompt space (0.937 ± 0.133), resulting in loss of semantic structure.\nThese results highlight that VDP produces **well-regularized, diversity-aware prompts** without requiring additional regularization losses.\n\n---\n\n### 2. On the robustness and sensitivity of RAP to concept quality\n\nWhile we acknowledge that RAP’s performance can be affected by the quality of LLM-generated concepts, **our analysis shows that the Sparsemax strategy is generally robust.** The slight drop in UO performance is not due to Sparsemax pruning itself, but rather to semantic noise arising from imperfect concept generation and region-feature alignment.\n\n### 2.1 Qualitative diagnosis of noisy object concepts\n\nThe real source of noise lies in **semantic leakage from the human region** during concept generation. Since our prompts follow the template “a photo of a person [verb]-ing **the object**”, LLMs often generate **object concepts with unintended human descriptions.**\n\nFor example, object concept pool for “lift” includes:\n\n> \"Person's legs are crossed or positioned comfortably.\", \"Person's feet are on the ground or resting on the object.\",  \"Person's arms are folded or resting on their lap.\", etc.\n> \n\nThese human-centric concepts are particularly problematic in **object concepts retrieval**, because **object region crops frequently include parts of the human (e.g., hands, arms)** due to physical proximity. This causes the region feature to match **human-like concepts**, leading Sparsemax to assign them higher weights—**pruning away actual object-relevant concepts**.\n\n### 2.3 Retrieval strategy sensitivity: τ-Sparsemax\n\nWe further test the effect of adaptive sparsity by applying τ-Sparsemax (τ ∈ {0.0, 0.05, 0.1}), where weights below τ are zeroed. As shown below, **our model remains robust across all τ values**, with only minor performance fluctuations.\n\n| τ value | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.00 (default) | 33.49 | 32.18 | 36.16 | 31.18 |\n| 0.05 | 33.51 | 32.38 | 35.76 | 31.53 |\n| 0.10 | 33.44 | 32.23 | 35.85 | 31.33 |\n\n### **RF-UC**\n\n| τ value | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.00 (default) | 32.62 | 33.55 | 31.24 | 34.13 |\n| 0.05 | 31.62 | 33.10 | 29.56 | 33.99 |\n| 0.10 | 32.50 | 33.59 | 30.92 | 34.26 |\n\n### **UO**\n\n| τ value | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.00 (default) | 33.64 | 32.79 | 35.04 | 32.34 |\n| 0.05 | 33.59 | 32.95 | 34.63 | 32.62 |\n| 0.10 | 33.98 | 32.90 | 35.84 | 32.31 |\n\n### **UV**\n\n| τ value | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| 0.00 (default) | 29.92 | 32.78 | 26.88 | 33.74 |\n| 0.05 | 29.35 | 32.73 | 25.91 | 33.84 |\n| 0.10 | **29.99** | **32.85** | **26.95** | **33.81** |\n\nThese results demonstrate that **RAP is robust to sparsity thresholds**, and that performance variations are more likely caused by **concept generation noise**, not the retrieval strategy itself.\n\n---\n\n### 3. Clarification on the positioning relative to prior work\n\nWe acknowledge that the **challenges of intra-class diversity and inter-class entanglement** have been previously recognized (e.g., Bongard-HOI, Open-World HOI). However, our contribution lies in:\n\n- Structuring these challenges into a **unified dual-module framework** (VDP + RAP),\n- Providing a **visually grounded solution** that addresses both variation and confusion explicitly through prompt learning,\n- And validating it under four **zero-shot settings** on HICO-DET with **consistent SOTA performance and low parameter count (4.5M)**.\n\nThus, while the problem is not entirely new, **our solution is novel in formulation, integration, and empirical strength**.\n\n---\n\n### 4. Citations and acknowledgements\nWe thank the reviewer for suggesting citations to ProDA and DAPT. **We will incorporate them in the revised version and clarify how our formulation builds upon and differs from these prior works.**\n\n--- \n\nWe sincerely thank the reviewer for pointing out the sensitivity of RAP to concept quality and the connection between VDP and prior prompt distribution methods. We hope our clarifications and analyses help address these concerns."}, {"review_id": "fH3FhDGcuZ", "reviewer": "Reviewer_7Lyn", "summary": "This paper proposes a Visual Diversity and Region-aware Prompt Learning framework for zero-shot HOI detection, aiming to address the visual complexity inherent in human-object interactions. To this end, the authors introduce a visual diversity-aware prompt learning strategy that incorporates group-wise visual variance into the context embeddings. Experimental results on the HICO-DET benchmark validate the effectiveness of the proposed method.", "questions": "1. In Fig. 2, the paper adopts a fixed template — 'A photo of a person is verbing an object' — as the verb prompt. Did the authors explore alternative or automatically generated prompts? If not, it would be helpful to investigate whether different prompt formulations could impact performance.\n2. The paper lacks an ablation study or empirical analysis to evaluate the contribution of Group-wise variance.\n3. Although this paper is designed for zero-shot HOI detection, it can still be compared with other supervised methods. However, the authors did not conduct a comprehensive comparison.\n4. The reported results for zero-shot HOI detection are not very competitive, and the paper lacks comparisons with recent state-of-the-art methods. For instance, SICHOI (CVPR 2024) reports a score of 36.27, and EZ-HOI (ECCV 2024) achieves 36.73 under the RF setting. Including such comparisons is essential to fairly evaluate the effectiveness of the proposed approach.\n5. Since SICHOI also proposes the concept of multi-view image-text alignment, the novelty of the current work appears to be somewhat limited.", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: This paper proposes a Visual Diversity and Region-aware Prompt Learning framework for zero-shot HOI detection, aiming to address the visual complexity inherent in human-object interactions.. Moreover, it attempts to generate labels using a text-image model. The experimental results also confirm the effectiveness of the proposed algorithm.\n\nWeaknesses: The paper aims to tackle the problem of zero-shot HOI detection. However, the proposed diversity-aware prompt learning strategy has already been explored in previous studies, limiting the novelty of the contribution. In addition, the experimental evaluation does not include a thorough comparison with current state-of-the-art methods, and the reported performance is not particularly competitive.", "was_revised": true, "final_justification": "Thanks for the response from the authors. I increase my score from 3 to 4. However, I really think that the contribution of prompting design is not novel enough. The concept of region-aware prompting are not new in the community. Thus, I think the paper is marginally above the acceptance threshold.", "rebuttal": "We thank the reviewer for the thoughtful assessment and helpful suggestions. We address the reviewer’s concerns regarding novelty, ablation, and comparative evaluation below.\n\n---\n\n## 1. On the novelty of VDRP\n\n**As noted by reviewer FDPj, who highlighted “novel technical contributions” including our dual-module design and the variance injection and Sparsemax-based concept retrieval, our method introduces several key advances.**\n\n### (1) Visual Diversity-aware Prompt (VDP)\n\nVDP provides a new perspective beyond prior distributional prompt methods such as ProDA and DAPT. While existing approaches operate in the text embedding space using non-parametric or regularization-based strategies, **VDP directly models prompt distributions from the visual modality by extracting union-region features per verb and injecting their group-wise variance into the prompt embedding.** This enables accurate modeling of intra-class diversity. Furthermore, unlike prior works that focus on seen class, VDP explicitly targets zero-shot HOI detection by transferring **visual variance from seen verbs to unseen ones via semantic grouping, improving generalization (Tab.5)**.\n\n### (2) Region-Aware Prompt (RAP)\n\nAlthough both RAP and SICHOI explore multi-view alignment, their formulations differ in both scope and implementation:\n- **SICHOI** aligns **linguistic triplets (subject, verb, object)** with global image features using **dependency parsing**.\n- **RAP**, in contrast, aligns **visual regions (human, object, union)** with **region concepts**, refining prompts via **Sparsemax-based selection**.\n\nOur RAP introduces **region-specific, retrieval-based prompt augmentation**, offering a complementary and orthogonal design to SICHOI’s text-driven attention alignment.\n\n---\n\n## 2. On the ablation study and robustness of group-wise variance\n\nWe respectfully disagree with the reviewer’s concern. **We have provided extensive analysis of our group-wise visual variance mechanism through ablations in both the main paper and supplementary.** Also, additional results are presented below for this rebuttal:\n\n### 2.1 Analyses in the main paper and the supplement\n\n- **The main paper (Tab. 5)** compares **Global** vs. **Group_3** vs. **Group_5** grouping strategies.\n- **The supplement (Tab. 3)** evaluates the contribution of **mean + variance (μ, σ²)** vs. **variance-only (σ²)** to assess which statistics are most impactful.\n- These results confirm that **group-wise variance** improves robustness to visual diversity while being more stable than per-class estimation.\n\n### 2.2 (New) Robustness to semantically isolated verbs\n\nTo evaluate robustness under reduced semantic support, we **removed the top-3 semantic neighbors** for each unseen verb during variance computation:\n\n|  | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- |\n| W/O $\\text{Top}-3$ | 32.64 | 26.21 | 33.69 |\n| W/ $\\text{Top}-3$ | 32.78 | 26.88 | 33.74 |\n\n→ Minimal performance drop (−0.67 mAP in unseen) confirms that our grouping retains generalization capacity even with limited support.\n\n### 2.3 (New) Robustness to few-shot variance sampling\n\nWe simulate low-data scenarios by limiting each verb class to **5 samples ($N_v$ = 5)** when estimating visual variance.\n\n| Setting | Sampling | Full | Unseen | Seen |\n| --- | --- | --- | --- | --- |\n| NF-UC | $N_v = 5$ | 32.23 | 35.90 | 31.31 |\n|  | All samples | 32.18 | 36.16 | 31.18 |\n| RF-UC | $N_v = 5$ | 33.40 | 30.59 | 34.10 |\n|  | All samples | 33.55 | 31.24 | 34.13 |\n| UO | $N_v = 5$ | 32.55 | 34.75 | 32.11 |\n|  | All samples | 32.79 | 35.04 | 32.34 |\n| UV | $N_v = 5$ | 32.89 | 25.88 | 34.04 |\n|  | All samples | 32.78 | 26.88 | 33.74 |\n\n→ We observe only minor drops (≤ 0.2 full mAP), confirming that our grouping strategy produces stable variance estimates even with few samples.\n\n---\n\n## 4. On comparisons with SoTA (e.g., EZ-HOI, SICHOI)\n\nWe thank the reviewer for pointing out the importance of comparison with recent SoTA methods such as EZ-HOI (NeurIPS 2024) and SICHOI (CVPR 2024). We provide a comprehensive and quantitative response below.\n\n### 4.1 Comparison with EZ-HOI under all zero-shot settings\n\n**In the main paper and the supplement, we already compared** our method with **EZ-HOI (NeurIPS 2024)** across all four zero-shot HOI settings.\n\n**Our method consistently outperform EZ-HOI across all settings. Also,** note that the **reviewer’s reference to 36.73** corresponds to **Full mAP in the RF-UC**, which is already reported in our paper (Main: Table 1–3, Supplementary: Table 2).\nUnder the same setting, our method achieves a higher **Full mAP of 38.13**, while also improving other key metrics such as **harmonic mean and unseen performance**.\n\n### **(1) ResNet50+ViT-B**\n\n| Setting | Method | #TP | HM | Full | Unseen | Seen |\n| --- | --- | --- | --- | --- | --- | --- |\n| NF-UC | EZ-HOI | 6.85M | 31.76 | 31.17 | 33.66 | 30.55 |\n|  | **Ours** | **4.50M** | **33.49** | **32.18** | **36.16** | **31.18** |\n| RF-UC | EZ-HOI | 6.85M | 31.18 | 33.13 | 29.02 | 34.15 |\n|  | **Ours** | **4.50M** | **32.62** | **33.55** | **31.24** | **34.13** |\n| UO | EZ-HOI | 6.85M | 32.38 | 32.27 | 33.28 | 32.06 |\n|  | **Ours** | **4.50M** | **33.64** | **32.79** | **35.04** | **32.34** |\n| UV | EZ-HOI | 6.85M | 29.09 | 32.32 | 25.10 | 33.49 |\n|  | **Ours** | **4.50M** | **29.92** | **32.78** | **26.88** | **33.74** |\n\n### **(2) ResNet50 + CLIP ViT-L**\n\n| **Setting** | **Method** | **#TP** | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- | --- | --- |\n| NF-UC | EZ-HOI | 14.07M | 35.38 | 34.84 | 36.33 | 34.47 |\n|  | **Ours** | **10.29M** | **36.83** | **36.46** | **37.48** | **36.21** |\n| RF-UC | EZ-HOI | 14.07M | 35.73 | 36.73 | 34.24 | 37.35 |\n|  | **Ours** | **10.29M** | **37.58** | **38.13** | **36.72** | **38.48** |\n| UO | EZ-HOI | 14.07M | 37.06 | 36.38 | 38.17 | 36.02 |\n|  | **Ours** | **10.29M** | **38.41** | **37.81** | **39.36** | **37.50** |\n| UV | EZ-HOI | 14.07M | 32.84 | 36.84 | 28.82 | 38.15 |\n|  | **Ours** | **10.29M** | **34.31** | **37.18** | **31.16** | **38.16** |\n\nThese results demonstrate that our method achieves **state-of-the-art performance across all four zero-shot settings** and two backbones, consistently improving generalization capability to unseen interactions.\n\n> We also clarify that EZ-HOI is a NeurIPS 2024 paper (not ECCV). Our comparisons are consistent with the official ViT-B and ViT-L configurations reported in their NeurIPS version.\n> \n\n---\n\n### 4.2 Clarification on SICHOI and its limited comparability\n\nThank you for sharing a recent and interesting work, **SICHOI (CVPR 2024).** We will **include this work in the final version.**  We acknowledge the recent advancement of SICHOI (CVPR 2024) performance. However, direct and fair comparison with our framework is problematic for several key reasons: **1)** **No code or model release, 2) No results on UO or UV settings, and more importantly 3) Unfairly high computational cost induced by a stronger backbone, additional foundation models (e.g., BLIP, SAM) and multiple decoders/branches.**\n\n| Model | Code? | UO/UV Reported? | External Modules |\n| --- | --- | --- | --- |\n| SICHOI | ✗ | ✗ | BLIP, SAM, CLIP |\n| Ours | ✓ | ✓ | CLIP |\n\nFor reference, the **reviewer’s cited score of 36.27 corresponds to Unseen mAP under RF-UC**, while SICHOI also reports **36.44 under NF-UC**. We include both below for comparison.\n\n| Model | Backbone | RF-UC Unseen | NF-UC Unseen |\n| --- | --- | --- | --- |\n| SICHOI | **ResNet101** + ViT-L | 36.27 | 36.44 |\n| Ours | ResNet50 + ViT-L | **36.72** | **37.48** |\n\n**Also, supervised comparison is already provided in Supplementary Tab.1,** showing that our method performs competitively under the **supervised setting.**\n\n---\n\nWe hope these address the reviewer’s concern regarding comparison fairness and highlight the **superior generalization capability** of our approach across multiple zero-shot settings.\n\n## 5. On the use of fixed prompt templates\n\nAs reviewer requested, we experimented with three formulations:\n\n- **BASE template**: \"A photo of a person is [verb]-ing an object.\"\n- **Simplified template**: \"Person [verb] the object.\"\n- **LLM-style template**: \"A example of the person [verb]-ing the object.\"\n\nResults show **only marginal differences**, indicating that our framework is **robust to the template design**:\n\n### **NF-UC**\n\n| Template | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| a photo of a person [verb] + [ing] object (default) | 33.49 | 32.18 | 36.16 | 31.18 |\n| person [verb] the object | 33.65 | 32.40 | 36.17 | 31.46 |\n| a example of the person [verb] + [ing] object | 33.34 | 32.15 | 35.72 | 31.25 |\n\n### **RF-UC**\n\n| Template | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| a photo of a person [verb] + [ing] object (default) | 32.62 | 33.55 | 31.24 | 34.13 |\n| person [verb] the object | 32.36 | 33.62 | 30.56 | 34.39 |\n| a example of the person [verb] + [ing] object | 31.59 | 33.08 | 29.52 | 33.97 |\n\n### **UO**\n\n| Template | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| a photo of a person [verb] + [ing] object (default) | 33.64 | 32.79 | 35.04 | 32.34 |\n| person [verb] the object | 33.82 | 32.99 | 35.21 | 32.54 |\n| a example of the person [verb] + [ing] object | 33.57 | 32.83 | 34.77 | 32.45 |\n\n### **UV**\n\n| Template | **HM** | **Full** | **Unseen** | **Seen** |\n| --- | --- | --- | --- | --- |\n| a photo of a person [verb] + [ing] object (default) | 29.92 | 32.78 | 26.88 | 33.74 |\n| person [verb] the object | 29.74 | 32.89 | 26.46 | 33.94 |\n| a example of the person [verb] + [ing] object | 29.95 | 33.05 | 26.70 | 34.08 |\n\nTemplate variation shows minimal performance impact, indicating that our prompt learning framework is robust to sentence structure.\n\n---\n\nWe truly appreciate the reviewer’s thoughtful comments and critical insights. We hope our responses and new analyses help resolve the concerns."}]

Author rebuttal:

Meta-review: The paper introduces a dual-module framework (VDRP) for zero-shot HOI detection, combining visual diversity-aware prompts with region-aware prompts. Reviewers agree on the solid motivation, technical soundness, and strong empirical results, with efficiency noted as a clear strength. While novelty was viewed as incremental by some, others recognized meaningful contributions in the variance injection and dual-module design. Concerns about originality and RAP sensitivity were addressed through additional experiments and clarifications. Overall, the consensus is positive, and the reasons to accept outweigh the reservations. I recommend acceptance.
Decision: Accept (poster)