Title: Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Zero-shot HOI detection

Vision Language Model

Prompt Learning
Abstract: Zero-shot Human-Object Interaction detection aims to localize humans and objects in an image and recognize their interaction, even when specific verb-object pairs are unseen during training. Recent works have shown promising results using prompt learning with pretrained vision-language models such as CLIP, which align natural language prompts with visual features in a shared embedding space. However, existing approaches still fail to handle the *visual complexity of interaction*—including (1) *intra-class visual diversity*, where instances of the same verb appear in diverse poses and contexts, and (2) *inter-class visual entanglement*, where distinct verbs yield visually similar patterns. To address these challenges, we propose **VDRP**, a framework for *Visual Diversity and Region-aware Prompt learning*. First, we introduce a visual diversity-aware prompt learning strategy that injects group-wise visual variance into the context embedding. We further apply Gaussian perturbation to encourage the prompt to capture diverse visual variations of a verb. Second, we retrieve region-specific concepts from the human, object, and union regions. These are used to augment the diversity-aware prompt embeddings, yielding region-aware prompts that improve verb-level discrimination. Experiments on the HICO-DET benchmark demonstrate that our method achieves state-of-the-art performance under four zero-shot evaluation settings, effectively addressing both intra-class diversity and inter-class visual entanglement. Code is available at https://github.com/mlvlab/VDRP.

Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection

Abstract

Zero-shot Human-Object Interaction detection aims to localize humans and objects in an image and recognize their interaction, even when specific verb-object pairs are unseen during training. Recent works have shown promising results using prompt learning with pretrained vision-language models such as CLIP, which align natural language prompts with visual features in a shared embedding space. However, existing approaches still fail to handle the visual complexity of interaction -including (1) intra-class visual diversity , where instances of the same verb appear in diverse poses and contexts, and (2) inter-class visual entanglement , where distinct verbs yield visually similar patterns. To address these challenges, we propose VDRP , a framework for Visual Diversity and Region-aware Prompt learning . First, we introduce a visual diversity-aware prompt learning strategy that injects group-wise visual variance into the context embedding. We further apply Gaussian perturbation to encourage the prompts to capture diverse visual variations of a verb. Second, we retrieve region-specific concepts from the human, object, and union regions. These are used to augment the diversity-aware prompt embeddings, yielding region-aware prompts that enhance verb-level discrimination. Experiments on the HICO-DET benchmark demonstrate that our method achieves state-of-the-art performance under four zero-shot evaluation settings, effectively addressing both intra-class diversity and inter-class visual entanglement. Code is available at https://github.com/mlvlab/VDRP .

1 Introduction

Human-Object Interaction (HOI) detection [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] aims to localize humans and objects in an image and recognize the interactions between them, serving as a cornerstone for fine-grained scene understanding. Unlike standard HOI detection, which assumes supervision over all interactions, zero-shot HOI detection must generalize to unseen combinations of verbs and objects. While recent advances in Vision-Language Models (VLMs) have significantly improved the zero-shot recognition of object and attribute classes [14, 15, 16, 17, 18, 19, 20], zero-shot HOI detection remains fundamentally more challenging. As previous works [21, 22] pointed out, the difficulty stems not only from the compositional novelty of interactions but also from the visual complexity of interactions-where each verb class exhibits large intra-class visual diversity, and different verbs frequently produce visually similar patterns.

First, verb classes exhibit substantial intra-class diversity. As shown in Fig. 1-(A), instances of the verb 'holding a baseball glove' may appear in drastically different poses, scales, or scene contexts, yet must be classified under the same label. To quantify this, we compute a diversity score using CLS features extracted from a frozen CLIP visual encoder. For verbs, we crop the union region; for objects, the object bounding box is used. The average pairwise cosine similarity within each class is

hyunwoojkim}@kaist.ac.kr

(B) Inter-class visual entanglement between the different interaction classes

: CLS feature

: Prototype

: boarding

: sitting at

: licking

: eating

: riding

[Inter-class visual overlap of verb classes in t-SNE]

(A) Intra-class diversity in the same interaction class

'Person

holding

a baseball glove'

⋯

Object

Verb

0.274

±0.048

0.364

±0.060

Object class

diversity score

Verb class

diversity score

※

Diversity score:  𝔼 1 - cos

∙

[Diversity score comparison between object and verb]

'Person

eating

an object.'

'Person

licking

an object.'

'Person

sitting

at

an object.'

0.40

0.35

0.30

0.25

0.20

0.15

0.10

0.05

0.00

335

48

电

30

20

10

0

-10

-20

X

-30

30

-20

-10

10

20

30

measured, and the diversity score is defined as the expectation of 1 -cos( · ) . This analysis shows that verbs have significantly higher intra-class diversity (0.364 ± 0.060) compared to objects (0.274 ± 0.048), indicating that a single prompt embedding is likely insufficient to capture such variation.

Second, interactions often exhibit inter-class visual entanglement , where different verbs produce highly similar visual patterns. As illustrated in Fig. 1-(B), semantically distinct verbs such as 'eating', 'licking', and 'sitting at' frequently share similar human-object layouts. In such cases, accurate classification depends on regional differences-often localized in the human or object region-which global or union-level features may fail to capture. To analyze this phenomenon, we extract unionregion CLS features from a frozen CLIP visual encoder across five verbs and project them into a 2D space using t-SNE. The resulting convex hulls show substantial overlap not only at the sample level but also across verb prototypes, revealing poor inter-class separability. This entanglement poses a major challenge for zero-shot HOI detection, where explicit class supervision is unavailable.

Recently, studies on zero-shot HOI detection have explored prompting strategies that leverage pretrained CLIP models [23, 24, 25, 26, 27]. These methods map HOI triplets to textual descriptions and embed them in a shared vision-language space. While effective for semantic alignment, most approaches assume a single static prompt per verb [23, 10, 26], making them inadequate for modeling the visual diversity within each class. Some incorporate spatial cues in the visual branch [24], but leave the text prompts agnostic to region-specific semantics. Others rely on LLM-generated descriptions [25], focusing on semantic differences across verbs but overlooking intra-class variation.

To overcome these limitations, we propose a new framework called VDRP ( V isual D iversity and R egion-aware P rompt learning) for zero-shot HOI detection. Our method enhances prompt representations in two complementary ways. First, we inject group-wise visual variance into the learnable context embeddings and apply Gaussian perturbation, allowing the prompts to reflect intra-class appearance diversity and better adapt to varied visual realizations during training. Second, we retrieve

region-specific concepts from the human, object, and union regions and use them to augment the prompt embeddings, producing region-aware prompts that improve verb discriminability.

Our contributions are summarized as follows:

We propose a visual diversity-aware prompt learning that models intra-class variation by injecting group-wise variance into the context embedding and applying Gaussian perturbation, enabling the prompts to generalize across diverse appearances of the same verb.

We introduce a region-aware prompt augmentation that leverages region-specific concept retrieval from human, object, and union regions to enrich the prompts, enhancing discriminability among visually similar verbs.

We integrate the two modules into a unified framework, VDRP , and demonstrate its effectiveness on the HICO-DET benchmark, achieving new state-of-the-art performance under multiple zero-shot evaluation settings.

2 Related works

HOI detection. Human-Object Interaction (HOI) detection typically involves three sub-tasks: object detection, human-object pairing, and interaction classification. Thanks to advances in largescale benchmarks [28, 29, 4, 21] and transformer-based architectures [30, 9, 31], a wide range of approaches have been proposed. Broadly, HOI methods fall into one-stage [23, 27, 4, 9, 11, 7] and two-stage [10, 26, 24, 25, 5, 6] paradigms. One-stage methods jointly predict object locations and interactions, often using set prediction frameworks such as DETR [30]. In contrast, two-stage methods decouple the task into object detection and interaction classification: a pre-trained detector localizes humans and objects, and a dedicated module predicts the verb label for each human-object pair. The division of the HOI task in two-stage approaches allows efficient training [5, 6], and shows promising results. Our work falls in the two-stage paradigm.

Zero-shot HOI detection. Zero-shot HOI detection aims to identify HOI triplets unseen during training, a task challenged by the long-tail distribution of compositional datasets. With the rise of Vision-Language Models (VLMs) [15, 32, 33, 14, 34] pretrained on large-scale image-text pairs, many works leverage their generalization for HOI. Several methods [10, 23, 27, 35] align HOI models with CLIP's pretrained representations to enable effective transfer. Recent approaches [24, 26, 25] adopt prompt learning to adapt CLIP with few learnable parameters for fine-grained interaction understanding. However, most rely on a single static prompt per verb [10, 23, 26], limiting their ability to capture intra-class visual diversity. Later works add spatial cues or LLM-generated descriptions [24, 25], but still lack region-level adaptation or concept-level grounding. We address these gaps with visual diversity-aware prompt learning and region-aware prompt augmentation.

Prompt learning. Prompt learning is a widely used strategy for adapting vision-language models (VLMs) like CLIP to downstream tasks [36, 37, 38, 39, 40, 41, 42]. Early works such as CoOp [18] and CoCoOp [19] optimize learnable or image-conditioned context vectors, while MaPLe [38] jointly tunes visual and textual prompts to improve cross-modal alignment. In zero-shot HOI detection, CMMP [24] and EZ-HOI [25] adopt multi-modal prompt learning, but do not account for visual diversity of verbs and use the same verb prompt across regions. This limits their ability to adapt to diverse verb appearances and handle visually similar interactions. Meanwhile, distribution-based prompt learning [37, 36, 40, 43] has shown that leveraging feature variance improves generalization. Inspired by this, we propose a method that injects group-wise visual variance-via modulation and perturbation-into prompt embeddings, and augments them with region-specific concepts, enabling effective handling of visual diversity and improving verb discriminability in zero-shot HOI detection.

3 Methods

In this section, we present our framework, VDRP , designed to address two key challenges in zero-shot Human-Object Interaction (HOI) detection: (1) intra-class visual diversity , where instances of the same verb exhibit a wide range of visual appearances, and (2) inter-class visual entanglement , where semantically distinct verbs appear visually similar. To tackle these challenges, VDRP introduces two complementary components. The first, visual diversity-aware prompt learning , addresses intra-class diversity by injecting group-wise visual variance into the context embeddings and applying varianceguided perturbation, resulting in visual diversity-aware prompts that better capture verb-specific

ǁ

ǁ

ǁ

Ƹ

Ƹ

Ƹ

Ƹ

Ƹ

Ƹ

Ƹ

Ƹ

Ƹ

Ƹ

Ƹ

Detector

(A) Overall pipeline

Concept retriever

𝐱h

𝐜 h

1

𝐜 h

2

𝐜 h

𝐾

⋮

Concept retriever

𝐱o

𝐜o

1

𝐜o

2

𝐜o

𝐾

⋮

Concept retriever

𝐱u

𝐜u

1

𝐜u

2

𝐜u

𝐾

⋮

ҧ 𝐜h

ҧ 𝐜u

Augmentation

Text

encoder

A photo of person is

zipping

an object.

A photo of person is

blocking

an object.

A photo of person is

assembling

an object.

⋮

A photo of person is

adjusting

an object.

ഥ

𝐏

Verb prompt

𝐏

Context

embedding

𝐄

injection

Concat.

⋮

(C) Region-aware

prompt augmentation

Image

𝐈

Image

encoder

Adapter

RoI Align

𝑏h

𝑏o

𝑏u

𝐱h

𝐱o

𝐱෥ u

Spatial

Head

𝐱u

𝑠

𝑙

𝑏

𝐭 o

𝑉

𝐭 o

1

𝐭 o

2

⋮

𝐓o:

𝐭

h

𝑉

𝐭

h

1

𝐭

h

2

⋮

𝐓h:

𝐭 u

𝑉

𝐭 u

1

𝐭 u

2

⋮

𝐭

h

𝑉

𝐭

h

1

𝐭

h

2

⋮

𝐓h:

𝐭 o

𝑉

𝐭 o

1

𝐭 o

2

⋮

𝐓o:

𝐭 u

𝑉

𝐭 u

1

𝐭 u

2

⋮

Logit h

Logit o

Logit u

Logit hoi

𝐭 𝑉

𝐭

1

𝐭

2

⋮

෩ 𝐓

Perturbation

Proj

down

(B) Visual diversity-aware

prompt learning

Mean

Ƹ

Ƹ

appearance variations. The second, region-aware prompt augmentation , enhances these prompts using region-specific concepts retrieved from the human, object, and union regions, yielding region prompts that improve inter-class discriminability. We begin by outlining the overall pipeline (Section 3.1), followed by detailed descriptions of the two modules in Sections 3.2 and 3.3.

3.1 Overall pipeline

As illustrated in Fig. 2-(A), our method follows a two-stage HOI detection framework [10, 24, 5, 6], consisting of (1) human-object detection and (2) interaction classification. In the first stage, a frozen object detector (DETR [30]) is applied to the input image I to identify object instances. Each detection yields a triplet ( s n , l n , b n ) , representing the confidence score, object class embedding, and bounding box of the n -th instance. In the second stage, we encode each instance into a prior embedding p = Proj down ([ s ; l ; b ]) ∈ R d down , which serves as guidance for task adaptation. Using this prior, we extract patch embeddings from a frozen CLIP encoder augmented with lightweight adapter layers inserted into multiple transformer blocks. Each adapter applies a bottleneck structure that projects the patch features, attends to the prior p , and reconstructs the output:

where X i ∈ R N × d up is the image feature map at layer i . The final image feature map X ∈ R N × d is obtained from the last transformer layer with projection. Given the image features and the boxes for the human ( b h), object ( b o), and union region ( b u), we extract region features via RoIAlign [44]:

where x h , x o , x ˜ u ∈ R d denote the region features for the human, object, and union regions, respectively. To enhance the union-region representation with spatial priors, we apply a spatial head that fuses x ˜ u with human and object features, as well as their bounding boxes:

More details on SpatialHead ( · ) are provided in the supplementary material. Next, each region featurex h, x o, and x u-is matched against a set of verb prompts.

Ƹ

Ƹ

Ƹ

Ƹ

Ƹ

ǁ

ǁ

ҧ

Ƹ

Concept pool

𝒞(∙)

𝑣

𝐜

(∙)

𝑣,1

𝐜

(∙)

𝑣,2

𝐜

(∙)

𝑣,𝐾

𝐜

(∙)

𝑣,1

⋮

Weighted

sum

Concept retriever

Concept

vector

Augmentation

𝐭

(∙)

𝑣

CLS feature

𝐳𝑣

1

𝐳𝑣

2

𝐳 𝑣

𝑁𝑣

⋮

𝐳𝑣

1

𝐳𝑣

2

𝐳 𝑣

𝑁𝑣

⋮

𝐳 𝑉

1

𝐳 𝑉

2

𝐳

𝑉

𝑁𝑉

⋮

Mean of CLS

𝐳 𝑉

1

𝐳 𝑉

2

𝐳

𝑉

𝑁𝑉

⋮

𝝁𝑽

𝝁𝑉

𝝁𝑉

𝝁𝑉

𝝈1

2

𝝈2

2

𝝈𝑉

2

𝝈𝑣

2

⋮

⋮

Group

𝒢(𝑣)

Mean

Learnable

embedding

𝐄

Token embedding

MLP

Text

Encoder

Concat.

Group-wise

variance

ഥ

𝝈𝑣

2

ഥ

𝐏𝑣

𝐭

𝑣

𝐭

𝑣

Visual diversity-

aware prompt

Region-aware

prompt

Group-wise variance estimation

Region feature

𝐱(∙)

W(∙)

𝑣

𝐒𝐩𝐚𝐫𝐬𝐞𝐦𝐚𝐱(∙)

Concept weight

𝐜 (∙)

𝑣

𝐭

𝑣

Variance injection

Perturbation

Noise

sampling

Modulation

(A) Visual diversity-aware prompt learning

(B) Region-aware prompt augmentation

As illustrated in Fig.2-(B) and (C), Each prompt ˆ t v ( · ) is constructed in two stages. Let T ( · ) = [ ˆ t 1 ( · ) , . . . , ˆ t V ( · ) ] ∈ R d × V denote the region-aware prompts for ( · ) ∈ { h , o , u } . We first generate a visual diversity-aware prompt ˜ t v via group-wise variance injection and Gaussian perturbation, then augment it with region-specific concepts retrieved based on region feature x ( · ) . This two-step design allows the final region-aware prompts to reflect both the visual diversity of verbs and localized context. See Sections 3.2 and 3.3 for details. The logits for each region prompts is computed as:

where Logit ( · ) ∈ R V . Finally, the overall HOI classification logit is obtained by averaging region-wise logits:

We train the model using focal loss [45] for multi-label verb classification.

3.2 Visual diversity-aware prompt learning

To address intra-class visual diversity in HOI detection, we propose a visual diversity-aware prompt learning method that incorporates visual variance into both the context modulation and prompt perturbation processes. Static prompts-optimized as single points-struggle to represent the wide variation of instances within the same verb class. Inspired by recent findings [40, 37, 36] showing that variance-aware representations improve generalization, we explicitly encode class-level visual variance to guide prompt adaptation and inject noise that reflects the extent of such diversity.

Group-wise variance estimation. As illustrated in Fig. 3-(A), we begin by extracting union-region features from the training set using a frozen CLIP image encoder. Specifically, we crop the union box of each human-object pair and extract the CLS token. For each verb v , let z ( j ) v denote the CLS feature of the j -th instance. We compute the mean µ v and variance σ 2 v over all N v samples. To obtain a stable estimate for rare or unseen verbs, we construct a group of verbs G ( v ) by selecting the

similar verbs based on cosine similarity between CLIP text embeddings. This grouping allows each verb to inherit variance statistics from its semantically similar neighbors. The group-wise variance is then computed as:

This group-wise variance serves as an inductive prior capturing the expected diversity of verb v .

Visual diversity-aware prompts. Then, we transform the group-wise variance into a modulation vector using a lightweight MLP to perform variance injection as follows:

This is added to the shared context embedding E ∈ R N ctx × d to produce a verb-specific context:

where α is a small scaling factor for stability. Given a verb prompt sentence P v (e.g., 'A photo of a person is [ v ] +ing an object.' ), we tokenize it into token embedding ¯ P v and concatenate it with the modulated context to form the final input for CLIP text encoder T ( · ) :

To further reflect visual variability, we perturb each prompt embedding using Gaussian noise scaled by the group-wise standard deviation. We normalize ¯ σ v across dimensions and modulate it to match the standard deviation of t v , producing ˜ σ v . Noise is then sampled and applied element-wise:

where β is a small scaling factor controlling the perturbation strength. This results in prompt embeddings that encode both the central semantics of the verb and its expected visual diversity. Finally, we collect all perturbed prompts across the V verbs to form the diversity-aware prompts ˜ T = [ ˜ t 1 , . . . , ˜ t V ] ∈ R d × V , which serves as the base for region-aware prompts.

3.3 Region-aware prompt augmentation

To address inter-class visual entanglement-where semantically distinct verbs exhibit similar visual patterns-we augment prompts using region-specific concepts from the human, object, and union regions. While diversity-aware prompts reflect class-level variation, they cannot capture the region concepts needed to distinguish visually similar verbs. To bridge this gap, we introduce retrieval-based region-aware prompt augmentation, which allows each prompt to specialize based on regions.

Region concept generation. To enrich prompt embeddings with localized semantics, we follow [46, 47, 20] and query LLMs (e.g., LLaMA-7B [48] and ChatGPT-4 [49]) to generate K region-level concepts for each verb v and region type R ∈ { human, object, union } . Each prompt follows the format: 'For the verb [ P v ] , give K short visual concepts for the [ R ] region.' The resulting concepts are encoded using the CLIP text encoder T ( · ) to form the concept pool:

where ( · ) ∈ { h , o , u } denotes the region. For more detailed examples, please refer to Fig.4-(A).

Region-aware prompts. As shown in Fig. 3-(B), given a region feature x ( · ) ∈ R d and its corresponding concept pool C v ( · ) , we compute cosine similarity between the feature and each concept:

To highlight the most informative concepts while ignoring irrelevant ones, we apply Sparsemax [50] to the similarity scores, which assigns exact zero weights to uninformative entries and retains only the most relevant concepts. Using the resulting scalar weight W v,k ( · ) , we compute a region concept vector:

This concept vector is then used to augment the diversity-aware prompt t v , producing the final region-aware prompt as follows:

where γ is a scalar controlling the degree of augmentation. The final set of region-aware prompts T ( · ) = [ ˆ t 1 ( · ) , . . . , ˆ t V ( · ) ] is used to compute classification logits, as described in Eq. (4). This improves discriminability of the model by aligning each prompt with region-specific semantics.

4 Experiments

4.1 Experimental settings

Datasets. We conduct experiments on the HICO-DET benchmark for HOI detection. HICO-DET contains 80 object categories from the COCO dataset [51] and 117 actions, forming 600 HOI classes. It includes 47,776 images, with 38,118 for training and 9,658 for testing.

Zero-shot setting on HICO-DET. Following prior works [3, 2, 1, 23], we evaluate under four settings: Non-rare First Unseen Composition (NF-UC), Rare First (RF-UC), Unseen Object (UO), and Unseen Verb (UV). NF-UC and RF-UC define 120 unseen and 480 seen HOI triplets from 600 total, with unseen compositions drawn from head and tail categories, respectively. UO uses 68 object classes to construct 500 seen and 100 unseen triplets. UV withholds 20 out of 117 verb classes, yielding 516 seen and 84 unseen triplets.

Evaluation metric. Mean Average Precision (mAP) is used to evaluate the model for HOI detection. Specifically, a sample is regarded as a true positive if two conditions are met: 1) the IoU of both human and object bounding boxes is larger than 0.5, and 2) the HOI triplet prediction is correct.

Implementation Details. Wefollow the standard training setup used in prior zero-shot two-stage HOI detection methods [24, 10, 25], where DETR is first fine-tuned on instance-level annotations from the HICO-DET training split. Unless otherwise noted, we use CLIP ViT-B/16 as the visual backbone. Additional implementation and training details are provided in the supplementary material.

4.2 Zero-shot HOI detection

We evaluate our method on four zero-shot HOI detection settings in HICO-DET: NF-UC, RF-UC, UO, and UV. Table. 1, 2, and 3 report mAP (Full / Unseen / Seen), harmonic mean (HM), and trainable parameters (#TP), comparing with state-of-the-art baselines. In NF-UC and RF-UC (Table. 1), our method achieves the best scores across all metrics. In NF-UC, we obtain 36.45 (Unseen) and 33.85 (HM), surpassing CLIP4HOI by +5.01 (Unseen) and +4.31 (HM). In RF-UC, we outperform EZ-HOI by +2.27 (Unseen) and +1.59 (HM). Our method uses only 4.50M trainable parameters, much fewer than EZ-HOI (6.85M) and CLIP4HOI (56.7M). In UO (Table. 2), we achieve 36.13 (Unseen) and 34.41 (HM), outperforming CMMP and EZ-HOI by +1.97 and +2.27 HM, respectively. In UV (Table. 3), we again achieve the best scores: 26.69 (Unseen), 32.72 (Full), and 29.80 (HM), with a +1.59 gain on Unseen verbs over EZ-HOI. These results confirm the effectiveness of our visual diversity-aware prompts and region-aware prompts in the zero-shot HOI detection.

4.3 Ablation studies

Component analysis. To assess each component's contribution, we conduct an ablation study across four zero-shot HOI settings (Table 4). Starting from a static prompt baseline, we add two modules: (1) VDP models intra-class variation using group-wise visual variance and Gaussian noise, and (2) RAP augments prompt embeddings with retrieved region-specific concepts. VDP improves generalization by capturing the variance of diverse verb appearances, while RAP enhances discrimination among visually similar verbs. The full model VDRP achieves the best performance across all settings, demonstrating the complementary benefits of both modules.

Group-wise modeling in VDP. To evaluate the impact of group-wise visual variance, we compare different configurations in the VDP module based on the number of verbs per group: (1) a single global variance shared across all verbs ( Global ), (2) group-wise variance using clusters of 3 verbs each ( Group_3 ), and (3) clusters of 5 verbs each ( Group_5 ). As shown in Table 5, we observe that group-wise variance modeling generally improves over a global estimate, but performance varies depending on the group size.

Impact of injection scale. We evaluate variance injection scale α ∈ { 0 . 01 , 0 . 02 , 0 . 10 } to analyze its influence on the delta scaling in Eq. 8. Results across all settings are summarized in Table 6. We

observe that α = 0 . 02 consistently achieves the best trade-off between unseen and seen performance. This value was chosen to match the initialization scale of CLIP's context embeddings, ensuring stability as excessively large α causes over-perturbed context tokens and unstable training. Given that context tokens are highly sensitive to initialization, keeping the scale aligned promotes stable optimization while maintaining sufficient diversity.

Effect of Gaussian perturbation. We study the effect of Gaussian perturbation on prompt embeddings by varying the scale β ∈ { 0 , 0 . 1 } . Table 7 summarizes the results across all zero-shot settings. Perturbation improves generalization in most unseen cases, confirming that stochastic sampling helps the model capture diverse visual realizations of each verb.

Retrieval strategy. We compare three retrieval methods for region concepts: Softmax, Top-3, and Sparsemax (Table 8). Overall, Sparsemax performs most consistently, benefiting from its ability to suppress irrelevant concepts and emphasize informative ones.

Impact of augmentation scaling factor. We evaluate γ ∈ { 0 . 2 , 0 . 5 , 1 . 0 } in Eq. 13 to study how strongly region-derived cues contribute to prompt adaptation. Results are reported in Table 9. A moderate scaling of γ = 0 . 2 provides the best overall balance, whereas excessive cue weighting ( γ = 1 . 0 ) slightly degrades performance across multiple settings, likely due to noisy region concepts.

4.4 Qualitative results

Fig. 4 shows qualitative examples of region-specific concept generation and retrieval. Given a verb prompt (e.g., 'licking an object'), our method uses LLMs (e.g., LLaMA-7B [48] and GPT4 [49]) to generate K region concepts, capturing fine-grained concepts such as facial expression or visual patterns. While LLaMA-7B provides diverse concept descriptions, we observed noise and redundancy in object-related concepts; therefore, GPT-4 was additionally used (see Supplementary for details). As illustrated, similar verbs like 'licking' and 'eating' are associated with distinct region-specific signals-enabling the model to better distinguish subtle visual differences, enhancing the discriminability.

Human

Object

Union

LLM

'For the '

A photo of person is licking an objec

t',

give

𝐾

short visual concepts for the [

human/object/union

] region'

'Tongue is extended and curled.'

'Cheeks are slightly puffed out.'

'Lips are parted and relaxed.'

𝐜 h

1

𝐜 h

2

𝐜 h

𝐾

⋮

'Object has a smooth, glossy surface.'

'Object's surface has a distinct flavor or scent.'

'Object's shape or contour is altered by the licking.'

𝐜o

1

𝐜o

2

𝐜o

𝐾

⋮

𝐜u

1

𝐜u

2

𝐜u

𝐾

⋮

'The object is being touched with the tongue for tasting.'

'A person leans in to lick the surface of a food item.'

'The object is partially melted as it is licked by the person.'

Licking an object.

- Person is directing their tongue toward the object.

- Lower jaw is slightly relaxed.

⋮

- Person's lips or mouth are close to the object.

- Traces of saliva or spit are visible.

⋮

- A close-up shows a person tasting the object with

the tip of their tongue.

⋮

- Person is bringing food to their mouth.

- Cheeks are puffed outward with each bite

⋮

- Person's mouth is near or on the object.

- Food is visibly being consumed from the object.

⋮

- The person's eyes are closed in concentration as they eat.

- The person's nose is wrinkled in pleasure as they

savor the object.

⋮

Eating an object.

Retrieved concepts

Retrieved concepts

: Human region concepts

: Object region concepts

: Union region concepts

(A) Generated region concepts

(B) Qualitative results of concepts retrieval

Figure 4: Qualitative examples of region concept generation and retrieval. (A) Given a verb prompt and region type, an LLM generates K region concepts per verb. (B) Retrieved concepts for 'Licking' and 'Eating' highlight subtle region concepts that help disambiguate visually similar interactions. Concepts are color-coded by region: blue (human), red (object), yellow (union).

5 Conclusion

We present VDRP , a prompt learning framework for zero-shot HOI detection that addresses the visual complexity of interaction , including intra-class diversity and inter-class entanglement among verb classes. To this end, we propose a visual diversity-aware prompt learning that injects groupwise visual variance into the prompt context and applies Gaussian perturbation, enabling prompt embeddings to better reflect diverse visual appearances. We also introduce region-aware prompt augmentation , which augments diversity-aware prompts with region-specific concepts from the human, object, and union regions to improve verb discriminability. Extensive experiments on HICODET show that our method outperforms prior work across four zero-shot settings while maintaining high parameter efficiency. These results underscore the value of combining distributional modeling and region-level augmentation for generalizable zero-shot HOI detection.

6 Acknowledgments

This work was partly supported by Korea Research Institute for defense Technology planning and advancement - Grant funded by Defense Acquisition Program Administration (DAPA) (KRIT-CT-23021, 45%), by the Institute of Information & Communications Technology Planning & Evaluation (IITP)-ITRC (Information Technology Research Center) grant funded by the Korea government (MSIT) (IITP-2025-RS-2024-00436857, 45%) and by the National Research Foundation of Korea (RS-2022-NR068758, 10%). We thank Jaewoon Byun for helpful discussion.

References

Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Detecting human-object interaction via fabricated compositional learning. In CVPR , 2021.

Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Affordance transfer learning for human-object interaction detection. In CVPR , 2021.

Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for human-object interaction detection. In ECCV , 2020.

Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng. Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In CVPR , 2020.

Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Efficient two-stage detection of humanobject interactions with a novel unary-pairwise transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 20104-20112, 2022.

Frederic Z Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould. Exploring predicate visual context in detecting of human-object interactions. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10411-10421, 2023.

Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise humanobject interaction detection with image-wide contextual information. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10410-10419, 2021.

Bumsoo Kim, Taeho Choi, Jaewoo Kang, and Hyunwoo J Kim. Uniondet: Union-level detector towards real-time human-object interaction detection. In European Conference on Computer Vision , pages 498-514. Springer, 2020.

Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-toend human-object interaction detection with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 74-83, 2021.

Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng, and Yang Liu. Efficient adaptive human-object interaction detection with concept-guided memory. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6480-6490, 2023.

Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models. Advances in Neural Information Processing Systems , 36:739-751, 2023.

Guangzhi Wang, Yangyang Guo, Ziwei Xu, and Mohan Kankanhalli. Bilateral adaptation for human-object interaction detection with occlusion-robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 27970-27980, 2024.

Jihwan Park, SeungJun Lee, Hwan Heo, Hyeong Kyu Choi, and Hyunwoo J Kim. Consistency learning via decoding path augmentation for transformers in human object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1019-1028, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022.

Sehyung Kim, Chanhyeong Yang, Jihwan Park, Taehoon Song, and Hyunwoo J Kim. Superclass guided transformer for zero-shot attribute classification. arXiv preprint arXiv:2501.05728 , 2025.

Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding-box labels. In ECCV , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16816-16825, 2022.

Jooyeon Kim, Eulrang Cho, Sehyung Kim, and Hyunwoo J Kim. Retrieval-augmented openvocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17427-17436, 2024.

Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard-hoi: Benchmarking few-shot visual reasoning for human-object interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 19056-19065, 2022.

Jie Yang, Bingliang Li, Ailing Zeng, Lei Zhang, and Ruimao Zhang. Open-world human-object interaction detection via multi-modal prompts. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition , pages 16954-16964, 2024.

Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In CVPR , 2022.

Ting Lei, Shaofeng Yin, Yuxin Peng, and Yang Liu. Exploring conditional multi-modal prompts for zero-shot hoi detection. In European Conference on Computer Vision , pages 1-19. Springer, 2024.

Qinqian Lei, Bo Wang, and Robby Tan. Ez-hoi: Vlm adaptation via guided prompt learning for zero-shot hoi detection. Advances in Neural Information Processing Systems , 37:55831-55857, 2024.

Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li. Clip4hoi: Towards adapting clip for practical zero-shot hoi detection. Advances in Neural Information Processing Systems , 36:45895-45906, 2023.

Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. In CVPR , 2023.

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision , 2020.

Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and Mohan S Kankanhalli. Learning to detect human-object interactions with knowledge. In CVPR , 2019.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , 2020.

Jongha Kim, Jihwan Park, Jinyoung Park, Jinyoung Kim, Sehyung Kim, and Hyunwoo J Kim. Groupwise query specialization and quality-aware multi-assignment for transformer-based visual relationship detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 28160-28169, 2024.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML , 2023.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS , 2022.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning , pages 4904-4916. PMLR, 2021.

Jinguo Luo, Weihong Ren, Weibo Jiang, Xi'ai Chen, Qiang Wang, Zhi Han, and Honghai Liu. Discovering syntactic interaction clues for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 28212-28222, 2024.

Eulrang Cho, Jooyeon Kim, and Hyunwoo J Kim. Distribution-aware prompt tuning for visionlanguage models. In Proceedings of the IEEE/CVF international conference on computer vision , pages 22004-22013, 2023.

Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5206-5215, 2022.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 19113-19122, 2023.

Wentao Bao, Lichang Chen, Heng Huang, and Yu Kong. Prompting language-informed distribution for compositional zero-shot learning. In European Conference on Computer Vision , pages 107-123. Springer, 2024.

Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. Advances in Neural Information Processing Systems , 37:2001-2025, 2024.

Jinyoung Park, Juyeon Ko, and Hyunwoo J Kim. Prompt learning via meta-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26940-26950, 2024.

Dongjun Lee, Seokwon Song, Jihee Suh, Joonmyeong Choi, Sanghyeok Lee, and Hyunwoo J Kim. Read-only prompt optimization for vision-language few-shot learning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1401-1411, 2023.

Songlin Dong, Zhengdong Zhou, Chenhao Ding, Xinyuan Gao, Alex Kot, and Yihong Gong. Diversity covariance-aware prompt learning for vision-language models. arXiv preprint arXiv:2503.01531 , 2025.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning , pages 5338-5348. PMLR, 2020.

Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19187-19197, 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International conference on machine learning , pages 16141623. PMLR, 2016.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Mingrui Wu, Jiaxin Gu, Yunhang Shen, Mingbao Lin, Chao Chen, and Xiaoshuai Sun. End-toend zero-shot hoi detection via vision and language knowledge distillation. In Proceedings of the AAAI Conference on artificial intelligence , volume 37, pages 2839-2846, 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Frederic Z. Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object interactions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 13319-13327, October 2021.

Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint arXiv:1505.04474 , 2015.

Hangjie Yuan, Shiwei Zhang, Xiang Wang, Samuel Albanie, Yining Pan, Tao Feng, Jianwen Jiang, Dong Ni, Yingya Zhang, and Deli Zhao. Rlipv2: Fast scaling of relational languageimage pre-training. In Proceedings of the IEEE/CVF international conference on computer vision , pages 21649-21661, 2023.

Danyang Tu, Wei Sun, Guangtao Zhai, and Wei Shen. Agglomerative transformer for humanobject interaction detection. In Proceedings of the IEEE/CVF international conference on computer vision , pages 21614-21624, 2023.

Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural-logic human-object interaction detection. Advances in Neural Information Processing Systems , 36:21158-21171, 2023.

Sedigheh Eslami and Gerard de Melo. Mitigate the gap: Improving cross-modal alignment in CLIP. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=aPTGvFqile .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly state the main contributions and scope in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our method-such as the Gaussian assumption in visual space and the reliance on LLM-generated concepts-in the supplemental material.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our work does not include any theoretical results or formal proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide implementation details and training configurations in the supplementary material to ensure reproducibility of all main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release our code and data upon publication, along with instructions in the supplementary material to reproduce all main results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe the training and evaluation protocols in Section 4 and provide additional details on implementation and hyperparameters in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our evaluation follows standard zero-shot HOI detection benchmarks with deterministic protocols, so statistical significance is not reported.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report training time and GPU specifications in the supplementary material to support reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm that our research adheres to the NeurIPS Code of Ethics, including responsible data usage and transparency in methodology and reporting.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We briefly discuss societal benefits and risks in the conclusion under 'Societal Impact.'

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not involve models or data with high risk of misuse, so this question is not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use publicly available datasets (HICO-DET) and pretrained models (e.g., CLIP, DETR) that are properly cited in the paper, and their licenses and usage terms are fully respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We properly credit and cite all existing datasets, models (e.g., CLIP, HICODET), and related assets used in our work, and follow their license terms as described in the paper and supplementary material.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our study does not involve human subjects and therefore does not require IRB approval.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: We use a large language model to generate region-level concepts (e.g., LLaMA7B) as part of our retrieval-based prompt augmentation module.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

A Implementation details

We conduct all experiments using PyTorch with mixed-precision training. For the base model using CLIP ViT-B/16, we train on two NVIDIA GeForce RTX 3090 GPUs with a batch size of 8 for 12 epochs. We use the AdamW [53] optimizer with an initial learning rate of 1 × 10 -3 , decayed to 1 × 10 -4 using a cosine scheduler and a weight decay of 8. We report the best results after applying weight decay. The framework adopts CLIP ViT-B/16 as the visual encoder, where the adapter applies a bottleneck transformation from d up = 768 to d down = 64 , followed by projection into a shared embedding space of dimension d = 512 . Human-object detection is performed by a frozen DETR model with Resnet50 [54], and detections with confidence below the threshold θ = 0 . 2 are discarded. The context embedding consists of N ctx = 24 learnable tokens, each initialized from a Gaussian distribution with standard deviation 0.02. To incorporate group-wise variance, we inject a modulation vector (scaled by α = 0 . 02 ) into the context embedding and apply Gaussian perturbation to the resulting prompt embedding with a noise scale β = 0 . 1 . For region-aware prompt augmentation, we generate K = 10 region concepts per verb and region type (human, object, union) using LLaMA7B [48] and ChatGPT-4 [49]. These are encoded by the frozen CLIP text encoder and aggregated via Sparsemax [50] to form a concept vector, which is added to the prompt embedding with scaling factor γ = 0 . 2 . We additionally conduct experiments with a scaled-up version using CLIP ViT-L/14 as the visual encoder. In this setting, the adapter transforms d up = 1024 to d down = 64 , with the final embedding dimension set to d = 768 . Due to memory constraints, these experiments are performed on two NVIDIA RTX 6000 Ada Generation GPUs with a reduced batch size of 4 per GPU.

B Details of our architectures

Details of spatial head. To enhance the union-region representation with geometric priors, we design a spatial head that encodes the spatial relationship between the human and object bounding boxes and fuses it with the region features, inspired by prior works [55, 5, 6]. Given a human box b h = [ x 1 , y 1 , x 2 , y 2 ] and an object box b o = [ x ′ 1 , y ′ 1 , x ′ 2 , y ′ 2 ] , we compute the center coordinates, widths, and heights:

We then extract spatial features including normalized positions, relative box areas, aspect ratios, intersection-over-union (IoU), and direction-aware relative distances:

These values are concatenated and passed through a small feedforward network to produce a spatial encoding vector E S . To incorporate this geometric context, we apply a multi-modal fusion module ϕ MMF ( · ) that combines the human and object features [ x h ; X o ] with the spatial encoding:

Finally, the spatial query f S is concatenated with the union feature x ˜ u, and passed through a projection layer to obtain the final spatially-enhanced representation:

which is used for verb classification with visual diversity and region-aware prompts.

Details of sparsemax. To select region-level concepts relevant to each human-object interaction, we adopt the Sparsemax [50] activation function in our region-aware prompt augmentation module. Unlike Softmax, which produces dense probability distributions where all entries are non-zero, Sparsemax enables sparse selection by assigning exact zeros to irrelevant entries. This property is particularly useful in our setting, as only a subset of the K generated region concepts are semantically aligned with the visual feature.

Formally, given a score vector s ∈ R K -representing the cosine similarity between a region feature and K region concept embeddings-we define Sparsemax as a projection onto the probability simplex:

This formulation yields a closed-form solution that projects the input vector onto the probability simplex ∆ K , resulting in a sparse vector a where low-scoring entries are assigned exact zeros. We apply Sparsemax to the similarity scores between each region feature and the corresponding concept pool, allowing the model to focus on the most informative region concepts while ignoring noisy or irrelevant ones.

C Limitations

While our method demonstrates strong performance across multiple zero-shot HOI detection settings, several broader limitations remain. First, the region-aware prompts (RAP) module builds upon region-level concepts generated by large language models (LLMs). Although effective in capturing contextual semantics, LLM-based concepts may be noisy or misaligned with visual concepts due to inherent limitations in language-vision grounding. Improving robustness through confidence-aware filtering or vision-aligned concept refinement is a promising direction. Second, our framework builds on prompt learning, which presents structural challenges when applied to a large number of classes or diverse interaction types. Prompt representations can be sensitive to scaling strategies, initialization, and optimization dynamics, making them less stable in settings with limited data or rare class compositions. This reflects a broader limitation of prompt-based models, where generalization can be affected by prompt granularity, representation collapse, or lack of compositional structure. Future directions may include more robust prompt initialization, adaptive scaling mechanisms, or compositional prompt construction to improve stability and generalization.

D Broader impacts

Our work builds upon vision-language models (VLMs) such as CLIP, which are trained on large-scale web-scraped image-text pairs. As a result, our model may inadvertently inherit biases present in the pretraining data, such as cultural stereotypes or imbalanced representations across demographic groups. While our method aims to improve generalization in zero-shot HOI detection, deployment in real-world applications should be done with caution-particularly in contexts involving surveillance, behavioral analysis, or human activity interpretation. Furthermore, the retrieval-based region prompt augmentation using large language models (e.g., LLaMA-7B and ChatGPT-4), which may also reflect biases or hallucinated associations. Misinterpretation of human-object interactions in sensitive domains (e.g., law enforcement or healthcare) could lead to harmful outcomes if such biases are not properly addressed. To mitigate potential misuse, we recommend restricting deployment to controlled environments with human oversight, and further advocate for evaluating fairness metrics across subpopulations when applying the model to downstream applications.

E Further experiments

V-COCO datasets. V-COCO [56] dataset has 80 object categories derived from COCO datasets [51] and includes 29 interactions, resulting 263 HOI compositions. The number of image samples is 10,396 (5,400/4,964 for train/test).

Fully supervised settings. We further evaluate our method under the fully supervised HOI detection setting on HICO-DET and V-COCO, as summarized in Table 10. Among two-stage methods, our model achieves the best performance on HICO-DET, with an mAP of 39.07, surpassing the prior state-of-the-art EZ-HOI [25]. While one-stage methods such as UniHOI and AGER benefit from end-to-end optimization and full supervision of all components, they typically rely on larger networks and longer training schedules. In contrast, our two-stage approach uses fewer learnable parameters and is designed primarily for zero-shot generalization, which likely limits the effectiveness of group-wise variance modeling and hinders the learning of cross-class structure. On

the V-COCO benchmark, our model achieves 60.6 and 66.2 AP under Scenario 1 and Scenario 2, respectively-comparable to other two-stage models. We attribute this to the small number of verb classes (24 vs. 117 in HICO-DET) and limited dataset size, which reduce the effectiveness of group-wise variance modeling and cross-class structure learning. Overall, these results demonstrate that although our method is designed to address zero-shot HOI detection, it generalizes well to fully supervised settings, retaining strong performance even when supervision is abundant.

Scaled-up CLIP (ViT-L) setting. To assess the effect of scaling the vision backbone, we evaluate our method using the CLIP ViT-L/14 encoder in place of the default ViT-B/16. As shown in Table 11, our model consistently improves performance across all four zero-shot evaluation splits-NF-UC, RF-UC, UO, and UV. Notably, our method achieves the highest harmonic mean (HM) across all settings, reflecting balanced generalization to both seen and unseen verb compositions. While prior methods such as CMMP [24] and EZ-HOI [25] exhibit competitive results on specific splits, their performance fluctuates across evaluation scenarios. In contrast, our scaled-up model maintains stable gains throughout, demonstrating that visual diversity-aware prompt learning and region-aware prompt augmentation remain effective even when paired with high-capacity vision-language encoders. These results highlight the scalability of our framework even when scaled to stronger backbones.

F More ablation studies

Effect of mean vs. variance in VDP. To evaluate the contribution of distributional statistics in our visual diversity-aware prompts (VDP) module, we compare two variants: one that uses only group-wise variance ( σ 2 ), and another that combines both the group-wise mean ( µ ) and variance ( σ 2 ) through concatenation followed by projection. Table 12 reports results across all four zero-shot settings. Overall, we find that the variance-only variant consistently matches or outperforms the mean-variance combination-especially on the Unseen splits in all settings. For instance, in NF-UC, using variance alone yields the highest unseen mAP of 36.45 , compared to 35.75 with µ + σ 2 . This suggests that variance alone serves as a stronger signal for modeling intra-class visual diversity, particularly under zero-shot generalization conditions. Prior studies such as Zhu et al. [40] also report that variance-centered prompt representations improve generalizability by capturing the dispersion of visual features without overfitting to sample means. Our results align with this observation, indicating that the inclusion of the mean may introduce redundant or unstable signals-especially for low-shot or noisy verb clusters, where the class prototype is poorly defined. An exception arises in the UV setting, where the combined variant slightly improves performance on seen metrics. Nonetheless, the overall trend supports the effectiveness and robustness of variance-only modeling for capturing distributional diversity in the prompt space.

Effect of branches in RAP. We compare the individual contributions of the human, object, and union region branches under four zero-shot HOI detection settings. The results are summarized in Table 13. Across all settings, the full model consistently outperforms each individual branch, confirming the complementary nature of region-level concepts. Notably, the proposed combination achieves the best unseen mAP in all settings, demonstrating the effectiveness of region-aware prompts in enhancing discriminability. These findings highlight that human and object concepts capture distinct but complementary contextual signals, while the union branch provides holistic spatial grounding that reinforces interaction understanding.

G Robustness of visual diversity-aware prompt learning

Zero-shot verb variance robustness. To evaluate the robustness of group-wise variance estimation for unseen verbs, we conduct an additional experiment under the UV setting. Specifically, we remove the top-3 semantic neighbors (based on CLIP similarity) for each unseen verb from the training set before computing the group-wise visual variance. This setup simulates a scenario where certain verbs lack semantically related support, allowing us to analyze the stability of variance estimation under reduced contextual guidance. As shown in Tab 14, the removal of neighboring classes results in only a marginal performance drop in unseen verbs (-0.48 mAP). This indicates that the proposed grouping mechanism maintains its generalization ability even when semantic support is limited.

τ -Sparsemax sensitivity. To investigate the effect of sparsity calibration in concept retrieval, we introduce τ -Sparsemax ( · ) , where all values below a threshold τ ∈ { 0 . 0 , 0 . 05 , 0 . 1 } are zeroed post-Sparsemax ( · ) . This modification controls the degree of sparsity in the region-concept selection process. The results in Tab 15 show that excessive pruning ( τ = 0 . 05 ) slightly decreases overall performance, while higher sparsity ( τ = 0 . 10 ) recovers seen-class precision at the cost of unseen generalization. The default τ = 0 . 0 provides the most balanced outcome, indicating that the original Sparsemax ( · ) formulation already achieves an effective level of adaptive sparsity.

Robustness of visual variance under few-shot sampling. We further test the stability of visual variance estimation when fewer visual samples are available. For this experiment, group-wise covariance matrices are computed using only five randomly sampled features per verb ( N v = 5 ), compared to using all available samples. As summarized in Tab 16, the performance reduction across all settings is minimal (typically less than 0.6 mAP), demonstrating that VDP remains robust even under limited sample diversity.

Visual

Prompt

Visual

Prompt

Visual

Prompt

𝟎. 𝟑𝟔𝟔

±𝟎.𝟎𝟗𝟎

𝟎. 𝟏𝟎𝟓

±𝟎.𝟎𝟑𝟏

𝟎. 𝟒𝟐𝟎

±𝟎.𝟎𝟖𝟗

𝟎. 𝟗𝟑𝟕

±𝟎.𝟏𝟑𝟑

𝟎. 𝟓𝟏𝟖

±𝟎.𝟏𝟎𝟓

𝟎. 𝟓𝟓𝟎

±𝟎.𝟏𝟎𝟑

CMMP (UV)

VDRP (UV)

Inter-class distances

Before training

After training

1.0

0.8

0.6

0.4

0.2

十

0.0

̸

H Inter-class alignment of visual and prompt representations

To assess inter-class alignment between visual and prompt representations, we analyze the distributional structure of each modality after training. Specifically, we examine whether our visual diversity-aware prompt learning preserves class-level relationships that are consistent with those observed in the visual embedding space. We measure the pairwise cosine distance between class-level prototypes in both visual and prompt embedding spaces:

̸

where z i denotes the prototype embedding of the i -th verb class. For the visual side, we extract features from the union region of each verb instance using the CLIP visual encoder. Before training, we use the CLS token of the encoder applied directly to the union-cropped image. After training, we adopt a region-specific pooling approach: patch embeddings within the union RoI-Align [44] are aggregated to match the training architecture of our method. To mitigate over-smoothing effects from averaging, we define the visual prototype for each class as the medoid -the instance whose embedding minimizes the average distance to others within the same class. For the prompt side, we collect the final trained verb embeddings for each class. Fig. 5 presents the average inter-class distance for both modalities across three conditions: (1) before training, (2) after training with CMMP [24], and (3) after training with our method (VDRP). We observe that CMMP significantly increases prompt diversity (0.937) while visual diversity remains moderate (0.420), indicating a modality mismatch. In contrast, VDRP maintains a comparable level of diversity in both modalities (prompt: 0.550, visual: 0.518), suggesting improved distributional alignment. This outcome implies that our prompt learning strategy-though primarily guided by intra-class visual variance-can indirectly induce a structured inter-class layout without over-amplifying semantic separation. We draw inspiration from recent works [60, 43, 36, 39, 40, 37], which emphasize distributional approaches to improving visual-prompt alignment. Motivated by this perspective, we interpret the improved visual-prompt alignment as a natural byproduct of our variance-aware design, even though cross-modal matching was not explicitly enforced.

Discussion. This inter-class alignment result further substantiates the novelty of our visual diversityaware design. Unlike prior distribution-based prompt learning methods such as ProDA [37] and DAPT [36], which regularize textual embeddings, VDP explicitly grounds the prompt distribution in the visual modality by injecting group-wise variance extracted from union-region features. This design enables structured yet balanced prompt distributions that mirror the visual embedding space, achieving semantic coherence without explicit regularization losses. Consequently, VDP naturally promotes inter-class consistency and zero-shot generalization-key objectives that motivated our dual-module framework.

I Qualitative Results

Retrieved concepts

Retrieved concepts

Straddling an object.

(0.39) Knees are bent, with feet flat on the ground.

(0.31) Buttocks are slightly raised off the ground.

(0.11) Ankles are flexed, with feet pointing toward the object.

(0.54) Saddle or seat is in use.

(0.19) Object appears to be a bicycle.

(0.16) Frame is in good condition.

(0.33) The object is being pierced in a deliberate downward motion.

(0.23) A pointed tool is used to puncture the surface of the object.

(0.17) A person is pushing a sharp tool into a soft material.

(0.44) Back is straight and upright.

(0.24) Legs are extended in front of them, pedaling.

(0.21) Chest is expanded with each breath.

(0.45) Object has a sleek or aerodynamic shape.

(0.28) Object has a distinctive shape or design.

(0.22) Object has a seat or saddle for the person.

(0.46) The person's feet are off the ground as they ride over a bump.

(0.25) A person is seated on the object, with their legs extended in

front of them.

(0.17) The person's legs are bent and ready to pedal the object.

Riding an object.

(0.49) Mouth is open and exhaling forcefully.

(0.38) Person is blowing air toward the object.

(0.14) Nose is wrinkled in concentration.

(0.32) Object appears to be lifted or moved by air.

(0.23) Object's position appears unstable or shifting.

(0.21) Object's shape or form changes due to air pressure.

(0.44) The person's lips are parted as they blow.

(0.38) A person is blowing air onto the object with their mouth.

(0.15) The person's cheeks are slightly puffed out as they blow.

Blowing an object.

(0.38) Hands are positioned near the top of the object.

(0.32) Shoulders are relaxed and slightly hunched.

(0.16) Person is grasping the object with both hands.

(0.30) Load is being inserted or removed.

(0.27) Load is balanced or centered within the object.

(0.26) Load is secured with straps or clips.

(0.17) Load is heavy or awkward to handle.

(0.33) The object is being lowered into a vehicle.

(0.26) The person's face is focused intently on the task at hand.

(0.24) The object is being lifted into position by the person's

arms.

Loading an object.

: Human region concepts

: Object region concepts

: Union region concepts

: Human box

: Object box

Retrieved concepts

Retrieved concepts

(0.42) Wrists are flexed during the swing.

(0.25) Hands are moving in a circular motion.

(0.13) Hair may be blown back by the motion.

(0.42) Object's weight or momentum creates a subtle arc or

curve in the swinging motion.

(0.26) Swinging motion creates blur or motion blur around

the object.

(0.53) The person's hand is gripping the object firmly as it is swung.

(0.30) The person's arm is extended and the object is at the peak of

its arc.

(0.10) A person is swinging the object in a wide arc.

(0.55) Mouth is open and water is flowing in.

(0.27) Cheeks are puffed out as they inhale.

(0.15) Chin is tilted upward, suggesting a sense of

satisfaction.

(0.42) Liquid level in the object is low.

(0.20) Person's lips are curled back in a sip motion.

(0.19) Object is tilted towards the person's mouth.

(0.44) The person's throat moves as they drink, their Adam's apple

visible.

(0.20) The person's head is tilted back slightly as they drink.

(0.18) A person brings a cup to their lips, sipping from the object.

(0.59) Arms are extended and pulling on oars.

(0.20) Legs are bent and feet are planted firmly on the

platform.

(0.14) Back is straight and facing forward.

(0.67) Rowing motion creates ripples in the water.

(0.20) Object is positioned at an angle in the water.

(0.13) Object is partially submerged in water.

(0.82) The person's legs are tucked in to maintain balance.

(0.07) The person's face is focused on the task at hand.

(0.05) The water is calm and reflective of the surrounding

scenery.

Rowing an object.

(0.33) Legs are bent, providing additional power for the pull.

(0.19) One foot is planted firmly on the ground, providing

stability.

(0.19) Fingers are spread wide, using maximum leverage.

(0.44) Object's surface is wet or dirty from dragging.

(0.26) Object is partially or fully dragged away from its

original position.

(0.13) Surface of object is scuffed or disturbed.

(0.21) The object is heavy, and the person is struggling to move it.

(0.18) The person's legs are moving in a slow, steady rhythm as

they pull the object.

(0.18) The object is bouncing and jolting as the person pulls it along

the ground.

Dragging an object.

Swinging an object.

Drinking with

an object.

: Human region concepts

: Object region concepts

: Union region concepts

: Human box

: Object box

We present qualitative examples in Fig. 6 and 7 to illustrate how region-specific concepts contribute to verb classification. Each example visualizes the top-weighted concepts retrieved from three region branches-human, object, and union-based on sparsemax-normalized concept weights. The full

Retrieved concepts

Retrieved concepts

(0.59) Hands are raised and gesturing toward the object.

(0.19) Shoulders are relaxed and slightly hunched.

(0.17) Chin is slightly raised, as if directing attention.

(0.22) Object is positioned near the person's hand.

(0.22) Object is positioned at eye level.

(0.17) Person's hand is pointing towards the object.

(0.14) Object is partially obscured by the person's body.

(0.33) The person's eyes are fixed intently on the object.

(0.28) The person's posture is relaxed, but their eyes are focused on

the object.

(0.14) The person's body language suggests urgency or importance.

Directing an object.

(0.33) Person is holding a glass or cup with one or both

hands.

(0.26) Hands are raised and angled toward the object.

(0.20) Chin is slightly raised.

(0.90) Person's body language suggests celebration or

appreciation.

(0.08) Person's hand or arm is raised towards the object.

(0.01) Object has a distinctive shape or design.

(0.45) The person's smile is subtle as they toast the object.

(0.29) The person's face is lit by the light of the object.

(0.14) The object's surface glows softly in the light of the

person's glass.

Toasting an object.

(0.35) Arms wrap around the object, enveloping it.

(0.34) Entire body is tense with emotion.

(0.24) Chest is pressed against the object, heart beating

against it.

(0.37) Object's shape is emphasized by the person's embrace.

(0.28) Object's surface is warm to the touch.

(0.14) Object is curved or rounded in shape.

(0.12) Person's face is close to the object's surface.

(0.53) The object is cradled in the person's arms, with their hands

holding it securely.

(0.19) The person's hug is gentle and affectionate, conveying a

sense of warmth and care.

(0.12) The person's hands are intertwined with the object's surface.

Hugging an object.

: Human region concepts

: Object region concepts

: Union region concepts

: Human box

: Object box

Kicking an object.

(0.27) Buttocks are tense and slightly lifted.

(0.27) Person is extending leg to make contact with object.

(0.25) Lower leg is straight and foot is flat on ground.

(0.21) Back is straight and shoulders are relaxed.

(0.50) Object appears to be in mid-air or suspended.

(0.35) Object's surface appears to be vibrating or

trembling.

(0.07) Object's position changes or shifts.

(0.27) The object is propelled through the air.

(0.14) The person's foot is cocked back before making contact.

(0.13) The object is sent flying in a diagonal direction.

(0.13) The object is struck with a firm, direct kick.

Retrieved concepts

Retrieved concepts

(0.31) Eyes are focused on the object and the hand.

(0.25) Arm is extended and bent at the elbow.

(0.22) Shoulder is relaxed or tensed depending on effort.

(0.20) Person is reaching out with one or both hands.

(0.33) Object's movement creates a sense of dynamism or

energy.

(0.27) Light sources create a sense of depth or motion.

(0.16) Object appears to be flying or floating in the air.

(0.28) The person's palm is facing upwards, ready to catch the

object.

(0.27) The object is suspended in mid-air as the person reaches for it.

(0.17) The object is falling towards the person's outstretched hand.

Catching  an object.

(0.54) Weight is evenly distributed across both feet.

(0.12) Chest expands subtly while wearing the object.

(0.11) Head is tilted slightly to accommodate the object.

(0.31) Clothing or accessories around object are well-

coordinated or match.

(0.26) Object is draped over person's body.

(0.26) Object's weight is evident on person's body.

(0.44) The person wears the object around their neck.

(0.27) The object is nestled in the person's pocket.

(0.20) The object is tucked into the person's shirt.

Wearing an object.

(0.46) Fingers are extended to feed the object.

(0.34) Hands are cupped around food, as if to scoop.

(0.20) Person is holding food in their hands.

(0.47) Food is visible in the mouth or opening.

(0.40) Object has a mouth or opening to receive food.

(0.13) Person's hand or utensil is inserted into the object.

(0.41) Food is being offered to the object with a gentle touch.

(0.24) The person's fingers are stained with food.

(0.17) A piece of food is being placed on the object's tongue.

(0.15) The person's hand is positioned near the object's mouth.

Feeding an object.

: Human region concepts

: Object region concepts

: Union region concepts

: Human box

: Object box

Lying on an object.

(0.59) Person is lying flat on their back.

(0.37) Eyelids are slightly closed.

(0.04) Arms are relaxed and slightly bent.

(0.30) Person's body is positioned at an angle.

(0.29) Object appears to be a soft or cushioned surface.

(0.26) Person's face is visible and relaxed.

(0.31) The person's face is relaxed and peaceful.

(0.28) The person's head is resting on a soft, cushioned surface.

(0.27) The object's surface is covered in a patterned blanket.

(0.12) The person's arms are folded behind their head.

concept pool contains K = 10 entries per region type and verb class, and weights are displayed in parentheses. Due to sparsemax activation, concepts with low similarity scores are often assigned zero weight, resulting in more selective and interpretable retrieval. For instance, in kicking an object , the object region retrieves dynamic descriptions such as '(0.50) Object appears to be mid-air or suspended,' while the human region emphasizes posture concepts like '(0.27) Buttocks are tense and slightly lifted.' These complementary signals capture both motion and pose indicative of a kicking action. Similarly, in distinguishing hugging from toasting , human concepts like '(0.35) Arms wrap around the object' help identify close physical contact, whereas in toasting, object and union concepts highlight celebratory gestures such as '(0.90) Person's body language suggests celebration' and '(0.45) The person's smile is subtle as they toast the object.' These examples show how our model selectively retrieves fine-grained region-specific concepts that support verb disambiguation in subtle interaction contexts. Overall, these results show that our region-aware prompt augmentation (RAP) selects meaningful concepts from distinct regions, supporting both interpretability and fine-grained HOI discrimination.

'For the '

A photo of person is

[

lifting

]

an objec

t',

give

𝐾

short visual concepts for the [

object

]

region'

ChatGPT-4

"

The object

is a shiny

metallic surface."

𝐜o

1

"

The object

has a solid,

dense appearance."

𝐜o

2

"

The object

is bulky

and oversized."

𝐜o

𝐾

⋮

"

The object

is angular

and boxy."

𝐜o

3

LLaMA-7B

"

Person's

muscles

tense as they lift"

𝐜o

1

"

Person's

arm or hand

strains under the load"

𝐜o

2

"

Person's

face contorts

in effort or strain."

𝐜o

𝐾

⋮

"

Person's

breathing quickens

or becomes labored."

𝐜o

3

J Analysis of object concept noise from LLaMA-7B

We further analyze the qualitative and quantitative effects of noisy object concepts generated by LLaMA-7B [48]. As discussed in the main paper, the prompt template 'a photo of a person [verb]ing the object' sometimes causes semantic leakage from the human region, producing human-centric descriptions for the object region. Fig. 8 visualizes this issue for the verb lift , where LLaMA-7B frequently associates object concepts with human body parts or actions. In contrast, ChatGPT-4 [49] produces more object-centered and visually grounded descriptions, effectively reducing such leakage. To quantitatively evaluate the impact of concept quality, we compare the model performance when object concepts are generated by LLaMA-7B versus ChatGPT-4. As shown in Table 17, GPT-4 generally achieves higher performance across all zero-shot settings, confirming that cleaner concept representations improve robustness without altering the model architecture.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Visual Diversity and Region-aware Prompt Learning framework for zero-shot HOI detection, aiming to address the visual complexity inherent in human-object interactions. To this end, the authors introduce a visual diversity-aware prompt learning strategy that incorporates group-wise visual variance into the context embeddings. Experimental results on the HICO-DET benchmark validate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper proposes a Visual Diversity and Region-aware Prompt Learning framework for zero-shot HOI detection, aiming to address the visual complexity inherent in human-object interactions.. Moreover, it attempts to generate labels using a text-image model. The experimental results also confirm the effectiveness of the proposed algorithm.\n\nWeaknesses: The paper aims to tackle the problem of zero-shot HOI detection. However, the proposed diversity-aware prompt learning strategy has already been explored in previous studies, limiting the novelty of the contribution. In addition, the experimental evaluation does not include a thorough comparison with current state-of-the-art methods, and the reported performance is not particularly competitive.\n\nQUESTIONS:\n1. In Fig. 2, the paper adopts a fixed template — 'A photo of a person is verbing an object' — as the verb prompt. Did the authors explore alternative or automatically generated prompts? If not, it would be helpful to investigate whether different prompt formulations could impact performance.\n2. The paper lacks an ablation study or empirical analysis to evaluate the contribution of Group-wise variance.\n3. Although this paper is designed for zero-shot HOI detection, it can still be compared with other supervised methods. However, the authors did not conduct a comprehensive comparison.\n4. The reported results for zero-shot HOI detection are not very competitive, and the paper lacks comparisons with recent state-of-the-art methods. For instance, SICHOI (CVPR 2024) reports a score of 36.27, and EZ-HOI (ECCV 2024) achieves 36.73 under the RF setting. Including such comparisons is essential to fairly evaluate the effectiveness of the proposed approach.\n5. Since SICHOI also proposes the concept of multi-view image-text alignment, the novelty of the current work appears to be somewhat limited.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents VDRP (Visual Diversity and Region-aware Prompt learning), for zero-shot Human-Object Interaction detection. The work is motivated by two well-defined challenges inherent to the HOI task. The first is intra-class visual diversity, where a single verb class can be realized through a wide variety of visually distinct poses, scales, and contexts. The second challenge is inter-class visual entanglement, where semantically different verbs often produce visually similar human-object configurations, making them hard to disambiguate from global features alone.\n\nTo address these challenges, VDRP proposes two complementary modules that enhance the learnable textual prompts used for interaction classification within a frozen Vision-Language Model like CLIP.\n\n1. Visual Diversity-aware Prompts (VDP): To tackle intra-class diversity, this module learns a distribution of prompts for each verb rather than a single static embedding. It first estimates the visual feature variance for each verb class from the training data. To create more robust estimates, especially for rare verbs, it groups semantically similar verbs and averages their variances. This \"group-wise variance\" is then used in two ways: it modulates the learnable context of the prompt via a small MLP, and it scales the magnitude of Gaussian noise that is added to the final prompt embedding. This process yields prompts that are designed to cover the diverse visual manifold of each verb class. \n\n2. Region-aware Prompts (RAP): To resolve inter-class entanglement, this module augments the prompts with fine-grained, localized visual information. For each verb, it pre-generates a pool of descriptive textual \"concepts\" for the human, object, and union regions using an external Large Language Model (LLaMA-7B). During inference, the model dynamically retrieves the most relevant concepts from this pool by comparing them to the visual features extracted from the corresponding image regions (human, object, and union). These retrieved concepts are then added to the diversity-aware prompt, creating a final region-aware prompt that is sensitive to the subtle, localized cues necessary to distinguish between visually similar interactions.  \n\nThe paper integrates these two modules into a unified, two-stage HOI detection pipeline. Extensive experiments on the HICO-DET benchmark demonstrate that VDRP achieves new state-of-the-art performance across four standard zero-shot settings, while being more parameter-efficient than previous methods.\n\nSTRENGTHS AND WEAKNESSES:\n# Quality\n\n## Strengths\n\nThe paper is well-motivated, grounding its contributions in a clear and compelling analysis of the core challenges in zero-shot HOI detection. The empirical evaluation is comprehensive and methodologically sound. The model presented in the paper achieved State-of-the-Art performance in all zero-shot settings on the HICO-DET dataset, with thorough ablation studies to validate the contributions of the proposed components.\n\n## Weaknesses\n\nAs the author stated in the \"Limitations\" section in the Supplementary material, The RAP module's effectiveness is critically dependent on the quality and consistency of the concepts generated by the LLM. This introduces several potential points of failure and a lack of robustness that are not adequately addressed in the paper. LLMs are known to be susceptible to generating generic, biased, or factually incorrect information, and their outputs can vary based on the specific model version and prompting strategy. The paper provides no analysis of the quality of the generated concepts, nor does it describe any mechanism for filtering or verifying them. The performance of retrieval-augmented models is known to be sensitive to the quality of the retrieved information. Providing noisy or irrelevant context can actively harm model performance, a phenomenon well-documented in retrieval-augmented NLP. The RAP module is a form of retrieval-augmented generation, yet the paper does not evaluate its robustness to a noisy concept pool.\n\n# Clarity\n\nThe paper is well-written, clearly structured, and easy to follow. The authors have done an excellent job of motivating the problem and presenting their solution.\n\n# Significance\n\n## Strengths\n\nThe work makes a significant and timely contribution to the field. Zero-shot HOI detection is a critical and challenging task for advancing scene understanding, and this paper pushes the state-of-the-art forward meaningfully.\n\n## Weaknesses\n\nHowever, the problem of intra-class visual diversity and inter-class visual entanglement has been identified by some previous works[1][2], although this paper proposed a meaningful method to improve the performance on this task, it is not the first paper to contribute on these issues.\n\n# Originality\n\nThe originality of the paper is good, but it is the area with the most notable weakness. The novelty lies more in the successful synthesis and specific adaptation of existing ideas to the HOI domain, rather than the invention of fundamentally new techniques. \n\nThe core idea of the VDP module is learning a distribution over prompts to capture the visual variance of a class, which is a well-established method. Paper \"Prompt Distribution Learning\" from CVPR 2022, explicitly proposes learning a Gaussian distribution over prompt embeddings to handle intra-class visual diversity in few-shot classification, also, I suggest the authors to cite this paper. The VDP module's use of variance to modulate a context vector and to scale Gaussian noise applied to the final prompt embedding is a direct application of this established principle. The main novel element here appears to be the \"group-wise variance estimation\", which is a clever engineering solution to handle data sparsity for rare verbs, but the conceptual mechanism of learning a prompt distribution is inherited. However, I must emphasize that the VDP module is well tailored for the HOI detection task.\n\nReferences:\n\n[1] Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions\n\n[2] Open-World Human-Object Interaction Detection via Multi-modal Prompts\n\nQUESTIONS:\n1. The VDP module is conceptually very similar to prior work in prompt distribution learning, such as ProDA[1] and DAPT[2], which also model distributions over prompts to capture visual variance. Could the authors please elaborate on the specific novel contributions of VDP beyond the application of these principles to the HOI detection task?\n\n2. As the authors stated in the Supplementary material, the performance of the RAP module is implicitly dependent on the quality of the concepts generated by the external LLaMA-7B model. Has there been any analysis of the model's sensitivity to the quality of these concepts? For example, Table 6 shows that Softmax outperforms your chosen Sparsemax strategy in the UO setting, which seems to contradict the motivation of aggressively pruning irrelevant concepts. Could this difference suggest the concept pool is noisy, making Sparsemax's pruning harmful in some cases?\n\nOverall, I believe this paper offers valuable insights into the longstanding issues of intra-class visual diversity and inter-class visual entanglement in the HOI detection task. My score currently falls between 4 and 5; if the authors can provide reasonable comments addressing my concerns, I am willing to raise my score.\n\nReferences:\n\n[1] Prompt Distribution Learning\n\n[2] Distribution-Aware Prompt Tuning for Vision-Language Models\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VDRP, a framework specifically designed to enhance Zero-shot Human-Object Interaction (HOI) detection by addressing two primary challenges: intra-class visual diversity and inter-class visual entanglement. VDRP employs a visual diversity-aware prompt learning strategy by injecting group-wise visual variations into prompts and applying Gaussian perturbation to capture diverse verb visualizations. Additionally, it incorporates region-specific embeddings from human, object, and union regions to generate region-aware prompts with LLM, thus improving verb-level discrimination. Experimental evaluation on the HICO-DET benchmark shows that VDRP achieves SOTA results on the zero-shot settings of HICO-DET.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper explores the intra-class visual information for zero-shot HOI detection. The proposed visual diversity-aware prompt learning strategy facilitates the verb co-occurrence representation of the verb embeddings. The retrieval-based region prompt augmentation, an advanced LLM guided prompt refinement method, fuses the region-level embeddings while filtering out the irrelevant concepts in the verb embeddings. The proposed method achieves SOTA results with comparable efficiency to the previous methods on the zero-shot settings of HICO-DET.\n\nWeaknesses:\n\nBoth the VDP and RAP is carefully designed for specific purposes, however, the effectiveness of some important part of the introduced components are not sufficient. In Eq. (8), adjusting the hyper-parameter $\\alpha$ directly affects the contribution of the proposed group-wise variance, the impact of the variance is not clear. Similarly, the hyper-parameter $\\beta$ in Eq. (13). In addition, the contribution of the Gaussian perturbation (with or without) according to recent findings is not verified in this paper.\n\nQUESTIONS:\nIn Line 150, the mean $\\mu_v$ and variance $\\sigma_v^2$ are computed over all $N_v$ samples, what if the $N_v$ is fixed to a small number, e.g., 5? Will the performance be affected significantly?\n\nAfter training, what is the distribution of the diversity-aware prompt embeddings $\\tilde{t}^v$? What is the different between the original token embeddings $\\bar{P}_v$?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VDRP, a novel framework for zero-shot Human-Object Interaction (HOI) detection. The method addresses two key challenges: (1) intra-class visual diversity via visual diversity-aware prompts (VDP) that inject group-wise variance and Gaussian noise into CLIP prompts, and (2) inter-class visual entanglement (distinct verbs appear similar) via region-aware prompts (RAP) that augment prompts with LLM-generated concepts retrieved from human/object/union regions. VDRP achieves SOTA results with only 4.5M trainable parameters.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1) Novel technical contributions: The dual-module design (VDP+RAP) elegantly solves core HOI challenges. Variance injection and Sparsemax-based concept retrieval are innovative.  \n2) Rigorous evaluation: Extensive experiments across all four standard zero-shot HOI settings with SOTA results.  \n3) High efficiency: Achieve SOTA with lightweight model.  \n4) Well-structured analysis: Ablations (Tables 4-6) and visualizations (Figures 1-4) effectively validate design choices.  \n\n\n**Weaknesses**\n1) Fragile Zero-Shot Verb Generalization: Group-wise variance estimation (Eq 6) assumes training-set verbs adequately cover all semantic neighbors. For truly novel verbs with no similar training-set counterparts (e.g., \"teeter\"), variance degenerates to a global mean, failing to model diversity.\n2) Sparsemax Sensitivity Unexplored: While Sparsemax aids concept filtering, its threshold sensitivity harms robustness in object-centric settings (UO in Table 6). No analysis confirms if failure stems from noisy object concepts (e.g., \"table\" for both \"sit\" and \"work\") or improper sparsity calibration.\n3) CLIP Phrase Encoding Bias: The text encoder handles compound phrases (\"hold+baseball+glove\") less effectively than atomic concepts, introducing prompt bias. The fixed template design lacks ablation against alternative formulations (e.g., \"person hold glove\").\n\nQUESTIONS:\n1) Zero-Shot Verb Variance Robustness\nSuggestion: Add experiments with intentionally isolated verbs (e.g., remove top-3 semantic neighbors from training). Report mAP on these verbs under UV setting.\n\n2) Sparsemax Failure Diagnosis\nSuggestion:\n(a) Quantify concept noise: Report % of shared concepts across verbs (e.g., \"table\" in verb pairs) for UO vs. UV.\n(b) Test adaptive thresholds: Replace Sparsemax with τ-Sparsemax (adjustable sparsity).\n\n3) Sensitivity Issues of Sparsemax:\nTable 6 shows that Softmax outperforms Sparsemax under the UO setting. The authors attribute this to “unreliable LLM concepts,” but no further analysis is provided:\n(1) Is this due to higher concept noise in object regions (e.g., “table” appears in both “sit at” and “work on”)?\n(2) Is the threshold selection sensitive? The authors did not attempt to adjust the sparsity level of Sparsemax.\n\nLIMITATIONS:\n1. **Computational costs**: No analysis of extra overhead from LLaMA queries/variance computation.  \n2. **Bias propagation**: LLM-generated concepts may encode societal biases (e.g., \"woman cooking\" for \"use stove\").  \n3. **Failure cases**: No discussion of scenarios where VDRP fails (e.g., heavy occlusion or rare verbs).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}