Title: EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Zero-shot HOI detection

Human-Object Interaction

Prompt Learning
Abstract: Detecting Human-Object Interactions (HOI) in zero-shot settings, where models must handle unseen classes, poses significant challenges. Existing methods that rely on aligning visual encoders with large Vision-Language Models (VLMs) to tap into the extensive knowledge of VLMs, require large, computationally expensive models and encounter training difficulties. Adapting VLMs with prompt learning offers an alternative to direct alignment. However, fine-tuning on task-specific datasets often leads to overfitting to seen classes and suboptimal performance on unseen classes, due to the absence of unseen class labels. To address these challenges, we introduce a novel prompt learning-based framework for Efficient Zero-Shot HOI detection (EZ-HOI). First, we introduce Large Language Model (LLM) and VLM guidance for learnable prompts, integrating detailed HOI descriptions and visual semantics to adapt VLMs to HOI tasks. However, because training datasets contain seen-class labels alone, fine-tuning VLMs on such datasets tends to optimize learnable prompts for seen classes instead of unseen ones. Therefore, we design prompt learning for unseen classes using information from related seen classes, with LLMs utilized to highlight the differences between unseen and related seen classes. Quantitative evaluations on benchmark datasets demonstrate that our EZ-HOI achieves state-of-the-art performance across various zero-shot settings with only 10.35\% to 33.95\% of the trainable parameters compared to existing methods. Code is available at https://github.com/ChelsieLei/EZ-HOI.

EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection

Abstract

Detecting Human-Object Interactions (HOI) in zero-shot settings, where models must handle unseen classes, poses significant challenges. Existing methods that rely on aligning visual encoders with large Vision-Language Models (VLMs) to tap into the extensive knowledge of VLMs, require large, computationally expensive models and encounter training difficulties. Adapting VLMs with prompt learning offers an alternative to direct alignment. However, fine-tuning on task-specific datasets often leads to overfitting to seen classes and suboptimal performance on unseen classes, due to the absence of unseen class labels. To address these challenges, we introduce a novel prompt learning-based framework for Efficient Zero-Shot HOI detection (EZ-HOI). First, we introduce Large Language Model (LLM) and VLM guidance for learnable prompts, integrating detailed HOI descriptions and visual semantics to adapt VLMs to HOI tasks. However, because training datasets contain seen-class labels alone, fine-tuning VLMs on such datasets tends to optimize learnable prompts for seen classes instead of unseen ones. Therefore, we design prompt learning for unseen classes using information from related seen classes, with LLMs utilized to highlight the differences between unseen and related seen classes. Quantitative evaluations on benchmark datasets demonstrate that our EZHOI achieves state-of-the-art performance across various zero-shot settings with only 10.35% to 33.95% of the trainable parameters compared to existing methods. Code is available at https://github.com/ChelsieLei/EZ-HOI .

1 Introduction

Human-Object Interaction (HOI) detection localizes human-object pairs and identifies the interactions. HOI has various practical applications, including robot manipulations, human-computer interaction, and human activity understanding [38, 41, 24, 32, 26, 1]. Additionally, it is a building block for related tasks such as action recognition, visual question answering, and image generation [18, 24, 4, 43, 11, 54]. However, the limited generalization ability of many existing HOI detectors makes zero-shot HOI detection particularly challenging, as it requires models to identify unseen HOI classes.

Vision-Language Models (VLMs) [47, 29, 2, 30] have gained popularity due to their extensive knowledge bases and effective ability to process and correlate complex patterns across visual and text data. Consequently, a group of methods has been developed to align HOI visual features with those of VLMs, leveraging the extensive knowledge from these models [51, 33, 44, 41, 4]. This alignment ensures that both the HOI model and the VLM can extract similar features to represent the same concept (e.g., an action). The degree of feature alignment can be quantified using cosine similarity, which measures the similarity between feature vectors. However, aligning with VLMs requires

12

90

66.2

42.1

14.1

7.1

28.8

17.2

24.3

21.0

28.7

31.2

26.3

36.8

30.2

32.2

27.8

38.2

GEN-VLKT

HOICLIP

MaPLe

Ours

(a)AligningHOl model with VLMs

(b)Prompttuningoverfitstoseenclasses

UnseenmAP

Allclasses

VLMtext米

Allclasses

VLMtext

HOlmodel

VLMvisual

Full

prediction

prediction

mAP

mAP

←Allclasses

VLMGuidance

LLM

VLMtext

VLMvisual

Unseen

Seen

Trainableparameters

Epochs

prediction

(c) Our efficient zero-shot HOl detection (EZ-HOl)

(d) Zero-shot Hol detection comparison

training transformer-based models, which is computationally expensive and leads to extended training time and significant difficulties.

An alternative approach involves adapting VLMs for HOI tasks, bypassing the demanding alignment process and leveraging VLM capabilities for action understanding in HOI detection. Prompt tuning, which adapts VLMs to various downstream tasks with a small number of learnable parameters, has shown considerable efficacy [62, 61, 19, 21]. One notable method, MaPLe [22], utilizes multi-modal prompt tuning specifically for image classification tasks. However, since adaptation typically involves only seen class labels, the adapted VLMs often overfit to seen classes but struggle to deal with unseen classes in zero-shot settings [62, 7, 9, 22]. Consequently, this limitation results in suboptimal performance for unseen classes in zero-shot HOI detection.

In this paper, we introduce a novel method, Efficient Zero-Shot HOI detection (EZ-HOI), to enhance VLM adaptation via guided prompt learning with information from foundation models (e.g., LLM and VLM). We design both learnable text and visual prompts, leveraging fixed LLM and VLM to guide the adaptation process. Specifically, the text prompts are designed to capture detailed HOI class insights from an LLM, while the visual prompts are tailored to incorporate external visual semantics from a VLM. However, as the training datasets contain images with seen-class labels alone, the trainable prompts are naturally optimized for these seen classes rather than the unseen ones.

To address this limitation, we develop the Unseen Text Prompt Learning (UTPL) to leverage prompts from related seen classes effectively. We begin by measuring the relationship between HOIs using cosine similarity of text embeddings, which helps identify closely related seen classes for each unseen one. After establishing these connections, we enhance the learning of unseen prompts based on those from selected seen classes. To capture the distinctions between unseen and seen classes, we incorporate an LLM. This LLM provides what we call 'disparity information', enhancing the learning of the distinctions and similarities between seen and unseen classes. Additionally, we enhance our approach by introducing intra- and inter-HOI feature fusion techniques following the visual encoder. Our method achieves competitive performance on established benchmarks in various zero-shot settings while requiring significantly fewer trainable parameters.

As illustrated in Fig. 1, we compare our method with two existing paradigms: (a) aligning HOI features to a fixed VLM and (b) adapting a VLM for HOI tasks. Unlike the existing approaches, our method facilitates VLM adaptation to HOI tasks and demonstrates effective generalization to unseen classes. Fig. 1(d) highlights that our method is competitive in terms of performance, model parameter efficiency, and training duration.

In summary, our contributions are as follows:

We introduce EZ-HOI, a novel framework for zero-shot HOI detection that adapts a VLM to HOI tasks via guided prompt learning with foundation model information, enhancing the adapted VLM's generalizability in zero-shot HOI detection.

We propose the UTPL module, which extracts information from related seen classes for the unseen learnable prompts. This module mitigates overfitting to seen classes, a common issue in task-specific VLM adaptations, improving performance on unseen classes.

Our EZ-HOI method achieves state-of-the-art performance in various zero-shot settings while significantly reducing trainable parameters and mitigating training challenges. It lowers trainable model parameters by 66% to 78% compared to existing methods, demonstrating enhanced efficiency and effectiveness in zero-shot HOI detection.

2 Related Work

Human-Object Interaction Detection Human-Object Interaction (HOI) detection involves identifying human-object pairs and recognizing their interactions, serving as a fundamental component for various downstream tasks in computer vision [53, 6]. HOI detection methods are typically categorized into two classes: one-stage and two-stage. One-stage methods simultaneously generate all outputs, including human bounding boxes, object bounding boxes and categories, and interaction classes. Recent advancements in one-stage detectors leverage transformer architectures, delivering promising performance [8, 46, 48, 63, 55, 52, 24]. An example is HOITrans [63], which utilizes the transformer's encoder and decoder to extract interaction features. The output features are then processed through multi-layer perceptrons to produce all output predictions at once. On the other hand, two-stage approaches divide HOI detection into two tasks: object detection and HOI classification [10, 14, 58]. Since separation allows each module to specialize, two-stage HOI detection is more memory-efficient [59]. Recent developments have seen the integration of transformer-based architectures into two-stage designs, which have shown promising results [59, 60, 45, 27]. Our method also falls into the two-stage design category.

Zero-Shot HOI Detection Prior zero-shot HOI detection efforts primarily address unseen composition settings, where models encounter action and object classes separately but not as combinations. To address this, several methods employ compositional learning strategies aimed at tackling the unseen-composition problem [14, 16, 15, 27]. However, compositional learning falls short in addressing unseen verb zero-shot HOI detection scenarios. Given the substantial image understanding capabilities of VLMs, they offer promising potential for enhancing HOI detection in zero-shot settings [2, 29, 30, 47, 37, 36], where labels for unseen HOI classes are absent from the training dataset. Recent studies have explored aligning HOI visual features with VLM [33, 44, 4, 41]. However, this approach requires training transformer-based models, which are computationally expensive. Consequently, aligning HOI models with VLMs is demanding, resulting in extended training times and significant training challenges.

Prompt Learning Prompt learning has gained popularity for adapting VLMs to downstream tasks [61, 62, 7, 9, 19, 23, 25, 40]. Context Optimization (CoOp) [62] refines the prompt input of the text encoder by combining learnable domain-shared prompt tokens with class prompt tokens. MaPLe [22] integrates learnable domain-shared text tokens with visual learnable tokens. This combination leverages the highly connected text and visual encoders in VLMs, facilitating the sharing of crossdomain information and benefiting both text and visual domains. However, fine-tuning VLM relies on training datasets with only seen class labels, which causes adapted VLMs to excel with seen classes but encounter difficulties with unseen ones in zero-shot scenarios. Consequently, this limitation results in suboptimal performance for unseen classes in zero-shot HOI detection.

3 Methodology

We start with adapting a pre-trained CLIP model [47] to zero-shot HOI tasks using an innovative prompt learning approach. We design two sets of learnable prompts: text and visual. The visual prompts are derived from the text prompts by using projection layers. We denote learnable text prompts as h T and learnable visual prompts as h V :

ℎ 1

, ℎ

2

𝑇1

𝑇𝑁

𝑉 1

𝑉𝑁

𝑊𝑡1

𝐸ℎ𝑜𝑖1

𝐸ℎ𝑜𝑖 2

𝐸ℎ𝑜𝑖𝑞

Text encoder

：frozen

Describe a

It describes aperson

:finetune

person

usingabaseball bat

swinginga

:concatenate

swinginga

tohitaball...

baseballbat.

:unseen class

baseballbat

LLM

prompt token

:seen class

UTPL

prompttoken

LLM Guidance

MHCA

break

hold

Learnableprompt

：：

UTPL

swing

Visual encoder

Image input

lensA WA

Proj

￥

HOl feature fusion

MHCA(

Object detector

HOl prediction

HOIprediction

where h T ∈ R p ∗ d t , and h V ∈ R p ∗ d v . d t is the dimension of the text feature, and d v is the dimension of the visual feature. p represents the number of tokens we design for each learnable prompt.

Let V = { v 1 , v 2 , · · · , v N v } as the verb set and O = { o 1 , o 2 , · · · , o N o } as the object set. Then the HOI set include all feasible verb-object pairs C = { hoi i = ( v i , o i ) | v i ∈ V ; o i ∈ O } . We set C as the total number of HOI classes. Since it is impractical for any datasets to include comprehensive HOI classes, researchers propose zero-shot HOI detection settings to encourage the generalization of HOI detectors to unseen classes C unseen in inference [33]. Denote the seen HOI class set as S and the unseen HOI class set as U = { hoi i | hoi i / ∈ S , hoi i ∈ C } . Please refer to Appendix 7.9 for detailed definitions of the different zero-shot setting unseen set splits.

3.1 LLMand VLM Guidance for Learnable Prompts

Prompt learning techniques for VLMs are characterized by their ability to adapt large, pre-trained models to specific tasks using relatively small amounts of task-specific data. This often leads to a diminished generalization ability of the fine-tuned models for unseen classes [62, 7, 9, 22]. Given that foundation models, such as LLMs and VLMs, have demonstrated substantial knowledge capacity [29, 47], leveraging guidance from these models can improve performance on unseen HOI classes that are absent in the training data.

Text Prompt Design As shown in Fig. 2, we design input prompts, 'Describe a person <acting> <object> and focus on the action' related to each HOI class, for an LLM. The output from LLM contains richer semantic information with specific HOI class descriptions. Please refer to Appendix Section 7.2 for a detailed explanation with examples.

Then, we use a CLIP text encoder to obtain the text embedding for this HOI class description. We process text embeddings for all HOI classes including both seen and unseen in parallel, so the whole text embeddings are denoted as F txt ∈ R C ∗ d t and F txt = [ f txt 1 , f txt 2 , · · · , f txt C ] . Then, the learnable text prompts ˆ h T can be obtained by:

ℎ𝑇

ℎ𝑜𝑠𝑒

ℎ𝑇

𝑤𝑎𝑠ℎ

ℎ𝑇

ℎ𝑜𝑠𝑒

ticsimilarity

MHCA

<unseen> Hose a dog.

<seen>Wash a dog

Promptfordisparity

VLMtext

LLM

description

Describethephrase:hosinga

“hosingadog”isdifferentfrom

dogandfocusonthedifference

“"washinga dog"

g”asthepersonisusing

fromwashingadog.

ahosetocleanthedog,ratherthan...

Semanticsimilarity

<unseen>

Hoseadog

<seen>

Washadog

Unseenclass:

VLMtext

Seenclass:

Washa dog

O:concatenate

:unseenclassprompttoken

:seenclassprompttoken

where ˆ h T = [ ˆ h T 1 , ˆ h T 2 , · · · , ˆ h T C ] , and ˆ h T i ∈ R p ∗ d t . Each class has its specific learnable text prompts after the process of Eq. 2. W up and W down indicate the up-projection and down-projection layers. MHCA means the multi-head cross attention [50]. In Eq. 2, we only aggregate the useful information from F txt using learnable attention, as the information provided by LLM may not all be relevant for our task. Moreover, to keep trainable parameters small, we apply a down-projection layer W down before MHCA to reduce the feature dimension, and an up-projection layer W up afterward. The same design strategy is used in Eq. 3 and Eq. 5. Later, the prompts corresponding to unseen classes are further refined, as detailed in Section 3.2.

Visual Prompt Design Visual prompt learning in our method is facilitated by a pre-trained Visual Language Model (VLM) visual encoder, specifically the CLIP visual encoder [47]. The pre-trained CLIP model inherently possesses knowledge of unseen HOI classes, offering richer visual semantics compared to models trained solely on task-specific datasets. The CLIP visual feature is denoted as f I vis for an image I , and we enhance the visual learnable prompts h V by:

Since the frozen VLM visual encoder can extract features for unseen HOIs, ˆ h V aggregates information from these visual features, improving performance on unseen HOIs.

3.2 Unseen-Class Text Prompt Learning (UTPL)

Since the training data has no labeled image for unseen HOI classes, learnable text prompts for seen classes are inevitably optimized better than unseen classes. Therefore, we refine the unseen-class text prompts by learning from closely related seen-class prompts. Specifically, we denote one unseen HOI class as u , and the learnable text prompt tokens as ˆ h T u , and denote one seen HOI class as s and the related seen text prompt token as ˆ h T s To identify the related seen class, we use text embeddings of the HOI class descriptions generated by the LLM, as explained in Text Prompt Design of Section 3.1. The related seen class is formulated as follows:

Directly refining prompts for unseen classes based solely on selected seen classes may be insufficient due to inherent differences between the seen and unseen classes. Thus, we propose utilizing disparity information, defined as the differences between unseen class u and seen class s , as provided by the LLM. The architecture of UTPL is shown in Fig. 3. Please refer to Appendix Section 7.3 for detailed explanation with examples.

Since the description can be too long to be encoded at once, we process each sentence separately by using the CLIP text encoder. Thus, set the text embedding of disparity descriptions for an unseen HOI class u as f disp txt u ∈ R m ∗ d t , where m is the number of sentences in the text description. Then we can compute the refined learnable text prompts ˜ h T u for unseen class u as:

where concat indicates concatenation. For seen classes, ˜ h T s = ˆ h T s . Although ground-truth labels for the unseen classes are not available during training, we propose two strategies to update ˜ h T u in Eq. 5. First, we design a class-relation loss (refer to Eq. 16 in the Appendix) to keep the relationship between seen and unseen classes, measured by cosine similarity between text features. As a result, unseen prompts can be refined based on their relation to seen classes. Second, the annotated training data serves as negative samples for unseen HOIs. If the prediction score for an unseen class is too high, the model is penalized (Eq. 17 in the Appendix).

3.3 Deep Visual-Text Prompt Learning

We denote a set of learnable text prompts as H T = [ ˜ h 1 T , ˜ h 2 T , · · · , ˜ h N T ] and learnable visual prompts as H V = [ ˆ h 1 V , ˆ h 2 V , · · · , ˆ h N V ] . ˜ h i T ∈ R C ∗ p ∗ d t , ˆ h i V ∈ R p ∗ d v , i = 1 , 2 , · · · , N . N means we intend to introduce new learnable prompts from the first layer to layer N .

Deep Text Prompt Learning The text encoder is composed of K transformer layers {T i } K i =1 .The CLIP text encoder generates text features by tokenizing the words and converting them into word embeddings W 0 = [ w 1 0 , w 2 0 , · · · , w P 0 ] ∈ R P ∗ d t . Text features W i will be processed by i th layer of the text transformer:

The final text representation W t can be obtained by:

where W t ∈ R 1 ∗ d a and d a is the feature dimension of the final aligned text and visual features in CLIP. Since our learnable text prompt ˜ h i T is specific for each HOI class, the whole text representation can be obtained W t = [ W t 1 , W t 2 , · · · , W t C ] .

Deep Visual Prompt Learning The visual encoder is also composed of K transformer layers {V i } K i =1 . After each layer V i , there is an adapter A i to inject object position and category information [27]. CLIP visual encoder splits image I into D = d p ∗ d p fixed-size patches, which are projected to obtain visual features E 1 ∈ R D ∗ d v . Visual features E i are processed by i th layer of the visual transformer:

where c i is the class token for the i th layer.

The final visual representation E v ∈ R 1 ∗ d a can be computed as follows:

E v ∈ R D ∗ d v can be re-sized to E v ∈ R d p ∗ d p ∗ d v . Since off-the-shelf detectors can provide object detection results, we select bounding boxes with confidence score sc > θ and apply RoI-Align [13] to obtain features for each detected human and object E hum , E obj . Then, the intra-HOI feature fusion extracts HOI feature E hoi ∈ R 1 ∗ d a from E hum , E obj :

where MLP represents multi-layer perception. In any given image I , multiple humans and objects may be present, leading to the detection of several potential human-object pairs. Assume there are q human-object (H-O) pairs in image I and denote all HOI features as E I hoi = [ E 1 hoi , E 2 hoi , · · · , E q hoi ] . Incorporating surrounding HOI features enriches context information for each HOI visual feature by capturing additional human-object relational details and interactions, thereby enhancing HOI features. We name this process inter-HOI feature fusion. Thus, final visual representation ˜ E I hoi is obtained by:

where MHSA means multi-head self-attention [50] and ˜ E I hoi = [ ˜ E 1 hoi , ˜ E 2 hoi , · · · , ˜ E q hoi ] . Then, we can calculate the prediction for each H-O pair hoi i .

Please refer to Appendix Section 7.5 for more details for training and inference.

4 Experiments

4.1 Zero-Shot HOI Setting Definition

Our method follows the existing zero-shot HOI setting, which involves predicting unseen HOI classes and typically includes using unseen class names during training [14, 15, 16, 44, 51]. In particular, VCL, FCL and ATL [14, 16, 15] 'compose novel HOI samples' during training with the unseen (novel) HOI class names. EoID [51] distills CLIP 'with predefined HOI prompts' including both seen and unseen class names. HOICLIP [44] introduces 'verb class representation' during training, including both seen and unseen classes.

4.2 Implementation Details

We evaluate our method on HICO-DET by following the established protocol of zero-shot two-stage HOI detection methods [14, 3, 27]. Our object detector utilizes a pre-trained DETR model [5] with a ResNet50 backbone [12]. As for our learnable prompts design, we set p = 2 , N = 9 . The LLaVA-v1.5-7b model [37] is used to provide text description, as explained in Section 3.1 and 3.2. For all experiments, our batch size is set as 16 on 4 Nvidia A5000 GPUs. We use AdamW [39] as the optimizer and the initial learning rate is 1e-3. For more implementation details, please refer to Appendix Section 7.1.

4.3 Comparison with State-of-the-Art Zero-Shot HOI Methods

Unseen-Verb Setting In Table 1, we compare our method to existing zero-shot HOI detection approaches under the unseen-verb zero-shot setting. The results demonstrate that our method not only achieves competitive performance but also requires only 10.35% to 33.95% of the trainable parameters compared to existing zero-shot HOI detection methods thanks to our novel prompt learning design. While our method shows a minor drop in performance compared to CLIP4HOI under the unseen verb setting, it is important to consider the significant reduction in trainable parameters that our method achieves-87.9% fewer than CLIP4HOI. In contrast, the existing HOI methods [33, 44, 41, 4] that align with VLMs unanimously require significantly more trainable

model parameters (e.g., 42.05M ∼ 66.18M with ViT-B visual encoder), whereas our method operates efficiently with only 6.85M trainable parameters.

UniHOI [4] is the state-of-the-art method in the unseen verb (UV) setting. Compared to UniHOI which aligns HOI features to BLIP [29] text embeddings, our method achieves improved performance, especially in the unseen category with a 2.77 increase in mAP. At the same time, our model requires only 26.9% of the trainable parameters compared to UniHOI. This demonstrates the effectiveness and efficiency of our proposed prompt learning framework in adapting the VLM for zero-shot HOI tasks.

Unseen-Composition Setting In Table 2, we provide the zero-shot performance comparison in rare-first unseen-composition (RF-UC) and nonrare-first unseen-composition (NF-UC) settings. With the ViT-B visual encoder, our method establishes a new standard for unseen-class performance across both RF and NF settings, outperforming the previous state-of-the-art method, CLIP4HOI [41], while requiring only 12.08% of its trainable parameters. Compared to UniHOI with ViT-L visual encoder [4], our method surpasses it by 5.56 mAP in unseen performance of the RF-UC setting and by 7.88 mAP in the NF-UC setting with only 26.9% of the trainable parameters of UniHOI.

Unseen-Object Setting We provide the unseen-object setting comparison in Table 3. Following previous methods [44, 33], we employ a DETR object detector pre-trained on MS-COCO [34] for object detection. To keep consistent with CLIP4HOI [41] and FCL [16], we also provide the results by using the DETR model pre-trained on HICO-DET [27] (marked with † in Table 3). CLIP4HOI [41] is the state-of-the-art method in unseen performance. With the same object detector pre-trained on HICO-DET, our method shows improved performance for unseen class prediction, outperforming CLIP4HOI by 1.49 mAP, with only 12.08% of its trainable model parameters. UniHOI [4] is the state-of-the-art method in seen performance. With the same object detector as UniHOI, our method outperforms UniHOI in unseen category by 13.36 mAP.

4.4 Ablation Studies

We conduct the ablation study for our text and visual prompt learning design in Table 4. The first row shows the performance of our baseline, MaPLe [22]. Our intra-HOI fusion module improves the seen HOI performance by 7.41 mAP, as shown in the second row. In the third row, the inclusion of the visual adapter [27] enhances performance for seen classes while negatively affecting unseen HOI performance.

Comparing the third row and fourth row, LLM guidance, shown in Fig. 2, notably enhances unseenclass learning, resulting in a 1.52 mAP improvement. LLM guidance, by integrating learnable text prompts with detailed HOI class descriptions, improves the model's understanding of unseen classes, providing richer information compared to simple class names. Additionally, the inclusion of the UTPL module significantly boosts unseen class performance, with a 2.42 mAP improvement. Later, the inter-HOI fusion benefits both the seen and unseen learning by providing more context

information for visual HOI features. Finally, the VLM guidance effectiveness is evaluated, which enhances the unseen performance by 1.33 mAP.

We also provide the experimental comparison with CLIP [47] and MaPLe [22] in Table 5. Since they are designed for general image classification, directly applying it to HOI tasks for comparison is unfair as they may not focus on the human-object region features. Therefore, we crop the union region from the original visual features for each human-object pair to obtain the HOI feature. HOI prediction is obtained by using cosine similarity between HOI and text features for each class.

As shown in Table 5, although with learnable prompts, MaPLe [22] outperforms CLIP on seen classes, its performance on unseen classes is far behind that of the seen classes, showing reduced generalization capability. Compared to our method, MaPLe lags significantly behind on unseen performance, with an 11.63 decrease in mAP. Additionally, the third row introduces another baseline that incorporates MaPLe with visual adapter (i.e., A i ) [27], which is also used in our framework, ensuring a fair comparison. The performance improvement from the third row to the fourth row, with gains of 5.93 mAP for unseen classes and 3.86 mAP for seen classes, clearly demonstrates the effectiveness of our method in zero-shot HOI detection.

4.5 Qualitative Results

Fig. 4 shows the qualitative results of MaPLe [22] and our method in the unseen-verb zero-shot setting. In particular, MaPLe struggles to detect unseen classes, either missing unseen HOI classes or predicting the wrong unseen HOI classes. For example, if an image only contains unseen classes, MaPLe tends to predict wrong seen classes and miss the correct ones. As shown in the bottom right of Fig. 4, this image contains unseen class only ("wear tie"), MaPLe predicts related wrong seen classes such as "pulling tie" and "adjusting tie", but fails to predict the ground-truth unseen HOI ("wear tie"). This shows the limited generalization ability of MaPLe to unseen classes. In contrast to MaPLe, our method can predict both seen and unseen classes more accurately.

4.6 Fully Supervised Setting for HOI Detection

We conducted the fully-supervised experiments on both the HICO-DET and V-COCO benchmarks [6, 34]. Our method achieves a competitive 38.61 mAP on the HICO-DET dataset, with a smaller performance drop between rare and non-rare classes (1.19 mAP) compared to AGER [49] (4.18 mAP), the current state-of-the-art one-stage method. Our method also outperforms the best-performing zero-shot HOI method, CLIP4HOI [41], by 3.28 mAP on HICO-DET. On the V-COCO benchmark, our method achieves state-of-the-art performance with 66.2 AP in Scenario 2. Detailed discussions and comparisons are provided in the Appendix 7.6.

MaPLe

MaPLe

MaPLe

xx

Ours

Ours

Ours

Wash

Noact

PIOH

Swing

Throw

Inspect

Repair

Direct

MaPLe

MaPLe

MaPLe

×

Ours

Ours

Ours

Feed

Pet

Milk

Feed

Watch

Pet

Wear

Ind

4.7 Discussion of Descriptors from LLMs

Descriptors, developed to leverage the LLMs for downstream tasks [42, 20], involve attribute decomposition to benefit category recognition. We adopt the attribute decomposition concept from DVDet [20] and tailor it to our EZ-HOI framework. Following DVDet, we generate action descriptors for each class and integrate them into HOI class text features, thus enhancing the detail and distinctiveness of class representations. Descriptors with low cosine similarity to the class text features are discarded to avoid noise. With our straightforward adoption of action descriptors, the result shows 0.31 mAP improvement compared to our EZ-HOI, achieving 32.63 mAP under the unseen-verb setting. This indicates that LLM-generated descriptions, such as action descriptors, hold potential to enhance HOI detection and are worth further exploration.

4.8 Limitations

Our method has some limitations. First, while our method significantly reduces the number of trainable parameters compared to existing approaches, it still requires fine-tuning on HOI-specific datasets such as HICO-DET. A training-free design, which could be used directly for HOI detection without the need for fine-tuning, would be more desirable. Second, zero-shot HOI detection requires the pre-definition of all HOI classes, both seen and unseen, following the current zero-shot HOI detection setting. This requirement restricts the generalizability to truly unseen classes. Consequently, our future work will aim to develop a robust open-category HOI detector that operates effectively without the need for predefined classes.

5 Conclusion

In this paper, we introduced the Efficient Zero-shot HOI Detection (EZ-HOI) approach, utilizing prompt learning to adapt visual-language models for zero-shot HOI detection. This method not only maintains competitive performance for unseen classes but also innovatively integrates learnable visual and text prompts. These prompts leverage foundation model information, thereby enriching prompt knowledge and enhancing the adaptability of VLMs. A significant challenge we addressed is the compromised performance on unseen classes, resulting from the training dataset containing only seen-class labels. To counter this, we developed a text prompt learning strategy that utilizes information from related seen classes to support the detection of unseen classes. We also employed an LLM to provide the nuanced differences between related seen and unseen classes, improving our method for unseen class prompt learning. Our method has demonstrated state-of-the-art performance across various zero-shot HOI detection settings while requiring only a third of the trainable parameters compared to existing methods.

References

Yihao Ai, Yifei Qi, Bo Wang, Yu Chen, Xinchao Wang, and Robby T. Tan. Domain-adaptive 2d human pose estimation via dual teachers in extremely low-light conditions. In ECCV , 2024.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35: 23716-23736, 2022.

Ankan Bansal, Sai Saketh Rambhatla, Abhinav Shrivastava, and Rama Chellappa. Detecting human-object interactions via functional generalization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 10460-10469, 2020.

Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models. Advances in Neural Information Processing Systems , 36, 2024.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020.

Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect humanobject interactions. In 2018 ieee winter conference on applications of computer vision (wacv) , pages 381-389. IEEE, 2018.

Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01253 , 2022.

Junwen Chen and Keiji Yanai. Qahoi: query-based anchors for human-object interaction detection. arXiv preprint arXiv:2112.08647 , 2021.

Eulrang Cho, Jooyeon Kim, and Hyunwoo J Kim. Distribution-aware prompt tuning for visionlanguage models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22004-22013, 2023.

Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human-object interactions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8359-8367, 2018.

Albert Gordo and Diane Larlus. Beyond instance-level image retrieval: Leveraging captions to learn a global visual representation for semantic retrieval. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6589-6598, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for humanobject interaction detection. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16 , pages 584-600. Springer, 2020.

Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Affordance transfer learning for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 495-504, 2021.

Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Detecting human-object interaction via fabricated compositional learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14646-14655, 2021.

Zhi Hou, Baosheng Yu, and Dacheng Tao. Discovering human-object interaction concepts via self-compositional learning. In European Conference on Computer Vision , pages 461-478. Springer, 2022.

ASM Iftekhar, Hao Chen, Kaustav Kundu, Xinyu Li, Joseph Tighe, and Davide Modolo. What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5353-5363, 2022.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision , pages 709-727. Springer, 2022.

Sheng Jin, Xueying Jiang, Jiaxing Huang, Lewei Lu, and Shijian Lu. Llms meet vlms: Boost open vocabulary object detection with fine-grained descriptors. arXiv preprint arXiv:2402.04630 , 2024.

Baoshuo Kan, Teng Wang, Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng. Knowledge-aware prompt tuning for generalizable vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15670-15680, 2023.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19113-19122, 2023.

Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15190-15200, 2023.

Sanghyun Kim, Deunsol Jung, and Minsu Cho. Relational context learning for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2925-2934, 2023.

Dongjun Lee, Seokwon Song, Jihee Suh, Joonmyeong Choi, Sanghyeok Lee, and Hyunwoo J Kim. Read-only prompt optimization for vision-language few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1401-1411, 2023.

Qinqian Lei, Bo Wang, and Robby T Tan. Few-shot learning from augmented label-uncertain queries in bongard-hoi. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 2974-2982, 2024.

Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng, and Yang Liu. Efficient adaptive human-object interaction detection with concept-guided memory. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6480-6490, 2023.

Ting Lei, Shaofeng Yin, and Yang Liu. Exploring the potential of large foundation models for open-vocabulary hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16657-16667, 2024.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023.

Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural-logic human-object interaction detection. Advances in Neural Information Processing Systems , 36, 2024.

Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human-object interaction. Advances in Neural Information Processing Systems , 33:5011-5022, 2020.

Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2012320132, June 2022.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 , 2023.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

Xinpeng Liu, Yong-Lu Li, Xiaoqian Wu, Yu-Wing Tai, Cewu Lu, and Chi-Keung Tang. Interactiveness field in human-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20113-20122, 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. Advances in Neural Information Processing Systems , 36, 2024.

Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li. Clip4hoi: Towards adapting clip for practical zero-shot hoi detection. Advances in Neural Information Processing Systems , 36, 2024.

Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183 , 2022.

Gyeongsik Moon, Heeseung Kwon, Kyoung Mu Lee, and Minsu Cho. Integralaction: Posedriven feature integration for robust human action recognition in videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3339-3348, 2021.

Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23507-23517, 2023.

Jeeseung Park, Jin-Woo Park, and Jong-Seok Lee. Viplo: Vision transformer based poseconditioned self-loop graph for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17152-17162, 2023.

Xian Qu, Changxing Ding, Xingao Li, Xubin Zhong, and Dacheng Tao. Distillation using oracle queries for transformer-based human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19558-19567, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise humanobject interaction detection with image-wide contextual information. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10410-10419, 2021.

Danyang Tu, Wei Sun, Guangtao Zhai, and Wei Shen. Agglomerative transformer for humanobject interaction detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 21614-21624, October 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Mingrui Wu, Jiaxin Gu, Yunhang Shen, Mingbao Lin, Chao Chen, and Xiaoshuai Sun. End-toend zero-shot hoi detection via vision and language knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 2839-2846, 2023.

Chi Xie, Fangao Zeng, Yue Hu, Shuang Liang, and Yichen Wei. Category query learning for human-object interaction classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15275-15284, 2023.

Bangpeng Yao and Li Fei-Fei. Recognizing human-object interactions in still images by modeling the mutual context of objects and human poses. IEEE transactions on pattern analysis and machine intelligence , 34(9):1691-1703, 2012.

Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In Proceedings of the European conference on computer vision (ECCV) , pages 684-699, 2018.

Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, and Mingqian Tang. Rlip: Relational language-image pre-training for human-object interaction detection. Advances in Neural Information Processing Systems , 35:37416-37431, 2022.

Hangjie Yuan, Shiwei Zhang, Xiang Wang, Samuel Albanie, Yining Pan, Tao Feng, Jianwen Jiang, Dong Ni, Yingya Zhang, and Deli Zhao. Rlipv2: Fast scaling of relational languageimage pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21649-21661, 2023.

Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225 , 2022.

Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object interactions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13319-13327, 2021.

Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Efficient two-stage detection of humanobject interactions with a novel unary-pairwise transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20104-20112, 2022.

Frederic Z Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould. Exploring predicate visual context in detecting of human-object interactions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10411-10421, 2023.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16816-16825, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, et al. End-to-end human object interaction detection with hoi transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11825-11834, 2021.

7 Appendix

7.1 Implementation Details

Here we provide detailed implementation information. For the pre-trained CLIP model with the ViT-B visual encoder, the visual feature dimension d v = 768 , while the text feature dimension d t = 512 and the final feature dimension for aligned visual and text features d a = 512 . For the CLIP model with the ViT-L visual encoder, d v = 1024 , d t = 768 , d a = 768 . We use an off-the-shelf object detector and add a threshold θ to filter out some low-confident predictions and we set θ = 0 . 2 . Since UniHOI [4] and CLIP4HOI [41] have not released their code, we estimate the trainable model parameters by modifying the HOICLIP [44] code according to the description in UniHOI and CLIP4HOI, which means that if some details are not mentioned in the UniHOI and CLIP4HOI papers, we use the HOICLIP design by default. In addition, we initialize the weights of all W up to 0, which stabilizes training by gradually fine-tuning the attention outputs in Eq. 2, Eq. 3, Eq. 5 and Eq. 11.

Dataset and Evaluation Metrics

We conducted extensive experiments using the widely-recognized HICO-DET [6] dataset for zeroshot HOI detection. HICO-DET comprises a total of 47,776 images, divided into 38,118 training images and 9,658 test images. This dataset features 600 Human-Object Interaction (HOI) classes, which are combinations derived from 117 action categories and 80 object categories. Our model's performance was evaluated in four distinct zero-shot HOI detection settings, categorized by the criterion for selecting the unseen HOI classes: rare-first unseen composition (RFUC), nonrare-first unseen composition (NFUC), unseen object (UO), and unseen verb (UV). These settings align with methodologies from previous research [33, 44, 41, 4].

Additionally, we also evaluate our model in the fully supervised setting on both the HICO-DET dataset and the V-COCO [34] dataset. V-COCO is a subset of COCO, with 10,396 images-split into 5,400 train-val images and 4,946 test images, and it encompasses 24 action classes and 80 object classes. Following standard evaluation protocols, we assessed our model using mean average precision (mAP) on the HICO-DET benchmark, while Average Precision (AP) in both Scenario 1 and Scenario 2 is used on the V-COCO benchmark [63, 44]. A prediction was deemed a true positive if the HOI classification was accurate and the Intersection over Union (IoU) between the predicted human and object bounding boxes and the ground-truth bounding boxes exceeded 0.5 [33, 27].

7.2 HOI Class Description

We utilize the LLM to provide detailed HOI class descriptions to guide the text prompt learning and we provide an example for detailed illustration. We take the HOI class 'Swing a baseball bat' as an example, which is one unseen HOI class in the unseen-verb zero-shot setting on the HICO-DET benchmark. The generated HOI class description is

'Swinging a baseball bat' describes a person using a baseball bat to hit a ball. This action typically involves the person holding the bat with both hands, standing in a stance with their feet shoulder-width apart, and using their body rotation to contact the ball.

7.3 Disparity Information for UTPL Module

The disparity information mentioned in Section 3.2 aims to explore the difference between the unseen class and the related seen class. Here we demonstrate it with a specific example. We take the unseen class ' hose a dog' as an example and its selected seen class is 'wash a dog'.The input text prompt for LLM to acquire the disparity description between the two classes is designed as ' Describe the definition of the phrase: hosing a dog and please focus on the attributes different from another phrase: washing a dog. ' Then we can obtain the detailed disparity description from LLM as

The phrase "a person hosing a dog" refers to the action of washing a dog using a hose. This action is different from "a person washing a dog" as the person is using a hose to clean the dog, rather than simply using their hands to wash the dog. The use of a hose adds an additional element of water pressure and flow, which can make the cleaning process more efficient and effective.

𝑊𝑡1

𝐸ℎ𝑜𝑖1

𝐸ℎ𝑜𝑖𝑞

𝐸ℎ𝑜𝑖2

HOl prediction

：frozen

:finetune

encoder

text

carry skis

ridecar

detector

HOltextclass

encoder

Concat.

visual

M

L

P

MHSAN

Intra-HOl Fusion

Inter-HOlFusion

7.4 HOI Feature Fusion

As shown in Fig. 5, we also provide the detailed architecture of the intra- and inter-HOI feature fusion design. The intra-HOI feature fusion integrates human and object features for each human-object pair. To enrich context information, all HOI features are processed together using multi-head self-attention (MHSA), allowing each feature to become aware of its surroundings. This enhancement improves the context and overall performance of each HOI feature.

7.5 Training and Inference

Since W t ∈ R C ∗ d t related to the number of total HOI classes, if the number is too large, the learnable prompts can be computationally expensive. Thus, we only select part of HOI classes and do action classification. Later, by combining action prediction with object detection results, we can obtain HOI prediction finally. Specifically, we select two HOI classes for each action with the following equation:

which means we select two HOI classes containing the same action with the most different semantic meanings. Since some actions can have multiple interpretations in different contexts (e.g., "hold apple" vs. "hold sheep"), randomly choosing two HOI classes with the same action does not cover the comprehensive meanings of the action. By selecting the most different HOI classes, we aim to capture a richer range of information for the action. Then, We calculate the HOI prediction score for i th H-O pair by:

The action scores can be obtained by:

where ˜ l align ∈ R C ∗ d a means the action label for each selected HOI class. We adopt focal loss [35] l focal to train the action prediction. We also design a class-relation loss shown in the following equation:

to keep the relation between adapted text class embeddings W t close to the original text embeddings of the HOI class description f txt . D KL [ ·||· ] denotes the KL divergence. Therefore, the training loss L train is computed by:

and α is a hyperparameter. In the inference stage, we can obtain the HOI score for each human-object pair by:

where s h , s o are the object detection confidence scores for humans and objects. τ is a hyperparameter. In Eq. 17, α = 150 , and in Eq. 18, τ = 1 during training and τ = 2 . 8 during inference [58, 59].

7.6 Quantitative Results

Fully Supervised HOI Detection Setting We conduct experiments under the fully supervised HOI detection setting on both the HICO-DET [6] and V-COCO [34] dataset, as shown in Table 6. In the UTPL module, we utilize common class prompts as inputs. This approach enables common HOIs to learn from rare ones, fostering a better differentiation between them. This strategy helps to alleviate overfitting to common HOI classes, thereby improving performance for both seen and unseen classes. As shown in Table 6, the performance drop between rare and non-rare classes ( 1.19 mAP) is much smaller than AGER (4.18 mAP), the best performance among the one-stage method. Additionally, our method achieves the best performance among the two-stage methods on the HICO-DET [6] dataset, outperforming CLIP4HOI [41] by 3.28 mAP.

As for the V-COCO benchmark, we achieve competitive performance among the two-stage methods with 66.2 AP performance under Scenario 2 evaluation. However, the smaller number of verb classes in V-COCO (24 classes), which have weaker connections (i.e., jumping vs. skateboarding skateboard), compared to HICO-DET (117 verb categories), which exhibits stronger connections (i.e., jumping vs. flipping skateboard), limits the potential of our method. Our UTPL design requires the learnable prompts to extract information from prompts of other classes, which also aids in better differentiation between similar classes. Due to the weakly connected classes in V-COCO, our UTPL design cannot work effectively. Despite this, we achieve performance comparable to CLIP4HOI [41], further demonstrating the effectiveness of our method.

7.7 Ablation Studies

Table 7 shows the ablation study for the hyperparameter N , where N means we introduce learnable prompts from the first layer until layer N in both visual and text encoders. We use N=9 in our main paper because it shows the best unseen performance.

Table 8 shows the ablation study for the different positions of learnable prompts. Position i -j means we insert learnable prompts from layer i to layer j in both the text encoder and visual encoder. Here we always insert learnable prompts into 9 layers and only change the position. We find that the position of learnable prompts does not affect the outcome too much. Thus, we follow [22] and fine-tune layers 1-9, which show slightly better unseen performance.

7.8 Qualitative Results

As shown in Fig. 6, we provide more qualitative results in three zero-shot HOI settings: unseenverb, rare-first unseen-composition, and nonrare-first unseen-composition settings. Compared to MaPLe [22], our method obtains better generalization capability to unseen HOI classes owing to our efficient prompt learning design.

In Fig. 7, we show more qualitative comparisons to illustrate the effectiveness of the LLM guidance and UTPL design. The baseline here refers to the method in the main paper's third row of Table 4. LLM guidance design utilizes the general description from LLM for each HOI class and the UTPL module integrates the distinctive description from LLM. As shown in Fig. 7, the LLM guidance provides detailed class information, improving the performance over the baseline. Distinctive descriptions from UTPL design help to distinguish unseen classes from related seen HOIs, enhancing unseen performance and challenging case predictions.

7.9 Discussion for Zero-Shot HOI Detection Definition

Our method follows the standard zero-shot HOI setting, where unseen class names are used during training [14, 15, 16, 51, 44], as discussed in Section 4.1. Specifically, before training, the whole verb set V = { V seen , V unseen } and the whole object set O = { O seen , O unseen } are all pre-defined. The four settings of zero-shot HOI detection include : 1) Rare-First Unseen Composition (RF-UC), where for all hoi i = ( v i , o i ) ∈ U , we have v i ∈ V seen , o i ∈ O seen and hoi i appears less than 10 times in the training set, belonging to rare HOI classes. 2) Nonrare-First Unseen Composition, where for all hoi i = ( v i , o i ) ∈ U , we have v i ∈ V seen , o i ∈ O seen and hoi i appears more than 10 times in the training set, belonging to nonrare HOI classes. 3) Unseen Verb (UV), where hoi i = ( v i , o i ) ∈ U , we have v i ∈ V unseen , o i ∈ O seen . 4) unseen Object (UO), where hoi i = ( v i , o i ) ∈ U , we have v i ∈ V seen , o i ∈ O unseen .

Beyond the zero-shot HOI setting, there are HOI unknown concept discovery [17] and openvocabulary HOI detection [28], where unseen class names cannot be used in training. The openvocabulary setting differs from HOI unknown concept discovery, with a much wider range of unseen HOI classes during testing.

7.10 Discussion of Broader Impacts

Our approach to zero-shot HOI detection reduces reliance on extensive annotated datasets, enhancing accessibility for organizations with limited resources and promoting inclusivity in technology adoption. This technology could notably improve assistive devices, offering more intuitive aids for individuals with disabilities by enabling better understanding of new environments.

However, these advancements also pose risks. The capability to interpret human-object interactions could be used for surveillance, potentially infringing on privacy. Additionally, reliance on existing visual-language models may maintain embedded biases, leading to discriminatory outcomes. To

(a) UV setting

(b) UV setting

(c) UV setting

(d) UV setting

(e) UV setting

(f) UV setting

(g) RF-UC setting

(h) RF-UC setting

(i) RF-UC setting

(j) NF-UC setting

(k) NF-UC setting

(l) NF-UC setting

MaPLe

MaPLe

MaPLe

Ours

Ours

Ours

Feed

PIOH

Hug

Inspect

Repair

PIOH

Wash

Pet

MaPLe

MaPLe

MaPLe

+

X

Ours

Ours

Ours

Inspect

Wash

Direct

Hug

PIOH

Straddle

AA68VAE

Board

Drive

Park

MaPLe

MaPLe

MaPLe

X

Ours

Ours

Ours

Wash

Park

No act

Adjust

Wear

Ride

Kiss

PIOH

Feed

MaPLe

MaPLe

MaPLe

xx

xx

Ours

Ours

Ours

Ride

dunr

Push

Ride

Walk

PIOH

Carry

Ride

address these concerns, we recommend developing rigorous ethical guidelines and governance frameworks to regulate the deployment of HOI detection technologies, alongside efforts to identify and mitigate biases in foundational models.

(a): Baseline; (b): Baseline w/ LLM guidance; (c): Baseline w/ LLM guidance& UTPL

Unseen class: hosecar Confidence score Increasing rate

22% 36% 100%

(a)

(b)

(c)

Unseen class: inspectfirehydrant Confidence score Increasing rate

22% 82% 100%

(a)

(b）

(c)

Unseen class: Weartie Confidence score Increasing rate

16% 34% 100%

(a)

(b)

(c)

Confusing class: board airplane Confidence score Increasing rate 12% 48% 100%

(a)

(b)

(c)

Confusing class: Race horse

Confidencescore Increasing rate

34% 72% 100%

(a)

(b)

(c)

Unseen class: swing baseball bat Confidence score Increasingrate

13% 53% 100%

(a)

(b)(c)

Unseen class: ride elephant Confidence score Increasing rate

29% 42% 100%

(a)

(b)

(c)

Unseen class: swing baseball bat Confidence score Increasingrate

8% 50% 100%

(a)

(b)

(c)

Confusing class: flip skateboard Confidence score Increasing rate 37% 59% 100%

(a)

(b）

(c)

Confusing class: grind skateboard

Confidence score Increasing rate

16% 52% 100%

(a)

(b)

(c)

'S'M'

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We emphasize our contributions in the Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We included the Limitation section in the main paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We don't include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide comprehensive implementation details both in main paper and in appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We included the website link for the code repository in the abstract and our evaluation is based on public datasets.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We included them in the implementation details in both main paper and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer:[No]

Justification: We follow the default evaluations in the HOI detection field, which doesn't require error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide them in implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conducted in the paper conform, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discussed the positive and negative societal impact in the supplementary material.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cited the original paper that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn order to solve the HOI detection problem in a zero-shot setting, the authors propose the Unseen-class Text Prompt Learning module. Using learnable visual prompts and textual prompts, it effectively utilizes the knowledge from large language models and Vision-Language Models and performs well in unseen class HOI detection.\n\nSTRENGTHS:\n1.The work is innovative, well-written and has clear diagrams.\n2.The authors' proposed learnable prompts scheme and UTPL module are novel.\n\nWEAKNESSES:\n1. In Qualitative Results, the authors use Figure 4 for visual illustration, but it is not possible to conclude from Figure 4 that \"MaPLe tends to predict seen classes with high confidence scores\". The authors should have given a clearer and more accurate illustration of the figure.\n2. The authors claim to have \"employed an LLM to provide the nuanced differences between related seen and unseen classes, improving our method for unseen class prompt learning. prompt learning.\" However, the information provided by the LLM may be misleading or inaccurate, did the authors filter the information? If not, what can be done to avoid HOI detection being negatively affected from the information provided by LLM?\n3. For the UTPL module, the authors split the LLM-generated description into multiple sentences; how do they control the length of individual statements?\n4. The author mentions: \" A prediction was deemed a true positive if the HOI classification was accurate and the Intersection over Union (IoU) between the predicted human and object bounding boxes and the ground-truth bounding boxes exceeded 0.5.\", why is the threshold value chosen to be 0.5 instead of 0.7 or other? What is the basis for the choice?\n5. WRITING DETAILS: Abbreviated nouns should be introduced the first time they appear.\n\nQUESTIONS:\n1. In Qualitative Results, the authors use Figure 4 for visual illustration, but it is not possible to conclude from Figure 4 that \"MaPLe tends to predict seen classes with high confidence scores\". The authors should have given a clearer and more accurate illustration of the figure.\n2. The authors claim to have \"employed an LLM to provide the nuanced differences between related seen and unseen classes, improving our method for unseen class prompt learning. prompt learning.\" However, the information provided by the LLM may be misleading or inaccurate, did the authors filter the information? If not, what can be done to avoid HOI detection being negatively affected from the information provided by LLM?\n3. For the UTPL module, the authors split the LLM-generated description into multiple sentences; how do they control the length of individual statements?\n4. The author mentions: \" A prediction was deemed a true positive if the HOI classification was accurate and the Intersection over Union (IoU) between the predicted human and object bounding boxes and the ground-truth bounding boxes exceeded 0.5.\", why is the threshold value chosen to be 0.5 instead of 0.7 or other? What is the basis for the choice?\n5. WRITING DETAILS: Abbreviated nouns should be introduced the first time they appear.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

comments: SUMMARY:
This work presents EZ-HOI, an innovative framework that addresses the zero-shot HOI detection challenge by employing prompt learning. It integrates LLM and VLM guidance to enrich prompts and adapt to HOI tasks effectively. By learning from related seen classes, EZ-HOI overcomes the limitation of lacking labels for unseen classes, enhancing its performance on them. The framework achieves state-of-the-art results with significantly fewer trainable parameters than existing methods, showcasing its efficiency and effectiveness in zero-shot HOI detection.

STRENGTHS:
- The paper is well-written.
- The Unseen Text Prompt Learning is well-designed.
- Overall, the zero-shot results with less trainable parameters are good.

WEAKNESSES:
- This paper primarily addresses the improvement of zero-shot HOI detection benchmark performance by training with the pre-definition of all HOI classes. This creates an unfair comparison with previous work.
- In light of the above, the reviewer believes that the authors should provide experimental results where the pre-definition of all HOI classes is unknown during training, allowing for a fairer comparison with previous work.

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of human-object interaction (HOI) detection. This paper introduces EZ-HOI, a method for efficient zero-shot HOI detection in an open-world setting. EZ-HOI also explores the use of LLM and VLM guidance for learnable prompts to enhance prompt knowledge and aid in adapting to HOI tasks. To better adapt to unseen classes, EZ-HOI establishes relationships between unseen and seen classes.\n\nSTRENGTHS:\n- The state-of-the-art experimental results in the zero-shot setting. EZ-HOI demonstrates good performance in zero-shot scenarios. More importantly, by leveraging prompt tuning, the number of trainable parameters in EZ-HOI is significantly smaller compared to other methodologies.\n- The code in the supplementary material is provided to ensure the reproducibility of the study.\n\nWEAKNESSES:\n- The overall method is straightforward but lacks sufficient novelty. Compared to UniHOI, which also claims to utilize Spatial Prompt Learning, the main difference in EZ-HOI lies in its use of more traditional prompt learning methods (like VPT-DEEP) and its additional modeling of relationships between unseen and seen classes when handling unseen categories. Since many previous works have already demonstrated that prompt tuning (ViT-adapter/CLIP-adapter) can achieve even higher performance than full fine-tuning, the claims of EZ-HOI seem somewhat weak. Additionally, modeling the relationship between unseen and seen classes as a technical contribution also appears somewhat limited.\n- The fully-supervised setting results are provided in Appendix Table 6, but the performance is worse than Uni-HOI. Firstly, this is a very important experiment and should be included in the main text. Secondly, it appears that under full supervision, EZ-HOI performs worse than Uni-HOI, which contradicts the claims in the main text. Have the authors provided an in-depth analysis of the reasons? If the performance difference is due to parameter count, what would the performance be with full fine-tuning?\n- The performance improvement seems to come from the baseline rather than the proposed modules. As shown in the ablation study (Table 4), the baseline of EZ-HOI achieves 37.44 on seen categories, which is already state-of-the-art (SOTA). I wonder what EZ-HOI’s baseline is and why it can achieve such strong performance on seen categories. Additionally, considering that the UTPL module increases unseen category performance by two points, and EZ-HOI shows a two-point increase compared to Uni-HOI, does this mean that EZ-HOI’s performance gain over Uni-HOI is primarily due to the UTPL module?\n\nQUESTIONS:\nWhat is the fully fine-tuned performance? What is the baseline?\n\nLIMITATIONS:\nThis submission discusses limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel prompt learning framework for zero-shot Human-Object Interaction (HOI) detection, which enhances the generalizability of Vision-Language Models (VLMs) by interacting with Large Language Models (LLMs) to obtain descriptions.\n\nSTRENGTHS:\n1. The proposed method appears logical and achieves general improvements in performance.\n2. Utilizing LLMs to generate descriptions is a promising direction.\n3. The paper is well-written, and the framework diagram is clear.\n\nWEAKNESSES:\n1. The prompt design seems to be a hybrid approach, combining visual prompt learning [1, 3] with text prompts, such as those used in COCOOP, without a specific focus on task-specific configurations. It is difficult to justify the necessity of this work for the field.\n\n2. Despite the claim of improved performance on novel classes, the method performs worse on unseen classes compared to CLIP4HOI, while it shows improved performance on seen classes. Given that novel class performance is crucial, this result seems not consistent with this task.\n\n3. W_down presents in both Equation 3 and Equation 5 is puzzling. Equation 5 attempts to integrate descriptions for unknown categories with similar known categories and LLM-generated details. More details on the training process would be helpful, as it appears that data training might be lacking since unknown categories are not included in the training.\n\n4. There is substantial related work on using LLMs for new class descriptions in zero-shot scenarios, such as CuPL [2] for classification and DVDet [3] for detection, which use attribute decomposition to aid in categorization. DVDet also generates more discriminative descriptors by distinguishing confusing categories. The paper lacks a discussion on such related LLM-based works. It would be interesting to see if using action descriptors might also yield some benefits.\n\n5. The paper could benefit from providing more visualization results to demonstrate the effectiveness of both general and discriminative descriptions generated by the method.\n\n[1] Visual Prompt Tuning\n\n[2] VISUAL CLASSIFICATION VIA DESCRIPTION FROM LARGE LANGUAGE MODELS\n\n[3] LLMS MEET VLMS: BOOST OPEN-VOCABULARY OBJECT DETECTION WITH FINE-GRAINED DESCRIPTORS\n\nQUESTIONS:\nPlease see the weakness.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the challenges behind successful adaption of pre-trained vision language models (VLMs), i.e., CLIP, to the problem of zero-shot Human Object Interaction (HOI) detection. Specifically, during finetuning, VLMs overfit to seen HOI classes observed during the training on HOI training data, preventing successful transfer on the unseen HOI classes. To overcome the aforementioned problem, authors propose prompt tuning mechanism coupled with guidance from a large language model (LLM). In particular, LLM is used to generate (i) descriptions of each HOI class; (ii) explain the difference between each unseen and the corresponding closest seen class to facilitate transfer on unseen classes. The prompt tuning technique itself learns a set of prompts that are shared between vision and text encoders. Authors employ attention mechanism to produce modality specific prompts via attending on class descriptions for text modality and via attending on the image embedding for visual modality correspondingly. Authors evaluate the proposed approach on the standard HOI benchmark and compare it to the recent baselines, showing improvements in terms of the parameter efficiency and the performance on unseen classes.\n\nSTRENGTHS:\n* The problem of enabling generalization to unseen classes during VLMs adaptation is important, even outside HOI detection field\n* To my knowledge, learning shared prompts and producing modality specific prompts via attention mechanism is novel\n* The evaluation is thorough, both comparing to HOI specific prompt tuning baselines and to general purpose prompt tuning approaches such as MaPLe.\n\nWEAKNESSES:\nMy main concern is that the current narration of the proposed methodology lacks intuition and structure. For example, \n\n**Structure**\n* Section 3 starts with mentioning that the method will learn prompts per layer, and the subsequent section also use notation considering N layers, but the first time the reader encounters interaction with the layers is Section 3.3. It makes the paper hard to follow.\n\n**Intuition**\n* The main idea of the approach builds on repeatedly applying MHCA for different purposes, however these equations just stated on the high-level in Eq. (2); (3); (5) and without any intuition behind the design.\n\n* Similarly to the above, Eq. (6) and Eq. (8) state how these learnable prompts are used in the corresponding encoder layers, however the narration is on the high level and does not elucidate the particular idea behind the equations.\n\nQUESTIONS:\n* How $N=9$ was chosen? Can you provide ablations for different number of $N$?\n* Similarly to the question above, Eq. (6) and (8) suggest that the learnable prompts are inserted in the first $N$ layers of textual and visual encoders, correspondingly. What is the motivation behind this design choice? I, in general, would assume that earlier layers already consumed broad knowledge during CLIP pre-training and task specific adaptation is required for penultimate layers. Can you please provide ablations on different positioning of the learnable prompts?\n* Currently, the paper narration mainly focuses on tackling generalization to unseen classes and, indeed, experimentally confirms the improvements offered by the proposed approach. The main component causing such improvements is seem to be UTPL that employs guidance from LLM. However, according to Table 4, other components also bring substantial improvements. Can you please provide the intuition behind other components' role and how they can bring these improvements?\n\nLIMITATIONS:\nAuthors discuss the limitations of the current zero-shot HOI detection setting. In particular, that to truly enable zero-shot HOI detection, one must exclude the assumption of having in advance a set of unseen HOI classes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}

comments: SUMMARY:
This paper tackles zero-shot HOI detection via prompt tuning. To address the challenge posed by the absence of novel classes, the authors first incorporate LLM and VLM guidance to enrich learnable prompt tokens. Further, the authors utilize LLM to provide nuanced differentiation between unseen classes and their related seen classes, termed Unseen Text Prompt Learning (UTPL), to alleviate overfitting to seen classes. In experiment, the proposed method achieves state-of-the-art performance under major zero-shot HOI detection settings while costing fewer training resources.

STRENGTHS:
1. This paper tackles an important issue in zero-shot / open-vocabulary learning, i.e., overfitting to seen classes.
2. The proposed method effectively explores guidance from large foundation models, including VLM and LLM.
3. The authors achieve state-of-the-art performance with significantly fewer training resources.

WEAKNESSES:
1. It seems the authors assume the prior knowledge of unseen class names when training the UTPL module, which conflicts with the zero-shot setting.

2. In Table 1, the proposed method underperforms CLIP4HOI (ResNet50+ViT-B) in unseen mAP although it is claimed to tackle the overfitting issue especially.

QUESTIONS:
1. In Figure 2, are the VLM visual and visual encoder the same thing (i.e., CLIP)?

2. Why did the authors use LLaVA for text description instead of choosing pure language models?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-31