EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection
Authors: Qinqian Lei
Bo Wang
Robby T. Tan
Venue: NeurIPS 2024 poster (2024)
Keywords: Zero-shot HOI detection
Human-Object Interaction
Prompt Learning
Primary area: machine_vision

Abstract: Detecting Human-Object Interactions (HOI) in zero-shot settings, where models must handle unseen classes, poses significant challenges. Existing methods that rely on aligning visual encoders with large Vision-Language Models (VLMs) to tap into the extensive knowledge of VLMs, require large, computationally expensive models and encounter training difficulties. Adapting VLMs with prompt learning offers an alternative to direct alignment. However, fine-tuning on task-specific datasets often leads to overfitting to seen classes and suboptimal performance on unseen classes, due to the absence of unseen class labels. To address these challenges, we introduce a novel prompt learning-based framework for Efficient Zero-Shot HOI detection (EZ-HOI). First, we introduce Large Language Model (LLM) and VLM guidance for learnable prompts, integrating detailed HOI descriptions and visual semantics to adapt VLMs to HOI tasks. However, because training datasets contain seen-class labels alone, fine-tuning VLMs on such datasets tends to optimize learnable prompts for seen classes instead of unseen ones. Therefore, we design prompt learning for unseen classes using information from related seen classes, with LLMs utilized to highlight the differences between unseen and related seen classes. Quantitative evaluations on benchmark datasets demonstrate that our EZ-HOI achieves state-of-the-art performance across various zero-shot settings with only 10.35\% to 33.95\% of the trainable parameters compared to existing methods. Code is available at https://github.com/ChelsieLei/EZ-HOI.

EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection
Qinqian Lei
Bo Wang" Robby Tan
National University of Singapore
2 University of Mississippi
3 ASSU Intelligent Cloud Services (AICS)
qinqian leicu nus edu, hawk rwwchhgmail......... com,
Abstract
Detecting Human-Object Interactions (HOI) in zero-shot settings, where models must handle unseen classes, poses significant challenges. Existing methods that rely on aligning visual encoders with large Vision-Language Models (VLMs) to tap into the extensive knowledge of VLMs, require large, computationally expensive models and encounter training difficulties. Adapting VLMs with prompt learning offers alternative direct alignment. However, fine-tuning on task-specific datasets often leads to overfitting to seen classes and suboptimal performance on unseen classes, due the absence of unseen class labels. To address these challenges, we introduce a novel prompt learning-based framework for Efficient Zero-Sho HOI detection (EZ-HOI). First, we introduce Large Language Model (LLM) and VLM guidance for learnable prompts, integrating detailed HOI descriptions and visual semantics adapt VLMs HOI tasks. However, because training datasets contain seen-class labels alone, fine-tuning VLMs on such datasets tends to optimize learnable prompts for seen classes instead of unseen ones. Therefore, we design prompt learning for unseen classes using information from related seen classes, with LLMs utilized to highlight the differences between unseen and related seen classes. Quantitative evaluations benchmark datasets demonstrate that our EZ- HOI achieves state-of-the-arr performance across various zero-shot settings with only 10.35% 33.95% of the trainable parameters compared to existing methods. com wwwm//hheeseeeeeEE---eee
Code is available at
Introduction
Human-Objec -bijact-- Interaction (HOI) detection localizes human-object pairs and identifies the interactions.
HOI has various practical applications, including robot manipulations, human--cmputee interaction,
26 M. Additionally, it is a building block for related tasks such as action recognition. visual question answering, and image generation [18] (24]
and human activity understanding [38] 41 24
32
[44] However, the limited generalization ability of many existing HOI detectors makes zero-shot HOI detection particularly challenging, it requires models to identify unseen HOI classes.
30 have gained popularity due their extensive knowledge bases and effective ability to process and correlate complex patterns across visual and text data. Consequently, group of methods has been developed align HOI visual features with those of VLMs, leveraging the extensive knowledge from these models [1 33
Vision-Language Models (VLMs) 44
29
4. This alignment ensures that both the HOI model and the VLM can extract similar features represent the same concept (e.g., an action). The degree of feature aligmment can be quantified using cosine similarity, which measures the similarity between feature vectors. However, aligning with VLMs requires
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Aligning HOI model with VLMs
(b) Prompt tuning overfits to seen classes
VEN-
Unseen mAP
HOGGUP
All classes
VLM text
classes
VLM
28
Metta
Our
HOI model
VLM visual
prediction
prediction
Seen mAP
2LD
Full mAP
313
classes
VLM Guidance
LLM
VLM text
VLM visual
Unseen
Seen
Trainable parameters
Epochs
prediction
(c) Our efficient zero-shot HOI detection (EZ-HOI)
(d) Zero-shot HOI detection comparison
Figure 1: Comparison of zero-shot HOI detection paradigms. (a) Methods that align HOI features with fixed VLMs 44 33
4 41I (b) Prompt learning methods to adapt VLMs for downstream tasks [22 57. (c) Our approach, which adapts VLMs HOI tasks without compromising VLM generation capabilities. (d) Unseen, seen, and full mAP indicate the performance for unseen-verb, seen-verb, and full sets on the HICO-DET dataset (6. Our EZ-HOI shows superior performance in these categories, with competitive trainable parameters and training epochs.
training transformer-Dos based models, which is computationally expensive and leads extended training time and significant difficulties.
An alternative approach involves adapting VLMs for HOI tasks, bypassing the demanding alignment process and leveraging VLM capabilities for action understanding in HOI detection. Prompt tuning, which adapts VLMs various downstream tasks with small number of learnable parameters, has shown considerable efficacy [62] [1] 199 21. One notable method, MaPL.e [22), utilizes multi-modal prompt tuning specifically for image classification tasks. However, since adaptation typically involves only seen class labels, the adapted VLMs often overfit to seen classes but struggle to deal with Consequently, this limitation results in suboptimal performance for unseen classes in zero-shot HOI detection.
unseen classes in zero-shot settings 62 7
22
In this paper, introduce novel method, Efficient Zero-Shot HOI detection (EZ-HOI), to enhance VLM adaptation via guided prompt learning with information from foundation models (e.g.. LLM and VLM). We design both learnable text and visual prompts, leveraging fixed LLM and VLM to guide the adaptation process. Specifically, the text prompts are designed to capture detailed HOI class insights from an L.M,, while the visual prompts semantics from a VLM. However, as the training datasets contain images with seen-class labels alone, the trainable prompts are naturally optimized for these seen classes rather than the unseen ones.
tailored to incorporate external visual
To address this limitation, we develop the Unseen Text Prompt Learning (UTPL) to leverage prompts from related seen classes effectively. We begin by measuring the relationship between HOIs using cosine similarity of text embeddings, which helps identify closely related seen classes for each unseen one. After establishing these connections, on those from selected seen classes. To capture the distinctions between unseen and seen classes, incorporate L.M.. This LLM provides what call "disparity information", enhancing the learning of the distinctions and similarities between seen and unseen classes. Additionally, we enhance our approach by introducing intra- and inter-HOI feature fusion techniques following the visual encoder. Our method achieves competitive performance on established benchmarks in various zero-shot settings while requiring significantly fewer trainable parameters.
enhance the learning of unseen prompts based
As illustrated in Fig. I we compare our method with two existing paradigms: (a) aligning HOI features fixed VLM and (b) adapting VLM for HOI tasks. Unlike the existing approaches, method facilitates VLM adaptation to HOI tasks and demonstrates effective generalization to unseen classes. Fig. dd) highlights that our method is competitive in terms of performance, model parameter efficiency, and training duration.
In summary, contributions are follows:

---

We introduce EZ-HOI, novel framework for zero-shot HOI detection that adapts VLM to HOI tasks via guided prompt learning with foundation model information, enhancing the adapted VI.M's generalizability in zero-shot HOI detection.
We propose the UTPL module, which extracts information from related seen classes for the unseen learnable prompts. This module mitigates overfitting to seen classes, common issue task-specific VLM adaptations, improving performance on unseen classes.
Our EZ-HOI method achieves state-of-the- art performance in various zero-shot settings while significantly reducing trainable parameters and mitigating training challenges. It low- ers trainable model parameters by 66% to 78% compared to existing methods, demonstrating enhanced efficiency and effectiveness in zero-shot HOI detection.
2 Related Work
Hluaan-Objed Interaction Detection Human-Object Interaction (HOI) detection involves identi- fying human-object pairs and recognizing their interactions, serving as a fundamental component for various downstream tasks in computer vision [53 G
HOI detection methods are typically categorized into two classes: one-stage and two-stage. One-stage methods simultaneously generate all outputs, including human bounding boxes, object bounding boxes and categories, and interaction classes. Recent advancements in one-stage detectors leverage transformer architectures, delivering promising performance [8] 46
24 An example HOITrans [63), which utilizes the transformer's s encoder and decoder to extract interaction features. The output features are then processed through multi-layer perceptrons to produce all output predictions at once. On the other hand, two-stage approaches divide HOI detection into two tasks: object detection and HOI classifi- cation [10, [14 58). Since separation allows each module to specialize, two-stage HOI detection is more memory efffccient [59]. Recent developments have seen the integration transformer-batt architectures into two-stage designs, which have shown promising results [59 method also falls into the two-stage design category.
55
27 Our
Zero-Shot HOI Detection Prior zero-shot HOI detection efforts primarily address unseen composi- tion settings, where models encounter action and object classes separately but not as combinations. To address this, several methods employ compositional learning strategies aimed tackling the unseen-compe composition problem [14, [6, [5] (7]. However, compositional learning falls short in ad- dressing unseen verb zero-shot HOI detection scenarios. Given the substantial image understanding capabilities of VLMs, they offer promising potential for enhancing HOI detection in zero-shot set- 47 37 36), where labels for unseen HOI classes are absent from the training dataset. 41 However,
tings 2 29
30
Recent studies have explored aligning HOI visual features with VLM 33 this approach requires training transformer-based models, which are computationally expensive. Consequently, aligning HOI models with VLMs is demanding, resulting in extended training times and significant training challenges.
Prompt Learning Prompt learning has gained popularity for adapting VL.Ms to downstream tasks 66 9
Context Optimization (CoOp) 622 refines the prompt input of the text encoder by combining learnable domain-shared prompt tokens with class prompt tokens. MaPL. (2] integrates learnable domain-shared text tokens with visual learnable tokens. This combination leverages the highly connected text and visual encoders in VLMs, facilitating the sharing of cross- domain information and benefiting both text and visual domains. However, fine-tuning VLM relies on training datasets with only seen class labels, which causes adapted VLMs to excel with seen classes but encounter difficulties with unseen ones in zero-shot scenarios. Consequently. this limitation results in suboptimal performance for unseen classes in zero-shot HOI detection.
62
19
23
255
40)
3 Methodology
We start with adapting pre-trained CLIP model 47 zero-shot HOI tasks using an innovative prompt learning approach. We design two sets of learnable prompts: text and visual. The visual prompts are derived from the text prompts by using projection layers. We denote learnable text prompts as hr and learnalle visual prompts as hy:
hr (hoj
(I)
3

---

Text encoder
@: frozen
Describe person swinging a baseball bar
It describes a person using baseball to hit ball..
é: finetune @: concatenate
swinging baseball
Encoder Layer1y
Encoder Layer
A unseen class
LLM
prompt token
seen class prompt token
UTPL
LLM Guidance
MHCA L
break
holid
Learnable prompt tokens ft
UTPL
swing
hose
Visual encoder
W
W
Image input
VLM Visual Pi
Proj
Encoder Layer Vagg
HOI feature fusion
E Epot
MHCA
Encoder Layer
Exst
VLM Guidance
Exoii
Object
HOI prediction
Figure 2: Overview of our EZ-HOI framework. Learnable text prompts capture detailed HOI class information from the LLM. To enhance their generalization ability, we introduce the Unseen Text Prompt Learning (UTPL) module. Meanwhile, visual learnable prompts are guided by a frozen VLM visual encoder. These learnable text and visual prompts then separately input into the text and visual encoder. Finally. HOI predictions are made by calculating the cosine similarity between the text encoder output and the HOI image features. MHCA denotes multi-head vrossaatention.
where fr E Rpad, and hy Rgnd. de is the dimension of the text feature, and d, is the dimension of the visual feature. p represents the number of tokens we design for each learnable prompt.
Let V = {21, 02 ... F.. } as the verb set and 0 = foi 02. ON. } as the object set. Then the HOI set include all feasible verb-object pairs C {hoi, (0,, o,) V: E O). We set C' the total number of HOI classes. Since it is impractical for any datasets to include comprehensive HOI classes, researchers propose zero-shot HOI detection settings to encourage the generalization of HOI detectors to unseen classes Cunseen in inference [33 Denote the seen HOI class set as $ and the unseen HOI class set as = [hoi, hoi, & S, hoi, E C). Please refer to Appendix 7.9 for detailed definitions of the different zero-shot setting unseen set splits.
3.1 LLM and VLM Guidance for Learnable Prompts
Prompt learning techniques for VLMs are characterized by their ability to adapt large, pre-trained models to specific tasks using relatively small amounts of task-speeific data. This often leads to diminished generalization ability of the fine-tuned models for unseen classes [62] g 22]
Given that foundation models, such as LLMs and VLMs, have demonstrated substantial knowledge capacity (29 477. leveraging guidance from these models improve performance unseen HOI classes that are absent in the training data.
Text Prompt Design As shown in Fig. we design input prompts, "Describe a person <acting> <object> and focus on the action" " related to each HOI class, for an LLM.. The output from LLM contains richer semantic information with specific HOI class descriptions. Please refer to Appendix Section 7.2 a detailed explanation with examples.
Then, we use a CLIP text encoder to obtain the text embedding for this HOI class description. We process text embeddings for all HOI classes including both seen and unseen in parallel, so the whole text embeddings are denoted as Fixt E Cede and Fixe = [fxte f ... fexto]. Then, the learnalee text prompts hr can be obtained by:
hr Wup MHCA(Q Wdown hr: K, V Wdown Fixt) + hT.
(2)

---

h
hose
Refined h
Unseen class: Hose dog
Semantic similarity
MHCA
VLM text
cunseen> Hose dog- <senn> Wash a dog
Seen class: Wash a dog
hyyash
Prompt for disparity description
VLM text
Kiss dog
LLM
! concatenate
Describe the phrase: hosing dog and on the difference from washing dog.
"hosing a dog" is different from "washinga dog" asthe person is using a hose to clean the dog, rather than
unseen prompt token seen class gromptokeer
Detailed architecture Unseen Text Prompt Learning (UTPL). the figure, we take the "hose a dog" unseen HOI class in the unseen-vert zero-shot setting as an example. We first utilize the HOI class text embeddings to identify the most connected seen HOI class to "hose a dog". After selecting the seen class, we generate input prompt obtain disparity information from LLM. Finally, the unseen learnable prompt learns from the selected seen class prompt and the disparity information through MHCA.
Figure 3:
where liT = [hTT lur hTTel and ly;; E RR Each class has its specific learnable text prompts after the process of Eq. [2] W up and Wdown indicate the up-projection and down-projection layers. MHCA means the multi-head cross attention [50]. Eq. 2 we only aggregate the useful information from Fier using learnable attention, the information provided by LLM may not all be relevant for our task. Moreover, to keep trainable parameters small, we apply a down- poon-prooection layer W Wdown before MHCA to reduce the feature dimension, and an up-projection layer W up afterward. The same Later, the prompts corresponding to unseen classes are
design strategy is used in Eq. 3 and Eq. 5
further refined, as detailed in Section 3.2
Visual Prompt Design Visual prompt learning in our method is facilitated by pre-trained Visual Language Model (VLM) visual encoder, specifically the CLIP visual encoder [47] The pre-trained CLIP model inherently possesses knowledge of unseen HOI classes, offering richer visual semantics compared to models trained solely on task-specific datasets. The CL.IP visual feature is denoted as fo. for an image I, and we enhance the visual learnable prompts hy by:
hy
Wup MHCA(Q = Wdown hv; K, V = Wdown f(..) + lv.
(3)
Since the frozen VLM visual encoder can extract features for unseen HOIs, hi aggregates information from these visual features, improving performance on unseen HOls.
3.2 Jinseen-Clase Text Prompt Learning (UTPL)
Since the training data has no labeled image for unseen HOI classes, learnable text prompts for seen classes inevitably optimized better than unseen classes. Therefore, we refine the unseen-class text prompts by learning from closely related seen-class prompts. Specifically, we denote one unseen HOI class as u, and the learnable text prompt tokens as hi., and denote one seen HOI class as and the related seen text prompt token /vT, To identify the related seen class, of the HOI class descriptions generated by the LLM, as explained in Text Prompt Design of Section The related class is formulated follows:
text embeddings
3.1
arg max (fext..,) .fxtt.
.ES
Directly refining prompts for unseen classes based solely on selected seen classes may be insufficient due to inherent differences between the seen and unseen classes. Thus, we propose utilizing disparity information, defined as the differences between unseen class and seen class s, as provided by the LLM. The architecture of UTPL is shown in Fig. 3 Please refer to Appendix Section 7.3 for detailed explanation with examples.
Since the description can be too long to be encoded at once, we process each sentence separately by using the CLIP text encoder. Thus, set the text embedding of disparity descriptions for an unseen HOI class u as ediap E Rmmd, where m is the number of sentences in the text description. Then we can compute the refined learnable text prompts hrr, for unseen class u as:
ht. Wup MHCA(Q Waown hr. K, V Wdown concat(fatte
ddisp lrr,, lvr.)) + hi.. (5)
5

---

where concat indicates concatenation. For seen classes, hr, = hrg, Although ground-trui truth labels for the unseen classes not available during training, we propose two strategies to update lr,, in Eq. 5 First, we design a class-relation loss (refer to Eq. [6 in the Appendix) to keep the relationship between seen and unseen classes, measured by cosine similarity between text features. As result, unseen prompts can be refined based on their relation to seen classes. Second, the annotated training data serves as negative samples for unseen HOIs. If the prediction score for unseen class is too high, the model is penalized (Eq. 17 in the Appendix).
3.3 Deep Visual-Text Prompt Learning
We denote a set of learnable text prompts as My= = 1 h2. , and learnable visual prompts as My = [h]. h2 n. hiz E RCpped, h2 E Reed, ; 1,2, ... ,N. N means we intend to introduce new learnable prompts from the first layer to layer N.
Deep Text Prompt Learning The text encoder is composed of K transformer layers (T:) The CLIP text encoder generates text features by tokenizing the words and converting them into word embeddings Wo- = [wb w ... ,wf'' E R Pede, Text features W; will be processed by 7th layer of the text transformer:
W1+1, J TWW, h2 i= 1,2. ,N, o 11 (W, ) N N + , K. [W h
(6)
The final text representation W; can be obtained by:
W, Tex(Pro((Wk))
where W. RInd, and d, is the feature dimension of the final aligned text and visual features in CLIP. Since our learnable text prompt hł specific for each HOI class, the whole text representation can be obtained W = [We, Wt,, W.el
Deep Visual Prompt Learning The visual encoder is also composed of K transformer layers { After each layer V: there is an adapter A; to inject object position and category information [27]. CLIP visual encoder splits image I into D dp * dp fixed-size patches, which are projected obtain visual features ES Ded_ Visual features F2 processed by zth layer of the visual transformer:
[G++1. Ei4 ] = A.Vil,, E. 11))), i= 1,2, -. .N,
[C++1, F441, 1 = A.(Vi(e, E,, 12))), i= - .K
where & is the class token for the 7th layer.
The final visual representation F., E Rlndd can be computed as follows:
Eg VisualProj(Ek).
E, R P-d, can be re-sized to E. Rd,,d Since off-the-shelf detectors provide object detection results, we select bounding boxes with confidence score se > 0 and apply Rol-Align [
obtain features for each detected human and object Frum, Eabj- Then, the intra-HOI feature fusion extracts HOI feature Fnoi IIdd from Faamx Eobj-
Ehoi = ML Eob)).
(10)
where MLP represents multi-layer perception. In any given image I, multiple humans and objects may be present, leading to the detection several potential human-object pairs. Assume there are human-object (H-O) pairs in image I and denote all HOI features as E [E E E Incorporating surrounding HOI features enriches context information for each HOI visual feature by capturing additional human-object relational details and interactions, thereby enhancing HOI features.
We name this process inter-HOI feature fusion. Thus, final visual representation & is obtained by:
Chi Wap MHSA(W down S Ex
(11)
where MHSA means multi-head self-attention (50) and Ehoi = hnii Eioin" calculate the prediction for each H-O pair hoi,.
E. Then, can
Exp(Ei (T,
Phoi(c/hoi,,
1=1 exp( Ei. (W,)7)) c= 1,2, C.
(12)
Please refer to Appendix Section
7.5 for more details for training and inference.

---

Experiments
Table 1:
Unseen-verb (UV) comparison HICO-DET with state-of-the-ar methods. indicates the
model size is estimated according to papers "TP" denotes the trainable parameters.
mAP
Method
Setting
Backbone
TP
Full
Unseen
Seen
GEN-VLKT (CVPR'22) 33
UV
ViSs
42.05M
28.74
20.96
30.23
EoID (AAAI'23) 6
UV
Resnet50
41.45M
29.61
22.71
30.73
HOICLIP (3VPR'23)
UV
Resnet50+ ViT-B
66.18M
31.09
24.30
32.19
CLIP4HOI (Ne
UV
Resnet50+ViT-2 ViT-B
56.7M*
30.42
26.02
31.14
Ours
UV
ViTT
6.85M
32.32
25.10
33.49
UniHOI (NeurlPS'23)
UV
Resnet50+ ViT-L
52.3M
34.68
26.05
36.78
Ours
Resnet50+ViT-LL ViT-L
14.07M
36.84
28.82
38.15
Table 2: Rare-first unseen-compost comppsicomooiiionnn (RF-UC) and Nonrare-firy unseen composition (NF-UC) comparison on HICO-DET with state-of-the-art methods.
mAP
mAP
Method
Backbone
Setting
Setting
Unseen
Seen
Full
Unseen
Seen
GEN-VLKT 33
Resnet50+ VTT-B
RF
30.56
21.36
32.91
23.71
25.05
23.38
BolD
Resnct50
RF
29.52
22.04
31.39
26.69
26.77
26.66
HOICLIP
Resnet50+ViT-B
RF
32.99
25.53
28.47
NF
27.75
26.39
28.10
ADA-CM
Resnet50+ViT-B
RF
33.01
27.63
34.35
NF
31.39
32.41
31.13
CLIP4HO1 41
Resnet50+Vii-B
RF
34,08
28.47
35.48
NF
28.90
31.44
28.26
Ours
VisT
RF
33.13
29.02
34.15
NF
31.17
33.66
30.55
UniHOI
VrS-
RF
32.27
28.68
33.16
NF
31.79
28.45
32.63
Ours
ViTLL
RF
36.73
34.24
37.35
NF
34.84
36.33
34.47
4.1
Zero-Shot HOI Setting Definition
Our method follows the existing zero-shot HOI setting, which involves predicting unseen HOI classes and typically includes using unseen class names during training [14] 15] 16 [1). In particular, VCL, FCL and ATL [4 16 [3] "compose novel HOI samples" during training with the unseen (novel) HOI class names. EolD [51] distills CLIP "with predefined HOI prompts" including both
seen and unseen class names. HOICLIP AF introduces "verb class representation" during training,
including both seen and unseen classes.
4.2 Implementation Details
We evaluate our method on HICO-DET by following the established protocol of zero-shot two-stage
HOI detection methods [14
3
27
Our object detector utilizes pre-trained DETR model [5]
with a ResNet50 backbone 12 As for our learnable prompts design, we set 2, N = 9. The
LLaVA-v1.5-7b model 37 used provide
description, as explained in Section 3.1 and 3.2
For all experiments, our batch size is set as 16 on Nvidia A5000 GPUs. We use Adam W [39 as the optimizer and the initial learning is le-3. For more implementation details, please refer to Appendix Section 7.1
4.3 Comparison with State-of-the-Art Zero-Shot HOI Methods
Unseen-Veet Verbb Setting In Table D we compare our method to existing zero-shot HOI detection approaches under the unseen-vedt verb zero-shot setting. The results demonstrate that our method not only achieves competitive performance but also requires only 10.35% to 33.95% of the trainable parameters compared to existing zero-shot HOI detection methods thanks to our novel prompt learning design. While our method shows a minor drop performance compared CLIP4HOI under the unseen verb setting, it is important to consider the significant reduction in trainable parameters that our method achieves--e -87.9% fewer than CLIP4HOL. contrast, the existing HOI methods B33 44 41 4 that align with VLMs unanimously require significantly more trainable
7

---

Table 3: Unseen-object (UO) comparison on HICO-DET with state-of-the-ee methods. indicates the model size is estimated according to papers 41 4 i denotes methods that use DETR object detection model pretrained on HICO-DET. Results without t indicate the use of a DETR object detection model pretrained on MS-COCO. "TP" denotes the trainable parameters.
mAP
Method
Selting
Backbone
TP
Full
U Unseen
Seen
FCL (CVPR'21)
UO
Resnet50
19.87
15.54
20.74
ATL (1)PR'21)
15
UO
Resnet50
20.47
15.11
21.54
GEN-VLKT (CVPR
UO
Resnet50
42.05M
25.63
10.51
28,92
HOICLIP (CVPR'23)
UO
Resnet50+ ViT-B
66. 18M
28.53
16.20
30.99
CLIP4HOI (3ea)
UO
Resmet50+ViT-2
56.7M
32.58
31.79
32.73
Ours
UO
Resnet50+ VTT-B
6.85M
27.90
31.63
27.16
Ours
UO
Resne150+ ViT-B
6.85M
32.27
33.28
32.06
UniHOI (NeurIPS'23)
UO
Resnet50+ VIT-L
52.3M
31.56
19.72
34.76
Ours
UO
Resnet50+ VYT-I.
14.07M
31.42
33.08
31.09
Ours
UO
Resnet50+ VIT-L
14.07M
36.38
38.17
36.02
model parameters (e.g., 42.05M ~ 18. with ViT-B visual encoder), whereas method operates efficiently with only 6.85M trainable parameters.
UniHOI 4 is the state-of-te srate-of-the-annd fi-thh--rt art method in the unseen verb (UV) setting. Compared to UniHOI which aligns HOI features to BLIP (29] text embeddings, our method achieves improved performance, especially in the unseen category with 2.77 increase in mAP. At the same time, our model requires only 26.9% of the trainable parameters compared UniHOI. This demonstrates the effectiveness and efficiency of our proposed prompt learning framework in adapting the VLM for zero-shot HOI tasks.
Dassen-Compossiiee Setting In Table we provide the zero-shot performance comparison in rare-first unseen-comp compossion (RF-UC) and nonrare-first unseen-compositonn (NF-UC) settings. With the VIT-B visual encoder, our method establishes new standard for unseen-class performance across both RF and NF settings, outperforming the previous state-of-the-art method, CLIP4HOI [41], while requiring only 12.0 22.8% of its trainable parameters. Compared UniHOI with VIT-L visual encoder [4), our method surpasses it by 5.56 mAP in unseen performance of the RF-UC setting and by 7.88 mAP the NF-UC setting with only 26.9% of the trainable parameters of UniHO1
Unseen-Object Setting We provide the unseen-object setting comparison in Table Following previous methods 44 33,, we employ DETR object detector pre-trained on MS-COCO 34] for
object detection. To keep consistent with CLIP4HOI [4] and FCL [16], we also provide the results by using the DETR model pre-trained is the ofate-of-the-ar method in unseen performance. With the same object detector pre-trained on HICO-DET, our method shows improved performance for unseen class prediction, outperforming CLIP4HOI by 1.49 mAP, with only 12.08% of its trainable model parameters. UniHOI 4 is the state-of-the the-th-----ttttt method in seen performance. With the same object detector as UniHOI, our method outperforms UniHOI in unseen category by 13.36 mAP.
HICO-DET
27 (marked with Table 3). CLIP4HOI [l
4.4 A Ablaioon Studies
We conduct the ablation study for our text and visual prompt learning design in Table The first row shows the performance of our baseline, MaPLe [22]. Our intra-HOI fusion module improves the seen HOI performance by 7.41 mAP, shown in the second row. In the third row, the inclusion of the visual adapter (27 enhances performance for seen classes while negatively affecting unseen HOI performance.
Comparing the third row and fourth row, LLM guidance, shown in Fig. 2 notably enhances unseen- class learning, resulting in 1.52 mAP improvement. LLM guidance, by integrating learnable text prompts with detailed HOI class descriptions, improves the model's understanding of unseen classes, providing richer information compared simple class names. Additionally, the inclusion of the UTPL module significantly boosts unseen class performance, with 2.42 mAP improvement.
Later, the Inter-HOI fusion benefits both the seen and unseen learning by providing more context

---

Table 4: Ablation study of our method in zero-shot unseen verb setting on HICO-DET.
Intra-Fi Intra-HOI fusion
visual adapter [27
Inter-HOl fusion
MAP
LLM Guide
UTPL
VLM Guide
Full
Unseen
Seen
X
X
26.26
17.19
27.73
X
X
X
X
33.52
23.54
35.14
X
X
X
X
35.40
22.91
37.44
V
X
X
X
35.62
24.43
37.44
V
V
X
X
36.23
26.85
37.76
V
V
X
36.70
27.49
38.10
4
V
V
V
36.84
28.82
38.15
Table 5: Prompt learning evaluation in zero-shot unseen verb setting on HICO-DET.
mAP
Method
Full
Unseen
Seen
CLIP [7
13.18
13.96
13.05
MaPLe 22
26.26
17.19
27.73
MaPI.e 22
visual adapter 227
32.70
22.89
34.29
Ours
36.84
28.82
38.15
information for visual HOI features. Finally, the VLM guidance effectiveness is evaluated, which enhances the unseen performance by 1.33 mAP.
We also provide the experimental comparison with CLIP 47 and MaPLe [2 in Table Since they are designed for general image classification, directly applying it to HOI tasks for comparison is unfair they may not focus on the human-object region features. Therefore, we crop the union region from the original visual features for each human-object pair to obtain the HOI feature. HOI prediction is obtained by using cosine similarity between HOI and text features for cach class.
As shown in Table although with learnable prompts, MaPLe 22 outperforms CLIP on seen classes, its performance on unseen classes is far behind that of the seen classes, showing reduced generalization capability. Compared
our method, MaPLe lags significantly behind on unseen performance, with an 11.63 decrease in mAP. Additionally, the third row introduces another baseline that incorporates MaPLe with visual adapter (i.e., A,, (27]. which is also used in our framework. ensuring fair comparison. The performance improvement from the third row to the fourth row, with gains of 5.93 mAP for unseen classes and 3.86 mAP for seen classes, clearly demonstrates effectiveness of our method in zero-shot HOI detection.
4.5
Qualitative Results
Fig. shows the qualitative results of MaPLe 22] and our method in the unseen-vert verb zero-shot setting. In particular, MaPLe struggles to detect unseen classes, either missing unseen HOI classes predicting the wrong unseen HOI classes. For example, if an image only contains unseen classes, MaPLe tends predict wrong seen classes and miss the correct ones. As shown in the bottom right of Fig. this image contains unseen class only ("wear tie"), MaPLe predicts related wrong seen classes such "pulling tie" and "adjusting tie". but fails predict the ground-truth unseen HOI ("wear tie"). This shows the limited generalization ability of MaPLe to unseen classes. In contrast to MaPLe, our method can predict both seen and unseen classes more accurately.
4.6
Fully Supervised Setting for HOI Detection
We conducted the uully-supervised experiments on both the HICO-DET and V-COCO benchmarks 6 Our method achieves competitive 38.61 mAP on the HICO-DET dataset, with a smaller performance drop between rare and non-rare classes (1.19 mAP) compared to AGER 49 (4.18 mAP), current state-of-the- one-stage method. Our method also outperforms the performiff zero-shot HOI method, CLIP4HOI [1), by 3.28 mAP HICO-DET. On the V-COCO benchmark, our method achieves state-of-the-arr performance with 66.2 AP in Scenario Detailed discussions and comparisons are provided in the Appendix 7.6
34

---

MaPLe
MaPLe
MaPLe
X X
Ours
Qurs
Weshi
YPA SARIE Phoo
inoppct
Repail
Direct
MaPLe
MaPLe
Ours
Ours
Ours
Food
pet
Milk
Feed Watch
Pet
Pan
Adjust
Figure 4: Qualitative comparison with MaPLe 22 for unseen- weerb zero-shot HOI detection. The
orange bar represents the unseen class prediction and the blue bar means the seen class prediction.
4.7 Discussion of Descriptors from LLMs
Descriptors, developed to leverage the LLMs for downstream tasks 42 20 involve attribute de- composition to benefit category recognition. We adopt the attribute decomposition concept from DVDet [20] and tailor it our EZ-HOI framework. Following DVDet, generate action descriptors for each class and integrate them into HOI class text features, thus enhancing the detail and distinc- tiveness of class representations. Descriptors with low cosine similarity to the class text features discarded to avoid noise. With our straightforward adoption of action descriptors, the result shows 0.31 mAP improvement compared setting. This indicates that LLM L---eeeertted descriptions, such as action descriptors, hold potential to enhance HOI detection and worth further exploration.
EZ-HOI, achieving 32.63 mAP under the unseen-verb
4.8 Limitations
Our method has some limitations. First, while our method significantly reduces the number of trainable parameters compared to existing approaches, it still requires fine-tuning on HOI-specific datasets such as HICO-DET. A training-free design, which could be used directly HOI detection without the need for fine-tuning would be more desirable. Second, zero-shot HOI detection requires the pre-definition of all HOI classes, both seen and unseen, following the current zero-shot HOI detection setting. This requirement restricts the generalizability to truly unseen classes. Consequently, our future work will aim to develop a robust open open-category HOI detector that operates effectively without the need for predefined classes.
5 Conclusion
In this paper, introduced the Efficient Zero-shot HOI Detection (EZ-HOI) approach, utilizing prompt learning to adapt visual-language models for zero-shot HOI detection. This method not only maintains competitive performance for unseen classes but also innovatively integrates learnable visual and text prompts. These prompts leverage foundation model information, thereby enriching prompt knowledge and enhancing the adaptability of VLMs. A significant challenge we addressed is the compromised performance on unseen classes, resulting from the training dataset containing only seen-class labels. To counter this, we developed text prompt learning strategy that utilizes information from related seen classes to support the detection of unseen classes. We also employed an LLM to provide the nuanced differences between related seen and unseen classes, improving our method for unseen class prompt learning. Our method has demonstrated state-of-the-ar performance across various zero-shot HOI detection settings while requiring only a third of the trainable parameters compared to existing methods.
10

---

6 Acknowledgements
This research/project supported by the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative. Any opinions, findings and conclusions or recommen- dations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
References
[1] Yihao Ai, Yifei Qi, Bo Wang, Yu Chen, Xinchao Wang, and Robby T. Tan. Domain-adaptive 2d human pose estimation via dual teachers extremely low-light conditions. ECCV, 2024.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716-23736, 2022.
[3] Ankan Bansal, Sai Saketh Rambhatla, Abhinav Shrivastava, and Rama Chellappa. Detecting human-object interactions via functional generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10460-10469, 2020.
[4] Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models. Advances in Neural Information Processing Systems, 36, 2024.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 21 213-229. Springer, 2020.
[6] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human- object interactions. In 2018 ieee winter conference applications of computer vision (wacv), pages 381-389. IEEE, 2018.
[7] Guangyi Chen, Weiran Yao, Xiangehen Song, Xinyue Li, Yongming Rao, and Kun Zhang Plot: Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.012533 2022.
[8] Junwen Chen and Keiji Yanai. Qahoi: query-based anchors for human-object interaction detection. arXiv preprint arXiv:21 2021.
[9] Euirang Cho, Jooyeon Kim, and Hyunwoo J Kim. Distribution-awarr prompt tuning for vision- language models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22004-22013, 2023.
[10] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human-object interactions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8359-8367, 2018.
[11] Albert Gordo and Diane Larlus. Beyond instance- image retrieval: Leveraging captions to learn global visual representation for semantic retrieval. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6589-6598, 2017.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[13] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[14] Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for human- object interaction detection. Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pages 584-600. Springer, 2020.

---

[15] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Affordance transfer learning for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 194-504, 2021.
[16] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Detecting human-object interaction via fabricated compositional learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14646-14655, 2021.
[17] Zhi Hou, Baosheng Yu, and Dacheng Tao. Discovering human-object interaction concepts via self-compositional learning. In European Conference on Computer Vision, pages 461-478. Springer, 2022.
[18] ASM Iftekhar, Hao Chen, Kaustav Kundu, Xinyu Li, Joseph Tighe, and Davide Modolo. What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5353-5363, 2022.
[19] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[20] Sheng Jin, Xueying Jiang, Jiaxing Huang, Lewei Lu, and Shijian Lu. Llms meet vlms: Boost open vocabulary object detection with fine- tne-ggrined descriptors. arXiv preprint arXiv:2402.04630, 2024.
[21] Baoshuo Kan, Teng Wang, Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng. Knowledge-aware prompt tuning for generalizable vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15670-15680, 2023.
[22] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[23] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedingy of the IEEE/CVE International Conference on Computer Vision, pages 151 15100-15200, 2023.
[24] Sanghyun Kim. Deunsol Jung, and Minsu Cho. Relational context learning for human-object interaction detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 1925-2934, 2023.
[25] Dongjun Lee, Scokwon Song. Jihee Suh, Joonmycong Choi, Sanghyeok Lee, and Hyunwoo J Kim. Read-only prompt optimization for langualanguage few-shot learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1401-1411, 2023.
[26] Qinqian Lei, Bo Wang, and Robby T Tan. Few-shot learning from augmented label-uncertain queries in bongard-hoi. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 38, pages 2974-2982 2024.
[27] Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng, and Yang Liu. Efficient adaptive human-object interaction detection with concept-guided memory. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 6480-6490, 2023.
[28] Ting Lei, Shaofeng Yin, and Yang Liu. Exploring the potential of large foundation models for open-vocabulary hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16657-16667, 2024.
[29] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-languag understanding and generation. In International Conference Machine Learning, pages 12888-12900. PMLR. 2022.
12

---

[30] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023.
[31] Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural-logic human-object interaction detection. Advances in Neural Information Processing Systems, 36, 2024.
[32] Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing numan-obbect interaction. Advances Neural Information Processing Systems, 33:5011-5022 2020.
[33] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vikt: Simplity association and enhance interaction understanding for hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20123- 20132, June 2022.
[34] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECC ECCC 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt 13, pages 740-755. Springer, 2014.
[35] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 1980-2988, 2017.
[36] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024.
[38] Xinpeng Liu, Yong-Lu Li, Xiaoqian Wu, Yu-Wing Tai, Cewu Lu, and Chi-Keung Tang. Interac- tiveness field in human-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10113-20122 2022.
[39] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051011 2017.
[40] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. Advances in Neural Information Processing Systems, 36, 2024.
[41] Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li. Clip4hoi: Towards adapting clip for practical zero-shot hoi detection. Advances in Neural Information Processing Systems, 36, 2024.
[42] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2110071183, 2022.
[43] Gyeongsik Moon, Heeseung Kwon, Kyoung Mu Lee, and Minsu Cho. Integralaction: Pose- driven feature integration for robust human action recognition videos. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3339-3348, 2021.
[44] Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 23507-23517 2023.
[45] Jeeseung Park, lin-Woo Park, and long-Seok Lee. Viplo: Vision transformer based pose- conditioned self-loop graph for human-object interaction detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 17152-17162, 2023.
13

---

[46] Xian Qu, Changxing Ding, Xingao Li, Xubin Zhong, and Dacheng Tao. Distillation using. oracle queries for transformer- human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19558-19567, 2022.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-87888 8763. PMLR, 2021.
[48] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpie: Query-based pairwise human- object interaction detection with image-wide contextual information. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10410-10415 2021.
[49] Danyang Tu, Wei Sun, Guangtao Zhai, and Wei Shen. Agglomerative transformer for human- object interaction detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages
October 2023.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[51] Mingrui Wu, Jiaxin Gu, Yunhang Shen, Mingbao Lin, Chao Chen, and Xiaoshuai Sun. End-to- end zero-shot hoi detection via vision and language knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2839-2846, 2023.
[52] Chi Xie, Fangao Zeng, Yue Hu, Shuang Liang, and Yichen Wei. Category query learning for human-object interaction classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15275-15284, 2023.
[53] Bangpeng Yao and Li Fei-Fei. Recognizing human-object interactions in still images by modeling the mutual context objects and human poses. IEEE transactions on pattern analysis and machine intelligence, 34(9):1691-1703, 2012.
[54] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In Proceedings of the European conference on computer vision (ECCV), pages 684-699, 2018.
[55] Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng. Ziyuan Huang, Dong Ni, and Minggian Tang. Rlip: Relational language-image pre-training for human-object interaction detection. Advances Neural Information Processing Systems, 35:37416-37431, 2022.
[56] Hangjie Yuan, Shiwei Zhang, Xiang Wang. Samuel Albanic, Yining Pan, Tao Feng, Jianwen Jiang, Dong Ni, Yingya Zhang, and Deli Zhao. Rlipv2: Fast scaling of relational language- image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21649-21661, 2023.
[57] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225, 2022.
[58] Frederic Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object interactions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13319-13327,20 2021.
[59] Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Efficient two-stage detection of human- object interactions with a novel unary-pairwise transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20104-201 12, 2022.
[60] Frederic Z Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould. Exploring predicate visual context in detecting of human-object interactions. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10411-10421, 2023. 2023.
14

---

[61] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816-16825, 2022.
[62] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9))2337-2348, 2022.
[63] Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, et al. End-to-end human object interaction detection with hoi transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11825-11834, 2021.
15

---

7 Appendix
7.1 Implementation Details
Here we provide detailed implementation information. For the pre-trained CLIP model with the ViT-B visual encoder, the visual feature dimension d, 768, while the text feature dimension d, = 512 and the final feature dimension for aligned visual and text features da = 512. For the CLIP model with the ViT-L visual encoder, d,- 1024, de 768, de 768. We use off-the-shelf object detector and add a threshold / to filter out some low-confident predictions and we set = 0.2. Since UniHOI [4] and CLIP4HOI 44 have not released their code, we estimate the trainable model parameters by modifying the HOICLIP [44] code according to the description in UniHOI and CLIP4HOI, which means that if some details not mentioned in the UniHOI and CLIP4HOI papers, we the HOICLIP design by default. In addition, we initialize the weights of all Wup to 0, which stabilizes training by gradually fine-tuning the attention outputs in Eq. Eq.
Eq. and Eq.
Dataset and Evaluation Metries
We conducted extensive experiments using the shot HOI detection. HICO-DET comprises a total of 47,776 images, divided into 38,118 training images and 9,658 test images. This dataset features 600 Human-Objec Interaction (HOI) classes, which are combinations derived from 117 action categories and 80 object categories. Our model's performance was evaluated in four distinct zero-shot HOI detection settings, categorized by the criterion for selecting the unseen HOI classes: rare- rare-first unseen composition (RFUC), nonrare-fire unseen composition (NFUC), unseen object (UO), and unseen verb (UV). These settings align with methodologies from previous research 33
HICO-DET (6] dataset for zero-
Additionally, we also evaluate our model in the fully supervised setting on both the HICO-DET dataset and the V-COCO [34] dataset. V-COCO is subset of COCO, with 10,396 images-split into 5,400 train-val images and 4.946 test images, and it encompasses 24 action classes and 80 object classes. Following standard evaluation protocols, we assessed our model using mean average precision (mAP) on the HICO-DET benchmark, while Average Precision (AP) in both Scenario I and Scenario is used on the V-COCO benchmark [63 MA A prediction was deemed true positive if the HOI classification was accurate and the Intersection over Union (IoU) between the predicted human and object bounding boxes and the ground-truth bounding boxes exceeded 0.5 [33 27].
7.2 HOI Class Description
We utilize the LLM to provide detailed HOI class descriptions to guide the text prompt learning and we provide example for detailed illustration. We take the HOI class "Swing a baseball bat" as an example, which is one unseen HOI class in the unseen-verb zero-shot setting on the HICO-DET benchmark. The generated HOI class description is
"Siinging baseball bat" describes a person using baseball bat to hit a ball. This action typically involves the person holding the bot with both hands, standing in stance with their feet shoulder-width apart, and using their body rotation to contact the ball.
7.3 Disparity Information for UTPL Module
The disparity information mentioned in Section 3.2 aims to explore the difference between the unseen class and the related seen class. Here demonstrate it with specific example. We take the unseen class hose a dog" as an example and its selected seen class is "wash a dog".The input text prompt for LLM to acquire the disparity description between the two classes is designed as Describe the definition of the phrase: hosing a dog and please focus on the attributes different from another phrase: obtain the detailed disparity description from LLM
washing dog. Then
The phrase "a person hosing a dog" refers to the action of washing a dog using a hose. This action is different from "a person washing dog" as the person is using a hose to clean the dog, rather than simply using their hands to wash the dog. The use of a hose adds an additional element of water pressure and flow, which can make the cleaning process more efficient and effective.
16

---

HOI prediction
@ frozen
finetune
W
text encoder
carry skis
ride car
Wec
detector
HOI text class
Enot, Enoiz
Exola
visual encoder
Concat.
MHSA
Intra-HOL Fusion
Inter-HOI Fusion
Figure 5: Detailed architecture for HOI feature fusion design. Intra-HOI feature fusion aims to extract HOI features from possible human region and object region features. Inter-HOl feature fusion aims to enhance the HOI features by incorporating the surrounding HOI feature context. "MHSA" refers to multi-head self-attention.
7.4 HOI Feature Fusion
As shown in Fig. we also provide the detailed architecture of the intra- and inter-HOI feature fusion design. The intra-HOI feature fusion integrates human and object features for each human-object pair. To enrich context information, all HOI features are processed together using multi-head self-attention (MHSA). allowing each feature become aware of its surroundings. This enhancement improves the context and overall performance of each HOI feature.
7.5 Training and Inference
Since W; RC+d, related the number of total HOI classes, the number is too large, the learnable prompts can be computationally expensive. Thus, we only select part of HOI classes and do action classification. Later, by combining action prediction with object detection results, we can obtain HOI prediction finally. Specifically, we select two HOI classes for each action with the following equation:
hoi; hoij arg min ftat, (Ftat,)* 043.SC
(13)
which means we select two HOI classes containing the same action with the most different semantic meanings. Since some actions can have multiple interpretations in different contexts (e.g., "hold apple" vs. "hold sheep"). randomly choosing two HOI classes with the same action does not cover the comprehensive meanings of the action. By selecting the most different HOI classes, aim to capture a richer range of information for the action. Then, We calculate the HOI prediction score for ęth H-O pair by:
exp(E) (W, Ei exp( (i (W,,)))
Phoi (cohoi.)
1,2, C
(14)
The action scores can be obtained by:
8a = Phoi * latigns
(15)
where lalign Ced, means the action label for each selected HOI class. We adopt focal loss 335 foccl to train the action prediction. We also design a class-relation loss shown in the following equation:
Irelation DDrllsim(W,, (i)
(16)
sim X, Y) TT.
keep the relation between adapted text class embeddings W. close to the original text embeddings of the HOI class description fix- D DxLl-11-1 denotes the KL divergence. Therefore, the training loss
Ltrain is computed by:
Lurain Loccl + alvelation:
(17)
17

---

and a is a hyperparameter. In the inference stage, we can obtain the HOI score for each human objjcct pair by:
$h (sh so)" (sa)):
(18)
where Sh. So are the object detection confidence scores for humans and objects. 7 is a hyperparameter.
In Eq. 17 a 150, and in Eq. 18 T during training and T = 2.8 during inference [58] 59].
7.6 Quantitative Results
Fully Supervised HOI Detection Setting We conduct experiments under the fully supervised HOI detection setting on both the HICO-DET [6] and V-COCO [34] dataset, as shown Table 6 In the UTPL module, utilize common class prompts inputs. This approach enables common HOIs to learn from rare ones, fostering a hetter differentiation between them. This strategy helps to alleviate overfitting common HOI classes, thereby improving performance for both seen and unseen classes. As shown in Table 6 the performance drop between rare and non-rare classes ( 1.19 mAP) is much smaller than AGER (4.18 mAP), the best performance among the one-stage method. Additionally, our method achieves the best performance among the two-stage methods on the HICO-DET [6 dataset, outperforming CLIP4HOI 4T by 3.28 mAP.
As for the V-COCO benchmark, we achieve competitive performance among the two-stage methods with 66.2 AP performance under Scenario 2 evaluation. However, the smaller number of verb classes in V-COCO (24 classes), which have weaker connections (i.e., jumping vs. skateboarding skateboard), compared HICO-DET (117 verb categories), which exhibits stronger connections (i.e., jumping vs. flipping skateboard), limits the potential of our method. Our UTPL. design requires the learnable prompts to extract information from prompts of other classes, which also aids in better differentiation between similar classes. Due to the weakly connected classes in V-COCO, our UTPL design cannot work effectively. Despite this, achieve performance comparable to CLIP4HOI 41] further demonstrating the effectiveness of our method.
Table 6: State-of-the-an Comparison on HICO-DET and V-COCO in the fully- supervised setting. Bold highlights the best rest-performing method within each of the two groups: one-stage and two-stage methods.
HICO-DET
V-COCO
Method
Full
Rare
Nonrare
APS; role
AP role
One-stage Methods
GEN-VLKT (CVPR'22)
33.75
29.25
35.10
62.4
64.5
HOICLIP (CVPR'23) 44
34.69
31.12
35.74
63.5
64.8
RLIPv2 (ICCV'23) (56
35.38
29.61
37.10
65.9
68.0
AGER (10CV'23) 50
36.75
33.53
37.71
65.7
69.7
LogicHOI (NearIPS
35.47
32.03
36.22
64.4
65.6
Two-stage Methods
UPT (CVPR'22) 58
32.62
28.62
33.81
59.0
64.5
ADA-CM (ICCV'23)
27
38.40
37.52
38.66
58.6
64.0
CLIP4HOI (NeurIPS
41
35.33
33.95
35.75
66.3
Ours
38.61
37.70
38.89
60.5
66.2
7.7 Ablation Studies
Table shows the ablation study for the hyperparameter N,, where N means we introduce learnable prompts from the first layer until layer N in both visual and text encoders. We use N=9 in our main paper because it shows the best unseen performance.
Table & shows the ablation study for the different positions of learnable prompts. Position i means we insert learnable prompis from layer layer j in both the text encoder and visual encoder. Here we always insert learnable prompts into 9 layers and only change the position. We find that the position of learnable prompts does not affect the outcome too much. Thus, we follow 22 and fine-tune layers 1-9, which show slightly better unseen performance.
18

---

Table 7: Ablation study for hyperparameter N under the unseen-verb setting.
mAP
Full
Unseen
Seen
32.60
24.10
33.99
32.32
25.10
33.49
12
32.76
23.98
34.19
Table 8: Ablation study for hyperparameter N under the unseen-verb setting.
mAP
Position
Full
Unseen
Seen
1-9
32.32
25.10
33.49
3-11
32.24
24.83
33.45
4-12
32.40
24.82
33.64
7.8 Qualitative Results
As shown in Fig. 6 we provide more qualitative results in three zero-shot HOI settings: unseen- verb, fare-first unseen-com onseen-composition, and nonrare-fing unseen-compont composiconooiioon settings. Compared to MaPLe [22], our method obtains better generalization capability to unseen HOI classes owing to our efficient prompt learning design.
In Fig. 7 we show more qualitative comparisons to illustrate the effectiveness of the LLM guidance and UTPL design. The baseline here refers the method in the main paper's third row of Table 4 LLM guidance design utilizes the general description from LLM for each HOI class and the UTPL module integrates the distinctive description from LLM. As shown in Fig. 7 the LLM guidance provides detailed class information, improving the performance over the baseline. Distinctive descriptions from UTPL design help distinguish unseen classes from related seen HOIs, enhancing unseen performance and challenging case predictions.
7.9 Discussion for Zero-Shot HOI Detection Definition
Our method follows the standard zero-shot HOI setting, where unseen class names are used during training [14 15
51 44 as discussed in Section 4.1 Specifically, before training, the whole verb set V = V seen V Vunseen }o and the whole object set 1 = [aeen O unnseen are all pre-defined. The four settings of zero-shot HOI detection include : 1) Rare-First Unseen Composition (RF-UC), where for all hoi; = (Ve, o;) E U, we have E V Veeen O, E Oreen and hoi; appears less than 10 times in the training set, belonging to rare HOI classes. Nonrare-Fiis First Unseen Composition, where for all hoi; = (v,, o;) E U, we have E V seen, O₂ E Oseen and hoi, appears more than 10 times in the training set, belonging to nonrare HOI classes. Unseen Verb (UV), where hoi, (1) 0:) U, we have E V unsseen Og E O 4) unseen Object (UO), where hoi; = (v;, 0;) E U, we have V/ E Vssen, Og E Opnspen-
T6
Beyond the zero-shot HOI setting, there are HOI unknown concept discovery 17 and open- vocabulary HOI detection (28). where unseen class names cannot be used in training. The open- vocabulary setting differs from HOI unknown concept discovery, with a much wider range of unseen HOI classes during testing.
7.10 Discussion of Broader Impacts
Our approach to zero-shot HOI detection reduces reliance on extensive annotated datasets, enhanc- ing accessibility for organizations with limited resources and promoting inclusivity in technology adoption. This technology could notably improve assistive devices, offering more intuitive aids for individuals with disabilities by enabling better understanding of new environments.
However, these advancements also pose risks. The capability to interpret human-object interactions could be used for surveillance, potentially infringing on privacy. Additionally, reliance on existing disual-languae models may maintain embedded biases, leading to discriminatory outcomes. To
19

---

MaPLe
MaPLe
MaPLe
Ours
Ours
Ours
Feed Hold
Hug
Inspect
Hold
Hose
Wath
UV setting
(b) UV setting
(c) UV setting
MaPLe
MaPLe
MaPLe
Ours
Ours
Inopect
Wash
Hold Stradole
Doard
Drive
Park
(d) UV setting
(e) UV setting
(l) UV setting
MaPLe
MaPLe
Ours
Ours
Ours
Adjust Wear T Mide
Wash
Park
No
RF-UC setting
(h) RF-UC setting
ii RF-UC setting
MaPLe
MaPLe
MaPLe
Ours
Ours
Rice
Jumo
Push
Sit
walk
Hold
Carry rode
i NF-UC setting
(k) NF-UC setting
(I) NF-UC setting
Figure 6: Qualitative comparison of zero-shot HOI detection between our method and MaPLe 12 We use orange color to represent unseen HOI classes and use blue color for seen ones. For images containing multiple HOI results, we only present one prediction for clearer demonstration and comparison.
address these concerns, we recommend developing rigorous ethical guidelines and governance frameworks to regulate the deployment of HOI detection technologies, alongside efforts identify and mitigate biases in foundational models.
20

---

(a): Baseline;
(b): Baseline LLM guidance; (cl: Baseline LLM guidance& UTPL
Unseen class: swing baseball bat
Unseen class:
hose car
Confidence score Increasing rate
Confidence score Increasing rate
13% 53% 100%
22% 36% 100%
(a)
(b) (c)
(a)
(b) (c)
Unseen class: ride elephant
Unseen class: inspect fire hydrant Confidence score Increasing rate
Confidence score Increasing rate
29%
42% 100%
22% 82% 100%
(a)
(b)
(c)
(a)
(c)
Unseen class: swing baseball bat Confidence score Increasing rate
Unseen class: Wear tie
Confidence score Increasing rate
8% 50%
100%
16%
34% 100%
(a)
(b) (c)
(a)
(b)
(c)
Confusing class: flip skateboard Confidence score Increasing rate
Confusing class: board airplane Confidence score Increasing rate
37%
59%
100%
12% 48% 100%
(a)
(b)
(c)
(a)
(b)
(c)
Confusing class: grind skateboard
Confusing class: Race horse
Confidence score Increasing rate
Confidence score Increasing rate
16% 52% 100%
34% 72% 100%
(a)
(b) (c)
(b) (c)
Figure 7: Qualitative results to show the effectiveness of the LLM guidance and UTPL design. We conduct the comparison under the unseen-verb zero-shot HOI setting.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We emphasize our contributions in the Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We included the Limitation section in the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
We don't 't include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide comprehensive implementation details both in main paper and in appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: We included the website link for the code repository the abstract and our evaluation is based on public datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nips.cc/public/giue guwde//ddee/ee/ Cot//ooodeeeiiseeeeeeee wwwpibmmsssioooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We included them in the implementation details in both main paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[No)
Justification: We follow the default evaluations in the HOI detection field, which doesn't t require error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide them in implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics stttps://eeurppee http://s...pppbccccccccccccdaaaa
Answer: [Yes]
Justification:
We conducted in the paper conform, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the positive and negative societal impact the supplementary
material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "4FP9fgTVlm", "reviewer": "Reviewer_UfLd", "summary": "This paper tackles zero-shot HOI detection via prompt tuning. To address the challenge posed by the absence of novel classes, the authors first incorporate LLM and VLM guidance to enrich learnable prompt tokens. Further, the authors utilize LLM to provide nuanced differentiation between unseen classes and their related seen classes, termed Unseen Text Prompt Learning (UTPL), to alleviate overfitting to seen classes. In experiment, the proposed method achieves state-of-the-art performance under major zero-shot HOI detection settings while costing fewer training resources.", "questions": "1. In Figure 2, are the VLM visual and visual encoder the same thing (i.e., CLIP)?\n\n2. Why did the authors use LLaVA for text description instead of choosing pure language models?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper tackles an important issue in zero-shot / open-vocabulary learning, i.e., overfitting to seen classes.\n2. The proposed method effectively explores guidance from large foundation models, including VLM and LLM.\n3. The authors achieve state-of-the-art performance with significantly fewer training resources.", "weaknesses": "1. It seems the authors assume the prior knowledge of unseen class names when training the UTPL module, which conflicts with the zero-shot setting.\n\n2. In Table 1, the proposed method underperforms CLIP4HOI (ResNet50+ViT-B) in unseen mAP although it is claimed to tackle the overfitting issue especially.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive feedback and insightful comments. The following is a detailed response to each of your questions. Note that references maintain their original numbering from the main paper, with new references denoted by letters and listed at the end. \n\n\n## Weakness-1. Use of unseen class names\n\nWe follow the common practices in the zero-shot HOI detection setting. Zero-shot HOI setting involves predicting unseen HOI classes, where unseen class names are typically used in training [13,15,14,45,38]. In particular, VCL, FCL and ATL [13,15,14] \"compose novel HOI samples\" during training with the unseen (novel) HOI class names. EoID [45] distills CLIP \"with predefined HOI prompts\" including both seen and unseen class names. HOICLIP [38] introduces \"verb class representation\" during training, including both seen and unseen classes. \n\nApart from using unseen class names, the prior knowledge of unseen class names is also used in existing work. UniHOI [3] utilizes LLM-generated descriptions, the prior knowledge of the unseen class names, to recognize unseen classes better with \"rich and detailed representation\". Thus, our method follows the common practices in the zero-shot HOI detection setting for conducting experiments.\n\nBeyond zero-shot HOI settings, there are HOI unknown concept discovery [a] and open-vocabulary HOI detection [b], where unseen class names cannot be used in training.\nThe open-vocabulary setting differs from HOI unknown concept discovery, with a much wider range of unseen HOI classes during testing.\nWe will explore these directions in our future work.  \n\n## Weakness-2. Comparison with CLIP4HOI \nWe acknowledge our method’s slightly lower performance when compared to CLIP4HOI under the unseen verb setting, where our method's unseen performance is 0.92 mAP lower. While it is true that there is a minor drop in accuracy, it is important to consider the significant reduction in trainable parameters that our method achieves—87.9% fewer than CLIP4HOI. Moreover, under the non-rare first unseen composition setting, we outperform CLIP4HOI by 2.22 mAP for unseen HOI classes (Table 2 second column from right to left, of the main paper).\n\nThis substantial decrease in model complexity not only lowers the computational cost but also makes it more accessible to researchers in the field with limited resources. Achieving a balanced trade-off between performance and efficiency is essential, which aligns with the growing need for more efficient models in the field. Our method provides a practical solution by offering competitive performance while drastically reducing the resource requirements. Thus, the minor compromise in accuracy is counterbalanced by significant gains in model efficiency and accessibility.\n\n## Question-1. Figure 2 visual encoder\nIn Figure 2, the \"VLM visual\" is different from the \"visual encoder\". The \"VLM visual\" is the frozen CLIP visual encoder, while the \"visual encoder\" shown in Figure 2 is fine-tuned during training and is based on our design, including visual learnable prompts $\\hat{\\mathcal{H}}_V$ and the HOI feature fusion module. \n\n## Question-2. Language model selection\n\nPure language models can also be applied to our methods for text description generation. We leverage LLaVA because it demonstrates quite strong reasoning results with GPT-4 level capabilities [32]. \nAs shown in the following, outputs from LLaVA and Chatgpt 3.5 are both informative and reasonable, benefiting the model for learning detailed representation.\n\n\nLLaVA output: _\"Swinging a baseball bat\" describes a person using a baseball bat to hit a ball. This action typically involves the person holding the bat with both hands, standing in a stance with their feet shoulder-width apart, and using their body rotation to contact the ball._\n\nChatgpt3.5 output: _\"Swing a baseball bat\" involves standing with feet apart, gripping the bat, and stepping forward as the pitch approaches. The batter rotates the torso, bringing the bat through the hitting zone to hit the ball._\n\n\n## References: \n\n[a] Discovering Human-Object Interaction Concepts via Self-Compositional Learning, ECCV22\n\n[b] Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection, CVPR24", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8fPjTcBX4i", "reviewer": "Reviewer_kbXP", "summary": "The paper studies the challenges behind successful adaption of pre-trained vision language models (VLMs), i.e., CLIP, to the problem of zero-shot Human Object Interaction (HOI) detection. Specifically, during finetuning, VLMs overfit to seen HOI classes observed during the training on HOI training data, preventing successful transfer on the unseen HOI classes. To overcome the aforementioned problem, authors propose prompt tuning mechanism coupled with guidance from a large language model (LLM). In particular, LLM is used to generate (i) descriptions of each HOI class; (ii) explain the difference between each unseen and the corresponding closest seen class to facilitate transfer on unseen classes. The prompt tuning technique itself learns a set of prompts that are shared between vision and text encoders. Authors employ attention mechanism to produce modality specific prompts via attending on class descriptions for text modality and via attending on the image embedding for visual modality correspondingly. Authors evaluate the proposed approach on the standard HOI benchmark and compare it to the recent baselines, showing improvements in terms of the parameter efficiency and the performance on unseen classes.", "questions": "* How $N=9$ was chosen? Can you provide ablations for different number of $N$?\n* Similarly to the question above, Eq. (6) and (8) suggest that the learnable prompts are inserted in the first $N$ layers of textual and visual encoders, correspondingly. What is the motivation behind this design choice? I, in general, would assume that earlier layers already consumed broad knowledge during CLIP pre-training and task specific adaptation is required for penultimate layers. Can you please provide ablations on different positioning of the learnable prompts?\n* Currently, the paper narration mainly focuses on tackling generalization to unseen classes and, indeed, experimentally confirms the improvements offered by the proposed approach. The main component causing such improvements is seem to be UTPL that employs guidance from LLM. However, according to Table 4, other components also bring substantial improvements. Can you please provide the intuition behind other components' role and how they can bring these improvements?", "limitations": "Authors discuss the limitations of the current zero-shot HOI detection setting. In particular, that to truly enable zero-shot HOI detection, one must exclude the assumption of having in advance a set of unseen HOI classes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The problem of enabling generalization to unseen classes during VLMs adaptation is important, even outside HOI detection field\n* To my knowledge, learning shared prompts and producing modality specific prompts via attention mechanism is novel\n* The evaluation is thorough, both comparing to HOI specific prompt tuning baselines and to general purpose prompt tuning approaches such as MaPLe.", "weaknesses": "My main concern is that the current narration of the proposed methodology lacks intuition and structure. For example, \n\n**Structure**\n* Section 3 starts with mentioning that the method will learn prompts per layer, and the subsequent section also use notation considering N layers, but the first time the reader encounters interaction with the layers is Section 3.3. It makes the paper hard to follow.\n\n**Intuition**\n* The main idea of the approach builds on repeatedly applying MHCA for different purposes, however these equations just stated on the high-level in Eq. (2); (3); (5) and without any intuition behind the design.\n\n* Similarly to the above, Eq. (6) and Eq. (8) state how these learnable prompts are used in the corresponding encoder layers, however the narration is on the high level and does not elucidate the particular idea behind the equations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive feedback and detailed reviews. The following is our response for your questions. Note that we use the same reference number for related papers as the main paper.\n\n## Weakness-1. Structure of the paper\nWe agree with the suggestion to introduce the \"encoder layer\" in Section 3.3 rather than at the beginning of Section 3, and will revise our paper.\n\n## Weakness-2. Design intuition for Eqs. (2), (3), (5)\nHere we provide detailed design intuition for Eq. (2) about LLM guidance, Eq. (3) about VLM guidance and Eq. (5) about UTPL.\n\nIn Eq. (2), the text learnable prompts $\\mathcal H_T$ are integrated with text embeddings $\\mathcal F_{txt}$ from LLM-generated class description. Through MHCA, $\\mathcal H_T$ only aggregate useful information from $\\mathcal F_{txt}$ with learnable attention. This is important because the information provided by LLM may not be equally crucial for our task. \n\nMoreover, specific design is introduced and used in Eqs. (3) and (5).\nTo keep trainable parameters small, we apply a down-projection layer $W_{down}$ before MHCA to reduce the feature dimension, and an up-projection layer $W_{up}$ afterward. In addition, initially Eq. (2) output equals to input $\\mathcal H_T$, by initializing $W_{up}$ to 0, which stabilizes training by gradually fine-tuning $\\mathcal H_T$.\n\nIn Eq. (3), visual features $f_{vis}^{\\mathcal I}$ from the VLM are integrated with visual learnable prompts $\\mathcal H_V$ by MHCA. Since the frozen VLM visual encoder can extract features for unseen HOIs, $\\mathcal H_V$ aggregates information from these visual features, improving performance on unseen HOIs.\n\nIn Eq. (5), the unseen learnable prompts $\\hat{h_{T_u}}$ are combined with disparity information $f_{ txt_u}^{\\rm disp}$, related seen class prompts $\\hat{h_{T_s}}$, and the unseen class prompts themselves $\\hat{h_{T_u}}$, in MHCA. The disparity information $f_{ txt_u}^{\\rm disp}$  provides distinctive attributes $\\hat{h_{T_u}}$. The related seen class prompts $\\hat{h_{T_s}}$ enhance $\\hat{h_{T_u}}$ by transferring shared features to unseen classes. The unseen class prompts $\\hat{h_{T_u}}$ retain self-information and are emphasized through MHCA's processing.\n\n\n## Weakness-3. Design intuition for Eqs. (6) and (8)\nHere we provide detailed design intuition for Eq. (6) for deep text prompt learning and Eq. (8) for deep visual prompt learning.\n\nIn deep text prompt learning, we put the learnable text prompts $\\tilde{h_T^i}$ at the end of the original text prompts $W_i$. \n$W_1$, in the first layer, is generated from \"a photo of a person \\<acting> a/an \\<object>\". \nSince we design $\\tilde{h_T^i}$ to be class-specific with HOI class information, its semantic information is very different from \"a photo of\", related to the beginning position of $W_1$.\nTherefore, the end position of $W_1$ is better suited for $\\tilde{h_T^i}$, where the semantic information is more connected to $W_1$. \n\n\nMoreover, after N layers, we stop introducing new learnable prompts.\nBasically, when introducing new learnable prompts to each layer, we observe the VLM deals with seen classes better with decreased unseen performance.\nMoreover, inserting learnable prompts into deeper layers of VLM, makes the performance sensitive, because the feature space is mature in those layers [19].\n\nSimilar to deep text prompt learning, deep visual prompt learning also only introduces new learnable prompts $\\hat{h_V^i}$ until layer N. \nThe position of $\\hat{h_V^i}$ does not significantly influence the outcome, as original frozen visual prompts $E_i$ correspond to different regions of the input image, but $\\hat{h_V^i}$ contains global image information, equally enhancing $E_i$.\n\n## Question-1. Ablation study for hyperparameter $N$\nThe following table shows the ablation study for the hyperparameter $N$, where $N$ means we introduce learnable prompts from the first layer until layer $N$. \n\n| N | Full | Seen | Unseen |\n| :-----| :----: | :----: |:----: |\n| 4 |32.60|33.99|24.10|\n|9|32.32|33.49|__25.10__|\n|12|32.76|34.19|23.98|\n\nWe use N=9 in our main paper because it shows the best unseen performance.\n\n## Question-2. Ablation study for different positions of learnable prompts\nThe following table shows the ablation study for the different positions of learnable prompts. We find that the position of learnable prompts does not affect the outcome too much. Thus, we follow [19] and fine-tune layers 1-12, where fine-tuning layers 1-9 shows slightly better unseen performance.\n\n| Position | Full | Seen | Unseen |\n| :-----| :----: | :----: |:----: |\n| 1-9 |32.32|33.49|__25.10__|\n|3-11| 32.24| 33.45 | 24.83 |\n|4-12|32.40 |33.64| 24.82 |\n\n## Question-3. Intuition for components tackling generalization, other than UTPL\nWe introduce components including LLM guidance, VLM guidance and inter-HOI fusion, other than the UTPL module, which can enhance generalization ability to unseen classes. We discuss the design intuition one by one. \n\nThe LLM guidance, mentioned in Section 3.1 \"Text Prompt Design\" of the main paper, integrates learnable text prompts with detailed HOI class descriptions. This approach enhances the model's understanding of unseen classes, which lack training data, by providing detailed information rather than simple class names.\n\nThe VLM guidance, detailed in Section 3.1 \"Visual Prompt Design,\" combines learnable visual prompts with image features from the frozen VLM visual encoder. Since the VLM includes unseen HOI information, the frozen encoder can extract unseen HOI representations during testing. The learnable visual prompts then aggregate these representations, enhancing unseen class prediction.\n\nInter-HOI fusion, as mentioned in Appendix Section 6.4, refines HOI visual features by considering surrounding HOI features in the image. For instance, to detect \"cut a cake,\" the model uses the surrounding visual context like \"cut with a knife,\" making recognition easier.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JApzbuc6E2", "reviewer": "Reviewer_1u4N", "summary": "The paper proposes a novel prompt learning framework for zero-shot Human-Object Interaction (HOI) detection, which enhances the generalizability of Vision-Language Models (VLMs) by interacting with Large Language Models (LLMs) to obtain descriptions.", "questions": "Please see the weakness.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method appears logical and achieves general improvements in performance.\n2. Utilizing LLMs to generate descriptions is a promising direction.\n3. The paper is well-written, and the framework diagram is clear.", "weaknesses": "1. The prompt design seems to be a hybrid approach, combining visual prompt learning [1, 3] with text prompts, such as those used in COCOOP, without a specific focus on task-specific configurations. It is difficult to justify the necessity of this work for the field.\n\n2. Despite the claim of improved performance on novel classes, the method performs worse on unseen classes compared to CLIP4HOI, while it shows improved performance on seen classes. Given that novel class performance is crucial, this result seems not consistent with this task.\n\n3. W_down presents in both Equation 3 and Equation 5 is puzzling. Equation 5 attempts to integrate descriptions for unknown categories with similar known categories and LLM-generated details. More details on the training process would be helpful, as it appears that data training might be lacking since unknown categories are not included in the training.\n\n4. There is substantial related work on using LLMs for new class descriptions in zero-shot scenarios, such as CuPL [2] for classification and DVDet [3] for detection, which use attribute decomposition to aid in categorization. DVDet also generates more discriminative descriptors by distinguishing confusing categories. The paper lacks a discussion on such related LLM-based works. It would be interesting to see if using action descriptors might also yield some benefits.\n\n5. The paper could benefit from providing more visualization results to demonstrate the effectiveness of both general and discriminative descriptions generated by the method.\n\n[1] Visual Prompt Tuning\n\n[2] VISUAL CLASSIFICATION VIA DESCRIPTION FROM LARGE LANGUAGE MODELS\n\n[3] LLMS MEET VLMS: BOOST OPEN-VOCABULARY OBJECT DETECTION WITH FINE-GRAINED DESCRIPTORS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed, helpful feedback. We address each of your concerns in the following. \n\n## 1. Task-specific configurations\n\nOur method is specifically designed for HOI settings with the following task-specific configurations. \n\nFirst, our method is configured for HOI detection by extracting visual features for each human-object pair in an image, different from using visual features from an entire image like general prompt learning methods [19,56,6,8]. We extract HOI visual features from each human-object pair to compare with text features, which is introduced in Appendix Section 6.4. \n\nSecond, the UTPL module is designed for HOI settings to recognize interactions with strong connections and subtle differences (i.e., \"straddle bike\" and \"ride bike\"). If \"ride bike\" is unseen, UTPL leverages related seen HOIs like \"straddle bike\" to help with recognizing \"ride bike\". \n\nThird, our proposed LLM-generated description is another task-specific configuration. In particular, the LLM-generated descriptions provide the nuances between unseen and related seen HOI classes, which further enhance the understanding of the unseen class.\n\n## 2. Comparison with CLIP4HOI \n\nWe acknowledge our method’s slightly lower performance when compared to CLIP4HOI under the unseen verb setting, where our method's unseen performance is 0.92 mAP lower. While it is true that there is a minor drop in accuracy, it is important to consider the significant reduction in trainable parameters that our method achieves—87.9% fewer than CLIP4HOI. Moreover, under the non-rare first unseen composition setting, we outperform CLIP4HOI by 2.22 mAP for unseen HOI classes (Table 2 second column from right to left, of the main paper).\n\nThis substantial decrease in model complexity not only lowers the computational cost but also makes it more accessible to researchers in the field with limited resources. Achieving a balanced trade-off between performance and efficiency is essential, which aligns with the growing need for more efficient models in the field. Our method provides a practical solution by offering competitive performance while drastically reducing the resource requirements. Thus, the minor compromise in accuracy is counterbalanced by significant gains in model efficiency and accessibility.\n\n\n## 3. Clarification and training details for Eqs. (3),(5)\n\n$W_{down}$ in Eqs. (3) and (5) refer to the down projection layers, but they are not the same projection layers with different weights. We provide more training details for Eq. (5) in the following.\n\nAlthough there is no annotated image for the unseen/unknown categories in training, we have two ways to optimize Eq. (5). First, we design a class-relation loss (Eq. (15) in the Appendix) to keep the relationship between seen and unseen classes, measured by cosine similarity between text features. This way, unseen prompts can also be refined based on their relation to seen classes. Second, while there are no annotated images for unseen/unknown classes, the annotated training data serves as negative samples. If the prediction score for an unseen class is too high, the model is penalized (Eq. (16) in the Appendix). \n\nAdditionally, in Eq. (5), each unseen learnable prompt is linked to a similar seen learnable prompt. If the seen prompts are optimized after each training step, the updated seen prompts will be used to refine the corresponding unseen prompts in UTPL. We will include the above discussion in the updated paper. \n\n\n## 4. Benefits from action descriptors\n\nWe thank the reviewers for pointing out the related LLM-based works such as action descriptors, which are potentially useful for HOI detection. We conducted the following preliminary experiment given the limited time in this rebuttal. \n\nDescriptors mentioned in CuPL[a] and DVDet [b] refer to attribute decomposition to benefit category recognition. We leverage the attribute decomposition idea from DVDet and tailor it to our method. We follow DVDet [b] to generate action descriptors for each class and integrate them into HOI class text features. This process enhances the detail and distinctiveness of the HOI class representations. Descriptors with low cosine similarity to the HOI class text features are discarded to avoid noisy information.\n\nThe following table presents our preliminary result under the unseen verb setting, showing improvement with our simple and direct adoption of the action descriptors. These results demonstrate that LLM-generated descriptions, such as action descriptors, are potentially useful for HOI detection. We will include a discussion of the related LLM-based works in our updated paper. \n\n| Methods |Full | Seen | Unseen |\n| :-----| :----: | :----: |:----: |\n|Ours|32.32|25.10|33.49|\n|Ours + action descriptor| 32.63 | 25.14 | 33.85|\n\n\n## 5. Visualization results\n\nWe acknowledge the reviewer’s request for additional visualizations. While we have made every effort to understand and address this, with all due respect, we are still unsure what visualization results we need to add. Based on our understanding, in this rebuttal, we provide the qualitative results to illustrate the impact of LLM guidance (using general description) and UTPL (using discriminative description) shown in Fig. 1 of our attached PDF.\n\nThe LLM guidance provides detailed class information, improving the performance over the baseline. Distinctive descriptions help to distinguish unseen classes from related seen HOIs, enhancing unseen performance and challenging case predictions.\n\n\n## References\n\n[a] Visual classification via description from large language models, ICLR23 \n\n[b] LLMs meet VLMs: Boost open vocabulary object detection with fine-grained descriptors, ICLR24", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6d2rUEPt1A", "reviewer": "Reviewer_RCVD", "summary": "This paper investigates the problem of human-object interaction (HOI) detection. This paper introduces EZ-HOI, a method for efficient zero-shot HOI detection in an open-world setting. EZ-HOI also explores the use of LLM and VLM guidance for learnable prompts to enhance prompt knowledge and aid in adapting to HOI tasks. To better adapt to unseen classes, EZ-HOI establishes relationships between unseen and seen classes.", "questions": "What is the fully fine-tuned performance? What is the baseline?", "limitations": "This submission discusses limitations.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The state-of-the-art experimental results in the zero-shot setting. EZ-HOI demonstrates good performance in zero-shot scenarios. More importantly, by leveraging prompt tuning, the number of trainable parameters in EZ-HOI is significantly smaller compared to other methodologies.\n- The code in the supplementary material is provided to ensure the reproducibility of the study.", "weaknesses": "- The overall method is straightforward but lacks sufficient novelty. Compared to UniHOI, which also claims to utilize Spatial Prompt Learning, the main difference in EZ-HOI lies in its use of more traditional prompt learning methods (like VPT-DEEP) and its additional modeling of relationships between unseen and seen classes when handling unseen categories. Since many previous works have already demonstrated that prompt tuning (ViT-adapter/CLIP-adapter) can achieve even higher performance than full fine-tuning, the claims of EZ-HOI seem somewhat weak. Additionally, modeling the relationship between unseen and seen classes as a technical contribution also appears somewhat limited.\n- The fully-supervised setting results are provided in Appendix Table 6, but the performance is worse than Uni-HOI. Firstly, this is a very important experiment and should be included in the main text. Secondly, it appears that under full supervision, EZ-HOI performs worse than Uni-HOI, which contradicts the claims in the main text. Have the authors provided an in-depth analysis of the reasons? If the performance difference is due to parameter count, what would the performance be with full fine-tuning?\n- The performance improvement seems to come from the baseline rather than the proposed modules. As shown in the ablation study (Table 4), the baseline of EZ-HOI achieves 37.44 on seen categories, which is already state-of-the-art (SOTA). I wonder what EZ-HOI’s baseline is and why it can achieve such strong performance on seen categories. Additionally, considering that the UTPL module increases unseen category performance by two points, and EZ-HOI shows a two-point increase compared to Uni-HOI, does this mean that EZ-HOI’s performance gain over Uni-HOI is primarily due to the UTPL module?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful feedback. Here is our response. Note all references are from the main paper’s citations.\n\n## Weakness-1. Our novelty and technical contribution\n__(1) Novelty__\n\nWe would like to clarify that our innovation lies in proposing a novel framework, rather than a new approach to prompt learning.  Existing methods such as UniHOI [3], HOICLIP [38] and GEN-VLKT [28] align HOI model's visual features to text features from a frozen VLM. However, aligning with VLM features requires training transformer-based models, which is computationally expensive.\n\nUnlike existing methods that utilize frozen VLM text features, we fine-tune both the visual and text encoders of the VLM with guidance from an LLM. This allows us to adapt both visual and text features to our HOI setting, rather than only adapting visual features. Consequently, our method, EZ-HOI, improves the alignment between visual and text representations, achieving SOTA performance in zero-shot HOI settings with significantly fewer trainable parameters.\n\nA comparison between our method and UniHOI is shown in the table below. Like most existing HOI detection methods [38,28,36,45], we use CLIP as the VLM. UniHOI [3], on the other hand, uses BLIP2, a more powerful VLM than CLIP. As shown in [25], the performance of BLIP2 is significantly superior to CLIP in zero-shot image-text retrieval tasks.\n\nDespite using CLIP, our method still achieves comparable performance to UniHOI under various zero-shot settings, which demonstrates the effectiveness of our novel framework, EZ-HOI.\n\n| Method |VLM|Setting| Full | Unseen | Seen |\n| :-----| :----: | :----: |:----: |:----: |:----: |\n| UniHOI|BLIP2|UV |34.68|26.05|36.78|\n|Ours |CLIP|UV|__36.84__|__28.82__|__38.15__|\n| UniHOI|BLIP2|RF-UC |32.27|28.68|33.16|\n|Ours |CLIP|RF-UC|__36.73__|__34.24__|__37.35__|\n| UniHOI|BLIP2|NF_UC|31.79|28.45|32.63|\n|Ours |CLIP|NF_UC|__34.84__|__36.33__|__34.47__|\n\n__(2) Technical contributions__\n\nRegarding modeling the relationship between unseen and seen classes, our novelty lies in the UTPL module. Our UTPL enhances unseen prompt learning by leveraging related seen classes with the help of LLM-generated descriptions, which to our knowledge is a novel idea. This is in contrast to existing methods, which either use LLM descriptions without seen class context [3] or rely on seen class information alone [13,15].\n\nOur technical contribution is centered around mitigating the overfitting problem when using prompt learning to adapt a VLM for zero-shot HOI detection. While existing prompt learning typically suffers from the overfitting problem to seen classes [56,6,8,19], our method achieves 11.6 mAP improvement on unseen HOI classes compared the prompt learning baseline (MaPLe [19]) as shown in Fig. 1(d). \n\n## Weakness-2. Performance under fully-supervised setting\n\nIn our Appendix, Table 6 presents the quantitative comparison under fully-supervised settings. To ensure a fair comparison, we used CLIP as our VLM, the same as most existing methods [38,28,36,45]. As mentioned earlier in this rebuttal, UniHOI employs BLIP2, which is a superior VLM compared to CLIP.\n\nTo provide a fair comparison, we show the performances of UniHOI and our method using the same VLM (CLIP). As observed, our method outperforms UniHOI when both use CLIP.\n\nThe results for UniHOI using CLIP were obtained from UniHOI's OpenReview rebuttal (<https://openreview.net/forum?id=pQvAL40Cdj&noteId=kI6HJnJ1KB>).\n\n| Method | Full | Rare | Non-rare |\n| :-----| :----: | :----: |:----: |\n| $UniHOI_l$ |36.84|35.71|37.05|\n|Ours |38.61|37.70|38.89|\n\nWe will follow the suggestion to move Table 6 of the appendix to the main text, and include our rebuttal discussion to the paper upon acceptance.\n\n## Weakness-3. Baseline method and performance gain \n\n__(1) Baseline and the strong seen performance__\n\nOur baseline is MaPLe [19], not the first row of Table 4 in our main text. The first row (Table 4) is part of our developed method, focusing on improving seen classes, which employs our intra-HOI fusion and visual adapter. Both modules are mentioned in Section 3.3: Intra-HOI is introduced in Appendix 6.4, and visual adapter is based on [23]. \n\nTo clarify the baseline performance, we provide an extended ablation study table below. The first row shows the baseline method, MaPLe [19]. Our intra-HOI fusion module improves the seen HOI performance by 7.41 mAP, as shown in the second row. The third row is the result of adding the visual adapter, which is the first row of Table 4 in our main text. We will follow the suggestion to update the ablation studies in Table 4 of the main paper and include this discussion. \n\n\n| Method |Intra-HOI fusion|visual adapter [23]| Full | Unseen | Seen |\n| :-----| :----: | :----: |:----: |:----: |:----: |\n| MaPLe |No|No| 26.26|17.19|27.73|\n|MaPLe|Yes|No|33.52|23.54|35.14|\n|MaPLe| Yes|Yes|35.40|22.91|37.44|\n|Ours (full model)|Yes|Yes|38.61|37.70|38.89|\n\n__(2) Performance gain from each designed modules__\n\nSince our baseline is MaPLe [19], the performance gain over UniHOI is not solely due to the UTPL module but results from all the modules we designed (including LLM guidance, VLM guidance, and intra- and inter-HOI fusion). Each module's contribution is shown in the table above and in Table 4 of the main paper, illustrating how each module incrementally enhances the final performance. While it is true that UTPL contributes 2.42 mAP to unseen class performance, other modules, such as our LLM guidance, also significantly improve unseen performance by 1.52 mAP, and VLM guidance increases unseen performance by 1.33 mAP.\n\n## Question-1. Fully fine-tuned method and baseline\nAs discussed in Question 3, our baseline is MaPLe [19]. Our framework is based on prompt tuning to enable a VLM to adapt to HOI tasks, so we do not use full fine-tuning in our method. We acknowledge that full fine-tuning might be beneficial, and we plan to explore this in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDfuOZKUrH", "reviewer": "Reviewer_kLJ1", "summary": "This work presents EZ-HOI, an innovative framework that addresses the zero-shot HOI detection challenge by employing prompt learning. It integrates LLM and VLM guidance to enrich prompts and adapt to HOI tasks effectively. By learning from related seen classes, EZ-HOI overcomes the limitation of lacking labels for unseen classes, enhancing its performance on them. The framework achieves state-of-the-art results with significantly fewer trainable parameters than existing methods, showcasing its efficiency and effectiveness in zero-shot HOI detection.", "questions": "N/A", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written.\n- The Unseen Text Prompt Learning is well-designed.\n- Overall, the zero-shot results with less trainable parameters are good.", "weaknesses": "- This paper primarily addresses the improvement of zero-shot HOI detection benchmark performance by training with the pre-definition of all HOI classes. This creates an unfair comparison with previous work.\n- In light of the above, the reviewer believes that the authors should provide experimental results where the pre-definition of all HOI classes is unknown during training, allowing for a fairer comparison with previous work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive feedback. The following is our response for your questions. Note that references maintain their original numbering from the main paper, with new references denoted by letters and listed at the end. \n\n## 1. Pre-definition of all HOI classes\n\nRegarding the comparison with previous work, we follow the common practices in the zero-shot HOI detection setting, as introduced in the following. Zero-shot HOI setting involves predicting unseen HOI classes, where unseen class names are typically used in training [13,15,14,45,38]. In particular, VCL, FCL and ATL [13,15,14] \"compose novel HOI samples\" during training with the unseen (novel) HOI class names. EoID [45] distills CLIP \"with predefined HOI prompts\" including both seen and unseen class names. HOICLIP [38] introduces \"verb class representation\" during training, including both seen and unseen classes. \n\nBeyond the zero-shot HOI setting, there are HOI unknown concept discovery [a] and open-vocabulary HOI detection [b], where unseen class names cannot be used in training. The open-vocabulary setting differs from HOI unknown concept discovery, with a much wider range of unseen HOI classes during testing. We will explore these directions in our future work. \n\n\n## 2. Experiments without pre-definition of all HOI classes\n\nWe appreciate the suggestion to conduct experiments without pre-definition of all HOI classes. As explained in the above question, our work follows the established zero-shot HOI detection protocols, where the use of unseen class names is a common practice, as illustrated by the existing HOI detection methods [13,15,14,45,38]. Nonetheless, we recognize the value of exploring alternative settings where unseen class names are not pre-defined, which could further demonstrate the effectiveness of our approach. While this falls outside the scope of the current submission, we are willing to include this experiment in the final version of our paper, to comprehensively validate our method across varying settings. \n\n\n## References: \n\n[a] Discovering Human-Object Interaction Concepts via Self-Compositional Learning, ECCV22 \n\n[b] Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection, CVPR24", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BbsCzFSaIM", "reviewer": "Reviewer_f78r", "summary": "In order to solve the HOI detection problem in a zero-shot setting, the authors propose the Unseen-class Text Prompt Learning module. Using learnable visual prompts and textual prompts, it effectively utilizes the knowledge from large language models and Vision-Language Models and performs well in unseen class HOI detection.", "questions": "1. In Qualitative Results, the authors use Figure 4 for visual illustration, but it is not possible to conclude from Figure 4 that \"MaPLe tends to predict seen classes with high confidence scores\". The authors should have given a clearer and more accurate illustration of the figure.\n2. The authors claim to have \"employed an LLM to provide the nuanced differences between related seen and unseen classes, improving our method for unseen class prompt learning. prompt learning.\" However, the information provided by the LLM may be misleading or inaccurate, did the authors filter the information? If not, what can be done to avoid HOI detection being negatively affected from the information provided by LLM?\n3. For the UTPL module, the authors split the LLM-generated description into multiple sentences; how do they control the length of individual statements?\n4. The author mentions: \" A prediction was deemed a true positive if the HOI classification was accurate and the Intersection over Union (IoU) between the predicted human and object bounding boxes and the ground-truth bounding boxes exceeded 0.5.\", why is the threshold value chosen to be 0.5 instead of 0.7 or other? What is the basis for the choice?\n5. WRITING DETAILS: Abbreviated nouns should be introduced the first time they appear.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.The work is innovative, well-written and has clear diagrams.\n2.The authors' proposed learnable prompts scheme and UTPL module are novel.", "weaknesses": "1. In Qualitative Results, the authors use Figure 4 for visual illustration, but it is not possible to conclude from Figure 4 that \"MaPLe tends to predict seen classes with high confidence scores\". The authors should have given a clearer and more accurate illustration of the figure.\n2. The authors claim to have \"employed an LLM to provide the nuanced differences between related seen and unseen classes, improving our method for unseen class prompt learning. prompt learning.\" However, the information provided by the LLM may be misleading or inaccurate, did the authors filter the information? If not, what can be done to avoid HOI detection being negatively affected from the information provided by LLM?\n3. For the UTPL module, the authors split the LLM-generated description into multiple sentences; how do they control the length of individual statements?\n4. The author mentions: \" A prediction was deemed a true positive if the HOI classification was accurate and the Intersection over Union (IoU) between the predicted human and object bounding boxes and the ground-truth bounding boxes exceeded 0.5.\", why is the threshold value chosen to be 0.5 instead of 0.7 or other? What is the basis for the choice?\n5. WRITING DETAILS: Abbreviated nouns should be introduced the first time they appear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive feedback and detailed reviews. The following is our response for your questions. Note all references are from the main paper’s citations.\n\n## 1. Illustration for Fig. 4\nWe provide more illustration for Fig. 4 in the following.\nFig. 4 shows the qualitative results of both our method and MaPLe. In particular, MaPLe struggles to detect unseen classes, either missing unseen HOI classes or predicting the wrong unseen HOI classes. For example, if an image only contains unseen classes, MaPLe tends to predict wrong seen classes and miss the correct ones. As shown in the bottom right of Fig. 4, this image contains unseen class only (\"wear tie\"), MaPLe predicts related wrong seen classes such as \"pulling tie\" and \"adjusting tie\", but fails to predict the ground-truth unseen HOI (\"wear tie\").\n\nThis shows the limited generalization ability of MaPLe to unseen classes. In contrast to MaPLe, our method can predict both seen and unseen classes more accurately. We will revise the discussion for Fig. 4 in the updated paper.\n\n## 2. Avoid negative effect of misleading information from LLM in UTPL\nWe agree that LLM can provide misleading information. \nTo deal with the problem, we utilize LLM description information with learnable attention.\nSpecifically, the UTPL module integrates the LLM information into the unseen text learnable prompts, through multi-head cross-attention (MHCA), where learnable attention is optimized during the training. \nThe training supervision for UTPL is from the class-relation loss in Eq. (15), to retain the relationship among each HOI class, indicated by cosine similarity between text class features.\nIf a wrong information is provided by LLM, then the unseen text features, integrated with the wrong information, display improper relations to other seen class text features. This will be penalized by the class-relation loss, and then the attention to the wrong information will be reduced in UTPL for unseen prompt learning. \nTherefore, the design can mitigate the possible negative effect from inaccurate information provided by LLM\n\nIn the future, we will also explore other approaches such as chain-of-thought (CoT) and retrieval-augmented generation (RAG) to mitigate the negative effect from LLM-generated misleading information.\n\n## 3. Control the length of individual statements in UTPL\nThe length control of individual statements is achieved by giving the prompt for LLM before  the generation for the desired description: _\"Please reply with multiple short sentences in the following, instead of a long sentence.\"_\n\n## 4. IoU threshold value selection\nWe follow existing papers [19,28,38,23] to choose the threshold value, the IoU between predicted human and object bounding boxes and the ground-truth bounding boxes, to be 0.5. \nThe threshold value is one criterion to judge whether a prediction is true positive or not. This is not only used in the evaluation during testing time, but also used in training to match predictions with ground-truth human-object pairs for model learning. \n\nWe conduct the ablation study for the threshold value selection in training time, as shown in the following table. We find that using 0.5 achieves the best performance for both seen and unseen classes. \n\n| Threshold | Full | Unseen | Seen |\n| :-----| :----: | :----: |:----: |\n| 0.2 |31.02 |23.05| 32.32|\n|0.5|32.32|25.10|33.49|\n|0.7| 29.59| 23.66 | 30.55|\n\n## Writing details\n\nWe will make sure the abbreviations are introduced with their full names the first time they appear in the revised version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thoughtful and constructive feedback. We appreciate that the reviewers found our work to be "innovative" (Reviewer f78r), "tackling an important issue in zero-shot learning and generalization to unseen classes during VLMs adaptation" (Reviewer UfLd, kbXP), and "well-written" (Reviewer 1u4N, kLJ1, f78r).

Regarding the major modules in our method, we thank the reviewers to recognize that "the proposed learnable prompts are novel" (Reviewer kbXP, f78r), and "well-designed Unseen Text Prompt Learning (UTPL)" (Reviewer kLJ1, f78r). 

In terms of evaluation and quantitative comparison, we appreciate the reviewers to point out that "the proposed method achieves state-of-the-art performance with significantly smaller trainable parameters" (Reviewer UfLd, RCVD, and kLJ1), and "the evaluation is thorough" (Reviewer kbXP).

Meta-review: The paper was evaluated by six reviewers. Authors provided strong rebuttal and most reviewers engaged in the author-reviewer discussion. As a result, five out of the six reviewers provided positive ratings, and the sixth reviewer who provided negative review did not respond to authors rebuttal. After reading the reviews and rebuttal, the AC agrees to accept this paper. Authors are encouraged to polish the paper further by considering the questions, suggestions and comments of the reviewers.
Decision: Accept (poster)