CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection
Authors: Yunyao Mao
Jiajun Deng
Wengang Zhou
Li Li
Yao Fang
Houqiang Li
Venue: NeurIPS 2023 poster (2023)
Keywords: human-object interaction detection
zero-shot learning
CLIP model adaptatiion
TL;DR: We introduce a novel two-stage zero-shot HOI detector, where the CLIP model is leveraged and adapted into a fine-grained HOI classifier.

Abstract: Zero-shot Human-Object Interaction (HOI) detection aims to identify both seen and unseen HOI categories. A strong zero-shot HOI detector is supposed to be not only capable of discriminating novel interactions but also robust to positional distribution discrepancy between seen and unseen categories when locating human-object pairs. However, top-performing zero-shot HOI detectors rely on seen and predefined unseen categories to distill knowledge from CLIP and jointly locate human-object pairs without considering the potential positional distribution discrepancy, leading to impaired transferability. In this paper, we introduce CLIP4HOI, a novel framework for zero-shot HOI detection. CLIP4HOI is developed on the vision-language model CLIP and ameliorates the above issues in the following two aspects. First, to avoid the model from overfitting to the joint positional distribution of seen human-object pairs, we seek to tackle the problem of zero-shot HOI detection in a disentangled two-stage paradigm. To be specific, humans and objects are independently identified and all feasible human-object pairs are processed by Human-Object interactor for pairwise proposal generation. Second, to facilitate better transferability, the CLIP model is elaborately adapted into a fine-grained HOI classifier for proposal discrimination, avoiding data-sensitive knowledge distillation. Finally, experiments on prevalent benchmarks show that our CLIP4HOI outperforms previous approaches on both rare and unseen categories, and sets a series of state-of-the-art records under a variety of zero-shot settings.

CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection
Yunyao Mao Jiajun Deng"
Wengang Zhou't Li Yao Fang² Houqiang Li" CCS Key Laboratory of Technology in GIPAS, EEIS Department,
University Science and Technology of China
The University of Adelaide, AIML Merchants Union Consumer Finance Company Limited myy20160mai1. wwt.uutce.. edu. en, jiajun dew.gadeennddddeeeeeeeeeeeee edu. au, zhag@ustc. edu.cn lili@ustc. en, fangyaodmucfe.. com, lihq@ustc. edu.
Abstract
Zero-shot Human-Object Interaction (HOI) detection aims to identify both seen and unseen HOI categories. A strong zero-shot HOI detector is supposed be not only capable of discriminating novel interactions but also robust to positional distribution discrepancy between seen and unseen categories when locating human- object pairs. However, nop-performing zero-shot HOI detectors rely seen and predefined unseen categories to distill knowledge from CLIP and jointly locate human-object pairs without considering the potential positional distribution discrep- ancy, leading to impaired transferability. In this paper, we introduce CLIP4HOI, novel framework for zero-shot HOI detection. CLIP4HOI developed the vision-language model CLIP and ameliorates the above issues in the following two aspects. First, avoid the model from overfitting to the joint positional distribu- tion of seen human-object pairs, we seek to tackle the problem of zero-shot HOI detection disentangled two-stage paradigm. To be specific, humans and objects are independently identified and all feasible human-objec! pairs are processed by Human-Object interactor for pairwise proposal generation. Second, to facilitate better transferability. the CLIP model is elaborately adapted into fine- -re-grained HOI classifier for proposal discrimination, avoiding data- sensitive knowledge dis- tillation. Finally, experiments on prevalent benchmarks show that our CLIP4HOI outperforms previous approaches on both rare and unseen categories, and sets a series of state-of-the-art records under variety of zero-shot settings.
Introduction
Human- Human-Object Interaction (HOI) detection aims at locating paired humans and objects and identifying their interactions. It finds applications in several downstream tasks, including surveillance, robotics, and human-computer interaction. Despite significant progress in recent years, in most previous works, the task of HOI detection has been conventionally restricted to predefined HOI categories. Given the flexibility of verb-object combinations in human-object interaction scenarios, obtaining a comprehensive dataset laborious and time-consuming. Moreover, the obtained detectors also struggle to generalize to unseen HOI categories, even if both verb and object in the novel category are present (but never composed together) in the training set. These facts motivate and highlight the exploration of zero-shot HOI detection, which targets to development of transferable HOI detectors.
Latest zero-shot HOI detectors [31, 44, 46, 48] seek to harness the general visual and linguistic knowledge of CLIP [38] for identifying novel HOIs. Despite notable advancements over conventional compositional learning approaches [2, 15, 14, 36], these methods exhibit certain limitations.
*Correponding authors: Wengang Zhou and Houqiang
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Detected: subj. obj.
seen unseen
10
gain (%)
Pairing
box pairs
4.25
F.S
CLIP
HOI
HOI text descriptions
scores
ja.s aJi
245
(0450.0
i0.00.35
jais (el
an
HOI Classifier
-100
-20
20
280
Angle distribution (surfboard)
Distribution discrepancy
(a) An overview of our CLIP4HOI
(b) Distribution discrepancy
(c) CLIP4HOI EolD
Figure 1: In our CLIP4HOI, the CLIP model is elaborately adapted for fine-grained pairwise HOI discrimination, avoiding data-sensitive knowledge distillation (a). Thanks to the distribution- agnostic proposal generation, our CLIP4HOI demonstrates better robustness when encountering large positional distribution discrepancy between seen and unseen HOI categories (b)(c). The distribution statistics are the angles between the line from the person to the object and the x-axis. The distribution discrepancy measured with KL divergence.
First, the prevailing approaches (31,46) adhere to the one-stage framework, where each query or pair thereof is required to jointly localize the human and the object. This strategy bears the inherent risk of overfitting the decoder to the joint positional distribution of human-object pairs for seen HOI categories, which results in compromised efficacy of the model when encountering novel HOIs that exhibit significant distribution discrepancy with the seen categories, as shown in Figure 1b. Second, the knowledge of CLIP is transferred to the detector through knowledge distillation, the effectiveness of which largely depends on the completeness of the training data. However, given the absence of unseen categories during training, the distillation process is dominated by samples from seen categories, resulting in impaired generalization of the model to unseen categories. Although the latest approach [46] tries to mine potential unseen categories during training, the text descriptions of unseen categories required in advance. As result, the generalization capability of the detector is restricted to predefined unseen HOI categories, which not conducive to practical applications.
To this end, we propose CLIP4HOI, a novel framework for zero-shot generalizable HOI detection. On the one hand, in view the overfitting issue associated with the joint localization of humans and objects, we seek to tackle the problem of zero-shot HOI detection in two-stage paradigm, as shown in Figure 1a. Specifically, with the help an off-the-shelf object detector, the first stage focuses on the accurate identification of humans and objects from images, followed by traversing all feasible human- object combinations in the Human-Object (HO) interactor for pairwise HOI proposal generation. On the other hand, instead of employing data-sensitive knowledge distillation from CLIP to existing detectors, we demonstrate that the adapted CLIP is a natural fit for generalizable fine mne-grained HOI discrimination. Specifically, based the generated pairwise proposals, transformer-batt based HOI decoder is designed to aggregate contextual visual clues from the output of the CLIP image encoder. HOI scores are then obtained in the HOI classifier by comparing the text embeddings of prompted HOI descriptions with the visual features from different granularities. The final scores for all human- object pairs the integration of HOI scores and the object class priors out of the object detector. Thanks to the distribution-agnosti pairwise proposal generation and the strong transferability of CLIP. the resulting HOI detector exhibits better robustness to positional distribution discrepancy and superior generalization capability to unseen HOI categories, as shown in Figure Ic.
To verify the effectiveness, we conducted extensive experiments two prevalent HOI detection benchmarks, i.e., HICO-DET [4] and V-COCO [14]. Results show that our approach exhibits superior performance under variety of zero-shot settings. Specifically, compared to previous state-of-ther of-thh-art methods GEN-VLKT [3]] and EolD [46], our CLIP4HOI achieves absolute mAP gains of 6.43, 4.67, and 8.41 on the unseen categories of RF-UC, NF-UC, and UO settings, respectively. Moreover, our CLIP4HOI also demonstrates exceptional performance under the fully supervised setting, particularly in rare categories where it outperforms Liu al. [32] by 3.65 mAP the HICO-DET dataset.
Overall, we make the following three-fold contributions:
Given the compromised generalization capability of the joint human-object localization, we propose a novel two-stage framework, termed CLIP4HÓI, for zero-shot HOI detection, which leverages the generalizable knowledge of CLIP for unseen interaction identification.

---

Instead of relying on data-sensitive knowledge distillation, we demonstrate that by carefully designing the adaptation modules, the CLIP model itself exhibits a strong capability for fine- grained HOI discrimination, enabling the unrivaled zero-shot transferability of our CLIP4HOL. We conduct extensive experiments on two prevalent benchmarks to verify the effectiveness of the proposed approach. Compared to previous methods, our CLIP4HOI exhibits remarkable performance and sets a series of state-of-the-ar records.
2
Related Work
Human-Object Interaction Detection:
There are two mainstream solutions for HOI detection,
one-stage and two-stage. Two-stage methods [4, 10-12, 15, 26,28, 29, 33, 32, 42, 47, 56] first detect human and object instances in the image with off-the-shelf object detector. Then relation modeling strategies like multi-steam fusion [4] and graph reasoning [10] are designed to identify interactions between each pair of detected humans and objects. One-stage methods, on the other hand, undertake detection, association, and classification in a single stage. Early one-stage methods adopt interaction point [30,45 45] or union box [22] as anchors to facilitate the joint localization of humans and objects. Recent one-stage approaches [5, 2,,,,12,2, 23, 24, 41,41,3,5555 $2,5222,4, 57, have been inspired by the query-based object detector [3], wherein group of learnable queries is employed in the transformer decoder to simultaneously predict (human, verb, object) triplets. Despite the promising performance under the fully-supervised setting, we argue that such joint localization strategy tends to overfit the decoder to the joint position distribution of humans and objects for known verb-object combinations. Therefore in this work, we seek solve the problem of zero-shot HOI detection in a more generalizable two-stage paradigm.
Contrastive Language-Iman Pre- Traininnn: Contrastive Language-Iman Image Pre-Training (CLIP) [38] is a multimodal learning framework that entails joint training of an image encoder and a text encoder on large-scale image-text pairs. While traditional approaches rely exclusively on either visual or textual cues for learning, CLIP utilizes both modalities to acquire a shared representation space that aligns images and their corresponding captions in common embedding space. The outcome of this joint training is the ability of CLIP to facilitate zero-shot and few-shot learning on a variety of downstream tasks, including image classification [38], object detection [8, 9, 13], and image retrieval [1]. The success of CLIP also opens up new avenues for zero-shot HOI detection.
Zero-shot Human-Objes Object Interaction Detection: Zero-shot human-object interaction (HOI) detection strives to develop a detector that can effectively generalize to HOI categories not encountered during training. It is of practical significance due the flexibility of verb-object combinations.
Early works [2. 15, 17-20, 20,,200,4,33 34,36) mainly adopt compositional learning for zero-shot HOI detection. VCL [17] decomposes HOI representation into object- and verb-specific features and then composes novel HOI samples in the feature space via stitching the decomposed features. FCL [19] introduces an object fabricator to generate large-scale HOI samples for rare and unseen categories. ATL [18] decouples HOI representation into a combination of affordance and object representation. Novel interactions are then discovered by combining affordance representation and novel object representation from additional object images. ConsNet [34] proposes a knowledge-awand framework for explicit relation modeling among objects, actions, and interactions in undirected graph. It leverages graph attention networks to enable knowledge propagation among HOI categories and their constituents. SCL [20] devises self- colposooooiii learning framework for HOI concept discovery.
Recently, the development of multimodal learning has led to surge of interest transferring knowledge from pre-trained vision-languag models like CLIP [38] to existing HOI detectors for high nigh-performance zero-shot HOI detection. GLK-VLKT [31] extracts CLIP text embeddings for prompted HOI labels to initialize the classifier and leverages the CLIP visual feature to guide the learning of interactive representation. EoID [46] distills the distribution of action probability from CLIP to the HOI model and detects potential action-agnostic interactive human-object pairs by applying an interactive score module combined with a two-stage bipartite matching algorithm. To avoid data-sensitive knowledge distillation and facilitate better transferability, in our approach, the CLIP model is directly adapted as a fine-grad -reiindd classifier for generalizable HOI identification.
As concurrent work, HOICLIP [35] adopts the one-stage design following GEN-VLKT [2) and proposes query-based knowledge retrieval for efficient knowledge transfer from CLIP to HOI detec- tion tasks. During evaluation, it exploits zero-shot CLIP knowledge as training- free enhancement.
3

---

Differently, our CLIP4HOI leverages the two-stage proposal generation strategy to mitigate the overfitting of the method the joint positional distribution human-objec pairs during training.
Preliminary
3.1 Prohlem Formulation
In this section, we introduce the problem formulation of zero-shot HOI detection. Let V (21,"e,"" ... VN. be the set of action verbs and O [01, 02, .N.. be the set of objects that can be interacted with. The conventional suply-sue solutions try to incorporate samples of all feasible verb-object pairs C = {(0,, 0,) |u, V: ) into the training set and assign each sample a discrete label for to learn closed-set classifier. Given the inherent flexibility of verb-object combinations, it impractical collect a complete HOI dataset. Consequently, researchers have pursued the zero-shot HOI detection paradigm as alternative direction, where the detector needs to generalize well to unseen categories Cunseen during the test. Let Vseen C V, Osen C C, and Creen C C Cunseen denote seen verbs, seen objects, and seen categories during training, respec- tively. According to whether verbs and objects in the unseen categories Cunseen exist during training, zero-shot HOI detection can be divided into three settings: (1) Unseen Composition (UC), where for all (14, 03) E Cunsoen we have &4 Vsen and 03 E Owann' (2) Unseen Object (UO), where for all (Us, oj) Cunseon. we have v, E Vsson and 0j & Oscen' (3) Unseen Verb (UV), where for all (vi, 0z) Cunseen, we have it & Vsenn and 0j E Oseen-
3.2 Revisiting DETR
DEtection TRansformer (DETR) [3] is end-to-end object detector that approaches object detection as direct set prediction problem. At the core of DETR is a transformer encoder-decoder architecture, which is responsible for processing the input image and producing set of object queries that correspond to potential objects in the image. DETR uses bipartite matching algorithm that associates the object queries with ground-truth objects during training. Thanks to this query- detection paradigm, we can easily obtain the features corresponding to each detected instance, enabling efficient extraction of pairwise representation for interaction recognition.
Methodology
4.1 Framework Overview
In this section, we give an overview of the proposed framework. As shown in Figure 2, The proposed CLIP4HOI comprises three key components: Human Hlmann-Obeet interactor (HO interactor), HOI decoder, and HOI classifier. Given a single image as input, all humans and objects are first detected using a pre-trained transformer- based detector DETR [3]. Then, the HO interactor takes the decoded query features that correspond to individual human-object pairs as input and derive pairwise HOI tokens through feature interaction and pairwise spatial information injection. After that, with pairwise HOI tokens serving as the input queries, the HOI decoder is engaged to generate pairwise HOI visual features by aggregating the contextual visual clues from the output of the CLIP image encoder. Finally, in the HOI classifier, global and pairwise HOI scores are obtained by comparing the text embeddings of prompted HOI descriptions with the class token from the CLIP image encoder and the decoded pairwise HOI visual features, respectively.
4.2 HO Interactor
Following [51], the detections from DETR are first post-processed with non-maximum suppression and thresholding. The filtered result is denoted as (B, (, C,S), where B E Nax4 X E R NaxCa Ce 1, Nº- Na, and [0, N denote bounding boxes, decoded query features, object classes, and confidence scores, respectively. Na denotes the number of detected instances. Then, we perform feature interaction between instances and traverse all valid human-object pairs to generate pairwise HOI tokens Q. These are done through the Human-Objec Interactor (HO Interactor). Specifically, the HO Interactor borrows the design of the lightweight interaction head UPT [51], which consists of cooperative layer and competitive layer. Given the detected boxes, unary and

---

HO Interactor
diokingg
HOI Decoder
untrella
boxes
DETR
huraams
objects
crding
FRN Block
Self-Attn Block
XN
HOls
Concat
HOI Decoder
(riding, bicycle) (holding, umbrella)
Cross-Attn Block
CLIP Image Encoder
[PREFIX] [verb] [CONIUN] [object]
CLIP Text
22
HOI
decoded DETR query
Encoder
scores
CLIP text class token
pairwise HOI token/feature
Classifier
global
poirwise
image feature/class
Figure 2: The pipeline of the proposed CLIP4HOL CLIP4HOI is developed based on the prevalent DETR [3] with three new components, i.e. HO interactor, HOI decoder, and HOI classifier. Based on the detections from DETR, the HO interactor generates pairwise HOI tokens by feature interaction and spatial information injection. The HOI decoder then aggregates the CLIP image representation to produce pairwise HOI visual features, with the pairwise HOI tokens serving input queries. Finally, in the HOI classifier, global and pairwise HOI scores are obtained by comparing the text embeddings of HOI descriptions with the CLIP image class token and the pairwise HOI visual features, respectively.
pairwise spatial features [50] (box center, width, height, pairwise IOU, relative area, and direction) first extracted and mapped into pairwise positional encodings E R Nax NaxC, Then, decoded query feature E R Naxaa of detected instances are fed into the cooperative layer along with E for feature interaction and pairwise spatial information injection:
X
CoopLayer X, E) E R.Na
(1)
After that, the competitive layer generates pairwise HOI tokens Q for all valid human-object pairs:
idxxvaidd {(u, v)/u + v. "human"),
X Xpuir E R.. where x pair = CAT(X [M] X1o1), (u, v) idx valid
(3)
Epair E IR Nurxx where E
il pair
Elu.vl, (u, v) = idx
(4)
Q CompLayer (X Xglob, Epair) R Nua x 20,
(5)
where idxvanid the index set of valid human-object pairs and has size Npait- Xglob E R LxC, the global visual feature from the backbone of DETR [3]. Please refer to [51] and also the supplementary material for more technical details of the HO interactor.
4.3 HOI Decoder
After obtaining the pairwise HOI tokens, we feed them into the HOI decoder to aggregate contextual features from the CLIP image encoder, yielding pairwise HOI visual features. Specifically, the HOI
decoder is stacked by N; customized transformer layers, each of which contains cross- crtss-attention block, a self-attention block, and feed-forward network (FFN) block. Given the collection of pairwise HOI tokens Q R Nyx x2C, patch level CLIP image features F E R N Noan
and the
global CLIP image class token R IIIC, the pairwise HOI visual features P E M x XC, are computed follows:
Hg Linear(())
(6)
H/- MHH,( LN(F), + H1-1
1 1, ,
(7)
H" (HAA H()) + CAT(L, H(),
l ... Ni
(8)
H, MLP(LN(H(" + H!!!
l 1 Ie...., Ni
(9)
P H
(10)
5

---

where Linear denotes the linear projection Proj (+) 2 + R that maps the pairwise HOI tokens into CLIP's joint feature space, LN, MHCA, MHSA, MLP, and CAT denote layer normalization, multi-heal head cross-attention, multi-head self-attention, multilayer perceptron, and concatenation, respectively. So far, the obtained pairwise HOI visual features P integrate the relative positional information between humans and objects as well as the contextual visual clues from the large-scale pre-trained CLIP model, greatly promoting the subsequent fine-grained HOI recognition.
4.4 HOI Classifier
Given the notable transferability the CLIP model, this study endeavors to adapt it as a fine-grained classifier for transferable HOI detection. Thanks to the shared embedding space of image and text learned by CLIP after large-scale contrastive pre-training, we can perform fine-graing aen-ggrinedd HOI recogni- tion for both seen and unseen HOI categories by transforming their verb-object combinations into textual descriptions and comparing their embeddings with pairwise HOI visual features. Considering that the manual prompt lacks task-specific heuristics and robustness [55], we choose train learn- able prompt for text embedding generation. Specifically, we insert several learnable tokens before the verb and object of each HOI category following [44,55], which we denote as [PREF IX] and [CONJUN], respectively. We denote the obtained text embedding as T E RNNxCo (Na denotes the number of HOI categories). As shown in Figure compute both global and pairwise HOI scores follows:
I' (Inear(I), P' Linear(P),
(11)
T = Norm (T), i= Norm(l'), P Norm PP'),
(12)
S glob Sigmoid /T), Spair = Sigmoid (T /T),
(13)
where Linear denotes the linear projection head (+) RC + Co, Norm denotes L2 normaliza- tion operation, and 7 is the learnable scale factor that rescales the intermediate logits.
The obtained global and pairwise scores Sylsb [0, 1| and Spair [0, M indicate the presence of HOI categories in the whole image and each human-object pair, respectively. Since each human-object pair may correspond multiple HOI categories, adopt the Sigmoid function when computing the HOI scores instead of Sof tmat with mutually exclusive properties. During inference, the HOI scores along with the class priors from the object detector are integrated produce the final classification result for each human-object proposal.
4.5 Training and Inference
Training: For the global HOI score, the training label is the union of all seen HOI categories present in the image. For the pairwise HOI score, we compare each proposal (detected human-object pair) to the seen HOI targets. When the IOU of both the human and the object exceeds certain threshold, we consider the proposal a positive sample for the compared target and assign the corresponding label to it. Note that single proposal may correspond to multiple HOI labels. Considering the scarcity of positive samples, we adopt binary focal loss for training. The final training loss is formulated as the combination of global loss and pairwise loss:
Linal = FocalBCEESSglo Y Yglob) + B FocalBCE(SSpul Y Ypair),
(14)
where Yglob E {0, 10,11120 1]1xN, and Y pair E {0, Mo...... are global and pairwise labels, respectively. is a hyper-parameter that adjusts the weight of pairwise loss.
Inference: Let Stuman E [0, 11 Nprx1 and Subject [0, 1] Npar be the confidence scores out of DETR
for paired humans and objects. During inference, the final HOI scores Sânal [0, 1] Nor X N. are obtained by element-wise multiplication (with broadcasting) Sglobs Spuir Snuman- and Subjeut
Sinal
glob
Spair
S.A. huuman
S. abjectt
(15)
where hyper-parametes X is introduced to suppress overconfident detections following [50, No equal to the number seen/full HOI categories during training/test. Based on the object class priors provided by DETR, scores of irrelevant HOIs in S manually set to O.

---

Table 1: Zero-shot HOI detection results on HICO-DET. UC, UO. and UV denote unseen composition.
unseen object, and unseen verb settings, respectively. RF- and NF- denote rare first and non-rare first.
(a) UC & UO & UV
(b) RF-UC NF-UC
Method
Setting
Full
Seen
Unseen
Method
Setting
Full
Seen
Unseen
Shen et al. [39]
UC
6.26
5.62
VCL
RF-UC 21.43
24.28
10.06
FG [2]
UC
12.26
12.60
10.93
ATL [18]
RE-UC 21.57
24.67
9.18
ConsNet [34]
UC
19.81
20.51
16.99
FCL [19]
RF-UC
22.01
24.23
13.16
EolD [46]
UC
28.91
30.39
23.01
SCL [20]
RF-UC 28.08
30.39
19.07
HOICLIP [35]
UC
32.99
34.85
25.53
RLIP [48]
RF-UC 30.52
33.35
19.19
CLIP4HOI
UC
32.11
33.25
27.71
GEN-VLKT [31]
RF-UC 30.56
32.91
21.36
FCL*
UO 11.43
13.71
EoID [46]
RF-UC 29.52
31.39
22.04
ATL [18]
UO
13.08
14.69
5.05
HOICLIP [35]
RF-UC 32.99
34,85
25.53
FCL [19]
UO 19.87
20.74
15.54
CLIP4HOT
RF-UC 34.08
35.48
28.47
[TL
UO
20.47
21.54
15.11
VCL(17)
NF-UC 18.06
18.52
16.22
GEN-VLKT [31]
UO
25.63
28.92
10.51
ATL [18]
NF-UC 18.67
18.78
18.25
HOICLIP [35]
UO
28.53
30.99
16.20
FCL [191
NF-UC 19.37
19.55
18.66
CLIP4HOI
UO
28.44
30.34
18.92
SCL [20]
NF-UC 24.34
25.00
21.73
CLIP4HOI
UO 32.58
32.73
31.79
RLIP [48]
NF-UC 26.19
27.67
20.27
GEN-VLKTISTI
UV 28.74
30.23
20.96
GEN-VLKT [31]
NF-UC 23.71
23.38
25.05
EoID [46]
UV
29.61
30.73
22.71
BoID [46]
NF-UC 26.69
26.66
26.77
HOICLIP [35]
UV
31.09
32.19
24.30
HOICLIP [35]
NF-UC 27.75
28.10
26.39
CLIP4HOI
UV
30.42
31.14
26.02
CLIP4HO1
NF-UC 28.90
28.26
31.44
Experiments
5.1 Implementation Details
We adopt pre-trained DETR as the object detector, with ResNet-50 [16] serving the backbone. The HOI classifier is built upon the CLIP model, which takes ViT-B/16 [7] as its visual encoder. During training, the parameters of both the DETR and CLIP models are frozen. The HOI decoder has N₁ 6 layers. In each layer, the embedding dimension is 768, the head number of the multi-head attention is 12, and the hidden dimension of the feed-forward network is 3072. The prompt lengths for [PREFIX] and [CONJUN] are 8 and 2, respectively. Following [51], the
À is set to 1 during training and 2.8 during inference. Please refer the supplementary material for the training details.
5.2 Compare with the State-of-the-art Methods
Zero-Shot HOI Detection: In Table 1, we evaluate the generalization capability of our approach on the HICO-DET [4] dataset. Performance is reported under three zero-shot settings.
Under the Unseen Composition (UC) setting. CLIP4HOI surpasses most previous cop-performing methods. Specifically, compared EolD [46], CLIP4HOI exhibits relative mAP improvements of 29.17% (28.47 vs 22.04) and 17.44% (31.44 vs 26.77) on unseen categories under rare first UC and non-rare first UC settings, respectively. Notably, our CLIP4HOI also outperforms the concurrent work HOICLIP [35] by considerable margins in terms of the unseen mAP.
For fair comparison with previous methods, under the Unseen Object (UO) setting, we provide an additional variant of our approach (marked with in Table 1), which only uses the bounding boxes of the detections subsequent inputs. Results show that both versions of our CLIP4HOI exhibit outstanding performance. Specifically, our CLIP4HOI surpasses HOICLIP [35] by a relative mAP improvement of 16.79% (18.92 vs 16.20) on the unseen categories.
Following GEN-VLKT [31], EoID [46], and HOICLIP [35], we report the performance our ap- proach under the Unseen Verb (UV) setting, where the proposed CLIP4HOI also exhibits promising methods, the advan- tages is mainly reflected in unseen categories, e.g., relative mAP improvements of 14.58% (26.02 vs 22.71) and 7.08% (26.02 vy 24.30) compared to EolD [46] and HOICLIP [35], respectively. This demonstrates the superior generalization capability of approach.
results. is worth mentioning that compared to the previous state-of-the

---

Table 2: Fully-supervised HOI detection results on HICO-DET and V-COCO sets.
HICO-DET
V-COCO
Method
Backbone
Default Rare
Known Object
AP
Full
Non-rare
Full
Rare
Non-rare
Male
(NowlPPP5300 [2]]
ResNct-50
23.36
22.47
23.63
26.43
25.01
26.85
60.3
Zou et al, [CVP921] [57]
ResNct-50
23.46
16.91
25.41
26.15
19.24
28.22
HVTR( [23]
ResNet-50
25.10
17.34
27.42
64.4
ATLICVPK21 [18]
ResNet-50
28.53
21.64
30.59
31.18
24.15
33.29
CV [6]
ResNet-50
28.87
24.25
30.25
31.74
27.07
33.14
QPIC(2 [40]
ResNct-50
29.07
21.85
31.23
31.68
24.14
33.93
61.0
FCL(CVPR211 [19]
ResNet-50
29.12
23.67
30.75
31.31
25.62
33.02
GGNetccVRR2 [53]
ResNet-50
29.17
22.13
30.84
33.50
26.67
34.89
[CG/oo [50]
ResNet-50
31.33
24.72
33.31
34.37
27.18
36.52
60.9
UPT [CVPR222 [51]
ResNet-50
31.66
25.94
33.36
35.05
29.27
36.77
64.5
CDNINeortPSaa [49]
ResNet-50
31.78
27.55
33.05
34.53
29.73
35.96
64.4
Iwin/eccv2a [41]
ResNet-50
32.03
27.62
34.14
35.17
28.79
35.91
Liu el al.ccV2222) [32]
ResNet-50
33.51
30.30
34.46
36.28
33.16
37.21
65.2
GEN-VLKT.cvwz2 [31]
5e
33.75
29.25
35.10
36.78
32.75
37.99
64.5
HOICLIP/cvrRa)) [35]
ResNct-50
34.69
31.12
35.74
37.61
34.47
38.54
64.8
CLIP4HOI
ResNet-50
35.33
33.95
35.74
37.19
35.27
37.77
66.3
Table 3: Robustness comparison between our CLIP4HOI and EoID against distribution discrepancy. Performance is evaluated the unseen categories of HICO-DET under the NF-UC setting.
KL divergence
(0,0.3)
(0.3,066)
(0.6,0.9
[0.9,oc)
EoID [46]
26.69
27.50
26.82
18.92
CLIP4HOI
30.97
33.06
34.15
28.23
mAP gain
4.28
5.56
7.33
9.31
Fully-supervised HOI Detection: This work focuses on the development of transferable HOI detector. Nonetheless, we also test the efficacy of our CLIP4HOI for fully-supervised HOI detection, as shown in Table 2. For the HICO-DET [4] dataset, the proposed approach demonstrates remarkable performance, outperforming GEN-VLKT [31] by margin of 1.11 mAP for full categories. It is worth mentioning that the performance improvement mainly comes from the rare categories, where our CLIP4HOI outperforms GEN-VLKT and [32] by margins of 4.70 mAP and 3.65 mAP, respectively, This indicates that the HOI classifier adapted from CLIP can better handle the long-tailed distribution of HOI categories. For the V-COCO [14] dataset, we ignore HOIs defined with no object labels and report the results Scenario As shown Table 2, our CLIP4HOI achieves 66.3 role AP, surpassing GEN-VLK' [31] and [32] by margins of 1.8 mAP and 1.1 mAP, respectively. Compared to the concurrent work HOICLIP [35], our approach also shows advantages on rare categories and performs better under the default setting.
Robustness Distribution Discrepancy: In Table 3, we report the mAP gain of CLIP4HOI over previous perpeffforing EolD [46]. Performance is evaluated on four unseen subsets of the HICO-DET dataset under the NF-UC setting, each of which exhibits varying degrees of positional distribution discrepancy (measured with KL divergence) from its corresponding seen subset. Results show that our CLIP4HOI exhibits better robustness to positional distribution discrepancy. detailed definition of the distribution discrepancy is in the supplementary material.
5.3
Ablation Study
In this section, we conduct several ablative experiments to verify the effectiveness of our CLIP4HOL. Performance is evaluated on HICO-DET under the RF-UC setting.
As shown in Table 4, we first verified the effectiveness of: (1) HO interactor. Without the HO interactor, the overall performance drops by 1.1 mAP, verifying the effectiveness of pairwise feature interaction and spatial information injection. (2) HOI decoder. The HOI decoder is designed to aggregate relevant CLIP visual features for better feature alignment in interaction recognition, without which the performance drops by 3.54 mAP and 4.67 mAP for seen and unseen categories, respectively.
Component Analysis:

---

Table Component analysis of CLIP4HOI on HICO-DET. The performance is evaluated on HICO- DET under the rare first UC (RF-UC) setting.
Configuration
Full
Seen
Unseen
Full version
34.08
35.48
28.47
Global:
w/o HO interactor
32.98
34.36
27.43
w/o HOI decoder
30.31
31.94
23.80
HOI decoder:
w/o CLIP class token
33.50
34.85
28,10
w/ Mid-level CLIP features
32.83
34.40
26.56
HOI classifier:
w/o Global HOI score
33.94
36.06
25,48
w/o Class priors from DETR
33.13
35.24
24.68
Table 5: Ablative experiments for hyper-parameters like (a) HOI decoder layer number, (b) text prompt length, and (c) training loss weight. The manual prompt is set to "A photo of a person [verb] [object]". The performance is evaluated HICO-DET under the rare first UC (RF-UC) setting.
(a) HOT decnder layer number
(h) Text prompt length
(c) Training loss weight
Ni
Full
Seen
Unseen
PREFIX [CONJUN]
Full
Seen
Unseen
g
Full
Seen
Unscen
2
33.45
34.80
28.08
Manual
32.77
34.99
23.92
0.2
33.46
35.06
27.04
33.65
34.96
28.41
33.60
35.11
27.57
0.5
33.59
35.17
27.30
34.08
35.48
28.47
4
33.37
35.15
26.23
1.0
34.08
35.48
28.47
8
33,66
35.32
27.01
8
2
34.08
35.48
28.47
2,0
33.82
35.18
28.37
10
33.94 35.30
28.46
8
4
33.59
34.96
28.13
5.0
33.33
35.06
26.41
Then, we explored the input of the HOI decoder: (1) CLIP class token. In the HOI decoder, the CLIP class token I incorporated as the input of the self-attention block. Without I, the overall performance decreases slightly by 0.58 mAP, which verifies the effectiveness of the CLIP class token for providing holistic representation of input images. (2) Mid-level CLIP features. We also train a variant of CLIP4HOI where each layer of the HOI decoder aggregates the corresponding mid-level features of the CLIP visual encoder layer. Experiments show that aggregating mid-level CLIP visual features results in overall performance drop of 1.25 mAP.
Finally, we discussed the design the HOI classifier: (1) Global HOI score. While the performance of CLIP4HOI on seen categories exhibits minor improvement of 0.58 mAP upon removing the global HOI score, the efficacy of the model to identify novel interactions deteriorates significantly by 2.99 mAP. (2) Class priors from DETR. The removal of the class priors out of DETR caused consistent decrease in performance, particularly drop of 3.79 mAP for unseen categories.
HOI Decoder Layer Number: In Table 5a, we experimented with different numbers of HOI decoder layers. Results show that a layer number of 6 exhibits the best overall performance. Adding more layers does not bring consistently better performance, but higher computational overhead.
Text Prompt Length: As shown in Table 5b, explored the impact of different prompt lengths for text embedding generation. Results show that the best performance is observed when the lengths of [PREFIX] and [CONJUN] prompt "A photo of person [verb] [object] whose performance is much lower than the learnable one, especially for the unseen categories (23.92 28.47 in mAP).
and 2, respectively. We also experimented with the manual
Training Loss Weight: We investigate the impact of varying weights for pairwise loss in Table Sc. Results show that the performance of CLIP4HOI is not substantially influenced by the weight of loss functions. A straightforward summation global and pairwise losses just works well.
5.4 Qualitative Visualization
As shown in Figure 3, we visualize some detection results and their corresponding attention maps in the HOI decoder. We can find that: (1) The HOI decoder can effectively attend to the regions associated with human-object interactions, resulting in accurate aggregation of contextual visual clues from the output of the CLIP image encoder. (2) The obtained HOI detector exhibits remarkable discrimination capability for unseen categories.

---

GT: <blow, cake Prod: <blow// calito
GT: <carry/ / hold throw, sports ball>
Gleewaar tie Predi swear /.,, Bie
GT: -drikkrik held sip, cup- Prod: shold drink with /ilp tup>
Pred: <hold carry/ throw / sports ball-
/laad pet, Pred: chold / / lood -o. horse>
shold/ Irspect, oranges Pred: shold inspect. ... orange>
GT. <holdhold// inapect smel, applex Pred: shold eat smell/ apple>
Figure Visualization of detection results and their corresponding attention maps the HOI decoder. Correctly classified seen and unseen categories are marked in blue and green, respectively. Incorrect recognition results are marked red. Images are sampled from the HICO-DET dataset. More qualitative results can be found in the supplementary material.
Conclusion
This paper presents CLIP4HO1, novel two-stage framework for zero-shot HOI detection, where the generalizable knowledge of the vision-language model CLIP leveraged for novel interaction identification. By disentangling the detection of humans and objects in the proposal generation phase, the proposed approach exhibits strong robustness to the positional distribution discrepancy between seen and unseen HOI categories. To avoid data-sensitive knowledge distillation and facilitate better knowledge transfer for novel human-object interactions, the CLIP model is elaborately adapted into fine- gre-grained classifier for pairwise HOI discrimination. Extensive experiments on two prevalent benchmarks verify the effectiveness of the proposed CLIP4HOI framework.
7 Acknowledgments
This work was supported by NSFC under Contract U20A20183 and 62021001. was also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC.
References
[1] Baldrati, A., Bertini, M., Uricchio, T., Del Bimbo, A.: Effective conditioned and composed
image retrieval combining CLIP-based features. In: CVPR. pp. 21466 -2474 (2022)
[2] Bansal, A., Rambhatla, S.S., Shrivastava, A., Chellappa. R.: Detecting human-object interac- tions via functional generalization. In: AAAL pp- 10460-10469 (2020) 1,
[3] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object
detection with transformers. In: ECCV. pp. 213-229 (2020) 3, 5,
[4] Chao, Y.W., Liu, Y. Liu, X., Zeng. H., Deng, J.: Learning to detect human-object interactions. In: WACV. pp. 381-389 (2018) 2, 7,
[5] Chen, J., Yanai, K.: QAHOI: query-based anchors for human-object interaction detection. arXiv preprint arXiv:21 (2021) 3
[6] Chen, M. Liao, Y.. Liu, S., Chen, Z.. Wang. Qian, C.: Reformulating hoi detection as adaptive set prediction. In: CVPR. pp. 9004-9013 (2021) 8
[7] Dosovitskiy, A., Beyer, I.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani,
M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16
words: Transformers for image recognition at scale. In: ICLR (2021) 7
[8] Du, Y., Wei, F. Zhang, 7., Shi, M., Gao, Y., Li, G.: Learning to prompt for open-vocabulary object detection with langunel model. In: CVPR. pp. 14084-14093 (2022) 3
10

---

[30] Liao, Y., Liu, S., Wang, F., Chen, Y., Qian, C., Feng, J.: PPDM: Parallel point detection and CVPR. pp. 482-490 (2020) [9] Feng, C.. Zhong, Y., Jie, Z., Chu, X., Ren, H., Wei, X., Xie, W., Ma, L.: PromptDet: Towards open-vocabulary detection using uncurated images. In: ECCV. pp. 701-717 (2022) 3
[10] Gao, C., Xu, J., Zou, Y., Huang, J.B.: DRG: Dual relation graph for human-objert interaction
detection. In: ECCV. pp. 696-712 (2020) 3
[11] Gao, C., Zou, Y., Huang. J.B.: iCAN: Instance-centric attention network for human-object interaction detection. In: BMVC (2018)
[12] Gkioxari, G., Girshick, R., Dollár, P., He, K.: Detecting and recognizing human-object interac- tions. In: CVPR. pp. 8359- 8367 (2018) 3
[13] Gu, X., Lin, T.Y., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation. In: ICLR (2022) 3
[14] Gupta, S., Malik, J.: Visual semantic role labeling. arXiv preprint arXiv: (2015) 2,
[15] Gupta, T., Schwing, A., Hoiem, D.: No-frills human-ohjer interaction detection: Factorization, layout encodings, and training techniques. In: ICCV. pp. 9677-9685 (2019) 1,3
[16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770-778 (2016)
[17] Hou, Z., Peng. X., Qiao, Y., Tao, D.: Visual compositional learning for human-object interaction detection. In: ECCV. pp. 584 600 (2020) 1, 3,7
[18] Hou, Z., Yu, B., Qiao. Y., Peng, X., Tao, D.: Affordance transfer learning for human-object
interaction detection. In: CVPR. pp. 495-504 5004 (2021) 3, 7,8
[19] Hou, Z., Yu, B.. Qiao, Y., Peng, X., Tao, D.: Detecting duman-objeed interaction via fabricated compositional learning. In: CVPR. pp. 16646-14655 (2021) 3, 7, 8
[20] Hou, Z., Yu, B.. Tao, D.: Discovering human-object interaction concepts via self-compositional learning. In: ECCV. pp. 461-478 (2022)* 7,7
[21] Iftekhar, A., Chen, H., Kundu, K., Li, X., Tighe, J., Modolo, D.: What to look at and where: Semantic and spatial refined transformer for detecting ouman-object interactions. In: CVPR. pp. 5353-5363 (2022)* 3
[22] Kim, B., Choi, T., Kang, J., Kim, H.J.: UnionDet: Union-level detector towards real-time
human-object interaction detection. In: ECCV. pp. 498-514 (2020) 3
[23] Kim, B., Lee, J., Kang. J., Kim, E.S., Kim, H.J.: HOTR: End-to-end human-object interaction detection with transformers. In: CVPR. pp. 74-83 (2021) 3,
[24] Kim, B., Mun, J., On, K. K.W., Shin, M., Lee, J., Kim, E.S.: MSTR: Multi-scale transformer for
end-to-end human-object interaction detection. In: CVPR. pp. 19578-19587 (2022) 3
[25] Kim, D.J., Sun, X., Choi, J., Lin, S., Kweon, L.S.: Detecting human-object interactions with
action co-occurrence priors. In: ECCV. pp. 718-736 (2020) 3
[26] Li, Y.L.,, Liu, X., Lu, H., Wang. S., Liu, J., Li, J., Lu, C.: Detailed 2d-3d joint representation for human-object interaction. In: CVPR. pp. 10166-10175 (2020) 3
[27] Li, Y.L.,, Liu, X., Wu, X., Li, Y., Lu, C.: Hoi analysis: Integrating and decomposing human- object interaction. In: NeurIPS. pp. 5011-5022 (2020)
[28] Li, Y.L., Zhou, S., Huang, X., Xu, L., Ma, Z., Fang, H.S., Wang, Y., Lu, C.: Transferable
interactiveness knowledge for numan-objec, interaction detection. In: CVPR. pp. 3585-3599 (2019) 3
[29] Li, Z., Zou, C., Zhao, Y., Li, B., Zhong, S.: Improving human-object interaction detection via phrase learning and label composition. In: AAAL pp. 1509-1517 (2022) 3
matching for real-time human-object interaction detection.
[31] Liao, Y., Zhang, A., Lu, M., Wang, Y., Li, X., Liu, S.: GEN-VLKT: Simplify association and
enhance interaction understanding for hoi detection. In: CVPR. pp. 20123-20132 (2022) 1, 2,
7,8
[32] Liu, X., Li, Y.I.,, Wu, X., Tai, Y.W., Lu, C., Tang, C.K.: Interactiveness field human-object interactions. In: CVPR. pp. 20113-20122 (2022) 2,

---

[33] Liu, Y., Chen, Q. Zisserman, A.: Amplifying key cues for human- cojecnt-ojeet--iie objecc-inteeraction detection.
In: ECCV. pp. 248-265 (2020) 3
[34] Liu, Y., Yuan, J., Chen, C.W.: ConsNet: Learning consistency graph for zero-shot human-object
interaction detection. In: ACM MM. pp. 4235-4243 (2020) 1, 7
[35] Ning. S., Qiu, L.,, Liu, Y., He, X.: HOICLIP: Efficient knowledge transfer for hoi detection with
vision-language models. In: CVPR. pp. 23507-235111 (2023) 3, 7, 8
[36] Peyre, J., Laptev, I,, Schmid, C., Sivic, J.: Detecting unseen visual relations using analogies. In:
ICCV. pp. 1981-1999 (2019) 1,3
[37] Qi. S., Wang. W. Jia, B.. Shen, J., Zhu, S.C.: Learning human-object interactions by graph parsing neural networks. In: ECCV. pp. 401-417 (2018) 3
[38] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell,
A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language
supervision. In: ICML.. pp. 8748-8763 (2021) 1,3
[39] Shen, L., Yeung. S., Hoffmman, J.. Mori, G., Fei-Fei, L.: Scaling human-object interaction recognition through zero-shot learning. In: WACV. pp. 1568-1576 (2018)
[40] Tamura, M., Ohashi, H., Yoshinaga, T.: QPIC: Query-based pairwise human-object interaction detection with image-wide contextual information. In: CVPR. pp. 10410-10419 (2021) 3, 8
[41] Tu, D., Min, X., Duan, H., Guo, G., Zhai, G., Shen, W.: Iwin: Human-object interaction
detection via transformer with irregular windows. In: ECCV. pp. 87-103 (2022) 3, 8
[42] Wan, B., Zhou, D., Liu, Y., Li, R., He, X.: Pose-aware multi-level feature network for human object interaction detection. In: ICCV. pp. 9469 9978 (2019) 3
[43] Wang. G., Guo, Y., Wong, Y., Kankanhalli, M.: Chairs can be stood on: Overcoming object bias in human-object interaction detection. In: ECCV. pp. 654-672 (2022) 3
[44] Wang. S., Duan, Y., Ding, H., Tan, Y.P., Yap. K.H., Yuan, J.: Learning transferable human-object
interaction detector with natural language supervision. In: CVPR. pp. 939-948 (2022) 1, 6
[45] Wang, T., Yang, T., Danelljan, M., Khan, F.S., Zhang, X., Sun, J.: Learning human-object
interaction detection using interaction points. In: CVPR. pp. 4116-4125 (2020) 3
[46] Wu, M., Gu, J.. Shen, Y., Lin, M., Chen, C., Sun, X., Ji, R.: End-to-end zero-shot hoi detection
via vision and language knowledge distillation. In: AAAI (2023) 2, 7,8
[47] Xu, B., Wong. Y., Li. J.. Zhao, Q., Kankanhalli, M.S.: Learning to detect human-object interactions with knowledge. In: CVPR. pp. 2019-2028 (2019)
[48] Yuan, H., Jiang, J., Albanie, S., Feng, T., Huang, Z., Ni, D., Tang, M.: RLIP: Relational language-image pre-training for human-object interaction detection. In: NeurlPS (2022) 1,7
[49] Zhang. A., Liao, Y., Liu, S., Lu, M., Wang, Y., Gao. C., Li, X.: Mining the benefits of two-stage and one- hoi detection. In: NeurIPS. pp. 17209-17220 (2021)
[50] Zhang, F.Z., Camphell, D., Gould, S.: Spatially conditioned graphs for detecting human-object interactions. In: ICCV. pp. 13319-13322 (2021) 5, 6, 8
[51] Zhang. FZ.. Campbell, D., Gould, S.: Efficient two-stage detection of human-object interactions
with a novel unary-pairwise transformer. In: CVPR. pp. 20104-201 12 (2022) 4, 5, 6, 7, 8
[52] Zhong, X., Ding, C., Li, 7.., Huang, S.: Towards hard-positive query mining for dese-
human-object interaction detection. In: ECCV. pp. 444-460 (2022) 3
[53] Zhong, X., Qu, X., Ding, C., Tao, D.: Glance and gaze: Inferring action-aware points for
one-stage human-object interaction detection. In: CVPR. pp. 13234-13243 (2021)
[54] Zhou, D., Liu, Z., Wang, J., Wang, L., Hu, T., Ding, E., Wang, J.: Human-object interaction detection via disentangled transformer. In: CVPR. pp. 19568 19687799777 (2022): 3
[55] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. IJCV 130(9), 2337-2348 (2022) 6
[56] Zhou, P., Chi, M.: Relation parsing neural network for human-object interaction detection. In: ICCV. pp. 843-851 (2019)3
[57] Zou, C., Wang, B., Hu, Y., Liu, J., Wa, Q. Zhao, Y., Li, B., Zhang, C., Zhang, C., Wei, Y., et al.: End-to-end human object interaction detection with hoi transformer. In: CVPR. pp. 11825-11834 (2021) 3. 8
12


Reviews:
[{"review_id": "lSTUPGARim", "reviewer": "Reviewer_dBok", "summary": "The paper addresses the problem of Zero-shot Human Object Interaction (HOI) detection, which aims to detect bounding boxes of both seen and unseen interactions.\nTo achieve this, the paper leverages semantic knowledge from pretrained CLIP models to recognize novel combinations of object-verb as well as unseen verbs or objects.\nThe main novelty of the work is to propose a two-stage detection framework that processes all pairs of human-object via a Human Object Interactor to avoid overfitting on interactional positions learned from seen HOI.\nMoreover, the paper introduces a fine-grained HOI classifier to directly use CLIP for detection without the need for knowledge distillation.\nThe paper experiments on HOI datasets of HICO-DET and V-COCO to show its effectiveness.", "questions": "Please refer to the weakness section", "limitations": "Sufficiently addressed", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed idea of using Human-Object Interactor and prompt learning with HOI classifier to directly adapt CLIP model toward the task of human-object interaction is sensible and interesting. This not only helps to significantly simplify the detection pipeline but also improves detection performance.\n+ The paper shows strong improvement in performance compared to SOTA on challenging datasets.\n+ The paper is self-contained and easy to follow.", "weaknesses": "+ The proposed method seems to be mostly effective with novel combinations of verbs and objects, which have been observed in seen interaction during training but becomes less effective for interactions with unseen objects and especially on unseen verbs. Thus, it seems to show that the proposed method is not yet generalizable toward truly novel HOI detection. Can the paper provide justifications for this?\n+ Can the paper explain why removing the Global HOI score achieves the best seen HOI performance in Table 4? Removing Global HOI score supposes to reduce seen class overfit, which should have improved unseen instead of seen class performance.\n+ The reviewer finds it surprising that prompt learning significantly improves unseen HOI detection, given that the prompt is learned with seen classes and is unaware of unseen classes. Can the paper justify how the proposed method can leverage this prompt for unseen classes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the insightful comments and clarify the concerns as follows.\n\n***\n\nQ1: Justification for the generalization ability of the proposed method.\n\nA1: We would like to kindly emphasize the advanced performance of our method that demonstrates the generalization ability. In Table 1, we report the zero-shot performance of our CLIP4HOI under three settings, i.e., unseen composition (UC), unseen object (UO), and unseen verb (UV). For each setting, we report full mAP (including seen HOI and unseen HOI), seen mAP (only including seen HOI), and unseen mAP (only including unseen HOI). It is unseen mAP that reflects the generalization ability of a method towards unseen HOI categories. We would like to clarify our CLIP4HOI achieves the best performance among the compared methods in terms of unseen mAP under all three settings (18.92 mAP vs. 10.51 mAP under the UO setting, 26.02 mAP vs. 22.71 mAP under the UV setting, and 27.71 mAP vs. 23.01 mAP under the UC setting).\n\n***\n\nQ2: Can the paper explain why removing the Global HOI score achieves the best seen HOI performance in Table 4? Removing Global HOI score supposes to reduce seen class overfit, which should have improved unseen instead of seen class performance.\n\nA2: The global HOI score is responsible for perceiving the HOIs existing in the entire receptive field, which provides discriminative knowledge from a global perspective. The pairwise HOI score only focuses on the perception of human-object interactions corresponding to a provided proposal, which is easily affected by the quality of the proposal generation process. In practice, we found that the pairwise HOI score may sometimes have a high response to the HOI categories that do not exist on the image. Such values are usually lower than the score of the seen HOI categories. Therefore, it hardly affects the discrimination of seen HOI categories. However, for the unseen HOI categories that typically exhibit lower HOI scores than the seen HOI categories, the impact of these noises is relatively large. The global HOI score helps mitigate this interference. Therefore, we argue that the degradations in discrimination capability for unseen categories are justifiable after removing the global HOI score.\n\n***\n\nQ3: The reviewer finds it surprising that prompt learning significantly improves unseen HOI detection, given that the prompt is learned with seen classes and is unaware of unseen classes. Can the paper justify how the proposed method can leverage this prompt for unseen classes?\n\nA3: The function of prompt learning is to tune the input of the CLIP text encoder into a sentence pattern suitable for a certain downstream task without finetuning the weights of the whole model, which may cause catastrophic forgetting. The learnable prompt we adopt is not limited to seen HOI categories (category-agnostic). Although only seen verbs and objects are available during training, thanks to the general knowledge owned by the large-scale pre-trained CLIP model, the learned prompt can also be used for unseen HOI categories. Moreover, the primary manual prompt such as \"a photo of a person xxx\" which is commonly used in image recognition was found to be not the optimal solution for the HOI task [3]. Therefore, compared to the manual prompt, the learnable prompt used in our approach leads to significant performance improvements.\n\n***\n\nReferences:\n\n[1] Zhou, Kaiyang, et al. \"Learning to prompt for vision-language models.\" International Journal of Computer Vision 130.9 (2022): 2337-2348.\n\n[2] Zhou, Kaiyang, et al. \"Conditional prompt learning for vision-language models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Wang, Suchen, et al. \"Learning transferable human-object interaction detector with natural language supervision.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n"}, {"review_id": "5In8Wv1woA", "reviewer": "Reviewer_baYx", "summary": "This paper introduces a new framework to leverage CLIP knowledge for zero-shot HOI detection. The paper proposes an HO interactor for pairwise HOI proposal generation to avoid the overfitting issue associated with the joint localization of humans and objects. Instead of using distillation, this paper directly utilizes CLIP knowledge to overcome the absence of unseen categories during training.", "questions": "* Regarding weakness (1), the author should demonstrate the amount of computational complexity introduced by the process of traversing all feasible HO combinations.", "limitations": "* Although the method in this paper is able to avoid the overfitting issue associated with the joint localization of humans and objects, thus improving the ability to detect novel UC, UV, UO HOI pairs, the zero-shot performance may still be limited by the pretrained DETR detector. If the object or human is not detected by the detector, there is no chance that the HOI proposal could be generated by the following modules.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The HOI proposal generation strategy (HO interactor) and the usage of CLIP knowledge (directly using CLIP representation instead of knowledge distillation) are reasonable and effective.\n* The overall performance of HOICLIP surpasses its baseline due to its ability to generate novel HOI proposals and efficiently use CLIP information.\n* The overall pipeline is clear and easy to understand.", "weaknesses": "* The HOI proposal method in this paper (HO interactor) requires traversing all feasible HO combinations for pairwise HOI proposal generation, which may lead to increased computational complexity.\n* The motivation behind the specific design of the HOI Decoder is not clear enough.\n* A discussion of recent works in CVPR2023 (HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models) is expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the insightful comments and clarify the concerns as follows.\n\n***\n\nQ1: The HOI proposal method in this paper (HO interactor) requires traversing all feasible HO combinations for pairwise HOI proposal generation, which may lead to increased computational complexity. The author should demonstrate the amount of computational complexity introduced by the process of traversing all feasible HO combinations.\n\nA1: In our approach, the traversal-based proposal generation is adopted to avoid the problem of over-fitting the joint positional distribution of seen human-object pairs during training. We agree that such a traversal-based strategy may bring non-negligible computational overhead. In practice, we follow [1] to filter out detections with scores lower than 0.2, and sample at least 3 and up to 15 humans and objects each. In extreme cases, up to 435 pairwise HOI proposals may be generated. But in most cases, the number of instances (humans + objects) input to the HO interactor is less than 10. Therefore, the computational overhead of our method remains within an acceptable range.\n\n***\n\nQ2: The motivation behind the specific design of the HOI Decoder is not clear enough.\n\nA2: In this work, the CLIP model is adapted for fine-grained HOI discrimination. To better exploit the joint visual-linguistic space obtained by CLIP pre-training, the visual features of HOI proposals need to be aligned with those of the HOI descriptions, which leads to the introduction of the HOI decoder. In the HOI decoder, given the pairwise HOI tokens produced by the HO interactor, we first perform cross-attention between the tokens and the patch-level CLIP image features to aggregate corresponding CLIP visual features for each HOI proposal. Then, considering that the CLIP image token contains rich global visual cues and is directly aligned with the CLIP text class token during pre-training, we incorporate it in the subsequent self-attention operation for better feature alignment.\n\n***\n\nQ3: A discussion of recent works in CVPR2023 (HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models) is expected.\n\nA3: Thanks for your valuable suggestion. As a common concern with Reviewer Rad1, we put the answer in the global response letter.\n\n***\n\nQ4: The zero-shot performance may still be limited by the pre-trained DETR detector. If the object or human is not detected by the detector, there is no chance that the HOI proposal could be generated by the following modules.\n\nA4: Thanks for the insightful concern. We agree it should be a common problem of to-date two-stage zero-shot HOI detectors. We also discuss this limitation of our method in Section B of the supplementary material. Fortunately, given the two-stage design of our CLIP4HOI, a promising solution is to integrate advanced open-vocabulary object detectors like OV-DETR [2] into our CLIP4HOI, as discussed in the supplementary material.\n\n***\n\nReferences:\n\n[1] Zhang, Frederic Z., Dylan Campbell, and Stephen Gould. \"Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Zang, Yuhang, et al. \"Open-vocabulary DETR with conditional matching.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n"}, {"review_id": "3RQAm2K3AB", "reviewer": "Reviewer_2e3Y", "summary": "This paper proposes a two-stage zero-shot HOI detection paradigm which uses information information from large-scale vision and language models like CLIP. The proposed approach, called CLIP4HOI, first extracts all feasible human-object pairs and generates pairwise proposals. In the second stage, CLIP4HOI uses the CLIP model to classify the pairwise proposals. The authors demonstrate the utility of the proposed approach through experiments on two datasets. ", "questions": "1. The authors claim that prior models over-fit to the join positional distribution of seen human-object pairs. However, I didn't see any convincing evidence to show if this actually happens and if it happens, is it actually a problem. The authors should first show why and how this is a problem. Without a clear demonstration, the motivation behind the proposed approach is not clear.\n\n2. Related to the above, the authors claim that the distribution discrepancy leads to compromised efficacy of the model when encountering novel HOIs. Have the authors actually tested the performance in such cases? It seems figure 1c is trying to do it, but the figure is not clear there's no clear description of the figure (what is the x-axis? what is the class?). The authors should take all classes which demonstrate such distribution dicrepancy and show if the performance is actually compromised. \n\n3. It's not clear how the proposal generation is different from prior works. Prior works also generate all human-object pairs as proposals. How is the proposed approach different from such works?\n\n4. In lines 169-170, the authors talk about \"feature interaction\". What exactly is feature interaction? \n\n5. Similarly, in several places, the authors talk about \"valid\" human-object pairs. This is not defined anywhere. What are valid human-object pairs? Are the authors removing any pairs using some heuristics? \n\n6. What is the CoopLayer? What is its form?", "limitations": "The authors have not discussed the limitations of their proposed approach. I would recommend that the authors include such a discussion. They can talk about the need to use large-scale vision-language models which require large amount of resources for training. They can also talk about the need for separate object detectors (DETR) which make the overall architecture clunky - there might be ways of reducing the dependence on a large number of additional models.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Utilizing the information contained in large-scale vision-language models for different tasks is the way to best use such models. The authors have taken inspiration from recent works along these lines and cleverly utilized the CLIP model for HOI proposal classification.", "weaknesses": "The motivation for the proposed approach and the differences with prior works are not clear. The paper contains several un-explained terms and claims. The authors should clearly address the questions below in a revision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the insightful comments and clarify the concerns as follows.\n\n***\n\nQ1: The authors claim that prior models over-fit to the join positional distribution of seen human-object pairs. However, I didn't see any convincing evidence to show if this actually happens and if it happens, is it actually a problem. The authors should first show why and how this is a problem. Without a clear demonstration, the motivation behind the proposed approach is not clear.\n\nA1: Thanks for raising this insight concern. We would like to kindly clarify we have discussed this problem in the Section of ``Introduction'' and showed the evidence in Figure 1b. Besides, some more examples are also included in Figure 2 of the supplementary material. \n\nIn the literature, previous top-performing methods generally follow the query-based one-stage paradigm, whose query is expected to be used for simultaneously localizing both the human and the object. This kind of prediction paradigm works well under the i.i.d assumption. However, in the zero-shot HOI detection, the joint positional distribution of HOIs is totally different between the training stage and the test stage, making it an o.o.d problem. Therefore, such one-stage methods bear the inherent risk of overfitting to the joint positional distribution of human-object pairs for seen HOI categories during training, which results in compromised efficacy of the model when encountering novel HOIs that exhibit significant distribution discrepancy with the seen categories. We have compared our proposed approach with the previous one-stage method in Figure 1c and Table 2. Results show that our CLIP4HOI exhibits better robustness to positional distribution discrepancy and superior generalization capability to unseen HOI categories.\n\n***\n\nQ2: Related to the above, the authors claim that the distribution discrepancy leads to compromised efficacy of the model when encountering novel HOIs. Have the authors actually tested the performance in such cases? It seems Figure 1c is trying to do it, but the figure is not clear there's no clear description of the figure (what is the x-axis? what is the class?). The authors should take all classes which demonstrate such distribution discrepancy and show if the performance is actually compromised.\n\nA2: The definition of the x-axis of Figure 1c is the distribution discrepancy between seen and unseen HOI categories. As stated in the caption of Figure. 1, the distribution statistics are the angles (quantized into 90 discrete bins) between the line from the person to the object and the x-axis. The discrepancy is measured with KL divergence. A more detailed explanation can be found in Section 5.2 “Robustness to Distribution Discrepancy” and also in Section A.2 in the supplementary material.\nWe indeed take all classes into consideration. Specifically, we first compute the positional distribution discrepancy between the seen and unseen HOI categories corresponding to each object. Then, according to the degree of distribution discrepancy, we divide the whole test set into seven subsets by object. On each subset, we report the performance improvement of our CLIP4HOI compared to the previous method EoID [1], and thus Figure 1c is obtained. Results show that our CLIP4HOI exhibits better performance when encountering larger positional distribution discrepancies (The mAP gain increased from 2.74 to 9.31 as the distribution discrepancy increased).\n\n***\n\nQ3: It's not clear how the proposal generation is different from prior works. Prior works also generate all human-object pairs as proposals. How is the proposed approach different from such works?\n\nA3: In fact, instead of designing a new HOI proposal generation strategy as the main contribution of this paper, we try to use a traversal-based prior-agnostic HOI proposal generation strategy in a two-stage framework to alleviate the detector from overfitting the joint positional distribution of human-object pairs. As stated in Section 4.2, we borrow the design of the lightweight interaction head in UPT [2] to realize the proposal generation process.\n\n***\n\nQ4: In lines 169-170, the authors talk about \"feature interaction\". What exactly is feature interaction?\n\nA4: Feature interaction is done with the HO interactor, which consists of a cooperative layer and a competitive layer. The technical details of these two layers are included in the supplementary material. In a nutshell, the cooperative layer operates on the features of individual human and object instances, while the competitive layer operates on the features of human–object pairs. Both of these two layers are based on the self-attention mechanism (with pairwise positional encoding).\n\n***\n\nQ5: Similarly, in several places, the authors talk about \"valid\" human-object pairs. This is not defined anywhere. What are valid human-object pairs? Are the authors removing any pairs using some heuristics?\n\nA5: The definition of valid human-object pairs can be found in Eq. (2)~Eq. (4). We do not heuristically remove any pairs. We just make sure that the subject must belong to the “human” class.\n\n***\n\nQ6: What is the CoopLayer? What is its form?\n\nA6: As stated in Section 4.2 (line 171~172), the adopted HO Interactor borrows the design of the lightweight interaction head in UPT [2], which consists of a cooperative layer (CoopLayer in Eq. (1)) and a competitive layer (CompLayer in Eq. (5)). For completeness, we have introduced them in detail in Section A.1 of the supplementary material.\n\n***\n\nReferences:\n\n[1] Wu, Mingrui, et al. \"End-to-end zero-shot hoi detection via vision and language knowledge distillation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 3. 2023.\n\n[2] Zhang, Frederic Z., Dylan Campbell, and Stephen Gould. \"Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n"}, {"review_id": "sAGEqDFHaF", "reviewer": "Reviewer_Rad1", "summary": "This paper proposes CLIP4HOI, a two-stage framework for zero-shot HOI detection, where the generalizable knowledge of CLIP is leveraged for interaction classification. To facilitate better transferability and avoid data-sensitive knowledge distillation, CLIP is tuned to a fine-grained HOI classifier. Extensive experiments are conducted on prevalent benchmarks from zero-shot and fully-supervised settings.", "questions": "1.\tMy main concerns and questions lie in the weaknesses. The author should discuss them in detail.\n2.\tThe authors could include more qualitative examples in the supplementary material to illustrate the success and failure cases of their method.\n", "limitations": "The authors addressed the limitations of the proposed method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe motivation of this paper is generally reasonable.\n2.\tThe proposed method is simple and easy to follow.\n3.\tThe experiment results show that the proposed method achieves competitive performance in both fully-supervised and zero-shot settings.\n", "weaknesses": "1.\tThe novelty of this work is marginal. Some key components are reused in existing HOI detection methods, e.g., HOI interactor is UPT[1]; the learnable text prompt in the HOI classifier is [2].\n2.\tThe paper omits some important baselines, such as HOICLIP[3]. Although HOICLIP is a CVPR2023 paper, it was uploaded to Arxiv in Mar. 2023. The authors should provide a detailed discussion and comparison with it.\n3.\tThe paper misses some experimental results, such as Scenario One on Fully-supervised V-COCO test sets.\n\n[1] Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer, CVPR 2022.\n[2]  Learning transferable human-object interaction detector with natural language supervision, CVPR 2022.\n[3]  HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models, CVPR 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the insightful comments and clarify the concerns as follows.\n\n***\n\nQ1: The novelty of this work is marginal. Some key components are reused in existing HOI detection methods.\n\nA1: This work targets to: 1) Alleviate the problem existing in existing top-performing one-stage zero-shot HOI detectors that are prone to overfitting the joint positional distribution of seen human-object pairs during training. 2) Better leverage the general knowledge of the CLIP model for fine-grained HOI discrimination. \nTo this end, we adopt a two-stage paradigm and introduce three modules: HO interactor to generate pairwise proposals, HOI decoder to aggregate related features from CLIP image encoder outputs, and HOI classifier for fine-grained HOI discrimination. \nIn the implementation of the HO interactor and HOI classifier, we follow some designs of UPT [1] (e.g., interaction module) and [2] (e.g., prompt learning). We would like to clarify that we clearly emphasize their contribution and never take these detailed designs as main innovations of ours.\n\n***\n\nQ2: The paper omits some important baselines, such as HOICLIP. Although HOICLIP is a CVPR2023 paper, it was uploaded to Arxiv in Mar. 2023. The authors should provide a detailed discussion and comparison with it.\n\nA2: Thanks for pointing out the missed reference. As a common concern with Reviewer baYx, we attempt to address this concern in the global response.\n\n***\n\nQ3: The paper misses some experimental results, such as Scenario One on Fully-supervised V-COCO test sets.\n\nA3: For fully-supervised testing on the V-COCO dataset, Scenario 1 needs to predict occlusion objects as [0,0,0,0] while Scenario 2 excludes no-object HOI categories during evaluation. This work focuses on the development of a practical zero-shot HOI detector for novel objects, verbs, and object-verb combinations, and thus no special mechanisms are designed to handle occluded objects. Therefore, Scenario 2 better reflects the real fully-supervised performance of the proposed CLIP4HOI on the V-COCO dataset. For completeness, we test the performance of our CLIP4HOI on Scenario 1, and the result is 59.7mAP. This performance is better than that of UPT [1], but lower than that of GEN-VLKT [3], which is reasonable.\n\n***\n\nQ4: The authors could include more qualitative examples in the supplementary material to illustrate the success and failure cases of their method.\n\nA4: Thanks for your valuable suggestion. We have included more qualitative examples in the one-page pdf of the global response.\n\n***\n\nReferences:\n\n[1] Zhang, Frederic Z., Dylan Campbell, and Stephen Gould. \"Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Wang, Suchen, et al. \"Learning transferable human-object interaction detector with natural language supervision.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Liao, Yue, et al. \"GEN-VLKT: Simplify association and enhance interaction understanding for hoi detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n"}]

Author rebuttal:
The following is a common question raised by Reviewer baYx and Rad1.

***

Q1: The paper omits some important baselines, such as HOICLIP. Although HOICLIP is a CVPR2023 paper, it was uploaded to Arxiv in Mar. 2023. The authors should provide a detailed discussion and comparison with it.

A1: Thanks for pointing out this concurrent work. HOICLIP [1] adopts the one-stage design following GEN-VLKT [2] and proposes query-based knowledge retrieval for efficient knowledge transfer from CLIP to HOI detection tasks. In addition, it exploits zero-shot CLIP knowledge as a training-free enhancement during evaluation. Differently, our CLIP4HOI leverages the two-stage proposal generation strategy to mitigate the overfitting of the method to the joint positional distribution of human-object pairs during training. As for the similarities, HOICLIP and our CLIP4HOI both retain the image encoder of CLIP to better exploit the general knowledge learned by large-scale pre-training, although the implementation details differ.
We compare the performance of our CLIP4HOI with HOICLIP under five zero-shot settings in the following table. Results show that: 1) Under UC, UO, and UV settings, our CLIP4HOI performs on par or slightly inferior in terms of seen mAP. 2) Under all five zero-shot settings, our CIP4HOI outperforms HOICLIP in terms of unseen mAP. This demonstrates that our proposed CLIP4HOI exhibits a stronger generalization ability for unseen HOI categories than HOICLIP. We will follow your suggestion to incorporate this discussion and comparison into the revised manuscript.

|  Method   | Setting  | Full | Seen | Unseen |
|  :----  | :----: |  :----:  | :----: |  :----:  |
| HOICLIP  | UC | **32.99** | **34.85** | 25.53 |
| **CLIP4HOI (Ours)**  | UC | 32.11 | 33.25 | **27.71** |
|   |  |  |  |  |
| HOICLIP  | UO | **28.53** | **30.99** | 16.20 |
| **CLIP4HOI (Ours)**  | UO | 28.44 | 30.34 | **18.92** |
|   |  |  |  |  |
| HOICLIP  | UV | **31.09** | **32.19** | 24.30 |
| **CLIP4HOI (Ours)**   | UV | 30.42 | 31.14 | **26.02** |
|   |  |  |  |  |
| HOICLIP  | RF-UC | 32.99 | 32.99 | 25.53 |
| **CLIP4HOI (Ours)**   | RF-UC | **34.08** | **35.48** | **28.47** |
|   |  |  |  |  |
| HOICLIP  | NF-UC | 27.75 | 28.10 | 26.39 |
| **CLIP4HOI (Ours)**  | NF-UC | **28.90** | **28.26** | **31.44** |

*** 

References:

[1] Ning, Shan, et al. "HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Liao, Yue, et al. "GEN-VLKT: Simplify association and enhance interaction understanding for hoi detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.


Meta-review: This submission proposes a method for Human-Object Interactor detection. The submission received one weak accept, two borderline accepts and one reject (2e3Y). After rebuttal, 2e3Y didn’t change his / her rating, but clearly mentioned ‘ If the paper is accepted, I would recommend that the authors provide more conclusive evidence for this being a problem.’ Reviewer 2e3Y mentioned that some concerns remain including motivation of the paper is not clear. The AC carefully read the submission, comments, and rebuttals, the AC agree with the majority of the reviewers. As for the last concern from reviewer 2e3Y of motivation, it’s clear this is an interesting topic as reviewer dBok mentioned ‘the proposed idea of using Human-Object Interactor and prompt learning with HOI classifier to directly adapt CLIP model toward the task of human-object interaction is sensible and interesting. This not only helps to significantly simplify the detection pipeline but also improves detection performance.’ Therefore, the AC suggests accepting this submission.
Decision: Accept (poster)