Title: CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: human-object interaction detection

zero-shot learning

CLIP model adaptatiion
Abstract: Zero-shot Human-Object Interaction (HOI) detection aims to identify both seen and unseen HOI categories. A strong zero-shot HOI detector is supposed to be not only capable of discriminating novel interactions but also robust to positional distribution discrepancy between seen and unseen categories when locating human-object pairs. However, top-performing zero-shot HOI detectors rely on seen and predefined unseen categories to distill knowledge from CLIP and jointly locate human-object pairs without considering the potential positional distribution discrepancy, leading to impaired transferability. In this paper, we introduce CLIP4HOI, a novel framework for zero-shot HOI detection. CLIP4HOI is developed on the vision-language model CLIP and ameliorates the above issues in the following two aspects. First, to avoid the model from overfitting to the joint positional distribution of seen human-object pairs, we seek to tackle the problem of zero-shot HOI detection in a disentangled two-stage paradigm. To be specific, humans and objects are independently identified and all feasible human-object pairs are processed by Human-Object interactor for pairwise proposal generation. Second, to facilitate better transferability, the CLIP model is elaborately adapted into a fine-grained HOI classifier for proposal discrimination, avoiding data-sensitive knowledge distillation. Finally, experiments on prevalent benchmarks show that our CLIP4HOI outperforms previous approaches on both rare and unseen categories, and sets a series of state-of-the-art records under a variety of zero-shot settings.

CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection

Abstract

Zero-shot Human-Object Interaction (HOI) detection aims to identify both seen and unseen HOI categories. A strong zero-shot HOI detector is supposed to be not only capable of discriminating novel interactions but also robust to positional distribution discrepancy between seen and unseen categories when locating humanobject pairs. However, top-performing zero-shot HOI detectors rely on seen and predefined unseen categories to distill knowledge from CLIP and jointly locate human-object pairs without considering the potential positional distribution discrepancy, leading to impaired transferability. In this paper, we introduce CLIP4HOI, a novel framework for zero-shot HOI detection. CLIP4HOI is developed on the vision-language model CLIP and ameliorates the above issues in the following two aspects. First, to avoid the model from overfitting to the joint positional distribution of seen human-object pairs, we seek to tackle the problem of zero-shot HOI detection in a disentangled two-stage paradigm. To be specific, humans and objects are independently identified and all feasible human-object pairs are processed by Human-Object interactor for pairwise proposal generation. Second, to facilitate better transferability, the CLIP model is elaborately adapted into a fine-grained HOI classifier for proposal discrimination, avoiding data-sensitive knowledge distillation. Finally, experiments on prevalent benchmarks show that our CLIP4HOI outperforms previous approaches on both rare and unseen categories, and sets a series of state-of-the-art records under a variety of zero-shot settings.

1 Introduction

Human-Object Interaction (HOI) detection aims at locating paired humans and objects and identifying their interactions. It finds applications in several downstream tasks, including surveillance, robotics, and human-computer interaction. Despite significant progress in recent years, in most previous works, the task of HOI detection has been conventionally restricted to predefined HOI categories. Given the flexibility of verb-object combinations in human-object interaction scenarios, obtaining a comprehensive dataset is laborious and time-consuming. Moreover, the obtained detectors also struggle to generalize to unseen HOI categories, even if both verb and object in the novel category are present (but never composed together) in the training set. These facts motivate and highlight the exploration of zero-shot HOI detection, which targets to development of transferable HOI detectors.

Latest zero-shot HOI detectors [31, 44, 46, 48] seek to harness the general visual and linguistic knowledge of CLIP [38] for identifying novel HOIs. Despite notable advancements over conventional compositional learning approaches [2,15,17-19,34,36], these methods exhibit certain limitations.

CLIP

Detected:

HOI Classifier

box

pairs

HOI

scores

Pairing

HOI text

descriptions

(a) An overview of our CLIP4HOI

180

90

0

90

180

Angle distribution (surfboard)

seen

unseen

(b) Distribution discrepancy

[0.0,0.15)

[0.15,0.3)

[0.3,0.45)

[0.45,0.6)

[0.6,0.75)

[0.75,0.9)

[0.9,inf)

Distribution discrepancy

0

2

4

6

8

10

mAP gain (%)

2.74

4.75

5.21

5.91 6.23

8.61

9.31

(c) CLIP4HOI

vs

EoID

First, the prevailing approaches [31, 46] adhere to the one-stage framework, where each query or pair thereof is required to jointly localize the human and the object. This strategy bears the inherent risk of overfitting the decoder to the joint positional distribution of human-object pairs for seen HOI categories, which results in compromised efficacy of the model when encountering novel HOIs that exhibit significant distribution discrepancy with the seen categories, as shown in Figure 1b. Second, the knowledge of CLIP is transferred to the detector through knowledge distillation, the effectiveness of which largely depends on the completeness of the training data. However, given the absence of unseen categories during training, the distillation process is dominated by samples from seen categories, resulting in impaired generalization of the model to unseen categories. Although the latest approach [46] tries to mine potential unseen categories during training, the text descriptions of unseen categories are required in advance. As a result, the generalization capability of the detector is restricted to predefined unseen HOI categories, which is not conducive to practical applications.

To this end, we propose CLIP4HOI, a novel framework for zero-shot generalizable HOI detection. On the one hand, in view of the overfitting issue associated with the joint localization of humans and objects, we seek to tackle the problem of zero-shot HOI detection in a two-stage paradigm, as shown in Figure 1a. Specifically, with the help of an off-the-shelf object detector, the first stage focuses on the accurate identification of humans and objects from images, followed by traversing all feasible humanobject combinations in the Human-Object (HO) interactor for pairwise HOI proposal generation. On the other hand, instead of employing data-sensitive knowledge distillation from CLIP to existing detectors, we demonstrate that the adapted CLIP is a natural fit for generalizable fine-grained HOI discrimination. Specifically, based on the generated pairwise proposals, a transformer-based HOI decoder is designed to aggregate contextual visual clues from the output of the CLIP image encoder. HOI scores are then obtained in the HOI classifier by comparing the text embeddings of prompted HOI descriptions with the visual features from different granularities. The final scores for all humanobject pairs are the integration of HOI scores and the object class priors out of the object detector. Thanks to the distribution-agnostic pairwise proposal generation and the strong transferability of CLIP, the resulting HOI detector exhibits better robustness to positional distribution discrepancy and superior generalization capability to unseen HOI categories, as shown in Figure 1c.

To verify the effectiveness, we conducted extensive experiments on two prevalent HOI detection benchmarks, i.e. , HICO-DET [4] and V-COCO [14]. Results show that our approach exhibits superior performance under a variety of zero-shot settings. Specifically, compared to previous state-of-the-art methods GEN-VLKT [31] and EoID [46], our CLIP4HOI achieves absolute mAP gains of 6.43, 4.67, and 8.41 on the unseen categories of RF-UC, NF-UC, and UO settings, respectively. Moreover, our CLIP4HOI also demonstrates exceptional performance under the fully supervised setting, particularly in rare categories where it outperforms Liu et al. [32] by 3.65 mAP on the HICO-DET dataset.

Overall, we make the following three-fold contributions:

Given the compromised generalization capability of the joint human-object localization, we propose a novel two-stage framework, termed CLIP4HOI, for zero-shot HOI detection, which leverages the generalizable knowledge of CLIP for unseen interaction identification.

Instead of relying on data-sensitive knowledge distillation, we demonstrate that by carefully designing the adaptation modules, the CLIP model itself exhibits a strong capability for finegrained HOI discrimination, enabling the unrivaled zero-shot transferability of our CLIP4HOI.

We conduct extensive experiments on two prevalent benchmarks to verify the effectiveness of the proposed approach. Compared to previous methods, our CLIP4HOI exhibits remarkable performance and sets a series of state-of-the-art records.

2 Related Work

Human-Object Interaction Detection: There are two mainstream solutions for HOI detection, one-stage and two-stage. Two-stage methods [4, 10-12, 15, 25, 26, 28, 29, 33, 37, 42, 47, 56] first detect human and object instances in the image with an off-the-shelf object detector. Then relation modeling strategies like multi-steam fusion [4] and graph reasoning [10] are designed to identify interactions between each pair of detected humans and objects. One-stage methods, on the other hand, undertake detection, association, and classification in a single stage. Early one-stage methods adopt interaction point [30, 45] or union box [22] as anchors to facilitate the joint localization of humans and objects. Recent one-stage approaches [5,6,21,23,24,40,41,43,52,54,57] have been inspired by the query-based object detector [3], wherein a group of learnable queries is employed in the transformer decoder to simultaneously predict ⟨ human,verb,object ⟩ triplets. Despite the promising performance under the fully-supervised setting, we argue that such a joint localization strategy tends to overfit the decoder to the joint position distribution of humans and objects for known verb-object combinations. Therefore in this work, we seek to solve the problem of zero-shot HOI detection in a more generalizable two-stage paradigm.

Contrastive Language-Image Pre-Training: Contrastive Language-Image Pre-Training (CLIP) [38] is a multimodal learning framework that entails joint training of an image encoder and a text encoder on large-scale image-text pairs. While traditional approaches rely exclusively on either visual or textual cues for learning, CLIP utilizes both modalities to acquire a shared representation space that aligns images and their corresponding captions in a common embedding space. The outcome of this joint training is the ability of CLIP to facilitate zero-shot and few-shot learning on a variety of downstream tasks, including image classification [38], object detection [8, 9, 13], and image retrieval [1]. The success of CLIP also opens up new avenues for zero-shot HOI detection.

Zero-shot Human-Object Interaction Detection: Zero-shot human-object interaction (HOI) detection strives to develop a detector that can effectively generalize to HOI categories not encountered during training. It is of practical significance due to the flexibility of verb-object combinations.

Early works [2,15,17-20,34,36] mainly adopt compositional learning for zero-shot HOI detection. VCL [17] decomposes HOI representation into object- and verb-specific features and then composes novel HOI samples in the feature space via stitching the decomposed features. FCL [19] introduces an object fabricator to generate large-scale HOI samples for rare and unseen categories. ATL [18] decouples HOI representation into a combination of affordance and object representation. Novel interactions are then discovered by combining affordance representation and novel object representation from additional object images. ConsNet [34] proposes a knowledge-aware framework for explicit relation modeling among objects, actions, and interactions in an undirected graph. It leverages graph attention networks to enable knowledge propagation among HOI categories and their constituents. SCL [20] devises a self-compositional learning framework for HOI concept discovery.

Recently, the development of multimodal learning has led to a surge of interest in transferring knowledge from pre-trained vision-language models like CLIP [38] to existing HOI detectors for high-performance zero-shot HOI detection. GEN-VLKT [31] extracts CLIP text embeddings for prompted HOI labels to initialize the classifier and leverages the CLIP visual feature to guide the learning of interactive representation. EoID [46] distills the distribution of action probability from CLIP to the HOI model and detects potential action-agnostic interactive human-object pairs by applying an interactive score module combined with a two-stage bipartite matching algorithm. To avoid data-sensitive knowledge distillation and facilitate better transferability, in our approach, the CLIP model is directly adapted as a fine-grained classifier for generalizable HOI identification.

As a concurrent work, HOICLIP [35] adopts the one-stage design following GEN-VLKT [2] and proposes query-based knowledge retrieval for efficient knowledge transfer from CLIP to HOI detection tasks. During evaluation, it exploits zero-shot CLIP knowledge as a training-free enhancement.

Differently, our CLIP4HOI leverages the two-stage proposal generation strategy to mitigate the overfitting of the method to the joint positional distribution of human-object pairs during training.

3 Preliminary

3.1 Problem Formulation

In this section, we introduce the problem formulation of zero-shot HOI detection. Let V = { v 1 , v 2 , · · · , v N v } be the set of action verbs and O = { o 1 , o 2 , · · · , o N o } be the set of objects that can be interacted with. The conventional fully-supervised solutions try to incorporate samples of all feasible verb-object pairs C = { ( v i , o j ) | v i ∈ V ; o j ∈ O } into the training set and assign each sample a discrete label for to learn a closed-set classifier. Given the inherent flexibility of verb-object combinations, it is impractical to collect a complete HOI dataset. Consequently, researchers have pursued the zero-shot HOI detection paradigm as an alternative direction, where the detector needs to generalize well to unseen categories C unseen during the test. Let V seen ⊂ V , O seen ⊂ O , and C seen ⊂ C \ C unseen denote seen verbs, seen objects, and seen categories during training, respectively. According to whether verbs and objects in the unseen categories C unseen exist during training, zero-shot HOI detection can be divided into three settings: (1) Unseen Composition (UC), where for all ( v i , o j ) ∈ C unseen, we have v i ∈ V seen and o j ∈ O seen; (2) Unseen Object (UO), where for all ( v i , o j ) ∈ C unseen, we have v i ∈ V seen and o j / ∈ O seen; (3) Unseen Verb (UV), where for all ( v i , o j ) ∈ C unseen, we have v i / ∈ V seen and o j ∈ O seen .

3.2 Revisiting DETR

DEtection TRansformer (DETR) [3] is an end-to-end object detector that approaches object detection as a direct set prediction problem. At the core of DETR is a transformer encoder-decoder architecture, which is responsible for processing the input image and producing a set of object queries that correspond to potential objects in the image. DETR uses a bipartite matching algorithm that associates the object queries with ground-truth objects during training. Thanks to this query-based detection paradigm, we can easily obtain the features corresponding to each detected instance, enabling efficient extraction of pairwise representation for interaction recognition.

4 Methodology

4.1 Framework Overview

In this section, we give an overview of the proposed framework. As shown in Figure 2, The proposed CLIP4HOI comprises three key components: Human-Object interactor (HO interactor), HOI decoder, and HOI classifier. Given a single image as input, all humans and objects are first detected using a pre-trained transformer-based detector DETR [3]. Then, the HO interactor takes the decoded query features that correspond to individual human-object pairs as input and derive pairwise HOI tokens through feature interaction and pairwise spatial information injection. After that, with pairwise HOI tokens serving as the input queries, the HOI decoder is engaged to generate pairwise HOI visual features by aggregating the contextual visual clues from the output of the CLIP image encoder. Finally, in the HOI classifier, global and pairwise HOI scores are obtained by comparing the text embeddings of prompted HOI descriptions with the class token from the CLIP image encoder and the decoded pairwise HOI visual features, respectively.

4.2 HOInteractor

Following [51], the detections from DETR are first post-processed with non-maximum suppression and thresholding. The filtered result is denoted as { B , X , C , S } , where B ∈ R N d × 4 , X ∈ R N d × C d , C ∈ { 0 , 1 , · · · , N c -1 } N d , and S ∈ [0 , 1] N d denote bounding boxes, decoded query features, object classes, and confidence scores, respectively. N d denotes the number of detected instances. Then, we perform feature interaction between instances and traverse all valid human-object pairs to generate pairwise HOI tokens Q . These are done through the Human-Object Interactor (HO Interactor). Specifically, the HO Interactor borrows the design of the lightweight interaction head in UPT [51], which consists of a cooperative layer and a competitive layer. Given the detected boxes, unary and

CLIP Text

Encoder

CLIP Image

Encoder

boxes

…

…

global

pairwise

DETR

HO Interactor

HOI Decoder

…

=

.94

.87

.92

.02

.09

.91

[PREFIX]

[verb]

[CONJUN]

[object]

(riding, bicycle)

(holding, umbrella)

…

pairwise HOI token/feature

decoded DETR query

CLIP image feature/class token

CLIP text class token

…

Cross-Attn Block

FFN Block

×

𝑁𝑙

HOI Decoder

Concat

×

humans

objects

HOI Classifier

HOI

scores

Self-Attn Block

HOIs

<holding,

umbrella>

<riding,

bicycle>

pairwise spatial features [50] (box center, width, height, pairwise IOU, relative area, and direction) are first extracted and mapped into pairwise positional encodings E ∈ R N d × N d × C r . Then, decoded query feature X ∈ R N d × C d of detected instances are fed into the cooperative layer along with E for feature interaction and pairwise spatial information injection:

After that, the competitive layer generates pairwise HOI tokens Q for all valid human-object pairs:

̸

where idx valid is the index set of valid human-object pairs and has size N pair . X glob ∈ R 1 × C b is the global visual feature from the backbone of DETR [3]. Please refer to [51] and also the supplementary material for more technical details of the HO interactor.

4.3 HOI Decoder

After obtaining the pairwise HOI tokens, we feed them into the HOI decoder to aggregate contextual features from the CLIP image encoder, yielding pairwise HOI visual features. Specifically, the HOI decoder is stacked by N l customized transformer layers, each of which contains a cross-attention block, a self-attention block, and a feed-forward network (FFN) block. Given the collection of pairwise HOI tokens Q ∈ R N pair × 2 C r , patch level CLIP image features F ∈ R N patch × C c , and the global CLIP image class token I ∈ R 1 × C c , the pairwise HOI visual features P ∈ R N pair × C c are computed as follows:

where Linear denotes the linear projection F proj ( · ) : R 2 C r ↦→ R C c that maps the pairwise HOI tokens into CLIP's joint feature space. LN , MHCA , MHSA , MLP , and CAT denote layer normalization, multi-head cross-attention, multi-head self-attention, multilayer perceptron, and concatenation, respectively. So far, the obtained pairwise HOI visual features P integrate the relative positional information between humans and objects as well as the contextual visual clues from the large-scale pre-trained CLIP model, greatly promoting the subsequent fine-grained HOI recognition.

4.4 HOI Classifier

Given the notable transferability of the CLIP model, this study endeavors to adapt it as a fine-grained classifier for transferable HOI detection. Thanks to the shared embedding space of image and text learned by CLIP after large-scale contrastive pre-training, we can perform fine-grained HOI recognition for both seen and unseen HOI categories by transforming their verb-object combinations into textual descriptions and comparing their embeddings with pairwise HOI visual features. Considering that the manual prompt lacks task-specific heuristics and robustness [55], we choose to train a learnable prompt for text embedding generation. Specifically, we insert several learnable tokens before the verb and object of each HOI category following [44,55], which we denote as [ PREFIX ] and [ CONJUN ] , respectively. We denote the obtained text embedding as T ∈ R N c × C o ( N c denotes the number of HOI categories). As shown in Figure 2, we compute both global and pairwise HOI scores as follows:

where Linear denotes the linear projection head F proj ( · ) : R C c ↦→ R C o , Norm denotes L2 normalization operation, and τ is the learnable scale factor that rescales the intermediate logits.

The obtained global and pairwise scores S glob ∈ [0 , 1] 1 × N c and S pair ∈ [0 , 1] N pair × N c indicate the presence of HOI categories in the whole image and each human-object pair, respectively. Since each human-object pair may correspond to multiple HOI categories, we adopt the Sigmoid function when computing the HOI scores instead of Softmax with mutually exclusive properties. During inference, the two HOI scores along with the class priors from the object detector are integrated to produce the final classification result for each human-object proposal.

4.5 Training and Inference

Training: For the global HOI score, the training label is the union of all seen HOI categories present in the image. For the pairwise HOI score, we compare each proposal (detected human-object pair) to the seen HOI targets. When the IOU of both the human and the object exceeds a certain threshold, we consider the proposal a positive sample for the compared target and assign the corresponding label to it. Note that a single proposal may correspond to multiple HOI labels. Considering the scarcity of positive samples, we adopt binary focal loss for training. The final training loss is formulated as the combination of global loss and pairwise loss:

where Y glob ∈ { 0 , 1 } 1 × N c and Y pair ∈ { 0 , 1 } N pair × N c are global and pairwise labels, respectively. β is a hyper-parameter that adjusts the weight of pairwise loss.

Inference: Let S human ∈ [0 , 1] N pair × 1 and S object ∈ [0 , 1] N pair × 1 be the confidence scores out of DETR for paired humans and objects. During inference, the final HOI scores S final ∈ [0 , 1] N pair × N c are obtained by element-wise multiplication (with broadcasting) of S glob , S pair , S human, and S object :

where a hyper-parameter λ is introduced to suppress overconfident detections following [50,51]. N c is equal to the number of seen/full HOI categories during training/test. Based on the object class priors provided by DETR, scores of irrelevant HOIs in S final are manually set to 0.

5 Experiments

5.1 Implementation Details

We adopt pre-trained DETR as the object detector, with ResNet-50 [16] serving as the backbone. The HOI classifier is built upon the CLIP model, which takes ViT-B/16 [7] as its visual encoder. During training, the parameters of both the DETR and CLIP models are frozen. The HOI decoder has N l = 6 layers. In each layer, the embedding dimension is 768, the head number of the multi-head attention is 12, and the hidden dimension of the feed-forward network is 3072. The prompt lengths for [ PREFIX ] and [ CONJUN ] are 8 and 2, respectively. Following [51], the hyper-parameter λ is set to 1 during training and 2.8 during inference. Please refer to the supplementary material for the training details.

5.2 Compare with the State-of-the-art Methods

Zero-Shot HOI Detection: In Table 1, we evaluate the generalization capability of our approach on the HICO-DET [4] dataset. Performance is reported under three zero-shot settings.

Under the Unseen Composition (UC) setting, CLIP4HOI surpasses most previous top-performing methods. Specifically, compared to EoID [46], our CLIP4HOI exhibits relative mAP improvements of 29.17% ( 28.47 vs 22.04) and 17.44% ( 31.44 vs 26.77) on unseen categories under rare first UC and non-rare first UC settings, respectively. Notably, our CLIP4HOI also outperforms the concurrent work HOICLIP [35] by considerable margins in terms of the unseen mAP.

For a fair comparison with previous methods, under the Unseen Object (UO) setting, we provide an additional variant of our approach (marked with * in Table 1), which only uses the bounding boxes of the detections as subsequent inputs. Results show that both versions of our CLIP4HOI exhibit outstanding performance. Specifically, our CLIP4HOI surpasses HOICLIP [35] by a relative mAP improvement of 16.79% ( 18.92 vs 16.20) on the unseen categories.

Following GEN-VLKT [31], EoID [46], and HOICLIP [35], we report the performance of our approach under the Unseen Verb (UV) setting, where the proposed CLIP4HOI also exhibits promising results. It is worth mentioning that compared to the previous state-of-the-art methods, the advantages is mainly reflected in unseen categories, e.g. , relative mAP improvements of 14.58% ( 26.02 vs 22.71) and 7.08% ( 26.02 vs 24.30) compared to EoID [46] and HOICLIP [35], respectively. This demonstrates the superior generalization capability of our approach.

Fully-supervised HOI Detection: This work focuses on the development of a transferable HOI detector. Nonetheless, we also test the efficacy of our CLIP4HOI for fully-supervised HOI detection, as shown in Table 2. For the HICO-DET [4] dataset, the proposed approach demonstrates remarkable performance, outperforming GEN-VLKT [31] by a margin of 1.11 mAP for full categories. It is worth mentioning that the performance improvement mainly comes from the rare categories, where our CLIP4HOI outperforms GEN-VLKT and [32] by margins of 4.70 mAP and 3.65 mAP, respectively. This indicates that the HOI classifier adapted from CLIP can better handle the long-tailed distribution of HOI categories. For the V-COCO [14] dataset, we ignore HOIs defined with no object labels and report the results in Scenario 2. As shown in Table 2, our CLIP4HOI achieves 66.3 role AP, surpassing GEN-VLKT [31] and [32] by margins of 1.8 mAP and 1.1 mAP, respectively. Compared to the concurrent work HOICLIP [35], our approach also shows advantages on rare categories and performs better under the default setting.

Robustness to Distribution Discrepancy: In Table 3, we report the mAP gain of CLIP4HOI over previous top-performing EoID [46]. Performance is evaluated on four unseen subsets of the HICO-DET dataset under the NF-UC setting, each of which exhibits varying degrees of positional distribution discrepancy (measured with KL divergence) from its corresponding seen subset. Results show that our CLIP4HOI exhibits better robustness to positional distribution discrepancy. A detailed definition of the distribution discrepancy is in the supplementary material.

5.3 Ablation Study

In this section, we conduct several ablative experiments to verify the effectiveness of our CLIP4HOI. Performance is evaluated on HICO-DET under the RF-UC setting.

Component Analysis: As shown in Table 4, we first verified the effectiveness of: (1) HO interactor . Without the HO interactor, the overall performance drops by 1.1 mAP, verifying the effectiveness of pairwise feature interaction and spatial information injection. (2) HOI decoder. The HOI decoder is designed to aggregate relevant CLIP visual features for better feature alignment in interaction recognition, without which the performance drops by 3.54 mAP and 4.67 mAP for seen and unseen categories, respectively.

Then, we explored the input of the HOI decoder: (1) CLIP class token . In the HOI decoder, the CLIP class token I is incorporated as the input of the self-attention block. Without I , the overall performance decreases slightly by 0.58 mAP, which verifies the effectiveness of the CLIP class token for providing holistic representation of input images. (2) Mid-level CLIP features. We also train a variant of CLIP4HOI where each layer of the HOI decoder aggregates the corresponding mid-level features of the CLIP visual encoder layer. Experiments show that aggregating mid-level CLIP visual features results in an overall performance drop of 1.25 mAP.

Finally, we discussed the design of the HOI classifier: (1) Global HOI score . While the performance of CLIP4HOI on seen categories exhibits a minor improvement of 0.58 mAP upon removing the global HOI score, the efficacy of the model to identify novel interactions deteriorates significantly by 2.99 mAP. (2) Class priors from DETR . The removal of the class priors out of DETR caused a consistent decrease in performance, particularly a drop of 3.79 mAP for unseen categories.

HOI Decoder Layer Number: In Table 5a, we experimented with different numbers of HOI decoder layers. Results show that a layer number of 6 exhibits the best overall performance. Adding more layers does not bring consistently better performance, but higher computational overhead.

Text Prompt Length: As shown in Table 5b, we explored the impact of different prompt lengths for text embedding generation. Results show that the best performance is observed when the lengths of [ PREFIX ] and [ CONJUN ] are set to 8 and 2, respectively. We also experimented with the manual prompt "A photo of a person [ verb ] [ object ] ", whose performance is much lower than the learnable one, especially for the unseen categories (23.92 vs 28.47 in mAP).

Training Loss Weight: We investigate the impact of varying weights for pairwise loss in Table 5c. Results show that the performance of CLIP4HOI is not substantially influenced by the weight of loss functions. A straightforward summation of global and pairwise losses just works well.

5.4 Qualitative Visualization

As shown in Figure 3, we visualize some detection results and their corresponding attention maps in the HOI decoder. We can find that: (1) The HOI decoder can effectively attend to the regions associated with human-object interactions, resulting in accurate aggregation of contextual visual clues from the output of the CLIP image encoder. (2) The obtained HOI detector exhibits remarkable discrimination capability for unseen categories.

GT: <hold / load / pet, horse> Pred: <hold / pet / load / ..., horse>

6 Conclusion

This paper presents CLIP4HOI, a novel two-stage framework for zero-shot HOI detection, where the generalizable knowledge of the vision-language model CLIP is leveraged for novel interaction identification. By disentangling the detection of humans and objects in the proposal generation phase, the proposed approach exhibits strong robustness to the positional distribution discrepancy between seen and unseen HOI categories. To avoid data-sensitive knowledge distillation and facilitate better knowledge transfer for novel human-object interactions, the CLIP model is elaborately adapted into a fine-grained classifier for pairwise HOI discrimination. Extensive experiments on two prevalent benchmarks verify the effectiveness of the proposed CLIP4HOI framework.

7 Acknowledgments

This work was supported by NSFC under Contract U20A20183 and 62021001. It was also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC.

References

Baldrati, A., Bertini, M., Uricchio, T., Del Bimbo, A.: Effective conditioned and composed image retrieval combining CLIP-based features. In: CVPR. pp. 21466-21474 (2022) 3

Bansal, A., Rambhatla, S.S., Shrivastava, A., Chellappa, R.: Detecting human-object interactions via functional generalization. In: AAAI. pp. 10460-10469 (2020) 1, 3, 7

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: ECCV. pp. 213-229 (2020) 3, 4, 5

Chao, Y.W., Liu, Y., Liu, X., Zeng, H., Deng, J.: Learning to detect human-object interactions. In: WACV. pp. 381-389 (2018) 2, 3, 7, 8

Chen, J., Yanai, K.: QAHOI: query-based anchors for human-object interaction detection. arXiv preprint arXiv:2112.08647 (2021) 3

Chen, M., Liao, Y., Liu, S., Chen, Z., Wang, F., Qian, C.: Reformulating hoi detection as adaptive set prediction. In: CVPR. pp. 9004-9013 (2021) 3, 8

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021) 7

Du, Y., Wei, F., Zhang, Z., Shi, M., Gao, Y., Li, G.: Learning to prompt for open-vocabulary object detection with vision-language model. In: CVPR. pp. 14084-14093 (2022) 3

Feng, C., Zhong, Y., Jie, Z., Chu, X., Ren, H., Wei, X., Xie, W., Ma, L.: PromptDet: Towards open-vocabulary detection using uncurated images. In: ECCV. pp. 701-717 (2022) 3

Gao, C., Xu, J., Zou, Y., Huang, J.B.: DRG: Dual relation graph for human-object interaction detection. In: ECCV. pp. 696-712 (2020) 3

Gao, C., Zou, Y., Huang, J.B.: iCAN: Instance-centric attention network for human-object interaction detection. In: BMVC (2018) 3

Gkioxari, G., Girshick, R., Dollár, P., He, K.: Detecting and recognizing human-object interactions. In: CVPR. pp. 8359-8367 (2018) 3

Gu, X., Lin, T.Y., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation. In: ICLR (2022) 3

Gupta, S., Malik, J.: Visual semantic role labeling. arXiv preprint arXiv:1505.04474 (2015) 2, 8

Gupta, T., Schwing, A., Hoiem, D.: No-frills human-object interaction detection: Factorization, layout encodings, and training techniques. In: ICCV. pp. 9677-9685 (2019) 1, 3

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770-778 (2016) 7

Hou, Z., Peng, X., Qiao, Y., Tao, D.: Visual compositional learning for human-object interaction detection. In: ECCV. pp. 584-600 (2020) 1, 3, 7

Hou, Z., Yu, B., Qiao, Y., Peng, X., Tao, D.: Affordance transfer learning for human-object interaction detection. In: CVPR. pp. 495-504 (2021) 1, 3, 7, 8

Hou, Z., Yu, B., Qiao, Y., Peng, X., Tao, D.: Detecting human-object interaction via fabricated compositional learning. In: CVPR. pp. 14646-14655 (2021) 1, 3, 7, 8

Hou, Z., Yu, B., Tao, D.: Discovering human-object interaction concepts via self-compositional learning. In: ECCV. pp. 461-478 (2022) 3, 7

Iftekhar, A., Chen, H., Kundu, K., Li, X., Tighe, J., Modolo, D.: What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions. In: CVPR. pp. 5353-5363 (2022) 3

Kim, B., Choi, T., Kang, J., Kim, H.J.: UnionDet: Union-level detector towards real-time human-object interaction detection. In: ECCV. pp. 498-514 (2020) 3

Kim, B., Lee, J., Kang, J., Kim, E.S., Kim, H.J.: HOTR: End-to-end human-object interaction detection with transformers. In: CVPR. pp. 74-83 (2021) 3, 8

Kim, B., Mun, J., On, K.W., Shin, M., Lee, J., Kim, E.S.: MSTR: Multi-scale transformer for end-to-end human-object interaction detection. In: CVPR. pp. 19578-19587 (2022) 3

Kim, D.J., Sun, X., Choi, J., Lin, S., Kweon, I.S.: Detecting human-object interactions with action co-occurrence priors. In: ECCV. pp. 718-736 (2020) 3

Li, Y.L., Liu, X., Lu, H., Wang, S., Liu, J., Li, J., Lu, C.: Detailed 2d-3d joint representation for human-object interaction. In: CVPR. pp. 10166-10175 (2020) 3

Li, Y.L., Liu, X., Wu, X., Li, Y., Lu, C.: Hoi analysis: Integrating and decomposing humanobject interaction. In: NeurIPS. pp. 5011-5022 (2020) 8

Li, Y.L., Zhou, S., Huang, X., Xu, L., Ma, Z., Fang, H.S., Wang, Y., Lu, C.: Transferable interactiveness knowledge for human-object interaction detection. In: CVPR. pp. 3585-3594 (2019) 3

Li, Z., Zou, C., Zhao, Y., Li, B., Zhong, S.: Improving human-object interaction detection via phrase learning and label composition. In: AAAI. pp. 1509-1517 (2022) 3

Liao, Y., Liu, S., Wang, F., Chen, Y., Qian, C., Feng, J.: PPDM: Parallel point detection and matching for real-time human-object interaction detection. In: CVPR. pp. 482-490 (2020) 3

Liao, Y., Zhang, A., Lu, M., Wang, Y., Li, X., Liu, S.: GEN-VLKT: Simplify association and enhance interaction understanding for hoi detection. In: CVPR. pp. 20123-20132 (2022) 1, 2, 3, 7, 8

Liu, X., Li, Y.L., Wu, X., Tai, Y .W., Lu, C., Tang, C.K.: Interactiveness field in human-object interactions. In: CVPR. pp. 20113-20122 (2022) 2, 8

Liu, Y., Chen, Q., Zisserman, A.: Amplifying key cues for human-object-interaction detection. In: ECCV. pp. 248-265 (2020) 3

Liu, Y., Yuan, J., Chen, C.W.: ConsNet: Learning consistency graph for zero-shot human-object interaction detection. In: ACM MM. pp. 4235-4243 (2020) 1, 3, 7

Ning, S., Qiu, L., Liu, Y ., He, X.: HOICLIP: Efficient knowledge transfer for hoi detection with vision-language models. In: CVPR. pp. 23507-23517 (2023) 3, 7, 8

Peyre, J., Laptev, I., Schmid, C., Sivic, J.: Detecting unseen visual relations using analogies. In: ICCV. pp. 1981-1990 (2019) 1, 3

Qi, S., Wang, W., Jia, B., Shen, J., Zhu, S.C.: Learning human-object interactions by graph parsing neural networks. In: ECCV. pp. 401-417 (2018) 3

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748-8763 (2021) 1, 3

Shen, L., Yeung, S., Hoffman, J., Mori, G., Fei-Fei, L.: Scaling human-object interaction recognition through zero-shot learning. In: WACV. pp. 1568-1576 (2018) 7

Tamura, M., Ohashi, H., Yoshinaga, T.: QPIC: Query-based pairwise human-object interaction detection with image-wide contextual information. In: CVPR. pp. 10410-10419 (2021) 3, 8

Tu, D., Min, X., Duan, H., Guo, G., Zhai, G., Shen, W.: Iwin: Human-object interaction detection via transformer with irregular windows. In: ECCV. pp. 87-103 (2022) 3, 8

Wan, B., Zhou, D., Liu, Y., Li, R., He, X.: Pose-aware multi-level feature network for human object interaction detection. In: ICCV. pp. 9469-9478 (2019) 3

Wang, G., Guo, Y., Wong, Y., Kankanhalli, M.: Chairs can be stood on: Overcoming object bias in human-object interaction detection. In: ECCV. pp. 654-672 (2022) 3

Wang, S., Duan, Y., Ding, H., Tan, Y.P., Yap, K.H., Yuan, J.: Learning transferable human-object interaction detector with natural language supervision. In: CVPR. pp. 939-948 (2022) 1, 6

Wang, T., Yang, T., Danelljan, M., Khan, F.S., Zhang, X., Sun, J.: Learning human-object interaction detection using interaction points. In: CVPR. pp. 4116-4125 (2020) 3

Wu, M., Gu, J., Shen, Y., Lin, M., Chen, C., Sun, X., Ji, R.: End-to-end zero-shot hoi detection via vision and language knowledge distillation. In: AAAI (2023) 1, 2, 3, 7, 8

Xu, B., Wong, Y., Li, J., Zhao, Q., Kankanhalli, M.S.: Learning to detect human-object interactions with knowledge. In: CVPR. pp. 2019-2028 (2019) 3

Yuan, H., Jiang, J., Albanie, S., Feng, T., Huang, Z., Ni, D., Tang, M.: RLIP: Relational language-image pre-training for human-object interaction detection. In: NeurIPS (2022) 1, 7

Zhang, A., Liao, Y., Liu, S., Lu, M., Wang, Y., Gao, C., Li, X.: Mining the benefits of two-stage and one-stage hoi detection. In: NeurIPS. pp. 17209-17220 (2021) 8

Zhang, F.Z., Campbell, D., Gould, S.: Spatially conditioned graphs for detecting human-object interactions. In: ICCV. pp. 13319-13327 (2021) 5, 6, 8

Zhang, F.Z., Campbell, D., Gould, S.: Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer. In: CVPR. pp. 20104-20112 (2022) 4, 5, 6, 7, 8

Zhong, X., Ding, C., Li, Z., Huang, S.: Towards hard-positive query mining for detr-based human-object interaction detection. In: ECCV. pp. 444-460 (2022) 3

Zhong, X., Qu, X., Ding, C., Tao, D.: Glance and gaze: Inferring action-aware points for one-stage human-object interaction detection. In: CVPR. pp. 13234-13243 (2021) 8

Zhou, D., Liu, Z., Wang, J., Wang, L., Hu, T., Ding, E., Wang, J.: Human-object interaction detection via disentangled transformer. In: CVPR. pp. 19568-19577 (2022) 3

Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. IJCV 130 (9), 2337-2348 (2022) 6

Zhou, P., Chi, M.: Relation parsing neural network for human-object interaction detection. In: ICCV. pp. 843-851 (2019) 3

Zou, C., Wang, B., Hu, Y., Liu, J., Wu, Q., Zhao, Y., Li, B., Zhang, C., Zhang, C., Wei, Y., et al.: End-to-end human object interaction detection with hoi transformer. In: CVPR. pp. 11825-11834 (2021) 3, 8

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CLIP4HOI, a two-stage framework for zero-shot HOI detection, where the generalizable knowledge of CLIP is leveraged for interaction classification. To facilitate better transferability and avoid data-sensitive knowledge distillation, CLIP is tuned to a fine-grained HOI classifier. Extensive experiments are conducted on prevalent benchmarks from zero-shot and fully-supervised settings.\n\nSTRENGTHS:\n1.\tThe motivation of this paper is generally reasonable.\n2.\tThe proposed method is simple and easy to follow.\n3.\tThe experiment results show that the proposed method achieves competitive performance in both fully-supervised and zero-shot settings.\n\nWEAKNESSES:\n1.\tThe novelty of this work is marginal. Some key components are reused in existing HOI detection methods, e.g., HOI interactor is UPT[1]; the learnable text prompt in the HOI classifier is [2].\n2.\tThe paper omits some important baselines, such as HOICLIP[3]. Although HOICLIP is a CVPR2023 paper, it was uploaded to Arxiv in Mar. 2023. The authors should provide a detailed discussion and comparison with it.\n3.\tThe paper misses some experimental results, such as Scenario One on Fully-supervised V-COCO test sets.\n\n[1] Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer, CVPR 2022.\n[2]  Learning transferable human-object interaction detector with natural language supervision, CVPR 2022.\n[3]  HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models, CVPR 2023.\n\nQUESTIONS:\n1.\tMy main concerns and questions lie in the weaknesses. The author should discuss them in detail.\n2.\tThe authors could include more qualitative examples in the supplementary material to illustrate the success and failure cases of their method.\n\nLIMITATIONS:\nThe authors addressed the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-stage zero-shot HOI detection paradigm which uses information information from large-scale vision and language models like CLIP. The proposed approach, called CLIP4HOI, first extracts all feasible human-object pairs and generates pairwise proposals. In the second stage, CLIP4HOI uses the CLIP model to classify the pairwise proposals. The authors demonstrate the utility of the proposed approach through experiments on two datasets.\n\nSTRENGTHS:\nUtilizing the information contained in large-scale vision-language models for different tasks is the way to best use such models. The authors have taken inspiration from recent works along these lines and cleverly utilized the CLIP model for HOI proposal classification.\n\nWEAKNESSES:\nThe motivation for the proposed approach and the differences with prior works are not clear. The paper contains several un-explained terms and claims. The authors should clearly address the questions below in a revision.\n\nQUESTIONS:\n1. The authors claim that prior models over-fit to the join positional distribution of seen human-object pairs. However, I didn't see any convincing evidence to show if this actually happens and if it happens, is it actually a problem. The authors should first show why and how this is a problem. Without a clear demonstration, the motivation behind the proposed approach is not clear.\n\n2. Related to the above, the authors claim that the distribution discrepancy leads to compromised efficacy of the model when encountering novel HOIs. Have the authors actually tested the performance in such cases? It seems figure 1c is trying to do it, but the figure is not clear there's no clear description of the figure (what is the x-axis? what is the class?). The authors should take all classes which demonstrate such distribution dicrepancy and show if the performance is actually compromised. \n\n3. It's not clear how the proposal generation is different from prior works. Prior works also generate all human-object pairs as proposals. How is the proposed approach different from such works?\n\n4. In lines 169-170, the authors talk about \"feature interaction\". What exactly is feature interaction? \n\n5. Similarly, in several places, the authors talk about \"valid\" human-object pairs. This is not defined anywhere. What are valid human-object pairs? Are the authors removing any pairs using some heuristics? \n\n6. What is the CoopLayer? What is its form?\n\nLIMITATIONS:\nThe authors have not discussed the limitations of their proposed approach. I would recommend that the authors include such a discussion. They can talk about the need to use large-scale vision-language models which require large amount of resources for training. They can also talk about the need for separate object detectors (DETR) which make the overall architecture clunky - there might be ways of reducing the dependence on a large number of additional models.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new framework to leverage CLIP knowledge for zero-shot HOI detection. The paper proposes an HO interactor for pairwise HOI proposal generation to avoid the overfitting issue associated with the joint localization of humans and objects. Instead of using distillation, this paper directly utilizes CLIP knowledge to overcome the absence of unseen categories during training.\n\nSTRENGTHS:\n* The HOI proposal generation strategy (HO interactor) and the usage of CLIP knowledge (directly using CLIP representation instead of knowledge distillation) are reasonable and effective.\n* The overall performance of HOICLIP surpasses its baseline due to its ability to generate novel HOI proposals and efficiently use CLIP information.\n* The overall pipeline is clear and easy to understand.\n\nWEAKNESSES:\n* The HOI proposal method in this paper (HO interactor) requires traversing all feasible HO combinations for pairwise HOI proposal generation, which may lead to increased computational complexity.\n* The motivation behind the specific design of the HOI Decoder is not clear enough.\n* A discussion of recent works in CVPR2023 (HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models) is expected.\n\nQUESTIONS:\n* Regarding weakness (1), the author should demonstrate the amount of computational complexity introduced by the process of traversing all feasible HO combinations.\n\nLIMITATIONS:\n* Although the method in this paper is able to avoid the overfitting issue associated with the joint localization of humans and objects, thus improving the ability to detect novel UC, UV, UO HOI pairs, the zero-shot performance may still be limited by the pretrained DETR detector. If the object or human is not detected by the detector, there is no chance that the HOI proposal could be generated by the following modules.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of Zero-shot Human Object Interaction (HOI) detection, which aims to detect bounding boxes of both seen and unseen interactions.\nTo achieve this, the paper leverages semantic knowledge from pretrained CLIP models to recognize novel combinations of object-verb as well as unseen verbs or objects.\nThe main novelty of the work is to propose a two-stage detection framework that processes all pairs of human-object via a Human Object Interactor to avoid overfitting on interactional positions learned from seen HOI.\nMoreover, the paper introduces a fine-grained HOI classifier to directly use CLIP for detection without the need for knowledge distillation.\nThe paper experiments on HOI datasets of HICO-DET and V-COCO to show its effectiveness.\n\nSTRENGTHS:\n+ The proposed idea of using Human-Object Interactor and prompt learning with HOI classifier to directly adapt CLIP model toward the task of human-object interaction is sensible and interesting. This not only helps to significantly simplify the detection pipeline but also improves detection performance.\n+ The paper shows strong improvement in performance compared to SOTA on challenging datasets.\n+ The paper is self-contained and easy to follow.\n\nWEAKNESSES:\n+ The proposed method seems to be mostly effective with novel combinations of verbs and objects, which have been observed in seen interaction during training but becomes less effective for interactions with unseen objects and especially on unseen verbs. Thus, it seems to show that the proposed method is not yet generalizable toward truly novel HOI detection. Can the paper provide justifications for this?\n+ Can the paper explain why removing the Global HOI score achieves the best seen HOI performance in Table 4? Removing Global HOI score supposes to reduce seen class overfit, which should have improved unseen instead of seen class performance.\n+ The reviewer finds it surprising that prompt learning significantly improves unseen HOI detection, given that the prompt is learned with seen classes and is unaware of unseen classes. Can the paper justify how the proposed method can leverage this prompt for unseen classes?\n\nQUESTIONS:\nPlease refer to the weakness section\n\nLIMITATIONS:\nSufficiently addressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}