SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where Linear denotes the linear projection head  $\mathcal{F}_{\mathrm{proj}}(\cdot):\mathbb{R}^{C_c}\mapsto \mathbb{R}^{C_o}$ , Norm denotes L2 normalization operation, and  $\tau$  is the learnable scale factor that rescales the intermediate logits.
The obtained global and pairwise scores  $\mathbf{S}_{\mathrm{glob}} \in [0,1]^{1 \times N_c}$  and  $\mathbf{S}_{\mathrm{pair}} \in [0,1]^{N_{\mathrm{pair}} \times N_c}$  indicate the presence of HOI categories in the whole image and each human-object pair, respectively. Since each human-object pair may correspond to multiple HOI categories, we adopt the Sigmoid function when computing the HOI scores instead of Softmax with mutually exclusive properties. During inference, the two HOI scores along with the class priors from the object detector are integrated to produce the final classification result for each human-object proposal.
4.5 Training and Inference
Training: For the global HOI score, the training label is the union of all seen HOI categories present in the image. For the pairwise HOI score, we compare each proposal (detected human-object pair) to the seen HOI targets. When the IOU of both the human and the object exceeds a certain threshold, we consider the proposal a positive sample for the compared target and assign the corresponding label to it. Note that a single proposal may correspond to multiple HOI labels. Considering the scarcity of positive samples, we adopt binary focal loss for training. The final training loss is formulated as the combination of global loss and pairwise loss:
$$
\mathcal {L} _ {\text {f i n a l}} = \operatorname {F o c a l B C E} \left(\mathbf {S} _ {\text {g l o b}}, \mathbf {Y} _ {\text {g l o b}}\right) + \beta \cdot \operatorname {F o c a l B C E} \left(\mathbf {S} _ {\text {p a i r}}, \mathbf {Y} _ {\text {p a i r}}\right), \tag {14}
$$
where  $\mathbf{Y}_{\mathrm{glob}} \in \{0,1\}^{1 \times N_c}$  and  $\mathbf{Y}_{\mathrm{pair}} \in \{0,1\}^{N_{\mathrm{pair}} \times N_c}$  are global and pairwise labels, respectively.  $\beta$  is a hyper-parameter that adjusts the weight of pairwise loss.
Inference: Let  $\mathbf{S}_{\mathrm{human}} \in [0,1]^{N_{\mathrm{pair}} \times 1}$  and  $\mathbf{S}_{\mathrm{object}} \in [0,1]^{N_{\mathrm{pair}} \times 1}$  be the confidence scores out of DETR for paired humans and objects. During inference, the final HOI scores  $\mathbf{S}_{\mathrm{final}} \in [0,1]^{N_{\mathrm{pair}} \times N_c}$  are obtained by element-wise multiplication (with broadcasting) of  $\mathbf{S}_{\mathrm{glob}}$ ,  $\mathbf{S}_{\mathrm{pair}}$ ,  $\mathbf{S}_{\mathrm{human}}$ , and  $\mathbf{S}_{\mathrm{object}}$ :
$$
\mathbf {S} _ {\text {f i n a l}} = \mathbf {S} _ {\text {g l o b}} \cdot \mathbf {S} _ {\text {p a i r}} \cdot \mathbf {S} _ {\text {h u m a n}} ^ {\lambda} \cdot \mathbf {S} _ {\text {o b j e c t}} ^ {\lambda}, \tag {15}
$$
where a hyper-parameter  $\lambda$  is introduced to suppress overconfident detections following [50, 51].  $N_{c}$  is equal to the number of seen/full HOI categories during training/test. Based on the object class priors provided by DETR, scores of irrelevant HOIs in  $\mathbf{S}_{\mathrm{final}}$  are manually set to 0.
Table 1: Zero-shot HOI detection results on HICO-DET. UC, UO, and UV denote unseen composition, unseen object, and unseen verb settings, respectively. RF- and NF- denote rare first and non-rare first.
Method	Setting	Full	Seen	Unseen
Shen et al. [39]	UC	6.26	-	5.62
FG [2]	UC	12.26	12.60	10.93
ConsNet [34]	UC	19.81	20.51	16.99
EoID [46]	UC	28.91	30.39	23.01
HOICLIP [35]	UC	32.99	34.85	25.53
CLIP4HOI	UC	32.11	33.25	27.71
FCL* [19]	UO	11.43	13.71	0.00
ATL* [18]	UO	13.08	14.69	5.05
FCL [19]	UO	19.87	20.74	15.54
ATL [18]	UO	20.47	21.54	15.11
GEN-VLKT [31]	UO	25.63	28.92	10.51
HOICLIP [35]	UO	28.53	30.99	16.20
CLIP4HOI*	UO	28.44	30.34	18.92
CLIP4HOI	UO	32.58	32.73	31.79
GEN-VLKT [31]	UV	28.74	30.23	20.96
EoID [46]	UV	29.61	30.73	22.71
HOICLIP [35]	UV	31.09	32.19	24.30
CLIP4HOI	UV	30.42	31.14	26.02
(b) RF-UC & NF-UC
Method	Setting	Full	Seen	Unseen
VCL [17]	RF-UC	21.43	24.28	10.06
ATL [18]	RF-UC	21.57	24.67	9.18
FCL [19]	RF-UC	22.01	24.23	13.16
SCL [20]	RF-UC	28.08	30.39	19.07
RLIP [48]	RF-UC	30.52	33.35	19.19
GEN-VLKT [31]	RF-UC	30.56	32.91	21.36
EoID [46]	RF-UC	29.52	31.39	22.04
HOICLIP [35]	RF-UC	32.99	34.85	25.53
CLIP4HOI	RF-UC	34.08	35.48	28.47
VCL [17]	NF-UC	18.06	18.52	16.22
ATL [18]	NF-UC	18.67	18.78	18.25
FCL [19]	NF-UC	19.37	19.55	18.66
SCL [20]	NF-UC	24.34	25.00	21.73
RLIP [48]	NF-UC	26.19	27.67	20.27
GEN-VLKT [31]	NF-UC	23.71	23.38	25.05
EoID [46]	NF-UC	26.69	26.66	26.77
HOICLIP [35]	NF-UC	27.75	28.10	26.39
CLIP4HOI	NF-UC	28.90	28.26	31.44
5.1 Implementation Details
We adopt pre-trained DETR as the object detector, with ResNet-50 [16] serving as the backbone. The HOI classifier is built upon the CLIP model, which takes ViT-B/16 [7] as its visual encoder. During training, the parameters of both the DETR and CLIP models are frozen. The HOI decoder has  $N_{l} = 6$  layers. In each layer, the embedding dimension is 768, the head number of the multi-head attention is 12, and the hidden dimension of the feed-forward network is 3072. The prompt lengths for [PREFIX] and [CONJUN] are 8 and 2, respectively. Following [51], the hyper-parameter  $\lambda$  is set to 1 during training and 2.8 during inference. Please refer to the supplementary material for the training details.
5.2 Compare with the State-of-the-art Methods
Zero-Shot HOI Detection: In Table 1, we evaluate the generalization capability of our approach on the HICO-DET [4] dataset. Performance is reported under three zero-shot settings.
Under the Unseen Composition (UC) setting, CLIP4HOI surpasses most previous top-performing methods. Specifically, compared to EoID [46], our CLIP4HOI exhibits relative mAP improvements of  $29.17\%$  (28.47 vs 22.04) and  $17.44\%$  (31.44 vs 26.77) on unseen categories under rare first UC and non-rare first UC settings, respectively. Notably, our CLIP4HOI also outperforms the concurrent work HOICLIP [35] by considerable margins in terms of the unseen mAP.
For a fair comparison with previous methods, under the Unseen Object (UO) setting, we provide an additional variant of our approach (marked with * in Table 1), which only uses the bounding boxes of the detections as subsequent inputs. Results show that both versions of our CLIP4HOI exhibit outstanding performance. Specifically, our CLIP4HOI surpasses HOICLIP [35] by a relative mAP improvement of  $16.79\%$  (18.92 vs 16.20) on the unseen categories.
Following GEN-VLKT [31], EoID [46], and HOICLIP [35], we report the performance of our approach under the Unseen Verb (UV) setting, where the proposed CLIP4HOI also exhibits promising results. It is worth mentioning that compared to the previous state-of-the-art methods, the advantages is mainly reflected in unseen categories, e.g., relative mAP improvements of  $14.58\%$  (26.02 vs 22.71) and  $7.08\%$  (26.02 vs 24.30) compared to EoID [46] and HOICLIP [35], respectively. This demonstrates the superior generalization capability of our approach.
Table 2: Fully-supervised HOI detection results on HICO-DET and V-COCO test sets.
Method	Backbone	HICO-DET	V-COCO
Default	Known Object	\(AP_{role}^{S2}\)
Full	Rare	Non-rare	Full	Rare	Non-rare
\(IDN_{[NeurIPS20]}\) [27]	ResNet-50	23.36	22.47	23.63	26.43	25.01	26.85	60.3
Zou et al.[CVPR21] [57]	ResNet-50	23.46	16.91	25.41	26.15	19.24	28.22	-
\(HOTR_{[CVPR21]}\) [23]	ResNet-50	25.10	17.34	27.42	-	-	-	64.4
\(ATL_{[CVPR21]}\) [18]	ResNet-50	28.53	21.64	30.59	31.18	24.15	33.29	-
AS-Net\(_{[CVPR21]}\) [6]	ResNet-50	28.87	24.25	30.25	31.74	27.07	33.14	-
\(QPIC_{[CVPR21]}\) [40]	ResNet-50	29.07	21.85	31.23	31.68	24.14	33.93	61.0
\(FCL_{[CVPR21]}\) [19]	ResNet-50	29.12	23.67	30.75	31.31	25.62	33.02	-
\(GGNet_{[CVPR21]}\) [53]	ResNet-50	29.17	22.13	30.84	33.50	26.67	34.89	-
\(SCG_{[ICCVC21]}\) [50]	ResNet-50	31.33	24.72	33.31	34.37	27.18	36.52	60.9
\(UPT_{[CVPR22]}\) [51]	ResNet-50	31.66	25.94	33.36	35.05	29.27	36.77	64.5
\(CDN_{[NeurIPS21]}\) [49]	ResNet-50	31.78	27.55	33.05	34.53	29.73	35.96	64.4
\(Iwin_{[ECCV22]}\) [41]	ResNet-50	32.03	27.62	34.14	35.17	28.79	35.91	-
Liu et al.[CVPR22] [32]	ResNet-50	33.51	30.30	34.46	36.28	33.16	37.21	65.2
GEN-VLKT\(_{[CVPR22]}\) [31]	ResNet-50	33.75	29.25	35.10	36.78	32.75	37.99	64.5
\(HOICLIP_{[CVPR23]}\) [35]	ResNet-50	34.69	31.12	35.74	37.61	34.47	38.54	64.8
CLIP4HOI	ResNet-50	35.33	33.95	35.74	37.19	35.27	37.77	66.3
Table 3: Robustness comparison between our CLIP4HOI and EoID against distribution discrepancy. Performance is evaluated on the unseen categories of HICO-DET under the NF-UC setting.
KL divergence	[0,0.3)	[0.3,0.6)	[0.6,0.9)	[0.9,∞)
EoID [46]	26.69	27.50	26.82	18.92
CLIP4HOI	30.97	33.06	34.15	28.23
mAP gain	4.28	5.56	7.33	9.31
Fully-supervised HOI Detection: This work focuses on the development of a transferable HOI detector. Nonetheless, we also test the efficacy of our CLIP4HOI for fully-supervised HOI detection, as shown in Table 2. For the HICO-DET [4] dataset, the proposed approach demonstrates remarkable performance, outperforming GEN-VLKT [31] by a margin of  $1.11\mathrm{mAP}$  for full categories. It is worth mentioning that the performance improvement mainly comes from the rare categories, where our CLIP4HOI outperforms GEN-VLKT and [32] by margins of  $4.70\mathrm{mAP}$  and  $3.65\mathrm{mAP}$ , respectively. This indicates that the HOI classifier adapted from CLIP can better handle the long-tailed distribution of HOI categories. For the V-COCO [14] dataset, we ignore HOIs defined with no object labels and report the results in Scenario 2. As shown in Table 2, our CLIP4HOI achieves 66.3 role AP, surpassing GEN-VLKT [31] and [32] by margins of  $1.8\mathrm{mAP}$  and  $1.1\mathrm{mAP}$ , respectively. Compared to the concurrent work HOICLIP [35], our approach also shows advantages on rare categories and performs better under the default setting.
Robustness to Distribution Discrepancy: In Table 3, we report the mAP gain of CLIP4HOI over previous top-performing EoID [46]. Performance is evaluated on four unseen subsets of the HICO-DET dataset under the NF-UC setting, each of which exhibits varying degrees of positional distribution discrepancy (measured with KL divergence) from its corresponding seen subset. Results show that our CLIP4HOI exhibits better robustness to positional distribution discrepancy. A detailed definition of the distribution discrepancy is in the supplementary material.
In this section, we conduct several ablative experiments to verify the effectiveness of our CLIP4HOI. Performance is evaluated on HICO-DET under the RF-UC setting.
Component Analysis: As shown in Table 4, we first verified the effectiveness of: (1) HO interactor. Without the HO interactor, the overall performance drops by  $1.1\mathrm{mAP}$ , verifying the effectiveness of pairwise feature interaction and spatial information injection. (2) HOI decoder. The HOI decoder is designed to aggregate relevant CLIP visual features for better feature alignment in interaction recognition, without which the performance drops by  $3.54\mathrm{mAP}$  and  $4.67\mathrm{mAP}$  for seen and unseen categories, respectively.
Table 4: Component analysis of CLIP4HOI on HICO-DET. The performance is evaluated on HICO-DET under the rare first UC (RF-UC) setting.
Configuration	Full	Seen	Unseen
Full version	34.08	35.48	28.47
Global:
w/o HO interactor	32.98	34.36	27.43
w/o HOI decoder	30.31	31.94	23.80
HOI decoder:
w/o CLIP class token	33.50	34.85	28.10
w/ Mid-level CLIP features	32.83	34.40	26.56
HOI classifier:
w/o Global HOI score	33.94	36.06	25.48
w/o Class priors from DETR	33.13	35.24	24.68
Table 5: Ablative experiments for hyper-parameters like (a) HOI decoder layer number, (b) text prompt length, and (c) training loss weight. The manual prompt is set to "A photo of a person [verb] [object]". The performance is evaluated on HICO-DET under the rare first UC (RF-UC) setting.
(a) HOI decoder layer number
Nl	Full	Seen	Unseen
2	33.45	34.80	28.08
4	33.65	34.96	28.41
6	34.08	35.48	28.47
8	33.66	35.32	27.01
10	33.94	35.30	28.46
(b) Text prompt length
[PREFIX]	[CONJUN]	Full	Seen	Unseen
Manual	/	32.77	34.99	23.92
4	2	33.60	35.11	27.57
4	4	33.37	35.15	26.23
8	2	34.08	35.48	28.47
8	4	33.59	34.96	28.13
(c) Training loss weight
β	Full	Seen	Unseen
0.2	33.46	35.06	27.04
0.5	33.59	35.17	27.30
1.0	34.08	35.48	28.47
2.0	33.82	35.18	28.37
5.0	33.33	35.06	26.41
Then, we explored the input of the HOI decoder: (1) CLIP class token. In the HOI decoder, the CLIP class token  $\mathbf{I}$  is incorporated as the input of the self-attention block. Without  $\mathbf{I}$ , the overall performance decreases slightly by  $0.58\mathrm{mAP}$ , which verifies the effectiveness of the CLIP class token for providing holistic representation of input images. (2) Mid-level CLIP features. We also train a variant of CLIP4HOI where each layer of the HOI decoder aggregates the corresponding mid-level features of the CLIP visual encoder layer. Experiments show that aggregating mid-level CLIP visual features results in an overall performance drop of  $1.25\mathrm{mAP}$ .
Finally, we discussed the design of the HOI classifier: (1) Global HOI score. While the performance of CLIP4HOI on seen categories exhibits a minor improvement of  $0.58\mathrm{mAP}$  upon removing the global HOI score, the efficacy of the model to identify novel interactions deteriorates significantly by  $2.99\mathrm{mAP}$ . (2) Class priors from DETR. The removal of the class priors out of DETR caused a consistent decrease in performance, particularly a drop of  $3.79\mathrm{mAP}$  for unseen categories.
HOI Decoder Layer Number: In Table 5a, we experimented with different numbers of HOI decoder layers. Results show that a layer number of 6 exhibits the best overall performance. Adding more layers does not bring consistently better performance, but higher computational overhead.
Text Prompt Length: As shown in Table 5b, we explored the impact of different prompt lengths for text embedding generation. Results show that the best performance is observed when the lengths of [PREFIX] and [CONJUN] are set to 8 and 2, respectively. We also experimented with the manual prompt "A photo of a person [verb] [object]", whose performance is much lower than the learnable one, especially for the unseen categories (23.92 vs 28.47 in mAP).
Training Loss Weight: We investigate the impact of varying weights for pairwise loss in Table 5c. Results show that the performance of CLIP4HOI is not substantially influenced by the weight of loss functions. A straightforward summation of global and pairwise losses just works well.
5.4 Qualitative Visualization
As shown in Figure 3, we visualize some detection results and their corresponding attention maps in the HOI decoder. We can find that: (1) The HOI decoder can effectively attend to the regions associated with human-object interactions, resulting in accurate aggregation of contextual visual clues from the output of the CLIP image encoder. (2) The obtained HOI detector exhibits remarkable discrimination capability for unseen categories.
GT:<blow,cake> Pred:<blow/..,cake>
carry / hold / throw, sports ball>  
hold / carry / throw / ..., sports ball>
GT: <wear, tie> Pred: <wear /..., tie>
GT: <drink with / hold / sip, cup> Pred: <hold / drink with / sip / ..., cup>
GT: <hold / load / pet, horse> Pred: <hold / pet / load ..., horse>
Figure 3: Visualization of detection results and their corresponding attention maps in the HOI decoder. Correctly classified seen and unseen categories are marked in blue and green, respectively. Incorrect recognition results are marked in red. Images are sampled from the HICO-DET dataset. More qualitative results can be found in the supplementary material.
GT: <hold / inspect, orange> Pred: <hold / inspect / ..., orange>
GT: <hold / inspect / smell, apple> Pred: <hold / eat / smell / ..., apple>
This paper presents CLIP4HOI, a novel two-stage framework for zero-shot HOI detection, where the generalizable knowledge of the vision-language model CLIP is leveraged for novel interaction identification. By disentangling the detection of humans and objects in the proposal generation phase, the proposed approach exhibits strong robustness to the positional distribution discrepancy between seen and unseen HOI categories. To avoid data-sensitive knowledge distillation and facilitate better knowledge transfer for novel human-object interactions, the CLIP model is elaborately adapted into a fine-grained classifier for pairwise HOI discrimination. Extensive experiments on two prevalent benchmarks verify the effectiveness of the proposed CLIP4HOI framework.
This work was supported by NSFC under Contract U20A20183 and 62021001. It was also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC.
[1] Baldrati, A., Bertini, M., Uricchio, T., Del Bimbo, A.: Effective conditioned and composed image retrieval combining CLIP-based features. In: CVPR. pp. 21466-21474 (2022) 3
[2] Bansal, A., Rambhatla, S.S., Shrivastava, A., Chellappa, R.: Detecting human-object interactions via functional generalization. In: AAAI. pp. 10460–10469 (2020) 1, 3, 7
[3] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: ECCV. pp. 213-229 (2020) 3, 4, 5
[4] Chao, Y.W., Liu, Y., Liu, X., Zeng, H., Deng, J.: Learning to detect human-object interactions. In: WACV. pp. 381-389 (2018) 2, 3, 7, 8
[5] Chen, J., Yanai, K.: QAHOI: query-based anchors for human-object interaction detection. arXiv preprint arXiv:2112.08647 (2021) 3
[6] Chen, M., Liao, Y., Liu, S., Chen, Z., Wang, F., Qian, C.: Reformulating hoi detection as adaptive set prediction. In: CVPR. pp. 9004–9013 (2021) 3, 8
[7] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021) 7
[8] Du, Y., Wei, F., Zhang, Z., Shi, M., Gao, Y., Li, G.: Learning to prompt for open-vocabulary object detection with vision-language model. In: CVPR. pp. 14084-14093 (2022) 3