SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We evaluate our model on two public benchmarks, HICO-Det [7] and V-COCO [15]. We adopt the BLIP2 [23] model as the VL foundation model in our UniHOI. We map the 768-dimensional features output by Q-Former in BLIP2 to 256 dimensions to adapt to the feature dimension of the visual HOI detector. The number of layers in the HO Prompt-guided Decoder is identical to that in the Interaction Decoder. The output features of these two decoders are concatenated to generate the final category prediction. The model's optimization method, learning rate, training epochs, weight decay, and loss weights, among other hyperparameters, are set in accordance with previous methods to ensure fair comparison. Aside from the ablation studies in Section 4.6 where we employed knowledge retrieval to further enhance model performance, in all other experiments we continue to use annotated phrases for model learning and inference to ensure fair comparison with other methods. All the experiments conducted with a batchsize of 16 on 8 Tesla V100 GPUs. The details of datasets, evaluation metrics, model parameters, computational overhead, etc. will be introduced in Appendix.
4.2 HOI Detection in the Closed World
We compare the performance of our UniHOI with the previous representative and state-of-the-art methods such as ViPLO [37], GEN-VLKT [30], and HOICLIP [35]. The detailed results of HICODET [7] and V-COCO [15] are summarized in Table 1 and 3, respectively. Generally, there are several observations drawn from different aspects: (i) The overall performance of our method is significantly superior to the previous state-of-the-art methods. (ii) Whether the visual feature extractor adopts the ResNet-50 or ResNet-101, our method yields excellent performance, especially a stronger backbone
Figure 3: The training process of UniHOI and previous State-of-the-Art GEN-VLKT [30] on V-COCO dataset.
Table 1: Experimental results on HICO-DET [7]. GEN-VLKT [30] is regarded as baseline.
Method	Backbone	Default Setting	Known Objects Setting
Full	Rare	Non-rare	Full	Rare	Non-rare
Two-stage Methods:
ATL [17]	ResNet-50	23.81	17.43	27.42	27.38	22.09	28.96
VSGNet [42]	ResNet-152	19.80	16.05	20.91	-	-	-
DJ-RN [25]	ResNet-50	21.34	18.53	22.18	23.69	20.64	24.60
VCL [16]	ResNet-50	23.63	17.21	25.55	25.98	19.12	28.03
DRG [12]	ResNet-50-FPN	24.53	19.47	26.04	27.98	23.11	29.43
IDN [26]	ResNet-50	24.58	20.33	25.86	27.89	23.64	29.16
FCL [18]	ResNet-50	25.27	20.57	26.67	27.71	22.34	28.93
SCG [48]	ResNet-50-FPN	29.26	24.61	30.65	32.87	27.89	34.35
UPT [49]	ResNet-50	31.66	25.90	33.36	35.05	29.27	36.77
UPT [49]	ResNet-101	32.31	28.55	33.44	35.65	31.60	36.86
ViPLO-s [37]	ViT-B/32	34.95	33.83	35.28	38.15	36.77	38.56
ViPLO-1 [37]	ViT-B/16	37.22	35.45	37.75	40.61	38.82	41.15
One-stage Methods:
PPDM [29]	Hourglass-104	21.94	13.97	24.32	24.81	17.09	27.12
HOI-Trans [54]	ResNet-101	26.61	19.15	28.84	29.13	20.98	31.57
AS-Net [8]	ResNet-50	28.87	24.25	30.25	31.74	27.07	33.14
QPIC [40]	ResNet-101	29.90	23.92	31.69	32.38	26.06	34.27
SSRT [19]	ResNet-101	31.34	24.31	33.32	-	-	-
CDN-S [47]	ResNet-50	31.44	27.39	32.64	34.09	29.63	35.42
CDN-L [47]	ResNet-101	32.07	27.19	33.53	34.79	29.48	36.38
Liu et al. [31]	ResNet-50	33.51	30.30	34.46	36.28	33.16	37.21
MUREN [22]	ResNet-50	32.87	28.67	34.12	35.52	30.88	36.91
GEN-VLKT-s [30]	ResNet-50	33.75	29.25	35.10	36.78	32.75	37.99
GEN-VLKT-m [30]	ResNet-101	34.78	31.50	35.77	38.07	34.94	39.01
GEN-VLKT-I [30]	ResNet-101	34.96	31.18	36.08	38.22	34.36	39.37
HOICLIP [35]	ResNet-101	34.69	31.12	35.74	37.61	34.47	38.54
Xie et al. (large) [44]	ResNet-101	36.03	33.16	36.89	38.82	35.51	39.81
UniHOI-s (w/ BLIP2)	ResNet-50	40.06	39.91	40.11	42.20	42.60	42.08
(+6.31)	(+10.66)	(+5.01)	(+5.42)	(+9.85)	(+4.09)
UniHOI-m (w/ BLIP2)	ResNet-101	40.74	40.03	40.95	42.96	42.86	42.98
(+5.96)	(+8.53)	(+5.18)	(+4.89)	(+7.92)	(+3.97)
UniHOI-l (w/ BLIP2)	ResNet-101	40.95	40.27	41.32	43.26	43.12	43.25
(+5.99)	(+9.09)	(+5.24)	(+5.04)	(+8.76)	(+3.88)
that can bring further improvements. (iii) As shown in Tables 2, our approach also has a significant performance lead compared to those methods that rely on additional data sets (i.e., Human Pose [32], and Vision-and-Language [45]). Figure 3 shows the training process of UniHOI-s and GEN-VLKT-s on the V-COCO dataset. Figure 4 shows some prediction cases of UniHOI on HICO-DET.
4.3 Comparisons with Methods that Utilize Extra Information
Table 2 presents several methods that utilize extra information (i.e., pose, and language) from additional datasets for HOI detection. Theoretically, the introduction of extra information can indeed further enhance visual models in specific domains. The results demonstrate that the proposed UniHOI still significantly outperforms previous methods.
Figure 4: Detection results of our UniHOI in HICO-DET [7].
Table 2: Comparison results with the methods using extra datasets on HICO-DET dataset [7]. For extra datasets, "P" indicates human pose and "L" indicates linguistic knowledge.
Method	Extras	HICO-DET	V-COCO
Default Setting	Known Objects Setting	AP#1role	AP#2role
Full	Rare	Non-rare	Full	Rare	Non-rare
FCMNet [32]	P+L	20.41	17.34	21.56	22.04	18.97	23.12	53.1	-
PD-Net [52]	L	20.76	15.68	22.28	25.59	19.93	27.28	-	-
DRG [12]	P	24.53	19.47	26.04	27.98	23.11	29.43	51.0	-
ConsNet-F [33]	P	24.39	17.10	26.56	-	-	-	53.2	-
RLIP-ParSeD [45]	L	30.70	24.67	32.50	-	-	-
RLIP-ParSe [45]	L	32.84	26.85	34.63	-	-	-	61.9	64.2
PhraseHOI [28]	L	30.03	23.48	31.99	33.74	27.35	35.64	-	-
OCN (large) [46]	L	31.43	25.80	33.11	-	-	-	-	-
HOICLIP [35]	L	34.69	31.12	35.74	37.61	34.47	38.54	63.5	64.8
UniHOI-s (w/ BLIP2)	L	40.06	39.91	40.11	42.20	42.60	42.08	65.58	68.27
UniHOI-m (w/ BLIP2)	L	40.74	40.03	40.95	42.96	42.86	42.98	67.95	70.61
UniHOI-l (w/ BLIP2)	L	40.95	40.27	41.32	43.26	43.12	43.25	68.05	70.82
4.4 Effectiveness for Zero-Shot HOI Detection
Unseen Composition (UC). As shown in Table 4, we follow the [30] to conduct three kind of zero-shot experiments on HICO-DET. We first evaluate UniHOI-s with UC setting. Compared with GEN-VLKT [30], our UniHOI-s substantially outperforms previous state-of-the-art on both RF-UC and NF-UC. Among them, UniHOI are  $7.32\mathrm{mAP}$  higher than GEN-VLKT on Unseen of RF-UC, and  $9.25\mathrm{mAP}$  higher than GEN-VLKT on Seen of NF-UC. This improvement is mainly due to our efficient prompt learning method for the foundation model, so UniHOI can cope well with some unseen categories. In fact, GEN-VLKT also performs some knowledge transfer from CLIP, but it lags behind UniHOI in the effectiveness of knowledge transfer.
Unseen Object (UO). For UO setting, compared with GEN-VLKT, UniHOI improves Unseen, Seen, and Full by 9.21, 5.84, and 5.93, respectively. This result proves that our method can still understand the interaction between human and novel objects well.
Unseen Verb (UV). On the challenging UV setting, compared to GEN-VLKT, UniHOI improves by 9.21, 5.84 and 5.93 on Unseen, Seen, and Full, respectively. In fact, on the three zero-shot settings mentioned above, the HOI recognition capability of UniHOI have almost surpassed some supervised
Table 3: Experimental results on V-COCO. Table 4: Zero-shot results on HICO-DET [7].
Method	AP#1role	AP#2role
VSGNet [42]	51.8	57.0
IDN [26]	53.3	60.3
UPT [49]	60.7	66.2
VIPLO-s [37]	60.9	66.6
VIPLO-I [37]	62.2	68.0
HOTR [21]	55.2	64.4
QPIC [40]	58.8	61.0
CDN [47]	63.91	65.89
Liu et al. [31]	63.0	65.2
GEN-VLKT-s [30]	62.41	64.46
GEN-VLKT-m [30]	63.28	65.58
GEN-VLKT-I [30]	63.58	65.93
HOICLIP [35]	63.50	64.80
Xie et al. (large) [44]	66.50	69.90
UniHOI-s (w/ BLIP2)	65.58 (+3.17)	68.27 (+3.81)
UniHOI-m (w/ BLIP2)	67.95 (+4.67)	70.61 (+5.03)
UniHOI-l (w/ BLIP2)	68.05 (+4.47)	70.82 (+4.89)
Shen et al. [39]	UC	5.62
FG [3]	UC	10.93
ATL [17]	UC	16.99
VCL [16]	RF-UC	10.06
ATL [17]	RF-UC	9.18
FCL [18]	RF-UC	13.16
GEN-VLKT [30]	RF-UC	21.36
UNIHOI-s (w/ BLIP2)	RF-UC	28.68 (+7.32)
VCL [16]	NF-UC	16.22
ATL [17]	NF-UC	18.25
FCL [18]	NF-UC	18.66
GEN-VLKT [30]	NF-UC	25.05
UniHOI-s (w/ BLIP2)	NF-UC	28.45 (+3.4)
GEN-VLKT [30]	UO	10.51
UniHOI-s (w/ BLIP2)	UO	19.72 (+9.21)
GEN-VLKT [30]	UV	20.96
UniHOI-s (w/ BLIP2)	UV	26.05 (+5.09)
Figure 5: HOI Detection in the Wild. UniHOI is capable of conducting open-category detection based on triplet phrases and also supports the input of descriptive texts retrieved by GPT [36].
Table 5: Ablation Studies on V-COCO [15]. Table 6: Ablation Studies on HICO-DET (UV).
Method	AP#1role	AP#2role	Method	Unseen	Seen	Full
baseline	62.41	64.46	baseline	20.96	30.23	28.74
+ VL Foundation Model	62.91	64.83	+ VL Foundation Model	21.57	31.62	29.85
+ HOPD	65.58	68.27	+ HOPD	26.05	36.78	34.68
+ Knowledge Retrieval	66.74	69.31	+ Knowledge Retrieval	27.41	37.82	35.89
methods in Table 1. This once again attests to the superiority of our UniHOI for open-category HOI detection and its powerful capability to understand interactions.
4.5 HOI Detection in the Wild
In addition to the zero-shot experiments on the HICO dataset, we also conducted open-category testing on open-world. Our UniHOI method supports flexible text input for detecting corresponding interactions. UniHOI can not only recognize category texts directly (e.g., "Juggle orange") but also take a descriptive text input (which could be human-written or knowledge retrieved from GPT). Consequently, interactions in images are identified through text representation. We present these detection results, along with some GPT-generated description texts, in Figure 5. Our test results indicate that open-vocabulary recognition based on interaction description texts is more reliable, and this phenomenon is also in line with our basic intuition. In the future, we will release our pretrained model and code to enable more extensive testing in the open world.
4.6 Ablation Studies
Effect of Foundation Model. We conduct ablation studies on the new components within UniHOI (as shown in Tables 5 and 6). The GEN-VLKT-s is regarded as the baseline, and then incorporated various components for testing on the fully supervised V-COCO dataset, as well as the HICO-DET dataset under a zero-shot setting. Initially, we simply integrated the BLIP2 into our method in a sequential manner. We then used learnable queries to extract interaction features from Q-Former and concatenate them with  $\mathcal{V}^i$ . However, the results did not yield a significant performance improvement.
Impact of HO Spatial Prompting. After realizing the alignment issue with the features of HO-pairs in the previous methods, we introduced the HO Prompt-guided Decoder, as presented in Section 3.3. The results clearly illustrate that the introduction of this component significantly improved all performance metrics, thereby validating the effectiveness of our prompting learning strategy.
Effect of Knowledge Retrieval. To substantiate the importance of utilizing LLM for knowledge retrieval in creating a more universal HOI detector, we also conduct supplementary experiments in this strategy. The results showed that the UniHOI model, when applying knowledge retrieval, achieved further performance improvements. We believe this approach opens new avenues for enhancement in the field of zero-shot HOI detection.
Table 7: The results of UniHOI equipped with different foundation models (i.e., CLIP [38] and BLIP2 [23]) on V-COCO [15].
Method	AP#1role	AP#2role
GEN-VLKT-s [30]	62.41	64.46
GEN-VLKT-m [30]	63.28	65.58
GEN-VLKT-1 [30]	63.58	65.93
UniHOI-s (w/ CLIP)	62.92 (+0.51)	65.67 (+1.21)
UniHOI-m (w/ CLIP)	64.85 (+1.57)	67.62 (+2.04)
UniHOI-l (w/ CLIP)	64.93 (+1.35)	67.86 (+1.93)
UniHOI-s (w/ BLIP2)	65.58 (+3.17)	68.27 (+3.81)
UniHOI-m (w/ BLIP2)	67.95 (+4.67)	70.61 (+5.03)
UniHOI-l (w/ BLIP2)	68.05 (+4.47)	70.82 (+4.89)
Effect of Different Foundation Model. To assess the impact of different Foundation Models on the performance of UniHOI, we also provide a comparative analysis between UniHOI equipped with CLIP and BLIP2. For CLIP, we feed its visual output representation into HOPD. However, given that CLIP's cross-modal contrastive training strategy leans more towards image-level representations [38], its output dimensionality is considerably lower than that of BLIP2, resulting in a less detailed feature representation. Experimental results indicate that both the UniHOI versions with CLIP and BLIP2 as the foundational models showed performance enhancements, with the latter exhibiting a more pronounced improvement. These findings underscore the efficacy and generalization of our approach.
In this work, we propose UniHOI, a universal HOI detection framework. By utilizing large foundation model, spatially prompting the large model, and generalizing the HOI detector to the open world. UniHOI supports open-category interaction relationship recognition from annotation texts or descriptions, which can be flexibly extended to applications in the open world. Extensive experiments on public datasets and diverse settings demonstrate its strong universality - it behaves the strongest open category interaction recognition ability so far. We believe our research will stimulate following research along the universal high-level relationship modeling direction in the future.
[1] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 (2022) 2, 3
[2] Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., Van Den Hengel, A.: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3674-3683 (2018) 3
[3] Bansal, A., Rambhatla, S.S., Shrivastava, A., Chellappa, R.: Detecting human-object interactions via functional generalization. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 10460-10469 (2020) 8
[4] Cao, Y., Su, X., Tang, Q., You, S., Lu, X., Xu, C.: Searching for better spatio-temporal alignment in few-shot action recognition. Advances in Neural Information Processing Systems 35, 21429-21441 (2022) 3
[5] Cao, Y., Tang, Q., Yang, F., Su, X., You, S., Lu, X., Xu, C.: Re-mine, learn and reason: Exploring the cross-modal semantic correlations for language-guided hoi detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 23492-23503 (2023) 2
[6] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European conference on computer vision. pp. 213-229. Springer (2020) 4, 5, 15
[7] Chao, Y.W., Liu, Y., Liu, X., Zeng, H., Deng, J.: Learning to detect human-object interactions. In: 2018 IEEE winter conference on applications of computer vision (wacv). pp. 381-389. IEEE (2018) 1, 3, 6, 7, 8, 16
[8] Chen, M., Liao, Y., Liu, S., Chen, Z., Wang, F., Qian, C.: Reformulating hoi detection as adaptive set prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9004–9013 (2021) 1, 3, 7
[53] Zhong, X., Qu, X., Ding, C., Tao, D.: Glance and gaze: Inferring action-aware points for one-stage human-object interaction detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13234-13243 (2021) 3
[54] Zou, C., Wang, B., Hu, Y., Liu, J., Wu, Q., Zhao, Y., Li, B., Zhang, C., Zhang, C., Wei, Y., et al.: End-to-end human object interaction detection with hoi transformer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11825-11834 (2021) 7