Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models
Authors: Yichao Cao
Qingfei Tang
Xiu Su
Song Chen
Shan You
Xiaobo Lu
Chang Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Human-object interaction
Commonsense Knowledge
Foundation Models

Abstract: Human-object interaction (HOI) detection aims to comprehend the intricate relationships between humans and objects, predicting <human, action, object> triplets, and serving as the foundation for numerous computer vision tasks. The complexity and diversity of human-object interactions in the real world, however, pose significant challenges for both annotation and recognition, particularly in recognizing interactions within an open world context. This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs). The proposed method is dubbed as UniHOI. We conduct a deep analysis of the three hierarchical features inherent in visual HOI detectors and propose a method for high-level relation extraction aimed at VL foundation models, which we call HO prompt-based learning. Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. Furthermore, we utilize a LLM (i.e. GPT) for interaction interpretation, generating a richer linguistic understanding for complex HOIs. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence.  Our efficient architecture design and learning methods effectively unleash the potential of the VL foundation models and LLMs, allowing UniHOI to surpass all existing methods with a substantial margin, under both supervised and zero-shot settings. The code and pre-trained weights will be made publicly available.

Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models
Yichao Cao Southeast University
Qingfei Tang Nanjing Enbo Tech.
Xiu Su* University of Sydney
Chen Song Nanjing Enbo Tech.
Shan You
Xiaobo Lu Southeast University
Chang Xu University of Sydney
SenseTime
Abstract
Human-object interaction (HOI) detection aims to comprehend the intricate relation- ships between humans and objects, predicting < luuman, action, object > triplets, and serving as the foundation for numerous computer vision tasks. The complexity and diversity of human-objee interactions in the real world, however, pose signif- icant challenges for both annotation and recognition, particularly in recognizing interactions within open world context. This study explores the universal inter- action recognition in an open- world setting through the use of LsiiunLanguage (VL. foundation models and large language models (LLMs). The proposed method is dubbed as UniHOI. We conduct a deep analysis of the three hierarchical features inherent in visual HOI detectors and propose method for high-level relation ex- traction aimed at VL foundation models, which we call HO prompt-based learning, Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the associa- tion of high-level relation representations in the foundation model with various HO pairs within the image. Furthermore, we utilize a LLM (i.e. GPT) for interaction interpretation, generating a richer linguistic understanding for complex HOIs. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence. Our efficient architecture design and learning methods effectively unleash the potential of the VL foundation models and LLMs, allowing UniHOI to surpass all existing methods with a substantial margin, under both supervised and zero-shot settings. The code and pre-trained weights are available at: https: : / thub. com/ Caoyi chao chao/UniHOI.
Introduction
Human-object interaction (HOI) detection [15, 7] is burgeoning field of research in recent years, which stems from object detection and makes higher demands on high-level visual understanding. plays crucial role in many vision tasks, e.g., visual question answering, numan-centria understanding. image generation, and activity recognition, to name few representative ones. An excellent HOI detector must accurately localize all the interacting Human-Object (HO) pairs and recognize their interaction, typically represented as an HOI triplet in the format of < lumman, action, object >
Despite the optimistic progress made by various HOI detectors [27, 12, 18, 49, 14, 29, 8] in recent years, numerous challenges still persist [45]. Naively mapping diverse interactions to one-hot labels an exceedingly labor-intensive and costly process, which is prone to the loss of semantic information.
*Corresponding author (xisu5992 @102220nniiyyooo .du...d First author (caoyichao seu.edu.cn)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Previous Vislon-and-angguage Modeling (HOI-VLM)
Traditional HOl Detection
Our Universal HOI Detector (UniHOI)
OR
HOI Knowledge Retrieual
GPTS-as-KB
Detector
HO Prompts
Rexxible
HOI Detector
Limited Knowleddge +
HOT Detector
VL Foundation
Transfer
Model
Language Mode
Q..
Unlimited Linguistic
+
Interaction Relationship
Knowledge
determinate
Annotation
<hurme, tilkon, hummn elephrr>>
Texts
Figure I: Comparison between HOI detection pipelines. Traditional HOI Detectors rely manually annotated image datasets for training, which limits their generality and scalability. Previous HOI-
VLM methods only achieved limited knowledge transfer from language models to visual HOI detectors, utilizing only a small amount of word embeddings for learning. The proposed UniHOI supports the input of any textual information (annotations or explanations) detect any interaction relationship, fully unleashing the potential of foundation models and LLMs.
As the number and complexity of interaction categories increase, model optimization becomes increasingly difficult, and the universality of the model is consequently restricted.
Recently, multi modal learning, especially Vision-and andno--nnn-LLLauggg (VL) learning, has become remark- ably popular, achieving far-reaching success across a variety of domains [1]. Multi-modal learning is capable of portraying multidimensional information in the same entity or spatio-temporal event through different modalities. Recently, a number of studies have endeavored employ VL models for the HOI task with the aim of constructing transferable HOI detectors [52, 19, 28, 43, 45, 5]. For instance, PhraseHOI [28] harnessed a pre-trained word embedding model to generate fixed-length linguistic representation for the HOI task. Unfortunately, these methods harness cross-modal knowledge in an overly constrained manner, falling short of fully exploiting the potential of cross-modal knowledge and LLMs within the HOI detection domain.
comprehensive manner by describing
Upon rigorous examination, we identify several intrinsic limitations in the existing methodologies: (1) Limited scalability: Model training overly reliant annotated data, which restricts them to finite set of categories. (2) Suboptimal adaptability in zero-shot settings: Even HOI-VLM approaches only harness limited number of word embeddings to accommodate unseen categories, thereby curtailing their adaptability. (3) Inability to quickly comprehend complex interactive behaviors from succinet descriptions, as humans can, indicating dearth of more universal and flexible framework for HOI detection.
Considering the recent outstanding performance of VL foundation models and LLMs in various fundamental tasks, in this paper, we attempt to harness the capabilities of such large models to facilitate more general HOI detector, shown in Figure 1. We conduct in in-depth analysis of the clear three-tier visual feature hierarchy in visual HOI detectors, effectively utilizing the foundation models boost the understanding of high-level interactive semantics within images. The employment of large models bridges the gap between textual and visual modalitics, making a knowledge- based universal HOI detector feasible. In summary, the contributions of this paper
follows:
We propose the first visual-textual foundation model-basee framework for HOI detection (UniHO1), which significantly improves the accuracy and universality compared previous HOI detectors that solely rely on training from specific datasets. Our UniHOI method is expected guide the HOI detection field into new research phase.
We treat the visual HOI detector as three-tier visual feature hierarchy and appropriately design an efficient -ojjctt prompt learning method, enabling visual HOI detectors to effectively query interaction tokens associated with specific HO pairs in foundation models, and thus obtain more universal HO relationship representations.
We devise a knowledge-basee zero-shot HOI recognition method that leverages large models such as GPT to generate descriptive knowledge for specific interaction categories, guiding visual detectors in zero-shot recognition of the corresponding categories. This approach

---

breaks free from the limitations of traditional zero-shot recognition methods, which can only learn from word embeddings, and offers insights for a more universal open-category HOI recognition system.
2
Related Work
Generic HOI Detection. Based on their architectural design, existing HOI (Human-Object In- teraction) detection approaches can be broadly categorized into groups: two-stage methods [49, 12, 13, 27, 18]. and one-stage methods [14, 20, 53, 29, 8]. One-stage methods typically adopt multitask learning strategy to simultancously execute instance detection and interaction relation modeling [29, 47, 30J. Conversely, two-stage methods initially conduct object detection, followed by interaction relation modeling for all candidate HO pairs. By fully exploiting the capabilities of each module, two-stage methods have showcased superior detection performance [47]. While these methods have significuntly propelled the progress of early HOI detection, they still grapple with the issue of limited generality.
Language Semantics for Vision. Driven by the notable success of large-scale models such as GPT-4 [36], the incorporation of language semantics to augment vision models has recently surfaced as promising avenue in the realm of computer vision tasks [38, 41, 1, 50]. Among these, Vision- and-Language Pre-training (VLP) [34, 24J has evolved into popular paradigm across numerous vision-and-language tasks, owing its ability learn generalizable multimodal representations from extensive image-text data [1, 9, 10]. These methods have recently found applications in multimodal retrieval [11], vision-and-len and-language navigation [2], among others [4]. Nevertheless, implementing effective prompt- based learning on base models, particularly extracting high-order order relationships in intricate scenarios, remains crucial for the successful deployment of large models. Thus, investigating how to efficaciously extract specific information from large models is of utmost importance.
Modeling (LMI-VLM). Despite the moderate success of previous HOI detectors [47, 49, 31], these often perceive interactions as discrete labels, overlooking the rich semantic text information encapsulated in triplet labels. Of late, handful of researchers Modeling in an effort to further enhance HOI detection performance. Among these, [52], [19], and [46] have primarily focused on incorporating language prior features into HOI recognition. Meanwhile, RLIP [45] and [43] have proposed the construction of a transferable HOI detector via the VLP approach. Serving as applications and extensions of Vision-ande Languag learning within the HOI domain, these HOI- VLM methodologies strive to comprehend the content and relations between visual interaction features and their respective triplet texts. Nevertheless, these methodologies harness cross-modal knowledge in markedly restricted manner, falling short of fully unleashing the potential of cross- modal knowledge and LLMs within the HOI domain.
HOI Vision-and-Langusee
[52, 19, 28, 43, 45, 46] have delved into HOI Vision-and-Languaa.
3 Method
3.1 Overview
In traditional HOI detection task, the objective is typically optimize detector Pop, such as the most commonly-used Transformer-based detectors in recent years. By inputting a query Qhe related to the Objeet (HO) pairs, the HOI detector learns the location of the HO pairs and corresponding interactions in images. This optimization objective can be formulated as follows:
min Bez)~x, [C(TT, &op (I, 2*)
where GT and C are ground-truie labels and overall loss function respectively, X₂ = { ! denotes the HOI image dataset, I represents the input image, and 0 indicates the weights in HOI detector d0 Thus, traditional HOI detectors rely on purely visual detection tasks for optimization.
In this work, our aim is to utilize superior VL foundation models and additional interaction interpre- tation T during the training and inference process. In this way, we can improve the optimization objective as follows:
min E [C(ST, De (I, hh²), 20, (I,T,Ph)))
(2)
3

---

Input Inare
fcociall back
Knowledge Retrieval for "Human tootball
Otgezt Rooms & Categodie
Borces
OT
Texts Chumann,iikk nootaal>>
LLM
center foot's
HO
Promps
HO Prompt Guided Decoder
Vision Module
Langagg Module S
*
CNN
Rusiiional Enaxding
MP
Instance Decodar
VI. Fuundation Model
Similaity
Textual Festures
x
HD spacitcciiic
F FEU
Image Enooder
pns
2ah
Fonturs
Human Citject Quanttes
Interaction Decoder
20
00
M
Sequired Embbekinngs
HOI Predictions
Figure 2: Overview of the UniHOL. Its feature hierarchy divided into three levels. After the first two levels achieve instance detection for humans and objects, we designed a HO spatial prompt learning method, which extracts more generic higher-level relationship representations associated with HO pairs from the VL foundation models. For open-category interaction detection, UniHOI supports annotations or retrieved texts input, and then determine whether the interaction described in the input text occurs between each HO pair through feature similarity measurement.
where ho denotes the Vision-Languuage (VL.) foundation models and 'pho represents the HOI- specific prompts for 4øy. X = {II, T.) X is an image-text corpus, where Z, denotes the i-th input image and Ti represents the phrase annotations (e.g. "Human ride bicycle")) in I,. In zero-shot setting, we also use knowledge retrieval enrich the texts Details will be described in Section 3.4.
In HOI detection, the detector not only needs to accurately locate the position of the interacting HO pair, but also needs to reliably model the interaction relationship occurring in the HO pair. But these two tasks have different requirements on the image feature level: the former focuses more on the edges and contours of instances, while the latter focuses more on high-level relational cues. To present and analyze this complicated process more clearly, we explicitly divide the feature learning HOI detector into three stages: basic visual feature extraction, instance-level feature learning. and high-level relationship modeling, as shown in Figure 2. The basic visual feature extraction is usually performed by the backbone (e.g. CNN). For the instance-level feature learning. we follow previous methodologies and design prompts related to HO locations to carry and transfer the position information, then apply the Hungarian algorithm for matching with the GT, and compute the loss of the bounding box. The third stage, which is the key focus of this study, involves using spatial location prompts related to the HO pairs to conduct targeted prompt learning from the visual model. The details of each component are explained in the following sections.
3.2 HO Spatial Prompts Generation
We follow the Transformer-ban detection architectures [6] to conduct basic visual feature extraction and instance detection. Fed with an input image I # Hxxxx
the CNN generates feature map X' E Rxxxxe Then, X is compressed by a projection convolution layer with a kernel size 1 x 1. Next, a flatten operator used to segment patch embeddings {x" x² number of patch embeddings. The patch embeddings {xY, x², .z..... 'xy is then linearly projected through a linear transformation E" R Thus, the input for instance detection are calculated via summing the projected embeddings and position embeddings Epos E DD"
- where No is the
X E R xD = """" ;..:"""""""""" + Epos
After patching and sequencing the features extracted by CNN, we follow previous methods to perform self-attention on XP: MM = ooze (XP)) E N" D"", This process is implemented by a Transformer encoder Pozz- which is composed of N Transformer encoder layers.
For instance-level feature learning, we deploy an instance decoder Teyp to locate the positions of humans and objects. Based the M, humans and objects are detected through the human query set

---

Q" E N and the object query set Q" E RN*" individually. Additionally, position guided embedding [30] Q E RNN*DD DD is added to assign the human and object queries at the same position as pair. The calculation process of instance decoder Perp can be formulated simply follows:
pho R2N**D" = [p* PO] Pozo (M, [2 + 2 + 2*)
where the pho is the spatial tokens highly correlated with HO pair location information, the N is the number of queries. With instance detection head FFNs,, s. we calculate the human bounding boxs Bh RN* from p", object bounding boxs B R N** and object categories Co R R-N*xN' X from po:
(5) where the feature tokens [ph pP] are excellent spatial position features for HO pairs. Driven by prompt learning methods, explore using the HO position features here to perform prompt learning on the VL foundation models.
= PP))
3.3 Prompting Foundation Models for HOI Modeling
Considering the excellent performance of BLIP2 [23] in cross-modal tasks and the fact that its Transformer architecture also integrates well with our HOI framework, in this work we use BLIP2 as the VL foundation model to support the recognition process of HOI. However, accurately extracting HOI related features from VL foundation model is the key to successfully applying VL to HOI tasks.
To reduce the computational cost of the VI. foundation model, we first downsample the input image to I'' with dimension W' x H', and then use an image encoder $6r to convert downsampled image into a feature map X E R xhu'xh'xe xc',",, We then employ Q-Former Poo in BLIP2 [23] to learn
higher-level content X E NR x in image:
X
Pay (I') Doz Do (I, Q²)
(6)
where the Qf denotes the query in Øøz. These modules in Pøz are more capable of understanding content in images than regular visiual detectors due to extensive training on a wider range of data.
Then, consider how extract the complex interaction representations between specific humans and objects in foundation model. To associate the higher-leve information in foundation model with HO pairs, we design HO Prompt-guide Decoder (HOPD) dgp to correlate the position information of HO pairs with feature tokens in foundation model:
y = = ((Ph + 12,
where the D denotes the high-level information tokens corresponding to HO pairs. In this manner, the HOI-specific tokens is generated under the guidance of the guidance of the human-object query pairs. This spatial prompts act as query embeddings self-attention layers, and interact with frozen image features through cross-attention layers (inserted every other transformer block in HOPD). In this way, these spatial prompts are fed into HOPD, which in turn goes on to query information about the relationships between their respective corresponding HO pairs. In addition, interaction decoding is performed in the visual HOI detector:
HOPD, interact with each other through
V Te ((Ph MM
(8)
Finally, we concatenate V and D as the visual representations the HO pairs and feed them to FFN to predict the interaction category. For model training, adhere the precedent set by query-based methods [6], utilizing the Hungarian algorithm to assign a corresponding prediction for each ground-truth instance. The loss and hyperparameters used for model optimization in this paper follow those of previous work [30]. More details are provided in the Appendix.
3.4 Knowledge Retrieval for HOT Reasoning Open World
In previous zero-shot HOI detectors, the recognition of unseen categories is usually achieved by directly using word embeddings and visual features for similarity measures. The problem with this approach is that it relies only on simple word embeddings, which may not be sufficiently detailed
5

---

and rich in the representation of HOIs. The model may not recognize some special categories well enough. Inspired by the human understanding of interactions, we propose method for HOI detection in an open world based on descriptive text.
The impressive performance of large-scale pretrained language models, as well the potentially enormous amount of implicitly stored knowledge, raises extensive attention about using language models as an alternative to conventional structured knowledge bases (LMs-as-KBs)) In particular, the rapid development of the chatGPT family of LLM models in the last two years inspired us to obtain richer and more comprehensive knowledge descriptions, called GPTs-as-KBB the performance of the visual model can be improved theoretically by transferring a larger amount of knowledge that can be effectively integrated into the visual model. Given an input image I' and corresponding descriptions Ti in X' = {I, T:) ! different phrase descriptions are linked with different HO pairs. We propose expanding the existing phrase descriptions T to obtain richer and more comprehensive knowledge descriptions K.' These descriptive knowledge facilitate more accurate interaction descriptions in zero-shot and open-vocabulary HOI recognition.
Theoretically,
To obtain richer and more comprehensive knowledge descriptions, we design knowledge retrieval process for LLM. We have designed a knowledge query statement for chatGPT, such as edge retrieve for verb_ crrb_ojjectoon limited to N words". to guide LLMs in explaining the in- teraction categories we are interested in. this way, the transferable textual knowledge will be richer and the interpretation of the interactions will be closer the way humans understand them. As an illustration, let us consider the phrase "Human ride bicycle" and "Human throw frisbee". The knowledge retrieval process yields the following example result:
Riding bicycle involves balance, coordination. and physical exertion. The rider mounts the bike, propels forward by pushing pedals with their feet. Steering is achieved by turning handlebars. Brakes slow or stop the bike. Helmets are worn for safety.
Throwing frishee involves grasping the disc, typically with forehand grip, then swinging the arm and releasing the frishee at the right moment for to glide through the air. Direction and distance depend on the angle and speed of the throw. It's a common recreational activity.
These detailed text descriptions can better guide the learning of visual features in open-category recognition. A more detailed open-category inference process is provided in the Appendix.
Experiments
4.1
Implementation Details
We evaluate our model on two public benchmarks, HICO-Det [7] and V-COCO [15]. We adopt the BLIP2 [23] model as the VL foundation model in our UniHOI. We map the 768-dimensional features output by Q-Former in BLIP2 to 256 dimensions to adapt to the feature dimension of the visual HOI detector. The number of layers in the HO Prompt-guided Decoder is identical to that in the Interaction Decoder. The output features of these two decoders are concatenated to generate the final category prediction. The model's optimization method, learning rate, training epochs, weight decay, and loss weights, among other hyperparameters, are set in accordance with previous methods to ensure fair comparison. Aside from the ablation studies in Section 4.6 where we employed knowledge retrieval to further enhance model performance, in all other experiments we continue to use annotated phrases for model learning and inference to ensure fair comparison with other methods. All the experiments conducted with batchsize of 16 on 8 Tesla V100 GPUs. The details of datasets, evaluation metrics, model parameters, computational overhead, etc. will be introduced in Appendix.
4.2 HOI Detection in the Closed World
We compare the performance of our UniHOI with the previous representative and of-te-of-the-ar methods such as ViPLO [37], GEN-VLKT [30]. and HOICLIP [35]. The detailed results of HICO- DET [7] and V-COCO [15] are summarized in Table and 3, respectively. Generally, there are several observations drawn from different aspects: (i) The overall performance of our method is significantly superior to the previous state-of-thed methods. (ii) Whether the visual feature extractor adopts the ResNet-50 or MosNet-101, our method yields excellent performance, especially stronger backbone

---

Caii CEN-TETT
Speci
Figure 3: The training process of UniHOI and previous ofateoofthe--Aee GEN-VLKT [30]
V-COCO dataset.
Table 1: Experimental results on HICO-DET [7]. GEN-VLKT [30] is regarded as baseline.
Default Setting
Known Objects Setting
Method
Backbone
Full
Non-rare
Full
Non-rare
Two-stage Methods:
ATL [17]
ResNet-50
23.81
17.43
27.42
27.38
22.09
28.96
VSGNet [42]
ResNet-155 152
19.80
16.05
20.91
DJ-RN [25]
ResNet-50
21.34
18.53
22.18
23.69
20.64
24.60
VCL [16]
ResNet-50
23.63
17.21
25.55
25.98
19.12
28.03
DRG [12]
ResNNet-50-FPN
24.53
19.47
26.04
27.98
23.11
29.43
IDN [26]
ResNet-50
24.58
20.33
25,86
27.89
23.64
29.16
FCL [18]
ResNet-50
25.27
20.57
26.67
27.71
22.34
28.93
SCG [48]
ResNet-50-EPN
29.26
24.61
30.65
32.87
27.89
34.35
UPT [49]
ResNet-50
31.66
25.90
33.36
35.05
29.27
36.77
UPT [49]
ResNet- 101
32.31
28.55
33,44
35,65
31.60
36.86
ViPLO-s [37]
ViT-B/32
34.95
33.83
35.28
38,15
36.77
38.56
ViPLO-1 [37]
ViT-B/16
37.22
35.45
37.75
40.61
38.82
41.15
One-stage Methods:
PPDM [29]
Hourglass-
21.94
13.97
24.32
24.81
17.09
27.12
HOI-Trans [54]
ResNet-101
26.61
19.15
28.84
29.13
20.98
31.57
AS-Net [8]
ResNet-50
28.87
24.25
30.25
31.74
27.07
33.14
QPIC [40]
ResNet-101
29.90
23.92
31.69
32.38
26.06
34.27
SSRT [19]
ResNet-101
31.34
24.31
33.32
CDN-S [47]
ResNet-50
31.44
27.39
32.64
34.09
29.63
35.42
CDN-L [47]
ResNet-101
32.07
27.19
33.53
34.79
29.48
36.38
Liu al. [31]
ResNet-50
33.51
30.30
34.46
36.28
33.16
37.21
MUREN [22]
ResNet-50
32.87
28.67
34.12
35.52
30.88
36,91
GEN-VLKT-s [30]
ResNet-50
33.75
29.25
35.10
36.78
32.75
37.99
GEN-VLKT-m [30]
ResNet-101
34.78
31.50
35.77
38.07
34.94
39.01
GEN-VLKT-I [30]
ResNet-101
34.96
31.18
36.08
38.22
34.36
39.37
HOICLIP [35]
ResNet-101
34.69
31.12
35.74
37.61
38.54
Xie al. (large) [44]
ResNet-101
36.03
33.16
36,89
38.82
35.51
39.81
UnitOO-s (w/ BLIP2)
ResNet-50
40.06
39.91
40.11
42.20
42.60
42.08
(+6.31)
(+10.66)
(+5.01)
(+5,42)
(+9.85)
(+4.09)
UniHO1-m (w/ BLIP2)
ResNet-101
40.74
40.03
40,95
42.96
42.86
42.98
(+5.96)
(+8.53)
(+5.18)
(+4,89)
(+7,92)
(+3,97)
UniHOr-1 (w/ BLIP2)
ResNet-101
40.95
40.27
41.32
43.26
43.12
43.25
(+5.99)
(+9.09)
(+5.24)
(+5.04)
(+8.76)
(+3.88)
that can bring further improvements. (iii) As shown in Tables 2, our approach also has significant performance lead compared to those methods that rely on additional data sets (i.e., Human Pose [32],
and Likion-and-Languae [45]). Figure 3 shows the training process of UniHOI-s and GEN-VLKT
on the V-COCO dataset. Figure 4 shows some prediction cases of UniHOI on HICO-DET.
4.3 Comparisons with Methods that Utilize Extra Information
Table presents several methods that utilize extra information (i.e., pose, and language) from additional datasets for HOI detection. Theoretically, the introduction of extra information can indeed further enhance visual models in specific domains. The results demonstrate that the proposed UniHOI still significantly outperforms previous methods.

---

anL
m
Figure 4: Detection results of our UniHOI in HICO-DET [7].
Table 2:
Comparison results with the methods using extra datasets HICO-DET dataset [7]. For extra datasets, "P" indicates human pose and "L" indicates linguistic knowledge.
HICO-DET
V-COCO
Default Setting
Known Objects Setting
Method
Extras
Full
Rare
Non-rare
Full
Rare
Non-rare
AP#1
rale
AP#3
rale
FCMNet [32]
P+L
20.41
17.34
21.56
22.04
18.97
23.12
53.1
PD-Net [52]
L
20.76
15.68
22.28
25.59
19.93
27.28
DRG [12]
P
24.53
19.47
26.04
27.98
23.11
29.43
51.0
ConsNet-F [33]
P
24.39
17.10
26.56
53.2
RLIP-ParSeD [45]
L
30.70
24.67
32.50
RLIP-ParSe [45]
L
32.84
26.85
34.63
61.9
64.2
PhraseHOI [28]
30.03
23.48
31.99
33.74
27.35
35.64
OCN (large) [46]
L
31.43
25,80
33.11
HOICLIP [35]
L
34.69
31.12
35.74
37.61
34.47
38.54
63.5
64.8
UniHOI-s (w/ BLIP2)
L.
40.06
39.91
40.11
42.20
42.60
42.08
65.58
68.27
UniHOI-m (w/ BLIP2)
40.74
40.03
40.95
42.96
42.86
42.98
67.95
70.61
UniHOI-I (w/ BLIP2)
L
40.95
40.27
41.32
43.26
43.12
43.25
68.05
70.82
4.4 Effectiveness for Zero-Shot HOI Detection
Unseen Composition (UC), As shown in Table 4, we follow the [30] to conduct three kind of zero-shot experiments on HICO-DET, We first evaluate UniHOI-s with UC setting. Compared with GEN-VLKT [30], our UniHOI-s substantially outperforms previous state-of-the-ar on both RF-UC and NF-UC. Among them, UniHOI are 7. 7.32mAP higher than GEN-VLKT Unseen of RF-UC, and 9.25mAP higher than GEN-VLKT on Seen of NF-UC. This improvement mainly due to our efficient prompt learning method for the foundation model, UniHOI can cope well with some unseen categories. In fact, GEN-VLKTT also performs some knowledge transfer from CLIP, but lags behind UniHOI in the effectiveness of knowledge transfer.
Unseen Object (UO). For UO setting, compared with GEN-VLKT, UniHOI improves Unseen, Seen, and Full by 9.21, 5.84, and 5.93, respectively. This result proves that our method can still understand the interaction between human and novel objects well.
Unseen Verb (UV). On the challenging UV setting, compared to GEN-VLKS UniHOI improves by 9.21, 5.84 and 5.93 on Unseen, Seen, and Full. respectively. fact, on the three zero-shot settings mentioned above, the HOI recognition capability of UniHOI have almost surpassed some supervised
Table 3: Experimental results on V-COCO.
Table 4: Zero-shot results on HICO-DET [7].
Method
AP raie
A
vale
Method
Type
Unseon
Soen
Full
VSONet [42]
51.8
$7.0
Shen el al, [39]
UC
3.62
6.26
IDN [26]
53,3
60.3
PG [3]
UC
10.93
12.60
12.26
UPT [49]
60.7
66.2
ATL [17]
UC
16.99
20.51
19.81
VIPLO-s [37]
66.6
VCL [[6]
RF-UC
10.06
24.28
21.43
VIPLO-I [37]
62.2
68.0
ATL
RE-UC
21.57
HOTR [2]]
55,2
64.4
FCL [18]
RF-UC
13.16
24.23
22.01
QPIC [40]
58.8
61.0
GEN-VLKT
RE-UC
21.36
32.91
30.56
CDN [47]
63.91
65.89
UnitOO-s (w/ BLIP2)
RF-UC
28.68
(+7.32) 33.16
(40.25)
32,27 (+L.71)
Liu [31]
63.0
65.2
VCL[16]
NE-UC
16.22
18.52
18.06
UEN-VLKT-a [30]
62.41
64.46
ATL
NE-UC
18.25
18.67
GEN VLKT [30]
FCL [18]
NF-UC
18.66
19.55
19.37
GIN.VI [30]
63.58
65.93
GEN-VLKT
NESUC
25.05
23.38
23.71
HOICCIP [35]
63.50
64,80
UnitIOLss BLIP2)
NF-UC
28.45
(+3.4)
$2.63 (49.25) 31.79 (+8.08)
Xie et at. (large) [44]
66.50
69.90
GEN-VLKT VLKT
UO
10.51
28.92
25.63
Unit (w/ BLIP2)
65.58
(+3.17)
68.27 (+3.81)
Unit1Ols BLIP2)
(+9.21)
(+5.84)
(+5.93)
UnittOI-m BLIP2)
(+4.67)
70.61 (+5.03)
MEN-VLKTI330
UV
20.96
30.23
28.74
UniHOl- (w) BLIP2)
68.05
(+4.47)
70.82 (+4.89)
UnitlOss (w/ BLIP2)
UV
26.05 (+5.09)
36.78
(46.55) 34.68 (+5.94)

---

with
Negodatee with Human
Juggling oranges involves tossing and catching multiple oranges in specific pettern. Typically, the juggler keeps more oranges in the air than they have hands, It requires good hand-eye coordination, rhythm, and timing, This activity be entertaining and challenging. Kickboxing is hybnd combat sport that combines elements punching from boxing kitking from karate or muay that. Humans participating sickboxing engage rigorous physical training, improving their strength, flexibility, and coordination. It's both competitive sport and fitness regimen. compromise reach
Negotiating with human inwolves communication patience, and persuasion. Negotiations can occur in various contexts, including business transactions, conflict resolution, and interpersonal relationships. Sleeping bost be unique experience, influenced by factors like the boat's size, stability, location. The gentle rocking be soothing, but conditions can vary. It's important ensure safety precautions, such wearing lifejackets, are place.
agreement. requires skills active listening, articulation,
Shaking hands with a human is a common form of greeting, agreement, or parting. The typical handshake involves a firm grip and brief up and down understanding
It's universal sign of goodwill, respect,
individuals.
Figure 5:
HOI Detection in the Wild. UniHOI is capable of conducting open-category detection based on triplet phrases and also supports the input of descriptive texts retrieved by GPT [36].
Table 5: Ablation Studies V-COCO [15].
Table 6: Ablation Studies on HICO-DET (UV).
Method
AP
AP#1
Method
Unseen
Seen
Full
baseline
62.41
64.46
baseline
20.96
30.23
28.74
Foundation Model
62.91
64.83
+ VL Foundation Model
21.57
31.62
29.85
HOPD
65.58
68.27
+ HOPD
26.05
36.78
34.68
Knowledge Retrieval
66.74
69.31
+ Knowledge Retrieval
27.41
37.82
35,89
methods in Table This once again attests to the superiority of our UniHOI for open-category HOI detection and its powerful capability understand interactions.
4.5 HOI Detection in the Wild
In addition to the zero-shot experiments on the HICO dataset, we also conducted open-category testing open-world. Our UniHOI method supports flexible text input for detecting corresponding interactions. UniHOI can not only recognize category texts directly (e.g., "Juggle orange") but also take descriptive text input (which could be human-written or knowledge retrieved from GPT). Consequently, interactions in images are identified through text representation. We present these detection results, along with some GPT-generated description texts, in Figure 5. Our test results indicate that open-vocabulary recognition based on interaction description texts is more reliable, and this phenomenon is also in line with our basic intuition. In the future, we will release our pretrained model and code to enable more extensive testing in the open world.
4.6 Ablation Studies
Effect of Foundation Model. We conduct ablation studies on the new components within UniHOI (as shown in Tables and 6). The GEN- VEK--S is regarded as the baseline, and then incorporated various components for testing the fully supervised V-COCO dataset, well as the HICO-DET dataset under a zero-shot setting. Initially, we simply integrated the BLIP2 into our method in a sequential manner. We then used learnable queries to extract interaction features from Q-Former and concatenate them with V However, the results did not yield a significant performance improvement.
Impact of HO Spatial Prompting, After realizing the alignment issue with the features of HO-pairs in the previous methods, we introduced the HO Prompt-guides Decoder, as presented in Section 3.3. The results clearly illustrate that the introduction of this component significantly improved all performance metrics, thereby validating the effectiveness of our prompting learning strategy.
Effect of Knowledge Retrieval. To substantiate the importance of utilizing LLM for knowledge retrieval in creating a more universal HOI detector, we also conduct supplementary experiments in this strategy. The results showed that the UniHOI model, when applying knowledge retrieval, achieved further performance improvements. We believe this approach opens new avenues for enhancement in the field of shot HOI detection.

---

Table 7: The results of UniHOI equipped with different foundation models (i.e., CLIP [38] and BLIP2 [23]) on V-COCO(15]
Method
AP
AP
rale
role
GEN- [30]
62.41
64.46
GEN-VLKT-m [30]
63.28
65.58
GEN-VLKT-I [30]
63.58
65,93
UniHOI-s (w/ CLIP)
62.92
(+0.51)
65.67
(+1.21)
UniHOI-m (w/ CLIP)
64.85 (+1.57)
67.62 (+2.04)
UniHOO-II (w/ CLIP)
64.93 (+1.35)
67.86 (+1.93)
UniHOI-s (w/ BLIP2)
65,58
(+3.17)
68.27 (+3.81)
UniHOI-m (w/ BLIP2)
67.95
(+4.67)
70.61 (+5.03)
UniHOI-I (w/ BLIP2)
68.05 (+4.47)
70.82 (+4.89)
Effect of Different Foundation Model. To assess the impact of different Foundation Models on the performance of UniHOI, we also provide comparative analysis between UniHOI equipped with CLIP and BLIP2. For CLIP, we feed its visual output representation into HOPD. However, given that CLIP's cross-modal contrastive training strategy leans more towards image-level representations [38], its output dimensionality is considerably lower than that of BLIP2, resulting in a less detailed feature representation. Experimental results indicate that both the UniHOI versions with CLIP and BLIP2 as the foundational models showed performance enhancements, with the latter exhibiting a more pronounced improvement. These findings underscore the efficacy and generalization of our approach.
5 Conclusion
In this work, we propose UniHOI, universal HOI detection framework. By utilizing large foun- dation model, spatially prompting the large model, and generalizing the HOI detector to the open world. UniHOI supports open- open-category interaction relationship recognition from annotation texts or descriptions, which can be flexibly extended to applications in the open world. Extensive experiments on public datasets and diverse settings demonstrate its strong universality it behaves the strongest open category interaction recognition ability far. We believe research will stimulate following research along the universal high-level relationship modeling direction in the future.
References
[1] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, L. Hasson, Y., Lenc. K., Mensch, A., Millican, K., Reynolds, M., al.: Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198 (2022) 2,
[2] Anderson, P., Wu, Q. Teney, D., Bruce, J. Johnson, M., Sünderhauf, N., Reid, I., Gould, S., Van Den Hengel, A.: Vision-au irsoo-and-aan and-language navigation: Interpreting ssually-grounde navigation instructions in real enviromments. Proceedings of the IEEE conference on computer vision and pattern recognition.
pp. 3674-3683 (2018)
[3] Bansal, A.. Rambhatla. S.S., Shrivastava, A., Chellappa, R.:
Detecting human-object interactions via functional generalization. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 10460- 10460 (2020)
[4] Cao, Y., Su, X., Tang, Q., You, S., Lu, X., Xu, C.: Searching for better spatio-temporal alignment in sew-shot action recognition. Advances in Neural Information Processing Systems 35, 21429 21441 (2022)
[5] Cao, Y. Tang, Q.. Yang, F., Su, X.. You, S., Lu, X., Xu, C.: Re-mine, learn reason: Exploring cross-modal semantic correlations for language-guided hoi detection. In: Proceedings of the IEEE/CVF
International Conference on Computer Vision. pp. 13492-23500 (2023) 2
[6] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection
with transformers. In: European conference on computer vision. pp. 213-229. Springer (2020) 4, 5, 15
[7] Chao, Y.W., Liu, Y., Liu, X., Zeng, H., Deng, J.: Learning to detect numan-object interactions. In: 2018 icce winter conferenee on applications of computer vision (wacv). pp. 381-389, IEEE (2018) 1, 6, 16
[8] Chen. M., Liao, Y., Liu, S., Chen, Z., Wang. F., Qian, C.: Reformulating hoi detection as adaptive set prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9004 9013 (2021) 3,
10

---

[9] Chen, Z., Li, G.. Wan. X.: Align, reason and learn: Enhancing medical vision-and- languuge pre-training with knowledge. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 5152-5161 (2022)
[10] Cheng, M., Sun, Y., Wang. L.,, Zhu, X., Yao, K., Chen, J., Song. G., Han, J., Liu, J., Ding, E., Wang, J.: Vista: Vision and scene text aggregation for cross-modal retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5184 1194-1193 (June 2022)
[11] Dzabraev, M., Kalashnikov, M., Komkov, S., Petiushko, A.: Mdmmt: Multidomain multimodal transformer for video retrieval. In: Proceedings of the IEEF/CVF Conference on Computer Vision and Pattem
Recognition. pp. 3354-3365 (2021) 3
[12] Gao, C., Xu, J. Zou, Y., Huang, J.B.: Drg: Dual relation graph for human-object interaction detection. In:
European Conference on Computer Vision. pp. 696-712. Springer (2020) 1, 3, 7, 8
[13] Gao, C., Zou, Y., Huang, J.B.; ican: Instance-centric attention network for human-object interaction
detection. arXiv preprint arXiv: 1011180..1100 10437 (2018) 3
[14] Gkioxari, G., Girshick, R., Dollár, P., He, K.: Detecting and recognizing human-object interactions. In: Proceedings of the IEEE conference computer vision and pattern recognition. pp. 8359 8367 (2018) 1,
[15] Gupta, S., Malik, J.: Visual semantic role labeling. arXiv preprint arXivv:505.044m (2015) 1, 6, 9, 10, 14,
15
[16] Hou, Z., Peng, X., Qiao, Y., Tao, D:: Visual compositional learning for human-ohject interaction detection. European Conference on Computer Vision. pp. 584-600 Springer (2020) 7,8
[17] Hou, Z., Yu, B., Qiao, Y., Peng, X., Tao, D.: Affordance transfer learning for human-ohject interaction detection. In: Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition. pp- 495-504 (2021) 7.8
[18] Hou, Z., Yu, B., Qiao, Y., Peng, X., Tao, D.: Detecting human-object interaction via fabricated composi- tional learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14646-14655 (2021) 3, 7,8
[19] Iftekhar, A., Chen, H., Kundu, K., Li, X., Tighe, J., Modolo, D.: What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions. In: Proceedings of the IEEE/CVF
Conference Computer Vision and Pattern Recognition, pp. 5353-5363 (2022) 2, 3,7
[20] Kim, B., Choi, T., Kang, J.. Kim, HJJ.: Uniondet: Union-level detector towards real-time human-object interaction detection. In: European Conference Computer Vision. pp. 498-514. Springer (2020) 3
[21] Kim, B., Lee, J., Kang. J.. Kim, E.S., Kim. H.J.: Hotr: End-to-end buman-object interaction detection with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision Pattern Recognition. pp. 74-83 (2021)
[22] Kim, S., Jung, D.. Cho, M.: Relational context learning for human- interaction detection. arXiv preprint #TXiv:2344.04997 (2023) 7
[23] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 122597 (2023) 5, 6, 10, 13
[24] Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang. L.,, Yuan, L., Zhang, L., Hwang, J.N., et al.: Grounded language-image pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965-10975 (2022) 3
[25] Li, Y.I.,, Liu, X., La, H., Wang, S., Liu, J., Li, J., Lu, C.: Detailed 2d-3d joint representation for human-object interaction. In: Proceedings of the TEEE/VV Conference Computer Vision and Pattern Recognition. pp. 10166 10166-10175 (2020) 7
[26] Li, Y.I.,, Liu, X., Wu, X., Li, Y., Lu, C.: Hoi analysis: Integrating and decomposing human-object interaction. Advances in Neural Information Processing Systems 33, 5011-5022 (2020) 7,8
[27] Li, Y.L., Zhou, S., Huang, X., Xu, L., Ma, Z., Fang, H.S., Wang, Y., Lu, C.: Transferable interactiveness knowledge for human-object interaction detcction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. $585-3594 (2019) 1, 3, 16
[28] Z., Zou, C., Zhao, Li, B., Zhong, S.: Improving human- interaction detection via phrase learning and label composition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36,
pp. 1509 1517 (2022) 2, 3, 8, 13, 14
[29] Liao, Y., Liu, S., Wang. F., Chen, Y., Qian, C., Feng. J.: Ppdm: Parallel point detection and matching for real-time human- human-obbect interaction detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 482 490 (2020) 3,

---

[30] Liao. Y., Zhang, A., Lu, M., Wang. Y. Li, X., Liu. S.: Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In: Proceedings of the IEEE/CVF Conference Computer
Vision and Pattern Recognition. pp. 20123-20132 (2022) 3, 5. 6, 7, 10, 13, 14, 15
[31] Liu, X., Li, Y.I., Wu, X., Tai, Y.W., Lu, C., Tang, C.K.: Interactiveness field in human-object interactions. In: Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13-
20122 (2022) 3, 7,8
[32] Liu, Y., Chen, Q.. Zisserman, A.: Amplifying key cues for human-objec Conference on Computer Vision. pp. 248 168.265.. Springer (2020) 7,8
cuman-objcctinttraccionn interaction detection. In: European
[33] Liu, Y., Yuan, Chen, C.W.: Consnet: Learning consistency graph for zero-shot human-object interaction detection. In: Proceedings the 28th ACM International Conference on Multimedia. pp. 4235-4243
(2020) 8
[34] Mu, N., Kirillov, A., Wagner, D., Xie, S.: Slip: Self-supervision meets sanuuge-imagg pre-training. arXiv preprint arXiv:21 12. 12750 (2021) 3
[35] Ning, S., Qiu, L., Liu, Y. He, X.: Hoiclip: Efficient knowledge transfer for hoi detection with vision-
language models. arXiv preprint arXiv:2303. 15786 (2023) 6, 7, 8, 14
[36] OpenAI: Gpt-4 technical report (2023) 3,9
[37] Park, J., Park, J.W., Lee, J.S.: Viplo: Vision transformer based pose-conditioned self-loop graph for human- interaction detection. arXiv preprint arXiv:2304.08 14 (2023) 6, 7,8
[38] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748-8763. PMLR (2021) 3, 10, 13
[39] Shen, L., Yeung. S., Hoffman, J.. Mori, G., Fei-Fei, L.: Scaling human-object interaction recognition through zero-shot learning. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 1568-155 1576. IEEE (2018) 8
[40] Tamura, M., Ohashi, H., Yoshinaga, T.: Qpic: Query-hased pairwise human-ohject interaction detection with image-wide contextual information. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 10410-10419 (2021) 7, 8,
[41] Tsimpoukelli, M., Menick, J.L., Cabi, S., Eslami, S., Vinyals, O., Hill, F:: Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems 34, 200-212 (2021) 3
[42] Ulutan. O., Iftekhar. A., Manjunath, B.S.: Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. In: Proceedings of the IEEE/CVF conference on computer vision
and pattemn recognition. pp. 13617-13622 (2020) 7, 8
[43] Wang, S., Duan, Y., Ding. H., Tan, Y.P., Yap, K.H.,, Yuan, J.: Learning transferable human-object interaction detector with natural language supervision. In: Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition. pp. 939-948 (2022) 2,
[44] Xie, C., Zeng, E. Hu, Y., Liang, S., Wei, Y.: Category query learning for human-object interaction classification. arXiv preprint arXiv:2303. 4005 (2023) 7,8
[45] Yuan, H., Jiang, J., Albanie, S., Feng. T., Huang, Z., Ni, D., Tang, M.: Rlip: Relational language-image pre-training for human-object interaction detection. arXiv preprint arXiv:2209, 001814 (2022) 3, 7,
[46] Yuan, H., Wang, M., Ni, D., Xu, L.: Detecting human-ohject interactions with object-guided cross-modal calibrated semantics. arXiv preprint arXiv:2202.002:
(2022) 3, 8
[47] Zhang, A., Liao, Y., Liu, S., Lu, M., Wang, Y.. Gao, C., Li, X.: Mining the benefits of two-stage and one- stage hoi detection. Advances in Neural Information Processing Systems 34, 17209 17220 (2021) 3,
7, 8
[48] Zhang, F.Z., Campbell, D., Gould, S.: Spatially conditioned graphs for detecting human-object interactions.
In: Proceedings of the IEEF/CVF International Conference on Computer Vision. pp. 13319-13327 (2021)
[49] Zhang, F.Z., Campbell, D., Gould, S.: Efficient two-stage detection of human-object interactions with a novel unary- unary-pairwise transformer. In: Proccedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20104-20112 (2022) 1, 3, 7,
[50] Zhang, H., Zhang, P., Hu, X., Chen, Y.C., Li. L.H., Dai, X., Wang, L. Yuan, L.. Hwang, J.N., Gao. J.: Glipv2: Unitying localization and vision- languae understanding. arXiv preprint arXiv:2206.00 228.65836660066666 (2022) 3
[51] Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M.. Chen, S., Dewan, C., Diab, M., Li. X., Lin, X.V., et al.: Opt: Open pre-trained transformer language models. arXiv preprint arXiv:220..00
(2022) 13
[52] Zhong, X., Ding, C. Qu. X., Tao, D.: Polysemy deciphering network for robust human human-object interaction detection. International Journal of Computer Vision 129(6), 1910 1929 (2021) 3.8
12

---

[53] Zhong, X., Qu, X., Ding. C.. Tao, D.: Glance and gaze: Inferring action-aware points for one-stage hujan-object interaction detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13234-13243 (2021)
[54] Zou, C., Wang, B., Hu, Y. Liu, J., Wu, Q. Zhao, Y., Li, B., Zhang, C., Zhang, C., Wei, Y., et al.: End-to-end human object interaction detection with hoi transformer. In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11825-11833 (2021) 7
13


Reviews:
[{"review_id": "YtooSCiEsT", "reviewer": "Reviewer_WxZv", "summary": "This paper proposes a universal HOID pipeline, which utilizes decoded ho-pair feature as spatial prompts to prompt the VL foundation model with the aim to implement effective prompt-based learning on base VL model and extract HOI related features from it. It also proposes knowledge retrieval for HOID in open-category manner by large-scale pretrained language models. Experiments show the effectiveness of this approach in both generic and zero-shot HOID.  \n\n", "questions": "Majorly the questions lies in the choice and interpretability of spatial prompts. It’s unclear why these prompts are ‘excellent spatial position features’. Or have you try some experiments on the choice of these prompts?\n\n\n", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It explores universal interaction recognition by tansferring the rich knowledge inside Vision-Language foundation models and LLMs to HOI pipeline, which broaden the research scope of HOID.\n2. The experimental results are promising. In both generic and zero-shot setting, this approach reaches a new state-of-the-art and surpasses previous methods by a substantial margin.\n", "weaknesses": "1.In line152, ‘P_h’ and ‘P_o’ are described as ‘excellent spatial position features’ and further utilized as HO spatial prompts. However, these features are generated by learnable position embedding and query, which is identical as many transformer-based HOID approaches before such as GEN-VLKT. Can you provide some evidence that these feature are indeed ‘excellent’, why could it provide accurate spatial information concerning ho pairs? Or are there some unique designs I overlook?   \n2.In Line 177, HOPD is designed for the alignment issue between VL foundation model and HOI pipeline. And the output V_f is incorperated with V_i for final predicition of interaction. But the performance of pure V_f, i.e., only utilizing V_f for prediction is unexamined. This results may more directly show the effectiveness of alignment between VL models and HOI pipeline.\n3.Some typos. Line 171, ‘the guidance of’ repeated twice. Line 177, ‘V_i’ is mismarked as ‘V_f’. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer WxZv,\n\nFirst and foremost, we extend our deepest gratitude for your thorough review and insightful feedback. Your recognition of our method is truly appreciated. We concur with your perspective that utilizing $V_f$ exclusively for prediction would offer a more direct testament to the effectiveness of alignment between VL models and the HOI pipeline. We have diligently conducted the pertinent experiments to this effect. Herein, we provide a detailed response to each of your concerns:\n\n\n**1. Regarding the 'excellent spatial position features' of $P_h$ and $P_o$**:\n\nIn our study, “$P_h$” and “$P_o$” are two 64*256 features used for predicting Human boxes $B^h$, Object boxes $B^o$, and categories $C^o$. Consequently, “$P_h$” and “$P_o$” are highly correlated with instance-level position and category information. This makes them exemplary spatial position features, perfectly tailored to serve as spatial location representations to prompt the VL foundation models.\n\nWe aim to explore the synergy between universal (e.g. BLIP2 and CLIP) and specialized (e.g. GEN-VLKT) models. This innovative design amalgamates traditional HOI detectors, encompassing instance localization and relational reasoning, with VL foundational models. It provides a novel avenue for large model-driven HOI detection, further inspiring the development and implementation of generic algorithms.\n\n\n**2. Regarding the performance of pure $V_f$**:\n\nWe are truly grateful for this insightful consideration. **Conducting this experiment does indeed elucidate the effectiveness of alignment between VL models and the HOI pipeline**. We have appended the results of the experiment on V-COCO dataset where only $V_f$ was used for prediction:\n\n|Model|Feature|${AP}^{1}_{role}$|${AP}^{2}_{role}$|\n|:--:|:--:|:--:|:--:|\n||Only $V_i$|62.41|64.46|\n|${UniHOI}_s$ (w/ BLIP2)|Only $V_f$|64.51|67.26|\n||$V_i$+$V_f$|**65.58**|**68.27**|\n||Only $V_i$|63.28|65.58|\n|${UniHOI}_m$ (w/ BLIP2)|Only $V_f$|66.18|68.95|\n||$V_i$+$V_f$|**67.95**|**70.61**|\n||Only $V_i$|63.58|65.93|\n|${UniHOI}_l$ (w/ BLIP2)|Only $V_f$|66.25|69.27|\n||$V_i$+$V_f$|**68.05**|**70.82**|\n\nThe experimental results on the V-COCO dataset show that only the features $V_f$ generated by the BLIP2 model have achieved impressive results, second only to the performance of the feature combination of \"$V_i $+$V_f $\".\n\nAdditionally, in light of the concerns raised by reviewer djfM regarding the performance difference between BLIP2 and CLIP, **we have also conducted experiments using UniHOI driven by CLIP**, specifically evaluating its performance when utilizing $V_f$ and $V_i$:\n\n|Model|Feature|${AP}^{1}_{role}$|${AP}^{2}_{role}$|\n|:--:|:--:|:--:|:--:|\n||Only $V_i$|62.41|64.46|\n|${UniHOI}_s$ (w/ CLIP)|Only $V_f$|62.87|64.91|\n||$V_i$+$V_f$|**63.79**|**65.91**|\n||Only $V_i$|63.28|65.58|\n|${UniHOI}_m$ (w/ CLIP)|Only $V_f$|63.74|66.25|\n||$V_i$+$V_f$|**64.47**|**67.83**|\n||Only $V_i$|63.58|65.93|\n|${UniHOI}_l$ (w/ CLIP)|Only $V_f$|63.79|66.83|\n||$V_i$+$V_f$|**64.86**|**67.98**|\n\nIn addition to the two close-set experiments mentioned above, **we have also conducted experiments in an open-set setting on the HICO-DET dataset**. The results of **UniHOI equipped with BLIP2** are as follows:\n\n|Type|Feature|Unseen|Seen|Full|\n|:--:|:--:|:--:|:--:|:--:|\n|RF-UC|Only $V_i$|21.36|32.91|30.56|\n|RF-UC|Only $V_f$|28.22|32.18|31.38|\n|RF-UC|$V_i$+$V_f$|**28.68**|**33.16**|**32.27**|\n|NF-UC|Only $V_i$|25.05|23.38|23.71|\n|NF-UC|Only $V_f$|**28.69**|32.51|31.74|\n|NF-UC|$V_i$+$V_f$|28.45|**32.63**|**31.79**|\n|UO|Only $V_i$|10.51|28.92|25.63|\n|UO\t|Only $V_f$|**20.91**|**34.88**|**31.78**|\n|UO\t|$V_i$+$V_f$|19.72|34.76|31.56|\n|UV|Only $V_i$|20.96|30.23|28.74|\n|UV\t|Only $V_f$|**26.81**|36.41|34.51|\n|UV\t|$V_i$+$V_f$|26.05|**36.78**|**34.68**|\n\n\nFrom the results of both our **close-set** and **open-set** experiments, we draw the following conclusions:\n\n- **The alignment between VL models and the HOI pipeline in our UniHOI is particularly effective**. Even when solely leveraging the features $V_f$ from the large models, our approach yields impressive results.\n\n- In the **close-set scenarios**, the feature combination of “$V_i$+$V_f$” emerges as a superior choice, rendering the model to function much like a specialized HOI detector. However, in the **open-set scenarios**, where there's a stronger emphasis on understanding the open world, it's predominantly the $V_f$ feature that plays a pivotal role.\n\nWe highly value your feedback and pledge to incorporate these novel experimental data into our final version. This will undoubtedly enhance our paper, facilitating a more comprehensive understanding and evaluation for readers.\n\n**3. Typos**:\n\nWe sincerely apologize for the oversight and are grateful for your meticulous attention to detail. Rest assured, these errors will be rectified in the revised manuscript. Furthermore, we commit to thoroughly scrutinizing the entire document to preclude any similar lapses.\n\n\nThank you again for your constructive feedback. **We will incorporate the corresponding modifications and expansions into the revised paper**. In addition, **the corresponding code and model weights will be open-source to ensure replication**. If you have any further questions, please let us know. We’d be very happy to do anything we can that would be helpful in the time remaining. Thanks!"}, {"review_id": "CN7IlNqxk0", "reviewer": "Reviewer_pXLy", "summary": "The paper addresses human-object interaction (HOI) detection task. The authors propose a new method named as UniHOI, achieved by prompting BLIP2 using human-object paired features, as well as linguistic semantics generated by a LLM. The proposed UniHOI demonstrates significant performance gain on HICO-DET and V-COCO in both fullly supervised and zero-shot scenarios.", "questions": "Actually, I am very much looking forward to the work of using fundamental models for HOI detection. However, I think an instrive work is to transfer a fundamental model pretrained on lower-level task to a high-level task. This work, however, seems to be the opposite. ", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "If the BLIP2 is a fundamental model pre-trained on a lower-level task than HOI detection (e.g., classification, object detection), I would say this paper is an excellent work  in terms of both model design and performance. Actually, I think this is the first work to transfer a fundamental model into the domain of HOI detection, which may open doors to further exploration on prompting learning  for HOI. However, I cannot accept the choice of using BLIP2 as the fundamental model for HOI detection. I will explain the reasons for this in the  weaknesses below.", "weaknesses": "1. Prompting engineering aims to transfer a fundamental model pretrained on **lower-level tasks** to **higher-level** tasks. At a minimum, **the task for pre-training needs to be decoupled from the downstream task**. Otherwise, transferring a model pertrained on higher-level tasks to low-level tasks is not prompting, but fine-tuning. BLIP2 is a powerful model pertrained for VQA, image captioning, and similar tasks. However,  as widely acknowledged,  HOI detection is a sub-problem for  these detailed scene understanding tasks.  Namely, BLIP2 itself is a powerful HOI detector (I have tried using BLIP2 directly for HOI detection, and the performance is impressive).  From this point, BLIP2 cannot be used as a fundamental model for HOI detection since it has a great capability for HOI detection by itself, and is capable of even higher-level tasks. Therefore, this paper is more like a work that fine-tunes BLIP2 on HICO-DET and V-COCO, at the cost of giving up the ability to use BLIP2 for other tasks, e.g., captioning.\n2. While direct use of BLIP2 for HOI detection may fail to achieve as impressive performance as that of UniHOI on HICO-DET and V-COCO, **BLIP2 has already achieved the goal of doing HOI detection, i.e., detailed scene understanding**.  Therefore, is it a case of putting the cart before the horse to use BLIP2 for HOI detection only?\n3. In a real open-world scenario, I think BLIP2 is more capable of HOI detection compared to UniHOI. After all, the  zero-shot HOI detection capability of UniHOI is mainly inherited from BLIP2.\n4. The performance of UniHOI on HICO-DET and V-COCO is impressive.  I think this is the first work that achieves a mAP  being larger than 40% on HICO-DET. However, the comparison is not so fair. As aforementioned, BLIP2 itself is a powerful HOI detector, which has been pre-trained with a large amount of **interaction-specific** data.  Note that, **on a fair comparison,  a fundamental model should not be pre-trained using data with annotations involving downstream tasks**. Otherwise, the authors need to report the results without using these extra data. For instance, if we first collect all data involving HOI detection from the dataset used for BLIP2 pre-training. Next, we use these data  to pre-train a HOI detector listed in Table 1, (e.g., GEN-VLKT) to get GEN-VLKT-2. Finally, we fine-tune the GEN-VLKT-2 on the HICO-DET and V-COCO. I think it can also get an excellent performance. This is another reason why I think that transferring a model pre-trained on a higher-level task to a lower- level task (especially when the lower-level task is a sub-task of the higher-level task) is not a prompting, but a fine-tuning.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer pXLy,\n\nFirst and foremost, we extend our deepest gratitude for your insightful feedback and hope our clarifications address your concerns. We're eager to highlight the significance and potential of our work.\n\n**1. Regarding BLIP2's Utilization**:\n\nWe agree with your statement about Prompting engineering. However, with the utmost respect, I'd like to disagree with the assertion that \"utilizing BLIP2 is a fine-tuning process on the HOI task.\" Please allow me to present several clarifications:\n\n- **Higher-level vs. Lower-level Task Pre-training**: BLIP2's pretraining revolves around global image-level tasks such as Image-Text Contrastive Learning (ITC), Image-grounded Text Generation (ITG), and Image-Text Matching (ITM). On the other hand, HOI detection involves detailed instance-level tasks like **instance localization** and interaction reasoning. Our UniHOI framework enhances BLIP2's **detection capabilities** beyond its original scope.\n\n- **Fine-tuning vs. Prompting**: Fine-tuning inherently modifies model parameters, as referenced in [1][2]. In our case, all parameters of BLIP2 are **frozen**. We leverage spatial features as prompts for BLIP2 without compromising its other functionalities (e.g. captioning). \n\n- **Motivations**: UniHOI presents a **robust synergy** between a specialized HOI model and a generic VL foundation model. With the foundation model retaining its core capabilities, it significantly enhances the performance of more specialized tasks.\n\n- **Using the Same Foundation Model - CLIP**: As additional evidence, we introduced **CLIP-driven UniHOI** to illustrate that our proposed method still achieves remarkable performance on VL foundation models like CLIP, which has been trained on (image, text) pairs for contrastive learning. Prior research like GEN-VLKT and HOICLIP also explored large model-driven methodologies based on CLIP. We've furnished detailed experiments and outcomes regarding this in the subsequent fourth response.\n\n[1] Visual Prompt Tuning, ECCV 2022.\n\n[2] Visual Tuning, arXiv 2023.\n\n**2. BLIP's ability to handle HOI tasks**:\n\nTo further explore the possibility of using BLIP2 for HOI detection, we employed its Image Encoder to derive image features, yielding a representation of [1024,1408]. We linearly interpolated the positional encodings in BLIP2 to handle larger images and used the Instance Decoder with learnable Queries of [64,256] to identify interacting HO pairs. **Unfortunately, achieving model convergence proved challenging.** Two primary constraints were observed:\n\n- BLIP2's training lacks emphasis on individual localization, limiting its precision in Instance Detection.\n\n- BLIP2's size necessitates a smaller input image. Enlarging this to match typical detection models would produce numerous tokens, adding substantial computational burden during LLM alignment.\n\n**3. Open-world Scenario**:\n\nThank you once again for your careful consideration of our work.\n\n- Firstly, we acknowledge that models like CLIP and BLIP2 exhibit superior adaptability in open-world scenarios, which aligns with our experimental findings. We endeavor to capitalize on these models' strengths while tailoring solutions for the nuanced requirements of HOI detection.\n\n\n- Presently, BLIP2's **text-driven interfacer** hinders nuanced interaction analysis. This limitation is evident when trying to describe complex visuals, such as distinguishing individuals in identical jerseys during a sports event, using just text.\n\n- The surge in large models driving specialized tasks, as seen with our UniHOI or the innovative SAM for image segmentation, indicates a promising direction. With UniHOI, we aim to bolster advancements in high-precision HOI detection.\n\n\n**4. Comparison Fairness**:\n\nBuilding on our earlier discussions, BLIP2's training tasks, namely ITC, ITG, and ITM, are rooted in the \"image-text\" paradigm, is hardly direct training for GEN-VLKT. For a fair comparison with CLIP-based approaches like GEN-VLKT, we introduced a CLIP-based UniHOI. Below, we present the results from our CLIP-driven UniHOI and GEN-VLKT on the VCOCO dataset:\n\n|Method|${AP}^{1}_{role}$|${AP}^{2}_{role}$|\n|:--:|:--:|:--:|\n|GEN-VLKT_s|62.41|64.46|\n|${UniHOI}_s$ (w/ CLIP)|63.79|65.91|\n|GEN-VLKT_m|63.28|65.58|\n|${UniHOI}_m$ (w/ CLIP)|64.47|67.83|\n|GEN-VLKT_l|63.58|65.93|\n|${UniHOI}_l$ (w/ CLIP)|64.86|67.98|\n\nWe also reported the performance of these two methods on the HICO-DET dataset:\n\n|||Default|||Known Obj.||\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|Method|Full|Rare|Non-rare|Full|Rare|Non-rare|\n|GEN-VLKT_s|33.75|29.25|35.10|36.78|32.75|37.99|\n|${UniHOI}_s$ (w/ CLIP)|35.92|34.39|36.26|38.84|37.19|40.18|\n|GEN-VLKT_m|34.78|31.50|35.77|38.07|34.94|39.01|\n|${UniHOI}_m$ (w/ CLIP)|36.71|35.42|36.91|39.16|39.23|40.56|\n|GEN-VLKT_l|34.96|31.18|36.08|38.22|34.36|39.37|\n|${UniHOI}_l$ (w/ CLIP)|36.84|35.71|37.05|39.28|39.31|40.79|\n\nMoreover, CLIP driven UniHOI is also significantly better than GEN-VLKT in zero shot settings:\n\n|Method|Type|Unseen|Seen|Full|\n|:--:|:--:|:--:|:--:|:--:|\n|GEN-VLKT_s|RF-UC|21.36|32.91|30.56|\n|${UniHOI}_s$ (w/ CLIP)|RF-UC|23.41|33.45|31.97|\n|GEN-VLKT_s|NF-UC|25.05|23.38|23.71|\n|${UniHOI}_s$ (w/ CLIP)|NF-UC|26.89|25.57|25.96|\n|GEN-VLKT_s|UO|10.51|28.92|25.63|\n|${UniHOI}_s$ (w/ CLIP)|UO|13.24|30.27|27.52|\n|GEN-VLKT_s|UV|20.96|30.23|28.74|\n|${UniHOI}_s$ (w/ CLIP)|UV|22.18|33.29|30.87|\n\n\nIn our additional experiments utilizing fair CLIP as the foundational model, UniHOI consistently demonstrated impressive results, further attesting to our method's efficacy.\n\n\nFinally, we humbly request you to consider the innovative spirit and potential impact of our work for the broader research community and kindly reconsider our submission. If you have any further questions, please let us know. We’d be very happy to do anything we can that would be helpful in the time remaining! Thanks!"}, {"review_id": "pAAiqHOQ22", "reviewer": "Reviewer_WF8N", "summary": "This paper investigates the problem of human-object interaction (HOI) detection. The authors introduced UniHOI, a method for universal HOI detection in an open-world setting. They also explored the universal interaction recognition with Vision-Language (VL) foundation models and large language models (LLMs), and proposed HO prompt-based learning for high-level relation extraction aimed at VL foundation models.  Experimental results show the effectiveness and significance of the proposed method.", "questions": "Please refer to the weakness section.", "limitations": "Please refer to the weakness section.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Overall, the manuscript is well-written and easy to follow. The figures are pretty and can convey the concepts clearly.\n2. Pushing the problem of human-object interaction detection toward an open-world setting is of great importance. This is also a trend for most existing computer vision applications.\n3. The extensive experimental results show the superiority of the proposed UniHOI method.", "weaknesses": "1. In Table 4, the results from the third row come from \"ConsNet [31]\" but not \"ATL [15]\", according to the paper of \"GEN-VLKT\".\n2. The conclusion part lacks objective reflections on the deficiencies of this study and future prospects for improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer WF8N,\n\nWe greatly appreciate your thoughtful review and the time you have taken to provide insights and feedback on our submission. We are encouraged by the positive aspects you've highlighted and grateful for the critical points you've raised. Here, we address the weaknesses mentioned and hope to provide clarifications that would enhance the clarity of our contribution.\n\n**Reference to Table 4 Results**\nWe apologize for the oversight. You are right. The result in the third row of Table 4 should reference \"ConsNet [31]\" and not \"ATL [15]\". We have duly noted this mistake and will rectify it in the final version of our manuscript. We understand that accuracy in presenting prior works is paramount, and we thank you for pointing out this error.\n\n**Reflections on the Study's Deficiencies and Future Directions**\nThank you for your constructive feedback. In light of your feedback, we will expand our conclusion section to incorporate the following points:\n\n1. **Deficiencies**: We acknowledge that while our UniHOI method has shown promise in various experimental settings, it also faces the issue of ***substantial computational demands*** when driving large models.\n2. **Future Prospects**: Building on our current findings, there are several promising directions: (1) At present, the focus of our UniHOI is mainly on ***Prompt Learning***. In the future, we will consider paradigms such as ***Adapter*** and ***Parameter-Efficient Fine-Tuning*** to further enhance the generic Visual-Language (VL) foundation models' capabilities in the specialized Human-Object Interaction (HOI) domain. (2) We will explore better ***knowledge transfer*** methods in the future to more efficiently implement the driving or collaboration of specialized HOI detectors by VL base models, thereby reducing the inference costs of large models.\n\nIn summary, we sincerely thank you for your valuable comments. We believe that by addressing these points, our work will be significantly improved and provide a solid contribution to the community. We hope our responses provide clarity, and we remain open to further feedback. Thanks!\n"}, {"review_id": "PnkkCJ3sUL", "reviewer": "Reviewer_djfM", "summary": "In view of the limited scalability and the suboptimal zero-shot performance of current HOI detection methods, the authors propose a novel method for HOI detection based on VL foundation models.\nWith in-depth analysis and adaptation of HOI detectors, the foundation model is effectively adopted to reason about HOI relationships based on human/object tokens.\nFurthermore, LLM is adopted as a knowledge base to diversify HOI descriptions, enabling open-vocabulary HOI detections.\nWith the VL foundation model and LLM, extraordinary HOI detection performance is achieved upon both conventional and zero-shot setting.", "questions": "- The description on the adopted Image Encoder and Instance Decoder is not clear. Is DETR adopted? Or some more advanced detectors? Are they frozen during training?\n\n- In L177, $V^f$ and $V^f$ seems to be a typo.\n\n- The performance of UniHOI-l in Tab. 1-2 is questionable. Please check whether there are typos.\n\n- In the ablation studies, it is still not very clear that how the VL Foundation model is simply added. Is it a simple removal of HO Spatial Prompting? Or replacing the input feature of HO Spatial Prompting with the learnable queries?\n\n- Is it possible to make the HOI detector share the backbone of VL foundation model? This could be related to the proposed insight that HOI detector is a three-tier visual feature hierarchy. Ablation studies on this would be preferrable.\n\n- The proposed method seems to be applicable to arbitrary HOI detectors (if the answer to Q1 is yes). Is it practical?\n\n- Results in Tab. 5 and Tab. 3 are not consistent. The result without Knowledge Retrieval is reported in Tab. 3.", "limitations": "Limitations of the paper is not well-discussed in the paper. \nI would like to see more discussion on extending the use of LLM further than a static knowledge base.\nAlso please refer to the weakness part.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed HO prompt-guided decoder is brilliant in addressing the feature alignment issue.\n\nAdopting GPT as knowledge base is an interesting idea to incorporate the recent progress in LLM with HOI detection.\n\nThe performance is amazing with significant margins upon previous SOTAs, especially for zero-shot setting.\n\nExtensive experiments are conducted, providing valuable insights on the effect of VL foundation models in HOI detection.\n\nIn the wild HOI detection illustration is quite impressive.\n", "weaknesses": "The comparison between GEN-VLKT and the proposed method is not totally fair to me. It might be better to replace BLIP-2 with CLIP for a fair comparison.\n\nFig. 2 is not very clear. It would help if annotating the encoders in the figure with corresponding notations. \n\nThe baseline of ablation experiments is chosen as GEN-VLKT. However, a major difference between GEN-VLKT and UniHOI is the VL foundation model used. And there are also other differences, like VLKT is not used for UniHOI. It might be better to change the baseline to make the ablation more reasonable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer djfM,\n\nWe truly appreciate the detailed feedback. Herein, we provide a detailed response to each of your concerns:\n\n**On Weaknesses**:\n\na. **Comparison with GEN-VLKT**: In response, we replace BLIP2 with CLIP and conducted experiments on V-COCO datasets:\n\n|Method|${AP}^{1}_{role}$|${AP}^{2}_{role}$|\n|:-:|:-:|:-:|\n|GEN-VLKT_s|62.41|64.46|\n|${UniHOI}_s$ (w/ CLIP)|63.79|65.91|\n|${UniHOI}_s$ (w/ BLIP2)|65.58|68.27|\n|GEN-VLKT_m|63.28|65.58|\n|${UniHOI}_m$ (w/ CLIP)|64.47|67.83|\n|${UniHOI}_m$ (w/ BLIP2)|67.95|70.61|\n|GEN-VLKT_l|63.58|65.93|\n|${UniHOI}_l$ (w/ CLIP)|64.86|67.98|\n|${UniHOI}_l$ (w/ BLIP2)|68.05|70.82|\n\nThe following table shows the results of UniHOI equipped with different foundation methods on the HICO-DET dataset:\n\n|||Default|||Known Obj.||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Method|Full|Rare|Non-rare|Full|Rare|Non-rare|\n|GEN-VLKT_s|33.75|29.25|35.10|36.78|32.75|37.99|\n|${UniHOI}_s$ (w/ CLIP)|35.92|34.39|36.26|38.84|37.19|40.18|\n|${UniHOI}_s$ (w/ BLIP2)|42.73|42.03|42.93|43.58|45.27|43.08|\n|GEN-VLKT_m|34.78|31.50|35.77|38.07|34.94|39.01|\n|${UniHOI}_m$ (w/ CLIP)|36.71|35.42|36.91|39.16|39.23|40.56|\n|${UniHOI}_m$ (w/ BLIP2)|43.18|42.98|43.57|44.32|45.96|43.89|\n|GEN-VLKT_l|34.96|31.18|36.08|38.22|34.36|39.37|\n|${UniHOI}_l$ (w/ CLIP)|36.84|35.71|37.05|39.28|39.31|40.79|\n|${UniHOI}_l$ (w/ BLIP2)|43.57|43.79|44.01|44.78|46.48|44.32|\n\nAdditionally, the results under zero-shot setting are as follows:\n\n|Method|Type|Unseen|Seen|Full|\n|:-:|:-:|:-:|:-:|:-:|\n|GEN-VLKT_s|RF-UC|21.36|32.91|30.56|\n|${UniHOI}_s$ (w/ CLIP)|RF-UC|23.41|33.45|31.97|\n|${UniHOI}_s$ (w/ BLIP2)|RF-UC|28.68|33.16|32.27|\n|GEN-VLKT_s|NF-UC|25.05|23.38|23.71|\n|${UniHOI}_s$ (w/ CLIP)|NF-UC|26.89|25.57|25.96|\n|${UniHOI}_s$ (w/ BLIP2)|NF-UC|28.45|32.63|31.79|\n|GEN-VLKT_s|UO|10.51|28.92|25.63|\n|${UniHOI}_s$ (w/ CLIP)|UO|13.24|30.27|27.52|\n|${UniHOI}_s$ (w/ BLIP2)|UO|19.72|34.76|31.56|\n|GEN-VLKT_s|UV|20.96|30.23|28.74|\n|${UniHOI}_s$ (w/ CLIP)|UV|22.18|33.29|30.87|\n|${UniHOI}_s$ (w/ BLIP2)|UV|26.05|36.78|34.68|\n\nThese results demonstrate that **regardless of whether we use CLIP or BLIP2, our approach significantly outperforms the current state-of-the-art methods**.\n\nb. **Clarification on Fig. 2**: We will add annotations to all the components in Fig. 2 to ensure clarity.\n\nc. **Baseline in Ablation Experiments**: Based on your suggestion, we adopted a GEN that doesn't use CLIP model as our baseline and performed ablation experiments by equipping BLIP2. The results are as follows:\n|Method|${AP}^{1}_{role}$|${AP}^{2}_{role}$|\n|:-:|:-:|:-:|\n|baseline|61.58|63.59|\n|+ BLIP2|62.91|64.83|\n|+ HOPD|65.58|68.27|\n|+ Knowledge Retrieval|66.74|69.31|\n\n Furthermore, we also conduct ablation experiments using CLIP as the VL foundation model:\n\n|Method|${AP}^{1}_{role}$|${AP}^{2}_{role}$|\n|:-:|:-:|:-:|\n|baseline|61.58|63.59|\n|+ CLIP|62.15|64.28|\n|+ HOPD|63.79|65.91|\n|+ Knowledge Retrieval|63.98|66.34|\n\n**On Questions**:\n\na. **Image Encoder & Instance Decoder**: We employed the image encoder and instance decoder from the single-stage HOI detector GEN, allowing their weights to adapt during training, while the weights of the VL foundation model remain frozen.\n\nb. **Typos in L177**: We deeply regret the oversight. These errors will be addressed in the revised manuscript.\n\nc. **Performance of UniHOI-l in Tab. 1-2**: In Table 1, UniHOI-l reports a mAP of 43.79 in the Rare category under the Default Setting, showing a \"12.61\" improvement over GEN-VLKT-l. However, there was an oversight where we mistakenly wrote \"2.61\". We will correct this mistake.\n\nd. **Clarification on Ablation Studies**: We use the BLIP2 to produce a representation $X^q$ of dimension [32,768]. Then, a two-layer Transformer Decoder processes a learnable query [64,256] to derive the feature $V^q$ from $X^q$. This is then combined with the output $V^i$ from the Interaction Decoder for our prediction. However, without UniHOI's spatial feature prompting, the feature alignment between $V^q$ and $V^i$ is suboptimal.\n\ne. **Sharing Backbone of VL Foundation Model**: Thanks for the thoughtful suggestion. We employ its Image Encoder to derive image features, yielding a representation of [1024,1408]. We linearly interpolate the positional encodings in BLIP2 to handle larger images and used the Instance Decoder with learnable Queries of [64,256] to identify interacting HO pairs. **Unfortunately, achieving model convergence proved challenging.** Two primary constraints were observed:\n\n- BLIP2's training lacks emphasis on individual localization, limiting its precision in Instance Detection.\n\n- BLIP2's size necessitates a smaller input image. Enlarging this to match typical detection models would produce numerous tokens, adding substantial computational burden during LLM alignment.\n\nf. **Applicability to Arbitrary HOI Detectors**: Our UniHOI is universally adaptable. Quality spatial information from any detector can prompt the VL model to extract advanced features, enhancing versatility across multiple HOI detectors and domains. In the future, we will try to release open source code that is compatible with classic HOI detectors, such as UPT (Unary–Pairwise Transformer). \n\ng. **Consistency in Tab. 5 and Tab. 3**: \n\n- In Table 3, we solely showcased the results without the utilization of knowledge retrieval, which has already achieved substantial performance enhancements.\n\n- In this paper, we treated knowledge retrieval as an ancillary mechanism to elevate the model's performance, specifically aiding in comprehending intricate interactions.\n\n- We commit to sharing the relevant code, weights, and results to support further exploration of knowledge-based techniques.\n\n**On Limitations**: We plan to enhance our conclusion, addressing potential areas such as model compression, knowledge representation, parameter-efficient tuning, and multi-turn dialogues.\n\nWe genuinely appreciate your time and expertise. If you have any further questions or concerns, please let us know. Thanks!"}, {"review_id": "s0gZ6xa4Ly", "reviewer": "Reviewer_X6bU", "summary": "This paper proposes a framework for human-object interaction (HOI) detection that leverages vision-language foundation models and large language models to achieve universal and flexible recognition of complex interactions in images. The framework, named UniHOI, consists of three main components: a visual HOI detector that extracts three levels of features from images, a HO prompt-guided decoder that queries the foundation model for high-level relation representations associated with human-object pairs, and a knowledge retrieval module that uses a large language model to generate descriptive texts for interaction categories. The framework supports both supervised and zero-shot settings, and can handle any textual input for open-category interaction detection. The paper demonstrates the effectiveness and superiority of UniHOI over existing methods on two public benchmarks, HICO-DET and V-COCO, as well as in the wild scenarios.", "questions": "Please refer to weaknesses.", "limitations": "There is no discussion on limitations or failure cases.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The performance is quite impressive.\n- Leveraging LLM and foundational models to augment CV tasks is the future, and this work gives a try to use them simultaneously.\n- The authors promise to release the code to ensure reproducibility.", "weaknesses": "1. There is no ablative study of each component (perhaps only one component, i.e., HO prompt decoder) under the close-set setup.\n2. This work uses BLIP2 with ViT-L while existing work like GEN-VLKT typically uses CLIP with ViT-B. It is evident that the former is much more powerful. Could you provide the performance on HICO-DET using HO prompt-based learning with CLIP ViT-B under the close-set setup? As the improvement may be brought by more advanced large visual-language pre-trained models.\n3. How about the inference speed? As shown in Table 8, **three** times longer than GEN-VLKT. Note that existing work like GEN-VLKT does not involve the computation of large visual-language pre-trained models at the inference stage, since all of the feature of objects or verbs is pre-computed. However, in this work, the feature for prompting should be computed for each image individually. Considering the extremely large backbone (e.g, ViT-L), there would be a heavy burden at inference.\n4. The core contribution of this work is actually the HO prompt-based decoder. However, there is nothing novel with it, i.e., directly using spatial location to get foundational model output features.\n5. Knowledge retrieval is solely used in open-world setup, is it possible for it to augment the closed-world setup?\n\nOverall, this is technically solid work, and LLM for knowledge retrieval is interesting. But the comparison is unfair (i.e., a much more powerful visual-language pre-trained model), the inference time is unacceptable, and the novelty of the prompt-based decoder is limited. I will be very happy to update my score if the authors can address my concerns above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer X6bU, \n\nFirst and foremost, we'd like to express our gratitude for your comprehensive review and insightful comments. We acknowledge the concerns you've raised and will attempt to address them point by point:\n\n\n**Ablations**: We appreciate the importance of ablation studies to establish the individual contributions of each component. We would like to kindly draw your attention to Table 5, which presents our ablation experiments conducted under the closed-set setup, and to Table 6, which depicts results from the open-set setup.\n\n\n**Model Comparison with GEN-VLKT using CLIP ViT-B**: Thank you for the keen observation regarding the comparative power of BLIP2 with ViT-L and CLIP with ViT-B. We wholeheartedly agree on the importance of a fair comparison. In line with this, we have conducted experiments using CLIP ViT-B not just in the close-set setup, but also in the open-set setup. We have conducted experiments on V-COCO datasets:\n\n|Method|${AP}^{1}_{role}$|${AP}^{2}_{role}$|\n|:-:|:-:|:-:|\n|GEN-VLKT_s|62.41|64.46|\n|${UniHOI}_s$ (w/ CLIP)|63.79|65.91|\n|${UniHOI}_s$ (w/ BLIP2)|65.58|68.27|\n|GEN-VLKT_m|63.28|65.58|\n|${UniHOI}_m$ (w/ CLIP)|64.47|67.83|\n|${UniHOI}_m$ (w/ BLIP2)|67.95|70.61|\n|GEN-VLKT_l|63.58|65.93|\n|${UniHOI}_l$ (w/ CLIP)|64.86|67.98|\n|${UniHOI}_l$ (w/ BLIP2)|68.05|70.82|\n\nThe following table shows the results of UniHOI equipped with different foundation methods on the HICO-DET dataset:\n\n|||Default|||Known Obj.||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Method|Full|Rare|Non-rare|Full|Rare|Non-rare|\n|GEN-VLKT_s|33.75|29.25|35.10|36.78|32.75|37.99|\n|${UniHOI}_s$ (w/ CLIP)|35.92|34.39|36.26|38.84|37.19|40.18|\n|${UniHOI}_s$ (w/ BLIP2)|42.73|42.03|42.93|43.58|45.27|43.08|\n|GEN-VLKT_m|34.78|31.50|35.77|38.07|34.94|39.01|\n|${UniHOI}_m$ (w/ CLIP)|36.71|35.42|36.91|39.16|39.23|40.56|\n|${UniHOI}_m$ (w/ BLIP2)|43.18|42.98|43.57|44.32|45.96|43.89|\n|GEN-VLKT_l|34.96|31.18|36.08|38.22|34.36|39.37|\n|${UniHOI}_l$ (w/ CLIP)|36.84|35.71|37.05|39.28|39.31|40.79|\n|${UniHOI}_l$ (w/ BLIP2)|43.57|43.79|44.01|44.78|46.48|44.32|\n\nAdditionally, the results under zero-shot setting are as follows:\n\n|Method|Type|Unseen|Seen|Full|\n|:-:|:-:|:-:|:-:|:-:|\n|GEN-VLKT_s|RF-UC|21.36|32.91|30.56|\n|${UniHOI}_s$ (w/ CLIP)|RF-UC|23.41|33.45|31.97|\n|${UniHOI}_s$ (w/ BLIP2)|RF-UC|28.68|33.16|32.27|\n|GEN-VLKT_s|NF-UC|25.05|23.38|23.71|\n|${UniHOI}_s$ (w/ CLIP)|NF-UC|26.89|25.57|25.96|\n|${UniHOI}_s$ (w/ BLIP2)|NF-UC|28.45|32.63|31.79|\n|GEN-VLKT_s|UO|10.51|28.92|25.63|\n|${UniHOI}_s$ (w/ CLIP)|UO|13.24|30.27|27.52|\n|${UniHOI}_s$ (w/ BLIP2)|UO|19.72|34.76|31.56|\n|GEN-VLKT_s|UV|20.96|30.23|28.74|\n|${UniHOI}_s$ (w/ CLIP)|UV|22.18|33.29|30.87|\n|${UniHOI}_s$ (w/ BLIP2)|UV|26.05|36.78|34.68|\n\n\nOur experiments demonstrate that **regardless of whether we use CLIP or BLIP2, our approach significantly outperforms the current state-of-the-art methods**.\n\nIn addition, one of the motivations we chose for BLIP is that **the training text for CLIP is too simple**, such as \"an image of an apple\". **We are concerned that the knowledge in CLIP may not be sufficient for challenging HOI detection**. In order to explore the potential of the latest large models, the default UniHOI has chosen BLIP2, which is pre trained with richer text.\n\n**Inference Speed**: Thank you for highlighting the computational aspects of UniHOI. In response:\n\n- We've mitigated computational demands by adjusting the image resolution for the VL foundation model, striking a balance between efficiency and performance.\n\n- For practical applications, we're exploring techniques like quantization to optimize inference speed without sacrificing results.\n\n- Addressing the challenges posed by large models remains an ongoing effort, with potential solutions in architectural refinements or improved knowledge transfer methods.\n\n\n**Novelty of HO Prompt-based Decoder**: Thank you for drawing attention to the novelty. While the core concept may appear direct, the effectiveness lies in its integration and synergy with other components. Here's a deeper dive into the innovative aspects of our approach:\n\n- UniHOI is the pioneering approach using prompt learning for VL foundation models in HOI detection, surpassing state-of-the-art methods in both supervised and zero-shot settings. This sets a more reliable benchmark for harmonizing specialized HOI detectors with large models.\n \n- UniHOI efficiently extracts spatial representations of humans and objects from professional HOI detectors. This enables us to conveniently and accurately capture instance-level features from large models. Previous approaches (e.g., GEN-VLKT, HOICLIP) driven by large models predominantly aligned only with image-level features, overlooking this critical facet.\n\n- UniHOI unleashes the potential of VL foundation models and LLMs (e.g., GPT-4), potentially inspiring future endeavors to explore more universal and intelligent HOI detectors.\n\n- The elegance of our core concept is in its simplicity and directness, making it adaptable across various VL foundation models.\n\n\n**Knowledge Retrieval in Closed-World Setup**: Thank you for the insightful suggestion. In response, we have added experiments on the Knowledge Retrieval (KR) in the closed-world setup. The results are as follows:\n\n|Method|${AP}^{1}_{role}$|${AP}^{2}_{role}$|\n|:-:|:-:|:-:|\n|GEN-VLKT_s|62.41|64.46|\n|${UniHOI}_s$|65.58|68.27|\n|${UniHOI}_s$+KR|66.12|68.73|\n|GEN-VLKT_m|63.28|65.58|\n|${UniHOI}_m$|67.95|70.61|\n|${UniHOI}_m$+KR|68.59|71.13|\n|GEN-VLKT_l|63.47|65.93|\n|${UniHOI}_l$|68.05|70.82|\n|${UniHOI}_l$+KR|69.39|71.24|\n\nThe results indicate that **knowledge retrieval can also promote model performance in Closed-World scenarios**. This is because more detailed textual description information can promote a deeper understanding of interactions.\n\nWe genuinely appreciate your time and expertise. If you have any further questions, please let us know. We’d be very happy to do anything we can that would be helpful in the time remaining! Thanks!"}]

Author rebuttal:

Meta-review: This paper presents work on human-object interaction detection.  The main contribution of the paper centers around vision-language foundation models in a prompting setup.  The reviewers appreciated the prompt-guided decoder, the leveraging of language data, and impressive empirical results.

In the rebuttal and subsequent discussion, the authors provided additional empirical results and clarifications on the novelty of the method.  Concerns were raised over the perceived use of higher-level tasks (generic language data) for relatively low-level tasks (human-object interactions).  While this is a valid point, the contributions of this paper remain intact.  In particular, the generic nature of the human-object interactions that can be detected using the proposed method, and the impressive empirical results are the reason for the accept decision.  This paper is a substantive contribution to the literature on human-object interaction detection.
Decision: Accept (poster)