Title: Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Human-object interaction

Commonsense Knowledge

Foundation Models
Abstract: Human-object interaction (HOI) detection aims to comprehend the intricate relationships between humans and objects, predicting <human, action, object> triplets, and serving as the foundation for numerous computer vision tasks. The complexity and diversity of human-object interactions in the real world, however, pose significant challenges for both annotation and recognition, particularly in recognizing interactions within an open world context. This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs). The proposed method is dubbed as UniHOI. We conduct a deep analysis of the three hierarchical features inherent in visual HOI detectors and propose a method for high-level relation extraction aimed at VL foundation models, which we call HO prompt-based learning. Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. Furthermore, we utilize a LLM (i.e. GPT) for interaction interpretation, generating a richer linguistic understanding for complex HOIs. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence.  Our efficient architecture design and learning methods effectively unleash the potential of the VL foundation models and LLMs, allowing UniHOI to surpass all existing methods with a substantial margin, under both supervised and zero-shot settings. The code and pre-trained weights will be made publicly available.

Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models

Abstract

Human-object interaction (HOI) detection aims to comprehend the intricate relationships between humans and objects, predicting < human,action,object > triplets, and serving as the foundation for numerous computer vision tasks. The complexity and diversity of human-object interactions in the real world, however, pose significant challenges for both annotation and recognition, particularly in recognizing interactions within an open world context. This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs). The proposed method is dubbed as UniHOI . We conduct a deep analysis of the three hierarchical features inherent in visual HOI detectors and propose a method for high-level relation extraction aimed at VL foundation models, which we call HO prompt-based learning. Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. Furthermore, we utilize a LLM ( i.e. GPT) for interaction interpretation, generating a richer linguistic understanding for complex HOIs. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence. Our efficient architecture design and learning methods effectively unleash the potential of the VL foundation models and LLMs, allowing UniHOI to surpass all existing methods with a substantial margin, under both supervised and zero-shot settings. The code and pre-trained weights are available at: https://github.com/Caoyichao/UniHOI .

1 Introduction

Human-object interaction (HOI) detection [15, 7] is a burgeoning field of research in recent years, which stems from object detection and makes higher demands on high-level visual understanding. It plays a crucial role in many vision tasks, e.g., visual question answering, human-centric understanding, image generation, and activity recognition, to name a few representative ones. An excellent HOI detector must accurately localize all the interacting Human-Object (HO) pairs and recognize their interaction, typically represented as an HOI triplet in the format of < human,action,object > .

Despite the optimistic progress made by various HOI detectors [27, 12, 18, 49, 14, 29, 8] in recent years, numerous challenges still persist [45]. Naively mapping diverse interactions to one-hot labels is an exceedingly labor-intensive and costly process, which is prone to the loss of semantic information.

[1, 0, … 0, 0]

[0, 0, … 1, 0]

Pre-determinate

One-hot Labels

HOI Detector

GPTs-as-KB

Traditional HOI Detection

VL Foundation

Model

<human, talk on, human>

<human, wash, elephant>

Annotation

Texts

HOI Detector

Previous HOI Vision-and-Language

Modeling (HOI-VLM)

Limited Knowledge

Transfer

HOI Detector

Our Universal HOI Detector (UniHOI)

Unlimited Linguistic

Knowledge

HO Prompts

Any Interaction

Relationship

Language

Model

HOI Knowledge

Retrieval

Category

Texts

OR

Flexible Text Input

As the number and complexity of interaction categories increase, model optimization becomes increasingly difficult, and the universality of the model is consequently restricted.

Recently, multi-modal learning, especially Vision-and-Language (VL) learning, has become remarkably popular, achieving far-reaching success across a variety of domains [1]. Multi-modal learning is capable of portraying multidimensional information in a more comprehensive manner by describing the same entity or spatio-temporal event through different modalities. Recently, a number of studies have endeavored to employ VL models for the HOI task with the aim of constructing transferable HOI detectors [52, 19, 28, 43, 45, 5]. For instance, PhraseHOI [28] harnessed a pre-trained word embedding model to generate a fixed-length linguistic representation for the HOI task. Unfortunately, these methods harness cross-modal knowledge in an overly constrained manner, falling short of fully exploiting the potential of cross-modal knowledge and LLMs within the HOI detection domain.

Upon rigorous examination, we identify several intrinsic limitations in the existing methodologies: (1) Limited scalability: Model training is overly reliant on annotated data, which restricts them to a finite set of categories. (2) Suboptimal adaptability in zero-shot settings: Even HOI-VLM approaches can only harness a limited number of word embeddings to accommodate unseen categories, thereby curtailing their adaptability. (3) Inability to quickly comprehend complex interactive behaviors from succinct descriptions, as humans can, indicating a dearth of a more universal and flexible framework for HOI detection.

Considering the recent outstanding performance of VL foundation models and LLMs in various fundamental tasks, in this paper, we attempt to harness the capabilities of such large models to facilitate more general HOI detector, as shown in Figure 1. We conduct an in-depth analysis of the clear three-tier visual feature hierarchy in visual HOI detectors, effectively utilizing the foundation models to boost the understanding of high-level interactive semantics within images. The employment of large models bridges the gap between textual and visual modalities, making a knowledge-based universal HOI detector feasible. In summary, the contributions of this paper are as follows:

We propose the first visual-textual foundation model-based framework for HOI detection (UniHOI), which significantly improves the accuracy and universality compared to previous HOI detectors that solely rely on training from specific datasets. Our UniHOI method is expected to guide the HOI detection field into a new research phase.

We treat the visual HOI detector as a three-tier visual feature hierarchy and appropriately design an efficient human-object prompt learning method, enabling visual HOI detectors to effectively query interaction tokens associated with specific HO pairs in foundation models, and thus obtain a more universal HO relationship representations.

We devise a knowledge-based zero-shot HOI recognition method that leverages large models such as GPT to generate descriptive knowledge for specific interaction categories, guiding visual detectors in zero-shot recognition of the corresponding categories. This approach

breaks free from the limitations of traditional zero-shot recognition methods, which can only learn from word embeddings, and offers insights for a more universal open-category HOI recognition system.

2 Related Work

Generic HOI Detection. Based on their architectural design, existing HOI (Human-Object Interaction) detection approaches can be broadly categorized into two groups: two-stage methods [49, 7, 12, 13, 27, 18], and one-stage methods [14, 20, 53, 29, 8]. One-stage methods typically adopt a multitask learning strategy to simultaneously execute instance detection and interaction relation modeling [29, 47, 30]. Conversely, two-stage methods initially conduct object detection, followed by interaction relation modeling for all candidate HO pairs. By fully exploiting the capabilities of each module, two-stage methods have showcased superior detection performance [47]. While these methods have significantly propelled the progress of early HOI detection, they still grapple with the issue of limited generality.

Language Semantics for Vision. Driven by the notable success of large-scale models such as GPT-4 [36], the incorporation of language semantics to augment vision models has recently surfaced as a promising avenue in the realm of computer vision tasks [38, 41, 1, 50]. Among these, Visionand-Language Pre-training (VLP) [34, 24] has evolved into a popular paradigm across numerous vision-and-language tasks, owing to its ability to learn generalizable multimodal representations from extensive image-text data [1, 9, 10]. These methods have recently found applications in multimodal retrieval [11], vision-and-language navigation [2], among others [4]. Nevertheless, implementing effective prompt-based learning on base models, particularly extracting high-order relationships in intricate scenarios, remains crucial for the successful deployment of large models. Thus, investigating how to efficaciously extract specific information from large models is of utmost importance.

HOI Vision-and-Language Modeling (HOI-VLM). Despite the moderate success of previous HOI detectors [47, 49, 31], these often perceive interactions as discrete labels, overlooking the rich semantic text information encapsulated in triplet labels. Of late, a handful of researchers [52, 19, 28, 43, 45, 46] have delved into HOI Vision-and-Language Modeling in an effort to further enhance HOI detection performance. Among these, [52], [19], and [46] have primarily focused on incorporating language prior features into HOI recognition. Meanwhile, RLIP [45] and [43] have proposed the construction of a transferable HOI detector via the VLP approach. Serving as applications and extensions of Vision-and-Language learning within the HOI domain, these HOIVLM methodologies strive to comprehend the content and relations between visual interaction features and their respective triplet texts. Nevertheless, these methodologies harness cross-modal knowledge in a markedly restricted manner, falling short of fully unleashing the potential of crossmodal knowledge and LLMs within the HOI domain.

3 Method

3.1 Overview

In traditional HOI detection task, the objective is typically to optimize a detector Φ θ D , such as the most commonly-used Transformer-based detectors in recent years. By inputting a query Q ho related to the Human-Object (HO) pairs, the HOI detector learns the location of the HO pairs and corresponding interactions in images. This optimization objective can be formulated as follows:

where GT and L are ground-truth labels and overall loss function respectively, X I = { ( I i ) } |X| i =1 denotes the HOI image dataset, I represents the input image, and θ indicates the weights in HOI detector Φ θ D . Thus, traditional HOI detectors rely on purely visual detection tasks for optimization.

In this work, our aim is to utilize superior VL foundation models and additional interaction interpretation T during the training and inference process. In this way, we can improve the optimization objective as follows:

Instance-level Learning

CNN

Image

Encoder

Sequenced Embeddings

𝒳𝒳

𝑝𝑝

…

Instance

Decoder

…

Human & Object Queries

Positional

Encoding

…

…

Human

Boxes

Object Boxes

& Categories

Interaction

Decoder

…

…

Kicking a football requires planting your

non-kicking foot beside the ball, slightly

ahead.

Lean

back

for

power,

extend

your kicking leg, make contact with the

ball's center using your foot's instep …

ℳ

𝓔𝓔

𝑝𝑝𝑝𝑝𝑝𝑝

𝑣𝑣

𝐼𝐼

𝒬𝒬

ℎ

𝒬𝒬

𝑝𝑝

𝒬𝒬

𝑔𝑔

𝒱𝒱

𝑓𝑓

𝒱𝒱

𝑖𝑖

ℳ 𝑝𝑝

ℬ ℎ

ℬ 𝑝𝑝

,

𝒞𝒞

𝑝𝑝

HO pair-specific

Features

Knowledge Retrieval for

'Human kick football'

HO Prompt-

Guided Decoder

Vision

Module

Language

Module

HO

Prompts

LLM

Annotation Texts

<

ℎ𝑢𝑢𝑚𝑚𝑎𝑎𝑛𝑛

,

𝑘𝑘𝑖𝑖𝑐𝑐𝑘𝑘

,

>

𝒳𝒳

𝑞𝑞

F

F

N

VL Foundation Model

𝐼𝐼

′

…

ℬ ℎ

ℬ 𝑝𝑝

,

𝒞𝒞

𝑝𝑝

𝒫𝒫

ℎ𝑝𝑝

or

𝒯𝒯

Similarity

HOI Predictions

Textual

Features

ℳ 𝑝𝑝

where Φ θ F denotes the Vision-Language (VL) foundation models and P ho represents the HOIspecific prompts for Φ θ F . X = { ( I i , T i ) } |X| i =1 is an image-text corpus, where I i denotes the i -th input image and T i represents the phrase annotations ( e.g. 'Human ride bicycle') in I i . In zero-shot setting, we also use knowledge retrieval to enrich the texts T . Details will be described in Section 3.4.

In HOI detection, the detector not only needs to accurately locate the position of the interacting HO pair, but also needs to reliably model the interaction relationship occurring in the HO pair. But these two tasks have different requirements on the image feature level: the former focuses more on the edges and contours of instances, while the latter focuses more on high-level relational cues. To present and analyze this complicated process more clearly, we explicitly divide the feature learning in HOI detector into three stages: basic visual feature extraction , instance-level feature learning , and high-level relationship modeling , as shown in Figure 2. The basic visual feature extraction is usually performed by the backbone ( e.g. CNN). For the instance-level feature learning, we follow previous methodologies and design prompts related to HO locations to carry and transfer the position information, then apply the Hungarian algorithm for matching with the GT , and compute the loss of the bounding box. The third stage, which is the key focus of this study, involves using spatial location prompts related to the HO pairs to conduct targeted prompt learning from the visual model. The details of each component are explained in the following sections.

3.2 HOSpatial Prompts Generation

We follow the Transformer-based detection architectures [6] to conduct basic visual feature extraction and instance detection. Fed with an input image I ∈ R H × W × C , the CNN generates a feature map X v ∈ R h × w × c . Then, X v is compressed by a projection convolution layer with a kernel size 1 × 1 . Next, a flatten operator is used to segment patch embeddings { x v 1 , x v 2 , . . . , x v N v } , where N v is the number of patch embeddings. The patch embeddings { x v 1 , x v 2 , . . . , x v N v } is then linearly projected through a linear transformation E v ∈ R c × D v . Thus, the input for instance detection are calculated via summing up the projected embeddings and position embeddings E v pos ∈ R N v × D v :

After patching and sequencing the features extracted by CNN, we follow previous methods to perform self-attention on X p : M = Φ θ IE ( X p ) ∈ R N v × D v . This process is implemented by a Transformer encoder Φ θ IE , which is composed of N Transformer encoder layers.

For instance-level feature learning, we deploy an instance decoder Φ θ ID to locate the positions of humans and objects. Based on the M , humans and objects are detected through the human query set

Q h ∈ R N q × D v and the object query set Q o ∈ R N q × D v individually. Additionally, a position guided embedding [30] Q g ∈ R N q × D v is added to assign the human and object queries at the same position as a pair. The calculation process of instance decoder Φ θ ID can be formulated simply as follows:

where the P ho is the spatial tokens highly correlated with HO pair location information, the N q is the number of queries. With instance detection head FFN s , we can calculate the human bounding boxs B h ∈ R N q × 4 from P h , object bounding boxs B o ∈ R N q × 4 and object categories C o ∈ R N q × N c from P o :

where the feature tokens [ P h , P o ] are excellent spatial position features for HO pairs. Driven by prompt learning methods, we explore using the HO position features here to perform prompt learning on the VL foundation models.

3.3 Prompting Foundation Models for HOI Modeling

Considering the excellent performance of BLIP2 [23] in cross-modal tasks and the fact that its Transformer architecture also integrates well with our HOI framework, in this work we use BLIP2 as the VL foundation model to support the recognition process of HOI. However, accurately extracting HOI related features from VL foundation model is the key to successfully applying VL to HOI tasks.

To reduce the computational cost of the VL foundation model, we first downsample the input image I to I ′ with dimension W ′ × H ′ , and then use an image encoder Φ θ I to convert downsampled image into a feature map X f ∈ R w ′ × h ′ × c ′ . We then employ Q-Former Φ θ Q in BLIP2 [23] to learn higher-level content X q ∈ R N f × D f in image:

where the Q f denotes the query in Φ θ I . These modules in Φ θ I are more capable of understanding content in images than regular visiual detectors due to extensive training on a wider range of data.

Then, we consider how to extract the complex interaction representations between specific humans and objects in foundation model. To associate the higher-level information in foundation model with HOpairs, we design a HO Prompt-guided Decoder (HOPD) Φ θ P to correlate the position information of HO pairs with feature tokens in foundation model:

where the V f denotes the high-level information tokens corresponding to HO pairs. In this manner, the HOI-specific tokens is generated under the guidance of the guidance of the human-object query pairs. This spatial prompts act as query embeddings in HOPD, interact with each other through self-attention layers, and interact with frozen image features through cross-attention layers (inserted every other transformer block in HOPD). In this way, these spatial prompts are fed into HOPD, which in turn goes on to query information about the relationships between their respective corresponding HO pairs. In addition, interaction decoding is performed in the visual HOI detector:

Finally, we concatenate V f and V f as the visual representations of the HO pairs and feed them to a FFN to predict the interaction category. For model training, we adhere to the precedent set by query-based methods [6], utilizing the Hungarian algorithm to assign a corresponding prediction for each ground-truth instance. The loss and hyperparameters used for model optimization in this paper follow those of previous work [30]. More details are provided in the Appendix.

3.4 Knowledge Retrieval for HOI Reasoning in Open World

In previous zero-shot HOI detectors, the recognition of unseen categories is usually achieved by directly using word embeddings and visual features for similarity measures. The problem with this approach is that it relies only on simple word embeddings, which may not be sufficiently detailed

and rich in the representation of HOIs. The model may not recognize some special categories well enough. Inspired by the human understanding of interactions, we propose a method for HOI detection in an open world based on descriptive text.

The impressive performance of large-scale pretrained language models, as well as the potentially enormous amount of implicitly stored knowledge, raises extensive attention about using language models as an alternative to conventional structured knowledge bases (LMs-as-KBs). In particular, the rapid development of the chatGPT family of LLM models in the last two years inspired us to obtain richer and more comprehensive knowledge descriptions, called GPTs-as-KBs. Theoretically, the performance of the visual model can be improved theoretically by transferring a larger amount of knowledge that can be effectively integrated into the visual model. Given an input image I i and corresponding descriptions T i in X = { ( I i , T i ) } |X| i =1 , different phrase descriptions are linked with different HO pairs. We propose expanding the existing phrase descriptions T i to obtain richer and more comprehensive knowledge descriptions K i . These descriptive knowledge facilitate more accurate interaction descriptions in zero-shot and open-vocabulary HOI recognition.

To obtain richer and more comprehensive knowledge descriptions, we design a knowledge retrieval process for LLM. We have designed a knowledge query statement for chatGPT, such as ' Knowledge retrieve for verb _ objection , limited to N words ', to guide LLMs in explaining the interaction categories we are interested in. In this way, the transferable textual knowledge will be richer and the interpretation of the interactions will be closer to the way humans understand them. As an illustration, let us consider the phrase "Human ride bicycle" and "Human throw frisbee" . The knowledge retrieval process yields the following example result:

Riding a bicycle involves balance, coordination, and physical exertion. The rider mounts the bike, propels forward by pushing pedals with their feet. Steering is achieved by turning handlebars. Brakes slow or stop the bike. Helmets are worn for safety.

Throwing a frisbee involves grasping the disc, typically with a forehand grip, then swinging the arm and releasing the frisbee at the right moment for it to glide through the air. Direction and distance depend on the angle and speed of the throw. It's a common recreational activity.

These detailed text descriptions can better guide the learning of visual features in open-category recognition. A more detailed open-category inference process is provided in the Appendix.

4 Experiments

4.1 Implementation Details

We evaluate our model on two public benchmarks, HICO-Det [7] and V-COCO [15]. We adopt the BLIP2 [23] model as the VL foundation model in our UniHOI. We map the 768-dimensional features output by Q-Former in BLIP2 to 256 dimensions to adapt to the feature dimension of the visual HOI detector. The number of layers in the HO Prompt-guided Decoder is identical to that in the Interaction Decoder. The output features of these two decoders are concatenated to generate the final category prediction. The model's optimization method, learning rate, training epochs, weight decay, and loss weights, among other hyperparameters, are set in accordance with previous methods to ensure fair comparison. Aside from the ablation studies in Section 4.6 where we employed knowledge retrieval to further enhance model performance, in all other experiments we continue to use annotated phrases for model learning and inference to ensure fair comparison with other methods. All the experiments conducted with a batchsize of 16 on 8 Tesla V100 GPUs. The details of datasets, evaluation metrics, model parameters, computational overhead, etc. will be introduced in Appendix.

4.2 HOI Detection in the Closed World

We compare the performance of our UniHOI with the previous representative and state-of-the-art methods such as ViPLO [37], GEN-VLKT [30], and HOICLIP [35]. The detailed results of HICODET [7] and V-COCO [15] are summarized in Table 1 and 3, respectively. Generally, there are several observations drawn from different aspects: ( i ) The overall performance of our method is significantly superior to the previous state-of-the-art methods. ( ii ) Whether the visual feature extractor adopts the ResNet-50 or ResNet-101, our method yields excellent performance, especially a stronger backbone

67

69

69

65

65

65

60

09

60

mAP

55

mAP

55

mAP

55

50

50

50

45

UniHOI-s

45

UniHOI-m

45

UniHOI-1

GEN-VLKT-s

GEN-VLKT-m

GEN-VLKT-1

40

10

20

30

40

50

60

70

80

90

40

10

20

30

40

50

09

70

80

90

40

10

20

30

40

50

60

70

08

06

Epoch

Epoch

Epoch

that can bring further improvements. ( iii ) As shown in Tables 2, our approach also has a significant performance lead compared to those methods that rely on additional data sets ( i.e. , Human Pose [32], and Vision-and-Language [45]). Figure 3 shows the training process of UniHOI-s and GEN-VLKT-s on the V-COCO dataset. Figure 4 shows some prediction cases of UniHOI on HICO-DET.

4.3 Comparisons with Methods that Utilize Extra Information

Table 2 presents several methods that utilize extra information ( i.e. , pose, and language) from additional datasets for HOI detection. Theoretically, the introduction of extra information can indeed further enhance visual models in specific domains. The results demonstrate that the proposed UniHOI still significantly outperforms previous methods.

Hold carrot

Ride skateboard

Catch ball

Carry bicycle

Straddle bicycle

Ride bicycle

Inspect fire hydrant

Paint fire hydrant

4.4 Effectiveness for Zero-Shot HOI Detection

Unseen Composition (UC). As shown in Table 4, we follow the [30] to conduct three kind of zero-shot experiments on HICO-DET. We first evaluate UniHOI-s with UC setting. Compared with GEN-VLKT [30], our UniHOI-s substantially outperforms previous state-of-the-art on both RF-UC and NF-UC. Among them, UniHOI are 7.32mAP higher than GEN-VLKT on Unseen of RF-UC, and 9.25mAP higher than GEN-VLKT on Seen of NF-UC. This improvement is mainly due to our efficient prompt learning method for the foundation model, so UniHOI can cope well with some unseen categories. In fact, GEN-VLKT also performs some knowledge transfer from CLIP, but it lags behind UniHOI in the effectiveness of knowledge transfer.

Unseen Object (UO). For UO setting, compared with GEN-VLKT, UniHOI improves Unseen, Seen, and Full by 9.21, 5.84, and 5.93, respectively. This result proves that our method can still understand the interaction between human and novel objects well.

Unseen Verb (UV). On the challenging UV setting, compared to GEN-VLKT, UniHOI improves by 9.21, 5.84 and 5.93 on Unseen, Seen, and Full, respectively. In fact, on the three zero-shot settings mentioned above, the HOI recognition capability of UniHOI have almost surpassed some supervised

Hold carrot

Catch ball

Straddle bicycle

Ride bicycle

Juggle orange

Kickboxing with human

Negotiate with human

Sleep on boat

Shake hands with human

Juggling oranges involves tossing and catching multiple oranges in a specific pattern. Typically, the juggler keeps more oranges in the air than they have hands. It requires good hand-eye coordination, rhythm, and timing. This activity can be entertaining and challenging.

Kickboxing is a hybrid combat sport that combines elements of punching from boxing and kicking from karate or muay thai. Humans participating in kickboxing engage in rigorous physical training, improving their strength, flexibility, and coordination. It's both a competitive sport and fitness regimen.

Negotiating with a human involves communication and compromise to reach a mutual agreement. It requires skills like active listening, articulation, patience, and persuasion. Negotiations can occur in various contexts, including business transactions, conflict resolution, and interpersonal relationships.

Sleeping on a boat can be a unique experience, influenced by factors like the boat's size, stability, and location. The gentle rocking can be soothing, but conditions can vary. It's important to ensure safety precautions, such as wearing lifejackets, are in place.

Shaking hands with a human is a common form of greeting, agreement, or parting. The typical handshake involves a firm grip and brief up and down movement. It's a universal sign of goodwill, respect, and mutual understanding between individuals.

methods in Table 1. This once again attests to the superiority of our UniHOI for open-category HOI detection and its powerful capability to understand interactions.

4.5 HOI Detection in the Wild

In addition to the zero-shot experiments on the HICO dataset, we also conducted open-category testing on open-world. Our UniHOI method supports flexible text input for detecting corresponding interactions. UniHOI can not only recognize category texts directly (e.g., "Juggle orange") but also take a descriptive text input (which could be human-written or knowledge retrieved from GPT). Consequently, interactions in images are identified through text representation. We present these detection results, along with some GPT-generated description texts, in Figure 5. Our test results indicate that open-vocabulary recognition based on interaction description texts is more reliable, and this phenomenon is also in line with our basic intuition. In the future, we will release our pretrained model and code to enable more extensive testing in the open world.

4.6 Ablation Studies

Effect of Foundation Model. We conduct ablation studies on the new components within UniHOI (as shown in Tables 5 and 6). The GEN-VLKT-s is regarded as the baseline, and then incorporated various components for testing on the fully supervised V-COCO dataset, as well as the HICO-DET dataset under a zero-shot setting. Initially, we simply integrated the BLIP2 into our method in a sequential manner. We then used learnable queries to extract interaction features from Q-Former and concatenate them with V i . However, the results did not yield a significant performance improvement.

Impact of HO Spatial Prompting. After realizing the alignment issue with the features of HO-pairs in the previous methods, we introduced the HO Prompt-guided Decoder, as presented in Section 3.3. The results clearly illustrate that the introduction of this component significantly improved all performance metrics, thereby validating the effectiveness of our prompting learning strategy.

Effect of Knowledge Retrieval. To substantiate the importance of utilizing LLM for knowledge retrieval in creating a more universal HOI detector, we also conduct supplementary experiments in this strategy. The results showed that the UniHOI model, when applying knowledge retrieval, achieved further performance improvements. We believe this approach opens new avenues for enhancement in the field of zero-shot HOI detection.

Ride skateboard

Effect of Different Foundation Model. To assess the impact of different Foundation Models on the performance of UniHOI, we also provide a comparative analysis between UniHOI equipped with CLIP and BLIP2. For CLIP, we feed its visual output representation into HOPD. However, given that CLIP's cross-modal contrastive training strategy leans more towards image-level representations [38], its output dimensionality is considerably lower than that of BLIP2, resulting in a less detailed feature representation. Experimental results indicate that both the UniHOI versions with CLIP and BLIP2 as the foundational models showed performance enhancements, with the latter exhibiting a more pronounced improvement. These findings underscore the efficacy and generalization of our approach.

5 Conclusion

In this work, we propose UniHOI, a universal HOI detection framework. By utilizing large foundation model, spatially prompting the large model, and generalizing the HOI detector to the open world. UniHOI supports open-category interaction relationship recognition from annotation texts or descriptions, which can be flexibly extended to applications in the open world. Extensive experiments on public datasets and diverse settings demonstrate its strong universality - it behaves the strongest open category interaction recognition ability so far. We believe our research will stimulate following research along the universal high-level relationship modeling direction in the future.

References

Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 (2022) 2, 3

Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., Van Den Hengel, A.: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3674-3683 (2018) 3

Bansal, A., Rambhatla, S.S., Shrivastava, A., Chellappa, R.: Detecting human-object interactions via functional generalization. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 10460-10469 (2020) 8

Cao, Y., Su, X., Tang, Q., You, S., Lu, X., Xu, C.: Searching for better spatio-temporal alignment in few-shot action recognition. Advances in Neural Information Processing Systems 35 , 21429-21441 (2022) 3

Cao, Y., Tang, Q., Yang, F., Su, X., You, S., Lu, X., Xu, C.: Re-mine, learn and reason: Exploring the cross-modal semantic correlations for language-guided hoi detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 23492-23503 (2023) 2

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European conference on computer vision. pp. 213-229. Springer (2020) 4, 5, 15

Chao, Y.W., Liu, Y., Liu, X., Zeng, H., Deng, J.: Learning to detect human-object interactions. In: 2018 ieee winter conference on applications of computer vision (wacv). pp. 381-389. IEEE (2018) 1, 3, 6, 7, 8, 16

Chen, M., Liao, Y., Liu, S., Chen, Z., Wang, F., Qian, C.: Reformulating hoi detection as adaptive set prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9004-9013 (2021) 1, 3, 7

Chen, Z., Li, G., Wan, X.: Align, reason and learn: Enhancing medical vision-and-language pre-training with knowledge. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 5152-5161 (2022) 3

Cheng, M., Sun, Y., Wang, L., Zhu, X., Yao, K., Chen, J., Song, G., Han, J., Liu, J., Ding, E., Wang, J.: Vista: Vision and scene text aggregation for cross-modal retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5184-5193 (June 2022) 3

Dzabraev, M., Kalashnikov, M., Komkov, S., Petiushko, A.: Mdmmt: Multidomain multimodal transformer for video retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3354-3363 (2021) 3

Gao, C., Xu, J., Zou, Y., Huang, J.B.: Drg: Dual relation graph for human-object interaction detection. In: European Conference on Computer Vision. pp. 696-712. Springer (2020) 1, 3, 7, 8

Gao, C., Zou, Y., Huang, J.B.: ican: Instance-centric attention network for human-object interaction detection. arXiv preprint arXiv:1808.10437 (2018) 3

Gkioxari, G., Girshick, R., Dollár, P., He, K.: Detecting and recognizing human-object interactions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8359-8367 (2018) 1, 3

Gupta, S., Malik, J.: Visual semantic role labeling. arXiv preprint arXiv:1505.04474 (2015) 1, 6, 9, 10, 14, 15

Hou, Z., Peng, X., Qiao, Y., Tao, D.: Visual compositional learning for human-object interaction detection. In: European Conference on Computer Vision. pp. 584-600. Springer (2020) 7, 8

Hou, Z., Yu, B., Qiao, Y., Peng, X., Tao, D.: Affordance transfer learning for human-object interaction detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 495-504 (2021) 7, 8

Hou, Z., Yu, B., Qiao, Y., Peng, X., Tao, D.: Detecting human-object interaction via fabricated compositional learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14646-14655 (2021) 1, 3, 7, 8

Iftekhar, A., Chen, H., Kundu, K., Li, X., Tighe, J., Modolo, D.: What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5353-5363 (2022) 2, 3, 7

Kim, B., Choi, T., Kang, J., Kim, H.J.: Uniondet: Union-level detector towards real-time human-object interaction detection. In: European Conference on Computer Vision. pp. 498-514. Springer (2020) 3

Kim, B., Lee, J., Kang, J., Kim, E.S., Kim, H.J.: Hotr: End-to-end human-object interaction detection with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 74-83 (2021) 8

Kim, S., Jung, D., Cho, M.: Relational context learning for human-object interaction detection. arXiv preprint arXiv:2304.04997 (2023) 7

Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023) 5, 6, 10, 13

Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.N., et al.: Grounded language-image pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10965-10975 (2022) 3

Li, Y.L., Liu, X., Lu, H., Wang, S., Liu, J., Li, J., Lu, C.: Detailed 2d-3d joint representation for human-object interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10166-10175 (2020) 7

Li, Y.L., Liu, X., Wu, X., Li, Y., Lu, C.: Hoi analysis: Integrating and decomposing human-object interaction. Advances in Neural Information Processing Systems 33 , 5011-5022 (2020) 7, 8

Li, Y.L., Zhou, S., Huang, X., Xu, L., Ma, Z., Fang, H.S., Wang, Y., Lu, C.: Transferable interactiveness knowledge for human-object interaction detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3585-3594 (2019) 1, 3, 16

Li, Z., Zou, C., Zhao, Y., Li, B., Zhong, S.: Improving human-object interaction detection via phrase learning and label composition. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 1509-1517 (2022) 2, 3, 8, 13, 14

Liao, Y., Liu, S., Wang, F., Chen, Y., Qian, C., Feng, J.: Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 482-490 (2020) 1, 3, 7

Liao, Y., Zhang, A., Lu, M., Wang, Y., Li, X., Liu, S.: Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20123-20132 (2022) 3, 5, 6, 7, 8, 10, 13, 14, 15

Liu, X., Li, Y .L., Wu, X., Tai, Y .W., Lu, C., Tang, C.K.: Interactiveness field in human-object interactions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2011320122 (2022) 3, 7, 8

Liu, Y., Chen, Q., Zisserman, A.: Amplifying key cues for human-object-interaction detection. In: European Conference on Computer Vision. pp. 248-265. Springer (2020) 7, 8

Liu, Y., Yuan, J., Chen, C.W.: Consnet: Learning consistency graph for zero-shot human-object interaction detection. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 4235-4243 (2020) 8

Mu, N., Kirillov, A., Wagner, D., Xie, S.: Slip: Self-supervision meets language-image pre-training. arXiv preprint arXiv:2112.12750 (2021) 3

Ning, S., Qiu, L., Liu, Y., He, X.: Hoiclip: Efficient knowledge transfer for hoi detection with visionlanguage models. arXiv preprint arXiv:2303.15786 (2023) 6, 7, 8, 14

OpenAI: Gpt-4 technical report (2023) 3, 9

Park, J., Park, J.W., Lee, J.S.: Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. arXiv preprint arXiv:2304.08114 (2023) 6, 7, 8

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748-8763. PMLR (2021) 3, 10, 13

Shen, L., Yeung, S., Hoffman, J., Mori, G., Fei-Fei, L.: Scaling human-object interaction recognition through zero-shot learning. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 1568-1576. IEEE (2018) 8

Tamura, M., Ohashi, H., Yoshinaga, T.: Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10410-10419 (2021) 7, 8, 15

Tsimpoukelli, M., Menick, J.L., Cabi, S., Eslami, S., Vinyals, O., Hill, F.: Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems 34 , 200-212 (2021) 3

Ulutan, O., Iftekhar, A., Manjunath, B.S.: Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13617-13626 (2020) 7, 8

Wang, S., Duan, Y., Ding, H., Tan, Y.P., Yap, K.H., Yuan, J.: Learning transferable human-object interaction detector with natural language supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 939-948 (2022) 2, 3

Xie, C., Zeng, F., Hu, Y., Liang, S., Wei, Y.: Category query learning for human-object interaction classification. arXiv preprint arXiv:2303.14005 (2023) 7, 8

Yuan, H., Jiang, J., Albanie, S., Feng, T., Huang, Z., Ni, D., Tang, M.: Rlip: Relational language-image pre-training for human-object interaction detection. arXiv preprint arXiv:2209.01814 (2022) 1, 2, 3, 7, 8

Yuan, H., Wang, M., Ni, D., Xu, L.: Detecting human-object interactions with object-guided cross-modal calibrated semantics. arXiv preprint arXiv:2202.00259 (2022) 3, 8

Zhang, A., Liao, Y., Liu, S., Lu, M., Wang, Y., Gao, C., Li, X.: Mining the benefits of two-stage and one-stage hoi detection. Advances in Neural Information Processing Systems 34 , 17209-17220 (2021) 3, 7, 8

Zhang, F.Z., Campbell, D., Gould, S.: Spatially conditioned graphs for detecting human-object interactions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13319-13327 (2021) 7

Zhang, F.Z., Campbell, D., Gould, S.: Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20104-20112 (2022) 1, 3, 7, 8

Zhang, H., Zhang, P., Hu, X., Chen, Y.C., Li, L.H., Dai, X., Wang, L., Yuan, L., Hwang, J.N., Gao, J.: Glipv2: Unifying localization and vision-language understanding. arXiv preprint arXiv:2206.05836 (2022) 3

Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X.V., et al.: Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022) 13

Zhong, X., Ding, C., Qu, X., Tao, D.: Polysemy deciphering network for robust human-object interaction detection. International Journal of Computer Vision 129 (6), 1910-1929 (2021) 2, 3, 8

Zhong, X., Qu, X., Ding, C., Tao, D.: Glance and gaze: Inferring action-aware points for one-stage human-object interaction detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13234-13243 (2021) 3

Zou, C., Wang, B., Hu, Y., Liu, J., Wu, Q., Zhao, Y ., Li, B., Zhang, C., Zhang, C., Wei, Y ., et al.: End-to-end human object interaction detection with hoi transformer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11825-11834 (2021) 7

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework for human-object interaction (HOI) detection that leverages vision-language foundation models and large language models to achieve universal and flexible recognition of complex interactions in images. The framework, named UniHOI, consists of three main components: a visual HOI detector that extracts three levels of features from images, a HO prompt-guided decoder that queries the foundation model for high-level relation representations associated with human-object pairs, and a knowledge retrieval module that uses a large language model to generate descriptive texts for interaction categories. The framework supports both supervised and zero-shot settings, and can handle any textual input for open-category interaction detection. The paper demonstrates the effectiveness and superiority of UniHOI over existing methods on two public benchmarks, HICO-DET and V-COCO, as well as in the wild scenarios.\n\nSTRENGTHS:\n- The performance is quite impressive.\n- Leveraging LLM and foundational models to augment CV tasks is the future, and this work gives a try to use them simultaneously.\n- The authors promise to release the code to ensure reproducibility.\n\nWEAKNESSES:\n1. There is no ablative study of each component (perhaps only one component, i.e., HO prompt decoder) under the close-set setup.\n2. This work uses BLIP2 with ViT-L while existing work like GEN-VLKT typically uses CLIP with ViT-B. It is evident that the former is much more powerful. Could you provide the performance on HICO-DET using HO prompt-based learning with CLIP ViT-B under the close-set setup? As the improvement may be brought by more advanced large visual-language pre-trained models.\n3. How about the inference speed? As shown in Table 8, **three** times longer than GEN-VLKT. Note that existing work like GEN-VLKT does not involve the computation of large visual-language pre-trained models at the inference stage, since all of the feature of objects or verbs is pre-computed. However, in this work, the feature for prompting should be computed for each image individually. Considering the extremely large backbone (e.g, ViT-L), there would be a heavy burden at inference.\n4. The core contribution of this work is actually the HO prompt-based decoder. However, there is nothing novel with it, i.e., directly using spatial location to get foundational model output features.\n5. Knowledge retrieval is solely used in open-world setup, is it possible for it to augment the closed-world setup?\n\nOverall, this is technically solid work, and LLM for knowledge retrieval is interesting. But the comparison is unfair (i.e., a much more powerful visual-language pre-trained model), the inference time is unacceptable, and the novelty of the prompt-based decoder is limited. I will be very happy to update my score if the authors can address my concerns above.\n\nQUESTIONS:\nPlease refer to weaknesses.\n\nLIMITATIONS:\nThere is no discussion on limitations or failure cases.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn view of the limited scalability and the suboptimal zero-shot performance of current HOI detection methods, the authors propose a novel method for HOI detection based on VL foundation models.\nWith in-depth analysis and adaptation of HOI detectors, the foundation model is effectively adopted to reason about HOI relationships based on human/object tokens.\nFurthermore, LLM is adopted as a knowledge base to diversify HOI descriptions, enabling open-vocabulary HOI detections.\nWith the VL foundation model and LLM, extraordinary HOI detection performance is achieved upon both conventional and zero-shot setting.\n\nSTRENGTHS:\nThe proposed HO prompt-guided decoder is brilliant in addressing the feature alignment issue.\n\nAdopting GPT as knowledge base is an interesting idea to incorporate the recent progress in LLM with HOI detection.\n\nThe performance is amazing with significant margins upon previous SOTAs, especially for zero-shot setting.\n\nExtensive experiments are conducted, providing valuable insights on the effect of VL foundation models in HOI detection.\n\nIn the wild HOI detection illustration is quite impressive.\n\nWEAKNESSES:\nThe comparison between GEN-VLKT and the proposed method is not totally fair to me. It might be better to replace BLIP-2 with CLIP for a fair comparison.\n\nFig. 2 is not very clear. It would help if annotating the encoders in the figure with corresponding notations. \n\nThe baseline of ablation experiments is chosen as GEN-VLKT. However, a major difference between GEN-VLKT and UniHOI is the VL foundation model used. And there are also other differences, like VLKT is not used for UniHOI. It might be better to change the baseline to make the ablation more reasonable.\n\nQUESTIONS:\n- The description on the adopted Image Encoder and Instance Decoder is not clear. Is DETR adopted? Or some more advanced detectors? Are they frozen during training?\n\n- In L177, $V^f$ and $V^f$ seems to be a typo.\n\n- The performance of UniHOI-l in Tab. 1-2 is questionable. Please check whether there are typos.\n\n- In the ablation studies, it is still not very clear that how the VL Foundation model is simply added. Is it a simple removal of HO Spatial Prompting? Or replacing the input feature of HO Spatial Prompting with the learnable queries?\n\n- Is it possible to make the HOI detector share the backbone of VL foundation model? This could be related to the proposed insight that HOI detector is a three-tier visual feature hierarchy. Ablation studies on this would be preferrable.\n\n- The proposed method seems to be applicable to arbitrary HOI detectors (if the answer to Q1 is yes). Is it practical?\n\n- Results in Tab. 5 and Tab. 3 are not consistent. The result without Knowledge Retrieval is reported in Tab. 3.\n\nLIMITATIONS:\nLimitations of the paper is not well-discussed in the paper. \nI would like to see more discussion on extending the use of LLM further than a static knowledge base.\nAlso please refer to the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper investigates the problem of human-object interaction (HOI) detection. The authors introduced UniHOI, a method for universal HOI detection in an open-world setting. They also explored the universal interaction recognition with Vision-Language (VL) foundation models and large language models (LLMs), and proposed HO prompt-based learning for high-level relation extraction aimed at VL foundation models.  Experimental results show the effectiveness and significance of the proposed method.

STRENGTHS:
1. Overall, the manuscript is well-written and easy to follow. The figures are pretty and can convey the concepts clearly.
2. Pushing the problem of human-object interaction detection toward an open-world setting is of great importance. This is also a trend for most existing computer vision applications.
3. The extensive experimental results show the superiority of the proposed UniHOI method.

WEAKNESSES:
1. In Table 4, the results from the third row come from "ConsNet [31]" but not "ATL [15]", according to the paper of "GEN-VLKT".
2. The conclusion part lacks objective reflections on the deficiencies of this study and future prospects for improvements.

QUESTIONS:
Please refer to the weakness section.

LIMITATIONS:
Please refer to the weakness section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses human-object interaction (HOI) detection task. The authors propose a new method named as UniHOI, achieved by prompting BLIP2 using human-object paired features, as well as linguistic semantics generated by a LLM. The proposed UniHOI demonstrates significant performance gain on HICO-DET and V-COCO in both fullly supervised and zero-shot scenarios.\n\nSTRENGTHS:\nIf the BLIP2 is a fundamental model pre-trained on a lower-level task than HOI detection (e.g., classification, object detection), I would say this paper is an excellent work  in terms of both model design and performance. Actually, I think this is the first work to transfer a fundamental model into the domain of HOI detection, which may open doors to further exploration on prompting learning  for HOI. However, I cannot accept the choice of using BLIP2 as the fundamental model for HOI detection. I will explain the reasons for this in the  weaknesses below.\n\nWEAKNESSES:\n1. Prompting engineering aims to transfer a fundamental model pretrained on **lower-level tasks** to **higher-level** tasks. At a minimum, **the task for pre-training needs to be decoupled from the downstream task**. Otherwise, transferring a model pertrained on higher-level tasks to low-level tasks is not prompting, but fine-tuning. BLIP2 is a powerful model pertrained for VQA, image captioning, and similar tasks. However,  as widely acknowledged,  HOI detection is a sub-problem for  these detailed scene understanding tasks.  Namely, BLIP2 itself is a powerful HOI detector (I have tried using BLIP2 directly for HOI detection, and the performance is impressive).  From this point, BLIP2 cannot be used as a fundamental model for HOI detection since it has a great capability for HOI detection by itself, and is capable of even higher-level tasks. Therefore, this paper is more like a work that fine-tunes BLIP2 on HICO-DET and V-COCO, at the cost of giving up the ability to use BLIP2 for other tasks, e.g., captioning.\n2. While direct use of BLIP2 for HOI detection may fail to achieve as impressive performance as that of UniHOI on HICO-DET and V-COCO, **BLIP2 has already achieved the goal of doing HOI detection, i.e., detailed scene understanding**.  Therefore, is it a case of putting the cart before the horse to use BLIP2 for HOI detection only?\n3. In a real open-world scenario, I think BLIP2 is more capable of HOI detection compared to UniHOI. After all, the  zero-shot HOI detection capability of UniHOI is mainly inherited from BLIP2.\n4. The performance of UniHOI on HICO-DET and V-COCO is impressive.  I think this is the first work that achieves a mAP  being larger than 40% on HICO-DET. However, the comparison is not so fair. As aforementioned, BLIP2 itself is a powerful HOI detector, which has been pre-trained with a large amount of **interaction-specific** data.  Note that, **on a fair comparison,  a fundamental model should not be pre-trained using data with annotations involving downstream tasks**. Otherwise, the authors need to report the results without using these extra data. For instance, if we first collect all data involving HOI detection from the dataset used for BLIP2 pre-training. Next, we use these data  to pre-train a HOI detector listed in Table 1, (e.g., GEN-VLKT) to get GEN-VLKT-2. Finally, we fine-tune the GEN-VLKT-2 on the HICO-DET and V-COCO. I think it can also get an excellent performance. This is another reason why I think that transferring a model pre-trained on a higher-level task to a lower- level task (especially when the lower-level task is a sub-task of the higher-level task) is not a prompting, but a fine-tuning.\n\nQUESTIONS:\nActually, I am very much looking forward to the work of using fundamental models for HOI detection. However, I think an instrive work is to transfer a fundamental model pretrained on lower-level task to a high-level task. This work, however, seems to be the opposite.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a universal HOID pipeline, which utilizes decoded ho-pair feature as spatial prompts to prompt the VL foundation model with the aim to implement effective prompt-based learning on base VL model and extract HOI related features from it. It also proposes knowledge retrieval for HOID in open-category manner by large-scale pretrained language models. Experiments show the effectiveness of this approach in both generic and zero-shot HOID.\n\nSTRENGTHS:\n1. It explores universal interaction recognition by tansferring the rich knowledge inside Vision-Language foundation models and LLMs to HOI pipeline, which broaden the research scope of HOID.\n2. The experimental results are promising. In both generic and zero-shot setting, this approach reaches a new state-of-the-art and surpasses previous methods by a substantial margin.\n\nWEAKNESSES:\n1.In line152, ‘P_h’ and ‘P_o’ are described as ‘excellent spatial position features’ and further utilized as HO spatial prompts. However, these features are generated by learnable position embedding and query, which is identical as many transformer-based HOID approaches before such as GEN-VLKT. Can you provide some evidence that these feature are indeed ‘excellent’, why could it provide accurate spatial information concerning ho pairs? Or are there some unique designs I overlook?   \n2.In Line 177, HOPD is designed for the alignment issue between VL foundation model and HOI pipeline. And the output V_f is incorperated with V_i for final predicition of interaction. But the performance of pure V_f, i.e., only utilizing V_f for prediction is unexamined. This results may more directly show the effectiveness of alignment between VL models and HOI pipeline.\n3.Some typos. Line 171, ‘the guidance of’ repeated twice. Line 177, ‘V_i’ is mismarked as ‘V_f’.\n\nQUESTIONS:\nMajorly the questions lies in the choice and interpretability of spatial prompts. It’s unclear why these prompts are ‘excellent spatial position features’. Or have you try some experiments on the choice of these prompts?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}