SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yichao Cao
Southeast University
Qingfei Tang
Nanjing Enbo Tech.
Xiu Su*
Chen Song
Shan You
Chang Xu
Human-object interaction (HOI) detection aims to comprehend the intricate relationships between humans and objects, predicting  $<$  human, action, object  $>$  triplets, and serving as the foundation for numerous computer vision tasks. The complexity and diversity of human-object interactions in the real world, however, pose significant challenges for both annotation and recognition, particularly in recognizing interactions within an open world context. This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs). The proposed method is dubbed as UniHOI. We conduct a deep analysis of the three hierarchical features inherent in visual HOI detectors and propose a method for high-level relation extraction aimed at VL foundation models, which we call HO prompt-based learning. Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. Furthermore, we utilize a LLM (i.e. GPT) for interaction interpretation, generating a richer linguistic understanding for complex HOIs. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence. Our efficient architecture design and learning methods effectively unleash the potential of the VL foundation models and LLMs, allowing UniHOI to surpass all existing methods with a substantial margin, under both supervised and zero-shot settings. The code and pre-trained weights are available at: https://github.com/ Caoyichao/ UniHOI.
Human-object interaction (HOI) detection [15, 7] is a burgeoning field of research in recent years, which stems from object detection and makes higher demands on high-level visual understanding. It plays a crucial role in many vision tasks, e.g., visual question answering, human-centric understanding, image generation, and activity recognition, to name a few representative ones. An excellent HOI detector must accurately localize all the interacting Human-Object (HO) pairs and recognize their interaction, typically represented as an HOI triplet in the format of  $<$  human, action, object  $\succ$ .
Despite the optimistic progress made by various HOI detectors [27, 12, 18, 49, 14, 29, 8] in recent years, numerous challenges still persist [45]. Naively mapping diverse interactions to one-hot labels is an exceedingly labor-intensive and costly process, which is prone to the loss of semantic information.
*Corresponding author (xisu5992@uni.sydney.edu.au). First author (caoyichao@seu.edu.cn).
Traditional HOI Detection
Previous HOI Vision-and-Language Modeling (HOI-VLM)
Our Universal HOI Detector (UniHOI)
Figure 1: Comparison between HOI detection pipelines. Traditional HOI Detectors rely on manually annotated image datasets for training, which limits their generality and scalability. Previous HOI-VLM methods only achieved limited knowledge transfer from language models to visual HOI detectors, utilizing only a small amount of word embeddings for learning. The proposed UniHOI supports the input of any textual information (annotations or explanations) to detect any interaction relationship, fully unleashing the potential of foundation models and LLMs.
As the number and complexity of interaction categories increase, model optimization becomes increasingly difficult, and the universality of the model is consequently restricted.
Recently, multi-modal learning, especially Vision-and-Language (VL) learning, has become remarkably popular, achieving far-reaching success across a variety of domains [1]. Multi-modal learning is capable of portraying multidimensional information in a more comprehensive manner by describing the same entity or spatio-temporal event through different modalities. Recently, a number of studies have endeavored to employ VL models for the HOI task with the aim of constructing transferable HOI detectors [52, 19, 28, 43, 45, 5]. For instance, PhraseHOI [28] harnessed a pre-trained word embedding model to generate a fixed-length linguistic representation for the HOI task. Unfortunately, these methods harness cross-modal knowledge in an overly constrained manner, falling short of fully exploiting the potential of cross-modal knowledge and LLMs within the HOI detection domain.
Upon rigorous examination, we identify several intrinsic limitations in the existing methodologies: (1) Limited scalability: Model training is overly reliant on annotated data, which restricts them to a finite set of categories. (2) Suboptimal adaptability in zero-shot settings: Even HOI-VLM approaches can only harness a limited number of word embeddings to accommodate unseen categories, thereby curtailing their adaptability. (3) Inability to quickly comprehend complex interactive behaviors from succinct descriptions, as humans can, indicating a dearth of a more universal and flexible framework for HOI detection.
Considering the recent outstanding performance of VL foundation models and LLMs in various fundamental tasks, in this paper, we attempt to harness the capabilities of such large models to facilitate more general HOI detector, as shown in Figure 1. We conduct an in-depth analysis of the clear three-tier visual feature hierarchy in visual HOI detectors, effectively utilizing the foundation models to boost the understanding of high-level interactive semantics within images. The employment of large models bridges the gap between textual and visual modalities, making a knowledge-based universal HOI detector feasible. In summary, the contributions of this paper are as follows:
- We propose the first visual-textual foundation model-based framework for HOI detection (UniHOI), which significantly improves the accuracy and universality compared to previous HOI detectors that solely rely on training from specific datasets. Our UniHOI method is expected to guide the HOI detection field into a new research phase.
- We treat the visual HOI detector as a three-tier visual feature hierarchy and appropriately design an efficient human-object prompt learning method, enabling visual HOI detectors to effectively query interaction tokens associated with specific HO pairs in foundation models, and thus obtain a more universal HO relationship representations.
- We devise a knowledge-based zero-shot HOI recognition method that leverages large models such as GPT to generate descriptive knowledge for specific interaction categories, guiding visual detectors in zero-shot recognition of the corresponding categories. This approach
breaks free from the limitations of traditional zero-shot recognition methods, which can only learn from word embeddings, and offers insights for a more universal open-category HOI recognition system.
Generic HOI Detection. Based on their architectural design, existing HOI (Human-Object Interaction) detection approaches can be broadly categorized into two groups: two-stage methods [49, 7, 12, 13, 27, 18], and one-stage methods [14, 20, 53, 29, 8]. One-stage methods typically adopt a multitask learning strategy to simultaneously execute instance detection and interaction relation modeling [29, 47, 30]. Conversely, two-stage methods initially conduct object detection, followed by interaction relation modeling for all candidate HO pairs. By fully exploiting the capabilities of each module, two-stage methods have showcased superior detection performance [47]. While these methods have significantly propelled the progress of early HOI detection, they still grapple with the issue of limited generality.
Language Semantics for Vision. Driven by the notable success of large-scale models such as GPT-4 [36], the incorporation of language semantics to augment vision models has recently surfaced as a promising avenue in the realm of computer vision tasks [38, 41, 1, 50]. Among these, Vision-and-Language Pre-training (VLP) [34, 24] has evolved into a popular paradigm across numerous vision-and-language tasks, owing to its ability to learn generalizable multimodal representations from extensive image-text data [1, 9, 10]. These methods have recently found applications in multimodal retrieval [11], vision-and-language navigation [2], among others [4]. Nevertheless, implementing effective prompt-based learning on base models, particularly extracting high-order relationships in intricate scenarios, remains crucial for the successful deployment of large models. Thus, investigating how to efficaciously extract specific information from large models is of utmost importance.
HOI Vision-and-Language Modeling (HOI-VLM). Despite the moderate success of previous HOI detectors [47, 49, 31], these often perceive interactions as discrete labels, overlooking the rich semantic text information encapsulated in triplet labels. Of late, a handful of researchers [52, 19, 28, 43, 45, 46] have delved into HOI Vision-and-Language Modeling in an effort to further enhance HOI detection performance. Among these, [52], [19], and [46] have primarily focused on incorporating language prior features into HOI recognition. Meanwhile, RLIP [45] and [43] have proposed the construction of a transferable HOI detector via the VLP approach. Serving as applications and extensions of Vision-and-Language learning within the HOI domain, these HOI-VLM methodologies strive to comprehend the content and relations between visual interaction features and their respective triplet texts. Nevertheless, these methodologies harness cross-modal knowledge in a markedly restricted manner, falling short of fully unleashing the potential of cross-modal knowledge and LLMs within the HOI domain.
In traditional HOI detection task, the objective is typically to optimize a detector  $\Phi_{\theta_D}$ , such as the most commonly-used Transformer-based detectors in recent years. By inputting a query  $Q^{ho}$  related to the Human-Object (HO) pairs, the HOI detector learns the location of the HO pairs and corresponding interactions in images. This optimization objective can be formulated as follows:
$$
\min  \mathbb {E} _ {(\mathcal {I}) \sim \mathcal {X} _ {\mathcal {I}}} \left[ \mathcal {L} \left(\mathcal {G T}, \Phi_ {\theta_ {\mathcal {D}}} \left(\mathcal {I}, \mathcal {Q} ^ {h o}\right)\right) \right] \tag {1}
$$
where  $\mathcal{G}\mathcal{T}$  and  $\mathcal{L}$  are ground-truth labels and overall loss function respectively,  $\mathcal{X}_{\mathcal{I}} = \{(I_i)\}_{i=1}^{|\mathcal{X}|}$  denotes the HOI image dataset,  $\mathcal{I}$  represents the input image, and  $\theta$  indicates the weights in HOI detector  $\Phi_{\theta_D}$ . Thus, traditional HOI detectors rely on purely visual detection tasks for optimization.
In this work, our aim is to utilize superior VL foundation models and additional interaction interpretation  $\mathcal{T}$  during the training and inference process. In this way, we can improve the optimization objective as follows:
$$
\min  \mathbb {E} _ {(\mathcal {I}, \mathcal {T}) \sim \mathcal {X}} \left[ \mathcal {L} \left(\mathcal {G T}, \Phi_ {\theta_ {\mathcal {D}}} (\mathcal {I}, \mathcal {Q} ^ {h o}), \Phi_ {\theta_ {\mathcal {F}}} (\mathcal {I}, \mathcal {T}, \mathcal {P} ^ {h o})\right) \right] \tag {2}
$$
Figure 2: Overview of the UniHOI. Its feature hierarchy is divided into three levels. After the first two levels achieve instance detection for humans and objects, we designed a HO spatial prompt learning method, which extracts more generic higher-level relationship representations associated with HO pairs from the VL foundation models. For open-category interaction detection, UniHOI supports annotations or retrieved texts input, and then determine whether the interaction described in the input text occurs between each HO pair through feature similarity measurement.
where  $\Phi_{\theta_{\mathcal{F}}}$  denotes the Vision-Language (VL) foundation models and  $\mathcal{P}^{ho}$  represents the HOI-specific prompts for  $\Phi_{\theta_{\mathcal{F}}}$ .  $\mathcal{X} = \{(\mathcal{I}_i, \mathcal{T}_i)\}_{i=1}^{|X|}$  is an image-text corpus, where  $\mathcal{I}_i$  denotes the  $i$ -th input image and  $\mathcal{T}_i$  represents the phrase annotations (e.g. "Human ride bicycle") in  $\mathcal{I}_i$ . In zero-shot setting, we also use knowledge retrieval to enrich the texts  $\mathcal{T}$ . Details will be described in Section 3.4.
In HOI detection, the detector not only needs to accurately locate the position of the interacting HO pair, but also needs to reliably model the interaction relationship occurring in the HO pair. But these two tasks have different requirements on the image feature level: the former focuses more on the edges and contours of instances, while the latter focuses more on high-level relational cues. To present and analyze this complicated process more clearly, we explicitly divide the feature learning in HOI detector into three stages: basic visual feature extraction, instance-level feature learning, and high-level relationship modeling, as shown in Figure 2. The basic visual feature extraction is usually performed by the backbone (e.g. CNN). For the instance-level feature learning, we follow previous methodologies and design prompts related to HO locations to carry and transfer the position information, then apply the Hungarian algorithm for matching with the  $\mathcal{G}\mathcal{T}$ , and compute the loss of the bounding box. The third stage, which is the key focus of this study, involves using spatial location prompts related to the HO pairs to conduct targeted prompt learning from the visual model. The details of each component are explained in the following sections.
3.2 HO Spatial Prompts Generation
We follow the Transformer-based detection architectures [6] to conduct basic visual feature extraction and instance detection. Fed with an input image  $\mathcal{I} \in \mathcal{R}^{H \times W \times C}$ , the CNN generates a feature map  $\mathcal{X}^v \in \mathcal{R}^{h \times w \times c}$ . Then,  $\mathcal{X}^v$  is compressed by a projection convolution layer with a kernel size  $1 \times 1$ . Next, a flatten operator is used to segment patch embeddings  $\{x_1^v, x_2^v, \ldots, x_{N^v}^v\}$ , where  $N^v$  is the number of patch embeddings. The patch embeddings  $\{x_1^v, x_2^v, \ldots, x_{N^v}^v\}$  is then linearly projected through a linear transformation  $\mathcal{E}^v \in \mathcal{R}^{c \times D^v}$ . Thus, the input for instance detection are calculated via summing up the projected embeddings and position embeddings  $\mathcal{E}_{pos}^v \in \mathcal{R}^{N^v \times D^v}$ :
$$
\mathcal {X} ^ {p} \in \mathcal {R} ^ {N ^ {v} \times D ^ {v}} = \left[ x _ {1} ^ {v} \mathcal {E} ^ {v}; x _ {2} ^ {v} \mathcal {E} ^ {v}; \dots ; x _ {N ^ {v}} ^ {v} \mathcal {E} ^ {v} \right] + \mathcal {E} _ {\text {p o s}} ^ {v} \tag {3}
$$
After patching and sequencing the features extracted by CNN, we follow previous methods to perform self-attention on  $\mathcal{X}^p$ :  $\mathcal{M} = \Phi_{\theta_{\mathcal{X}\varepsilon}}(\mathcal{X}^p) \in \mathcal{R}^{N^v \times D^v}$ . This process is implemented by a Transformer encoder  $\Phi_{\theta_{\mathcal{X}\varepsilon}}$ , which is composed of  $N$  Transformer encoder layers.
For instance-level feature learning, we deploy an instance decoder  $\Phi_{\theta_{\mathcal{I}\mathcal{D}}}$  to locate the positions of humans and objects. Based on the  $\mathcal{M}$ , humans and objects are detected through the human query set
$\mathcal{Q}^h \in \mathcal{R}^{N^q \times D^v}$  and the object query set  $\mathcal{Q}^o \in \mathcal{R}^{N^q \times D^v}$  individually. Additionally, a position guided embedding [30]  $\mathcal{Q}^g \in \mathcal{R}^{N^q \times D^v}$  is added to assign the human and object queries at the same position as a pair. The calculation process of instance decoder  $\Phi_{\theta_{\mathcal{I}^D}}$  can be formulated simply as follows:
$$
\mathcal {P} ^ {h o} \in \mathcal {R} ^ {2 N ^ {q} \times D ^ {v}} = \left[ \mathcal {P} ^ {h}, \mathcal {P} ^ {o} \right] = \Phi_ {\theta_ {\mathcal {I D}}} \left(\mathcal {M}, \left[ \mathcal {Q} ^ {h} + \mathcal {Q} ^ {g}, \mathcal {Q} ^ {o} + \mathcal {Q} ^ {g} \right]\right) \tag {4}
$$
where the  $\mathcal{P}^{ho}$  is the spatial tokens highly correlated with HO pair location information, the  $N^q$  is the number of queries. With instance detection head  $\mathcal{FFN}s$ , we can calculate the human bounding boxes  $\mathcal{B}^h\in \mathcal{R}^{N^q\times 4}$  from  $\mathcal{P}^h$ , object bounding boxes  $\mathcal{B}^o\in \mathcal{R}^{N^q\times 4}$  and object categories  $\mathcal{C}^o\in \mathcal{R}^{N^q\times N^c}$  from  $\mathcal{P}^o$ :
$$
\left[ \mathcal {B} ^ {h}; \mathcal {B} ^ {o}; \mathcal {C} ^ {o} \right] = \mathcal {F F N} s \left(\left[ \mathcal {P} ^ {h}, \mathcal {P} ^ {o} \right]\right) \tag {5}
$$
where the feature tokens  $\left[\mathcal{P}^h,\mathcal{P}^o\right]$  are excellent spatial position features for HO pairs. Driven by prompt learning methods, we explore using the HO position features here to perform prompt learning on the VL foundation models.
3.3 Prompting Foundation Models for HOI Modeling
Considering the excellent performance of BLIP2 [23] in cross-modal tasks and the fact that its Transformer architecture also integrates well with our HOI framework, in this work we use BLIP2 as the VL foundation model to support the recognition process of HOI. However, accurately extracting HOI related features from VL foundation model is the key to successfully applying VL to HOI tasks.
To reduce the computational cost of the VL foundation model, we first downsample the input image  $\mathcal{I}$  to  $\mathcal{I}'$  with dimension  $W' \times H'$ , and then use an image encoder  $\Phi_{\theta_{\mathcal{I}}}$  to convert downsampled image into a feature map  $\mathcal{X}^f \in \mathcal{R}^{w' \times h' \times c'}$ . We then employ Q-Former  $\Phi_{\theta_{\mathcal{Q}}}$  in BLIP2 [23] to learn higher-level content  $\mathcal{X}^q \in \mathcal{R}^{N^f \times D^f}$  in image:
$$
\mathcal {X} ^ {q} = \Phi_ {\theta_ {\mathcal {F}}} \left(\mathcal {I} ^ {\prime}\right) = \Phi_ {\theta_ {\mathcal {I}}} \circ \Phi_ {\theta_ {\mathcal {Q}}} \left(\mathcal {I}, \mathcal {Q} ^ {f}\right) \tag {6}
$$
where the  $Q^{f}$  denotes the query in  $\Phi_{\theta_{\mathcal{I}}}$ . These modules in  $\Phi_{\theta_{\mathcal{I}}}$  are more capable of understanding content in images than regular visual detectors due to extensive training on a wider range of data.
Then, we consider how to extract the complex interaction representations between specific humans and objects in foundation model. To associate the higher-level information in foundation model with HO pairs, we design a HO Prompt-guided Decoder (HOPD)  $\Phi_{\theta_{\mathcal{P}}}$  to correlate the position information of HO pairs with feature tokens in foundation model:
$$
\mathcal {V} ^ {f} = \Phi_ {\theta_ {\mathcal {P}}} \left(\left(\mathcal {P} ^ {h} + \mathcal {P} ^ {o}\right) / 2, \mathcal {X} ^ {q}\right) \tag {7}
$$
where the  $\mathcal{V}^f$  denotes the high-level information tokens corresponding to HO pairs. In this manner, the HOI-specific tokens is generated under the guidance of the guidance of the human-object query pairs. This spatial prompts act as query embeddings in HOPD, interact with each other through self-attention layers, and interact with frozen image features through cross-attention layers (inserted every other transformer block in HOPD). In this way, these spatial prompts are fed into HOPD, which in turn goes on to query information about the relationships between their respective corresponding HO pairs. In addition, interaction decoding is performed in the visual HOI detector:
$$
\mathcal {V} ^ {i} = \Phi_ {\theta_ {\mathcal {I N}}} \left(\left(\mathcal {P} ^ {h} + \mathcal {P} ^ {o}\right) / 2, \mathcal {M}\right) \tag {8}
$$
Finally, we concatenate  $\mathcal{V}^f$  and  $\mathcal{V}^f$  as the visual representations of the HO pairs and feed them to a FFN to predict the interaction category. For model training, we adhere to the precedent set by query-based methods [6], utilizing the Hungarian algorithm to assign a corresponding prediction for each ground-truth instance. The loss and hyperparameters used for model optimization in this paper follow those of previous work [30]. More details are provided in the Appendix.
3.4 Knowledge Retrieval for HOI Reasoning in Open World
In previous zero-shot HOI detectors, the recognition of unseen categories is usually achieved by directly using word embeddings and visual features for similarity measures. The problem with this approach is that it relies only on simple word embeddings, which may not be sufficiently detailed
and rich in the representation of HOIs. The model may not recognize some special categories well enough. Inspired by the human understanding of interactions, we propose a method for HOI detection in an open world based on descriptive text.
The impressive performance of large-scale pretrained language models, as well as the potentially enormous amount of implicitly stored knowledge, raises extensive attention about using language models as an alternative to conventional structured knowledge bases (LMs-as-KBs). In particular, the rapid development of the chatGPT family of LLM models in the last two years inspired us to obtain richer and more comprehensive knowledge descriptions, called GPTs-as-KBs. Theoretically, the performance of the visual model can be improved theoretically by transferring a larger amount of knowledge that can be effectively integrated into the visual model. Given an input image  $\mathcal{I}^i$  and corresponding descriptions  $\mathcal{T}_i$  in  $\mathcal{X} = \{(\mathcal{I}_i,\mathcal{T}_i)\}_{i = 1}^{|\mathcal{X}|}$ , different phrase descriptions are linked with different HO pairs. We propose expanding the existing phrase descriptions  $\mathcal{T}^i$  to obtain richer and more comprehensive knowledge descriptions  $\mathcal{K}^i$ . These descriptive knowledge facilitate more accurate interaction descriptions in zero-shot and open-vocabulary HOI recognition.
To obtain richer and more comprehensive knowledge descriptions, we design a knowledge retrieval process for LLM. We have designed a knowledge query statement for chatGPT, such as "Knowledge retrieve for verbOLAction, limited to  $N$  words", to guide LLMs in explaining the interaction categories we are interested in. In this way, the transferable textual knowledge will be richer and the interpretation of the interactions will be closer to the way humans understand them. As an illustration, let us consider the phrase "Human ride bicycle" and "Human throw frisbee". The knowledge retrieval process yields the following example result:
- Riding a bicycle involves balance, coordination, and physical exertion. The rider mounts the bike, propels forward by pushing pedals with their feet. Steering is achieved by turning handlebars. Brakes slow or stop the bike. Helmets are worn for safety.
- Throwing a frisbee involves grasping the disc, typically with a forehand grip, then swinging the arm and releasing the frisbee at the right moment for it to glide through the air. Direction and distance depend on the angle and speed of the throw. It's a common recreational activity.
These detailed text descriptions can better guide the learning of visual features in open-category recognition. A more detailed open-category inference process is provided in the Appendix.