SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yunyao Mao $^{1}$  Jiajun Deng $^{2}$  Wengang Zhou $^{1\dagger}$  Li Li $^{1}$  Yao Fang $^{3}$  Houqiang Li $^{1\dagger}$
$^{1}$ CAS Key Laboratory of Technology in GIPAS, EEIS Department,
University of Science and Technology of China
$^{2}$ The University of Adelaide, AIML  $^{3}$ Merchants Union Consumer Finance Company Limited
myy2016@mail.ustc.edu.cn, jiajun.deng@adelaide.edu.au, zhwg@ustc.edu.cn
lil1@ustc.edu.cn, fangyao@mucfc.com, lihq@ustc.edu.cn
Zero-shot Human-Object Interaction (HOI) detection aims to identify both seen and unseen HOI categories. A strong zero-shot HOI detector is supposed to be not only capable of discriminating novel interactions but also robust to positional distribution discrepancy between seen and unseen categories when locating human-object pairs. However, top-performing zero-shot HOI detectors rely on seen and predefined unseen categories to distill knowledge from CLIP and jointly locate human-object pairs without considering the potential positional distribution discrepancy, leading to impaired transferability. In this paper, we introduce CLIP4HOI, a novel framework for zero-shot HOI detection. CLIP4HOI is developed on the vision-language model CLIP and ameliorates the above issues in the following two aspects. First, to avoid the model from overfitting to the joint positional distribution of seen human-object pairs, we seek to tackle the problem of zero-shot HOI detection in a disentangled two-stage paradigm. To be specific, humans and objects are independently identified and all feasible human-object pairs are processed by Human-Object interactor for pairwise proposal generation. Second, to facilitate better transferability, the CLIP model is elaborately adapted into a fine-grained HOI classifier for proposal discrimination, avoiding data-sensitive knowledge distillation. Finally, experiments on prevalent benchmarks show that our CLIP4HOI outperforms previous approaches on both rare and unseen categories, and sets a series of state-of-the-art records under a variety of zero-shot settings.
Human-Object Interaction (HOI) detection aims at locating paired humans and objects and identifying their interactions. It finds applications in several downstream tasks, including surveillance, robotics, and human-computer interaction. Despite significant progress in recent years, in most previous works, the task of HOI detection has been conventionally restricted to predefined HOI categories. Given the flexibility of verb-object combinations in human-object interaction scenarios, obtaining a comprehensive dataset is laborious and time-consuming. Moreover, the obtained detectors also struggle to generalize to unseen HOI categories, even if both verb and object in the novel category are present (but never composed together) in the training set. These facts motivate and highlight the exploration of zero-shot HOI detection, which targets to development of transferable HOI detectors.
Latest zero-shot HOI detectors [31, 44, 46, 48] seek to harness the general visual and linguistic knowledge of CLIP [38] for identifying novel HOIs. Despite notable advancements over conventional compositional learning approaches [2, 15, 17-19, 34, 36], these methods exhibit certain limitations.
†Corresponding authors: Wengang Zhou and Houqiang Li
(a) An overview of our CLIP4HOI
Figure 1: In our CLIP4HOI, the CLIP model is elaborately adapted for fine-grained pairwise HOI discrimination, avoiding data-sensitive knowledge distillation (a). Thanks to the distribution-agnostic proposal generation, our CLIP4HOI demonstrates better robustness when encountering large positional distribution discrepancy between seen and unseen HOI categories (b)(c). The distribution statistics are the angles between the line from the person to the object and the x-axis. The distribution discrepancy is measured with KL divergence.
(b) Distribution discrepancy
(c) CLIP4HOI vs EoID
First, the prevailing approaches [31, 46] adhere to the one-stage framework, where each query or pair thereof is required to jointly localize the human and the object. This strategy bears the inherent risk of overfitting the decoder to the joint positional distribution of human-object pairs for seen HOI categories, which results in compromised efficacy of the model when encountering novel HOIs that exhibit significant distribution discrepancy with the seen categories, as shown in Figure 1b. Second, the knowledge of CLIP is transferred to the detector through knowledge distillation, the effectiveness of which largely depends on the completeness of the training data. However, given the absence of unseen categories during training, the distillation process is dominated by samples from seen categories, resulting in impaired generalization of the model to unseen categories. Although the latest approach [46] tries to mine potential unseen categories during training, the text descriptions of unseen categories are required in advance. As a result, the generalization capability of the detector is restricted to predefined unseen HOI categories, which is not conducive to practical applications.
To this end, we propose CLIP4HOI, a novel framework for zero-shot generalizable HOI detection. On the one hand, in view of the overfitting issue associated with the joint localization of humans and objects, we seek to tackle the problem of zero-shot HOI detection in a two-stage paradigm, as shown in Figure 1a. Specifically, with the help of an off-the-shelf object detector, the first stage focuses on the accurate identification of humans and objects from images, followed by traversing all feasible human-object combinations in the Human-Object (HO) interactor for pairwise HOI proposal generation. On the other hand, instead of employing data-sensitive knowledge distillation from CLIP to existing detectors, we demonstrate that the adapted CLIP is a natural fit for generalizable fine-grained HOI discrimination. Specifically, based on the generated pairwise proposals, a transformer-based HOI decoder is designed to aggregate contextual visual clues from the output of the CLIP image encoder. HOI scores are then obtained in the HOI classifier by comparing the text embeddings of prompted HOI descriptions with the visual features from different granularities. The final scores for all human-object pairs are the integration of HOI scores and the object class priors out of the object detector. Thanks to the distribution-agnostic pairwise proposal generation and the strong transferability of CLIP, the resulting HOI detector exhibits better robustness to positional distribution discrepancy and superior generalization capability to unseen HOI categories, as shown in Figure 1c.
To verify the effectiveness, we conducted extensive experiments on two prevalent HOI detection benchmarks, i.e., HICO-DET [4] and V-COCO [14]. Results show that our approach exhibits superior performance under a variety of zero-shot settings. Specifically, compared to previous state-of-the-art methods GEN-VLKT [31] and EoID [46], our CLIP4HOI achieves absolute mAP gains of 6.43, 4.67, and 8.41 on the unseen categories of RF-UC, NF-UC, and UO settings, respectively. Moreover, our CLIP4HOI also demonstrates exceptional performance under the fully supervised setting, particularly in rare categories where it outperforms Liu et al. [32] by  $3.65\mathrm{mAP}$  on the HICO-DET dataset.
Overall, we make the following three-fold contributions:
- Given the compromised generalization capability of the joint human-object localization, we propose a novel two-stage framework, termed CLIP4HOI, for zero-shot HOI detection, which leverages the generalizable knowledge of CLIP for unseen interaction identification.
- Instead of relying on data-sensitive knowledge distillation, we demonstrate that by carefully designing the adaptation modules, the CLIP model itself exhibits a strong capability for fine-grained HOI discrimination, enabling the unrivaled zero-shot transferability of our CLIP4HOI.
- We conduct extensive experiments on two prevalent benchmarks to verify the effectiveness of the proposed approach. Compared to previous methods, our CLIP4HOI exhibits remarkable performance and sets a series of state-of-the-art records.
Human-Object Interaction Detection: There are two mainstream solutions for HOI detection, one-stage and two-stage. Two-stage methods [4, 10–12, 15, 25, 26, 28, 29, 33, 37, 42, 47, 56] first detect human and object instances in the image with an off-the-shelf object detector. Then relation modeling strategies like multi-steam fusion [4] and graph reasoning [10] are designed to identify interactions between each pair of detected humans and objects. One-stage methods, on the other hand, undertake detection, association, and classification in a single stage. Early one-stage methods adopt interaction point [30, 45] or union box [22] as anchors to facilitate the joint localization of humans and objects. Recent one-stage approaches [5, 6, 21, 23, 24, 40, 41, 43, 52, 54, 57] have been inspired by the query-based object detector [3], wherein a group of learnable queries is employed in the transformer decoder to simultaneously predict  $\langle \text{human}, \text{verb}, \text{object} \rangle$  triplets. Despite the promising performance under the fully-supervised setting, we argue that such a joint localization strategy tends to overfit the decoder to the joint position distribution of humans and objects for known verb-object combinations. Therefore in this work, we seek to solve the problem of zero-shot HOI detection in a more generalizable two-stage paradigm.
Contrastive Language-Image Pre-Training: Contrastive Language-Image Pre-Training (CLIP) [38] is a multimodal learning framework that entails joint training of an image encoder and a text encoder on large-scale image-text pairs. While traditional approaches rely exclusively on either visual or textual cues for learning, CLIP utilizes both modalities to acquire a shared representation space that aligns images and their corresponding captions in a common embedding space. The outcome of this joint training is the ability of CLIP to facilitate zero-shot and few-shot learning on a variety of downstream tasks, including image classification [38], object detection [8, 9, 13], and image retrieval [1]. The success of CLIP also opens up new avenues for zero-shot HOI detection.
Zero-shot Human-Object Interaction Detection: Zero-shot human-object interaction (HOI) detection strives to develop a detector that can effectively generalize to HOI categories not encountered during training. It is of practical significance due to the flexibility of verb-object combinations.
Early works [2, 15, 17-20, 34, 36] mainly adopt compositional learning for zero-shot HOI detection. VCL [17] decomposes HOI representation into object- and verb-specific features and then composes novel HOI samples in the feature space via stitching the decomposed features. FCL [19] introduces an object fabricator to generate large-scale HOI samples for rare and unseen categories. ATL [18] decouples HOI representation into a combination of affordance and object representation. Novel interactions are then discovered by combining affordance representation and novel object representation from additional object images. ConsNet [34] proposes a knowledge-aware framework for explicit relation modeling among objects, actions, and interactions in an undirected graph. It leverages graph attention networks to enable knowledge propagation among HOI categories and their constituents. SCL [20] devises a self-compositional learning framework for HOI concept discovery.
Recently, the development of multimodal learning has led to a surge of interest in transferring knowledge from pre-trained vision-language models like CLIP [38] to existing HOI detectors for high-performance zero-shot HOI detection. GEN-VLKT [31] extracts CLIP text embeddings for prompted HOI labels to initialize the classifier and leverages the CLIP visual feature to guide the learning of interactive representation. EoID [46] distills the distribution of action probability from CLIP to the HOI model and detects potential action-agnostic interactive human-object pairs by applying an interactive score module combined with a two-stage bipartite matching algorithm. To avoid data-sensitive knowledge distillation and facilitate better transferability, in our approach, the CLIP model is directly adapted as a fine-grained classifier for generalizable HOI identification.
As a concurrent work, HOICLIP [35] adopts the one-stage design following GEN-VLKT [2] and proposes query-based knowledge retrieval for efficient knowledge transfer from CLIP to HOI detection tasks. During evaluation, it exploits zero-shot CLIP knowledge as a training-free enhancement.
Differently, our CLIP4HOI leverages the two-stage proposal generation strategy to mitigate the overfitting of the method to the joint positional distribution of human-object pairs during training.
In this section, we introduce the problem formulation of zero-shot HOI detection. Let  $\mathbb{V} = \{v_1,v_2,\dots ,v_{N_v}\}$  be the set of action verbs and  $\mathbb{O} = \{o_1,o_2,\dots ,o_{N_o}\}$  be the set of objects that can be interacted with. The conventional fully-supervised solutions try to incorporate samples of all feasible verb-object pairs  $\mathbb{C} = \{(v_i,o_j)|v_i\in \mathbb{V};o_j\in \mathbb{O}\}$  into the training set and assign each sample a discrete label for to learn a closed-set classifier. Given the inherent flexibility of verb-object combinations, it is impractical to collect a complete HOI dataset. Consequently, researchers have pursued the zero-shot HOI detection paradigm as an alternative direction, where the detector needs to generalize well to unseen categories  $\mathbb{C}_{\mathrm{unseen}}$  during the test. Let  $\mathbb{V}_{\mathrm{seen}}\subset \mathbb{V},\mathbb{O}_{\mathrm{seen}}\subset \mathbb{O},$  and  $\mathbb{C}_{\mathrm{seen}}\subset \mathbb{C}\setminus \mathbb{C}_{\mathrm{unseen}}$  denote seen verbs, seen objects, and seen categories during training, respectively. According to whether verbs and objects in the unseen categories  $\mathbb{C}_{\mathrm{unseen}}$  exist during training, zero-shot HOI detection can be divided into three settings: (1) Unseen Composition (UC), where for all  $(v_{i},o_{j})\in \mathbb{C}_{\mathrm{unseen}}$ , we have  $v_{i}\in \mathbb{V}_{\mathrm{seen}}$  and  $o_j\in \mathbb{O}_{\mathrm{seen}}$ ; (2) Unseen Object (UO), where for all  $(v_{i},o_{j})\in \mathbb{C}_{\mathrm{unseen}}$ , we have  $v_{i}\in \mathbb{V}_{\mathrm{seen}}$  and  $o_j\notin \mathbb{O}_{\mathrm{seen}}$ ; (3) Unseen Verb (UV), where for all  $(v_{i},o_{j})\in \mathbb{C}_{\mathrm{unseen}}$ , we have  $v_{i}\notin \mathbb{V}_{\mathrm{seen}}$  and  $o_j\in \mathbb{O}_{\mathrm{seen}}$ .
3.2 Revisiting DETR
DEtection Transformer (DETR) [3] is an end-to-end object detector that approaches object detection as a direct set prediction problem. At the core of DETR is a transformer encoder-decoder architecture, which is responsible for processing the input image and producing a set of object queries that correspond to potential objects in the image. DETR uses a bipartite matching algorithm that associates the object queries with ground-truth objects during training. Thanks to this query-based detection paradigm, we can easily obtain the features corresponding to each detected instance, enabling efficient extraction of pairwise representation for interaction recognition.
4.1 Framework Overview
In this section, we give an overview of the proposed framework. As shown in Figure 2, The proposed CLIP4HOI comprises three key components: Human-Object interactor (HO interactor), HOI decoder, and HOI classifier. Given a single image as input, all humans and objects are first detected using a pre-trained transformer-based detector DETR [3]. Then, the HO interactor takes the decoded query features that correspond to individual human-object pairs as input and derive pairwise HOI tokens through feature interaction and pairwise spatial information injection. After that, with pairwise HOI tokens serving as the input queries, the HOI decoder is engaged to generate pairwise HOI visual features by aggregating the contextual visual clues from the output of the CLIP image encoder. Finally, in the HOI classifier, global and pairwise HOI scores are obtained by comparing the text embeddings of prompted HOI descriptions with the class token from the CLIP image encoder and the decoded pairwise HOI visual features, respectively.
4.2 HO Interactor
Following [51], the detections from DETR are first post-processed with non-maximum suppression and thresholding. The filtered result is denoted as  $\{\mathbf{B},\mathbf{X},\mathbf{C},\mathbf{S}\}$ , where  $\mathbf{B} \in \mathbb{R}^{N_d \times 4}$ ,  $\mathbf{X} \in \mathbb{R}^{N_d \times C_d}$ ,  $\mathbf{C} \in \{0,1,\dots,N_c - 1\}^{N_d}$ , and  $\mathbf{S} \in [0,1]^{N_d}$  denote bounding boxes, decoded query features, object classes, and confidence scores, respectively.  $N_d$  denotes the number of detected instances. Then, we perform feature interaction between instances and traverse all valid human-object pairs to generate pairwise HOI tokens  $\mathbf{Q}$ . These are done through the Human-Object Interactor (HO Interactor). Specifically, the HO Interactor borrows the design of the lightweight interaction head in UPT [51], which consists of a cooperative layer and a competitive layer. Given the detected boxes, unary and
Figure 2: The pipeline of the proposed CLIP4HOI. CLIP4HOI is developed based on the prevalent DETR [3] with three new components, i.e., HO interactor, HOI decoder, and HOI classifier. Based on the detections from DETR, the HO interactor generates pairwise HOI tokens by feature interaction and spatial information injection. The HOI decoder then aggregates the CLIP image representation to produce pairwise HOI visual features, with the pairwise HOI tokens serving input queries. Finally, in the HOI classifier, global and pairwise HOI scores are obtained by comparing the text embeddings of HOI descriptions with the CLIP image class token and the pairwise HOI visual features, respectively.
pairwise spatial features [50] (box center, width, height, pairwise IOU, relative area, and direction) are first extracted and mapped into pairwise positional encodings  $\mathbf{E} \in \mathbb{R}^{N_d \times N_d \times C_r}$ . Then, decoded query feature  $\mathbf{X} \in \mathbb{R}^{N_d \times C_d}$  of detected instances are fed into the cooperative layer along with  $\mathbf{E}$  for feature interaction and pairwise spatial information injection:
$$
\hat {\mathbf {X}} = \operatorname {C o o p L a y e r} (\mathbf {X}, \mathbf {E}) \in \mathbb {R} ^ {N _ {d} \times C _ {r}}. \tag {1}
$$
After that, the competitive layer generates pairwise HOI tokens  $\mathbf{Q}$  for all valid human-object pairs:
$$
\mathbf {i d x} _ {\text {v a l i d}} = \{(u, v) | u \neq v, c _ {u} = “ \text {h u m a n}" \}, \tag {2}
$$
$$
\mathbf {X} _ {\text {p a i r}} \in \mathbb {R} ^ {N _ {\text {p a i r}} \times 2 C _ {r}}, \quad \text {w h e r e} \mathbf {X} _ {\text {p a i r}} ^ {[ i ]} = \operatorname {C A T} \left(\hat {\mathbf {X}} ^ {[ u ]}, \hat {\mathbf {X}} ^ {[ v ]}\right), (u, v) = \mathbf {i d x} _ {\text {v a l i d}} ^ {[ i ]}, \tag {3}
$$
$$
\mathbf {E} _ {\text {p a i r}} \in \mathbb {R} ^ {N _ {\text {p a i r}} \times C _ {r}}, \quad \text {w h e r e} \mathbf {E} _ {\text {p a i r}} ^ {[ i ]} = \mathbf {E} ^ {[ u, v ]}, (u, v) = \mathbf {i d x} _ {\text {v a l i d}} ^ {[ i ]}, \tag {4}
$$
$$
\mathbf {Q} = \operatorname {C o m p L a y e r} \left(\mathbf {X} _ {\text {p a i r}}, \mathbf {X} _ {\text {g l o b}}, \mathbf {E} _ {\text {p a i r}}\right) \in \mathbb {R} ^ {N _ {\text {p a i r}} \times 2 C _ {r}}, \tag {5}
$$
where  $\mathbf{idx}_{\mathrm{valid}}$  is the index set of valid human-object pairs and has size  $N_{\mathrm{pair}}$ .  $\mathbf{X}_{\mathrm{glob}} \in \mathbb{R}^{1 \times C_b}$  is the global visual feature from the backbone of DETR [3]. Please refer to [51] and also the supplementary material for more technical details of the HO interactor.
4.3 HOI Decoder
After obtaining the pairwise HOI tokens, we feed them into the HOI decoder to aggregate contextual features from the CLIP image encoder, yielding pairwise HOI visual features. Specifically, the HOI decoder is stacked by  $N_{l}$  customized transformer layers, each of which contains a cross-attention block, a self-attention block, and a feed-forward network (FFN) block. Given the collection of pairwise HOI tokens  $\mathbf{Q} \in \mathbb{R}^{N_{\mathrm{pair}} \times 2C_r}$ , patch level CLIP image features  $\mathbf{F} \in \mathbb{R}^{N_{\mathrm{patch}} \times C_c}$ , and the global CLIP image class token  $\mathbf{I} \in \mathbb{R}^{1 \times C_c}$ , the pairwise HOI visual features  $\mathbf{P} \in \mathbb{R}^{N_{\mathrm{pair}} \times C_c}$  are computed as follows:
$$
\mathbf {H} _ {0} = \operatorname {L i n e a r} (\mathbf {Q}), \tag {6}
$$
$$
\mathbf {H} _ {l} ^ {\prime} = \operatorname {M H C A} \left(\operatorname {L N} \left(\mathbf {H} _ {l - 1}\right), \operatorname {L N} (\mathbf {F}), \operatorname {L N} (\mathbf {F})\right) + \mathbf {H} _ {l - 1}, \quad l \in 1, \dots , N _ {l} \tag {7}
$$
$$
\mathbf {H} _ {l} ^ {\prime \prime} = \operatorname {M H S A} (\ln (\operatorname {C A T} (\mathbf {I}, \mathbf {H} _ {l} ^ {\prime}))) + \operatorname {C A T} (\mathbf {I}, \mathbf {H} _ {l} ^ {\prime}), \quad l \in 1, \dots , N _ {l} \tag {8}
$$
$$
\mathbf {H} _ {l} = \operatorname {M L P} \left(\operatorname {L N} \left(\mathbf {H} _ {l} ^ {\prime \prime [ 1: ]}\right)\right) + \mathbf {H} _ {l} ^ {\prime \prime [ 1: ]}, \quad l \in 1, \dots , N _ {l} \tag {9}
$$
$$
\mathbf {P} = \mathbf {H} _ {N _ {l}}, \tag {10}
$$
where Linear denotes the linear projection  $\mathcal{F}_{\mathrm{proj}}(\cdot):\mathbb{R}^{2C_r}\mapsto \mathbb{R}^{C_c}$  that maps the pairwise HOI tokens into CLIP's joint feature space. LN, MHCA, MHSA, MLP, and CAT denote layer normalization, multi-head cross-attention, multi-head self-attention, multilayer perceptron, and concatenation, respectively. So far, the obtained pairwise HOI visual features  $\mathbf{P}$  integrate the relative positional information between humans and objects as well as the contextual visual clues from the large-scale pre-trained CLIP model, greatly promoting the subsequent fine-grained HOI recognition.
4.4 HOI Classifier
Given the notable transferability of the CLIP model, this study endeavors to adapt it as a fine-grained classifier for transferable HOI detection. Thanks to the shared embedding space of image and text learned by CLIP after large-scale contrastive pre-training, we can perform fine-grained HOI recognition for both seen and unseen HOI categories by transforming their verb-object combinations into textual descriptions and comparing their embeddings with pairwise HOI visual features. Considering that the manual prompt lacks task-specific heuristics and robustness [55], we choose to train a learnable prompt for text embedding generation. Specifically, we insert several learnable tokens before the verb and object of each HOI category following [44, 55], which we denote as  $\mathsf{[PREFIX]}$  and [CONJUN], respectively. We denote the obtained text embedding as  $\mathbf{T} \in \mathbb{R}^{N_c \times C_o}$  ( $N_c$  denotes the number of HOI categories). As shown in Figure 2, we compute both global and pairwise HOI scores as follows:
$$
\mathbf {I} ^ {\prime} = \operatorname {L i n e a r} (\mathbf {I}), \quad \mathbf {P} ^ {\prime} = \operatorname {L i n e a r} (\mathbf {P}), \tag {11}
$$
$$
\hat {\mathbf {T}} = \operatorname {N o r m} (\mathbf {T}), \quad \hat {\mathbf {I}} = \operatorname {N o r m} \left(\mathbf {I} ^ {\prime}\right), \quad \hat {\mathbf {P}} = \operatorname {N o r m} \left(\mathbf {P} ^ {\prime}\right), \tag {12}
$$
$$
\mathbf {S} _ {\text {g l o b}} = \operatorname {S i g m o i d} \left(\hat {\mathbf {I}} \hat {\mathbf {T}} ^ {\top} / \tau\right), \quad \mathbf {S} _ {\text {p a i r}} = \operatorname {S i g m o i d} \left(\hat {\mathbf {P}} \hat {\mathbf {T}} ^ {\top} / \tau\right), \tag {13}
$$