Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 07:31:00

Abstract: Zero-shot human-object interaction (HOI) detector is capable of generalizing to HOI categories even not encountered during training.
Inspired by the impressive zero-shot capabilities offered by CLIP, latest methods strive to leverage CLIP embeddings for improving zero-shot HOI detection. However, these embedding-based methods train the classifier on seen classes only, inevitably resulting in seen-unseen confusion of the model during testing. Besides, we find that using prompt-tuning and adapters further increases the gap between seen and unseen accuracy. To tackle this challenge, we present the first generation-based model using CLIP for zero-shot HOI detection, coined HOIGen. It allows to unlock the potential of CLIP for feature generation instead of feature extraction only. To achieve it, we develop a CLIP-injected feature generator in accordance with the generation of human, object and union features. Then, we extract realistic features of seen samples and mix them with synthetic features together, allowing the model to train seen and unseen classes jointly. To enrich the HOI scores, we construct a generative prototype bank in a pairwise HOI recognition branch, and a multi-knowledge prototype bank in an image-wise HOI recognition branch, respectively. Extensive experiments on HICO-DET benchmark demonstrate our HOIGen achieves superior performance for both seen and unseen classes under various zero-shot settings, compared with other top-performing methods.

Main paper: 
 related work 2.1 hoi detection and beyond: 
 HOI detection task is an integration of object detection, humanobject pairing and interaction recognition. Broadly speaking, existing work can be categorized into one-stage and two-stage methods.Initially, the two-stage methods [6, 11, 12, 27, 32, 36, 42, 45] predominated in HOI detection. Its first stage utilizes off-the-shelf object detectors to identify humans and objects according to their class labels. Subsequently, independent modules are designed in the second stage to classify interactions between each human-object pair.On the other hand, thanks to the recent emergence of DEtection TRansformer (DETR) [4] , one-stage methods have witnessed a rapid evolution [7, 10, 22-24, 29, 41, 44, 54] . The methods are typically built upon a pre-trained DETR, predicting HOI triples directly in an end-to-end fashion.Zero-shot HOI. Beyond the conventional setting above, an increasing line of work tends to research zero-shot HOI detection, allowing the model to infer some new HOI categories being not seen during training. To tackle this problem, early endeavors developed combinatorial learning, enabling the prediction of HOI triplets during inference [2, 20, 21, 35] . Inspired by the rapid advancements in visual-language pre-trained models like CLIP [38] , more recent work has tapped into zero-shot generalization capabilities of these models to transfer prior knowledge for identifying unseen HOI categories [3, 27, 33, 36, 37, 42, 43, 48] . Specifically, these methods employ CLIP image encoder to encode raw images and subsequently extract features related to human-object pairs using bounding boxes acquired from an off-the-shelf detector. Subsequently, we achieve the HOI score with a given image and a candidate human-object interaction triplet. Although these embedding-based methods have shown promise of CLIP embeddings for zero-shot learning, they are inefficient for addressing the bias between seen and unseen classes. Different from them, we devise a generative-based paradigm for zero-shot HOI detection, with an aim of exploiting CLIP for feature generation and alleviating the seen-unseen bias explicitly.Generative zero-shot learning allows the model to generate unseen samples from their corresponding attributes, converting the conventional zero-shot learning to a classic supervised learning problem. Generative adversarial networks (GANs) [16] and variational autoencoders (VAEs) [25] are two prominent members of generative models used for generalized zero-shot learning task which needs to infer both seen and unseen classes. Through this generative paradigm, the model can capture underlying data distributions and generate diverse samples resembling unseen categories [1, 9, 17, 28, 40, 49, 56] . In order to improve the generation quality and stability, a few methods combine the advantages of VAE and GAN for joint training to generate high-quality visual features for unseen categories [14, 15, 26] . Recently, some pioneering works study integrating CLIP to feature generation [19, 34, 39, 47] . For instance, CLIP-Forge [39] obtains the latent space of shapes by training an auto-encoder and then trains a normalized flow network to align the distribution of shape embeddings with image features from a pre-trained CLIP image encoder. Likewise, CLIP-GEN [47] obtains cross-modal embeddings through CLIP and converts the image into a sequence of discrete tokens in the VQGAN codebook space, training an autoregressive transformer and generating coherent image tokens. A recent work similar to ours is SHIP [46] , which inserts pre-trained CLIP image-text encoders into a variational autoencoder (VAE) model, so as to generate synthetic image features given arbitrary names of unseen classes. However, this generation method in SHIP is designed for simple image classification, but is challenged the complex compositions of human, objects and actions in HOI detection. In this work, we develop a new CLIP-injected feature generation model which is tailored specifically for zero-shot HOI detection effectively. 
preliminaries 3.1 problem formulation: 
 Given an image 𝐼 as input, its detection results are in the form of fivetuple (𝑏 ℎ , 𝑠 ℎ , 𝑏 𝑜 , 𝑠 𝑜 , 𝑐 𝑜 ), where 𝑏 ℎ , 𝑏 𝑜 ∈ R 4 represent the bounding boxes of detected human and object instances; 𝑐 𝑜 ∈ {1, ..., 𝑂 } indicates the object category; 𝑠 ℎ and 𝑠 𝑜 are the confidence scores of detected human and objects. The action category 𝑐 𝑎 ∈ {1, ..., 𝐴} is then identified and a confidence score 𝑠 is used for each human-object pair. The final result is presented in the form of 𝐻𝑂𝐼 𝑖,𝑗 =< ℎ, 𝑣 𝑖 , 𝑜 𝑗 > triplet, where ℎ represents the human, 𝑣 𝑖 represents the category of the verb, and 𝑜 𝑗 represents the category of the object. According to whether verbs and objects in the unseen categories C 𝑢𝑛𝑠𝑒𝑒𝑛 exist during training, zero-shot HOI detection can be further divided into three settings: (1) Unseen Composition (UC), where for all (𝑣 𝑖 , 𝑜 𝑗 ) ∈ C 𝑢𝑛𝑠𝑒𝑒𝑛 , we have 𝑣 𝑖 ∈ V 𝑠𝑒𝑒𝑛 and 𝑜 𝑗 ∈ O 𝑠𝑒𝑒𝑛 ; (2) Unseen Object (UO), where for all (𝑣 𝑖 , 𝑜 𝑗 ) ∈ C 𝑢𝑛𝑠𝑒𝑒𝑛 , we have 𝑣 𝑖 ∈ V 𝑠𝑒𝑒𝑛 and 𝑜 𝑗 ∉ O 𝑠𝑒𝑒𝑛 ; (3) Unseen Verb (UV), where for all (𝑣𝑖, 𝑜 𝑗) ∈ C 𝑢𝑛𝑠𝑒𝑒𝑛 , we have 𝑣 𝑖 ∉ V 𝑠𝑒𝑒𝑛 and 𝑜 𝑗 ∈ O 𝑠𝑒𝑒𝑛 . Note that, C 𝑢𝑛𝑠𝑒𝑒𝑛 , O 𝑠𝑒𝑒𝑛 and V 𝑠𝑒𝑒𝑛 represent unseen combination categories, seen object categories, and seen verb categories, respectively.Generalized zero-shot learning suffers from inherently over-fitting on seen classes and weakly generalization on unseen ones. This problem becomes more difficult for HOI, as the number of HOI categories is much larger than that of object categories only. After a thorough examination of existing CLIP-based methods for zeroshot HOI detection [27, 36, 37, 42, 43] , a consistent phenomenon we observe is they follow an embedding-based paradigm to infer unseen categories. Yet, these methods fail to avoid confusion between seen and unseen classes during testing, because the primary challenge persists in the absence of unseen categories during training. In addition, some of them leverage prompt tuning and adapters to better adapt CLIP on HOI benchmarks, resulting in an improved performance of both seen and unseen categories. Nevertheless, they overlook a significant consequence that the seen-unseen bias becomes more severe, because of adapting CLIP with seen data only. For instance, ADA-CM [27] , which is a recent state-of-theart, introduces two different settings. One is directly using CLIP in a training-free (TF) fashion, and the other is adding a learnable adapter to fine-tune (FT) the image embeddings obtained by CLIP. As shown in Fig. 2 , thanks to the use of such an adapter, the seen accuracy obtains a substantial boost from 24.54% to 34.35%, and meanwhile the unseen accuracy increases from 26.83% to 27.63%. Nonetheless, we should be aware that the gap between seen and unseen accuracy also improves a lot, from -2.29% to 6.72%. Through such a relative comparison, we can conjecture that fine-tuning the CLIP embeddings benefits the adaptation ability with seen data, whereas degenerates the generalization ability on unseen data. Differently, our method takes both adaptation and generalization into account. It can be seen that the unseen accuracy by our method has a more remarkable gain, making the seen-unseen gap decrease to 3.56%. We will elaborate on the details of our method below. 
methodology 4.1 overview framework: 
 The framework of our proposed HOIGen, as depicted in Fig. 3 , comprises three primary parts: CLIP-injected feature generation (Sec. 4.2), pairwise HOI recognition (Sec. 4.3) and image-wise HOI recognition (Sec. 4.4). Firstly, during the feature generation stage, we inject pre-trained CLIP into a variational autoencoder model, so as to produce synthetic features from CLIP text encoder and make them consistent with the real ones captured from CLIP image encoder. Secondly, for the pairwise HOI recognition, we adopt an off-the-shelf object detector (i.e. DETR) for bounding box detection. Then we employ CLIP image encoder for ROI-Align calculation and achieve real image features of seen samples, and collaborate the synthetic features to the real ones for training seen and unseen categories jointly. Besides, we use the CLIP-injected feature generator to construct a generative prototype bank, which aims to calculate the pairwise HOI recognition scores. Thirdly, the image-wise HOI recognition entails combining CLIP image encoder and DINO image encoder, and also involves the use of synthetic features. It develops a multi-knowledge prototype bank for computing the image-wise HOI recognition scores. Finally, we fuse the HOI scores of different branches for classifying the HOI category.The idea of extracting CLIP feature embeddings for zero-shot HOI detection has been studies in previous work [27, 36, 37, 42] . These methods, however, fail to overcome the scarcity of unseen samples explicitly, resulting in a severe bias between seen and unseen HOI categories. Beyond that, our objective is to fully harness the potential of CLIP for feature generation, thereby leveraging prior (eat, pizza) (eat) (pizza) (…, …) (…) (…) (ride, bicycle) (ride) (bicycle) (…, …) (…) (…) ü Seen ü Unseen MLP … … … 𝑃 !"#$ 𝑃 %""& (𝑃!, 𝑃", 𝑃#) 𝑃 '&%""& (𝑃!, 𝑃", 𝑃#) … 𝑃 ()*+ … 𝑃 ,*-."action of the person is jumping" "action of the person is riding" "action of the person is ……" knowledge learned by CLIP to improve the generation quality. As depicted in Fig. 3 , in contrast to traditional feature generation methods, our feature generation model comprises three branches (i.e. union, human, and object), in accordance with zero-shot HOI task. Besides, we thoroughly impose the correlation among the three branches and learn adaptive prompts to enhance their interdependence. Our feature generation is performed in two stages below.𝑠 " 𝑠 # 𝑠 $ 𝑠 % 𝑠 & 𝑠 'Stage I: Train VAE encoder 𝐸 (•) and generator 𝐺 (•). In this stage, the feature generation process needs to deal with image regions of human-object union, human only, and object only, respectively, which are denoted as 𝑖𝑚𝑔 𝑢 , 𝑖𝑚𝑔 ℎ and 𝑖𝑚𝑔 𝑜 . For brevity, we present the regions with 𝑖𝑚𝑔 𝑘 where 𝑘 ∈ (𝑢, ℎ, 𝑜). First of all, the CLIP image encoder 𝐼 (•) extracts realistic image features 𝑥 𝑘 = 𝐼 (𝑖𝑚𝑔 𝑘 ) given 𝑖𝑚𝑔 𝑘 . Afterwards, we build a variational autoencoder model, where the encoder 𝐸 (•) encodes 𝑥 𝑘 into a latent code 𝑧 𝑘 , and the generator 𝐺 (𝑧 𝑘 , 𝑐 𝑘 ) then reconstructs 𝑥 𝑘 from the latent code 𝑧 𝑘 and the corresponding learnable class name 𝑐 𝑘 . Note that, different from previous work on generic image classification [46] , our approach considers emphasizing meticulous consideration of "human" categories, where we observe distinct human expressions interacting with various objects. To achieve it, we categorize "human" prompts based on the corresponding object categories and introduce a novel prompt template (e.g. "person who interacts with <OBJECT>"). For example, if a person interacts with a bus, we label this person as "person who interacts with bus". This fine-grained categorization of humans facilitates more precise associations between humans and objects, and captures nuanced interactions between humans and specific objects. During this stage, the optimization of both the encoder 𝐸 (𝑥 𝑘 ) and the generator 𝐺 (𝑧 𝑘 , 𝑐 𝑘 ) is achieved by minimizing the following evidence lower bound (ELBO) cost:EQUATIONwhere 𝐾𝐿 denotes the Kullback-Leibler divergence, 𝑝 (𝑧 𝑘 |𝑐 𝑘 ) is a prior distribution that is assumed to be 𝑁 (0, 1), andlog 𝐺 (𝑧 𝑘 , 𝑐 𝑘 ) estimates the reconstruction loss.Stage II: Freeze encoder 𝐸 (•) and generator 𝐺 (•), and finetune MLP. This stage is dedicated to bridging the gap between synthetic and realistic images in HOIGen. It is noteworthy that the first stage aligns with features extracted by the CLIP image encoder, which may differ from the image features required in the following detector. To solve it, the same generator 𝐺 (•) learned in Stage I remains frozen during Stage II training. The input transitions from a different dataset to a category prompt 𝑐 𝑘 and a randomly initialized normal distribution 𝑁 (0, 1), and through the frozen generator 𝐺 (•). Additionally, a new multilayer perceptron (MLP) is employed to align the reconstructed synthetic features with the realistic features. Throughout this process, only the MLP is optimized, while the other components remain frozen. This optimization process is realized through minimizing mean square error:EQUATIONwhere 𝑥 ′ 𝑘 denotes the reconstructed features through generator 𝐺 (•) and 𝑥 𝑘 indicates the realistic features extracted by CLIP in HOIGen. To show the quality of our generated features, we illustrate their distributions using t-SNE in Fig 4 . We can see that, the feature distributions of different categories distinguish from each other clearly. In addition, synthesized features are distribution-aligned well with the corresponding realistic features in the space.Since most of existing methods are heavily affected by long-tail distributions, it becomes challenging to effectively manage complex scenes featuring a diverse range of interactive objects. To this end, we construct a generative prototype bank by using the generated features as key. It can alleviate the long-tail distribution, and avoid the time-consuming project of loading keys in the training samples. As shown in Fig 4 , we generate 100 synthetic features for each HOI category through the CLIP-injected feature generator, and also generate the corresponding human and object synthetic features.We then extract the seen and unseen features from them as the key of the generative prototype bank. This design provides a powerful generative prototype bank that can better capture feature distributions. We convert the training labels into multi-hot encodings 𝐿 𝑚ℎ ∈ R 𝑁 ×𝐴 as the value for generative prototype bank. Inspired by previous research [27, 36] , while extracting a rich visual feature prototypes, we also supplement semantic feature prototypes. Specifically, we first use handcrafted prompts (i.e. A photo of a person is <ACTION> an object) to generate the raw text description of interactions. Then we pass such prompt into CLIP text encoder T and obtain semantic prototype, thus enriching the prototype bank.In training phase, given the feature map M extracted by the CLIP image encoder, and simultaneously utilizing DETR to extract the bounding boxes of the images. ROI-Align [18] is then applied to derive the union feature 𝑣 𝑢 , human feature 𝑣 ℎ and object feature 𝑣 𝑜 . These elements are collectively used in the computation of pairwise HOI recognition scores 𝑠 𝑃 :EQUATIONwhere 𝐺(𝑧, 𝑐) 𝑁~(0,1)Values: 𝐿 !" … … … Generative Prototype BankGlobal context information is a crucial aspect that cannot be overlooked in HOI detection. Hence, we not only utilize the global features provided by CLIP but also leverage the self-supervision capability of DINO [5] to enhance the contextual interactions in the images. Concretely, we construct a key-value multi-knowledge prototype bank for global knowledge ensemble. This bank contains the pre-learned knowledge from both CLIP and DINO because of caching two kinds of keys. Formally, we first utilize CLIP and DINO to independently extract visual features of training images, formulated as:EQUATIONEQUATIONwhere 𝐼 𝑁 denotes training images, 𝐶𝐿𝐼𝑃 𝑣𝑖𝑠 denotes the visual encoder of CLIP in HOIGen and 𝑃 𝐶𝐿𝐼 𝑃 , 𝑃 𝐷𝐼 𝑁 𝑂 ∈ R 𝑁 ×𝐴 . In addition to the key, we utilize the same 𝐿 𝑚ℎ used in generative prototype bank as the value for the multi-knowledge prototype bank.In training phase, in order to fully extract global features, we use the image encoder of CLIP and DINO to generate feature vectors 𝑣 𝐶𝐿𝐼 𝑃 , 𝑣 𝐷𝐼 𝑁 𝑂 for the input image 𝐼 𝑁 , respectively. Then use the multi-knowledge prototype bank 𝑃 𝐶𝐿𝐼 𝑃 , 𝑃 𝐷𝐼 𝑁 𝑂 to perform imagewise HOI recognition. Finally, we obtain image-wise HOI scores 𝑠 𝐼 ∈ 𝑅 𝑁 byEQUATIONwhere 𝜆 𝐶𝐿𝐼 𝑃 , 𝜆 𝐷𝐼 𝑁 𝑂 balance the weights of different prototypes based on CLIP and DINO. To classify the HOI categories, we need to combine the HOI scores from the pairwise and image-wise branches together. The whole objective is to optimize the network parameters 𝜃 via the crossentropy loss L 𝑡𝑜𝑡𝑎𝑙 below: 𝜃 * = arg min 𝜃 L 𝑡𝑜𝑡𝑎𝑙 (𝑠 ( 
experiments 5.1 experimental protocol: 
 Datasets and Metric. We conduct extensive experiments on the most benchmarked and challenging dataset, HICO-DET [6] . It is composed of 47,776 images, with 38,118 designated for training and 9,658 for testing. The annotations for HICO-DET encompass 600 categories of HOI triplets, derived from 80 object categories and 117 action categories. Besides, 138 out of the 600 HOI categories are classified as Rare due to having fewer than 10 training instances, while the remaining 462 categories are labeled as Non-Rare. Typically, the methods are evaluated with mean Average Precision (mAP) metric.Zero-shot Settings. Following the protocols established in previous work [30, 36, 37] , To make a fair and comprehensive comparison with previous work, we construct zero-shot HOI detection with five manners: Rare First Unseen Combination (RF-UC), Non-rare First Unseen Combination (NF-UC), Unseen Combination (UC), Unseen Verb (UV), and Unseen Object (UO). Specifically, UC indicates all action and object categories are included during training, but some HOI triplets (i.e. combinations) are absent; Under the RF-UC setting, the tail HOI categories are selected as unseen categories, while NF-UC uses head HOI categories being unseen. The UV (or UO) setting indicate that some action (or object) categories are not concluded in the training set.Implementation Details. We employ DETR with ResNet-50 as backbone, and the CLIP variant with an image encoder based on ViT-B/16. The DINO model we use is also pre-trained with ResNet-50 backbone. First of all, we need to train the CLIP-injected VAE model separately. We optimize it via the AdamW optimizer with a learning rate of 10 -3 for 50 epochs. The batch size is set to 256. The dimension of the latent code 𝑧 is equal to that of the token embeddings. Afterwards, we generate 100 features for each of both seen and unseen HOI categories, and randomly choose some of them to merge with some realistic features. Remarkably, we do not use any external data or model since the generated model is also trained on the same training dataset. We then optimize the whole HOI detection model for 15 training epochs through AdamW with an initial learning rate of 10 -3 . This batch size in this training stage is 4. All experiments are conducted on a single NVIDIA RTX 4090 GPU card. Following [27] , 𝜆 𝐶𝐿𝐼 𝑃 , 𝜆 𝐷𝐼 𝑁 𝑂 , 𝜆 𝑢 , 𝜆 ℎ , 𝜆 𝑜 are all set to 0.5, while 𝜆 𝑡 is set to be 1.0, for a fair comparison.Zero-shot HOI Detection. We show a comprehensive comparison in Table 1 , where our proposed HOIGen is superior to or on par with the competitors under five zero-shot settings. First, in terms of UC, RF-UC and NF-UC settings, our approach marks a significant advancement, particularly for the performance of unseen categories. Relative to the state-of-the-art methods, CLIP4HOI [36] and ADA-CM [27] , our unseen accuracy achieves a consistent gain of 2.55%, 2.54%, 1.57% for UC, RF-UC, and NF-UC, respectively. We note that our results are also very competitive performance in both the seen categories and overall metrics. Moreover, our approach benefits improving the mAP accuracy of Full and Seen, as we also generate additional features for seen categories apart from unseen ones. Second, considering the UV setting, our relatively inferior performance in the unseen accuracy may be attributed to the abstract nature of actions, devoid of specific characteristics like objects. Besides, the same action can be applied to various objects, posing a significant [23] ResNet-50 25.10 17.34 27.42 ATL [20] ResNet-50 28.53 21.64 30.59 AS-Net [8] ResNet-50 28.87 24.25 30.25 QPIC [41] ResNet-50 29.07 21.85 31.23 MSTR [24] ResNet-50 31.17 25.31 32.92 UPT [50] ResNet-50 31.66 25.94 33.36 PVIC [51] ResNet-50 34.69 32.14 35.45 GEN-VLKT [33] ResNet-50 33.75 29. 25 35.10 ADA-CM [27] ResNet-50 33.80 31.72 34.42 HOICLIP [37] ResNet-50 34.69 31.12 35.74 CLIP4HOI [36] ResNet- 50 [37] . Third, we conduct a comparison with CLIP4HOI [36] under the UO setting. It reveals that our approach obtains superior results for all the three mAP metrics, particularly evident for the unseen results, achieving a significant increase of 4.56%. Fully-supervised HOI Detection. Apart from zero-shot settings, our feature generation can be also applicable in a fullysupervised setting. We present the comparative results in Table 2 . Overall, LOGICHOI [30] achieves the highest Full accuracy among other methods because it is designed specifically for normal HOI detection. However, this method fails to retain the superior performance under zero-shot setting (see Table 1 ). Notably, our HOIGen becomes the new state-of-the-art for rare categories, improved by approximately 0.6%. This suggests that the generated features effectively complement the limited number of real samples of rare categories, even within this fully-supervised setting.We undertake a series of ablation experiments to verify the effectiveness of the HOIGen model. All experiments are conducted under the context of the NF-UC setting.Component analysis. First of all, we conduct experiments to delineate the contribution of each component to the model. As reported in Table 3 , we study the impact of feature generation, CLIPbased and DINO-based image feature individually. It can be seen that the feature generation module acts as the most influential factor in raising the unseen accuracy performance, resulting in a 0.9% improvement over the baseline (i.e. the first row in the table). This finding underscores our motivation of exploring feature generation for zero-shot HOI detection. In addition, the use of CLIP-based image feature is observed to augment the seen accuracy by 0.61%. Moreover, by integrating DINO-based image feature, we further obtain 0.8%-0.9% gain in seen accuracy. These results reveal the 4a , we investigate the influence of varying the number of generated features on the HOIGen model. Specifically, we generate 𝑁 𝑏𝑠 features for each sample in the batch. In this way, the total of samples in each training batch becomes (𝑁 𝑏𝑠 + 1) × 𝑏𝑎𝑡𝑐ℎ -𝑠𝑖𝑧𝑒. We see that, the unseen accuracy yields improvement along with the addition of generated features, suggesting a positive impact on unseen categories. However, as more generated features are integrated, the Full and Seen performance gradually declines. The main reason is that the feature distributions of real data becomes disrupted, when more and more generated features dominate the feature space. Finally, we choose 𝑁 𝑏𝑠 = 1 in the experiments.Construction of prototype bank. This experiment is to validate the construction of our generative prototype bank. In Table 4b , we compare several construction methods, including using realistic features alone ('R'), adding realistic and generated features ('R+G'), concatenating realistic and generated features ('R ⊕ G'), and using generated features alone ('G'). Experimental results demonstrate that after adding generated features, the performance of the network increases significantly. This may be because the generated features contain unseen sample features, making the generative prototype bank richer. As for the reason why using only generated features as a prototype bank performs best, it can be considered that the fusion of both realistic and generated features may lead to the confusion in the feature distribution.Size of generative prototype bank. This experiment aims to study the impact of the size 𝑁 𝑠𝑖𝑧𝑒 of generative prototype bank. As shown in Table 4c , the highest scores occurs when 𝑁 𝑠𝑖𝑧𝑒 = 2. When we expand the bank size further (i.e. 3 and 4), the performance witnesses a decline on all the three metrics. The main reason might be the noisy prototypes involved in the bank. Fortunately, we can control the size to avoid this problem effectively.In addition to quantitative results above, we further elaborate some qualitative results as shown in Fig. 5 . We observe the following: (1) Our method demonstrates effectiveness in identifying samples of rare categories within the dataset. For example <person,wash,bus> and <person,smell,broccoli>, they are rare HOI combinations, especially <person, smell,broccoli> is easily confused with <person,eat,broccoli>. (2) Even in the multi-label scenario, our method exhibits notable discriminative capability towards unseen categories. In particular for the second image in the first row containing five HOIs , four of which belong to the unseen categories, our method recognizes them accurately. (3) Some semantically similar actions may lead to recognition errors, such as <person, throw, frisbee> and <person, spin, frisbee>. However, other semantically similar actions can still be accurately recognized, such as <person, hold, hot dog> and <person, carry, hot dog>. The majority of these human-object pairs are accurately recognized by our method. 
conclusion: 
 We have proposed HOIGen, a pioneering method for zero-shot HOI detection that harnesses the capabilities of CLIP for feature generation. By fully leveraging the generated synthetic features, our approach effectively captures the feature distributions in the dataset, facilitating robust performance in zero-shot HOI detection scenarios across both seen and unseen categories of HOI. The generated features are fed into a generative prototype bank for computing pairwise HOI scores. Remarkably, we do not use any external data or model since the generated model is also trained on the same training dataset. Besides, the image-wise HOI recognition branch is responsible for extracting global features by combining CLIP and DINO encoders, thereby constructing a multi-knowledge prototype bank for image-wise HOI scores. Finally, the scores from the two branches are fused together to predict the HOI category. Extensive experiments conducted on the HICO-DET benchmark validate the effectiveness of HOIGen. In future work, it is promising to generate more diversified visual features aligning better with the distributions of realistic features.  
✗: 
 Figure 1 : Differences between existing embedding-based paradigm and our generation-based paradigm for zero-shot HOI detection. The former exploits CLIP to train visual and semantic embeddings of seen HOI categories only. Beyond that, our generation-based paradigm develops a new CLIP-injected feature generation module given either seen or unseen class names. The generated features enable the model to train seen and unseen HOI categories jointly. Besides, we construct generative prototype bank and multi-knowledge prototype bank to enrich the HOI scores.in images [6, 11, 12, 23, 24, 32, 41] . Existing studies on HOI detection are typically devoted to closed-domain scenarios, assuming all test classes have been seen at training stage. However, the methods are hardly applied to open-domain scenarios where some test classes are unseen and disjoint from seen ones. Considering the substantial time and effort involved in acquiring all classes in advance, zero-shot HOI detector remains a significant goal, as it is capable of generalizing to unseen HOI categories not encountered during training at all.Thanks to its impressive generalization capability, contrastive language-image pre-training (CLIP) [38] has been a key remedy for various low-shot learning tasks [52, 53, 55] . Following this trend, a few works [27, 36, 37, 48] position CLIP as a valuable source of prior knowledge for zero-shot HOI detection, effectively discerning and understanding unseen interactions between human and objects. As depicted in Fig. 1(a) , these methods follow an embedding-based paradigm, which utilizes CLIP embeddings to recognize seen HOI categories during training, and transfers the learned knowledge to unseen categories at inference. Despite recent advancements, this paradigm trains the classifier only on seen classes, inevitably leading the model to seen-unseen confusion during testing. That means some samples of unseen categories might be mis-classified into the set of seen categories. Moreover, some efforts suggest fine-tuning CLIP embeddings by incorporating learnable prompts or adding adapters, so as to further strengthen the adaptation on downstream tasks [13, 52, 53] . For instance, ADA-CM [27] integrates a learnable adapter into CLIP image encoder and achieves a significant boost in the performance of both seen and seen HOI categories. However, we note that current methods overlook a key phenomenon: fine-tuning CLIP embeddings enlarges the performance gap between seen and unseen HOI categories. The primary reason is that, the fine-tuned embeddings become more influenced by image samples of seen categories, resulting in impaired generalization of the model to unseen categories. Hence, the remaining challenge for zero-shot HOI detection is how we can adapt CLIP on seen categories and meanwhile retain its generalization on unseen ones.Instead of the embedding-based paradigm above, we reveal that leveraging generation-based paradigm for zero-shot HOI detection can alleviate the seen-unseen bias explicitly. To achieve it, we propose the first feature generation model for zero-shot HOI detection, coined HOIGen. As shown in Fig. 1 (b), HOIGen aims to harness the potential of CLIP for generating image features, rather than utilizing CLIP for feature extraction solely. In this way, it trains the model for all HOI categories in a supervised learning manner. Specifically, we devise a CLIP-injected feature generator based on a variational autoencoder model, contributing the prior knowledge learned in CLIP to the feature generation of human-object pairs, humans, and objects. After the feature generator is trained, we use it to produce a variety of synthetic image features used by a HOI detection model, which comprises a pairwise HOI recognition branch and an image-wise HOI recognition branch. For the former branch, we integrate an off-the-shelf object detector (i.e. DETR) with a CLIP image encoder to obtain realistic image features of seen samples. Then we merge both synthetic and realistic features together and pass them into a generative prototype bank, resulting in several pairwise HOI scores. Subsequently, in the image-wise HOI recognition branch, we obtain two global image features from CLIP and DINO [5] image encoders, respectively, and thereby construct a multi-knowledge prototype bank, offering more comprehensive image-wise HOI scores. Eventually, we fuse all the HOI scores from the two branches to classify seen and unseen categories jointly.Overall, this work has three-fold contributions below:• This work is the first to address zero-shot HOI detection via a generation-based paradigm using CLIP. It alleviates the model overfitting on seen categories and improves the generalization on unseen ones. • We devise a new CLIP-injected feature generator, synthesizing image features of humans, objects and their unions simultaneously. Besides, we utilize the synthetic features to construct a generative prototype bank for pairwise HOI recognition, and a multi-knowledge prototype bank for image-wise HOI recognition, respectively. • Extensive experiments on HICO-DET benchmark exhibit superior performance of our HOIGen over prior state-of-thearts. Our method achieves an absolute mAP gain of 2.54 on unseen categories and of 1.56 on seen categories. 
 
 Appendix:  


Decision: The initial ratings for this paper were ba, ba, br. After reading the authors' feedback, all reviewers learned to accept this paper. AC is therefore happy to recommend acceptance and invites the authors to incorporate the materials from the rebuttal to the camera ready.

Reviews: review: paper_summary: paper_summary: The paper present HOIGen, the first generation-based model using CLIP for zero-shot HOI detection. Unlike traditional methods that only use CLIP for feature extraction, HOIGen leverages CLIP for feature generation. By mixing realistic features of seen samples with synthetic features, the model can train on both seen and unseen classes jointly. To enhance HOI scores, the paper constructs a generative prototype bank for pairwise HOI recognition and a multi-knowledge prototype bank for image-wise HOI recognition. Extensive experiments on the HICO-DET benchmark demonstrate that HOIGen achieves superior performance.
strength_weakness: strength_weakness: 1.	The paper first proposes a novel generation-based paradigm in HOI detection.
2.	The experimental results are impressive.
3.	Ablation study verifies the effectiveness of each module.
limitations: limitations: 1. When training the VAE for human categories, why choose the prompt template "person who interacts with <OBJECT>" instead of "person who <VERB> an object"?
2. Is the generator implemented as the CLIP text encoder? If so, how does it generate $ x_{k}'$?
3. Which dataset is used to train stage 2 of "CLIP-injected Feature Generation"?
4. The paper claims that "the features of stage 1 may differ from the image features required in the following detector." How does stage 2 training address this problem?
5. After loading the generated features into the prototype bank, how is supervision provided for the unseen classes during training?
6. The experiment is only conducted on a single dataset. What is the model's performance on V-COCO?
paper_summary: paper_summary: This paper propose a CLIP-injected feature generator to train models on seen and unseen features which improves models’ generalizability. Also, the paper develops two banks, generative prototype bank and multi-knowledge prototype bank. The former solves pairwise HOI recognition and the later focus on image-wise HOI recognition.
strength_weakness: strength_weakness: 1 Clear writing and good representation:
Both figures and tables are well designed and the captions are easy to read and help to understand the main ideas.

2 Good performance:
In Table 1, this paper achieves significant improvement on the unseen of UC and UO settings which show the good generalizability of HOIGen. Additionally, on RF-UC and NF-UC, HOIGen also show the good generalizable performance with 2.54% and 1.57% improvements compared with previous methods.

3 Novel idea:
The paper proposes two banks--generative prototype bank and multi-knowledge prototype bank, to solve pairwise HOI recognition and image-wise HOI recognition, which is interesting to me.
limitations: limitations: 1, The discussion is missing about the relatively low performance on UV unseen. Can authors show more detail discussion about the performance degradation in UV settings, such as the specific examples?
paper_summary: paper_summary: This paper presents a generation-based model using CLIP for zero-shot HOI detection It uses CLIP for feature generation instead of feature extraction only. To achieve it, a CLIP-injected feature generator is proposed in accordance with the generation of human, object and union features. Then,   realistic features of seen samples  are extracted and mixed with synthetic features together, allowing the model to train
seen and unseen classes jointly. To enrich the HOI scores, a generative prototype bank is constructed in a pairwise HOI recognition branch, and a multi-knowledge prototype bank in an image-wise HOI recognition branch, respectively. Experiments show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The paper is overall well-written and easy to follow.

2. The experiments are comprehensive and demonstrated the effectiveness of the proposed method.
limitations: limitations: 1. The reviewer concerns about the scope of the paper. As the paper is submitting to ACM MM, but there is barely literature connection to previously ACM MM papers. To better align with the scope of ACM MM, the reviewer recommends adding more MM literatures for comparison and discussion to align with the scope of ACM MM.

2. In Fig.2, the authors claim that by taking both adaption and generalization into account, the performance is increased compared with CLIP-based method. However, the reviewer notice that the proposed method also adopts a few off-the-shelf models which incorporate external training information into the task. Thus, the authors need to clarify how the adaption and generalization is enhanced, rather than simply relying on external knowledge leaked from other large models.

3. The reviewer noticed that for unseen verb, the unseen performance of the proposed method underperforms all the baseline methods. Particularly, the unseen performance is around 6% lower than the best method CLIP4HOI. Why is this phenomenon? Is the method good at detecting nouns but bad at verbs?


Typos: Line342 "The idea of extracting CLIP feature embeddings for zero-shot HOI
detection has been studies in previous work" -> "studied."