Towards Open-vocabulary HOI Detection with Calibrated Vision-language Models and Locality-aware Queries

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 07:08:16

Abstract: The open-vocabulary human-object interaction (Ov-HOI) detection aims to identify both base and novel categories of humanobject interactions while only base categories are available during training. Existing Ov-HOI methods commonly leverage knowledge distilled from CLIP to extend their ability to detect previously unseen interaction categories. However, our empirical observations indicate that the inherent noise present in CLIP has a detrimental effect on HOI prediction. Moreover, the absence of novel humanobject position distributions often leads to overfitting on the base categories within their learned queries. To address these issues, we propose a two-step framework named, CaM-LQ, Calibrating visual-language Models, (e.g., CLIP) for open-vocabulary HOI detection with Locality-aware Queries. By injecting fine-grained HOI supervision from the calibrated CLIP into the HOI decoder, our model can achieve the goal of predicting novel interactions. Extensive experimental results demonstrate that our approach performs well in open-vocabulary human-object interaction detection, surpassing state-of-the-art methods across multiple metrics on mainstream datasets and showing superior open-vocabulary HOI detection performance, e.g., with 4.54 points improvement on the HICO-DET dataset over the SoTA CLIP4HOI on the UV task with the same backbone ResNet-50.

Main paper: 
 introduction: 
 Human-object Interaction (HOI) detection, a fundamental task of human-centered scene understanding, has gained considerable attention across diverse domains, such as Image Captioning [27] , Visual Question Answering (VQA) [33] , and video analysis [5] . Notably, prevailing methods [28, 40] have demonstrated stunning 570 8& 89 82)XOOP$3 &/,3 5HJLRQ&/,3 &D&/,3RXUVFigure 1 : Noise of V&L mode comparison. We investigate three different V&L models on RTM, UC, UV, UO tasks, which denotes the region text matching and HOI detection under settings where compositions, verbs, and objects are unseen, respectively. CaCLIP denotes our calibrated CLIP.performance within the confines of closed-set scenarios. Nevertheless, the inherent complexity of real-world interactions between humans and objects gives rise to a multitude of interaction categories that are not comprehensively encapsulated by existing HOI datasets, such as V-COCO [8] and HICO-DET [2] . Hence, there is a compelling necessity to shift attention toward the exploration of Open-vocabulary Human-object Interaction (Ov-HOI) detection.With the widespread adoption of large-scale pretrained visionlanguage models (VLMs), such as CLIP [29] , prior works [20, 32, 34] have showcased the impressive generalization capabilities of VLMs in open-vocabulary scenarios [7, 42] . However, these models encounter three primary limitations: (1) struggling in fine-grained HOI detection, especially for local regions, due to their adoption of a global image-level knowledge distillation strategy [20, 32] to guide their image encoders towards learning CLIP-like embeddings; (2) prone to overfitting on the spatial features of base categories, making them incapable of handling scenarios with substantial spatial distribution disparities between seen and unseen HOI categories;(3) misalignment of HOI visual and language knowledge, attributed to intrinsic noise introduced by pretrained multimodal models like CLIP. To empirically assess these limitations, Figure 1 illustrates the performance evaluation of three multimodal features: CLIP, Re-gionCLIP [42] , and our calibrated CLIP, in HOI classification tasks under various scenarios. The former two features exhibit obviously inferior performance compared to our calibrated CLIP. For more analysis, please refer to Sec. 5.4.To overcome the aforementioned issues, we propose a novel model, denoted as CaM-LQ, aimed at calibrating visual language models with fine-grained HOI priors and then training an openvocabulary HOI detector with locality-aware queries. More concretely, we develop a two-step training mechanism: (1) calibrating CLIP with HOI priors, and ( 2) performing open-vocabulary HOI detection. In the first step, we suppress the intrinsic noise in visual language models, such as CLIP, by calibrating CLIP features with HOI knowledge through training two parallel adapters [6] . This calibration ensures that the learned visual-semantic space is equipped with corrective HOI concepts, thereby enhancing its compatibility for HOI detection.Moving on to the second step, we initially employ a pre-trained object detector, e.g., DETR [1] , to identify objects and construct pairwise queries for feasible human-object pairs. Specifically, we utilize spatial embeddings of human-object pairs to refine human and object visual features and employ human-object queries to decode the global image feature with spatial priors. Subsequently, for each detected pair, we encode union box embeddings and compute their similarities with pre-defined text embeddings using our calibrated pre-trained model at the first stage. These similarities serve as finegrained supervision to guide the training of the open-vocabulary HOI detection network through a logit distillation mechanism.In summary, our contributions can be outlined as follows:• Through comprehensive empirical experiments of V&L models, we observe the presence of significant intrinsic noise in their embeddings, which proves to be detrimental to open-vocabulary HOI detection. • We devise an approach to calibrate CLIP with HOI priors, coupled with a fine-grained logit distillation strategy, to alleviate the impact of intrinsic noise in the embeddings. • We propose to inject spatial priors into HOI queries to decode pairwise HOI features, which will help the mode focus on the interaction point and capture a nuanced relationship. • Our model surpasses state-of-the-art approaches with a large margin on many metrics of Ov-HOI detection, e.g., surpassing the SoTA CLIP4HOI with 4.54 mAP on the HICO-DET dataset on the UV task with the backbone ResNet-50. 
related work: 
 In this section, we will carefully review literature from the aspects of HOI detection, open-vocabulary HOI detection and HOI detection with CLIP.The current mainstream HOI detection methods can be categorized into two paradigms: one-stage and two-stage. One-stage approaches [14, 17, 30] treat HOI detection as a set prediction problem, adopting parallel prediction methods to locate objects and predict interaction categories. This requires a decoder to extract effective interaction information from the learned query. However, these methods are often constrained by excessive noise in the learned queries, leading to an inability to decode the correct interaction categories. Additionally, as the query learns from the training set, it may result in over-fitting to the position distribution of human and base object [25] . On the other hand, two-stage methods [18, 28] often utilize a pre-trained detector to locate objects and predict categories. Therefore, they can leverage off-the-shelf features to construct the query, imbuing it with sufficient interaction information. Our approach adopts a two-stage step, based on a pre-trained detector [1] , to enhance interaction prediction capabilities.Open-vocabulary HOI detection [10, 38] aims to predict base and novel HOI categories with only base HOI labels during training. Due to the complexity of the composition of human and object interaction, it is almost impossible to construct all HOI categories. Therefore, open-vocabulary HOI detection can be more effectively applied in real-world scenarios. The task currently has three main scenarios: Unseen Composition(UC), Unseen Object(UO), Unseen Verb(UV). Recently, ConsNet [23] employed explicit and implicit knowledge of HOI from consistency graphs and word embeddings. However, these methods are constrained by the limitations of HOI datasets and cannot fully explore the capability of open-vocabulary HOI detection. In light of this, recent methods [20, 25] extract knowledge from vision-language pre-trained model [29] to improve open-vocabulary performance and have achieved promising scores.Recently, CLIP has successfully implemented contrastive learning with large-scale image-text pair data gathered from the internet and achieved powerful zero-shot performance. The approaches to leverage CLIP knowledge in HOI tasks can be divided into two groups. The first group utilizes CLIP to extract global features. For example, GEN-VLKT [20] encodes the entire image with CLIP and applies supervised learning on the global image features to the image encoder in the network. HOICLIP [26] , integrates CLIP global embeddings with backbone features, followed by a decoder to predict interaction categories. CLIP4HOI [25] employs the global feature from CLIP to introduce prior knowledge to aid the final prediction of interaction categories. The other group, however, involves utilizing CLIP to extract region features. For instance, EoID [34] proposed to inject more regional supervision information into the predictions by union box features. Similarly, we borrow the idea of EoID to inject fine-grained supervision into the decoder. However, none of the aforementioned methods consider the noise brought by CLIP which is trained on large-scale image-caption datasets, leading the bias towards the object category, rather than HOI category. It is necessary to calibrate CLIP features with HOI priors and eliminate noise. In light of this, we propose a pre-training step to calibrate CLIP with HOI priors, enabling a visual semantic space with rich HOI priors. 
problem statement: 
 In this section, we introduce the setting of HOI detection in both fully supervised and open-vocabulary settings. Denote O = {𝑜 1 , 𝑜 2 , ..., 𝑜 𝑁 𝑜 } be the object category, V = {𝑣 1 , 𝑣 2 , ..., 𝑣 𝑁 𝑣 } be the interaction verbs and C denote all the feasible composition of verb-object pairs, i.e., C = {(𝑣 𝑖 , 𝑜 𝑗 )|𝑣 𝑖 ∈ V; 𝑜 𝑗 ∈ O}. Let I denote an input image, with corresponding labels 𝑇 = {B, Y} where B is a set of bounding boxes including human bounding boxes B ℎ and object bounding boxes B 𝑜 , and Y denote a set of HOI triplets. Each ⟨𝑏 ℎ , 𝑏 𝑜 , 𝑐 𝑖 ⟩ in Y is a HOI triplet, where 𝑐 𝑖 ∈ C.  
method: 
 In this section, we introduce our proposed CaM-LQ, Calibrating visual-language Models, and training an open-vocabulary HOI detector with Locality-aware Queries for open-vocabulary HOI detection. As illustrated in Figure 2 , our CaM-LQ employs a twostep training framework, encompassing four primary components: the teacher model CaCLIP, a pre-trained object detector, an HOI encoder, and a pairwise interaction decoder. Firstly, we conduct fine-tuning of the CLIP model, incorporating HOI priors to imbue HOI-specific knowledge and yielding a specialized V&L teacher model denoted as CaCLIP. The HOI transformer-based encoder is employed, utilizing query embeddings and box coordinates as inputs, integrating visual and spatial features to construct HOI queries. Subsequently, the interaction decoder integrates these queries with box spatial priors, and backbone features serving as key-value pairs, to predict the action in each HOI triplet. Ultimately, we compute cosine similarities between visual features and HOI text embeddings.These similarity scores are utilized as soft labels to guide the action prediction, facilitating knowledge transfer to the HOI decoder.Our empirical observations (seeing Sec.5.4 for more detail.) indicate that the raw CLIP feature can not handle HOI prediction well, primarily due to intrinsic noise stemming from the training on extensive web image-caption pairs. To address this limitation, we propose a calibration strategy to imbue CLIP with HOI-specific knowledge and align HOI visual and text information at a finer granularity, specifically at the region level.However, due to the limited data availability, direct fine-tuning of all parameters of CLIP proves challenging. Inspired by [6] , we adopt a more lightweight fine-tuning strategy by introducing two adapters, for the image and text encoders. In default, our adapters are implemented by MLPs and are positioned subsequent to the original CLIP image and text encoders. In practice, we also test other alternative adapter architectures, and for more details, please refer to the supplementary materials.We employ human-annotated HOI labels to calculate the union box coordinates for each human-object pair, i.e., the minimum bounding rectangle. These union boxes are utilized to crop the image as:EQUATIONEQUATIONWhere 𝑣 ′ 𝑖 denotes the embedding of the union region, Φ 𝑢𝑛𝑖𝑜𝑛 represents the operation computing the union box for the human-object pair, Φ 𝑐𝑟𝑜𝑝 signifies the operation of cropping the bounding box region from the image, CLIP 𝑖𝑚𝑔 is the function responsible for encoding the image using the CLIP visual encoder and Adapter 𝑖𝑚𝑔 denotes the image adapter.As for the text branch, we utilize CLIP to encode a predefined set of HOI texts, by a template, e.g., "a photo of a person verb object, " where "verb" and "object" represent labels in an HOI triplet. Subsequently, we establish an HOI knowledge bank denoted as 𝑾 = [𝑤 1 , 𝑤 2 , ..., 𝑤 |𝐶 | ], which serves as a classifier. By feeding text features into the HOI adapter, we derive a refined HOI knowledge bank 𝑾 ′ . The cosine similarity between the obtained human-object region feature and predefined text features is then computed:EQUATIONWhere𝑠𝑖𝑚(𝑣 ′ 𝑖 , 𝑤 ′ 𝑗 ) = (𝑣 ′ 𝑖 •𝑤 ′ 𝑖 )/(||𝑣 ′ 𝑖 || • ||𝑤 ′ 𝑖 ||).Finally, a binary crossentropy loss is employed for multi-label classification, aligning pairwise region features with corresponding HOI texts. Following this calibration process, we acquire a calibrated CLIP endowed with HOI priors, dubbed CaCLIP, which attains proficiency in a specialized HOI visual-semantic space, rendering it amenable for subsequent applications in open-vocabulary HOI detection.In this section, we present how we encode visual features and inject spatial priors into the HOI queries. Following [39] , we adopt a two-stage paradigm for HOI detection. Specifically, we first input an image into DETR [1] and obtain the global visual features and a set of region proposals. As [39] , we post-process the region proposals with non-maximum suppression (NMS) and filter the DETR queries with confidence scores lower than a threshold. The postprocessed results are denoted as ( B, Ŝ, Ô, 𝐻 ), where B ∈ 𝑅 𝑁 𝑝𝑟𝑒𝑑 ×4 , Ŝ ∈ [0, 1] 𝑁 𝑝𝑟𝑒𝑑 , Ô ∈ {0, 1, ..., |O| -1} 𝑁 𝑝𝑟𝑒𝑑 and 𝐻 ∈ 𝑅 𝑁 𝑝𝑟𝑒𝑑 ×𝐶 𝑑 refer to predicted boxes, confidence scores, object categories and query features respectively, where 𝑁 𝑝𝑟𝑒𝑑 denotes the number of queries after filtering.Since the queries obtained from a pre-trained object detector [1] are designed for localization and classification, directly employing them as interaction queries can not obtain ideal results, due to the fact that they typically focus on object-level information rather than the pairwise interactive concepts. Hence, to further adapt them to represent HOI visual embeddings, we employ a selfattention mechanism to refine them. The objective of self-attention mechanism is to enable human and object queries to focus on each other's features and complement each other.Concretely, we compute the sinusoidal embeddings [31] of box center 𝑐 𝑥 , 𝑐 𝑦 , width 𝑐 𝑤 and height 𝑐 ℎ , by Θ :EQUATION)where 𝑑 is half of the 𝐶 𝑑 , 𝑖 = 1, 2, ..., 𝑑/2 and 𝜏 is a temperature parameter.With the coordinates embeddings, we concatenate them to form the spatial embeddings 𝐸 𝑠𝑝 ∈ 𝑅 𝑁 𝑝𝑟𝑒𝑑 ×2𝐶 𝑑 and the object features are refined through self-attention layers as follows:EQUATIONwhere 𝐻 ′ ∈ 𝑅 𝑁 𝑝𝑟𝑒𝑑 ×𝐶 𝑑 , SelfAttn and Proj denote box spatial embeddings, refined object features, self-attention layers and projection layers respectively. Subsequently, we keep all viable pairs, i.e., excluding those where the subject category is not human. Thus, human-object queries 𝑄 ℎ𝑜 ∈ R 𝑁 pair ×𝐶 𝑟 for feasible pairs are constructed through concatenation and a two-layers MLP to fuse human and object features by the following operations:EQUATIONwhere 𝐻 ′ human and 𝐻 ′ object represent the human and object features of feasible pairs in the spatially refined features 𝐻 ′ . So far, 𝑄 ℎ𝑜 ∈ R 𝑁 pair ×𝐶 𝑟 is enriched with detailed spatial information, which can facilitate the HOI decoder to pay more attention to the objects close to a human.In this section, we seek to use refined interaction queries to predict HOIs from visual features via a locality-aware interaction decoder. Following [40] , we apply the backbone ResNet C5 [9] features in default and further refine it with lightweight window attention layers [24] , where the refined features are denoted as 𝐹 ′ ∈ R ℎ×𝑤×𝐶 𝑑 .One-stage HOI approaches [20, 26] widely incorporate learnable position embeddings to ensure that the initialized queries capture interaction pairs at different locations. However, for a two-stage counterpart, since objects have already been localized and form human-object pairs, adding learnable vectors to capture position information becomes less meaningful. Inspired by this, we argue that the localization information of humans and objects helps the decoder in learning the interaction between human-object pairs. Hence, we concatenate the sinusoidal spatial information of humans and objects in feasible pairs to obtain pairwise spatial embeddings 𝐸 ′ 𝑠𝑝 ∈ R 𝑁 𝑝𝑎𝑖𝑟 ×4𝐶 𝑑 . Each pairwise spatial embedding 𝑒 ′ 𝑠𝑝 is calculated as follows:EQUATIONwhere 𝑒 human , 𝑒 object ∈ 𝐸 𝑠𝑝 denote spatial embedding of human and object in each feasible pair. Subsequently, we develop a spatial prior enhancement module in each decoder layer by the sinusoidal embeddings (Eq.4) again. Concretely, we inject the box spatial priors 𝐸 ′ 𝑠𝑝 from Eq.( 7) and sinusoidal positional embeddings 𝐸 ′ 𝑝𝑠 into each decoder layer to augment the query 𝑄 ℎ𝑜 and image features F ′ respectively. Taking the query in the 𝑙-th layer as an example, the augmentation operation is calculated as follows:EQUATIONwhere LN, MHSA, MHCA, CAT, FFN denotes layer normalization, multi-head self-attention, multi-head cross attention, concatenation and feed-forward network, respectively. Projection layers and dropout operations are omitted for simplicity. It is worth noting that the first layer 𝑙 = 0, 𝑄 0 is set to 𝑄 ℎ𝑜 from Eq.( 6). Finally, the output hidden state embedding of the last layer, i.e., 𝑄 𝑁 should incorporates interactive feature information of human-object pairs and is employed to predict interactions via a binary classifier, yielding𝑆 𝑣 ∈ R 𝑁 𝑝𝑎𝑖𝑟 × | V | .Owing to the unavailability of novel categories during training, we turn to CaCLIP, serving as a teacher model, to exploit its openvocabulary HOI knowledge. With the huge discrepancy of the features between the teacher and the HOI decoder, we adopt the logits from CaCLIP as the guidance knowledge instead of its V&L features, to avoid the noise feature in the V&L teacher. Note that though EoID [34] adopts the similar strategy to inject CLIP signals as supervisory information, the significant noise presented in CLIP severely impedes the accurate alignment between image regions and their corresponding HOI texts. This results in the supervision scores providing less satisfactory guidance than ours provided by CaCLIP.In implementation, we use the union box of each feasible pair to crop the input image and compute visual embedding through CaCLIP. Afterward, the cosine similarities with the predefined HOI texts, i.e., (verb, object), are normalized by a softmax operation. Subsequently, those similarity scores are mapped to action logits𝑆 𝑐𝑙𝑖𝑝 ∈ R 𝑁 𝑝𝑎𝑖𝑟 × | V |where invalid actions are masked out with the priors provided from DETR. Then, we treat the logits 𝑆 𝑐𝑙𝑖𝑝 from CaCLIP as a soft label to guide the interaction prediction. Note that 𝑆 𝑐𝑙𝑖𝑝 only influences categories not covered by the ground truth. We conduct a mask operation to choose the CLIP logit for computing losses:SoftMask (𝑖,𝑗 ) =1 if𝑆 𝑐𝑙𝑖𝑝 (𝑖,𝑗 ) ≠ 0 and 𝑆 𝑔𝑡 (𝑖,𝑗 ) ≠ 1 0 otherwise (9) where 𝑆 𝑔𝑡 ∈ [0, 1] 𝑁 𝑝𝑎𝑖𝑟 × | V | denote ground truth labels of action categories. In practice, we use SoftMask (𝑖,𝑗 ) to multiply losses calculated by the soft CLIP label.Training: Given action logits, the total loss comes from two parts: the hard loss with ground truth labels and the soft loss with CLIP supervision labels. We apply focal loss [21] to mitigate the impact of imbalanced data distribution and hyper-parameter 𝜆 to balance the two terms:EQUATIONInference: We combine the confidence scores (𝑆 ℎ , 𝑆 𝑜 ) from DETR as prior knowledge to obtain the final prediction scores following [40] : 11) where 𝑆 𝑝 ∈ 𝑁 𝑝𝑎𝑖𝑟 × | V | and 𝜑 ∈ [0, 1] is a hyperparameter and 𝑆 𝑣 is the predicted action score by the HOI decoder.EQUATION 
experiments: 
 In this section, we will carefully elaborate our comprehensive experimental evaluation on three benchmark datasets. More results can be found in the supplementary materials.Datasets: we conduct downstream experiments on two popular benchmarks: HICO-DET [2] and V-COCO [8] . Open-vocabulary Setups: As described in Section 3, we conducted experiments based on the UO, UV, and UC settings. Additionally, the UC setting is extended to RF-UC, where tail HOI categories are selected as the novel, and NF-UC setting, which contains head HOI categories as the novel. For HICO-DET, the division between base and novel classes for all the Ov-HOI tasks follows the EoID [34] protocol. As for V-COCO, we are the first to report the UV results, please refer to the appendix for more details.We use the fine-tuned DETR with the backbone ResNet-50 [9] as our small model CaM-LQ 𝑠 in default and a large counterpart CaM-LQ 𝑙 with more powerful object detector DETR [15] of Swin-L [24] backbone. Following [39] , we adopt the same filtering scheme only keeping the detections with a confidence score higher than 0.2. Besides, the distillation is built upon the CLIP, which takes ViT-B/16 [4] as its backbone. The CLIP adapters are instantialized as a twolayer MLP for both image and text branch. The hyper-parameter 𝜆 and 𝜑 are set to 400 and 0.26 respectively. For the selection of 𝜏, we follow [22] and set it to 20 to make the sinusoidal encoding in the Transformer more suitable for vision tasks. We use 𝛼 = 0.5 and 𝛾 = 0.1 in our focal loss, the same as previous work [40] . The object hidden features from DETR have a dimension of 𝐶 𝑑 of 256, and the query features of the decoder are set to 𝐶 𝑟 of 384. For more details, please refer to the appendix.5.3.1 Effectiveness for Open-vocabulary HOI detection. In Table 2 , we report the results on HICO-DET under five different settings as [25] : UC (Unseen Composition), RF-UC (Rare First Unseen Combination), NF-UC (Non-rare First Unseen Combination), UO (Unseen Object), UV (Unseen Verb). On the UC setting, our approach achieves SoTA for small version and even gain an mAP gain of 10.05, 10.84 and 10.89 for unseen, seen, and full results with a larger backbone. As for the RF-UC setting, we demonstrate the robust generalization capability of our approach to achieve a remarkable improvement of 3.82 and 6.99 points on unseen categories with different backbones. When it turns to the NF-UC setting, our CaM-LQ can also outperform all the previous models. Besides, our model also shows considerable improvements on the UO and UV, confirming its strong generalizability to novel objects and actions. We defined a base network, i.e., without the addition of any spatial prior information in the encoder or decoder and V&L Teacher. By introducing spatial priors to the queries, the full performance improves by 1.09 points, validating the effectiveness of spatial priors in shaping the features adapted to HOI detection. We further investigate the role of spatial information in the cross-attention mechanism. It can be observed that the full score increases by 1.74 points. Notably, adding the spatial embeddings in the encoder and decoder simultaneously can contribute more mAP gain of 4.83, demonstrating the efficacy of the spatial information.Finally, we perform ablation on the CaCLIP structure. With the guidance of V&L Teacher, our model exhibits a substantial improvement of 9.62 points in the performance, affirming the significance of our calibrated CLIP for open-vocabulary HOI detection task.Introduced by V&L Models. To investigate the potential noise introduced by V&L models in HOI tasks, we conduct alignment ability experiments on V&L models. RegionCLIP [42] is designed to train a region-level image-text alignment network, with a similarity to our HOI task. Figure 1 illustrates the comparative results of the three models on RTM, UC, UV and UO. RTM denotes region-text matching tasks where we compute the classification precision of region embeddings. It can be observed that RegionCLIP's performance is slightly better than CLIP. However, our CaCLIP significantly outperforms the other two models in HOI classification tasks and open vocabulary tasks, demonstrating the presence of noise in V&L models.Besides, we also compare different CLIP variants and their calibrated version under UC tasks on the HICO-DET, adding Region-CLIP as well, referring to Table 5 . It is noteworthy that we use the weights from pre-trained Region-CLIP to initialize CLIP. Experimental results demonstrate that using ViT-B/16 as the backbone achieves the best performance after fine-tuning. Furthermore, it surpasses Region-CLIP by a large margin of 3.56 points under UC setting. Plus, we extend our analysis to the V-COCO dataset to provide additional evidence of misalignment in large models regarding HOI visual-semantic information. We provide, for the first time, an Ov-HOI metric on V-COCO and conduct experiments under the UV setting. Specifically, we select 10 out of 24 interaction categories as novel classes, unseen during training, while the model is required to predict both base and novel classes during inference.The experimental results are presented in Table 6 . Our CaM-LQ the best performance in both scenarios of 𝐴𝑃 𝑆1 𝑟𝑜𝑙𝑒 and 𝐴𝑃 𝑆2 𝑟𝑜𝑙𝑒 , demonstrating the existence of noise in V&L models and the necessity of calibration and the superiority of our methods in handling Ov-HOI detection.To visually demonstrate the improvement in the performance of the refined V&L models on Ov-HOI tasks, we conduct pre-training for varying numbers of epochs. Subsequently, we apply the obtained CaCLIP model to the UC task on the HICO-DET dataset and UV task on the V-COCO, with the results presented in Figure 3 and Figure 4 . As the refinement process progresses, it is evident that the performance of the HOI detector utilizing the V&L model gradually improves and hit the peak when training with about 20 epochs and 27 epoch for HICO-DET and V-COCO respectively. This affirms that eliminating noise from V&L models facilitates progress in HOI tasks. Finally, we choose the checkpoints of 20 and 27 epochs as our CLIP models for HICO-DET and V-COCO dataset respectively to deal with all Ov-HOI tasks mentioned above.Weight. Although we fine-tune CLIP during the pre-training phase, the quality of soft labels is still not as robust as their corresponding hard labels, due to limitations in data scale and significant noise in CLIP's training data. However, soft labels can introduce novel HOI features, which is crucial for Ov-HOI detection tasks. We believe that different loss weights guide the model's attention to different extents for soft labels L 𝑠𝑜 𝑓 𝑡 and hard labels L ℎ𝑎𝑟𝑑 . We compare the network performance under different loss weight 𝜆 𝑐𝑙𝑖𝑝 for CLIP, as shown in Table 7 . CLIP achieves 30.40 full 𝑟𝑜𝑙𝑒 gain with the process of calibrating CLIP under UV task on V-COCO. mAP when the weight is set to 1 and the score grows continuously as loss weight increases, finally hitting the peak at 38.00 mAP as 𝜆 𝑐𝑙𝑖𝑝 equals to 400. In the case of 700 points, the performance degrades a little bit. Experimental results indicate that under a loss weight of 400, the model can inject the most beneficial openvocabulary information to aid learning while ensuring effective guidance from hard labels.To exploit general knowledge from V&L models, we employ the logits distillation instead of embedding distillation. We conduct preliminary experiments to compare the performance of the two distillation methods in Ov-HOI detection. For logits distillation, we utilize the approach described Sec. 4.4. Regarding embedding distillation, we use the way of GEN-VLKT [20] . Specifically, for an image, we apply CLIP to encode the entire image I and supervise the average of the query features of the decoder's last layer, i.e., 𝑄 𝑁 ∈ 𝑅 𝑁 𝑝𝑎𝑖𝑟 ×𝐶 𝑟 , calculating the L1 loss L 𝑚𝑖𝑚𝑖𝑐 as follows:EQUATIONwhere Figure 5 presents some visualization results of CaM-LQ. We showcase the localization map and average attention maps from the pairwise decoder, demonstrating CaM-LQ's accurate object localization. By incorporating spatial priors, our attention maps better capture the interaction points between human-object pairs. In terms of category predictions, CaM-LQ can forecast novel categories not present in the training set and even predict reasonable interaction predicates not included in the data labels. This highlights the robust Ov-HOI detection capability of our CaM-LQ. 
conclusion: 
 In this work, we propose an open-vocabulary human-object interaction detection network, CaM-LQ. We identify potential noise issues in the V&L model that negatively influence the HOI task. To this end, we propose a two step open-vocabulary HOI detection model. Firstly, we suppress the intrinsic noise in the CLIP by calibrating the visual-language space of CLIP with HOI priors. Secondly, we inject fine-grained supervision as soft labels deriving from our calibrated CLIP and leverage spatial priors to enhance the target detector's query and assist in extracting feature map information for pairwise queries. Our approach achieves SoTA results on the Ov-HOI setting on the V-COCO and HICO-DET datasets across a wide range of evaluation metrics. For future work, we will focus on integrating the knowledge from large language models (LLM) to provide more explicit cues for CLIP. This endeavor aims to further optimize Ov-HOI tasks, contributing to their enhancement.  
 
 Appendix:  


Decision: This paper proposes a two-step framework for open-vocabulary human-object interaction (HOI) detection. Initially, the paper received 3 borderline accept and 1 borderline reject (Reviewer VG54).  Reviewer VG54 used to have concern on the definition of the open-vocabulary HOI task, the performance on a specific dataset, the effectiveness of some component. During rebuttal, the authors addressed these concerns by providing clarification on the task, the required experimental results, as well as the additional ablation study. All the reviewers reached consensus that this paper is acceptable.

Reviews: review: paper_summary: paper_summary: The paper proposes a two-step framework called CaM-LQ for open-vocabulary human-object interaction (HOI) detection. This approach incorporates fine-grained HOI supervision from a calibrated CLIP model into the HOI decoder and injects spatial priors into HOI queries. Extensive experiments and ablation studies demonstrate the effectiveness of this method.
strength_weakness: strength_weakness: 1.	The 2-step framework is intuative and tenichally sound for the HOI deteciton task.
2.	The method achieves impressive performance on various benckmarks.
3.	Ablation study confirm the effectivness of various design choices.
limitations: limitations: 1.	What is the difference between the claimed open-vocabulary setting and previous zero-shot setting in the HOI detection community?
2.	What’s the method’s performance on the open-vocabulary HOI dataset: SWIG-HOI [1]?
3.	The calibration procedure may lead to overfitting to the seen classes. I find that in Table 5 and 6, the improvement brought by CaCLIP is mainly on the seen split. I also notice that when utilizing R50 backbone, CaCLIP shows inferior perforamnce on the unseen split compared with RegionCLIP, which supports my hypothesis. Please provide a detailed analysis of the phenomenon.
4.	I doubt the effectiveness of the designed spatial prior injection. Please provide more ablation study on this design.
5.	What’s the total training time of the proposed method? Please compare it with previous methods.
6.	Can the proposed training paradigm be applied to different VLMs, such BLIP or BLIP2?
7.	Writing issue: the equation (8) should be center-aligned.


[1] Wang, Suchen, et al. "Discovering human interactions with large-vocabulary objects via query and multi-scale detection." ICCV 2021
paper_summary: paper_summary: The authors argue that existing methods leveraging knowledge distilled from CLIP (Contrastive Language-Image Pre-training) struggle with fine-grained HOI detection and are prone to overfitting on spatial features of base categories. Moreover, they highlight the issue of misalignment between HOI visual and language knowledge due to noise in pretrained models like CLIP. To address these challenges, the CaM-LQ framework first calibrates CLIP using HOI priors by training two parallel adapters, which helps suppress the intrinsic noise and aligns the visual and text information at a finer granularity. The second step involves using a pre-trained object detector to identify objects and construct pairwise queries for potential human-object pairs. These queries are then refined using spatial embeddings and fed into a locality-aware decoder to predict actions in each HOI triplet.
strength_weakness: strength_weakness: 1. The CaM-LQ framework introduces a novel two-step approach to open-vocabulary HOI detection, which is an innovative solution to a challenging problem in computer vision. The idea of calibrating vision-language models like CLIP with HOI priors and using locality-aware queries for fine-grained interaction detection is a fresh perspective in the field.
2. The use of adapters to fine-tune CLIP and the logit distillation mechanism are correct applications of contemporary machine learning techniques.
3. The paper provides a comprehensive evaluation of the CaM-LQ model on mainstream datasets like HICO-DET and V-COCO. The comparison with state-of-the-art methods across multiple metrics demonstrates the effectiveness of the proposed approach.
limitations: limitations: 1. While the paper compares the proposed CaM-LQ model with CLIP and its variants, it may benefit from a broader comparison with other SOTA VLMs that have been released around the same time or after. This would provide a more comprehensive understanding of where CaM-LQ stands in the broader context of VLM advancements.
2. The paper reports strong performance on the HICO-DET and V-COCO datasets. However, it is not clear how well the model generalizes to other datasets or real-world scenarios. There might be a risk of overfitting to the specific characteristics of the datasets used for evaluation.
3. While the paper demonstrates the model's effectiveness, there is limited discussion on the interpretability of the model's predictions. Understanding how the model makes its predictions, especially in an open-vocabulary setting, is crucial for trust and adoption in practical applications.
4. The paper relies on automated metrics for evaluation. Including human subject evaluation, such as user studies, could provide additional insights into the model's performance from a practical standpoint.
paper_summary: paper_summary: The paper presents a novel framework named CaM-LQ (Calibrating visual-language Models with Locality-aware Queries) for open-vocabulary human-object interaction (HOI) detection. It aims to detect both known (base) and unknown (novel) categories of human-object interactions using only base categories for training. Extensive experiments demonstrate that CaM-LQ achieves state-of-the-art performance on standard datasets like HICO-DET and V-COCO across various open-vocabulary settings. In summary, the paper proposes a calibrated vision-language model with locality-aware queries to address the challenge of open-vocabulary HOI detection, demonstrating significant improvements over prior arts. The core innovation lies in refining the visual-semantic space of CLIP for HOI task and leveraging spatial priors to model interactions.
strength_weakness: strength_weakness: The paper detailing the CaM-LQ framework for open-vocabulary human-object interaction detection exhibits several strengths:
1. The authors demonstrate through extensive experiments that CaM-LQ outperforms state-of-the-art methods on multiple metrics across mainstream datasets like HICO-DET and V-COCO. This shows the empirical strength and effectiveness of the proposed model.
2. The authors conduct detailed ablation studies to understand the contribution of different components of the model, which helps in validating the design choices and the effectiveness of the proposed framework.
3. The paper presents a technically sound approach with a clear explanation of the methodology, including the calibration of CLIP, the use of spatial embeddings, and the locality-aware interaction decoder.
limitations: limitations: The method proposed in this paper is commendable. However, it would be beneficial to present a more comprehensive analysis by including additional cases where the model's performance was suboptimal. Extending the discussion to cover a wider array of scenarios, particularly those that highlight the limitations or areas for improvement of the proposed method, would greatly enhance the academic rigor and practical applicability of the research. This could involve detailing specific instances where the model encountered difficulties in accurately predicting human-object interactions, or where the calibration of the vision-language model did not yield the expected results. By doing so, the authors can provide a more nuanced understanding of the model's robustness, its susceptibility to certain types of errors, and offer insights into potential avenues for future research and development.
paper_summary: paper_summary: The paper presents a novel two-step framework named CaM-LQ for open-vocabulary human-object interaction (HOI) detection. The framework aims to identify base and novel categories of human-object interactions during training, addressing the limitations of existing methods that struggle with fine-grained detection and are prone to overfitting. CaM-LQ calibrates visual-language models like CLIP with HOI priors and employs locality-aware queries to predict interactions, demonstrating superior performance over state-of-the-art methods on mainstream datasets. The approach involves fine-tuning CLIP to align it with HOI knowledge, using the calibrated CLIP to guide an open-vocabulary HOI detection network through a logit distillation mechanism.
strength_weakness: strength_weakness: + The code is released anonymously.

+ The proposed model surpasses state-of-the-art approaches.

+ The experiments are sufficient.

+ The method of calibrating the CLIP using the HOI priors seems to be novel.
limitations: limitations: - The latex format is not up to date.

- The paper needs to be proofread. For example, "mode" --> "model" in L147, "denote" --> "denotes" in L230, formulas are not standardized (Eq. 9). 

- The grammar used in the paper is confusing. "defiend" in L720, and "improves" in L723, "investigate" in L725.