EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection

Authors: qinqian  lei
bo  wang
robby t tan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-29 09:27:00

Abstract: Detecting Human-Object Interactions (HOI) in zero-shot settings, where models must handle unseen classes, poses significant challenges. Existing methods that rely on aligning visual encoders with large Vision-Language Models (VLMs) to tap into the extensive knowledge of VLMs, require large, computationally expensive models and encounter training difficulties. Adapting VLMs with prompt learning offers an alternative to direct alignment. However, fine-tuning on task-specific datasets often leads to overfitting to seen classes and suboptimal performance on unseen classes, due to the absence of unseen class labels. To address these challenges, we introduce a novel prompt learning-based framework for Efficient Zero-Shot HOI detection (EZ-HOI). First, we introduce Large Language Model (LLM) and VLM guidance for learnable prompts, integrating detailed HOI descriptions and visual semantics to adapt VLMs to HOI tasks. However, because training datasets contain seen-class labels alone, fine-tuning VLMs on such datasets tends to optimize learnable prompts for seen classes instead of unseen ones. Therefore, we design prompt learning for unseen classes using information from related seen classes, with LLMs utilized to highlight the differences between unseen and related seen classes. Quantitative evaluations on benchmark datasets demonstrate that our EZ-HOI achieves state-of-the-art performance across various zero-shot settings with only 10.35\% to 33.95\% of the trainable parameters compared to existing methods. Code is available at https://github.com/ChelsieLei/EZ-HOI.

Main paper: 
 introduction: 
 Human-Object Interaction (HOI) detection localizes human-object pairs and identifies the interactions. HOI has various practical applications, including robot manipulations, human-computer interaction, and human activity understanding [38, 41, 24, 32, 26, 1] . Additionally, it is a building block for related tasks such as action recognition, visual question answering, and image generation [18, 24, 4, 43, 11, 54] . However, the limited generalization ability of many existing HOI detectors makes zero-shot HOI detection particularly challenging, as it requires models to identify unseen HOI classes.Vision-Language Models (VLMs) [47, 29, 2, 30] have gained popularity due to their extensive knowledge bases and effective ability to process and correlate complex patterns across visual and text data. Consequently, a group of methods has been developed to align HOI visual features with those of VLMs, leveraging the extensive knowledge from these models [51, 33, 44, 41, 4] . This alignment ensures that both the HOI model and the VLM can extract similar features to represent the same concept (e.g., an action). The degree of feature alignment can be quantified using cosine similarity, which measures the similarity between feature vectors. However, aligning with VLMs requires 38th Conference on Neural Information Processing Systems (NeurIPS 2024). [44, 33, 4, 41] . (b) Prompt learning methods to adapt VLMs for downstream tasks [22, 57] . (c) Our approach, which adapts VLMs to HOI tasks without compromising VLM generation capabilities. (d) Unseen, seen, and full mAP indicate the performance for unseen-verb, seen-verb, and full sets on the HICO-DET dataset [6] . Our EZ-HOI shows superior performance in these categories, with competitive trainable parameters and training epochs.training transformer-based models, which is computationally expensive and leads to extended training time and significant difficulties.An alternative approach involves adapting VLMs for HOI tasks, bypassing the demanding alignment process and leveraging VLM capabilities for action understanding in HOI detection. Prompt tuning, which adapts VLMs to various downstream tasks with a small number of learnable parameters, has shown considerable efficacy [62, 61, 19, 21] . One notable method, MaPLe [22] , utilizes multi-modal prompt tuning specifically for image classification tasks. However, since adaptation typically involves only seen class labels, the adapted VLMs often overfit to seen classes but struggle to deal with unseen classes in zero-shot settings [62, 7, 9, 22] . Consequently, this limitation results in suboptimal performance for unseen classes in zero-shot HOI detection.In this paper, we introduce a novel method, Efficient Zero-Shot HOI detection (EZ-HOI), to enhance VLM adaptation via guided prompt learning with information from foundation models (e.g., LLM and VLM). We design both learnable text and visual prompts, leveraging fixed LLM and VLM to guide the adaptation process. Specifically, the text prompts are designed to capture detailed HOI class insights from an LLM, while the visual prompts are tailored to incorporate external visual semantics from a VLM. However, as the training datasets contain images with seen-class labels alone, the trainable prompts are naturally optimized for these seen classes rather than the unseen ones.To address this limitation, we develop the Unseen Text Prompt Learning (UTPL) to leverage prompts from related seen classes effectively. We begin by measuring the relationship between HOIs using cosine similarity of text embeddings, which helps identify closely related seen classes for each unseen one. After establishing these connections, we enhance the learning of unseen prompts based on those from selected seen classes. To capture the distinctions between unseen and seen classes, we incorporate an LLM. This LLM provides what we call "disparity information", enhancing the learning of the distinctions and similarities between seen and unseen classes. Additionally, we enhance our approach by introducing intra-and inter-HOI feature fusion techniques following the visual encoder. Our method achieves competitive performance on established benchmarks in various zero-shot settings while requiring significantly fewer trainable parameters.As illustrated in Fig. 1 , we compare our method with two existing paradigms: (a) aligning HOI features to a fixed VLM and (b) adapting a VLM for HOI tasks. Unlike the existing approaches, our method facilitates VLM adaptation to HOI tasks and demonstrates effective generalization to unseen classes. Fig. 1 (d) highlights that our method is competitive in terms of performance, model parameter efficiency, and training duration.In summary, our contributions are as follows:• We introduce EZ-HOI, a novel framework for zero-shot HOI detection that adapts a VLM to HOI tasks via guided prompt learning with foundation model information, enhancing the adapted VLM's generalizability in zero-shot HOI detection. • We propose the UTPL module, which extracts information from related seen classes for the unseen learnable prompts. This module mitigates overfitting to seen classes, a common issue in task-specific VLM adaptations, improving performance on unseen classes. • Our EZ-HOI method achieves state-of-the-art performance in various zero-shot settings while significantly reducing trainable parameters and mitigating training challenges. It lowers trainable model parameters by 66% to 78% compared to existing methods, demonstrating enhanced efficiency and effectiveness in zero-shot HOI detection. 
related work: 
 Human-Object Interaction Detection Human-Object Interaction (HOI) detection involves identifying human-object pairs and recognizing their interactions, serving as a fundamental component for various downstream tasks in computer vision [53, 6] . HOI detection methods are typically categorized into two classes: one-stage and two-stage. One-stage methods simultaneously generate all outputs, including human bounding boxes, object bounding boxes and categories, and interaction classes. Recent advancements in one-stage detectors leverage transformer architectures, delivering promising performance [8, 46, 48, 63, 55, 52, 24] . An example is HOITrans [63] , which utilizes the transformer's encoder and decoder to extract interaction features. The output features are then processed through multi-layer perceptrons to produce all output predictions at once. On the other hand, two-stage approaches divide HOI detection into two tasks: object detection and HOI classification [10, 14, 58] . Since separation allows each module to specialize, two-stage HOI detection is more memory-efficient [59] . Recent developments have seen the integration of transformer-based architectures into two-stage designs, which have shown promising results [59, 60, 45, 27] . Our method also falls into the two-stage design category.Zero-Shot HOI Detection Prior zero-shot HOI detection efforts primarily address unseen composition settings, where models encounter action and object classes separately but not as combinations.To address this, several methods employ compositional learning strategies aimed at tackling the unseen-composition problem [14, 16, 15, 27] . However, compositional learning falls short in addressing unseen verb zero-shot HOI detection scenarios. Given the substantial image understanding capabilities of VLMs, they offer promising potential for enhancing HOI detection in zero-shot settings [2, 29, 30, 47, 37, 36] , where labels for unseen HOI classes are absent from the training dataset.Recent studies have explored aligning HOI visual features with VLM [33, 44, 4, 41] . However, this approach requires training transformer-based models, which are computationally expensive. Consequently, aligning HOI models with VLMs is demanding, resulting in extended training times and significant training challenges.Prompt Learning Prompt learning has gained popularity for adapting VLMs to downstream tasks [61, 62, 7, 9, 19, 23, 25, 40] . Context Optimization (CoOp) [62] refines the prompt input of the text encoder by combining learnable domain-shared prompt tokens with class prompt tokens. MaPLe [22] integrates learnable domain-shared text tokens with visual learnable tokens. This combination leverages the highly connected text and visual encoders in VLMs, facilitating the sharing of crossdomain information and benefiting both text and visual domains. However, fine-tuning VLM relies on training datasets with only seen class labels, which causes adapted VLMs to excel with seen classes but encounter difficulties with unseen ones in zero-shot scenarios. Consequently, this limitation results in suboptimal performance for unseen classes in zero-shot HOI detection.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We included the Limitation section in the main paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 We start with adapting a pre-trained CLIP model [47] to zero-shot HOI tasks using an innovative prompt learning approach. We design two sets of learnable prompts: text and visual. The visual prompts are derived from the text prompts by using projection layers. We denote learnable text prompts as h T and learnable visual prompts as h V : where h T ∈ R p * dt , and h V ∈ R p * dv . d t is the dimension of the text feature, and d v is the dimension of the visual feature. p represents the number of tokens we design for each learnable prompt. EQUATION)ℎ 1 , ℎLet V = {v 1 , v 2 , • • • , v Nv }= {hoi i = (v i , o i )|v i ∈ V; o i ∈ O}.We set C as the total number of HOI classes. Since it is impractical for any datasets to include comprehensive HOI classes, researchers propose zero-shot HOI detection settings to encourage the generalization of HOI detectors to unseen classes C unseen in inference [33] . Denote the seen HOI class set as S and the unseen HOI class set as U = {hoi i | hoi i / ∈ S, hoi i ∈ C}. Please refer to Appendix 7.9 for detailed definitions of the different zero-shot setting unseen set splits.Prompt learning techniques for VLMs are characterized by their ability to adapt large, pre-trained models to specific tasks using relatively small amounts of task-specific data. This often leads to a diminished generalization ability of the fine-tuned models for unseen classes [62, 7, 9, 22] . Given that foundation models, such as LLMs and VLMs, have demonstrated substantial knowledge capacity [29, 47] , leveraging guidance from these models can improve performance on unseen HOI classes that are absent in the training data.Text Prompt Design As shown in Fig. 2 , we design input prompts, "Describe a person <acting> <object> and focus on the action" related to each HOI class, for an LLM. The output from LLM contains richer semantic information with specific HOI class descriptions. Please refer to Appendix Section 7.2 for a detailed explanation with examples.Then, we use a CLIP text encoder to obtain the text embedding for this HOI class description. We process text embeddings for all HOI classes including both seen and unseen in parallel, so the whole text embeddings are denoted asF txt ∈ R C * dt and F txt = [f txt1 , f txt2 , • • • , f txt C ].Then, the learnable text prompts ĥT can be obtained by: In the figure, we take the "hose a dog" unseen HOI class in the unseen-verb zero-shot setting as an example. We first utilize the HOI class text embeddings to identify the most connected seen HOI class to "hose a dog". After selecting the seen class, we generate an input prompt to obtain disparity information from LLM. Finally, the unseen learnable prompt learns from the selected seen class prompt and the disparity information through MHCA.EQUATIONwhere ĥT = [ ĥ T1 , ĥ T2 , • • • , ĥ T C ], and ĥTi ∈ R p * dt . Each class has its specific learnable text prompts after the process of Eq. 2. W up and W down indicate the up-projection and down-projection layers. MHCA means the multi-head cross attention [50] . In Eq. 2, we only aggregate the useful information from F txt using learnable attention, as the information provided by LLM may not all be relevant for our task. Moreover, to keep trainable parameters small, we apply a down-projection layer W down before MHCA to reduce the feature dimension, and an up-projection layer W up afterward. The same design strategy is used in Eq. 3 and Eq. 5. Later, the prompts corresponding to unseen classes are further refined, as detailed in Section 3.2.Visual Prompt Design Visual prompt learning in our method is facilitated by a pre-trained Visual Language Model (VLM) visual encoder, specifically the CLIP visual encoder [47] . The pre-trained CLIP model inherently possesses knowledge of unseen HOI classes, offering richer visual semantics compared to models trained solely on task-specific datasets. The CLIP visual feature is denoted as f I vis for an image I, and we enhance the visual learnable prompts h V by:EQUATIONSince the frozen VLM visual encoder can extract features for unseen HOIs, ĥV aggregates information from these visual features, improving performance on unseen HOIs.Since the training data has no labeled image for unseen HOI classes, learnable text prompts for seen classes are inevitably optimized better than unseen classes. Therefore, we refine the unseen-class text prompts by learning from closely related seen-class prompts. Specifically, we denote one unseen HOI class as u, and the learnable text prompt tokens as ĥ Tu , and denote one seen HOI class as s and the related seen text prompt token as ĥ Ts To identify the related seen class, we use text embeddings of the HOI class descriptions generated by the LLM, as explained in Text Prompt Design of Section 3.1. The related seen class is formulated as follows:EQUATIONDirectly refining prompts for unseen classes based solely on selected seen classes may be insufficient due to inherent differences between the seen and unseen classes. Thus, we propose utilizing disparity information, defined as the differences between unseen class u and seen class s, as provided by the LLM. The architecture of UTPL is shown in Fig. 3 . Please refer to Appendix Section 7.3 for detailed explanation with examples.Since the description can be too long to be encoded at once, we process each sentence separately by using the CLIP text encoder. Thus, set the text embedding of disparity descriptions for an unseen HOI class u as f disp txtu ∈ R m * dt , where m is the number of sentences in the text description. Then we can compute the refined learnable text prompts h Tu for unseen class u as:EQUATIONwhere concat indicates concatenation. For seen classes, h Ts = ĥ Ts . Although ground-truth labels for the unseen classes are not available during training, we propose two strategies to update h Tu in Eq. 5. First, we design a class-relation loss (refer to Eq. 16 in the Appendix) to keep the relationship between seen and unseen classes, measured by cosine similarity between text features. As a result, unseen prompts can be refined based on their relation to seen classes. Second, the annotated training data serves as negative samples for unseen HOIs. If the prediction score for an unseen class is too high, the model is penalized (Eq. 17 in the Appendix).We denote a set of learnable text prompts asH T = [ h1 T , h2 T , • • • , hN T ] and learnable visual prompts as H V = [ ĥ1 V , ĥ2 V , • • • , ĥN V ]. hi T ∈ R C * p * dt , ĥi V ∈ R p * dv , i = 1, 2, • • • , N. N means we intend to introduce new learnable prompts from the first layer to layer N .The text encoder is composed of K transformer layers {T i } K i=1 .The CLIP text encoder generates text features by tokenizing the words and converting them into word embeddingsW 0 = [w 1 0 , w 2 0 , • • • , w P 0 ] ∈ R P * dt .Text features W i will be processed by i th layer of the text transformer:EQUATIONThe final text representation W t can be obtained by:EQUATIONwhere W t ∈ R 1 * da and d a is the feature dimension of the final aligned text and visual features in CLIP. Since our learnable text prompt hi T is specific for each HOI class, the whole text representation can be obtainedW t = [W t1 , W t2 , • • • , W t C ].The visual encoder is also composed of K transformer layers {V i } K i=1 . After each layer V i , there is an adapter A i to inject object position and category information [27] . CLIP visual encoder splits image I into D = d p * d p fixed-size patches, which are projected to obtain visual features E 1 ∈ R D * dv . Visual features E i are processed by i th layer of the visual transformer:EQUATIONwhere c i is the class token for the i th layer.The final visual representation E v ∈ R 1 * da can be computed as follows:E v = VisualProj(E K ). (9) E v ∈ R D * dv can be re-sized to E v ∈ R dp * dp * dv .Since off-the-shelf detectors can provide object detection results, we select bounding boxes with confidence score sc > θ and apply RoI-Align [13] to obtain features for each detected human and object E hum , E obj . Then, the intra-HOI feature fusion extracts HOI featureEQUATIONwhere M LP represents multi-layer perception. In any given image I, multiple humans and objects may be present, leading to the detection of several potential human-object pairs. Assume there are q human-object (H-O) pairs in image I and denote all HOI features asE I hoi = [E 1 hoi , E 2 hoi , • • • , E q hoi ].Incorporating surrounding HOI features enriches context information for each HOI visual feature by capturing additional human-object relational details and interactions, thereby enhancing HOI features. We name this process inter-HOI feature fusion. Thus, final visual representation ẼI hoi is obtained by:EQUATIONwhere MHSA means multi-head self-attention [50] andẼI hoi = [ Ẽ1 hoi , Ẽ2 hoi , • • • , Ẽq hoi ].Then, we can calculate the prediction for each H-O pair hoi i .EQUATIONPlease refer to Appendix Section 7.5 for more details for training and inference. Our method follows the existing zero-shot HOI setting, which involves predicting unseen HOI classes and typically includes using unseen class names during training [14, 15, 16, 44, 51] . In particular, VCL, FCL and ATL [14, 16, 15 ] "compose novel HOI samples" during training with the unseen (novel) HOI class names. EoID [51] distills CLIP "with predefined HOI prompts" including both seen and unseen class names. HOICLIP [44] introduces "verb class representation" during training, including both seen and unseen classes.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discussed the positive and negative societal impact in the supplementary material.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
implementation details: 
 We evaluate our method on HICO-DET by following the established protocol of zero-shot two-stage HOI detection methods [14, 3, 27] . Our object detector utilizes a pre-trained DETR model [5] with a ResNet50 backbone [12] . As for our learnable prompts design, we set p = 2, N = 9. The LLaVA-v1.5-7b model [37] is used to provide text description, as explained in Section 3.1 and 3.2. For all experiments, our batch size is set as 16 on 4 Nvidia A5000 GPUs. We use AdamW [39] as the optimizer and the initial learning rate is 1e-3. For more implementation details, please refer to Appendix Section 7.1.Unseen-Verb Setting In Table 1 , we compare our method to existing zero-shot HOI detection approaches under the unseen-verb zero-shot setting. The results demonstrate that our method not only achieves competitive performance but also requires only 10.35% to 33.95% of the trainable parameters compared to existing zero-shot HOI detection methods thanks to our novel prompt learning design. While our method shows a minor drop in performance compared to CLIP4HOI under the unseen verb setting, it is important to consider the significant reduction in trainable parameters that our method achieves-87.9% fewer than CLIP4HOI. In contrast, the existing HOI methods [33, 44, 41, 4] that align with VLMs unanimously require significantly more trainable UniHOI [4] is the state-of-the-art method in the unseen verb (UV) setting. Compared to UniHOI which aligns HOI features to BLIP [29] text embeddings, our method achieves improved performance, especially in the unseen category with a 2.77 increase in mAP. At the same time, our model requires only 26.9% of the trainable parameters compared to UniHOI. This demonstrates the effectiveness and efficiency of our proposed prompt learning framework in adapting the VLM for zero-shot HOI tasks.In Table 2 , we provide the zero-shot performance comparison in rare-first unseen-composition (RF-UC) and nonrare-first unseen-composition (NF-UC) settings. With the ViT-B visual encoder, our method establishes a new standard for unseen-class performance across both RF and NF settings, outperforming the previous state-of-the-art method, CLIP4HOI [41] , while requiring only 12.08% of its trainable parameters. Compared to UniHOI with ViT-L visual encoder [4] , our method surpasses it by 5.56 mAP in unseen performance of the RF-UC setting and by 7.88 mAP in the NF-UC setting with only 26.9% of the trainable parameters of UniHOI.Unseen-Object Setting We provide the unseen-object setting comparison in Table 3 . Following previous methods [44, 33] , we employ a DETR object detector pre-trained on MS-COCO [34] for object detection. To keep consistent with CLIP4HOI [41] and FCL [16] , we also provide the results by using the DETR model pre-trained on HICO-DET [27] (marked with † in Table 3 ). CLIP4HOI [41] is the state-of-the-art method in unseen performance. With the same object detector pre-trained on HICO-DET, our method shows improved performance for unseen class prediction, outperforming CLIP4HOI by 1.49 mAP, with only 12.08% of its trainable model parameters. UniHOI [4] is the state-of-the-art method in seen performance. With the same object detector as UniHOI, our method outperforms UniHOI in unseen category by 13.36 mAP.We conduct the ablation study for our text and visual prompt learning design in Table 4 . The first row shows the performance of our baseline, MaPLe [22] . Our intra-HOI fusion module improves the seen HOI performance by 7.41 mAP, as shown in the second row. In the third row, the inclusion of the visual adapter [27] enhances performance for seen classes while negatively affecting unseen HOI performance.Comparing the third row and fourth row, LLM guidance, shown in Fig. 2 , notably enhances unseenclass learning, resulting in a 1.52 mAP improvement. LLM guidance, by integrating learnable text prompts with detailed HOI class descriptions, improves the model's understanding of unseen classes, providing richer information compared to simple class names. Additionally, the inclusion of the UTPL module significantly boosts unseen class performance, with a 2.42 mAP improvement. Later, the inter-HOI fusion benefits both the seen and unseen learning by providing more context We also provide the experimental comparison with CLIP [47] and MaPLe [22] in Table 5 . Since they are designed for general image classification, directly applying it to HOI tasks for comparison is unfair as they may not focus on the human-object region features. Therefore, we crop the union region from the original visual features for each human-object pair to obtain the HOI feature. HOI prediction is obtained by using cosine similarity between HOI and text features for each class.As shown in Table 5 , although with learnable prompts, MaPLe [22] outperforms CLIP on seen classes, its performance on unseen classes is far behind that of the seen classes, showing reduced generalization capability. Compared to our method, MaPLe lags significantly behind on unseen performance, with an 11.63 decrease in mAP. Additionally, the third row introduces another baseline that incorporates MaPLe with visual adapter (i.e., A i ) [27] , which is also used in our framework, ensuring a fair comparison. The performance improvement from the third row to the fourth row, with gains of 5.93 mAP for unseen classes and 3.86 mAP for seen classes, clearly demonstrates the effectiveness of our method in zero-shot HOI detection.Fig. 4 shows the qualitative results of MaPLe [22] and our method in the unseen-verb zero-shot setting. In particular, MaPLe struggles to detect unseen classes, either missing unseen HOI classes or predicting the wrong unseen HOI classes. For example, if an image only contains unseen classes, MaPLe tends to predict wrong seen classes and miss the correct ones. As shown in the bottom right of Fig. 4 , this image contains unseen class only ("wear tie"), MaPLe predicts related wrong seen classes such as "pulling tie" and "adjusting tie", but fails to predict the ground-truth unseen HOI ("wear tie"). This shows the limited generalization ability of MaPLe to unseen classes. In contrast to MaPLe, our method can predict both seen and unseen classes more accurately.We conducted the fully-supervised experiments on both the HICO-DET and V-COCO benchmarks [6, 34] . Our method achieves a competitive 38.61 mAP on the HICO-DET dataset, with a smaller performance drop between rare and non-rare classes (1.19 mAP) compared to AGER [49] (4.18 mAP), the current state-of-the-art one-stage method. Our method also outperforms the best-performing zero-shot HOI method, CLIP4HOI [41] , by 3.28 mAP on HICO-DET. On the V-COCO benchmark, our method achieves state-of-the-art performance with 66.2 AP in Scenario 2. Detailed discussions and comparisons are provided in the Appendix 7.6. Descriptors, developed to leverage the LLMs for downstream tasks [42, 20] , involve attribute decomposition to benefit category recognition. We adopt the attribute decomposition concept from DVDet [20] and tailor it to our EZ-HOI framework. Following DVDet, we generate action descriptors for each class and integrate them into HOI class text features, thus enhancing the detail and distinctiveness of class representations. Descriptors with low cosine similarity to the class text features are discarded to avoid noise. With our straightforward adoption of action descriptors, the result shows 0.31 mAP improvement compared to our EZ-HOI, achieving 32.63 mAP under the unseen-verb setting. This indicates that LLM-generated descriptions, such as action descriptors, hold potential to enhance HOI detection and are worth further exploration.Our method has some limitations. First, while our method significantly reduces the number of trainable parameters compared to existing approaches, it still requires fine-tuning on HOI-specific datasets such as HICO-DET. A training-free design, which could be used directly for HOI detection without the need for fine-tuning, would be more desirable. Second, zero-shot HOI detection requires the pre-definition of all HOI classes, both seen and unseen, following the current zero-shot HOI detection setting. This requirement restricts the generalizability to truly unseen classes. Consequently, our future work will aim to develop a robust open-category HOI detector that operates effectively without the need for predefined classes. 
conclusion: 
 In this paper, we introduced the Efficient Zero-shot HOI Detection (EZ-HOI) approach, utilizing prompt learning to adapt visual-language models for zero-shot HOI detection. This method not only maintains competitive performance for unseen classes but also innovatively integrates learnable visual and text prompts. These prompts leverage foundation model information, thereby enriching prompt knowledge and enhancing the adaptability of VLMs. A significant challenge we addressed is the compromised performance on unseen classes, resulting from the training dataset containing only seen-class labels. To counter this, we developed a text prompt learning strategy that utilizes information from related seen classes to support the detection of unseen classes. We also employed an LLM to provide the nuanced differences between related seen and unseen classes, improving our method for unseen class prompt learning. Our method has demonstrated state-of-the-art performance across various zero-shot HOI detection settings while requiring only a third of the trainable parameters compared to existing methods. 
implementation details: 
 Here we provide detailed implementation information. For the pre-trained CLIP model with the ViT-B visual encoder, the visual feature dimension d v = 768, while the text feature dimension d t = 512 and the final feature dimension for aligned visual and text features d a = 512. For the CLIP model with the ViT-L visual encoder, d v = 1024, d t = 768, d a = 768. We use an off-the-shelf object detector and add a threshold θ to filter out some low-confident predictions and we set θ = 0.2. Since UniHOI [4] and CLIP4HOI [41] have not released their code, we estimate the trainable model parameters by modifying the HOICLIP [44] code according to the description in UniHOI and CLIP4HOI, which means that if some details are not mentioned in the UniHOI and CLIP4HOI papers, we use the HOICLIP design by default. In addition, we initialize the weights of all W up to 0, which stabilizes training by gradually fine-tuning the attention outputs in Eq. 2, Eq. 3, Eq. 5 and Eq. 11.We utilize the LLM to provide detailed HOI class descriptions to guide the text prompt learning and we provide an example for detailed illustration. We take the HOI class "Swing a baseball bat" as an example, which is one unseen HOI class in the unseen-verb zero-shot setting on the HICO-DET benchmark. The generated HOI class description is "Swinging a baseball bat" describes a person using a baseball bat to hit a ball. This action typically involves the person holding the bat with both hands, standing in a stance with their feet shoulder-width apart, and using their body rotation to contact the ball.The disparity information mentioned in Section 3.2 aims to explore the difference between the unseen class and the related seen class. Here we demonstrate it with a specific example. We take the unseen class " hose a dog" as an example and its selected seen class is "wash a dog".The input text prompt for LLM to acquire the disparity description between the two classes is designed as " Describe the definition of the phrase: hosing a dog and please focus on the attributes different from another phrase: washing a dog. " Then we can obtain the detailed disparity description from LLM asThe phrase "a person hosing a dog" refers to the action of washing a dog using a hose. This action is different from "a person washing a dog" as the person is using a hose to clean the dog, rather than simply using their hands to wash the dog. The use of a hose adds an additional element of water pressure and flow, which can make the cleaning process more efficient and effective. As shown in Fig. 5 , we also provide the detailed architecture of the intra-and inter-HOI feature fusion design. The intra-HOI feature fusion integrates human and object features for each human-object pair.To enrich context information, all HOI features are processed together using multi-head self-attention (MHSA), allowing each feature to become aware of its surroundings. This enhancement improves the context and overall performance of each HOI feature.Since W t ∈ R C * dt related to the number of total HOI classes, if the number is too large, the learnable prompts can be computationally expensive. Thus, we only select part of HOI classes and do action classification. Later, by combining action prediction with object detection results, we can obtain HOI prediction finally. Specifically, we select two HOI classes for each action with the following equation:EQUATIONwhich means we select two HOI classes containing the same action with the most different semantic meanings. Since some actions can have multiple interpretations in different contexts (e.g., "hold apple" vs. "hold sheep"), randomly choosing two HOI classes with the same action does not cover the comprehensive meanings of the action. By selecting the most different HOI classes, we aim to capture a richer range of information for the action. Then, We calculate the HOI prediction score for i th H-O pair by:EQUATIONThe action scores can be obtained by:EQUATIONwhere lalign ∈ R C * da means the action label for each selected HOI class. We adopt focal loss [35] l f ocal to train the action prediction. We also design a class-relation loss shown in the following equation: EQUATIONEQUATIONand α is a hyperparameter. In the inference stage, we can obtain the HOI score for each human-object pair by:EQUATIONwhere s h , s o are the object detection confidence scores for humans and objects. τ is a hyperparameter. In Eq. 17, α = 150, and in Eq. 18, τ = 1 during training and τ = 2.8 during inference [58, 59] .Fully Supervised HOI Detection Setting We conduct experiments under the fully supervised HOI detection setting on both the HICO-DET [6] and V-COCO [34] dataset, as shown in Table 6 . In the UTPL module, we utilize common class prompts as inputs. This approach enables common HOIs to learn from rare ones, fostering a better differentiation between them. This strategy helps to alleviate overfitting to common HOI classes, thereby improving performance for both seen and unseen classes.As shown in Table 6 , the performance drop between rare and non-rare classes ( 1.19 mAP) is much smaller than AGER (4.18 mAP), the best performance among the one-stage method. Additionally, our method achieves the best performance among the two-stage methods on the HICO-DET [6] dataset, outperforming CLIP4HOI [41] by 3.28 mAP.As for the V-COCO benchmark, we achieve competitive performance among the two-stage methods with 66.2 AP performance under Scenario 2 evaluation. However, the smaller number of verb classes in V-COCO (24 classes), which have weaker connections (i.e., jumping vs. skateboarding skateboard), compared to HICO-DET (117 verb categories), which exhibits stronger connections (i.e., jumping vs. flipping skateboard), limits the potential of our method. Our UTPL design requires the learnable prompts to extract information from prompts of other classes, which also aids in better differentiation between similar classes. Due to the weakly connected classes in V-COCO, our UTPL design cannot work effectively. Despite this, we achieve performance comparable to CLIP4HOI [41] , further demonstrating the effectiveness of our method. Table 7 shows the ablation study for the hyperparameter N , where N means we introduce learnable prompts from the first layer until layer N in both visual and text encoders. We use N=9 in our main paper because it shows the best unseen performance.Table 8 shows the ablation study for the different positions of learnable prompts. Position i -j means we insert learnable prompts from layer i to layer j in both the text encoder and visual encoder.Here we always insert learnable prompts into 9 layers and only change the position. We find that the position of learnable prompts does not affect the outcome too much. Thus, we follow [22] and fine-tune layers 1-9, which show slightly better unseen performance. As shown in Fig. 6 , we provide more qualitative results in three zero-shot HOI settings: unseenverb, rare-first unseen-composition, and nonrare-first unseen-composition settings. Compared to MaPLe [22] , our method obtains better generalization capability to unseen HOI classes owing to our efficient prompt learning design.In Fig. 7 , we show more qualitative comparisons to illustrate the effectiveness of the LLM guidance and UTPL design. The baseline here refers to the method in the main paper's third row of Table 4 . LLM guidance design utilizes the general description from LLM for each HOI class and the UTPL module integrates the distinctive description from LLM. As shown in Fig. 7 , the LLM guidance provides detailed class information, improving the performance over the baseline. Distinctive descriptions from UTPL design help to distinguish unseen classes from related seen HOIs, enhancing unseen performance and challenging case predictions.7.9 Discussion for Zero-Shot HOI Detection DefinitionOur method follows the standard zero-shot HOI setting, where unseen class names are used during training [14, 15, 16, 51, 44] , as discussed in Section 4.1. Specifically, before training, the whole verb set V = {V seen , V unseen } and the whole object set O = {O seen , O unseen } are all pre-defined. The four settings of zero-shot HOI detection include : 1) Rare-First Unseen Composition (RF-UC), where for all hoi i = (v i , o i ) ∈ U, we have v i ∈ V seen , o i ∈ O seen and hoi i appears less than 10 times in the training set, belonging to rare HOI classes. 2) Nonrare-First Unseen Composition, where for all hoi i = (v i , o i ) ∈ U, we have v i ∈ V seen , o i ∈ O seen and hoi i appears more than 10 times in the training set, belonging to nonrare HOI classes. 3) Unseen Verb (UV), wherehoi i = (v i , o i ) ∈ U, we have v i ∈ V unseen , o i ∈ O seen . 4) unseen Object (UO), where hoi i = (v i , o i ) ∈ U, we have v i ∈ V seen , o i ∈ O unseen .Beyond the zero-shot HOI setting, there are HOI unknown concept discovery [17] and openvocabulary HOI detection [28] , where unseen class names cannot be used in training. The openvocabulary setting differs from HOI unknown concept discovery, with a much wider range of unseen HOI classes during testing.Our approach to zero-shot HOI detection reduces reliance on extensive annotated datasets, enhancing accessibility for organizations with limited resources and promoting inclusivity in technology adoption. This technology could notably improve assistive devices, offering more intuitive aids for individuals with disabilities by enabling better understanding of new environments.However, these advancements also pose risks. The capability to interpret human-object interactions could be used for surveillance, potentially infringing on privacy. Additionally, reliance on existing visual-language models may maintain embedded biases, leading to discriminatory outcomes. To Justification: We emphasize our contributions in the Introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We conducted extensive experiments using the widely-recognized HICO-DET [6] dataset for zeroshot HOI detection. HICO-DET comprises a total of 47,776 images, divided into 38,118 training images and 9,658 test images. This dataset features 600 Human-Object Interaction (HOI) classes, which are combinations derived from 117 action categories and 80 object categories. Our model's performance was evaluated in four distinct zero-shot HOI detection settings, categorized by the criterion for selecting the unseen HOI classes: rare-first unseen composition (RFUC), nonrare-first unseen composition (NFUC), unseen object (UO), and unseen verb (UV). These settings align with methodologies from previous research [33, 44, 41, 4] .Additionally, we also evaluate our model in the fully supervised setting on both the HICO-DET dataset and the V-COCO [34] dataset. V-COCO is a subset of COCO, with 10,396 images-split into 5,400 train-val images and 4,946 test images, and it encompasses 24 action classes and 80 object classes. Following standard evaluation protocols, we assessed our model using mean average precision (mAP) on the HICO-DET benchmark, while Average Precision (AP) in both Scenario 1 and Scenario 2 is used on the V-COCO benchmark [63, 44] . A prediction was deemed a true positive if the HOI classification was accurate and the Intersection over Union (IoU) between the predicted human and object bounding boxes and the ground-truth bounding boxes exceeded 0.5 [33, 27] . 


Decision: The paper was evaluated by six reviewers. Authors provided strong rebuttal and most reviewers engaged in the author-reviewer discussion. As a result, five out of the six reviewers provided positive ratings, and the sixth reviewer who provided negative review did not respond to authors rebuttal. After reading the reviews and rebuttal, the AC agrees to accept this paper. Authors are encouraged to polish the paper further by considering the questions, suggestions and comments of the reviewers.

Reviews: review: paper_summary: paper_summary: This paper tackles zero-shot HOI detection via prompt tuning. To address the challenge posed by the absence of novel classes, the authors first incorporate LLM and VLM guidance to enrich learnable prompt tokens. Further, the authors utilize LLM to provide nuanced differentiation between unseen classes and their related seen classes, termed Unseen Text Prompt Learning (UTPL), to alleviate overfitting to seen classes. In experiment, the proposed method achieves state-of-the-art performance under major zero-shot HOI detection settings while costing fewer training resources.
strength_weakness: strength_weakness: 1. It seems the authors assume the prior knowledge of unseen class names when training the UTPL module, which conflicts with the zero-shot setting.

2. In Table 1, the proposed method underperforms CLIP4HOI (ResNet50+ViT-B) in unseen mAP although it is claimed to tackle the overfitting issue especially.
questions: questions: 1. In Figure 2, are the VLM visual and visual encoder the same thing (i.e., CLIP)?

2. Why did the authors use LLaVA for text description instead of choosing pure language models?
limitations: limitations: N/A
paper_summary: paper_summary: The paper studies the challenges behind successful adaption of pre-trained vision language models (VLMs), i.e., CLIP, to the problem of zero-shot Human Object Interaction (HOI) detection. Specifically, during finetuning, VLMs overfit to seen HOI classes observed during the training on HOI training data, preventing successful transfer on the unseen HOI classes. To overcome the aforementioned problem, authors propose prompt tuning mechanism coupled with guidance from a large language model (LLM). In particular, LLM is used to generate (i) descriptions of each HOI class; (ii) explain the difference between each unseen and the corresponding closest seen class to facilitate transfer on unseen classes. The prompt tuning technique itself learns a set of prompts that are shared between vision and text encoders. Authors employ attention mechanism to produce modality specific prompts via attending on class descriptions for text modality and via attending on the image embedding for visual modality correspondingly. Authors evaluate the proposed approach on the standard HOI benchmark and compare it to the recent baselines, showing improvements in terms of the parameter efficiency and the performance on unseen classes.
strength_weakness: strength_weakness: My main concern is that the current narration of the proposed methodology lacks intuition and structure. For example, 

**Structure**
* Section 3 starts with mentioning that the method will learn prompts per layer, and the subsequent section also use notation considering N layers, but the first time the reader encounters interaction with the layers is Section 3.3. It makes the paper hard to follow.

**Intuition**
* The main idea of the approach builds on repeatedly applying MHCA for different purposes, however these equations just stated on the high-level in Eq. (2); (3); (5) and without any intuition behind the design.

* Similarly to the above, Eq. (6) and Eq. (8) state how these learnable prompts are used in the corresponding encoder layers, however the narration is on the high level and does not elucidate the particular idea behind the equations.
questions: questions: * How $N=9$ was chosen? Can you provide ablations for different number of $N$?
* Similarly to the question above, Eq. (6) and (8) suggest that the learnable prompts are inserted in the first $N$ layers of textual and visual encoders, correspondingly. What is the motivation behind this design choice? I, in general, would assume that earlier layers already consumed broad knowledge during CLIP pre-training and task specific adaptation is required for penultimate layers. Can you please provide ablations on different positioning of the learnable prompts?
* Currently, the paper narration mainly focuses on tackling generalization to unseen classes and, indeed, experimentally confirms the improvements offered by the proposed approach. The main component causing such improvements is seem to be UTPL that employs guidance from LLM. However, according to Table 4, other components also bring substantial improvements. Can you please provide the intuition behind other components' role and how they can bring these improvements?
limitations: limitations: Authors discuss the limitations of the current zero-shot HOI detection setting. In particular, that to truly enable zero-shot HOI detection, one must exclude the assumption of having in advance a set of unseen HOI classes.
paper_summary: paper_summary: The paper proposes a novel prompt learning framework for zero-shot Human-Object Interaction (HOI) detection, which enhances the generalizability of Vision-Language Models (VLMs) by interacting with Large Language Models (LLMs) to obtain descriptions.
strength_weakness: strength_weakness: 1. The prompt design seems to be a hybrid approach, combining visual prompt learning [1, 3] with text prompts, such as those used in COCOOP, without a specific focus on task-specific configurations. It is difficult to justify the necessity of this work for the field.

2. Despite the claim of improved performance on novel classes, the method performs worse on unseen classes compared to CLIP4HOI, while it shows improved performance on seen classes. Given that novel class performance is crucial, this result seems not consistent with this task.

3. W_down presents in both Equation 3 and Equation 5 is puzzling. Equation 5 attempts to integrate descriptions for unknown categories with similar known categories and LLM-generated details. More details on the training process would be helpful, as it appears that data training might be lacking since unknown categories are not included in the training.

4. There is substantial related work on using LLMs for new class descriptions in zero-shot scenarios, such as CuPL [2] for classification and DVDet [3] for detection, which use attribute decomposition to aid in categorization. DVDet also generates more discriminative descriptors by distinguishing confusing categories. The paper lacks a discussion on such related LLM-based works. It would be interesting to see if using action descriptors might also yield some benefits.

5. The paper could benefit from providing more visualization results to demonstrate the effectiveness of both general and discriminative descriptions generated by the method.

[1] Visual Prompt Tuning

[2] VISUAL CLASSIFICATION VIA DESCRIPTION FROM LARGE LANGUAGE MODELS

[3] LLMS MEET VLMS: BOOST OPEN-VOCABULARY OBJECT DETECTION WITH FINE-GRAINED DESCRIPTORS
questions: questions: Please see the weakness.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper investigates the problem of human-object interaction (HOI) detection. This paper introduces EZ-HOI, a method for efficient zero-shot HOI detection in an open-world setting. EZ-HOI also explores the use of LLM and VLM guidance for learnable prompts to enhance prompt knowledge and aid in adapting to HOI tasks. To better adapt to unseen classes, EZ-HOI establishes relationships between unseen and seen classes.
strength_weakness: strength_weakness: - The overall method is straightforward but lacks sufficient novelty. Compared to UniHOI, which also claims to utilize Spatial Prompt Learning, the main difference in EZ-HOI lies in its use of more traditional prompt learning methods (like VPT-DEEP) and its additional modeling of relationships between unseen and seen classes when handling unseen categories. Since many previous works have already demonstrated that prompt tuning (ViT-adapter/CLIP-adapter) can achieve even higher performance than full fine-tuning, the claims of EZ-HOI seem somewhat weak. Additionally, modeling the relationship between unseen and seen classes as a technical contribution also appears somewhat limited.
- The fully-supervised setting results are provided in Appendix Table 6, but the performance is worse than Uni-HOI. Firstly, this is a very important experiment and should be included in the main text. Secondly, it appears that under full supervision, EZ-HOI performs worse than Uni-HOI, which contradicts the claims in the main text. Have the authors provided an in-depth analysis of the reasons? If the performance difference is due to parameter count, what would the performance be with full fine-tuning?
- The performance improvement seems to come from the baseline rather than the proposed modules. As shown in the ablation study (Table 4), the baseline of EZ-HOI achieves 37.44 on seen categories, which is already state-of-the-art (SOTA). I wonder what EZ-HOI’s baseline is and why it can achieve such strong performance on seen categories. Additionally, considering that the UTPL module increases unseen category performance by two points, and EZ-HOI shows a two-point increase compared to Uni-HOI, does this mean that EZ-HOI’s performance gain over Uni-HOI is primarily due to the UTPL module?
questions: questions: What is the fully fine-tuned performance? What is the baseline?
limitations: limitations: This submission discusses limitations.
paper_summary: paper_summary: This work presents EZ-HOI, an innovative framework that addresses the zero-shot HOI detection challenge by employing prompt learning. It integrates LLM and VLM guidance to enrich prompts and adapt to HOI tasks effectively. By learning from related seen classes, EZ-HOI overcomes the limitation of lacking labels for unseen classes, enhancing its performance on them. The framework achieves state-of-the-art results with significantly fewer trainable parameters than existing methods, showcasing its efficiency and effectiveness in zero-shot HOI detection.
strength_weakness: strength_weakness: - This paper primarily addresses the improvement of zero-shot HOI detection benchmark performance by training with the pre-definition of all HOI classes. This creates an unfair comparison with previous work.
- In light of the above, the reviewer believes that the authors should provide experimental results where the pre-definition of all HOI classes is unknown during training, allowing for a fairer comparison with previous work.
questions: questions: N/A
limitations: limitations: Yes
paper_summary: paper_summary: In order to solve the HOI detection problem in a zero-shot setting, the authors propose the Unseen-class Text Prompt Learning module. Using learnable visual prompts and textual prompts, it effectively utilizes the knowledge from large language models and Vision-Language Models and performs well in unseen class HOI detection.
strength_weakness: strength_weakness: 1. In Qualitative Results, the authors use Figure 4 for visual illustration, but it is not possible to conclude from Figure 4 that "MaPLe tends to predict seen classes with high confidence scores". The authors should have given a clearer and more accurate illustration of the figure.
2. The authors claim to have "employed an LLM to provide the nuanced differences between related seen and unseen classes, improving our method for unseen class prompt learning. prompt learning." However, the information provided by the LLM may be misleading or inaccurate, did the authors filter the information? If not, what can be done to avoid HOI detection being negatively affected from the information provided by LLM?
3. For the UTPL module, the authors split the LLM-generated description into multiple sentences; how do they control the length of individual statements?
4. The author mentions: " A prediction was deemed a true positive if the HOI classification was accurate and the Intersection over Union (IoU) between the predicted human and object bounding boxes and the ground-truth bounding boxes exceeded 0.5.", why is the threshold value chosen to be 0.5 instead of 0.7 or other? What is the basis for the choice?
5. WRITING DETAILS: Abbreviated nouns should be introduced the first time they appear.
questions: questions: 1. In Qualitative Results, the authors use Figure 4 for visual illustration, but it is not possible to conclude from Figure 4 that "MaPLe tends to predict seen classes with high confidence scores". The authors should have given a clearer and more accurate illustration of the figure.
2. The authors claim to have "employed an LLM to provide the nuanced differences between related seen and unseen classes, improving our method for unseen class prompt learning. prompt learning." However, the information provided by the LLM may be misleading or inaccurate, did the authors filter the information? If not, what can be done to avoid HOI detection being negatively affected from the information provided by LLM?
3. For the UTPL module, the authors split the LLM-generated description into multiple sentences; how do they control the length of individual statements?
4. The author mentions: " A prediction was deemed a true positive if the HOI classification was accurate and the Intersection over Union (IoU) between the predicted human and object bounding boxes and the ground-truth bounding boxes exceeded 0.5.", why is the threshold value chosen to be 0.5 instead of 0.7 or other? What is the basis for the choice?
5. WRITING DETAILS: Abbreviated nouns should be introduced the first time they appear.
limitations: limitations: Yes

Comments: title: Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.
Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.
Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.
Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.
kicking off reviewer-author discussion
Acknowledgement
comment: Dear reviewer, with the deadline for our discussion approaching, we kindly ask if you have any remaining concerns or feedback. Your insights are highly valuable to us, and we would be grateful for any further comments or questions you may have.
Thank you for your valuable support. We're pleased that our rebuttal has addressed most of the concerns. We will incorporate the clarification of zero-shot HOI setting configurations in our updated paper.

Regarding the previous feedback from the reviewer on the experiment without pre-defining all HOI classes during training, we believe it would be helpful to expand on this point. Although this experiment, involving unseen class names during training, is beyond the scope of our submission, we would still like to provide the results below to demonstrate our method's effectiveness beyond the zero-shot setting.

As shown in the table below, without unseen class names in training, our model still outperforms the SOTA method UniHOI [3], especially for unseen performances. This experiment highlights the potential applicability of our method beyond the zero-shot HOI setting. 


| Method |Setting| Full | Unseen | Seen |
| :-----| :----: | :----: |:----: |:----: |
| UniHOI|UV |34.68|26.05|36.78|
|Ours w/o unseen class names |UV|__36.12__ |__28.24__| __37.41__ |
| UniHOI|RF-UC |32.27|28.68|33.16|
|Ours w/o unseen class names |RF-UC|__36.96__ |__32.10__ |__38.17__|
| UniHOI|NF_UC|31.79|28.45|32.63|
|Ours w/o unseen class names |NF_UC|__34.26__| __34.80__ | __34.13__|
Thank you for the positive comments and for raising the score. We're pleased that our rebuttal addressed most of your concerns. Based on your suggestions, we will include the visualized results and discussions in our paper.
Thanks for the authors' response. The rebuttal has addressed most of my concerns. I keep my positive score.
The author addressed most of my concerns, and the visualized results are also reliable, so I have decided to increase my score.
Hi dear reviewer,

Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.

Thanks,

AC
Hi dear reviewer,

Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.

Thanks,

AC
Hi dear reviewer,

Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.

Thanks,

AC
Hi dear reviewer,

Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.

Thanks,

AC
Thank you for your positive feedback. We're pleased that our rebuttal has addressed your concerns.  We will include the detailed discussion for zero-shot HOI setting configuration in our paper.
Thank you for the positive feedback and valuable suggestions. We're glad that our rebuttal has addressed your concerns. We will include the provided clarifications in our paper, and also expand the discussion on existing prompt-tuning techniques in our paper as suggested. If there are any needs for further information or clarification, please let us know.
Thanks for the authors' response. My major concerns about the zero-shot setting have been resolved.
Thanks the authors for the rebuttal, and reviewers' for the comments. We are in the period of author-reviewer discussion.
- Reviewers, please read through all the reviews and rebuttal and see if authors' responses have addressed your and others' important questions.
- Authors, please follow up with any reviewers' further comments.

Cheers,

AC
I thank the authors for the provided clarifications. After reading the other reviewers responses and the rebuttal I will maintain my positive score. I strongly suggest authors to include the provided clarifications in the future revision of the manuscript. Given that, partially, the proposed approach employs well-established techniques in the prompt tuning community, I also believe that the paper will greatly benefit from expanding discussion and positioning with respect to these existing techniques.