Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models

Authors: liulei  li
wenguan  wang
yi  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 19:39:29

Abstract: Prevalent human-object interaction (HOI) detection approaches typically leverage large-scale visual-linguistic models to help recognize events involving humans and objects. Though promising, models trained via contrastive learning on text-image pairs often neglect mid/low-level visual cues and struggle at compositional reasoning. In response, we introduce DIFFUSIONHOI, a new HOI detector shedding light on text-to-image diffusion models. Unlike the aforementioned models, diffusion models excel in discerning mid/low-level visual concepts as generative models, and possess strong compositionality to handle novel concepts expressed in text inputs. Considering diffusion models usually emphasize instance objects, we first devise an inversion-based strategy to learn the expression of relation patterns between humans and objects in embedding space. These learned relation embeddings then serve as textual prompts, to steer diffusion models generate images that depict specific interactions, and extract HOI-relevant cues from images without heavy finetuning. Benefited from above, DIFFUSIONHOI achieves SOTA performance on three datasets under both regular and zero-shot setups.

Main paper: 
 introduction: 
 As a crucial topic in the field of visual scene understanding, human-object interaction (HOI) detection demands not only inferring the semantics and locations of entities but also should comprehend the ongoing events happening between them [1, 2] . Given the complexity and diversity of human activities in object-rich realistic scenes, this task presents challenges in long-tailed distributions and zero-shot discovery [3] . A set of studies seek to tackle these two issues by leveraging large-scale visual-linguistic models (e.g., CLIP [4] ) which show strong generalization ability on dozens of tasks. Though strides made, it has been observed that models trained by aligning high-level text-image semantics face difficulties in discerning spatial locations [5] , and struggle at compositionality [6] which is a fundamental ability for human to capture new concepts by combining known parts. In fact, both middle-level visual cues (e.g., spatial relation) and compositionality are essential facets for HOI detection. The former can help deduce feasible interactions according to locations between instances, while compositionality contributes significantly to zero-shot generalization. For example, we can easily understand humanhold-horse by composing human-hold-dog and class horse that have encountered previously.In contrast, the text-to-image diffusion models [7] [8] [9] [10] [11] [12] [13] [14] also pre-trained on large-scale image-text pairs, are demonstrating superior capabilities outperforming models like CLIP. Concretely, they are able to generate diverse high-quality images conditioned on textual inputs, showing proficiency in understanding high-level semantics [15, 16] . In addition, the generated images convey reasonable shape, texture, layout, and structure, indicating the comprehension in mid/low-level visual concepts as generative models [17] . More importantly, the descriptions are typically organized in a compositional manner, with phrases such as "happy","near a bridge", or "hugged by a man" continually appended to objects like "a dog". This suggests that diffusion models inherently possess compositionality, to systematically adapt to newly encountered user requirements by composing known visual concepts.The above analysis motivates us to explore diffusion models for HOI detection. Nonetheless, to fully unlock the potential of diffusion models and accommodate the unique characteristic of HOI detection task, the following questions naturally arise: ❶ With diffusion models typically emphasizing instance generation, how to steer it to prioritize the relationships between humans and objects? ❷ How to transfer the extensive knowledge obtained from large-scale pre-training in diffusion models to assist the recognition of interactions? To address ❶, we harness textual inversion [18] which conceptualizes a user-provided object by inverting it to a text embedding. However, this method focuses solely on instance objects. To facilitate a smooth shift from object-centric to relation-centric modeling, we devise a human-object relation inversion strategy grounded in the disentanglement of HOI. Concretely, given the HOI latent describing human-action-object, we build a cycle-consistency objective to reconstruct it from a intermediate relation latent derived from the original HOI latent. This reconstruction process is guided by a set of learnable relation embeddings as text prompts, for which we use the placeholder R * to denote the textual form before encoded into embedding space. These relation embeddings further involves in a relation-centric contrastive learning to enhance the awareness of high-level relational semantics. To answer ❷, we leverage both the text promoted image generation and conditioned feature extraction abilities of diffusion models. We realize relation-driven image generation by compositionally organizing R * with other linguistic elements to formulate new text prompts (Fig. 1(b) ). This allows for the generation of novel interactions with unseen objects, and extends the training set for HOI detectors. Moreover, we directly utilize diffusion models as backbone to extract HOI-relevant features conditioned on R * (Fig. 1(c )). After a single noise-free forward step, features distinct for each interaction can be obtained. Finally, to establish a loop for mutual boosting between above relation-inspired HOI detection and relation modeling, we devise an online update strategy to facilitate the continual evolving of relation embeddings during HOI detection learning.Benefited from controllable image generation and knowledge transfer from diffusion models, our method named DIFFUSIONHOI enjoys several appealing advantages: First, it steers diffusion models to focus on complex relationships rather than single objects in an efficient way. This offers a robust foundation for HOI modeling. Second, from the perspective of relation-driven, it unlocks the image generation power of diffusion models tailored for the HOI detection task. This enriches the pool of training samples, particularly for long-tailed/unseen interaction classes. Third, the relation-inspired prompting improves both the flexibility and accuracy of HOI detectors. It adapts to each individual image to extract action or object related cues, while CLIP-based methods [3, 19] produce action/object features merely from texts (i.e., Fig. 1(a) ), remaining static and unresponsive to image content.By embracing text-to-image diffusion models as well as facilitating relation-driven image generation and prompting, our method demonstrates superior performance. It surpasses all top-leading solutions on HICO-DET [20] and V-COCO [21] , and sets new state-of-the-arts. In addition, it yields up to 6.43% mAP improvements on SWiG-HOI [22] under the zero-shot HOI discovery setup. These promising performance evidences the great potential of integrating diffusion models for visual relation understanding. We hope this work could foster the broader exploration of large-scale pre-trained diffusion models on more computer vision tasks beyond mere image generation. 
related work: 
 Human-Object Interaction Detection. According to the architecture design of networks, existing solutions for HOI detection can be broadly categorized into two groups: one-stage and two-stage.The one-stage methods [23] [24] [25] [26] typically employ a multi-task learning pipeline that jointly undertake the tasks of human-object detection and interaction classification in an end-to-end manner, therefore distinguished by fast inference. In contrast, two-stage methods [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] first detect entities with offthe-shelf detectors such as Faster R-CNN [38] , and the predict the dense relationships among possible human-object pairs. This paradigm effectively disentangles the HOI detection process and results in improved performance. Inspired by DETR [39] , recent advancements shift to adopt Transformerbased architectures [3, [40] [41] [42] [43] [44] [45] [46] . Several studies [3, [47] [48] [49] [50] [51] also supplement the Transformer-based HOI detectors with large-scale visual-linguistic models like CLIP [4] or visual knowledge [52, 53] to conduct logic-induced reasoning [54] . However, these models focus solely on aligning high-level semantics and overlooking mid/low-level visual cues. To tackle this, we redirect our attention to diffusion models, which perfectly address the aforementioned challenges and possess the capacity to handle previously unseen concepts through their strong compositionality.Controllable Image Generation. To facilitate customized image generation with respect to predefined class, attribute, text or image [55] , various approaches based on GANs [56] have been proposed. For instance, [57, 58] develop a photo realistic hairstyle transfer method through latent space optimization. However, these methods typically show limited diversity when compared to likelihood-based models [59] . In response, diffusion models [60] [61] [62] have emerged that not only demonstrate remarkable synthesis quality but also offer enhanced controllability. The core idea behind is to transform a simple and known distribution (e.g., Gaussian) into the desired data distribution. These models have proven to be highly effective in various conditional scenarios. According to the conditional targets, the prevalent work can be grouped into class-driven [63, 64] text-driven [7, 8] , exemplar image-driven [65, 66] , etc.. These advances have found application in a wide range of domains such as super resolution [66, 67] , image editing [13, 68] . Recently, a new approach achieves guided image generation by learning a single word embedding through a frozen text-to-image model to properly describe the desired target objects [18] . Take inspiration from it, we achieve relation-driven image generation by extending such object-centric concept modeling approach to relation-centric.Knowledge Transfer from Diffusion Models. In light of the notable success achieved by diffusion models in applications, there is a growing interest in transferring knowledge acquired from large-scale pre-training to various tasks [17, [69] [70] [71] [72] [73] [74] [75] . For example, given the limited availability of data for constructing NeRFs and the unprecedented generalizability of diffusion models, researchers are motivated to explore generating 3D NeRFs via a 2D text-to-image diffusion model using diverse input text [69] [70] [71] . More recently, a notable trend has emerged where efforts are dedicated towards learning semantic representations from diffusion models by extracting intermediate feature maps. It finds diverse application in image segmentation [17] , semantic correspondence learning [72] [73] [74] , and general representation learning [75] . In this work, we extensively harness both the image generation and semantic representation abilities of diffusion models, by using relation-centric embeddings to control the generation and prompt semantic extraction from images with respect to specific interactions. 
preliminary: textual inversion: 
 Latent diffusion models [14] represent an evolution of diffusion models which offer significant enhancements in both computational and memory efficiency by executing denosing in the latent space. It comprises two primary components. The first is a pre-trained generator equipped with an encoder E to map the input image x into a latent vector z = E(x), from which the original data can be reconstructed via a decoder D by x = D(z) ≈ x. The second is a diffusion model to generate latent codes z conditioned on user guidance y which can be text, image, etc.The latent codes then serve as inputs to D for image generation w.r.t. y. The training objective is given as:EQUATIONwhere c θ is a conditioning model to encode y, z t is the noised latent at time t, ϵ is sampled noise, ϵ θ is the denoising network. Based on latent diffusion models, inversion-based diffusion [18] seeks to learn a text embedding v * that accurately describes novel concepts in user provided images. This is achieved by optimizing v * with Eq.1 to iteratively reconstruct the latent code z of user provided images with text prompts y like "an image of S * ", where S * is the placeholder of new concept: As such, it enables image generation w.r.t. target concepts in diverse scenes by using the learned embedding v * to replace the tokenized placeholder S * in text prompts.EQUATIONDisentanglement-based Relation Embedding Learning. To facilitate above inversion technology for relation modeling, two options present: i) directly optimizing embeddings describing interactions (i.e., human-action-object), which risks overfitting with limited samples for long-tailed categories and cannot generalize to novel concepts, and ii) learning action embeddings with diverse images sharing a common action but different objects, which seems feasible but poses significant convergence issues due to the complex content, and the optimization target cannot be fixed to actions but not other unrelated elements. In contrast, drawn from the compositional nature of HOI, we adopt a disentangled solution (i.e., Fig. 2 ) where HOI triplets are broken into human-action and object. Here humanaction is considered to describe the relation between human and object, as action is executed by and strictly adheres to human involved. Then, denoting the text describing human-action as R * , encoded relation embeddings as v Rel * = c θ (R * ), and the latent of one happening HOI in image as z HOI , a relation latent z Rel 0 could be reconstructed (i.e., denoising with ϵ θ from time T to 0) by:EQUATIONHere ( * ) T is the noised version at time T , and z Obj is retrieved by encoding the cropped object from image with provided bounding box annotations. We consider z HOIz Obj is able to describe the human-action component by subtracting the object from human-action-object. Then, we can reconstruct the latent representing the complete HOI image by adding z Obj back to z Rel 0 :EQUATIONwhere P o is the CLIP encoded text embedding of object, and it is combined with the relation embedding v Rel * to generate the prompt that describes the entire HOI image. In this way, with only one learnable relation embedding (i.e., v Rel * ), we build a cycle to generate relation latent z Rel 0 from the HOI image latent z HOI , and subsequently, the original HOI image latent is reconstructed from the generated relation latent. The learning of v Rel * can be supervised without human annotation, but just ensuring the consistency between the original HOI latent and the reconstructed one:EQUATIONwhere all latents are ℓ 2 -normalized for improved training stability [76] . Through such a disentanglement-based relation modeling and cycle-consistency training, the optimization objective become clearer and easier to learn. It enables using same action from different interactions to enhance the comprehension of a relation, and generalizing to new interactions by combining it with other object.Relation-Centric Contrastive Learning. Eq.5 is a pixel-level reconstruction loss which prioritizes aligning low-level cues. We supplement it with a relation-centric contrastive loss to enhance the awareness of high-level semantics. Instead of directly engaging learning with relation latents, we combine them with object latents to form new HOI latents, thus significantly enriching the diversity of samples:EQUATIONwhere x is the anchor sample, x + is the positive sample composed of a different object latent p Obj sharing the same class as z Obj . Conversely, x - Obj and x - Rel are negative samples, with x - Obj composed of a different class object latent n Obj compared to x, and x - Rel composed of any other relation latent n Rel 0 and arbitrary object latent s Obj . The final optimization objective is given as:EQUATIONto optimize v Rel * which involves in reconstructing z Rel 0 . τ = 0.07 is the temperature parameter.Text Prompts Preparation. We harness the captions provided in the MS COCO Caption dataset [77] to generate diverse prompts. Compared to text synthesized by GPT-4, these captions are more precise and closer to real visual scenes as they are annotated by human subjects. The preparation initiates with a filtration where captions not containing pronouns indicating human (e.g., man, woman, boy) or action words are removed. To further enrich the diversity of prompts, given two randomly selected sentences that share the same action, we exchange the clauses following the action word. Prompts are exclusively generated with GPT-4 only when actions or objects not present in COCO Caption. This results in 33,834 text prompts in total. Finally, action words in prompts are replaced with placeholders corresponding to learned relation embeddings, so as to empower the diffusion model with enhanced awareness of relation patterns between human and object during generation.Image and Annotation Generation. Denoting text prompts as P = {P 1 , • • •, P N }, we aim to construct a dataset X = {(I 1 , A 1 ), • • •, (I N , A N )} where I i ∈ R H×W×3 represents the synthesized image and A i = {B h i , B o i , C o i , C a i }P i = c θ (P i ) ∈ R d ,where the placeholder string is directly replaced with relation embedding v Rel * . Then, a random sampled noise tensor z T ∈ R h×w×d is iteratively denoised to yield a new latent z 0 . I i is generated by a single pass through D, i.e., I i = D(z 0 ). For the generation of A i , C o i and C a i can be easily determined by referring to the action and object words in P i , while B h i and B o i are derived from the cross-attention maps computed within the U-shape denoising network ϵ θ . Specifically, to effectively tackle various input modalities, ϵ θ is equipped with cross-attention mechanisms in each layer to inject P i into z conforming to the similarity between them. For the l-th layer at the last denoising step 0, the cross-attention map is computed as:M l i,0 = softmax(z 0 •P ⊤ i / √ d) ∈ R h×w .According to prior work [17, 78] , here M l i,0 signifies the correspondence between text prompt P i and regions in generated image. Thus, we explicitly concatenate words describing human and object with P i (i.e., [P i ; word human ; word object ]), resulting in a new text embedding Pi ∈ R d×3 and corresponding cross-attention maps M l i,0 ∈ R h×w×3 where the last two items along the third dimension channel are probability maps of human and object. Finally, we leverage the implementation in weakly supervised object localization [79] to outline bounding boxes from these probability maps.While prior studies [3, [48] [49] [50] have investigated knowledge transfer from visual-linguistic models such as CLIP, they utilize visual knowledge solely during training. The prediction relies on a confined set of CLIP encoded word embeddings, which leads to limited knowledge transfer and rigid inference unresponsive to image content. In contrast, we propose directly leveraging diffusion models as the feature extractor and build HOI detector on this basis. Moreover, given the conditioning property of diffusion model, relation embeddings can serve as text prompts to guide the retrieval of interaction-relevant visual cues from images, further benefiting HOI detection.Pioneering studies [17, 78, 80] have empirically demonstrated that the output of frozen text-to-image diffusion models possesses rich visual features to tackle complex perception tasks. Next we illustrate how to build a HOI detector on this basis. As shown in Fig. 3 , our method is a one-stage solution composed of: a visual encoder with diffusion models serving as the backbone, and a HOI decoder consisting of two parallel decoders for instance and interaction detection. For the visual encoder, given an image I, it is encoded into latent space with the encoder E of a pre-trained generator (e.g., VQGAN): z = E(I). Then, z is fed into ϵ θ through a single noise-free forward pass to derive text conditioned features: , and considers those at the same index (i.e., i = j) as a pair to initialize queries q r for the interaction decoder D HOI . In fact, the HOI decoder can be replaced with any other one-stage models. We do not claim the detector architecture as the contribution, but focus on how to derive HOI-relevant feature to assist in HOI detection.ϵ θ (z, T, c θ (y)) → {z l T } 4 l=1 . AllRelation-Inspired HOI Detection. As the relation embeddings are optimized towards modeling the interactions between human and object, we use them as conditions to inspire the extraction of HOI-relevant cues. This can eliminate the potential domain gap between general-purpose diffusion models and the downstream HOI detection task. Specifically, all feasible HOI phrases (e.g., "human feed horse") are encoded with CLIP text encoder into embedding space and concatenated together, with the human-action component replaced with learned relation embeddings (e.g., "R * horse"). This results in HOI prompts P r ∈ R Nr×d l which further participates into the cross-attention in ϵ θ via:EQUATIONwhere K Pr and V Pr are key and value embeddings projected from P r . As seen, P r contributes to: i) encourage the denoising network ϵ θ to extract visual features z l T w.r.t. HOI prompts, and ii) guide the derivative of cross-attention maps M l r in response to ongoing interactions in I. The final interaction maps are computed as the average value of {M l r } 4 l=1 . we also derive cross-attention maps for human M h and object M o in a similar way as M r , by without update to z l T . Then, these cross-attention maps are used to initialize queries from the aggregated visual feature z ′ T via mask pooling:EQUATION)Note we conduct Hungarian matching between qk r and qi o + qj h , so as to arrange HOI, and combined human-object queries that are most similar to the same index in their respective query lists. Following [17] , the classification for interaction and instance are jointly supervised by:EQUATIONwhere y r and y o are ground truth for interaction and object categories, qr and qo are queries after decoding through D HOI and D Ins , CE and FFN denote the cross entropy loss and feed-forward network. Beyond the score delivered by conventional linear classifier (i.e., softmax(FFN( qo ))), here softmax( q • P /τ ) with learnable parameters τ r and τ o computes the similarity between decoded queries and conditioning prompts, thereby facilitating the recognition for unseen categories.Online Update for Relation Embedding. To enable the continual evolution of relation embeddings v Rel * throughout the supervised HOI detection learning, an additional loss considering the compositional nature of HOI is devised. Specifically, we concatenate all N a relation embeddings into a new prompt P a , from which a set of relation query embeddings qa can be initialized in the same way as qo (c.f., Eq.9). In addition, another set of embeddings describing relations can be derived from qr and qo by: qa = qrqo . The goal is to align qa directly derived from visual features with relation embeddings as conditions, and qa computed from interaction and object queries after decoding:EQUATION)Here L Rel solely optimizes v Rel * to render a mutual boost between HOI detection and relation embedding learning. Concretely, enhanced relation embeddings inspires improved HOI feature discovery, and in turn, the more precise query decoding benefits the update of relation embeddings. Network Architecture. DIFFUSIONHOI is built upon Stable Diffusion v1.5 with xFormers [82] installed. The denoising UNet ϵ θ receives input latents at a downsampling factor of 1/8, with four encoder blocks output feature at a size of 1/2 l+3 where l is the block index. For the final visual feature z ′ after FPN aggregation, it is interpolated to a size of 1/32 and then projected to 256 channels to enhance computing efficiency. Both D HOI and D Ins consist of six Transformer decoding layers with hidden dimension of 768. The query number N q is uniformly set to 64 for both D HOI and D Ins .Training Objective. The inversion-based HOI modeling is jointly optimized by two embedding learning losses: L Inversion = L Consistency + λ 1 L Contrastive , where λ 1 ∈ [0, 0.2] is scheduled following a cosine annealing policy. For HOI detection learning, we follow DETR [39] to match predictions and ground truths with Hungarian algorithm. Denoting the bounding box detection loss as L Det , the final training objective is given as: L = L HOI + L Ins + L Det + λ 2 L Rel where λ 2 is fixed to 0.5. 
experimental setup: 
 Datasets. We conduct extensive experiments on three datasets.• HICO-DET [20] is a large-scale HOI detection benchmark with 38,118/9,658 images for training/testing, respectively. This dataset includes 80 object categories as in MS-COCO [96] and 117 action categories, formulating a rich vocabulary of 600 human-object interactions in total. • V-COCO [21] is a curated subset of MS-COCO [96] including 2,533/2,867/4,946 images in train/val/ test sets. It also contains 80 object categories from MS-COCO [96] and a much smaller set of 29 action classes, resulting in a total of 263 human-object interactions. • SWiG-HOI [22] is assembled from SWiG [97] and DOH [98] with about 45,000/14,000 for training/testing. This dataset covers 406 human actions and 1,000 object categories.Zero-Shot HOI Discovery. In accordance with prior research [3, 19, [99] [100] [101] [102] , the zero-shot HOI discovery on HICO-DET [20] uses four setups: Rare First Unseen Combination (RF-UC), Non-rare First Unseen Combination (NF-UC), Unseen Verb (UV), and Unseen Object (UO). The RF-UC and NF-UC configurations excluded the 120 most frequent/infrequent interaction categories from the training sets for testing purposes only. The UV and UO setups reserve 20 verb classes and 12 object classes never encountered during training for testing. For SWiG-HOI [22] , the test set includes approximately 5,500 interactions, with around 1,800 of them not present in the training set. Evaluation Metric. Following conventions [3, 41, 42] , we adopt mAP as metrics. For HICO-DET, we report performance according to Default and Known Object two setups. The former computes mAP across all testing images, while the latter is tailored for each object class. For each setup, the scores are reported in Full/Rare/Non-Rare three types. For V-COCO, we evaluate the performance under scenario 1 (S1) which contains all 29 actions and scenario 2 (S2) which excludes 4 actions interact with no objects. For zero-shot setup, the evaluation is divided into Seen/Unseen/Full three sets for HICO-DET, and Non-Rare/Rare/Unseen/Full four sets for SWiG-HOI.Training and Testing. The diffusion model and CLIP text encoder are kept frozen during training. For inversion-based HOI modeling, the only learnable parameters are relation embeddings, which are updated for 40,000 steps using images sampled from HICO-DET. Following [18] , we employ a base learning rate of 8e -2 with a batch size of 32. For HOI detection learning, we train the interaction decoder D Ins and object decoder D HOI for 60 epochs with a base learning rate of 1e -4 and batch size of 16, using both synthesized data and the target dataset. Subsequently, the model is trained only on the target dataset for an additional 30 epochs with a base learning rate of 1e -5 . During inference, no data augmentation is used to ensure fair comparison. Following [3, 103] , the inputs are resized to maximum of 1,333 pixels on long sides, and the shortest sides falls between 480 and 800 pixels.Reproducibility. DIFFUSIONHOI is implemented in PyTorch and trained on 8 Tesla A40 GPUs with 48GB memory per card.Regular Setup. We first compare DIFFUSIONHOI with top-leading solutions on HICO-DET [20] and V-COCO [21] under the regular setup. As shown in Table 1 , for HICO-DET, our method achieves the best performance on both Default and Known Object setups. Notably, with the encoder of VQGAN as the backbone, it surpasses the previous SOTA, RemLR [93] , which employs a similar level backbone (i.e., ResNet-50) by 1.19% and 2.64% on the Full categories. Benefited from synthesized data and comprehensive knowledge transfer from diffusion models, the performance on Rare categories improves significantly, achieving higher scores than on Non-Rare categories for the first time. Finally, with a more powerful VL model (i.e., Stable unCLIP [11] ) and backbone (i.e., ViT-L), the performance is boosted to 42.54% under the Default setup, surpassing nearly all existing work by a considerable margin. Please note that PViC [89] with Swin-L as the backbone leverages H-Deform-DETR [95] as the detector which achieves 48.7 mAP on MS COCO [96] by running merely 12 epochs, significantly higher than DETR [39] which achieves 36.2 mAP by running 50 epochs.Zero-Shot Setup. Next we investigate the effectiveness of DIFFUSIONHOI under the zero-shot generalization setup. As shown in Table 2 , our method yields remarkable performance across all four setups on HICO-DET. In particular, it surpasses the previous SOTA (i.e., HOICLIP [19] ) by 2.90% under the RF-UC setup. This setup emphasizes compositional generalization which requires models to comprehend new types of interactions using known actions and objects. It aligns well with the strengths of text-to-image diffusion models to generate images conditioned on compositionally organized textual descriptions. Moreover, due to the effective knowledge transfer, DIFFUSIONHOI also achieves satisfactory improvement under the UV and UO setups which focus on the recognition of novel actions and objects. Table 3 further confirms the exceptional ability of our method, showing 5.97%/8.23% mAP improvements over CMD-SE [104] under Rare and Unseen two categories.Model Efficiency. We compare the trainable parameter number and inference time in Table 4 . As seen, DIFFUSIONHOI demonstrates significantly fewer trainable parameters compared to the one-stage counterparts. This is attributed to our inversion-based HOI modeling, which avoids finetuning diffusion models like previous work [78] , while effectively capturing task-specific properties. Regarding inference speed, even with stable diffusion for feature extraction, our method still achieves 9.49 FPS, a rate similar to two-stage models. This is due to the inference involving only one single forward pass, and the downsampling factor of stable diffusion from 1/8 to 1/64 is smaller than conventional backbones typically from 1/4 to 1/32. Moreover, thank to the flourishing community of stable diffusion, a variety of optimized inference solutions have emerged. By running at fp16 precision and using traced UNet, the FPS increases to 24.77, surpassing most one-stage methods.Key Component Analysis. We first examine the essential components of DIFFUSIONHOI in Table 5 . Here BASELINE denotes HOI detector built upon stable diffusion without text prompting. Through jointly training with the synthesized data, both Default and RF-UC setups observe notable improvements (e.g., up to 2.25% and 3.32% on Full categories ). This verifies the effectiveness of our relation-driven HOI image generation strategy. in addition, after imposing relation embedding to prompt the feature extraction and HOI detection processes, the performance boosts to 36.45% and 34.25% under two setups. Finally, after combining these two core components together, our DIFFUSIONHOI delivers consistent improvements and sets new SOTA across all setups.Conditioning Input. To assess the effectiveness of learned relational embeddings, we present the experimental results using different conditional inputs to stimulate HOI detection in Table 6 . As seen, though action words offer limited improvement, they are far surpassed by relation embeddings which enables HOI-oriented feature extraction and enhance query initialization through cross-attention.Relation Embedding Learning. Next we probe the impact of different strategies for relation embedding learning. The results regarding relation-inspired HOI detection are summarized in Table 7 . It can be observed that textual inversion, which directly uses different images sharing the same action for relation embedding learning, is inferior to our cycle-consistency learning strategy that considers the disentanglement nature of HOI interactions. On this basis, the relation-centric contrastive learning and online update strategies consistently bring improvement in both setups.Prompt for Dataset Generation. Finally we study the impact of data synthesized by different types of textual prompts in Table 8 . As observed, data generated with purely textual description using plain action words like "The man at bat readies to swing at the pitch" gives negative improvement over baseline. The potentially indicates that diffusion models cannot understand the relations between human-object pairs and generate meaningful images when provided with straightforward textual description. In contrast, through relation modeling, data generated with relation embeddings to replace the plain action words provides high-quality samples for the training of HOI detectors.Analysis on Training Cost. For our inversion-based HOI modeling to learn relation-centric embeddings, unlike the original textual inversion technology that learns text embeddings within the image space, we optimize relation embeddings within the latent space by reconstructing interaction features. This lead to reduced training costs. Consequently, the 117 relation embeddings in HICO-DET [20] can be learned within 5.7 hours (23 minutes per relation embedding) which is more efficient than textual inversion (i.e., 32 minutes per embedding). For the main training of HOI detection on HICO-DET, since our method utilizes significantly fewer trainable parameters compared to existing work (e.g., 27.6M v.s. 50.4M for STIP [85] , 41.9M for QPIC [42] , and 42.8M for GEN-VLKT [3] in Table 4 ), the training process can be completed in just 11.5 hours. The comparison of the whole training time with some representative work is summarized in Table 9 , with all experiments conducted on 8 Tesla A40 cards. It can be observed that DIFFUSIONHOI requires less training time than most existing work. 
conclusion: 
 We present DIFFUSIONHOI, a new HOI detector built upon diffusion models. By explicitly modeling the relations between humans and objects in an inversion-based manner, we enable effective knowledge transfer from diffusion models while adapting unique characteristics of the HOI detection task. This is achieved in two aspects: i) relation-driven image generation using diffusion models to enrich the training set with more HOI-oriented samples, and ii) relation-inspired HOI detection with learned relation embeddings as prompts to retrieve task-specific features from images, thereby enhancing the recognition of ongoing interactions. Extensive experiments demonstrate that DIFFUSIONHOI excels in both regular or zero-shot setups and sets new SOTAs. We believe this work provides insights to unleash the power of diffusion models for downstream visual perception tasks in an efficient manner. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The border impacts is provided in Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The proposed method uses pre-trained models. This proposed methods is safe under the safeguards of adopted pre-trained models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.1 LimitationOne potential limitation of our method is that it does not run as quickly as previous one-stage methods due to the adoption of diffusion models. A detailed comparison is provided in Table 4 .As shown, DIFFUSIONHOI achieves inference speeds similar to two-stage methods. However, currently there is a trend towards leveraging large-scale pre-trained models to assist in various perception tasks. These models offer substantial enhancements in performance and accuracy. For instance, after utilizing diffusion models, our method has demonstrated an 8.23% improvement on unseen categories in the SWiG-HOI benchmark. Though it is inevitable that these advancements introduce additional computational overhead, the trade-off is generally considered acceptable given the significant improvements in task performance and the ability to generalize better to previously unseen scenarios. Therefore, an important consideration in future research is the balance between computational cost and the enhanced capabilities provided by large-scale pre-trained models. In this work, we implement acceleration technology for Stable Diffusion which successfully boosts the inference speed of DIFFUSIONHOI to 24.77 FPS, surpassing most one-stage methods.As shown in Fig. S1 , we found that failure cases primarily manifest in the following scenarios: i) scenes featuring only partial human bodies, such as arm or leg, which introduces challenges for person detection; and ii) chaotic scenes teeming with people, which causes occlusion and difficulties in identifying interactions. Despite these challenges, DIFFUSIONHOI has shown remarkable improvement over existing approaches. Additionally, the patterns of these failure cases provide valuable insights for future research.On the positive side, this work proposes a more powerful solution for recognizing complex interactions between humans and objects in a scene. It is particularly effective for few-and zero-shot setups that are common in real-world scenarios. This advancement can significantly contribute to a range of applications, such as healthcare and assistive systems, smart homes and IoT (Internet of Things), security, and more. However, there are also potential negative aspects. Our method carries the risk of being used for continuous monitoring, which could raise concerns over intrusive surveillance and the unauthorized collection of personal data. Additionally, the inversion-based modeling strategy for adjusting diffusion models could be misused to create harmful or false information about individuals. Hence, it is essential to rigorously consider ethical standards and legal regulations to address privacy concerns, so as to avoid potential negative societal impacts.We present a more detailed comparison of DIFFUSIONHOI with other methods in Table S1 . As demonstrated, DIFFUSIONHOI consistently achieves state-of-the-art performance.We present the qualitative results of relation-driven sample generation in Fig. S2 . It can be observed that the synthesized images exhibit realistic shapes and textures for individual instances, as well as coherent structure and layout for entire scenes. Additionally, the generated images accurately depict the interactions between humans and objects, demonstrating the effectiveness of our proposed inversion-based HOI modeling strategies. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper introduces DIFFUSIONHOI, a novel method for Human-Object Interaction (HOI) detection utilizing text-to-image diffusion models to improve visual cue recognition and compositional reasoning. The approach demonstrates significant performance improvements across multiple datasets, including HICO-DET and V-COCO. Despite concerns about training efficiency, parameter count, and technical complexity, the authors provided comprehensive analyses and additional comparisons, reinforcing the method's effectiveness. Reviewers acknowledged the innovative integration of diffusion models and clear presentation, although they noted the architecture's complexity and suggested further exploration of failure cases. Overall, the paper is recommended for acceptance.

Reviews: review: paper_summary: paper_summary: This paper tackles the human-object interaction (HOI) detection task.

It aims to utilize the feature of generative models like diffusion models to help human-object interaction classification.

More specifically, it utilize the inversion process in the diffusion model to learn the embedding for human-object interaction.

The proposed method obtains obvious improvement over existing methods.
strength_weakness: strength_weakness: - The cost of training with the inversion process. As the inversion process is computationally costly and slow, it could affect the training speed. I would like to see more clarification or analysis over the cost of the proposed method.
- The proposed model does not achieves SOTA as it claims. For example, it falls behind PViC (iccv23) with swin-L backbone by a large margin. As the proposed method have a ViT-L backbone, I think it is not fair to not mentioning PViC-swin-L and some other methods. Besides, PViC utilize no external knowledge like CLIP or SD while this method does.
- The general technical design is complicated. This is not a big problem and does not affect my rating.
questions: questions: - The analysis of the cost.

I will look into the rebuttal and adjust my rating accordingly.
limitations: limitations: Discussion over the limitation is provided.
paper_summary: paper_summary: This paper introduces DIFFUSIONHOI, a new HOI detector leveraging text-to-image diffusion models. Unlike previous one-stage or two-stage models, diffusion models excel at discerning mid/low-level visual concepts as generative models and possess strong compositionality to handle novel concepts expressed in text inputs. To steer the focus of diffusion models from instance generation to the relationships between humans and objects, this paper exploits textual inversion and devises a human-object relation inversion strategy grounded in the disentanglement of HOI. Furthermore, to transfer extensive knowledge from large-scale diffusion models to assist in recognizing interactions, the paper leverages both text-prompted image generation and conditioned feature extraction capabilities of diffusion models. By embracing text-to-image diffusion models and facilitating relation-driven image generation and prompting, this method demonstrates superior performance.
strength_weakness: strength_weakness: 1. I am curious about the number of parameters in the VQGAN used as the backbone of DiffusionHOI. According to Table 1, the authors use ViT-L as their backbone, which appears to be significantly larger compared to previous works, especially those using the Res50 network. Could the authors provide more results using a smaller backbone, such as ViT-B, to offer a clearer and fairer comparison between DiffusionHOI and previous state-of-the-art models?
2. Another concern is that the improvement primarily stems from the additional knowledge provided by the powerful diffusion model. Could this be considered unfair when comparing it to previous works that do not have this significant benefit? In this case, how does this approach differ from simply adding more data for training? As I am familiar with HOI but not as familiar with diffusion models, could the authors provide more details about the data scale and parameter scale of the diffusion model used in this paper? Additionally, I look forward to feedback from other reviewers on this matter.
questions: questions: Please refer to weakness.
limitations: limitations: Please refer to weakness.
paper_summary: paper_summary: This paper introduces DIFFUSIONHOI, a novel Human-Object Interaction (HOI) detector that utilizes text-to-image diffusion models for HOI detection. It efficiently focuses on complex relationships between objects, providing a strong basis for HOI modeling. The relation-driven approach enhances image generation capabilities for HOI detection, enriching training samples for rare interactions. Additionally, it improves detector flexibility and accuracy, achieving good performance on HICO-DET and V-COCO benchmarks.
strength_weakness: strength_weakness: - Despite having fewer trainable parameters overall, DIFFUSIONHOI has a large total parameter count, which raises concerns about unfair comparisons with existing methods.
- The authors claim to set a new state-of-the-art on HICO-DET, but the reviewer found that there are better models available, as also mentioned in the paper [79].
- Quantitative results and analysis of failure cases are needed.

[79] Frederic Z Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould. Exploring predicate visual context in detecting of human-object interactions. In ICCV, 2023
questions: questions: N/A
limitations: limitations: Yes

Comments: comment: Thanks for the authors' response. The rebuttal has addressed most of my concerns. I keep my positive score.
Hi, All,
Please make the best use of the discussion period to learn better from each other's ideas. Note that the discussion ends on Aug 13. Thank you.

Best,
AC