Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models
Authors: Liulei Li
Wenguan Wang
Yi Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: HOI detection
relation understanding
diffusion models
Primary area: machine_vision

Abstract: Prevalent human-object interaction (HOI) detection approaches typically leverage large-scale visual-linguistic models to help recognize events involving humans and objects. Though promising, models trained via contrastive learning on text-image pairs often neglect mid/low-level visual cues and struggle at compositional reasoning. In response, we introduce DIFFUSIONHOI, a new HOI detector shedding light on text-to-image diffusion models. Unlike the aforementioned models, diffusion models excel in discerning mid/low-level visual concepts as generative models, and possess strong compositionality to handle novel concepts expressed in text inputs. Considering diffusion models usually emphasize instance objects, we first devise an inversion-based strategy to learn the expression of relation patterns between humans and objects in embedding space. These learned relation embeddings then serve as textual prompts, to steer diffusion models generate images that depict specific interactions, and extract HOI-relevant cues from images without heavy finetuning. Benefited from above, DIFFUSIONHOI achieves SOTA performance on three datasets under both regular and zero-shot setups.

Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models
Liulei Li', Wenguan Wang?;
Yi Yang²
RcLLER, AAlI, University of Technology Sydney
CCAL Zhejiang University
Abstract
Prevalent human-object interaction (HOI) detection approaches typically leverage lurge-scale iisual-linguiste models to help recognize events involving humans and objects. Though promising, models trained via contrastive learning on text-image pairs often neglect mid/low-level visual cues and struggle at compositional reason- ing. In response, we introduce DIFFUSIONHOL, a new HOI detector shedding light on text-to-iman models excel in discerning mid/low-level visual concepts as generative models, and possess strong compositionality to handle novel concepts expressed in text inputs. Considering diffusion models usually emphasize instance objects, we first devise inversion-based strategy to learn the expression of relation patterns between humans and objects embedding space. These learned relation embeddings then serve textual prompts, steer diffusion models generate images that depict specific interactions, and extract HOI-relevant cues from images without heavy fine-tuning. Benefited from above, DIFFUSIONHO1 achieves SOTA performance on three datasets under both regular and zero-shot setups.
diffusion models. Unlike the aforementioned models, diffusion
Introduction
As a crucial topic in the field visual scene understanding, human-object interaction (HOI) detection demands not only inferring the semantics and locations of entities but also should comprehend the ongoing events happening between them[1, 2]. Given the complexity and diversity of human activities in object-rich realistic scenes, this task presents challenges in long-tailed distributions and zero-shot discovery [3]. A set of studies seek to tackle these two issues by leveraging large-scale risual-linguise models (e.g., CL]P(4]) which show strong generalization ability dozens of tasks. Though strides made, it has been observed that models trained by aligning high-level text-image semantics face diffi- culties in discerning spatial locations [5], and struggle compositionality which is a fundamental ability for human to capture new concepts by combining known parts. In fact, both middle-level visual cues (e.g., spatial relation) and compositionality are essential facets for HOI detection. The former can help deduce feasible interactions according to locations between instances, while compositionality contributes significantly zero-shot generalization. For example, we can easily understand human- hold-horse by composing homan-hold-dog and class horse that have encountered previously.
In contrast, the eoxt-to-image diffusion models [7-14] also pre-trained large-scale image-text text pairs, are demonstrating superior capabilities outperforming models like CLIP. Concretely, they are able to generate diverse high-quality images conditioned textual inputs, showing proficiency in understanding high-level semantics [15, 16]. In addition, the generated images convey reasonable shape, texture, layout, and structure, indicating the comprehension in mid/low-level visual concepts as generative models| [1]]. More importantly, the descriptions are typically organized in a compositional
"Corresponding Author: Wenguan Wang.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A man R. motorcycle on the road
Detector
Fide
Feature Matching
cRing
Denoising UNet
CLI Text
Stand
Enccoder
hronnn-jargg janp-n------oooo
51.0
Juunnn 2024 colse
humann Ribiicyle
Figure 1: Existing solutions utilize mere linguistic knowledge (a). Our solution utilizes both text- prommpt image generation (b) and conditioned feature extraction (c) abilities of diffusion models for knowledge transfer.
manner, with phrases such as "happy", "near to objects like "a dog". This suggests that diffusion models inherently possess compositionality. to systematically adapt to newly encountered user requirements by composing known visual concepts.
"near a bridge", or "hugged by man" continually appended
The above analysis motivates us to explore diffusion models for HOI detection. Nonetheless, fully unlock the potential of diffusion models and accommodate the unique characteristic of HOI detection task, the following questions naturally arise: 1 With diffusion models typically emphasizing instance generation, how to steer it to prioritize the relationships between humans and objects? How to transfer the extensive knowledge obtained from large-scale pre-training in diffusion models assist the recognition of interactions? To address O, we harness textual inversion [18] which conceptualizes user-provided object by inverting it text embedding. However, this method focuses solely on instance objects. To facilitate a smooth shift from object-centric to relation-centle modeling, we de- vise a human-object relation inversion strategy grounded in the disentanglement of HOI. Concretely, given the HOI latent describing human-act www.aaaciiiioeeeeee we build vycle-consistence objective to reconstruct it from a intermediate relation latent derived from the original HOI latent. This reconstruc- tion process is guided by set of learnable relation embeddings as text prompts, for which we use the placeholder R. to denote the textual form before encoded into embedding space. These relation embeddings further involves in a relation-centee
contrastive learning to enhance the awareness of high-level relational semantics. To answer @ we leverage both the text promoted image generation and conditioned feature extraction abilities of diffusion models. We realize relation-driven image generation by compositionally organizing R. with other linguistic elements to formulate new text prompts (Fig. (b)). This allows for the generation of novel interactions with unseen objects, and extends the training set for HOI detectors. Morcover, we directly utilize diffusion models as backbone to extract HOI-relevant features conditioned on R., (Fig. I())). After single noise-free forward step, features distinct for each interaction can be obtained. Finally, to establish a loop for mutual boosting between above relation-insppe iittiionn--p-eedddddeee HOI detection and relation modeling, we devise online update strategy to facilitate the continual evolving of relation embeddings during HOI detection learning.
Benefited from controllable image generation and knowledge transfer from diffusion models, our method named DIFFUSIONHOI enjoys several appealing advantages: First, it steers diffusion models to focus on complex relationships rather than single objects in an efficient way. This offers a robust foundation for HOI modeling. Second, from the perspective of relation-driven, unlocks the image generation power of diffusion models tailored for the HOI detection task. This enriches the pool of training samples, particularly for bong-tailed/unseeee interaction classes. Third, the relation- iaation-inspired prompting improves both the flexibility and accuracy of HOI detectors. It adapts each individual image to extract action or object related cues, while CLIP-based methods[3, 19] produce action/object features merely from texts (i.e., Fig. ((a)), remaining static and unresponsive to image content.
By embracing text-to-image diffusion models well as facilitating celation-drvver image generation and prompting, our method demonstrates superior performance. It surpasses all top-leading solutions on HICO-DET [20] and '-Coco(21), and sets new state-of-the-arts. In addition, it yields up to 6.43% mAP improvements on SWiG-HOI [22] under the zero-shot HOI discovery setup. These promising performance evidences the great potential of integrating diffusion models for visual relation understanding. We hope this work could foster the broader exploration of large-scale pre-trained diffusion models on more computer vision tasks beyond mere image generation.
2 Related Work
Human-Object Interaction Detection. According to the architecture design of networks, existing solutions for HOI detection can be broadly categorized into two groups: one-stage and two-stage.

---

The one-stage methods[23 typically employ a multi-task learning pipeline that jointly undertake the tasks of numan-object detection and interaction classification in an end-to-end manner, therefore distinguished by fast inference. In contrast, two-stage methods [27-37] first detect entities with off- the-shelf detectors such as Faster R-CNN[38) and the predict the dense relationships among possible human-object pairs. This paradigm effectively disentangles the HOI detection process and results in improved performance. Inspired by DETR [39], recent advancements shift to adopt Transformer- based architectures [3, 40-46]. Several studies [3, 47-51] also supplement the Transformer-based HOI detectors with large-scale visual-linguistic models like CLIP [4] or visual knowledge [52, 53] to conduct logic-induced reasoning [54]. However, these models focus solely on aligning high-level semantics and overlooking mid/low-level visual cues. To tackle this, we redirect our attention to diffusion models, which perfectly address the aforementioned challenges and possess the capacity to handle previously unseen concepts through their strong compositionality.
Controllable Image Generation. To facilitate customized image generation with respect to prede- fined class, attribute, text or image[55], various approaches based on GANs[56] have been proposed. For instance, [57, 58] develop a photo realistic hairstyle transfer method through latent space optimiza- tion. However, these methods typically show limited diversity when compared to Aikelihood-basee [5dels[59]. In response, diffusion models [60-62] have emerged that not only demonstrate remark- able synthesis quality but also offer enhanced controllability. The core idea behind is to transform simple and known distribution (e.g., Gaussian) into the desired data distribution. These models have proven to be highly effective in various conditional scenarios. According to the conditional targets, the prevalent work can be grouped into class-driven [63, 64] text-driven [7, 8], exemplar image-driven [65, 66), etc.. These advances have found application in a wide range of domains such as super resolution [66, 67], image editing [13, 68]. Recently, new approach achieves guided image generation by learning a single word embedding through a frozen text-to-image model to properly describe the desired target objects[18] Take inspiration from it, we achieve relation-driven image generation by extending such object-centric concept modeling approach to relation-centric.
Knowledge Transfer from Diffusion Models. In light of the notable success achieved by diffusion models in applications, there is a growing interest in transferring knowledge acquired from large-scale pre-traini ree-training various tasks [17, 69-75]. For example, given the limited availability of data for constructing NeRFs and the unprecedented generalizability of diffusion models, rescarchers are motivated to explore generating 3D NeRFs via 2D text-to-image diffusion model using diverse input text[69-71]. More recently, a notable trend has emerged where efforts are dedicated towards learning semantic representations from diffusion models by extracting intermediate feature maps. It finds diverse application in image segmentation[17] semaatiii semantic correspondence learning[ [72-74], and general representation learning [75]. In this work, we extensively harness both the image generation and semantic representation abilities of diffusion models, by using relation-centric embeddings to control the generation and prompt semantic extraction from images with respect to specific interactions.
3 Methodology
3.1 Preliminary: Textual Inversion
Latent diffusion models [14] represent evolution of diffusion models which offer significant enhancements in both computational and memory efficiency by executing denosing in the latent space. It comprises two primary components. The first is pre- pre-trained generator equipped with an encoder E to map the input image x into a latent vector 2 = (z), from which the original data can be reconstructed via decoder D by 2 = D(z) latent codes x conditioned on user guidance y which can be text, image, etc. The latent codes then serve inputs to D for image generation w.r.f. y. The training objective is given as:
z. The second is diffusion model to generate
LLDM := (((,))) ||e - c0(2,, t, co
(1)
where co is a conditioning model to encode y, 2, is the noised latent at time t, is sampled noise, co is the denoising network. Based on latent diffusion models, inversion- based diffusion[188 seeks to learn a text embedding v, that accurately describes novel concepts in user provided images. This is achieved by optimizing v, with Eq. iteratively reconstruct the latent code of user provided images with text prompts y like "an image of S.", where S, is the placeholder of new concept:
U- arg min (E (x)...... ||e ea Ea t, co(v))
(2)
3

---

R.
Denoisiig UNet Eg
Consistensy
Dessising UNet 6
[R.:P.).
H01
Obi
Rel
Figure (Left) Disentanglement based coce--oonssttecyy learning. (Right) Relation-centie centrrc contrastive learning.
As such, it enables image generation w.r.t. target concepts diverse scenes by using the learned
embedding u. to replace the tokenized placeholder S. in text prompts.
3.2 Inversion-Based HOI Modeling
Disentangleeentt-asse Relation Embedding Learning. To facilitate above inversion technology for relation modeling, two options present: i) directly optimizing embeddings describing interactions (i.e., moman--cttionobjeeee
which risks overfitting with limited samples for long-tailed categories and cannot generalize to novel concepts, and ii) learning action embeddings with diverse images sharing a common action but different objects, which seems feasible but poses significant convergence issues due the complex content, and the optimization target cannot be fixed actions but not other unrelated elements. In contrast, drawn from the compositional nature of HOI, we adopt a disentangled broken into human-actior and object. Here human-
solution (i.e., Fig. 2) where HOI triplets action considered to describe the relation between human and object, as action o is executed by and strictly adheres human involved. Then, denoting the text describing human-action
R,, encoded relation embeddings as v.t = co (R.),, and the latent of one happening HOI in image as H0I relation latent z0 could be reconstructed (i.e., denoising with eo from time T to 0) by:
c HOI = )T, T, v/ Rel)
Rel
20
Here (*)r the noised version time T, and 260 retrieved by encoding the cropped object from image with provided bounding box annotations. We consider 101 2 is able to describe the human-action component by subtracting the object from human-act wat.aaaciiino.ec......... Then, we can reconstruct the latent representing the complete HOI image by adding x Obj back to Rel.
ca((= Rel + 2 20) T, [v Rel, P.l) + 20 HOI
where P. is the CLIP encoded text embedding of object, and it is combined with the relation embedding B to generate the prompt that describes the entire HOI image. this way, with only one learnable relation embedding (i.e., dRRa),, build cycle to generate relation latent 2 from the HOI image latent x H01 and subsequently, the original HOI image latent is reconstructed from the generated relation latent. The learning of R can be supervised without human annotation, but just ensuring the consistency between the original HOI latent and the reconstructed one:
Coonsistency = |1 (2(2))) )
(5)
where all latents are Ez-normalized for improved training stability [76]. Through such a disentan- glement-basee relation modeling and sycle-consistency training, the optimization objective become clearer and easier to learn. It enables using same action from different interactions to enhance the comprehension of relation, and generalizing to new interactions by combining with other object
Relation-Centric Contrastive Learning. Eq.5 is a pixel-level reconstruction loss which prioritizes aligning low-level cues. We supplement it with relation-centric contrastive loss enhance the awareness of high-level semantics. Instead of directly engaging learning with relation latents, we combine them with object latents to form new HOI latents, thus significantly enriching the diversity of samples:
x 20
Rel
z
Ohj
x
Rel + Obj =
(6)
Obj
Rel
n
Obj
0, + $
where a is the anchor sample, t the positive sample composed of different object latent p Obj sharing the same class as z Obj Conversely, oozj and ØRel negative samples, with Zooj composed

---

of different class object latent n thi compared to and Ret composed of any other relation latent Rel and arbitrary object latent sbi The final optimization objective is given as:
no
exp(æ : xx///////
Coontrastive log exp(x x /rt/ + Σn exp(x Toby/ /T) + E. exp(x =Ra/T)'
to optimize uf which involves in reconstructing Rel T 0.07 is the temperature parameter.
3.3 Relation-Driven Sample Generation
Text Prompts Preparation. We harness the captions provided in the MS COCO Caption dataset [77] to generate diverse prompts. Compared to text synthesized by GPT-4, these captions are more precise and closer to real visual scenes as they are annotated by human subjects. The preparation initiates with filtration where captions not containing pronouns indicating human (e.g., man, woman, bay) or action words are removed. To further enrich the diversity of prompts, given two randomly selected sentences that share the same action on, we exchange the clauses following the action word. Prompts are exclusively generated with GPT-4 only when actions or objects not present in COCO Caption. This results in 33,834 text prompts in total. Finally, action words in prompts are replaced with placeholders corresponding learned relation embeddings, so as to empower the diffusion model with enhanced awareness of relation patterns between human and object during generation.
Image and Annotation Generation. Denoting text prompts as P (PI,..., PN}, we aim to (IN,AN)) where I₁ E R represents the synthesized image and A, (B% B3; C, C.) } is the pseudo annotation containing bounding boxes Bhth for human, BE for object, and class labels C. for object, C. for action. For the generation of I, the text prompts P, is first encoded by CLIP text encoder to obtain the conditioning vector P. cg= E ERd where the placeholder string is directly replaced with relation embedding Rel Then, a random sampled noise tensor 27 E Rxxwxd is iteratively denoised to yield a new latent žg. I, is generated by single pass through DD i.e., I = D(zo). For the generation of Ai, C₁ and C. can be easily determined by referring to the action and object words in P. while Bi and B3 are derived from the coos-attention maps computed within the U-shape denoising network co. Specifically, to effectively tackle various input modalities, co is equipped with cross-attentin mechanisms in each layer to inject P; into z conforming to the similarity between them. For the 1-th layer at the last denoising step 0, the cross-attentin
construct a dataset X' = (I),,
map computed as: M = sof tmax Z0 P.T/ 2T/vv //V)) E h According to prior work [17, 78), here M signifies the correspondence between text prompt P; and regions in generated image. Thus, we explicitly concatenate words describing human and object with P. (i.e., (P;; wordhumnn: wordsbject), resulting in a new text embedding P. E Rdxx3 and corresponding cross-attention maps M E Rixxux3 where the last two items along the third dimension channel are probability maps of human and object. Finally, we leverage the implementation in weakly supervised object localization [79] to outline bounding boxes from these probability maps.
3.4 HOI Knowledge Transfer from Diffusion Models
While prior studies [3, 48-50] have investigated knowledge transfer from visual-linguistc models such as CLIP, they utilize visual knowledge solely during training. The prediction relies on confined set of CLIP encoded word embeddings, which leads to limited knowledge transfer and rigid inference unresponsive to image content. In contrast, we propose directly leveraging diffusion models as the feature extractor and build HOI detector on this basis. Moreover, given the conditioning property of diffusion model, relation embeddings can serve as text prompts guide the retrieval of interaction-relevant visual cues from images, further benefiting HOI detection.
HOI Detector Built Upon Diffusson Models. Pioneering studies [17, 78, 80] have empirically demonstrated that the output of frozen text-to-image diffusion models possesses rich visual features to tackle complex perception tasks. Next we illustrate how to build a HOI detector on this basis. As shown in Fig. our method is a one-stage solution composed of: a visual encoder with diffusion models serving as the backbone, and a HOI decoder consisting of two parallel decoders for instance and interaction detection. For the visual encoder, given an image I, it is encoded into latent space with the encoder & of a pre-trained generator (e.g., VQGAN): ET). Then, is fed into eg through single noise-free free forward pass to derive text conditioned features: ea (z, T, co(y)) + {-/ All
5

---

P. P
4,
"R. -bat"
"R. glove"
co
Qr
Daoi
"baseball" "glove"
ql
3
Denoising UNet CD
Diss
go
qh. qo
Po
Figure 3: The overall pipeline of DIFFUSIONHOL, See $3.4 for details.
scales of features are aggregated with [PN[81], yielding z/p downsampling factor of 32. The architecture of HOI decoder is similar to GEN-VLKT(3], Concretely, the instance decoder Pins em- ploys a set of human queries [q) and object queries { and considers those at the same index (f.e., decoder can be replaced with any other one-stage models. We do not claim the detector architecture the contribution, but focus on how to derive HOI-relevant feature assist in HOI detection.
as a pair initialize queries q, for the interaction decoder DHOI- In fact, the HOI
Relation-Inspired HOI Detection, As the relation embeddings are optimized towards modeling the interactions between human and object, we use them conditions to inspire the extraction of HOI-relevant cues. This can climinate the potential domain gap between seneral-purpose diffusion models and the downstream HOI detection task. Specifically, all feasible HOI phrases (e.g., "human feed horse") are encoded with CLIP text encoder into embedding space and concatenated together, with the human-action component replaced with learned relation embeddings (e.g., "R. horse"). This results in HOI prompts P, E R N,,i which further participates into the cross-attention in cø via:
* = M; Vp, E R
M sof tmax 2 K.p//a// Ixxwxx,
(8)
where Kp, and Vp. are key and value embeddings projected from P,.. As seen, P. contributes to: i) encourage the denoising network 6g to extract visual features 2 w.r.f. HOI prompts, and ii) guide the derivative of cross-attention maps M in response to ongoing interactions in I. The final interaction maps are computed as the average value of [M, we also derive cross-attention maps for human M, and object M. in a similar way as M,, by without update to 2/. Then, these cross-attention maps are used to initialize queries from the aggregated visual feature 27 via mask pooling:
ي. = MaskPoolingl ing 2/y, ) 4. = MaskPoolingg 2/7, Mo), qn. = MaskPooling(24 z/r. MR). (9)
Note we conduct Hungarian matching between d: and á 4/, so as to arrange HOI, and com- bined human-object queries that are most similar to the same index in their respective query lists. Following[ [7], the classification for interaction and instance are jointly supervised by:
CHI tmax(q. P..TT)).bh + Cis CE( tmax(.. Po/To), No) + CE (E(
Vr),
(10)
yo),
where yr and are ground truth for interaction and object categories, qr and To are queries after decoding through Dhoi and Diss, CE and FFN denote the cross entropy loss and foed-forward network. Beyond the score delivered by conventional linear classifier (i.e., sof sof tmaxxFFFF (FFN(Q))) here soft P/T) with learnable parameters T, and T, computes the similarity between decoded queries and conditioning prompts, thereby facilitating the recognition for unseen categories.
Online Update for Relation Embedding. To enable the continual evolution of relation embeddings Rel throughout the supervised HOI detection learning, an additional loss considering the composi- tional nature of HOI devised. Specifically, prompt Pa. from which set of relation query embeddings Qa can be initialized in the same way as á. (c.f., Eq. 9). In addition, another set of embeddings describing relations
concatenate all N₂ relation embeddings into new
be derived from ğ. and by: 4. = q. qo- The goal is to align qu directly derived from visual features with relation embeddings as conditions, and qu computed from interaction and object queries after decoding:
L (g.)) lg (g.))))
(11)
Here LRe solely optimizes Ral to render a mutual boost between HOI detection and relation embed- ding learning. Concretely, enhanced relation embeddings inspires improved HOI feature discovery, and in turn, the more precise query decoding benefits the update of relation embeddings.

---

Table 1: Quantitative results for regular HOT detection HICO-DET(20] and V-COCO
Default
Known Object
V-COCO
Method
Backhone
Pretrain
Full
Rare
Non- Aan-Rare
Full
Non-Rare
A A7S
[CAN[83] [плvсCя]]
R50
14.84
10.45
16.150
16.26
11.33
17.73
45.3
PPDM[2] 24
HG104
21.73
13.78
24.10
24.58
16.65
26.84
HOTR [1]
JEV0R211
R50
23.46
16.21
25.60
55.2
64.4
QPICI
42
JCVPR211
R101
29.90
23.92
31.69
32.38
26.06
34.27
58.3
60.7
CDN144
44
R101
32.07
27.19
33.53
34.79
29.48
36,38
63.9
65.9
CPChoi
84
JCVPR771
R50
29.63
23.14
31.57
63.1
65.4
STIP[85
CCVR221
R50
32.22
28.15
33.43
35.29
31.43
36.45
66.0
70.7
86
(VVR22)
R101
32.62
28.62
33.81
36.08
31.41
37.47
61.3
67.1
1win|8 88 BOCV22)
R101
32,79
27.84
35.40
35.84
28.74
36,09
60.9
88 BCCV22)
R50
35.15
33.71
35.58
37.56
35.87
38.06
63.0
65.1
8VIC[89
[OCV23
R50
34.69
32.14
35.45
38.14
35.38
38.97
62.8
67.8
PVIC [89]
BOCV21
Swin-L
44.32
44.61
44.24
47.81
48.38
47.64
64.1
70.2
GEN-VLK[
[CVPR22]
RIO1
CLIP
34,95
31.18
36.08
38.22
34.36
39.37
63.6
65,9
HOICLIP 19]
[EVPR7]]
R50
CLIP
34.69
31.12
35.74
37.61
34.47
38.54
63.5
64.8
CQL[99]
R50
CLIP
35.36
32,97
36.07
38.43
34.85
39.50
66.4
69.2
ViPI.0[91]
VIT-B
CLIP
37.22
35.45
37.75
40.61
38.82
41.15
62.2
68.0
AGER[92]
R50
CLIP
36.75
33.53
37.71
39.84
35.58
40.2
65.7
69.7
RmLR[93]
noCv21)
R101
MobileBERT
37.41
28.81
39.97
38.69
31.27
40,91
64,2
70,2
ADA-CM[94]
nocv22)
ViT-L
CLIP
38.40
37.52
38.66
58.6
64.0
DiFFUSIONHÓI
VQGAN
Stable Diffusion
38.12
38.93
37.84
40.93
42.87
40.04
66.8
70.9
DIFUSIOOHO1
VIT-I
Stable unCI.IP
42.54
42.95
42.35
44.91
45.18
44.83
67.1
71.1
1 Models built upon advanced object dector, i.e., H--effom-DDee DEEinn---ERRII (95].
3.5 Implementation Details
Network Architecture. DIFFUSIONHOI is built upon Stable Diffusion v1.5 with xFormers [82] installed. The denoising UNet eg receives input latents at downsampling factor of 1/8, with four encoder blocks output feature at a size of 1/24+3 where [ is the block index. For the final visual feature z' after FPN aggregation, it is interpolated a size of 1/32 and then projected to 256 channels to enhance computing efficiency. Both Phoi and Dim consist of six Transformer decoding layers with hidden dimension of 768. The query number No uniformly set to 64 for both DhOi and Din-
Training Objective. The inversion-based HOI modeling is jointly optimized by two embedding learning losses: Linversion LConsistency + X Conntastive where \₁ 00, 0.2) is scheduled following cosine annealing policy. For HOI detection learning, we follow DETR [39] to match predictions and ground truths with Hungarian algorithm. Denoting the bounding box detection loss LDa, the final training objective is given as: L LHOI + Lius + Le + X L where X2 is fixed to 0.5.
4 Experiment
4.1 Experimental Setup
Datasets. We conduct extensive experiments on three datasets.
HICO-DET [20] large-scale HOI detection benchmark with 38, 18,11800 18/9,658 images for train- ing/testing, respectively. This dataset includes 80 object categories in MS-COCO[96] and 117 action categories, formulating a rich vocabulary of 600 human-object interactions in total.
V-COCO [21] is a curated subset of MS-COCO [96] including 05333/2866/44999 9,,66 images in train/val/ test sets. It also contains 80 object categories from MS-COCO [96] and much smaller set of 29 action classes, resulting in total of 263 buman-objee interactions.
SWiG-HO1[22. is assembled from S SWiG[97] and [OH[98] with about 45, 0000/14,000 for train- ing/testing. This dataset covers 406 human actions and 1,000 object categories.
Zero-Shot HOI Discovery. In accordance with prior research [3, 19, 99-102], the zero-shot HOI discovery on HICO-DET(20] uses four setups: Rare First Unseen Combination (RF-UC), Non-ram First Unseen Combination (NF-UC), Unseen Verb (UV), and Unseen Object (UO). The RF-UC and NF-UC configurations excluded the 120 most frequent/inffrequent interaction categories from the training sets for testing purposes only. The UV and UO setups reserve 20 verb classes and 12 object classes never encountered during training for testing. For SWiG-HO[[2]] the test set includes approximately 5,500 interactions, with around 1,800 of them not present in the training set.
7

---

Table 2: Zero-shot generalization on HICO-DET [20].
Table Zero-shot generalization on SWiG-DET [22].
Method
Type
Unseen
Seen
Full
Method
Non-rare
Rare
Unseen
Full
ATL[[O0]
CVV922
RF-UC
9.18
24.67
21.57
QPIC|42] (CVP221)
16,95
10.84
6.21
11.12
[CL[
(CVR22
RF-UC
13.16
24.23
22.01
[H]D[48] (CV9922)
17.67
12.82
10.04
13.26
|02
[BOCV22]
RF-UC
19.07
30.39
28.08
CMD-SE 104] [CVPR24]
21.46
14.64
10.70
15.26
GEN-VLKT[3]
CVP922)
RF-UC
21.36
32.91
30.56
DIFFUSIONHOI
25.59
20.61
18.93
21.69
OpenCad [03] CV99223
RF-UC
21.46
33.86
31.38
HOICLIP[19 cV1922
RF-UC
25.53
34.85
32.99
Table 4:
Comparison parameters and running effi-
DIFFUSIONHHI
RE-U
32.06
36.77
35.89
ciency. means applying accelerated technology.
[TL[
CV1922
NF-UC
18.25
18.78
18.67
FCL[100] CVIS2E
NF-UC
18.66
19.55
19.37
Method
Backbone
Trainable Params (M)
HICO-DET
SCL (02)
NF-UC
21.73
25.00
24.34
GEKTVL
CVPR2
NF-UC
25.05
23.38
23.71
Two-stages Dotoctors:
OpenCat 103]
NF-UC
23.25
28.04
27.08
iCAN[S]]
R50
39.3
6.23
14.84
HOICLIP(1) cv1e1)
NF-UC
26.39
28. 10
27.75
DRG 105] [XXX133)
R550
46.1
6.05
19.26
DEVUSIONHOI
NF-UC
30.04
30.29
30.25
STIP[85] (EVER77)
R50
50.4
32.22
[iPLO191]
VIT
118.2
5.66
37.22
ATL[101] CVPR21
UO
5.05
14.69
13.08
ADA-CM[94] (00V11)
VIT.I
6.6
3.24
38.40
UO
15.54
20.74
19.87
One-stages Detectors:
GEN-VLKTI3| (VPR2
UO
10.51
28.92
25.63
PPDM[24] (EVERTN
HGI04
194.9
17.58
21.73
OpenCui [03] CVPR23
UO
23.84
28.49
27.72
HOTRI 41 [CVPR21]
R550
51.2
15.92
23.46
HOICLIP 19] CV2R21
UO
16.20
30.99
28.53
QPIC[42) [ESPREI]
R550
17.41
DIFFUSIONHÖI
UO
22.37
32.03
31.12
CDN +-] (NcatPS211)
R50
42.1
16.24
31.78
GEN-VLKT[]] CVPR22)
UV
20.96
30.23
28.74
GEN-VLEE GEN-VLKT(3] [ESPR22]
R50
42.8
18.23
33.75
HOICLIP 19] CV9R22)
UV
24.30
32.19
31.09
DIFFUSIONHO! VQ
27.6
DIFFUSIONHOI
UV
28.05
33.24
32,67
*DIFFUSIONH01 VQ-GAN
27.6
24.77
38.12
Evaluation Metric. Following conventions [3, 41, 42], we adopt mAP as metrics. For HICO-DET, we report performance according to Default and Known Object two setups. The former computes mAP across all testing images, while the latter is tailored for each object class. For each setup, the scores reported in Full/Rare/Non-Ran under scenario (S1) which contains all 29 actions and scenario 2 (S2) which excludes 4 actions interact with no objects. For zero-shot setup, the evaluation is divided into Ceen//nscen/Full three sets for HICO-DET, and Non-Rare/Rared Caeiaaaaeee/neeeeaua four sets for SWIG-HOL.
three types. For V-COCO, we evaluate the performance
Training and Testing. The diffusion model and CLIP text encoder are kept frozen during training. For inversion based HOI modeling, the only learnable parameters are relation embeddings, which are updated for 40,000 steps using images sampled from HICO-DET. Following [18]. we employ base learning rate of -2 with a batch size of 32. For HOI detection learning, we train the interaction decoder Din and object decoder DHoi for 60 epochs with a base learning rate of le-1 and batch size of 16, using both synthesized data and the target dataset. Subsequently, the model is trained only on the target dataset for an additional 30 epochs with base learning rate of le 55 During inference, no data augmentation is used to ensure fair comparison. Following [3, 103], the inputs are resized to maximum of 1,333 pixels on long sides, and the shortest sides falls between 480 and 800 pixels.
Reproducibility. DifFUSIONHOI is implemented in PyTorch and trained on 8 Tesla A40 GPUs with 48GB memory per card.
4.2 Comparison with State-of-the-Arts
Regular Setup. We first compare DIFFUSIONHO! with top-leading solutions on HICO-DET(20) and V-COCO[21] under the regular setup. As shown in Table 1, for HICO-DET, our method achieves the best performance both Default and Known Object setups. Notably, with the encoder of VQGAN as the backbone, it surpasses the previous SOTA, RemLR which employs a similar level backbone (ie., ResNet-50) by 1.19% and 2.4% on the Full categories. Benefited from synthesized data and comprehensive knowledge transfer from diffusion models, the performance on Rare categories improves significantly, achieving higher scores than on Non-Rare categories for the first time. Finally, with a more powerful VL model (i.e.. Stable unCLIP I)) and backbone (i.e., VIT-L), the performance is boosted to 42.54% under the Default setup, surpassing nearly all existing work by considerable margin. Please note that [V9][89] with Swin- as the backbone leverages 7-- W--Deform-DETT [95] as the detector which achieves 48.7 mAP on MS COCO [96] by running merely 12 epochs, significantly higher than DETR[39) which achieves 36.2 mAP by running 50 epochs.
Zero-Shot Setup. Next we investigate the effectiveness of DIFFUSIONHOI under the zero-shot generalization setup. As shown in Table 2, our method yields remarkable performance across all four setups on HICO-DET. In particular, it surpasses the previous SOTA (i.e., HOICLIP [19]) by
8

---

Table 5: Detailed analysis of essential components of Table 7: Analysis of relation embeddings with different DIFFUSIONHOI on (20))
learning strategies for relation-inspired HOI detection.
Algorithm Component
Delault
RF-UC
Learnnng Strategy
Defauit
RF-DI
Full Rare
Non Rare
Full Unseen
Seen
Full Rure
Non Rure
Full
Unseen
Seen
BASELINE
33.24 20,25
34.32
30.47
20.63 33.09
Textual Inversion
34.03 32.17
34.61
10.93
21.55 33.45
+ Synitheaized Data only
35.49 36.27
35.02
33.79
28.22 34.85
Cycle- Consistecsy
35,23 34,56
35.46
32.96
24.24 34,54
Relation Prompting only
36.45.35,78
36.71
34.25
26.57 35.58
Relationn-Cuutric CL
35.94 35.06
36.32
33.72
25.73
35.17
DIFFUSIONHO
36.1238.993
37.84
35,89 52,06
36.77
Online Update
36.45 35.78
36.71
14.25
26.57
35.58
Table 6: Analysis of conditioning input for relation- inspired HOI detection on HICO-DET(20]
Table 8: Analysis of prompts for dataset generation. TD: textual description, RE: relation embedding.
Conditioning Input
Default
RE-UC
Training Set
Default
RF-UC
Full Rare
Rure
Full Unseen Seen
Full Non-Rare Full Unseen Seen
33.24 50.25
34.32
30.47
20.63 33.09
HICO DET
33.24 30.25
34.32
30.47
20.63 33,09
Textual Description
33.71 30.98
34.73
30.72
21.29
33.24
+ TD Synthesized Data 32.54 30.04
33.49
30.12
20.55 32.57
Relation Einbedding
36.45 35.78
36.71
34.25
26.57 35.58
+ RE Synthesized Data 35,49 36.27
35.02
33.79 28.22 34.85
2.90% % under the RF-UC setup. This setup emphasizes compositional generalization which requires models comprehend new types of interactions using known actions and objects. It aligns well with the strengths of text-to-image diffusion models to generate images conditioned on compositionally organized textual descriptions. Moreover, due to the effective knowledge transfer, DIFFUSIONHOI also achieves satisfactory improvement under the UV and UO setups which focus on the recognition of novel actions and objects. Table 3 further confirms the exceptional ability of our method, showing 5.97% %/8.23% % mAP improvements over CMD-SE(104 under Rare and Unseen two categories.
Model Efficiency. We compare the trainable parameter number and inference time in Table 4. As seen, DIFFUSIONHOI demonstrates significantly fewer trainable parameters compared to the one-stage counterparts. This is attributed to our mnversion-based HOI modeling, which avoids fine- tuning diffusion models like previous work [78], while effectively capturing task-specific properties. Regarding inference speed, even with stable diffusion for feature extraction, our method still achieves 9.49 FPS, a rate similar to two-stage models. This is due to the inference involving only one single forward pass, and the downsampling factor of stable diffusion from 1/8 to 1/64 is smaller than conventional backbones typically from 1/4 to 1/32. Moreover, thank to the flourishing community of stable diffusion, a variety of optimized inference solutions have emerged. By running at fp16 precision and using traced UNet, the FPS increases to 24.77, surpassing most one-stage methods.
4.3 Diagnostic Analysis
Key Component Analysis. We first examine the essential components of DIFFUSIONHOI in Table 5. Here BASELINE denotes HOI detector built upon stable diffusion without text prompting. Through jointly training with the synthesized data, both Default and RF-UC setups observe notable improvements (e.g.. up to 2.25% and 3.32% % on Full categories ). This verifies the effectiveness of our relation-driven HOI image generation strategy. in addition, after imposing relation embedding to prompt the feature extraction and HOI detection processes, the performance boosts to 36.45 % and 34.25% % under two setups. Finally, after combining these two core components together, our DIFFUSIONHOI delivers consistent improvements and sets new SOTA across all setups.
Conditioning Input. To assess the effectiveness of learned relational embeddings, we present the experimental results using different conditional inputs to stimulate HOI detection in Table 6. As seen, though action words offer limited improvement, they are far surpassed by relation embeddings which enables HOI-oriented feature extraction and enhance query initialization through cross-attention.
Relation Embedding Learning. Next we probe the impact of different strategies for relation embedding learning. The results regarding relation-ine relation-inspired HOI detection are summarized in Table 7. It can be observed that textual inversion, which directly uses different images sharing the same action for relation embedding learning, is inferior to our cycle-consistency learning strategy that considers the disentanglement nature of HOI interactions. On this basis, the relation-centric contrastive learning and online update strategies consistently bring improvement in both setups.
Prompt for Dataset Generation. Finally we study the impact of data synthesized by different types of textual prompts Table 8. As observed, data generated with purely textual description using plain action words like "The man at bat readies to swing at the pitch" gives negative improvement over baseline. The potentially indicates that diffusion models cannot understand the relations between numan-object pairs and generate mcaningful images when provided with straightforward textual
9

---

Table 9: Comparison of total training time HICO-DET
Method
CDN[44]
HOTR[41]
UPT[86]
STIP[85]
GEN-VLLKT[3] HOICLIPІ19] CQL|99] DIFFUSIONHOI
Time (Hour)
17.9
16.4
28.4
5.7+11.5
description. In contrast, through relation modeling, data generated with relation embeddings to replace the plain action words provides high-quality samples for the training of HOI detectors.
Analysis Training Cost. For our inversion-based HOI modeling learn relation-centric embed- dings, unlike the original textual inversion technology that learns text embeddings within the image space, we optimize relation embeddings within the latent space by reconstructing interaction features. This lead to reduced training costs. Consequently, the 117 relation embeddings in HICO-DET can be learned within 5.7 hours (23 minutes per relation embedding) which is more efficient than textual inversion (i.e., 32 minutes per embedding). For the main training of HOI detection on HICO-DET, since our method utilizes significantly fewer trainable parameters compared existing work (e.g.. 27.6M v.s. 50.4M for STIP [85], 41.9M for QPIC and 42. 42.8M for GEN- VLNT-VLK333 [3] in Table 4), the training process can be completed in just 11.5 hours. The comparison of the whole training time with some representative work is summarized in Table 9, with all experiments conducted on 8 Tesla A40 cards. It can be observed that DIFFUSIONHOI requires less training time than most existing work.
Conclusion
We present DIFFUSIONHOL, new HOI detector built upon diffusion models. By explicitly modeling the relations between humans and objects in an inversion- aeversion-based manner, we enable effective knowl- edge transfer from diffusion models while adapting unique characteristics of the HOI detection task. This is achieved in two aspects: i) relation-die -eliiionddriven image generation using diffusion models to enrich the training with more HOI-oriented samples, and ii) eelation-inspiree HOI detection with learned relation embeddings as prompts to retrieve spec-fie features from images. thereby enhancing the recognition of ongoing interactions. Extensive experiments demonstrate that DIFFUSIONHOI excels in both regular or zero-shot setups and sets new SOTAs. We believe this work provides insights to unleash the power of diffusion models for downstream visual perception tasks in efficient manner.
Acknowledgement. This work was supported by the National Science and Technology Major Project (No. 20232D0121300) the National Natural Science Foundation of China (No. 62372405), the Fundamental Research Funds for the Central Universities 226-2024-00058, National Key Laboratory Hybrid Augmented Intelligence, Xi' an Jiaotong University (No. HMHAI- of 202403), and CIPSC-SMP Zhipu Large Model Cross- Cross-Disciplinary Fund.
10

---

References
[1] Bangpeng Yao and Li Fei-Fei. Modeling mutual context of object and human pose in human-object interaction activities. In CVPR, 2010.
[2] Lifeng Fan, Wenguan Wang. Siyuan Huang, Xinyu Tang, and Song-Chun -huun Zhu. Understanding human gaze communication by spatio-temporal graph reasoning. In ICCV, 2019.
[3] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang. Xiaobo Li, and Si Liu. Gen-vikt: Simplify association and enhance interaction understanding for hoi detection. In CVPR, 2022. 1, 3, 7,
10, 17
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI,, 2021. 3,3
[5] Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. In ACL, 2022.
[6] Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? In CVPR, 2023.
[7] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. In NeurlPS, 2022. 1,3
[8] Nan Liu. Shuang Li. Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In ECCV, 2022. 3
[9] Xihui Liu, Dong Huk Park, Samanch Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In WACV, 2023.
[10] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, 2022.
[11] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional
image generation with clip latents. arXiv preprint arXiv:2204.0612 1(2)3, 2022. 8
[12] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In ICLR, 2022.
[13) Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022. 3
[14] Robin Romhach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 1,3
[15] Koupat Preechakul. Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakorn. Diffusion autocncoders: Toward mcaningful and decodable representation. In CVPR, 2022.
[16] Mingi Kwon, Jaescok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. ICLR, 2022.
[17] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023. 3,
[18] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. ICLR, 2022. 2, 3,8
[19] Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-languee models. In CVPR, 2023. 2, 7, 8, 10, 17
[20] fei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning detect homen-object interactions. In WACV, 2018. 7, 8, 9, 10, 17
[21] Saurahh Gupta and Jitendra Malik. Visual semantic role laheling, arXiv preprint arXiv: 2015.
2. 7, 8, 17
[22] Suchen Wang, Kim-Hui Yap, Henghui Ding, Jiyan Wa, Junsong Yuan, and Yap-Pong Tan. Discovering human interactions with large-vocabulary objects via query and multi-scalc detection. In ICCV, 2021. 2, 7, 8, 17
[23] Bumsoo Kim, Tacho Choi, Jaewoo Kang, and Hyunwoo J Kim. Uniondet: Union-level detector towards
real-time human-object interaction detection. In ECCV, 2020. 3, 17
[24] Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng. Ppdm: Parallel point detection and matching for real-time humen-object interaction detection. In CVPR, 2020. 7, 8, 17
11

---

[25] Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun. Learning human-object interaction detection using interaction points. In CVPR, 2020.
[26] Hao-Shu Fang, Yichen Xie, Dian Shao, and Cewu Lu. Dirv: Dense interaction region voting for end-to-end human-object interaction detection. In AAAI, 2021. 3
[27] Siyuan Qi. Wenguan Wang. Baoxiong Jia, Jianbing Shen, and Song-Chun Chun Zhu. Learning human-object interactions by graph parsing neural networks. In ECCV, 2018. 3
[28] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xumingg Hc.. Pose-aware multi-level feature network for human object interaction detection. In /CCV, 2019.
[29] Hai Wang, Wei-shi Zheng, and Ling Yingbiao. Contextual heterogeneous graph network for human-object interaction detection. In ECCV, 2020.
[30] Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human-object interaction. NeurIPS, 2020.
[31] Yong-Lu Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang. Jungi Liu, Jiefeng Li, and Cewu Lu. Detailed 2d-3d joint representation for human-object interaction. In CVPR, 2020.
[32] Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object interactions. In ICCV, 2021.
[33] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing buman-object interactions. In CVPR, 2018.
[34] Tianfei Zhou, Wenguan Wang. Siyuan Qi. Haibin Ling, and Jianbing Shen. Cascaded human-object interaction recognition. In CVPR, 2020.
[35] Tianfci Zhou, Siyuan Qi, Wenguan Wang, Jianbing Shen, and Song-Chun Zhu. Cascaded parsing of human-object interaction recognition. IEEE TPAMI, 4886)22800- 2021.
[36] Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Hydra-sgg: Hybrid relation assignment for one stage seene graph generation. arXiv preprint arXiv:2409. 10262, 2024.
[37] Jianan Wei, Tianfei Zhou, Yi Yang, and Wenguan Wang. Nonverbal interaction detection. In ECCV, 2024.
[38] Shaoqing Ren, Kaiming He, Ross Girshick. and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurlPS, 2015. 3
[39] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 3, 8
[40] Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian. Reformulating hoi detection adaptive set prediction. In CVPR, 2021. 3, 17
[41] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to-end
human-object interaction detection with transformers. In CVPR, 2021. 7, 8, 10, 17
[42] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise human-object interaction detection with imago-wide contextual information. In CVPR, 2021. 7,8, 10, 17
[43] Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, et al. End-to- end human object interaction detection with hoi transformer. In CVPR, 2021.
[44] Aixi Zhang, Yue Liao, Si Liu, Miao La, Yongliang Wang, Chen Gao, and Xiaoho Li. Mining the henefits of two-stage and one-stage hoi detection. In NeurlPS, 2021. 7, 8, 10, 17
[45] Desen Zhou, Zhichao Liu, Jian Wang, Leshan Wang, Tao Hu, Errui Ding, and Jingdong Wang. Human- object intoraction detection via disentangled transformer. In CVPR, 2022.
[46] Xubin Zhong, Changxing Ding, Zijian Li, and Shaoli Huang. Towards hard-positive query mining for date- human-object interaction detection. ECCV, 2022.
[47] ASM Iftekhar, Hao Chen, Kaustav Kundu, Xinyu Li, Joseph Tighe, and Davide Modolo. What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions. In CVPR, 2022. 3, 17
[48] Suchen Wang. Yueqi Duan, Henghui Ding, Yap-Peng Tan, Kim-Hui Yap, and Junsong Yuan. Learning transferable human-object interaction detector with natural language supervision. In CVPR, 2022. 5, 8
[49] Xian Qu, Changxing Ding, Xingao Li, Xubin Zhong, and Dacheng Tao. Distillation using oracle queries for transformer-based human-object interaction detection. In CVPR, 2022. 17
[50] Leizhen Dong, Zhimin Li, Kunlun Xu, Zhijun Zhang, Luxin Yan. Sheng Zhong, and Xu Zou. Category- aware transformer network for better human-object interaction detection. In CVPR, 2022. 5, 17
12

---

[51] Guikun Chen, Jin Li, and Wenguan Wang. Scene graph generation with role-playing large language models. In NewrIPS, 2024.
[52] Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural-logic human-object interaction detection. In NeurIPS, 2023. 3
[53] Wenguan Wang, Yi Yang, and Yunhe Pan. Visual knowledge in the big model era: Retrospect and prospect. Frontiers Information Technology & Electronic Engineering, 2024. 3
[54] Liulei Li, Wenguan Wang, and Yi Yang. Logicseg: Parsing visual semantics with neural logic learning and reasoning. In ICCV, 2023.
[55] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. 3
[56] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neur/PS, 2014. 3
[57] Rohit Saha, Brendan Duke, Florian Shkurti, Graham W Taylor, and Parham Aarabi. Loho: Latent optimization of hairstyles via orthogonalization. In CVPR, 2021. 3
[58] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICMI., 2022. 3
[59] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. NeurIPS, 2019. 3
[60] Jaseha Sohl- Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised lcarning using nonequilibrium thermodynamies. ICML, 2015. 3
[61] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[62] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020. 3
[63] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS,
2021. 3
[64] Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-dccoddin models for few-shot conditional generation. In NeurlPS, 2021. 3
[65] Georgios Batzolis, Jan Stanczuk, Carola Bibbi-e Schönlich, and Christian Ermann. Conditional image generation with bosee diffusion models. arXiv preprint arXiv:21 1...6666,,,66 2021. 3
[66] Max Daniels, Tyler Maunu, and Paul Hand. Score-based generative neural networks for large-scale optimal transport. 2021. 3
[67] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE TPAMI, 45(4):4713-4726, 2022. 3
[68] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2021. 3
[69] Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2022.
[70] Haochen Wang. Xiaodan Du, Jiahao Li, Raymond Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffuusio models for 3d generation. In CVPR, 2023.
[71] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023. 3
[72] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfea- tures: Searching through time and space for semantic correspondence. In NeurlPS, 2023. 3
[73] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. arXiv preprint arXiv:2306.03881, 2023.
[74] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Miuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic corresonndece. In NewrIPS, 2023. 3
[75] Xingyi Yang and Xinchao Wang Diffusion model as representation learner. In ICCV, 2023. 3
[76] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantize image modeling with improved vggan. In ICLR. 2022. 4
13

---

[77] Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00525, 2015. 5
[78] Wenliang Zhao. Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to- text-to-image diffusion models for visual perception. In ICCV, 2023. 5,
[79] Junsuk Choe. Seong Joon Oh, Seungho Lee. Sanghyuk Chun, Zeynep Akata, and Hyunjung Shim. Evaluating weakly supervised object localization methods right. In CVPR, 2020. 5
[80] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. ICLR, 2022.
[81] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017. 6
[82] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jicru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. https://gt gi thub.
wttprcch///eeee xfrmerrs 2022. 7
[83] Chen Gao, Yuliang Zou, and Bia-Bin Huang. ican: Instance-centric attention network for human-ohjeet
interaction detection. In BMVC, 2018. 7,8,17 17
[84] Jihwan Park, SeungJun Lee, Hwan Heo, Hyeong Kyu Choi, and Hyunwoo J Kim. Consistency learning via decoding path augmentation for transformers in human object interaction detcetion. In CVPR, 2022.
7, 17
[85] Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang. Tao Mei, and Chang- Wen Chen. Exploring structure- aware transformer over interaction proposals for human-object interaction detection. In CVPR, 2022. 7,
10, 17
[86] Frederic Z Zhang, Dylan Campbell, and Stephen Gould, Efficient two-stage detection of human-object interactions with a novel unary- pairwisc transformer. In CVPR, 2022. 7, 10, 17
[87] Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. Twin: Human- ohject interaction detection via transformer with irregular windows. In ECCV, 2022. 7, 17
[88] Xiaogian Wu, Yong-Lu Li, Xinpeng Liu. Junyi Zhang. Yuzhe Wu, and Cewu Lu. Mining cross-person cues for body-part interactiveness learning hoi detection. In ECCV, 2022. 7, 17
[89] Frederic Z Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould, Exploring predicate visual context in detecting of human-object interactions. In ICCV, 2023. 7, 8, 17
[90] Chi Xie, Fangao Zeng, Yue Hu, Shuang Liang, and Yichen Wei. Category query learning for human-object
interaction classification. In CVPR, 2023. 7, 10, 17
[91] Jeeseung Park, Jin-Woo Park, and Jong-Scok Lee. Viplo: Vision transformer based posc-conditioned
self-loop graph for hujan-object interaction detection. In CVPR, 2023. 7, 8, 17
[92] Danyang Tu, Wei Sun, Guangtao Zhai, and Wei Shen. Agglomerative transformer for human-object interaction detection. In ICCV, 2023. 7, 17
[93] Yichao Cao, Qingfei Tang, Feng Yang, Xiu Su, Shan You, Xiaobo Lu, and Chang Xu. Re-mine, learn and reason: Exploring the cross-modal semantic correlations for language-guide hoi detection. In ICCV, 2023. 7, 8, 17
[94] Ting Lei, Fabian Caba, Qingehao Chen, Hailin Jin, Yuxin Peng, and Yang Liu. Efficient adaptive human-object interaction detection with camcept-guided memory. In ICCV, 2023. 7,8, 17
[95] Ding Jia, Yuhui Yuan, Haodi He. Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In CVPR, 2023. 7, 8,8,1 17
[96] Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 8,
[97] Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. Grounded situation recognition. ECCV, 2020.
[98] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In CVPR, 2020.
[99] 7hi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for human-ohject interaction detection. In ECCV, 2020. 7
[100] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Detecting human-object interaction via fabricated compositional learning. CVPR, 2021.
[101] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao, Affordance transfer learning for hujen- interaction detection. In CVPR, 2021. 8
14

---

[102] Zhi Hou, Baosheng Yu, and Dacheng Tao. Discovering human-object interaction concepts via self-
compositional learning. In ECCV, 2022. 8.8
[103] Sipeng Zheng, Boshen Xu, and Qin Jin. Open-category human-object interaction pre-training via language modeling framework. In CVPR, 2023.
[104] Ting Lei, Shaofeng Yin, and Yang Liu, Exploring the potential of large foundation models open- vocabulary hoi detection. In CVPR, 2024. 8,9
[105] Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang. Drg: Dual relation graph for human-object interaction detection. In ECCV, 2020. 17
[106] Ye Liu, Junsong Yuan, and Chang Wen Chen. Consnet: Learning consistency graph for zero-shot human-object interaction detection. In ACM MM, 2020. 17
[107] Bumsoo Kim, Jonghwan Mun, Kyoung- Woon On, Minchul Shin, Junhyun Lec, and Eun-Sol Kim. Mstr:
Multi-scale transformer for end-to-end human-object interaction detection. In CVPR, 2022. 17
[108] Xinpeng Liu, Yong-Lu Li, Xiaoqian Wu, Yu-Wing Wing Tai, Cewu Lu, and Chi- Chi-Koung Tang. Interactiveness field human-object interactions. CVPR, 2022. 17
[109] Guangzhi Wang. Yangyang Guo, Yongkang Wong, and Mohan Kankanhalli. Chairs can be stood on: Overcoming object bias in human-object interaction detection. In ECCV, 2022. 17
15

---

A Discussion
A.1 Limitation
One potential limitation of our method is that it does not run as quickly as previous one-stage methods due the adoption of diffusion models. A detailed comparison is provided in Table As shown, DIFFUSIONHOI achieves inference speeds similar to two-stage methods. However, currently there is trend towards leveraging large-scale pre-trained models assist in various perception tasks. These models offer substantial enhancements in performance and accuracy. For instance, after utilizzig diffusion models, our method has demonstrated an 8.23% improvement on unseen categories in the SWIG-HOI benchmark. Though it is inevitable that these advancements introduce additional computational overhead, the trade-off generally considered acceptable given the significant improvements in task performance and the ability to generalize better to previously unseen scenarios. Therefore, an important consideration in future research is the balance between computational cost and the enhanced capabilities provided by large-scale pre-trained models. In this work, we implement acceleration technology for Stable Diffusion which successfully boosts the inference speed of DIFFUSIONHOI to 24.77 FPS, surpassing most one-stage methods.
A.2 Failure Case
As shown Fig.S1, we found that failure cases primarily manifest the following scenarios: i) leg, which introduces challenges for
scenes featuring only partial human bodies, such as person detection; and ii) chaotic scenes teeming with people, which causes occlusion and difficulties in identifying interactions. Despite these challenges, DIFFUSIONHOI has shown remarkable im- provement over existing approaches. Additionally, the patterns of these failure cases provide valuable insights for future research.
A.3 Broader Impact
On the positive side, this work proposes more powerful solution for recognizing complex interactions between humans and objects a scene. is particularly effective for few- and zero-shot setups that are common in real-world scenarios. This ad advancement can significantly contribute to a range of applications, such as healthcare and assistive systems, smart homes and IoT (Internet of Things), security, and more. However, there are also potential negative aspects. Our method carries the risk of being used for continuous monitoring, which could raise concerns over intrusive surveillance and the unauthorized collection of personal data. Additionally, the inversion-based modeling strategy for adjusting diffusion models could misused to create harmful or false information about individuals. Hence, it is essential to rigorously consider ethical standards and legal regulations to address privacy concerns, so as to avoid potential negative societal impacts.
B Detailed Comparison
We present more detailed comparison of DIFFUSIONHOI with other methods in Table S1. As demonstrated, DIFFUSIONHOI consistently achieves state-of-the-arr performance.
C Qualitative Results for Image Generation
We present the qualitative results of relation-driver sample generation in Fig. It can be observed that the synthesized images exhibit realistic shapes and textures for individual instances, as well as coherent structure and layout for entire scenes. Additionally, the generated images accurately depict the interactions between humans and objects, demonstrating the effectiveness of our proposed inversion-based HOI modeling strategies.
16

---

Person ride cow
Person hug sheep
Person teach person
Porsee-hold poriddo-oooo
Person hold knife
Pesses hold wine wine-gless
Person pel dog
Person drag suitcase
Figure Typical failure case on HICO 1990-.ET[19]] Actions highlighted in indicate missing predictions that should be detected, while text with strkkthrough means wrong predictions that should be removed.
D License
The V-C0CO[21] and SWiG-HOI [22] datasets are released under the MIT license. The HICO- DET[20] dataset is released under the CCO: Public Domain license. The weight of Stable Diffusion released under CreativeML Open RAIL M License.
Table S1: Detailed comparison for regular HOI detection on HICO (ICO-DEE[20] and V-COCO121].
VL
Default Rare
Known Object
V-COCO
Method
Backbone
Pretruin
Full
Non-Rare
Full
Rare
Non-Rure
APrdn
rale
[83
(8SVC16)
R50
14.84
10.45
16.150
16.26
11.33
17.73
45.3
UnionDett22
R50
17.58
11.72
19.33
19.76
14.68
21.27
47.5
56.2
DRG [OS
R50-FPN
19.26
17.74
19.71
23.40
21.75
23.89
51.0
PPDM 24
CVPR22
HG104
21.73
13.78
24.10
24.58
16.65
26.84
HOTR
(S9121)
R50
23.46
16.21
25.60
55.2
64.4
ConsNetl
106
R50
24.39
17.10
26.56
AS-Net
[0
CXPR21
R50
28.87
24.25
30.25
31.74
27.07
33.14
53.9
QPIC 42
(X9921
R50
29.07
21.85
31.23
31.68
24.14
33.93
58.8
61.0
CDN[44]
27.55
62.3
64.4
CPChoi 84
R50
29.63
23.14
31.57
63.1
65.4
MSTR[10 107
R50
31.17
25.31
32.92
34.02
28.83
35.57
62.0
65.2
866
R50
31,66
25.94
33.36
35.05
29.27
36.77
59.0
64.5
STIPL85
R50
32.22
28.15
33.43
35.29
31.43
36.45
66.0
70.7
IF-HOT [08
(VPR22)
R50
33.51
30.30
34.46
36.28
33.16
37.21
63.0
65.2
ODM[ 109
B2CC222
R50-FPN
31.65
24.95
33.65
Iwin[S S7
BO0C222
R50-FPN
32.03
27.62
34.14
35.17
28.79
35.91
60.5
MCPC 88
33.71
35.58
37.56
35.87
63.0
BXCV223
R50
32.14
35.45
38.14
35.38
62.8
67.8
P9
Swin-L
44.32
44.61
44.24
47.81
48.38 33.96
47.64
64.1
70.2
CTÀNI
R50
CLIP
31.71
24.82
33.77
26.37
36.23
60.1
SSRT
R50
CLIP
30.36
25.42
31.83
63.7
65.9
DOQE49 49
R50
CLIP
33.28
29.19
34.50
63.5
GEN VLK[3]
R50
CLIP
33.75
29.25
35.10
37.80
34.76
38.71
62.4
64.4
HOICLIP [9]
CVPR22)
R50
CLIP
34.69
31.12
35.74
37.61
34.47
38.54
63.5
64.8
[QU
C59925)
R50
CLIP
35.36
32.97
36.07
38.43
34.85
39.50
66.4
69.2
[1
exPR25
VIT-B
CLIP
37.22
35.45
37.75
40.61
38.82
41.15
62.2
68.0
AGER(92
(oco25)
R50
CLIP
36.75
33.53
37.71
39.84
35.58
40.2
65.7
69,7
RmLR(93]
pocv22
R50
MobileBERT
29.03
39.29
38.29
31.41
63.8
69.8
ADA-CM[94]
VIT--
CLIP
38.40
37.52
38.66
58.6
64.0
DiNTUSIoNHÖI
VOGAN
Stuble Diffusion
38.12
38.93
37.88
40.93
42.87
40.04
66.8
70.9
DIFFUSIONHOI
VIT--
Stable unCLIP
42.54
42.95
42.35
44.91
45.18
44.83
67.1
71.1
Models built upon advanced object dector, i.e., H-Deform-DETR. (95].
17

---

human-swing wtteesswwwi-bbbbballeee baseball_baa
human-walk-dog
humaa--opprattemmcccovvvv
human-hold surfboard
human-jump-skateele
suman-hug-shess
Figure S2: Qualitative results for relation-driven image generation.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main contributions have been claimed in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The computational efficiency has been discussed in $4.2. The limitation the proposed algorithm has been discussed in Appendix.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
There is no theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The implementation details have been provided in $3.5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

reproducible, they Answer: [Yes]
Justification:
Code is available at httppp//gggaaab..c/////aaaaaaa/a
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm
wwssssooooicyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ / wttpccccppppccccccccc wwwpc//ggg////eeeeeeeeeee /oodeSubmeeeeeeee cerbbbissioooo aagionooliie
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setup, including data splits, training and testing detailed, provided in $4.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The prior work included into comparison does not provide error bar. could be too expensive and time-consuming to run the experiments on large-scale datasets for multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
21

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources are described in $4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / http///eerri..eeee httbbiiiccchhicccccieeeeeee.
Answer: [Yes]
Justification:
This work conforms the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The border impacts provided Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The proposed method uses pre- pre-trained models. This proposed methods is safe under the safeguards of adopted pre-trained models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The V-COCO and SWiG-HOI datasets are released under the MIT license. The HICO-DET dataset is released under the CCO: Public Domain license. The weight of Stable Diffusion is released under CreativeMI. Open RAIL M License.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode.tcoe...c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
There is new assets released in this work.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
There is no research with human subjects in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is research with human subjects in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "CY3CagSAlh", "reviewer": "Reviewer_MQdd", "summary": "This paper tackles the human-object interaction (HOI) detection task.\n\nIt aims to utilize the feature of generative models like diffusion models to help human-object interaction classification.\n\nMore specifically, it utilize the inversion process in the diffusion model to learn the embedding for human-object interaction.\n\nThe proposed method obtains obvious improvement over existing methods.", "questions": "- The analysis of the cost.\n\nI will look into the rebuttal and adjust my rating accordingly.", "limitations": "Discussion over the limitation is provided.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The improvement on performance is significant compared with the baseline.\n- The figures are illustrative and helpful for understanding this paper.\n- The writing and overall demonstration is good. The problems to tackle are clearly explained, for example, in line 39-61. This does affect my rating.\n- The ablation study is though and well organized.", "weaknesses": "- The cost of training with the inversion process. As the inversion process is computationally costly and slow, it could affect the training speed. I would like to see more clarification or analysis over the cost of the proposed method.\n- The proposed model does not achieves SOTA as it claims. For example, it falls behind PViC (iccv23) with swin-L backbone by a large margin. As the proposed method have a ViT-L backbone, I think it is not fair to not mentioning PViC-swin-L and some other methods. Besides, PViC utilize no external knowledge like CLIP or SD while this method does.\n- The general technical design is complicated. This is not a big problem and does not affect my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Analysis on the training cost.**\n\n**A1:** For relation-centric inversion, unlike the original textual inversion technology that learns text embeddings within the image space, we optimize relation embeddings within the latent space by reconstructing interaction features. This results in reduced training costs. Consequently, the 117 relation embeddings in HICO-DET can be learned within 5.7 hours (23 minutes per relation embedding), using 8 Tesla A40 cards.\n\n|Method| Time (Minute)|\n|:-|:-|\n|Textual Inversion|32|\n|Relation-centric Inversion|23|\n\nFor the main training of HOI detection on HICO-DET, as shown in Table 4, our method utilizes significantly fewer trainable parameters compared to existing work ($\\textit{e.g.}$, 27.6M $\\textit{vs.}$ 50.4M for STIP [77], 41.9M for QPIC [37], and 42.8M for GEN-VLKT [2]), and therefore the training process is completed in just 11.5 hours.\nThe comparison of the whole training time with some representative work is summarized below, with all experiments conducted on 8 Tesla A40 cards. It can be observed that our method is more efficient.\n\n|Method|Time (Hour)|\n|:-|:-|\n|CDN[39]|25.2|\n|HOTR[36]|23.6|\n|UPT[76]|17.9|\n|STIP[77]|16.4|\n|GEN-VLKT[2]|28.4|\n|HOICLIP[18]|29.1|\n|CQL[80]|29.7|\n|DIFFUSIONHOI|$\\textbf{5.7+11.5}$|\n\nThe discussion and results above will be integrated into $\\S$4.2.\n\n---\n\n**Q2: Comparison to SOTA work.**\n\n**A2:** To solidly address your concern, we provide a comprehensive comparison with PViC below. Here PViC with Swin-L leverages $\\mathcal{H}$-Deform-DETR as the detector which achieves 48.7 mAP on MS COCO by running merely 12 epochs, significantly higher than DETR which achieves 36.2 mAP by running 50 epochs.\nFor a fair comparison, we i) reimplement PViC with ViT-L as the backbone and DETR as the detector, and ii) augment the two parallel decoders for instance and interaction detection of DIFFUSIONHOI with the technology used in $\\mathcal{H}$-Deform-DETR, resulting in DIFFUSIONHOI with the decoder holds a similar detection ability to $\\mathcal{H}$-Deform-DETR. The detailed results are summarized below.\n\n|Model| Backbone| Detector| Full|Rare|Non-rare|AP$^{S1}_{role}$|AP$^{S2}_{role}$|\n|:-|:-|:-|:-|:-|:-|:-|:-|\n|PViC|ResNet-50|DETR| 34.69 |32.14 |35.45|62.8 |67.8|\n|DIFFUSIONHOI|VQGAN|$\\approx$DETR|$\\textbf{38.12}$|$\\textbf{38.93}$|$\\textbf{37.84}$|$\\textbf{66.8}$ |$\\textbf{70.9}$|\n|PViC|ViT-L|DETR| 39.89| 40.36|39.98|63.4|68.9|\n|DIFFUSIONHOI|ViT-L|$\\approx$DETR|$\\textbf{42.54}$|$\\textbf{42.95}$|$\\textbf{42.35}$|$\\textbf{67.1}$|$\\textbf{71.1}$|\n|PViC|Swin-L|$\\mathcal{H}$-Deform-DETR|44.32 |44.61| 44.24|64.1| 70.2|\n|DIFFUSIONHOI |ViT-L|$\\approx$$\\mathcal{H}$-Deform-DETR|$\\textbf{45.46}$|$\\textbf{45.78}$|$\\textbf{45.37}$|$\\textbf{67.9}$|$\\textbf{71.5}$|\n\nAs seen, with a similar backbone ($\\textit{i.e.}$, ResNet-50 and VQGAN), our method surpasses PViC by 3.43 mAP on HICO-DET, 4.0 mAP on V-COCO. Next, after reimplementing PViC with DETR and ViT-L as the backbone, the performance is still inferior to DIFFUSIONHOI ($\\textit{i.e.}$, 42.54 $\\textit{vs.}$ 39.89 on HICO-DET). Finally, after augmenting DIFFUSIONHOI with tricks used in $\\mathcal{H}$-Deform-DETR, the performance boosts to 45.46 mAP on HICO-DET. Please note that ViT-L (employed by DIFFUSIONHOI) is inferior to Swin-L (employed by PViC) for downstream tasks.\n\nWe will provide a discussion in the experiment section, to highlight methods not using additional knowledge ($\\textit{e.g.}$, PViC) during comparison.\n \n---\n\n**Q3: Complicated technical design.** \n\n**A3:** This study represents the first attempt to utilize diffusion models for HOI detection, from the perspective of both data generation and conditioned feature extraction. This may bring a fundamental shift in architectural design and training strategy. \nHowever, we agree that simplicity in design is crucial for reimplementation and extension by the community. We will definitely consider this as our future work. \n\n---\n\nThank you so much for your careful review and suggestive comments. We will update our manuscript with the training time analysis and a more detailed comparison with the SOTA approaches. Please feel free to post your feedback if you have any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ip3Emf2oI7", "reviewer": "Reviewer_UENt", "summary": "This paper introduces DIFFUSIONHOI, a new HOI detector leveraging text-to-image diffusion models. Unlike previous one-stage or two-stage models, diffusion models excel at discerning mid/low-level visual concepts as generative models and possess strong compositionality to handle novel concepts expressed in text inputs. To steer the focus of diffusion models from instance generation to the relationships between humans and objects, this paper exploits textual inversion and devises a human-object relation inversion strategy grounded in the disentanglement of HOI. Furthermore, to transfer extensive knowledge from large-scale diffusion models to assist in recognizing interactions, the paper leverages both text-prompted image generation and conditioned feature extraction capabilities of diffusion models. By embracing text-to-image diffusion models and facilitating relation-driven image generation and prompting, this method demonstrates superior performance.", "questions": "Please refer to weakness.", "limitations": "Please refer to weakness.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and easy to follow. It proposes a new diffusion-based solution, DiffusionHOI, for the human-object interaction (HOI) task. Unlike traditional one-stage and two-stage methods, DiffusionHOI benefits from controllable image generation and HOI knowledge transfer from diffusion models. Without making design changes to the HOI decoder, this method achieves superior performance compared to previous state-of-the-art models by deriving HOI-relevant features to assist in HOI detection.", "weaknesses": "1. I am curious about the number of parameters in the VQGAN used as the backbone of DiffusionHOI. According to Table 1, the authors use ViT-L as their backbone, which appears to be significantly larger compared to previous works, especially those using the Res50 network. Could the authors provide more results using a smaller backbone, such as ViT-B, to offer a clearer and fairer comparison between DiffusionHOI and previous state-of-the-art models?\n2. Another concern is that the improvement primarily stems from the additional knowledge provided by the powerful diffusion model. Could this be considered unfair when comparing it to previous works that do not have this significant benefit? In this case, how does this approach differ from simply adding more data for training? As I am familiar with HOI but not as familiar with diffusion models, could the authors provide more details about the data scale and parameter scale of the diffusion model used in this paper? Additionally, I look forward to feedback from other reviewers on this matter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.1: Number of parameters in the VQGAN.**\n\n**A1:** We only utilize the encoder of VQGAN as the backbone which contains 39.2M parameters and is smaller than that of ResNet-101 (42.8M). To address your concern about unfair comparison with models using ResNet-50 as the backbone, we provide a detailed comparison with top-leading approaches using ResNet-101 below.\n\n|Model| Backbone| Full|AP$^{S1}_{role}$|AP$^{S2}_{role}$|\n|:-|:-|:-|:-|:-|\n|QPIC|ResNet-101|29.90|58.3|60.7|\n|CDN|ResNet-101|32.07|63.91|65.89|\n|UDP|ResNet-101|32.62|61.3|67.1|\n|Iwin|ResNet-101|32.79|60.9|-|\n|GEN-VLKT|ResNet-101|34.95|63.6|66.0|\n|RmLR|ResNet-101|37.41|64.2|70.2|\n|DIFFUSIONHOI|VQGAN|$\\textbf{38.12}$|$\\textbf{66.8}$ |$\\textbf{70.9}$|\n\nAs seen, our DIFFUSIONHOI achieves SOTA performance on both HICO-DET and V-COCO. This verifies the superiority of our proposed method. Table 1 will be updated accordingly.\n\n---\n\n**Q1.2: Comparison with large backbones.** \n\n**A2:** The implementation of DIFFUSIONHOI relies on pre-trained Diffusion models that do not provide small backbones such as ViT-B. Therefore, we provide a detailed comparison with models using large backbones like ViT-L below to ensure a fair comparison.\n\n|Model| Backbone| Full|AP$^{S1}_{role}$|AP$^{S2}_{role}$|\n|:-|:-|:-|:-|:-|\n|ADA-CM|ViT-L|38.40|58.6|64.0|\n|PViC|ViT-L|39.89|63.4|68.9|\n|DIFFUSIONHOI|ViT-L|$\\textbf{42.54}$|$\\textbf{67.1}$| $\\textbf{71.1}$|\n\nCurrently, only a few models provide results on large backbones such as ViT-L, and DIFFUSIONHOI can achieve the best performance on two datasets. \n\n---\n\n**Q2.1: Unfair comparison to previous work without using additional knowledge.**\n\n**A2.1:** Though prior work does not utilize diffusion models for knowledge transfer, they adopted other visual-linguistic models like CLIP ($\\textit{e.g.}$, CTAN [45], SSRT [42], DOQ [44], GEN-VLK [2], HOICLIP [18], CQL [80], AGER[82], RmLR[83], and ADA-CM[84]). We provide a detailed comparison with approaches using pre-training models such as CLIP or MobileBERT in Table 1. It can be seen that when provided with additional knowledge, DIFFUSIONHOI can achieve a clear gap in performance ($\\textit{e.g.}$, 38.12 mAP $\\textit{vs.}$ 33.75 mAP for GEN-VLK[2], 35.35 mAP for CQL[80], 36.75 mAP for AGER[82], and 36.93 mAP for RmLR[83]), which highlights the effectiveness of our design.\n\n**Q2.2: The difference from methods simply adding more data for training.**\n\n**A2.2:** The primary distinction lies in three aspects:\n\nFirst, adding training data from other datasets is impractical in real-world applications, as it requires significant time and labor to gather annotated data, which include bounding boxes, object classes, and the interactions between humans and objects. This also imposes significant challenges in generating HOI samples using diffusion models due to the complexity of annotations.\n\nSecond, the training data in our work is generated with respect to the relations between humans and objects, resulting in images that depict specific human-object interactions more accurately ($\\S$3.3 Relation-Driven Sample Generation). As outlined in L178-180, for a text prompt such as \"The man at bat readies to swing at the pitch,\" the action word \"swing\" is replaced with the relation embedding learned through relation-centric inversion. This steers diffusion models to focus on complex relationships rather than single objects, without heavy retraining. \n\nThird, as stated in L53-61, we achieve knowledge transfer from diffusion models in two aspects. In addition to data generation, features distinguished for each interaction can be derived with learned relational embeddings as prompts ($\\S$3.4 Relation-Inspired HOI Detection). Therefore, we enhance HOI detection with diffusion models in both training data and hypothesis set ($\\textit{i.e.}$, feature extraction), while adding more data can only affect the training set.\n\nAdditional experiments on HICO-DET are given below to verify the effectiveness of our design. As seen, simply adding more data from SWiG-DET for training (row #2) delivers limited improvement compared to using additional data synthesized with relation-driven sample generation (row #3), or features extracted with relational embeddings as prompts (row #4). \n\n|Method|Full|Rare| Non-Rare|\n|:-|:-|:-|:-|\n|BASELINE|33.24|30.25|34.32| \n|+ data from SWiG-DET only |33.89 |31.13 |34.72 |\n|+ Synthesized Data only|35.49 |36.27| 35.02|\n|+ Relation Prompting only|36.45|35.78|36.71| \n|DIFFUSIONHOI|$\\textbf{38.12}$|$\\textbf{38.93}$|$\\textbf{37.84}$|\n\n**Q2.3: More details about the data and parameter scale of the diffusion model used.**\n\n**A2.3:** As specified in L254, DIFFUSIONHOI is built upon Stable Diffusion v1.5 pre-trained on laion2B-en, a dataset containing 2.3B samples, and subsequently fine-tuned on high-quality subsets of laion2B-en. The total parameter size is 1066M. Despite the large parameter size, as summarized in Table 4, the number of trainable parameters is significantly smaller compared to existing works ($\\textit{e.g.}$, 27.6M $\\textit{vs.}$ 50.4M for STIP [77], 41.9M for QPIC [37], and 42.8M for GEN-VLKT [2]), and the inference speed is comparable to leading counterparts ($\\textit{e.g.}$, 9.49 FPS $\\textit{vs.}$ 6.23 for iCAN [74], 7.12 for STIP [77], 3.24 for ADA-CM [84]). This indicates that the usage of diffusion models does not impose a heavy burden on our approach.\n\nMoreover, as stated in L76-77 and L365-366, this work presents an efficient way to transfer knowledge from diffusion models, with human-object relational embeddings learned via relation-centric inversion. This can offer valuable insights to the community, and further advance the exploration of large visual-linguistic pre-training models for downstream tasks.\n\n---\n\nThank you for providing such a comprehensive and detailed review. We are committed to revising the experiment section in accordance with your valuable suggestions. Please do not hesitate to post comments if there are any further questions we can help.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0LZCmrNVLB", "reviewer": "Reviewer_Xw3E", "summary": "This paper introduces DIFFUSIONHOI, a novel Human-Object Interaction (HOI) detector that utilizes text-to-image diffusion models for HOI detection. It efficiently focuses on complex relationships between objects, providing a strong basis for HOI modeling. The relation-driven approach enhances image generation capabilities for HOI detection, enriching training samples for rare interactions. Additionally, it improves detector flexibility and accuracy, achieving good performance on HICO-DET and V-COCO benchmarks.", "questions": "N/A", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The experiments are comprehensive.\n- The design of Inversion-Based HOI Modeling and Relation-Driven Sample Generation are intriguing.- \n- Achieving zero-shot generalization on SWiG-DET is impressive.", "weaknesses": "- Despite having fewer trainable parameters overall, DIFFUSIONHOI has a large total parameter count, which raises concerns about unfair comparisons with existing methods.\n- The authors claim to set a new state-of-the-art on HICO-DET, but the reviewer found that there are better models available, as also mentioned in the paper [79].\n- Quantitative results and analysis of failure cases are needed.\n\n[79] Frederic Z Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould. Exploring predicate visual context in detecting of human-object interactions. In ICCV, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Concerns about unfair comparisons as DIFFUSIONHOI has a larger total parameter count.**\n\n**A1:** Though DIFFUSIONHOI contains a larger count of parameters, this does not incur heavy consumption in computation resources compared to existing work.\nFirst, the training time ($\\textit{i.e.}$, 5.7 hours for relation embedding learning and 11.5 hours for HOI detection learning) is more efficient than most existing work, which can be evidenced by the comparison of training time below. Second, as the discussion provided in L321-330 and the results summarized in Table 4, the inference time of DIFFUSIONHOI is faster than the two-stage HOI detectors, and comparable to the one-stage HOI detectors without a significant gap.\nBased on the analysis above, we believe we render a fair comparison to existing work.\n\n|Method|Time (Hour)|\n|:-|:-|\n|CDN[39]|25.2|\n|HOTR[36]|23.6|\n|UPT[76]|17.9|\n|STIP[77]|16.4|\n|GEN-VLKT[2]|28.4|\n|HOICLIP[18]|29.1|\n|CQL[80]|29.7|\n|DIFFUSIONHOI|$\\textbf{5.7+11.5}$|\n\nMore importantly, as stated in L75-77, one valuable contribution of this work is the effective knowledge transfer from diffusion models without heavy fine-tuning. We address this challenge by learning task-oriented embeddings ($\\textit{i.e.}$, relational embeddings) as textual prompts, and further transfer the knowledge from relation-driven sample generation and feature extraction two aspects.\nConsidering the current trend towards leveraging large-scale pre-trained models to assist in various downstream perception tasks, we believe this work could offer valuable insights to the border community, and thus the larger count of parameters cannot be considered as the drawback of our work.\n\n---\n\n**Q2: Comparsion to [79].** \n\n**A2:** We provide the comparison between DIFFUSIONHOI with VQGAN and PViC with ResNet-50 as the backbone in Table 1. \nFor PViC with Swin-L as the backbone, it leverages $\\mathcal{H}$-Deform-DETR as the detector which achieves 48.7 mAP on MS COCO by running merely 12 epochs, significantly higher than DETR which achieves 36.2 mAP by running 50 epochs.\nFor a fair comparison, we i) reimplement PCiV with ViT-L as the backbone and DETR as the detector, and ii) augment the two parallel decoders for instance and interaction detection of DIFFUSIONHOI with the technology used in $\\mathcal{H}$-Deform-DETR, resulting in DIFFUSIONHOI with the decoder holds a similar detection ability to $\\mathcal{H}$-Deform-DETR. The detailed results are summarized below.\n\n|Model| Backbone| Detector| Full|Rare|Non-rare|AP$^{S1}_{role}$|AP$^{S2}_{role}$|\n|:-|:-|:-|:-|:-|:-|:-|:-|\n|PViC|ResNet-50|DETR| 34.69 |32.14 |35.45|62.8 |67.8|\n|DIFFUSIONHOI|VQGAN|$\\approx$DETR|$\\textbf{38.12}$|$\\textbf{38.93}$|$\\textbf{37.84}$|$\\textbf{66.8}$ |$\\textbf{70.9}$|\n|PViC|ViT-L|DETR| 39.89| 40.36|39.98|63.4|68.9|\n|DIFFUSIONHOI|ViT-L|$\\approx$DETR|$\\textbf{42.54}$|$\\textbf{42.95}$|$\\textbf{42.35}$|$\\textbf{67.1}$|$\\textbf{71.1}$|\n|PViC|Swin-L|$\\mathcal{H}$-Deform-DETR|44.32 |44.61| 44.24|64.1| 70.2|\n|DIFFUSIONHOI |ViT-L|$\\approx$$\\mathcal{H}$-Deform-DETR|$\\textbf{45.46}$|$\\textbf{45.78}$|$\\textbf{45.37}$|$\\textbf{67.9}$|$\\textbf{71.5}$|\n\nAs seen, for models with similar backbones, DIFFUSIONHOI achieves much better performance. For instance, 38.12 $\\textit{vs.}$ 34.69, and 42.54 $\\textit{vs.}$ 39.89.\nOn this basis, as observed in the last two rows of the table above, though ViT-L is inferior to Swin-L in downstream tasks, DIFFUSIONHOI still surpasses PViC by 1.14 mAP on the Full set of HICO-DET and 3.8/1.3 mAP on the Scenario1/Scenario2 setup of V-COCO, respectively.\n\nTable 1 and $\\S$4.2 will be updated accordingly.\n\n---\n\n**Q3: Quantitative results and analysis of failure cases.**\n\n**A3:** Thank you for pointing this out. We will include the quantitative results regarding HOI detection in the Appendix for detailed analysis. Additional quantitative results for relation-driven sample generation have already been provided in Figure S1. \n\nRegarding failure case analysis, the visualized results are provided in the PDF attached with \"global\" response.\nAfter reviewing the predictions delivered by DIFFUSIONHOI, we found that failure cases primarily manifest in the following scenarios: i) scenes featuring only partial human bodies, such as arm or leg, which introduces challenges for person detection; and ii) chaotic scenes teeming with people, which causes occlusion and difficulties in identifying interactions.\nDespite these challenges, our algorithm has shown remarkable improvement over existing approaches. Additionally, the patterns of these failure cases provide valuable insights for future research. \n\n---\n\nThank you for your valuable time and constructive feedback. We will revise the main text with discussions and experimental results according to your review. It is welcome if there are any further questions or suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**To all reviewers**

We express our sincere gratitude to all reviewers for their valuable time and thorough assessment of our manuscript. In response, we have carefully addressed each concern raised, and provided point-to-point clarifications which shall be integrated into the new version of our manuscript.

We are gratified by the positive feedback from all reviewers, particularly regarding the significant improvement in performance, the quality of presentation, and the comprehensiveness of our experiments.

Foremost among the concerns is the comparison to top-leading approaches. We provide additional experiments to ensure an all-inclusive comparison, of which our method can still achieve the best performance.

Additionally, Reviewer MQdd raises concerns about the training efficiency of DIFFUSIONHOI. We appreciate this feedback and have included a detailed analysis regarding training costs to clarify this issue. 
Similarly, Reviewer UENt and Reviewer Xw3E highlight concerns about the parameter count of our approach. In response, we have provided additional experiments and discussions to clarify the motivation and contribution of using diffusion models for downstream tasks.

Other questions, such as the complexity of the technical design of our approach, the differences from simply adding training data, more details about the diffusion model used in this work, and analysis on failure cases, have also been addressed accordingly.

Meta-review: The paper introduces DIFFUSIONHOI, a novel method for Human-Object Interaction (HOI) detection utilizing text-to-image diffusion models to improve visual cue recognition and compositional reasoning. The approach demonstrates significant performance improvements across multiple datasets, including HICO-DET and V-COCO. Despite concerns about training efficiency, parameter count, and technical complexity, the authors provided comprehensive analyses and additional comparisons, reinforcing the method's effectiveness. Reviewers acknowledged the innovative integration of diffusion models and clear presentation, although they noted the architecture's complexity and suggested further exploration of failure cases. Overall, the paper is recommended for acceptance.
Decision: Accept (poster)