ReCorD: Reasoning and Correcting Diffusion for HOI Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 11:24:43

Abstract: Diffusion models revolutionize image generation by leveraging natural language to guide the creation of multimedia content. Despite significant advancements in such generative models, challenges persist in depicting detailed human-object interactions, especially regarding pose and object placement accuracy. We introduce a training-free method named Reasoning and Correcting Diffusion (ReCorD) to address these challenges. Our model couples Latent Diffusion Models with Visual Language Models to refine the generation process, ensuring precise depictions of HOIs. We propose an interaction-aware reasoning module to improve the interpretation of the interaction, along with an interaction correcting module to refine the output image for more precise HOI generation delicately. Through a meticulous process of pose selection and object positioning, ReCorD achieves superior fidelity in generated images while efficiently reducing computational requirements. We conduct comprehensive experiments on three benchmarks to demonstrate the significant progress in solving text-to-image generation tasks, showcasing ReCorD's ability to render complex interactions accurately by outperforming existing methods in HOI classification score, as well as FID and Verb CLIP-Score.

Main paper: 
 introduction: 
 In recent years, diffusion models have become a cornerstone in the field of multimedia processing, demonstrating remarkable success across a wide array of generative tasks [26, 35, 53, 61, 63, 71] . Among them, text-to-image (T2I) generation has attracted significant attention [8, 12, 40, 54, 62] due to its user-friendly nature leveraging natural language guidance. While state-of-the-art (SOTA) T2I diffusion models, such as SDXL [42] and DALL-E 3, have greatly improved image realism and expanded the conceptual possibilities of generation, ongoing challenges persist. In particular, they often encounter difficulties with text prompts that contain intricate human-object interactions (HOI) [23] .As illustrated in Figure 1 , despite their training on extensive datasets, T2I methods like SDXL and DALL-E 3 exhibit flaws in rendering human poses or object placements. In the example of "a boy is lying on a bench", although DALL-E 3 precisely captures the lying pose, it incorrectly locates the boy, whereas SDXL places the boy on the bench accurately but does not manage to render the lying down pose. These inaccuracies may stem from the inherent biases or assumptions about the interaction between the given human and object, embedded within the large-scale datasets [50] used to train T2I models [16] . Such biases can lead to hallucination problems [34] , resulting in models failing to generate images matching the intended interactions accurately. For instance, given the prompt "a man is carrying a bicycle" as in Figure 1 , SDXL and DALL-E 3 might err in posture and object placement because the most common association in the datasets is "riding," leading to inaccuracies in depicting the intended interaction.To enhance the accuracy of T2I models in generating interactions, a possible avenue is to ensure the correct positioning of both the human and the object within the image. Previously, several layout-to-image (L2I) models [13, 27, 29, 59, 69] have been proposed to include the layout of each object as additional input for diffusion models, aiming to gain more precise control over the output images. For instance, GLIGEN [29] retrain the model with the layout-annotated dataset by the Gated Self-Attention. DenseDiffusion [27] and BoxDiff [59] exemplify training-free L2I methods that necessitate supplementary inputs for operation. However, the requirement for user-specific layouts can be time-consuming and inconvenient for users. In addition, especially in scenarios involving HOI, simplistic inputs like boxes prove inadequate in capturing complex attributes such as posture and body orientation, which are crucial for the accurate depiction of interaction, ultimately resulting in suboptimal images. The deficiency is evident in Figure 1 , where BoxDiff struggles to generate realistic interactions despite being provided with layout information as additional input.On the other hand, alternative approaches have integrated Large Language Models (LLMs) to augment diffusion models, aiming better to grasp the nuances of textual prompts in image generation [15, 30, 43, 58, 60, 70] . Innovations such as LMD [30] and LayoutLLM-T2I [43] have pioneered to employ LLMs for creating more intuitive and accurate image outputs. LMD utilizes a dualphase approach, initially using a pre-trained LLM to create a scene layout with captioned bounding boxes. It further proceeds with a layout-grounded controller to guide diffusion models. Additionally, LayoutLLM-T2I starts by generating a coarse layout and then implements a specially trained transformer module within the denoising UNet for fine-grained generations. Despite the progress made with LLM-assisted methods in image generation, a pivotal limitation arises when handling HOI. Predominantly dependent on textual prompts, these methods exhibit two possible shortcomings. Firstly, they may overlook the intricate spatial dynamics and nuanced interactions within an image due to the limited information the prompt provides. Secondly, in some cases, the LLMs may over-analyze the textual prompt, leading to hallucinations where they generate fabricated content that is not grounded in reality. An evident illustration of this phenomenon is presented in the first row of Figure 1 . Despite the prompt suggesting a man carrying a bicycle, the LLM-assisted LayoutLLM-T2I tends to overanalyze and assume an additional man riding the bike instead, resulting in the generation of human-like artifacts riding the bicycle, which is not the intended interaction. Such deficiency underscores the critical need for advanced approaches to interpret information from the concise HOI prompts better, ensuring image generation that closely aligns with the intended interactions.In this paper, we propose Reasoning and Correcting Diffusion (ReCorD) for generating human-object interaction. We argue that proper HOI generation requires both the correct human posture and precise object positioning to facilitate realistic interaction. To achieve this, we introduce an innovative pipeline depicted in Figure 2. Our approach includes three key steps. First, we employ the Latent Diffusion Model (LDM) to produce a set of human candidates performing the verb in the text prompt, emphasizing the correct posture by employing intransitive prompts. Next, as Visual Language Models (VLMs) excel at comprehending image contents, we harness their visual reasoning capabilities to select candidates with optimal posture and determine the appropriate placement of the object based on their interpretation of the interaction scene. Finally, we introduce a refinement mechanism to adjust object positions while preserving accurate human posture. By implementing inverse attention masks and bounding box constraints, we prevent the overlap of attention maps between humans and objects during image generation, enhancing the fidelity of the final image output. Our proposed ReCorD guarantees precise control over the depicted interactions, effectively mitigating the risks of hallucinatory inaccuracies. To the best of our knowledge, this work is the first attempt to introduce interaction generation in a training-free fashion which eliminates the need for additional HOI-labeled data and avoiding computational challenges associated with training. We summarize our main contributions as follows:(  
related work 2.1 conditioned t2i diffusion models: 
 Recent advancements in diffusion models [9, 11, 21, 25, 36, 52] have significantly improved the capabilities of large-scale T2I generation models such as DALL-E [45, 46] , Imagen [49] , and Stable Diffusion [42, 48] . While these models guided solely by plain text demonstrate the ability to generate high-quality images, they struggle with prompts that demand detailed attribute specifications and a nuanced understanding of spatial relationships [47] . Consequently, recent research on diffusion models extended beyond text-based conditions and incorporated advanced conditioning mechanisms such as inpainting masks [27] , sketches [56] , key points [29] , segmentation maps [10] and layouts [16] , facilitating enhanced spatial manipulations. The modification of models marks this evolution to include additional encoders, achieved through strategies like fine-tuning [3, 29, 39, 65] , or constructing new models from scratch [24] . For instance, SpaText [3] and GLIGEN [29] introduce spatial modulations into pre-trained models, employing fine-tuning with adapters to perform layout constraints. Despite the enhancements, the requirement for model retraining for each new condition type remains a significant challenge. Building on the foundation of the previous diffusion models, we propose a training-free framework designed explicitly to enhance the proficiency of LDM in interpreting and visualizing the intricate relationships of HOIs.As T2I models are traditionally trained on datasets featuring brief text captions, they frequently struggle to capture the nuances of more complex captions that contain multiple phrases [51] . Following the insight from Prompt-to-Prompt [18] that trained T2I models inherently provide token-region associations through their attention maps, several works [2, 4, 7, 16, 22, 27, 59 ] have been proposed to mitigate this issue. For instance, MultiDiffusion [4] choose to conduct individual denoising procedures for every phrase, respectively, at every timestep. However, this independent generation technique frequently stumbles over lifelike compositions and is easily hindered by bias toward specific actions or objects. Attendand-Excite [7] strategically manipulates the noise map to enhance the activation of previously overlooked tokens in cross-attention maps. Yet, a limitation arises since the mere intensification of attention to certain tokens does not always lead to a holistic representation of the intended information within the generated output. BoxDiff [59] comes up with three spatial constraints to optimize the cross-attention layers given instinctive inputs, e.g., bounding box or scribble, during the sampling process. InteractDiffusion [22] introduces a novel approach to generating images with precise HOI by tokenizing interaction using a conditioning self-attention layer for the accuracy of the complexities of interaction representation. To better impose regulations on the generated human, the rectangular-shaped constraints cannot be applied since bounding boxes cannot properly convey the details of human action, such as body orientation, facial direction, etc. Building on these trainingfree approaches, our pipeline assures that the object is matched with a sufficiently strong attention map and the human can be generated in a legitimate pose. Thus, a complex scene containing interaction can be precisely rendered.The integration of Large Language Models (LLMs) with diffusion models has significantly transformed T2I generation, capitalizing on the superior generalization abilities of LLMs [14, 30, 31, 58, 60, 66] .LayoutGPT [14] adopt LLMs for layout generation via in-context learning. VisorGPT [60] takes one step further by fine-tuning to embrace diverse modalities, including key points, semantic masks, etc. LMD [30] represents pioneering efforts in this integration, utilizing LLMs to interpret object locations from text prompts, thus enhancing the accuracy and quality of generated images. LayoutLLM-T2I [43] query ChatGPT for text-to-layout induction and introduce a Layout-aware Spatial transformer with a view to improve layout and image generation simultaneously. Emphasizing the central role of LLMs, these advancements bypass the need for additional information inputs, allowing LLMs to shape the initial layout configurations and interpret user prompts directly. Despite the successful outcomes of using LLMs, they fail to address the challenge of generating the explicit posture of humans given specific actions.Compared to LLMs, LDMs demonstrate a better understanding of scene kinetics based on simple text prompts. Therefore, we propose to leverage such advantage of LDMs alongside the robust visual reasoning abilities of VLMs to generate accurate HOI. 
method: 
 We introduce the ReCorD, an interaction-aware model that maintains training-free superiority. The resulting images hinge on the human adopting the appropriate pose and ensure the object is located in a suitable position according to the given text prompt. Our generative pipeline comprises three modules: Coarse Candidates Generation, Interaction-aware Reasoning, and Interaction Correcting. To abbreviate these modules, we term them as M 𝑔 , M 𝑟 , and M 𝑐 , respectively. We decomposed the denoising process 𝑇 into two stages, i.e. 𝑇 1 and 𝑇 2 , by observing that the diffusion model captures the initial layout during the early denoising steps and refines the details in the later iterations [4] . In the former stage, M 𝑔 generates 𝑘 coarse candidates, while M 𝑟 suggests the ideal pose and layout w.r.t. the text prompt. Subsequently, M 𝑐 corrects object locations while preserving selected poses to refine the cursory images into desired ones. Importantly, ReCorD empowers the diffusion model to create images aligned with text prompts, highlighting complex spatial conditions and intricate interactions without additional training. The overall pipeline is depicted in Figure 2 , and we elaborate on the details of each module in the following sections.Given a text prompt 𝑦 containing the HOI attribute, i.e., a statement describes an interaction between a single human and an object phrased as "a subject is verbing an object", we enhance interaction representation by adopting distinct attention mechanisms [55] within M 𝑔 . More precisely, we manipulate cross-attention and selfattention maps to generate candidate images 𝑘 associated with the action subject to the prompt. EQUATIONwhere Q = 𝜙 𝑞 (𝜑 (𝑧)) and K = 𝜙 𝑘 (𝜓 (𝑦)) represent the query and key embeddings derived by corresponding projection functions. 𝜑 and 𝜓 are spatial normalization [68] and the text encoder from CLIP [44] , which yields intermediate representations and𝑁 text tokens 𝜓 (𝑦) = {𝑤 1 • • • 𝑤 𝑁 }.For simplicity, we omit the subscript 𝑡 that represents the denoising step while manipulating attention maps.Modeling the cross-attention map for the interaction (verbing in the 𝑦) is challenging when generating HOI scenes, leading to an ambiguous representation of the verb token. To address this issue, we propose an alternative intransitive prompt ỹ, which typically excludes object-related descriptions in 𝑦. The cross-attention maps Ã using ỹ can be derived by substituting K in eq. ( 1) with K = 𝜙 𝑘 (𝜓 ( ỹ)). As illustrated in Figure 3 , Ã captures more informative clues, especially for the verb token, compared to A when using ỹ as the text prompt, resulting in interactive representations. Consequently, we formulate the final cross-attention maps by rearranging the maps as follows:EQUATIONwhere 𝑛 denotes the index of text tokens. Ideally, we embrace the attention maps if the text token exists in the intransitive prompt.Self-Attention Maps Manipulation. In contrast to cross-attention maps, self-attention maps lack direct token associations but still influence the spatial layout and appearance of generated images [33] .Therefore, we manipulate the self-attention maps similarly to eq. ( 2) for latent representation once the denoising step 𝑡 > 𝛾 for obtaining A 𝑠𝑒𝑙 𝑓 , where 𝛾 is a predefined parameter ensuring that scenes and objects from original tokens 𝜓 (𝑦) can be generated effectively.As an intermediate module bridging the others, we present the Interaction-Aware Reasoning module M 𝑟 (see Figure 4 ) following the generation of coarse candidates in M 𝑔 . This module comprises two components powered by VLM: the Pose Selection Agent and the Layout Agent. Specifically, the Pose Selection Agent selects an image aligning with the prompt 𝑦, while the Layout Agent adjusts the object's location and preserves human key points P and further determines the target position b𝑜 for the correction module M 𝑐 .Pose Selection Agent. As the pose plays a vital characteristic in the HOI generation, we first couple an agent to select the appropriate pose conditioned on the prompt. The Pose Selection Agent integrates the initial prompt 𝑦 with previously generated candidates to create the pose template. Leveraging the visual comprehension capabilities of VLMs, this agent excels in identifying the precise pose corresponding to 𝑦, enhancing the model's ability to interpret visual data beyond relying solely on textual cognition as in LLMs. This pivotal step ensures that the pose information initially obtained from LDMs is meticulously refined for subsequent phases. Layout Agent. To address the issue of LLM-assisted methods being overly reliant on prompts for sampling layouts, we incorporate the identified key points P and the bounding box 𝑏 ℎ for humans as additional data. Recognizing that an interaction involves both the relation to human and the object, we collect the crucial information of P using 33 key points in (𝑥, 𝑦) coordinates and represent 𝑏 ℎ = (𝑥 𝑚𝑖𝑛 , 𝑦 𝑚𝑖𝑛 , 𝑥 𝑚𝑎𝑥 , 𝑦 𝑚𝑎𝑥 ). Additionally, we use the image selected by the former agent as inputs to VLMs for layout suggestion tasks. We first extract the object's location 𝑏 𝑜 using Otsu's algorithm [41] , an automatic thresholding technique applied to the object's crossattention map A 𝑐𝑟𝑜𝑠𝑠 to isolate regions with higher values. Subsequently, we detect human key points using MediaPipe Pose Landmarker to create the segmentation mask 𝑚 ℎ . Consecutively, we establish a series of guidelines and fixed protocols for VLMs adherence, including constraints on 𝑏 ℎ to maintain the integrity of the intended human poses and an overlapping reduction strategy to improve the quality of generated images containing multiple objects. Furthermore, inspired by the Chain-of-Thought approach [57] , we enhance the logical coherence by guiding VLMs to construct visual attribute information for human posture. We ground the VLMs in logical reasoning across multiple factors such as pose types, body orientation, object relations, etc. Drawing on insights from previous research [5, 37] , we prepare VLMs with three examples, aiding in the clarification of visual representation and reducing ambiguity to construct the interaction template. Eventually, we extract the proposed location b𝑜 for M 𝑐 and integrate a checking mechanism to determine whether the alteration in b𝑜 falls within a predetermined threshold. If the change is minimal which indicates a minor difference, M 𝑟 would signal no changes to M 𝑐 . This mechanism is crucial for maintaining a streamlined and resource-efficient generation process, ensuring only significant location adjustments prompt further action. Please refer to the supplementary material for the details of pose and interaction templates.We delicately refine the candidate image provided by the dual agents while preserving the original human pose in M 𝑐 , as shown in Figure 5. To combine the generative capabilities of LDMs with the reasoning abilities of VLMs, we incrementally update the latent 𝑧 𝑡 to adjust the object's position and size based on bounding boxes b𝑜 related to the interaction. Notably, we conduct the denoising process for 𝑡 ∈ [𝑇 2 , • • • , 0], including modulation of cross-attention and self-attention maps, as described in Section 3.1.Simultaneously modifying the object's location requires consideration of potential overlap with the human body since crossattention maps from different tokens may exhibit strong values in the same region which will deteriorate image quality. To address this challenge, we introduce a mechanism for eliminating attention overlap. Specifically, given the token index of the object denoted as 𝑚, we use the cross-attention map 𝐴 𝑚 to construct an inverse mask at each time step 𝑡, denoted as Ā𝑚 = 1 -𝐴 𝑚 , where 1 is the tensor of the same dimension as 𝐴 𝑚 containing all elements equal to 1. This inverse attention map is then applied to the remaining maps using an element-wise product operation defined asEQUATIONThrough this correction mechanism outlined in eq. ( 3), we can mitigate the issue of attention overlap between humans and objects while updating the object positions, ensuring the successful generation of updated objects. Conditioned Spatial Constraints Since our ReCorD is trainingfree and does not involve additional learnable networks for knowledge transfer, we employ box constraints [59] to regularize the denoiser, which can be formulated asEQUATIONwhere each term in sequence order represents the inner-box, outerbox, and corner constraint, respectively. We apply eq. ( 4) to update the latent at each time step 𝑡 with corresponding weight 𝛼 𝑡 as ẑ𝑡 ← 𝑧 𝑡 -𝛼 𝑡 • ∇L.(EQUATIONThrough slight update 𝑧 𝑡 at each step, we ensure that the object holds sufficient mutual information with the box region and conforms to the specified size, i.e. b𝑜 , thereby accurately correcting the object's position to represent the interaction. As LDM aims to denoise iteratively and involve the attention maps as intermediate. We denote that LDM(𝑧 𝑡 , 𝑦, 𝑡, 𝑠) is the diffusion process at time step 𝑡 before manipulation, which seeks the corresponding attention maps. After going through our proposed ReCorD, the denoising UNet is reused to predict the latent representation at next step. Formally LDM( ẑ𝑡 , 𝑦, 𝑡, 𝑠) symbolizes the diffusion model that adopts the manipulated attention maps, resulting in the prediction, i.e. 𝑧 𝑡 -1 . The complete algorithm can be found in the supplementary.  
experiments 4.1 experimental setup: 
 Datasets. Given the absence of a standard benchmark crafted for HOI generation, we assess the efficacy of our approach by extracting HOI triplets from two established HOI detection datasets, namely HICO-DET [6] and VCOCO [17] , to form the input text prompts. HICO-DET includes 600 triplets across 80 object categories and 117 verb classes, while VCOCO contains 228 triplets, spanning 80 object classes and 29 verb types. For a comprehensive assessment, we incorporate the non-spatial relationship category in T2I-CompBench [23] , which is characterized by 875 interaction terms. We select prompts that exclusively involve HOIs in T2I-CompBench.To enhance diversity, we apply random subject augmentation to each verb and object pair extracted from the datasets to form the input prompt. Accordingly, our experiments are conducted across three datasets: HICO-DET, with 7,650 HOI prompts; VCOCO, contributing 2,550 prompts; and the non-spatial relationship category of T2I-CompBench, adding 465 prompts. More details are described in the supplementary.Baselines. We report comparisons with nine strong-performing models, 1) T2I models: Stable Diffusion (SD) [48] , Attend-and-Excite (A&E) [7] , SDXL [42] , and DALL-E 3. 2) L2I models: BoxDiff [59] , MultiDiffusion [4] and InteractDiffusion [22] . 3) LLM-assisted T2I models: LayoutLLM-T2I [43] and LMD [30] . We utilized the official implementations and the default settings for each baseline. For L2I models, we provide the actual bounding box data from HICO-DET and VCOCO datasets in addition to the text prompts as inputs. For LLM-assisted methods, the input layouts are exclusively generated by LLMs, rather than being sourced from the datasets. Evaluation Metrics. To measure the interaction in the generated images, we utilize the CLIP-Score S CLIP [19] evaluating the similarity between the input text and the generated images. While this metric is commonly applied to estimate fidelity to text prompts, we note its inclination towards a noun or object bias, with CLIP often unable to differentiate among verbs, relying instead on nouns [38, 64] . To address this, we specifically extract verbs from the text prompts and calculate the Verb CLIP-Score S 𝑣𝑒𝑟𝑏 CLIP . In addition, we introduce a HOI classification score to evaluate the interaction depiction. By transforming a pre-trained, SOTA HOI detector [67] into a classifier, we evaluate HOI instances in generated images and compare them against the ground truth of HICO-DET and VCOCO. The accuracy of HOI classification is evaluated based on the top three accuracy scores. HOI Full and HOI Rare represent scores for the full and rare set, respectively, on the HICO-DET dataset. The rare set is selected based on having fewer than 10 instances across the dataset. Moreover, we employ the Fr échet Inception Distance (FID) [20] and PickScore [28] to assess image quality. FID compares the Fr échet distance distribution of Inception features between real and generated images, whereas PickScore, a text-image scoring metric, exceeds human performance in predicting user preferences. Implementation Details. We choose the Stable Diffusion [48] model as the default backbone and GPT4V [1] as the VLM in M 𝑟 . We set the ratio of classifier-free guidance to 7.5, denoising steps 𝑇 1 = 10, 𝑇 2 = 50, and use the DDIM [52] scheduler within denoising steps. The number of coarse candidates 𝑘 = 5, and the hyperparameter 𝛾 = 5 initials the operation for self-attention maps manipulation. For evaluation, one image is generated per triplet for HICO-DET and VCOCO datasets and three images per triplet for T2I-CompBench. We provide a qualitative comparison to assess the generated HOIs. As depicted in Figure 6 , ReCorD outperforms other SOTA methods by generating realistic human poses and object placements that align with text prompts, proofing its proficiency in depicting object interactions with high fidelity. In contrast, baseline methods often tend to misplace objects or fail to capture the nuances of intended actions. For L2I models with additional layout inputs, BoxDiff achieves object size requirements but struggles to accurately depict interaction poses; InteractDiffusion fails to accurately portray subtle activities despite the fine-tuning, as seen in (a), (d), (e), and (f); Mul-tiDiffusion strives for precise object placement despite generating images in various sizes. On the other hand, LayoutLLM-T2I, despite leveraging language models for improved layout generation, often produces objects disproportionate to humans, which is evident in (e) and (f). Furthermore, the tendency of the refiner module in SDXL as well as LMD to prioritize nouns over verbs compromises their interaction depiction capabilities. Especially, SDXL struggles with action poses (a), (b), (d), and (e), and DALL-E 3 with object sizing and placement (a), (c), (e), and (f), illustrating key areas where ReCorD advances beyond the limitations of existing solutions.We present the quantitative comparison of the generated results, with prompts sourced from HICO-DET and VCOCO in Table 1 , and results using prompts formed from T2I CompBench in Table 2 . CLIP-based Image-Text Similarity. The results of CLIP-Score S CLIP reveal that our ReCorD outperforms other methods on HICO-DET and T2I-CompBench, and it is comparable to MultiDiffusion on VCOCO. Furthermore, our ReCorD achieves the best results across all three datasets in terms of Verb CLIP-Score S 𝑣𝑒𝑟𝑏 CLIP , this confirms our ability to generate more closely matched interactions. Evaluation of Image Quality. As per PickScore evaluation, our ReCorD model is comparable with the SDXL model and outperforms other methods. This demonstrates that after incorporating our designed Interaction-Correcting Module with SD models. Our ReCorD can maintain the image generation quality of the model while achieving more authentic interaction. Furthermore, when comparing the generated images with real ones in HICO-DET and VCOCO datasets using FID scores, our model outperforms other methods except InteractDiffusion. Notably, given that InteractDiffusion is fined-tuned using HICO-DET and COCO datasets [32] , the performance of our ReCorD is particularly remarkable as it operates without the need for training or additional HOI data. Evaluation of Interaction Accuracy. Table 1 validates our method significantly enhances the accuracy of HOI generation in both datasets, revealing the efficacy in synthesizing more precise HOI.We conduct a user survey with 95 participants who are asked to evaluate images generated by baseline methods and our ReCorD based on four criteria: Interaction, Posture, Position&Size, and Overall Quality. We generated eight distinct images for each of 24 randomly selected prompts from the HICO-DET and VCOCO datasets, using methods including A&E, LMD, SD, SDXL, MultiDiffusion, InteractDiffusion, BoxDiff, and ReCorD. According to the results represented in Figure 7 , ReCorD emerged as the clear favorite, garnering between 60% to 80% of the votes across the criteria, indicating its effectiveness in accurately generating the correct postures and positioning objects within reasonable areas, which are key factors ensuring the accurate depiction of interactions. The results of the user study illustrate that our ReCorD can better align with human understanding of HOIs, underscoring its advanced capability to not only recognize but also precisely delineate the nuances of HOIs.For the ablation studies, Figure 8 displays the HOI generation results of the SD incorporating modules in our ReCorD : (a) with only SD, (b) with SD and M 𝑔 , and (c) with SD and M 𝑔 +M 𝑟 +M 𝑐 . With only SD, the generated outputs appear to be suboptimal, likely influenced by biases inherent in its training data, leading to misinterpretation of the intended interaction described in the text prompts. From the results in (b), it shows that with the inclusion of M 𝑔 , the accuracy of the generated human action is significantly improved due to our intransitive prompt altering technique, highlighting that simplifying the prompt to focus on the core action enables the model to generate the intended human poses with enhanced precision. However, it still struggles to accurately position objects in relation to humans in the images, resulting in a mistaken interaction. For our ReCorD, M 𝑟 helps select appropriate poses and retain suitable candidates while M 𝑐 refines images to obtain accurate interaction with the correct object size and position while maintaining the selected pose. As a result, HOI generation of our complete pipeline in (c) exemplifies the most successful outcomes achieved.  
conclusion: 
 We have introduced ReCorD framework, tailored explicitly for HOI image generation. This method comprises three interaction-specific modules that synergistically interact with each other. Our core idea revolves around reasoning layout and correcting attention maps using VLM-based agents and an LDM to address this challenge. Extensive experiments demonstrate the effectiveness of our approach in enhancing image accuracy and semantic fidelity to the input text prompts, particularly in capturing intricate concepts of interactions that several baseline generative models struggle with. Additionally, we quantify our improvements through various protocols and a user survey focused on HOI generation, providing valuable insights and paving the way for future explorations in this domain.  
 
 Appendix:  


Decision: This paper presents an approach to T2I generation by integrating diffusion models with VL models to tackle the intricate challenge of human-object interaction (HOI). By utilizing LDMs for generating a variety of pose options and VLMs for selecting the optimal poses and object placements based on visual cues and textual descriptions, the proposed method ensures the fidelity and contextual coherence of generated images.

The paper receives the initial rating of four borderline accept. After the rebuttal, two reviewers raise the rating to weak accept (4 4 5 5). The AC agrees with the reviewers' comments that this paper provides interesting insights to HOI generation.

Reviews: review: paper_summary: paper_summary: Diffusion models revolutionize image generation by leveraging natural language to guide the creation of multimedia content. Despite significant advancements in such generative models, challenges persist in depicting detailed human-object interactions, especially
regarding pose and object placement accuracy. We introduce a training-free method named Reasoning and Correcting Diffusion (ReCorD) to address these challenges. Our model couples Latent
Diffusion Models with Visual Language Models to refine the generation process, ensuring precise depictions of HOIs. We propose an interaction-aware reasoning module to improve the interpretation of the interaction, along with an interaction correcting module
to refine the output image for more precise HOI generation delicately. Through a meticulous process of pose selection and object positioning, ReCorD achieves superior fidelity in generated images
while efficiently reducing computational requirements.
strength_weakness: strength_weakness: 1） a novel reasoning framework that integrates LDM with VLMs to overcome the challenges of generating realistic HOI, mitigating issues presented in previous approaches, such as LLMs overanalyzing simple text prompts and training data biases in LDM.
2） a correction mechanism within LDM for dynamic image adjustment, enabling precise control and refinement of human interactions in generated images as well as enhancing
the portrayal accuracy significantly.
limitations: limitations: 1）The image quality scores are not state-of-the-art
2)  The visual effect is not unreal
paper_summary: paper_summary: This paper introduces a framework, ReCorD, aimed at generating Human-Object Interaction (HOI) images. The primary innovation of this framework lies in its integration of the Latent Diffusion Model (LDM) with the Visual Language Model (VLM) to handle complex spatial conditions and intricate interactions without additional training. The LDM generates a set of human figures as coarse candidates. The VLM selects candidates with optimal postures and determines the precise placement of objects within the scene. The correcting module adjusts object positions accurately while maintaining the integrity of human postures. Both qualitative and quantitative experiments demonstrate the framework's superiority over existing methods in certain aspects.
strength_weakness: strength_weakness: 1. The authors innovatively integrate the VLM and LDM for generating realistic HOI. With object bounding box refinement, it enables precise control and refinement of human interactions in generated images. No training is required, which enhances its usability and scalability.
2. The experimental design is relatively comprehensive, with a selection of quantitative metrics, multiple visual comparisons showcased through various case studies, and user studies conducted, validating the effectiveness of the method from multiple perspectives. 
3. The article is well-written, with clear descriptions and aesthetically pleasing and informative visual representations.
limitations: limitations: 1.	Visual effects are often the most direct way to assess the capabilities of a generative model. However, compared to other methods, the visual effects presented in this paper mostly feature dark images. The authors did not explain it, leaving it unclear if there are specific reasons for this phenomenon. Are there any better examples?
2.	[59] is the key point in the correcting module, thus, the refined object bounding box is very important. The bounding box replacement should be illustrated in detail, and the connection with the visual attributes should be reported. 
3.	In the quantitative experiments, the difference between the proposed method and its backbone is subtle. In table 1, “Ours” is better than SD but worse than SDXL. Does this mean the HOI challenge could be handled by upgrading model scales and training data. 
4.	The proposed method involves a set of candidate generation, attention map manipulation, auxiliary modules, etc. The total generation speed and memory usage should be reported.
paper_summary: paper_summary: The *ReCorD* proposed in the paper is to make HOI generation more reasonable. LLM selects the most appropriate pose from the coarse noisy images generated with object-related descriptions. Using the layout suggested by LLM, attention maps are corrected, after which the HOI image is synthesized. Fruitful experiments show the capability of the framework in HOI generation objectively.
strength_weakness: strength_weakness: 1. A novel framework for reasoning HOI generation is proposed in the paper. The reasoning layout is suggested by LLM with the heuristic of input candidates, which simultaneously enhances the efficiency of the LLM rather than directly generating it.

2. The paper offers a framework for LLMs to operate the attention map during the procedure of LDMs, which is remarkable. The framework generally outperforms the others in metrics ranging from CLIP-Score to FID.

3. A specific dataset is constructed for HOI evaluation and extensive ablation studies have been implemented in the paper, which demonstrates the efficiency of the *ReCorD*.

4. The paper is clearly written and is easy to follow. Good paper. Source codes are provided in supplementary material which is commendable.
limitations: limitations: 1. I find that from the results shown in the paper, the images generated by *ReCorD* always have a lower brightness than the others. However, this result is from the previous research where token-level amending results in less fruitful color. The discussion of this phenomenon would increase your contributions and innovations.

2. More ablations could be implemented to compare the ability of different LLMs or MLLMs in layout suggestions.

3. GPT4-V can be prompted with designed heuristics to play the role of the HOI generation evaluator.
paper_summary: paper_summary: The paper focuses on the inaccurate representation of prompt content in existing diffusion model-based generative models. It proposes a paradigm based on the decoupling and reconstruction of internal relationships within human, object, and interaction triplets. This is achieved by introducing the reasoning-capable LDM and VLM interaction to determine the optimal pose and layout represented by the prompt and provide guidance in the new generation process. Multiple indicators have proven the effectiveness of the approach.
strength_weakness: strength_weakness: 1. The paper is well-written, allowing readers to quickly understand the authors' ideas and focus.

2. The focus of the paper is meaningful because human, object, and interaction are the most common elements found in prompt texts, and a correct understanding of their relationship is essential.

3. The idea of going from coarse to fine is reasonable as it allows for minimal changes to the original layout of the image, preventing overall layout changes caused by modifications to objects or interactions.

4. The design of the Interaction-Aware Reasoning Module has a certain level of reliability and can further eliminate ambiguity.

5. The visualization results are quite impressive, surpassing the image quality of some fine-tuning based approaches to a great extent.
limitations: limitations: 1. The introduction of VLM and multiple noisy images requires further discussion on time and resource costs.

2. The process of Attention Maps Correcting can be further refined.

3. The guiding role of suggest Layout throughout the denoising time steps can be further explored.

4. For instructions used to generate images, it is necessary to consider cases with more factors in the prompt, such as the appearance of image styles, the presence of animals as subjects, new backgrounds, and other scenarios.