<figure><img src="image_1.png" /></figure>

HAL
open science

# ComposeAnything: Composite Object Priors for Text-to-Image Generation

Zeeshan Khan, Shizhe Chen, Cordelia Schmid

<figure><img src="image_2.png" /></figure>

## To cite this version:

Zeeshan Khan, Shizhe Chen, Cordelia Schmid. ComposeAnything: Composite Object Priors for Text-to-Image Generation. 2026. hal-05448488

HAL Id: hal-05448488

https://hal.science/hal-05448488v1

Preprint submitted on 8 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# ComposeAnything: Composite Object Priors for Text-to-Image Generation

Zeeshan Khan Shizhe Chen Cordelia Schmid

Inria, École normale supérieure, CNRS, PSL Research University
https://zeeshank95.github.io/composeanything/ca.html

<figure><img src="image_4.png" /><figcaption>Figure 1: The proposed *ComposeAnything* framework enables text-to-image generation for complex compositions involving surreal spatial relationships and high object counts. It enhances both visual quality and faithfulness to the input text compared to diffusion-based models (e.g., SD3 [11], FLUX [3]) and 2D layout conditioned models (e.g., RPG [56] and CreatiLayout [58]).</figcaption></figure>

## Abstract

Generating images from text involving complex and novel object arrangements remains a significant challenge for current text-to-image (T2I) models. Although prior layout-based methods improve object arrangements using spatial constraints with 2D layouts, they often struggle to capture 3D positioning and sacrifice quality and coherence. In this work, we introduce *ComposeAnything*, a novel framework for improving compositional image generation without retraining existing T2I models. Our approach first leverages the chain-of-thought reasoning abilities of LLMs to produce 2.5D semantic layouts from text, consisting of 2D object bounding boxes enriched with depth information and detailed captions. Based on this layout, we generate a spatial and depth aware coarse composite of objects that captures the intended composition, serving as a strong and interpretable prior that replaces stochastic noise initialization in diffusion-based T2I models. This prior guides the denoising process through object prior reinforcement and spatial-controlled denoising, enabling seamless generation of compositional objects and coherent backgrounds, while allowing refinement of inaccurate priors. *ComposeAnything* outperforms state-of-the-art methods on the T2I-CompBench and NSR-1K benchmarks for prompts with 2D/3D spatial arrangements, high object counts, and surreal compositions. Human evaluations further demonstrate that our model generates high-quality images with compositions that faithfully reflect the text.
# 1 Introduction

Text-to-image (T2I) models, particularly diffusion-based ones such as SDXL [45], SD3 [11] and Flux [3], have achieved remarkable success in generating individual concepts with high fidelity. However, they struggle with complex object compositions [21], especially novel arrangements that deviate from their training distribution, often resulting in unnatural mixing of objects, incorrect 2D/3D spatial positioning, and inaccurate object counts, as shown in Figure 1.

To improve compositional T2I generation, layout control has emerged as a key strategy [13, 29, 41]. This approach leverages various 2D layout representations, such as bounding boxes [29] and blobs [41], typically generated by large language models (LLMs) [42] from textual prompts. Layout-controlled T2I methods can be broadly categorized into training-based and training-free approaches. Training-based methods [58, 29, 59, 52] incorporate layout conditioning by training adapter modules over pretrained T2I models. While they provide strong spatial control, they require extensive training and often degrade image coherence and quality [60] due to rigid layout constraints, as seen with CreatiLayout [58] in Figure 1. In contrast, training-free methods mainly apply inference-time layout control [56] or latent optimization [10], to guide spatial aware generation, such as manipulating region-to-text cross-attention maps [4, 10] or employing region-wise denoising [56]. These methods better preserve image quality but offer weaker control, making it difficult to follow complex compositional instructions such as RPG [56] in Figure 1. Furthermore, all existing training-free methods rely solely on coarse 2D layouts, which not only lack 3D spatial relationships but also fail to visually represent object appearance, limiting their effectiveness in guiding T2I generation.

To address these limitations, we propose *ComposeAnything*, a novel inference-time T2I framework that enhances compositional generation without requiring model retraining. Our approach leverages LLMs with chain-of-thought reasoning to decompose the input text into a 2.5D semantic layout comprising objects and background. Each object is represented by a caption, a 2D bounding box, and a depth value to reflect relative 3D spatial relations. We then synthesize a coarse image by generating individual object images from their captions with existing T2I models [11] and arranging them spatially according to this layout. This results in a *composite object prior* that captures object appearance, count, size and 2.5D spatial positioning. To effectively utilize it we replace random noise initialization with noisy object prior and propose a *prior-guided diffusion* module that combines object prior reinforcement and spatial-controlled denoising. The prior reinforcement preserves the influence of foreground object priors in early diffusion steps, while automatically generating background to enable coherent generation. The spatial-controlled denoising further strengthens the spatial arrangement of the composite prior via mask-guided attention in early diffusion steps where global structure is determined. After these initial steps, we switch to standard diffusion to refine the image quality and coherence, achieving both faithful composition and high visual fidelity.

*ComposeAnything* outperforms state-of-the-art methods on two challenging compositional T2I benchmarks: T2I-CompBench [21] and NSR-1K [13], under automatic metrics. It achieves both high image quality and strong faithfulness to input text, as further validated by human evaluations. Ablation studies highlight the effectiveness of the composite object prior and prior-guided diffusion.

To summarize, our contributions are as follows:

(i) We introduce *ComposeAnything*, a training-free framework that enhances the compositional capabilities of existing diffusion-based T2I models by generating structured 2.5D semantic layouts and composite object priors from text using LLMs with chain-of-thought reasoning.

(ii) We propose a prior-guided diffusion method that integrates composite object priors into the denoising process via object prior reinforcement and spatially-controlled denoising in initial diffusion steps to balance faithful object compositions and high-quality image synthesis.

(iii) *ComposeAnything* enables interpretable and robust image generation, and outperforms the state of the art on two compositional T2I benchmarks, especially for surreal compositions. We will publicly release our code.

# 2 Related Works

**Compositional generation** aims to produce images that are faithfully aligned with complex texts [21, 60, 22, 25, 52, 13, 56, 53, 9, 31]. While diffusion models [45, 11, 3] have demonstrated strong
<figure><img src="image_5.png" /><figcaption>Figure 2: The *ComposeAnything* framework, which enhances text-to-image diffusion models e.g. SD3-M [11] with layouts and composite object priors for complex compositional generation.</figcaption></figure>

generative capabilities, they still struggle with compositional generation. To address this, prior work in this area can be broadly categorized into two main directions.

**Training-based methods** fine-tune pretrained diffusion models [45, 11, 3, 5, 6, 43] on large-scale datasets to improve semantic alignments [7, 57]. Some approaches introduce additional training objectives, such as grounding loss [53] or text-to-image alignment [23], to strengthen compositional accuracy, while others enrich semantic features by using LLM-generated information [19]. To incorporate explicit spatial control, training-based layout-controlled methods train adapter modules to inject new conditioning for spatial control [29, 59, 52, 14, 57, 58, 32, 61, 62, 15, 40, 27] such as bounding boxes, segmentation masks, and keypoints. Despite their effectiveness, these methods face two key limitations: a) large-scale training is required to learn object compositions comprehensively; and b) hard conditioning often degrades image quality as indicated in [60].

**Training-free methods** provide an alternative by manipulating diffusion models at inference time, without requiring retraining. Inspired by text-based image editing methods [18, 50, 51] which manipulate attention maps for fine-grained region-level control, many of training-free approaches operate by adjusting text embeddings or cross-attention activations to influence object placement and composition [4, 12, 39, 49, 30, 46, 34, 1, 16].

Within this category, training-free layout-controlled methods steer generation using layout guidance without additional training. These methods often use LLMs [42] to generate 2D layouts of object bounding boxes, and then modulate cross-attention to emphasize regions corresponding to input boxes [60, 55, 10, 24, 36, 9, 8, 22, 44]. RPG [56] and MuLan [28] introduce region-wise diffusion, which decomposes the image latent into spatial regions and performs localized denoising. However, controlling cross-attention alone is often insufficient for handling complex multi-object compositions.

Another class of training-free approaches focuses on initial noise search and optimization, motivated by the observation that the initial noise can significantly influence the generation outcome. Several works have proposed inference-time noise search and optimization [35, 17], which involve generating multiple candidates and selecting the best one using heuristics based on attention dynamics [17] or external verifiers [35]. While promising, these methods tend to be computationally expensive and unreliable when generating highly complex or out-of-distribution compositions.

To overcome these limitations, we propose to directly generate noise in the form of composite object priors — coarse RGB images representing the scene layout, semantics and appearance. This is inspired by image editing methods [38, 2, 37] that use noisy initialization and image inversion. However, we generate object priors based on LLM’s chain-of-thought reasoning for noise initialization and propose a novel prior-guided diffusion method in the image generation process.

# 3 Proposed Method

As illustrated in Figure 2, our *ComposeAnything* framework consists of three key components for compositional text-to-image generation: 1) LLM Planning (Section 3.1): We employ LLMs to transform the input prompt into a structured 2.5D semantic layout, including object captions, bounding boxes and relative depths; 2) Composite Object Prior (Section 3.2): Based on the layout, we generate a coarse composite image that serves as a strong semantic and spatial prior for guiding image synthesis; and 3) Prior Guided Diffusion (Section 3.3): We iteratively initialize noises with the object prior and apply spatially-controlled self-attention to preserve structure in early denoising steps.
## 3.1 LLM Planning

Recent advancements in LLMs have demonstrated their effectiveness in generating high-quality scene layouts from textual descriptions [13, 56, 20]. Hence, we harness GPT-4.1 [42] to produce a structured 2.5D semantic layout from the original text. The layout includes the following elements: Object captions {$y_{o_i}$}$_{i=1}^K$ that describe size, orientation and appearance for each identified object; Bounding boxes {$box_i$}$_{i=1}^K$ that specify 2D spatial configuration for each object; Depth values {$depth_i$}$_{i=1}^K$ that reflect relative depth orders for each object to support 3D-aware composition; Background caption $y_{bg}$ describing the background scene; and Compositional caption $y_{base}$ which is a concise summary of the entire image. This process involves several key steps for chain-of-thought reasoning, as illustrated in Figure 3. More details are provided in section F of the appendix.

<figure><img src="image_6.png" /><figcaption>Figure 3: Chain-of-thought LLM planning for generating 2.5D semantic layouts from text.</figcaption></figure>

## 3.2 Composite Object Prior

**2.5D position-aware composite image generation.** Given the isolated object captions from LLM, we first generate individual objects using Stable Diffusion-3 Medium (SD3-M) [11]. Next, we use a referring expression segmentation model Hyperseg [54] to extract objects {$o_i$}$_{i=1}^K$ along with their segmentation masks {$m_i$}$_{i=1}^K$. Each object and its corresponding mask are resized to fit within the designated bounding box generated from the LLM according to a scaling factor $scale_i$. Objects are then composited in a depth-aware order, where objects with smaller depth values are placed above those with larger depth values, thereby establishing occlusion-correct layering in the final scene. This process is formulated as follows:

$$o'_i, m'_i = \text{Resize}(o_i, m_i, scale_i), \qquad (1)$$

$$o_p, m_p = \text{Compose}(\{o'_i\}, \{m'_i\}, \{box_i\}, \{depth_i\}). \qquad (2)$$

Finally, all objects are composited on a $N \times N$ sized canvas, denoted as $o_p$. Its corresponding composited mask is denoted as $m_p$. Figure 2 shows an example of the composite image and mask. The composition of all objects forms the foreground, and the rest is considered the background.

**Initializing object prior for diffusion-based models.** Our work builds upon existing T2I diffusion models, aiming to enhance its ability to generate images with complex object compositions. Our method is compatible with both denoising diffusion probabilistic models like SDXL [45] and recent flow-matching based models like SD3-M [11].

The core idea of diffusion models is to learn a generative process by simulating and then reversing a gradual noising procedure. Given an image $x_0$ from the real data distribution $p(x)$, the forward process transforms $x_0$ into $x_T \sim \mathcal{N}(0, I)$ through a predefined noise schedule:

$$x_t = \alpha(t)x_0 + \sigma(t)z, \quad z \sim \mathcal{N}(0, I), \qquad (3)$$

where $t \in [0, T]$ indexes the diffusion timestep. A denoising network $\epsilon(\theta)$ is trained to predict the added noise at each step in the forward process. During inference, image generation starts from pure Gaussian noise $x_T$ and denoises it back to $x_0$ via the reverse process, which is an ordinary differential model (ODE) on time $t \in [T, 0]$ guided by the noise prediction network $\epsilon(\theta)$:

$$x_{t-\Delta t} \leftarrow x_t - \epsilon(\theta)(x_t, t) \Delta t. \qquad (4)$$

Our method is inspired by the fact that the reverse ODE can be solved from any $t \in (0, T)$ [38]. Instead of starting from pure Gaussian noise at $t = T$, we initialize the process with a noisy object prior at an intermediate timestep $t_p < T$, providing a stronger starting point for generation.
Specifically, we follow latent diffusion models [47, 45, 11, 3] where the denoising is applied on the latent space. We use the above composite image $o_p$ to generate an initial noise in the latent space. The image $o_p$ is first encoded through a Variational Autoencoder (VAE) to get the prior latent. Then, we apply the forward process from Eq (3) at a high noise timestep $t_p$ to get the latent object prior:

$$z^{o_p} = \text{VAE}(o_p), \qquad (5)$$

$$\hat{z}_{t_p}^{o_p} = \alpha(t_p)z^{o_p} + \sigma(t_p)z, \quad z \sim \mathcal{N}(0, I). \qquad (6)$$

Since the background in $o_p$ is empty, we avoid conditioning the generation process on the uninformative background region of the latent $\hat{z}_{t_p}^{o_p}$. To achieve this, we use the mask $m_p$ to reinitialize the background with pure Gaussian noise:

$$z_{t_p}^{o_p} = \hat{z}_{t_p}^{o_p} \odot m_p + z_{bg} \odot (1 - m_p), \qquad (7)$$

where $z_{bg} \sim \mathcal{N}(0, I)$ and $\odot$ indicates element multiplication. This ensures that only the object regions are guided by a prior, while the background remains free to be generated based on the caption. The reverse process still starts from $t = T$, but uses the composite object prior $z_{t_p}^{o_p}$ as initialization.

## 3.3 Prior-guided Diffusion

We propose two mechanisms to incorporate the guidance from the composite object prior in the denoising process. Figure 4 illustrates the prior-guided diffusion method.

**Object prior reinforcement.** To prevent excessive corruption of the foreground object prior, we initialized the foreground with noise at time $t_p$, while the background is still initialized with pure Gaussian noise at $T$.

However, during denoising from $t = T$, this mismatch in noise levels leads to inaccurate noise predictions for the foreground region, which potentially distort its semantics and structure. To address this, we propose a novel foreground prior reinforcement algorithm. During the denoising steps from $T$ to $t_p$, we repeatedly restore the original object prior in the foreground regions to protect them from degradation. Specifically, we overwrite the foreground region in the current latent $z_{t-1}$ with the initial object prior, while retaining the denoised background:

<figure><img src="image_7.png" /><figcaption>Figure 4: Overview of prior-guided diffusion. The spatial-controlled denoising is applied for each aligned text and region pair to strengthen spatial control, and we further re-inject the object prior $z_{t_p}^{o_p}$ into predicted $z_{t-1}$ to reinforce the prior.</figcaption></figure>

$$z_{t-1} \leftarrow z_{t_p}^{o_p} \odot m_p + z_{t-1} \odot (1 - m_p). \qquad (8)$$

This iterative replacement ensures that the semantic integrity and spatial structure of the object prior are preserved throughout the early diffusion steps. At the same time, the background is progressively refined in the presence of a fixed foreground, allowing for coherent integration between the two.

Once the latent reaches time $t_p$, both foreground and background are aligned in terms of noise level and the global structure becomes stable. From this point onward, denoising proceeds without any additional intervention, allowing for natural refinement and generative flexibility. Notably, decreasing $t_p$ strengthens the object prior while reducing generative flexibility.

**Spatial-controlled denoising.** To further enhance object-level spatial control in T2I generation, we propose a spatial-controlled attention mechanism that explicitly strengthens the alignment between specific image regions and their corresponding region textual descriptions.

Our method builds on Multi-Modal Diffusion Transformers, a dual-stream architecture used in Stable Diffusion [11], which processes text and image modalities in parallel. In addition to the base prompt embeddings $y^{base}$, we introduce a set of K object prompt embeddings $\{y^{o_i}\}_{i=1}^K$ and one background prompt embedding $y^{bg}$. These are independently processed by the text stream, while the image stream receives only the latent image embeddings.
<table><caption>Table 1: State-of-the-art comparison on the T2I-CompBench and NSR-1k benchmarks.</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">Base</th><th colspan="3">T2I-CompBench</th><th colspan="2">NSR-1K</th></tr><tr><th>Model</th><th>2D-Spatial</th><th>Count</th><th>3D-Spatial</th><th>Complex</th><th>Spatial</th><th>Count</th></tr></thead><tbody><tr><td>LayoutGPT [13]</td><td>SD-v1</td><td>45.81</td><td>60.27</td><td>-</td><td>-</td><td>60.6</td><td>55.6</td></tr><tr><td>CreatiLayout[58]</td><td>SD3-M</td><td>47.36</td><td>62.15</td><td>68.85</td><td>34.6</td><td>59.8</td><td><b>63.4</b></td></tr><tr><td>SD-v1 [47]</td><td>SD-v1</td><td>12.46</td><td>44.61</td><td>-</td><td>30.80</td><td>16.89</td><td>31.45</td></tr><tr><td>Attend-Excite v2 [4]</td><td>SD-v2</td><td>14.55</td><td>47.67</td><td>-</td><td>34.01</td><td>26.86</td><td>39.41</td></tr><tr><td>SDXL [45]</td><td>SDXL</td><td>21.33</td><td>49.88</td><td>47.12</td><td>32.37</td><td>31.57</td><td>30.62</td></tr><tr><td>RealCompo [60]</td><td>SDXL</td><td>31.73</td><td>65.92</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>SD3-M [11]</td><td>SD3-M</td><td>31.32</td><td>60.22</td><td>49.43</td><td>37.71</td><td>44.43</td><td>44.61</td></tr><tr><td>FLUX-Schnell [3]</td><td>FLUX</td><td>26.13</td><td>60.58</td><td>59.51</td><td>37.03</td><td>39.29</td><td>55.97</td></tr><tr><td>RPG<fnref n="1" /> [56]</td><td>SDXL</td><td>40.26</td><td>56.39</td><td>50.93</td><td>36.53</td><td>52.24</td><td>39.81</td></tr><tr><td>Inference-scale [35]</td><td>FLUX</td><td>31.51</td><td>67.89</td><td>-</td><td>38.10</td><td>-</td><td>-</td></tr><tr><td>ComposeAnything (Ours)</td><td>SD3-M</td><td><b>48.24</b></td><td><b>68.21</b></td><td><b>77.16</b></td><td><b>38.66</b></td><td><b>63.80</b></td><td>59.36</td></tr></tbody></table>

During the self-attention, the image latent $z_t$ is split into two latents: 1) a base latent $z_t^{base}$, and 2) an object-background latent $z_t^{ob}$. Given object masks $\{m_i\}_{i=1}^K$ and background mask $m_{bg}$, we segment $z_t^{ob}$ into separate objects and background latents:

$$ \{z_t^{o_i}\}_{i=1}^K = \text{Segment}(z_t^{ob}, \{m_i\}_{i=1}^K), \qquad (9) $$

$$ \{z_t^{bg}\} = \text{Segment}(z_t^{ob}, m_{bg}). \qquad (10) $$

Each object latent $z_t^{o_i}$ and its corresponding prompt embedding $y_{o_i}$ are concatenated and passed through a Joint Self-Attention (JSA) module:

$$ q_t^i = [(W_{qy} \cdot y_t^{o_i}); (W_{qz} \cdot z_t^{o_i})], \qquad (11) $$

$$ k_t^i = [(W_{ky} \cdot y_t^{o_i}); (W_{kz} \cdot z_t^{o_i})], \qquad (12) $$

$$ v_t^i = [(W_{vy} \cdot y_t^{o_i}); (W_{vz} \cdot z_t^{o_i})], \qquad (13) $$

$$ [y_t^{o_i}, z_t^{o_i}] \leftarrow \text{Softmax}\left(\frac{q_t^i(k_t^i)}{\sqrt{d}}\right) \cdot v_t^i, \qquad (14) $$

where $W_{qy}$, $W_{ky}$, $W_{vy}$ are linear projects for the prompt embeddings, and $W_{qz}$, $W_{kz}$, $W_{vz}$ for the image latent. For simplicity, we reuse the same notation for the input and output of the transformer layer. This spatial-controlled self-attention is applied at each transformer layer, enabling precise control over object placement and appearance while preserving global visual consistency. The same mechanism is applied to the background: $[y_t^{bg}, z_t^{bg}] \leftarrow \text{JSA}(y_t^{bg}, z_t^{bg})$. The original base attention is applied on the base prompt and the base latent embeddings $[y_t^{base}, z_t^{base}] \leftarrow \text{JSA}(y_t^{base}, z_t^{base})$.

After the last transformer layer, the object and background latents are denoised from $t \to t-1$. The updated object latents $\{z_{t-1}^{o_i}\}_{i=1}^K$ and background latent $z_{t-1}^{bg}$ are then composed back into $z_{t-1}^{ob}$ using the segmentation masks.

Finally, we merge the base latent and object-background latent with a weighted sum:

$$ z_{t-1} = z_{t-1}^{base} * \text{ratio}_{base} + z_{t-1}^{ob} * (1 - \text{ratio}_{base}). \qquad (15) $$

It balances global coherence from the base latent and fine-grained spatial control from the object-background latent. We apply the spatial control for the initial $N_{sc}$ denoising steps.

# 4 Experiments

## 4.1 Experimental Setup

<footnote n="1">¹We run RPG using the released code and report the reproduced results on all categories of the two benchmarks.</footnote>
<figure><img src="image_8.png" /><figcaption>Figure 5: State-of-the-art comparison against SD3-M [11], FLUX [3], RPG [56], and Creatilay-out [58], on complex surreal prompts.</figcaption></figure>

**Evaluation benchmarks.** We evaluate our method on T2I-CompBench [21] and NSR-1k [13] datasets. They contain prompts rich in spatial, 3D, numeric and generally complex and often surreal compositions. We evaluate our method on four categories from T2I-CompBench: 2D Spatial, Numeracy (Count), 3D Spatial, and Complex, each containing 300 prompts. For NSR-1K, we report results on the Spatial (283 prompts) and Count (672 prompts) categories.

**Evaluation metrics.** For the 2D-spatial and numeracy categories, we follow the standard evaluation protocols from T2I-CompBench and NSR-1k. Object detectors — UniDet [63] for T2I-CompBench and GLIP [26] for NSR-1k — are first used to identify objects in the generated images. The metric penalizes missing objects, incorrect counts, and spatial errors, with the latter measured by the center-point distance for each specified 2D relation. For the 3D-spatial category, the original T2I-CompBench metric relies on depth estimation and bounding box detection, which we found inaccurate and overly punitive. To address this, we introduce an MLLM-based metric using GPT-4.1 [42]. The model is prompted to first identify all required objects and then assess their 3D spatial relations. Scores are assigned as follows: 0 if objects are missing or 3D relations are wrong, 1 if all objects are present but the 3D relations are ambiguous, and 2 if everything is correct. We normalize the total score to a 0–100 scale and average over all examples. Full details are provided in section G of the appendix. For the complex category, we adopt the 3-in-1 metric from T2I-CompBench, which averages the CLIP similarity score, spatial accuracy (via object detection), and BLIP-VQA accuracy. This composite score better aligns with human judgment.

**Implementation details.** We use GPT-4.1 [42] for LLM planning and SD3-Medium (SD3-M) [11] as the base diffusion model with Flow matching Euler discrete scheduler. We fix total 28 steps for denoising. We also perform experiments with SDXL as the base model, in section D of the appendix. The generation process is controlled by two key hyper-parameters: (1) $t_p$ — The time at which noise is sampled and applied to the prior image in the forward diffusion. As $t_p$ goes from (T to 0), prior strength increases, which increases faithfulness while reducing generative flexibility. (2) $N_{sc}$ — The number of steps for spatially controlled denoising. A higher value enforces stronger spatial control.
The two hyper-parameters enable highly controllable generation and can be tuned to balance the composition performance and image quality, as demonstrated in section B of the appendix. For the experiments in this section, we sample $t_p$ corresponding to a high noise of 91.3% from the Flow matching schedule and set $N_{sc} = 3$ steps.

## 4.2 Comparison with State-of-the-Art Methods

**Compared methods.** We compare our method against both training-based and inference-only approaches. Training-based methods include layout-to-image models with box conditioning such as Gligen [29] and CreatiLayout [58]. The inference-only methods include general pretrained T2I models (SDv1 [47], SDXL [45], SD3-M [11], and FLUX [3]), layout-guided training-free approaches (RPG [56] and RealCompo [60]), and a noise search method inference time scaling [35].

**Quantitative results.** Table 1 presents the evaluation results on the T2I-CompBench and NSR-1K benchmarks. Our method outperforms all prior approaches across all categories on T2I-CompBench by a significant margin. Compared to the base model SD3-M, ComposeAnything achieves absolute gains of 16.9% on 2D-Spatial, 7.9% on Count, 27.7% on 3D-Spatial, and 0.9% on Complex. On NSR-1K, we also outperform SD3-M with improvements of absolute 19.0% and 14.7% on the Spatial and Count categories, respectively. Our method surpasses all state-of-the-art methods except being slightly worse than CreatiLayout [58] in the Count category.

**Qualitative results.** Figure 5 compares our method against state-of-the-art approaches on challenging compositional prompts. Our method consistently adheres to the input prompt, generating coherent and realistic images. In contrast, models such as SD3-M [11], FLUX [3], and RPG [56] often fail to follow the prompt. For instance, all three baselines struggle with placing a chicken on a balloon and frequently fail to maintain the correct object count. As object count increases, SD3-M and FLUX tend to “cartoonify” the output, sacrificing realism. CreatiLayout [58], a fine-tuned layout-conditioned model, performs slightly better in maintaining spatial relations when provided with bounding box inputs. However, as discussed in RealCompo [60], hard box-conditioning methods, while strong in compositional accuracy, often compromise image quality. These methods are inherently limited by the spatial distributions seen during training and struggle to generalize to unusual or surreal layouts. In contrast, by leveraging object priors and integrating them into the denoising process, our method achieves a better balance between compositional fidelity and visual quality.

**Human evaluations.** To further assess generation quality, we conduct human evaluations on three categories in T2I-CompBench: 2D-, 3D-Spatial and Count. We compare ComposeAnything against two state-of-the-art methods: RPG [56] (inference-only) and CreatiLayout [58] (training-based). For each category, we ran-

<figure><img src="image_9.png" /><figcaption>Figure 6: Human evaluations against RPG and CreatiLayout.</figcaption></figure>

domly sample 30 prompts and perform pairwise comparisons, resulting in 180 unique image comparisons (30 prompts × 3 categories × 2 baselines). Human raters were instructed to evaluate based on both prompt alignment and image quality (see appendix for details). Five raters participated in the evaluation, with 30% of images overlapping across raters to measure inter-annotator agreement. The average agreement scores were around 80% for all categories. As shown in Figure 6, our method significantly outperforms both methods across the three categories.

## 4.3 Ablations

**Object prior quality.** The correctness of object priors is crucial for high-quality image generation. We use the same metrics for evaluating the final image to evaluate the generated composite object prior. As shown in Table 2, the performance of the object priors is closely correlated with that of the final generated images. Our method achieves state-of-the-art results even at the prior

<table><caption>Table 2: Performance of Object Priors (OP) and the corresponding Final Image (FI) on T2I-CompBench.</caption><thead><tr><th></th><th>Spatial</th><th>Numeracy</th><th>3D</th><th>Complex</th></tr></thead><tbody><tr><td>OP</td><td>45.79</td><td>68.06</td><td>86.71</td><td>35.04</td></tr><tr><td>FI</td><td>48.24</td><td>68.21</td><td>77.16</td><td>38.66</td></tr></tbody></table>
<figure><img src="image_10.png" /><figcaption>Figure 7: LLM generated object prior and their corresponding final image generation.</figcaption></figure>

stage, demonstrating the effectiveness and robustness of the LLM-driven pipeline for automatic prior generation. Notably, our prior-guided diffusion is applied only during the initial denoising steps. This design preserves the structural benefits of the prior while allowing subsequent steps to introduce generative flexibility, which helps correct inaccuracies in the initial composite. As illustrated in Figure 7(b), the final images can resolve issues present in the priors such as missing elements, incorrect orientation, size, and overall incoherence of the composite. This improvement is also reflected in the quantitative results in Table 2, where the final generations outperform the priors across all categories except in 3D. Although the standard diffusion step improves image quality and coherence, it sometimes compromises the correctness of 3D spatial relationships for difficult compositions, due to lack of explicit 3D guidance in the diffusion process, as seen in Figure 7(c). However composition correctness and image quality/coherence can be easily balanced by tuning the hyper-parameters as shown in section B of the appendix. More detailed human evaluations on the quality of prior image and the corresponding generation is presented in section C of the appendix.

**Prior-guided diffusion.** Table 3 shows the contribution of object prior reinforcement and spatial-controlled denoising in the prior-guided diffusion process. Each component significantly enhances performance over the base SD3-M model [11]. Specifically, the object prior reinforcement yields absolute gains of over 14 and 6% in spatial and numeric categories, respectively,

<table><caption>Table 3: Impact of object prior reinforcement and spatial-controlled denoising on T2I-CompBench.</caption><thead><tr><th>Prior Reinforce</th><th>Spatial Control</th><th>Spatial</th><th>Numeracy</th></tr></thead><tbody><tr><td>×</td><td>×</td><td>31.32</td><td>60.22</td></tr><tr><td>✓</td><td>×</td><td>45.33</td><td>66.08</td></tr><tr><td>×</td><td>✓</td><td>43.56</td><td>64.42</td></tr><tr><td>✓</td><td>✓</td><td><strong>48.24</strong></td><td><strong>68.21</strong></td></tr></tbody></table>

while spatial-controlled denoising improves performance by over absolute 12 and 4%. When combined, these components further boost results, demonstrating their complementary roles in achieving precise and controlled text-to-image generation.

# 5 Conclusion

In this work, we introduce *ComposeAnything*, a novel inference-time framework for compositional text-to-image generation that leverages object-level guidance derived from LLM-generated 2.5D semantic layouts. By introducing a composite object prior for structured initialization and prior-guided diffusion, our approach enables precise object placement and robust semantic grounding without any additional training. *ComposeAnything* achieves state-of-the-art performance on T2I-CompBench and NSR-1K, effectively balancing image quality and prompt fidelity even under complex or surreal scenarios. Our results highlight the potential of LLM-driven reasoning and composite prior guidance in advancing compositional T2I generation.

**Limitations.** The main failure mode of our approach are errors of the LLM for spatial layout generation, as shown in Table 2. While our approach is robust to some of these failures (Fig. 7(b)), a completely incorrect object prior can not be recovered (Fig. 7(d)). Future work could improve the
layout generation with LLMs by fine-tuning the models for this task or by adding explicit 3D scene information. In very rare cases, we observe a failure even though the object prior is correct (Fig. 7(c)). This is due to missing 3D knowledge in the diffusion model and requires further improvement of the base model.

## References

[1] Aishwarya Agarwal, Srikrishna Karanam, K J Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. A-star: Test-time attention segregation and retention for text-to-image synthesis. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 2283–2293, October 2023.

[2] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022.

[3] Black Forest Labs. FLUX. https://blackforestlabs.ai, 2024.

[4] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. *ACM Trans. Graph.*, 2023.

[5] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023. URL https://arxiv.org/abs/2310.00426.

[6] Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-δ: Fast and controllable image generation with latent consistency models, 2024. URL https://arxiv.org/abs/2401.05252.

[7] Kai Chen, Enze Xie, Zhe Chen, Yibo Wang, Lanqing HONG, Zhenguo Li, and Dit-Yan Yeung. Geodiffusion: Text-prompted geometric control for object detection data generation. In *The Twelfth International Conference on Learning Representations*, 2024.

[8] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, 2024.

[9] Guillaume Couairon, Marlène Careil, Matthieu Cord, Stéphane Lathuilière, and Jakob Verbeek. Zero-shot spatial layout conditioning for text-to-image diffusion models. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2023.

[10] Omer Dahary, Or Patashnik, Kfir Aberman, and Daniel Cohen-Or. Be yourself: Bounded attention for multi-subject text-to-image generation. In *European Conference on Computer Vision (ECCV)*, 2024.

[11] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In *International Conference on Machine Learning (ICML)*, 2024.

[12] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In *The Eleventh International Conference on Learning Representations*, 2023.

[13] Weixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Xuehai He, S Basu, Xin Eric Wang, and William Yang Wang. LayoutGPT: Compositional visual planning and generation with large language models. In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.

[14] Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou. Ranni: Taming text-to-image diffusion for accurate instruction following. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024.

[15] Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, Salman Khan, and Peter Wonka. LLM blueprint: Enabling text-to-image generation with complex and detailed prompts. In *The Twelfth International Conference on Learning Representations*, 2024.

[16] Biao Gong, Siteng Huang, Yutong Feng, Shiwei Zhang, Yuyuan Li, and Yu Liu. Check locate rectify: A training-free layout calibration system for text-to-image generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 6624–6634, June 2024.
[17] Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting text-to-image diffusion models via initial noise optimization. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024.

[18] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. In *The Eleventh International Conference on Learning Representations*, 2023.

[19] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. *CoRR*, 2024.

[20] Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A Ross, Cordelia Schmid, and Alireza Fathi. Scenecraft: An LLM agent for synthesizing 3D scenes as blender code. In *Forty-first International Conference on Machine Learning*, 2024.

[21] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhengguo Li, and Xihui Liu. T2I-compBench: A comprehensive benchmark for open-world compositional text-to-image generation. *Advances in Neural Information Processing Systems*, 36:78723-78747, 2023.

[22] Vikram Jamwal and Ramaneswaran S. Composite diffusion: whole sparts. In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, 2024.

[23] Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, and Hongsheng Li. Comat: Aligning text-to-image diffusion model with image-to-text concept matching. In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.

[24] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2023.

[25] Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Evaluating and improving compositional text-to-visual generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops*, 2024.

[26] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022.

[27] Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnet ++: Improving conditional controls with efficient consistency feedback. In *European Conference on Computer Vision (ECCV)*, 2024.

[28] Sen Li, Ruochen Wang, Cho-Jui Hsieh, Minhao Cheng, and Tianyi Zhou. Mulan: Multimodal-llm agent for progressive multi-object diffusion. *arXiv preprint arXiv:2402.12741*, 2024.

[29] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In *Conference on Computer Vision and Pattern Recognition (CVPR)*, 2023.

[30] Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for improved generative semantic nursing. In *BMVC*, 2023.

[31] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LLM-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. *Transactions on Machine Learning Research*, 2024. ISSN 2835-8856.

[32] Han Lin, Jaemin Cho, Abhay Zala, and Mohit Bansal. Ctrl-adapter: An efficient and versatile framework for adapting diverse controls to any diffusion model. In *The Thirteenth International Conference on Learning Representations*, 2025.

[33] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. *arXiv preprint arXiv:2210.02747*, 2022.

[34] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In *European Conference on Computer Vision (ECCV)*, 2022.
[35] Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, and Saining Xie. Inference-time scaling for diffusion models beyond denoising steps. arXiv preprint arXiv:2501.09732, 2025.

[36] Wan-Duo Kurt Ma, Avisek Lahiri, J.P. Lewis, Thomas Leung, and W. Bastiaan Kleijn. Directed diffusion: direct control of object placement through attention guidance. In *Association for the Advancement of Artificial Intelligence (AAAI)*, 2024.

[37] Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. Guided image synthesis via initial image editing in diffusion model. In *Proceedings of the 31st ACM International Conference on Multimedia*, 2023.

[38] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In *International Conference on Learning Representations (ICLR)*, 2022.

[39] Tuna Han Salih Meral, Enis Simsar, Federico Tombari, and Pinar Yanardag. Conform: Contrast is all you need for high-fidelity text-to-image diffusion models. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024.

[40] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models. In *Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence*, 2024.

[41] Weili Nie, Sifei Liu, Morteza Mardani, Chao Liu, Benjamin Eckart, and Arash Vahdat. Compositional text-to-image generation with dense blob representations. In *Forty-first International Conference on Machine Learning*, 2024.

[42] OpenAI. GPT-4.1. https://openai.com/index/gpt-4-1/, 2025.

[43] Pablo Pernias, Dominic Rampas, Mats L. Richter, Christopher J. Pal, and Marc Aubreville. Wuerstchen: An efficient architecture for large-scale text-to-image diffusion models, 2023. URL https://arxiv.org/abs/2306.00637.

[44] Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024.

[45] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. URL https://arxiv.org/abs/2307.01952.

[46] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.

[47] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In *Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022.

[48] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In *International Conference on Learning Representations (ICLR)*, 2021.

[49] Maria Mihaela Trusca, Wolf Nuyts, Jonathan Thomm, Robert Honig, Thomas Hofmann, Tinne Tuytelaars, and Marie-Francine Moens. Object-attribute binding in text-to-image generation: Evaluation and control. arXiv preprint arXiv:2404.13766, 2024.

[50] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2023.

[51] Kai Wang, Fei Yang, Shiqi Yang, Muhammad Atif Butt, and Joost van de Weijer. Dynamic prompt learning: Addressing cross-attention leakage for text-based image editing. In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.

[52] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control for image generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024.

[53] Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu. Tokencompose: Text-to-image diffusion with token-level supervision. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024.
[54] Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Zheng Zhao, Jie Hu, and Yujiu Yang. Hyperseg: Towards universal visual segmentation with large language model, 2024. URL https://arxiv.org/abs/2411.17606.

[55] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2023.

[56] Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin CUI. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal LLMs. In *International Conference on Machine Learning (ICML)*, 2024.

[57] Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Reco: Region-controlled text-to-image generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2023.

[58] Hui Zhang, Dexiang Hong, Tingwei Gao, Yitong Wang, Jie Shao, Xinglong Wu, Zuxuan Wu, and Yu-Gang Jiang. Creatilayout: Siamese multimodal diffusion transformer for creative layout-to-image generation. *arXiv preprint arXiv:2412.03859*, 2024.

[59] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In *International Conference on Computer Vision (ICCV)*, 2023.

[60] Xinchen Zhang, Ling Yang, YaQi Cai, Zhaochen Yu, Kai-Ni Wang, xie jiake, Ye Tian, Minkai Xu, Yong Tang, Yujiu Yang, and Bin CUI. Realcompo: Balancing realism and compositionality improves text-to-image diffusion models. In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.

[61] Yibo Zhao, Liang Peng, Yang Yang, Zekai Luo, Hengjia Li, Yao Chen, Wei Zhao, Qinglin Lu, Wei Liu, and Boxi Wu. Local conditional controlling for text-to-image diffusion models. *CoRR*, abs/2312.08768, 2023.

[62] Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2023.

[63] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022.
# Appendix

In the appendix, we first discuss the broader impact of our work in Section A. Section B analyzes the impact of two key hyper-parameters in our framework. Section C provides more detailed results of the correctness of object prior and final images, including human evaluation and qualitative examples. Section D shows the result of our ComposeAnything method with SDXL as the base model. Section E introduces the human evaluation details for state-of-the-art comparison. Section F presents the LLM prompt for 2.5D semantic layout generation and illustrates an output example. Finally, Section G details the LLM prompt for evaluating the 3D-Spatial category.

## A Broader Impact

This work proposes a novel text-to-image (T2I) framework for compositional image generation, advancing interpretability, faithfulness and controllability of deep generative models in image generation from complex textual descriptions. This has potential applications in creative design, digital content creation, education and human-computer interaction to serve as an assistive tool for artists and designers. It can also benefit robotics in generating synthetic datasets.

Since our method focuses on improving the compositional capabilities of existing diffusion models without training, we do not foresee significant additional ethical risks associated with this work compared to the base models. These T2I base models come with significant misinformation risks (deepfakes). Furthermore, copyright infringement and biases need to be carefully checked when training them.

## B Impact of Key Hyper-parameters

We analyze the effect of two key hyper-parameters $t_p$ and $N_{sc}$, as discussed in Section 4.1 of the main paper. $t_p$ is the time at which noise is sampled and applied to the prior image in the forward diffusion. It controls the object prior reinforcement strength (OPR). Lower values denote stronger priors. $N_{sc}$ is the number of steps for spatially controlled denoising. It controls the spatial-controlled denoising strength (SCD). Larger values result in stronger control.

Figure 8 presents object priors and corresponding generated images for two text prompts, under varying $t_p$ and $N_{sc}$ values. In the first row of each example, $N_{sc}$ is fixed at 3 to isolate the impact of OPR with different $t_p$. At low OPR strength, the final image fails to preserve the *appearance and semantics* of the prior, for example, the butterfly appears in front of the cup not on top of it, and the number of clocks and microwaves is incorrect. As the strength of OPR increases, objects' semantics and appearance such as color, shape and number are more strongly retained. However, excessive reinforcement reduces generative flexibility, leading to over-constrained and less natural outputs.

In the second row of each example, $t_p$ is fixed at 0.91 to examine the effect of $N_{sc}$. Low SCD strength leads to limited spatial control, with objects leaking in background (extra cup), objects getting merged (both dogs merged), and incorrect object counts. As we increase the SCD strength, object positions and sizes from the prior are more faithfully preserved in the final image. However, too strong spatial control results in low-quality compositions such as rigid placements, incoherent scene, floating objects similar to training-based box-conditioned methods [58].

Therefore, in our experiments, we set $t_p = 0.91$ and $N_{sc} = 3$ to strike a balance between faithful prompt adherence, generative flexibility, and overall scene coherence. Both hyper-parameters are beneficial and complementary to reliably produces correct spatial relations, accurate object counts and high-quality images.
<figure><img src="image_11.png" /><figcaption>Figure 8: Effect of Object Prior Reinforcement and Spatial-Controlled Denoising. Increasing either strength enhances appearance fidelity and spatial precision, but reduces generative flexibility.</figcaption></figure>

## C Detailed Evaluation of Object Priors and Corresponding Final Images

To better assess the quality of object priors and their influence on final image generation, we categorize results into four combinations: i) correct prior – correct image, ii) correct prior – incorrect image, iii) incorrect prior – correct image, iv) incorrect prior – incorrect image.

We conduct a human evaluation using 30 samples, i.e., pairs of the prior and final image, across the 2D-Spatial, 3D-Spatial, and Numeracy categories in T2I-Compbench. For sample annotators perform
<figure><img src="image_12.png" /><figcaption>Figure 9: Human evaluation results on the correctness of prior and final image pairs on the three categories of T2I-Compbench dataset.</figcaption></figure>

a 4-way classification task, judging the correctness of both the prior and the resulting image. The annotation interface is illustrated in Figure 10.

Figure 9 presents the results of the human evaluation. As seen in the bar plots for the 2D and 3D-Spatial categories, the majority of samples fall into the correct-prior and correct-image category. In contrast, the Numeracy category shows a higher occurrence of correct-prior but incorrect-image cases. This is primarily due to the increasing number of objects, where relative object sizes in the prior affect its realism and quality. While our model's generative flexibility helps correct such visual artifacts during image synthesis, it often does so at the expense of count accuracy, leading to mismatches in object quantities, as illustrated in Figure 10 (bottom). Moreover, as the number of objects increases, the priors tend to become less coherent overall, resulting in a greater frequency of incorrect-prior and incorrect-image cases.

Figure 11 - 14 provide more examples of the object priors and corresponding final images on T2I-Compbench dataset, including 2D-spatial, 3D-spatial, non-spatial, numeracy and complex prompt categories.
<figure><img src="image_13.png" /><figcaption>Figure 10: Labeling interface for evaluating object prior and the final image.</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>Figure 11: Object prior and the corresponding generation for 2D-Spatial compositions from T2I-compbench</figcaption></figure>
<figure><img src="image_15.png" /><figcaption>Figure 12: Object prior and the corresponding generation for Complex compositions from T2I-Compbench.</figcaption></figure>
<figure><img src="image_16.png" /><figcaption>Figure 13: Object prior and the corresponding generation for 3D-Spatial compositions from T2I-compbench</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>Figure 14: Object prior and the corresponding generation for Numeracy compositions from T2I-compbench</figcaption></figure>
<table><caption>Table 4: Evaluating our framework on T2I-Compbench for different base models SDXL (a score based diffusion model), and SD3-M (a flow matching based diffusion model).</caption><thead><tr><th>Model</th><th>Base Model</th><th>2D-Spatial</th><th>Numeracy</th><th>3D-Spatial</th><th>Complex</th></tr></thead><tbody><tr><td>SDXL</td><td>SDXL</td><td>21.33</td><td>49.88</td><td>47.12</td><td>32.37</td></tr><tr><td>SD3-M</td><td>SD3-M</td><td>31.32</td><td>60.22</td><td>49.43</td><td>37.71</td></tr><tr><td>ComposeAnything(ours)</td><td>SDXL</td><td>44.64</td><td>57.22</td><td>71.03</td><td>36.20</td></tr><tr><td>ComposeAnything(ours)</td><td>SD3-M</td><td>48.24</td><td>68.21</td><td>77.16</td><td>38.66</td></tr></tbody></table>

## D Additional results on DDIM based SDXL as a base model

In our main experiments, we adopt Stable Diffusion 3 (SD3) as the base model. SD3 is built on the flow matching framework [33], which replaces the traditional score-based training objective with a learned velocity field that enables more stable and efficient generative dynamics. For sampling, it utilizes the Flow Matching Euler (FME) discrete sampler, enabling faster image generation with fewer steps. Architecturally, SD3 incorporates the MM-DiT (Multimodal DiT) Transformer [11] with 2-way, joint self attention, allowing for improved scalability and better handling of complex conditioning signals.

In this section, we further report results on T2I-Compbench, using SDXL [45] a prior diffusion model trained with the conventional score-based denoising objective. SDXL employs DDIM (Denoising Diffusion Implicit Models) [48] sampling and incorporates a U-Net backbone with unidirectional cross-attention layers for conditioning on text prompts. Our model enhances the performance of both SDXL and SD3 base architectures by a large margin.
E Labeling interface for human evaluations

Figure 15 shows the human evaluation instructions and interface. The instructions focuses on prioritizing both correctness to prompt focusing on 2D-3D spatial relations and object count. Also to select images with higher quality. As can be seen in the "six horses" image, both images have 6 horses, but the first image has better quality.

<figure><img src="image_18.png" /><figcaption>Figure 15: Labeling interface for human evaluations.</figcaption></figure>
## F LLM Planning Instructions

Figures 16 - 20 show the detailed instructions for LLM planning.

Figure 21 shows the output from the LLM for an example alongside the prior and final image.

<figure><img src="image_19.png" /><figcaption>Figure 16: Instructions for LLM planning (to be continued).</figcaption></figure>
**3. Foreground object Extraction:**

a. Extract only foreground objects without severe entanglements as planned.

b. Strictly keep the original counts, and enumerate every object one at a time.

c. Make sure to extract the accurate counts and enumerate every object one at a time.

**4. Object Descriptions (isolated objects):**

a. Describe each object individually with strictly no information leak from other objects. Only focus on object's own attributes, and the way it interacts with the scene, but not other objects.

b. Strictly Don't mention any detail of other objects or background, while describing the current object.

c. Strictly no mention of other objects count, positions, attributes etc. in any way.

d. Description should be very concise in one line.

**5. Relative Depth:**

a. For every extracted object predict the relative depth of the object based on the given prompt or imagination. If there are a total of 3 objects, 2 in the foreground and 1 behind them depth should be: object1: 1, object2: 1, object3: 2.

**6. Bounding Box Layout (Relative Sizes):**

a. Scale bounding boxes to reflect any surreal or exaggerated proportions.

b. Bounding boxes should be generally big, covering major parts of the image.

c. Bounding boxes must be placed such that every object remains visibly distinct on a 2D canvas. Even when objects differ in depth, they must not fully occlude or obscure one another.

d. For objects close to the viewer in the foreground size should be bigger than the objects in far back (to emphasize depth). Also consider the original size of the objects.

**7. Composition Caption (Scene-Level):**

a. Craft a unified caption that highlights how all extracted objects interact within the scene. Objects should cover most of the part of the image. Mention a one phrase background.

b. If the input caption is a surreal composition, then imagine the object interactions accordingly and compose a coherent scene.

c. Emphasize object attributes, spatial relationships, and background.

**8. Background (isolated background):**

a. Strictly mention a one phrase background, with no object information.

b. If provided, extract the background information from the input caption. Example: A white door in a pink bathroom. Here pink bathroom should be the background.

**9. Output Format rules:**

Use the following format strictly:

Object class heading: "Objects:"

1. [object_name]

2. [object_name]

Object descriptions heading: "Objectitenstions"

For each object, use the following structure:

Caption Heading: "Caption_object_n(isolated):"

Depth Heading: "Relative_depth_n:"

Bounding Box Heading: "Boooz object_n:"

Compositional caption heading: "Compositional caption:"

Background caption heading: "Background caption:"

c. Bounding boxes should include absolute positions in the format [x_top, y_top, x_bottom, y_bottom] for a 1024x1024 resolution canvas.

d. Strictly both compositional caption and object captions should not be more than 70 words.

<caption>Figure 17: Instructions for LLM planning (to be continued).</caption>
**Examples:**

**Example 1:**

a green balloon on the bottom of a cat

**Planning:** "a green balloon on the bottom of a cat" could mean that the cat is sitting on a balloon in a surreal or a playful scene. Both the objects can be easily put in a 2D layout as they are not heavily entangled. Cat is sitting on a balloon so both have the same depth.

**Rewritten caption:**

a white cat is sitting on top of a green balloon

**Objects:**

1. green balloon

2. cat

**Object>Description:**

Caption_object_1(isolated): A green balloon is lying on the floor.

Relative_depth_1: 1

Box_object_1: [300, 700, 724, 1024]

Caption_object_2(isolated): A white cat, sits calmly with its legs and tail folded.

Relative_depth_2: 1

Box_object_2: [200, 300, 824, 750]

**Compositionalaren't:** A fluffy white cat, full of vibrant energy is sitting playfully on top of a blue balloon in a cosy room.

Backgroundaren't: A cosy room

**Example 2:**

The soft, fluffy texture of the cotton candy melted in the mouth, a sugary treat of childhood nostalgia.

**Planning:** Since both the objects "boy" and "cotton candy" are interacting closely in an intrecate way "melted in the mouth", it is hard to put the two objects in a 2D layout as the object is in the mouth of the boy, there are occlusions and orientation complexities, qualifying for severe entanglement. So considering both the objects as one. The boy could be holding a cotton candy on his left and eating it.

**Rewritten caption:**

A boy is eating a soft fluffy cotton candy.

**Objects:**

1. boy

**Object Description:**

Caption_object_1(isolated): A little boy facing left is eating a pink cotton candy.

Relative_depth_1: 1

Box_object_1: [300, 150, 700, 874]

**Compositional caption:** A little boy facing left eagerly enjoys a fluffy pink cotton candy in a lively park, his eyes sparkle with excitement.

**Background:** a lively park

<caption>Figure 18: Instructions for LLM planning (to be continued).</caption>
**Example 3:**

two helmets and three ships

**Planning:** There are three ships and two helmets, each treated as individual, rigid objects to be placed independently in the scene. Given their differing scales, the ships (being much larger) should be positioned in the background, while the smaller helmets should be placed in the foreground.

**Depth Assignment:** Helmets: depth = 1 (foreground). Ships: depth = 2 (background)

**Size Ratio:** To reflect realistic scale in a 2D layout, the bounding boxes for helmets and ships should follow an approximate size ratio of 1:2, with ships appearing larger.

**Placement Details:** The helmets should be located near the bottom of the scene, close to the shore in the foreground. The ships should appear near the horizon, conveying depth and distance.

Among the three ships: two can be large and face forward (toward the viewer), while the third, a smaller wooden ship, can be placed in the center, angled left to reveal more of its body and shape.

**Rewritten caption:** Three ships are sailing in the sea near the horizon, while two helmets are placed on the shore in the foreground

**Objects:**

1. ship

2. ship

3. ship

4. helmet

5. helmet

**Objectiten Descriptions:**

Caption_object_1(isolated): A metalic ship facing front.

Relative_depth_1: 2

Box_object_1: [50, 0, 350, 510]

Caption_object_2(isolated): A metalic ship facing front.

Relative_depth_2: 2

Box_object_2: [650, 90, 1000, 520]

Caption_object_3(isolated): A small wooden ship facing left.

Relative_depth_3: 2

Box_object_3: [234, 550, 824, 750]

Caption_object_4(isolated): A black helmet.

Relative_depth_4: 1

Box_object_4: [100, 790, 400, 1020]

Caption_object_5(isolated): A red helmet.

Relative_depth_5: 1

Box_object_5: [600, 780, 900, 1020]

**Compositionalarenas:** Two majestic ships sail across the horizon with towering masts and curved hulls in the sea. One small wooden ship is in the middle. In the foreground, two helmets, one sleek black and the other bold red rest side by side on the shore.

**Background caption:** a sea shore

<caption>Figure 19: Instructions for LLM planning (to be continued).</caption>
**Example 4:**

A plane is hidden by a blue frog

Planning: The frog and the plane are separate, non-entangled objects and can be laid out independently in a 2D composition. Since the frog hides the plane, the frog should be large and positioned in the foreground (depth 1), while the plane appears smaller in the background (depth 2). To preserve visibility, the frog is placed slightly left of center, and the plane is positioned more toward the right. The frog partially overlaps the left edge of the plane, but at least half of the plane remains clearly visible.

**Rewritten Caption:** a blue frog in the front hides a plane in the back

**Objects:**

1. blue frog

2. plane

Caption_object_1(isolated): A blue frog facing front sitting calmly

Relative_depth_1: 1

Box_object_1: [200, 300, 700, 900]

Caption_object_2(isolated): A commercial airplane on ground facing left

Relative_depth_2: 2

Box_object_2: [520, 350, 920, 550]

**Compositionalarenas:** A frog is sitting in close in the foreground hiding a commercial airplane in far back

**Background caption:** A runway

<caption>Figure 20: Instructions for LLM planning.</caption>
**Input Prompt:** The soft, warm glow of the campfire illuminated the faces of the hikers, as they roasted marshmallows and swapped stories.

**Planning:** The "faces of the hikers" are illuminated by the campfire, while they roast marshmallows and swap stories. Since roasting marshmallows requires close hand-object interaction, this counts as a heavy entanglement. The group of hikers is treated as a single foreground object, with the campfire as another. Both can be easily placed in a visually coherent 2D layout: the campfire in the foreground (at depth = 1) the group of seated or standing hikers((at depth = 2) arranged in a semicircle behind it, both occupying significant portions of the image. Background is implied as outdoors at night, likely in a forest or clearing.

**Rewritten caption:**

A group of hikers sit around a glowing campfire, their faces lit warmly as they roast marshmallows and share stories.

**Objects:**

1. campfire

2. group of hikers

**Object Descriptions:**

**Caption_object_1(isolated):** A glowing campfire with bright orange flames and flickering light.

**Relative_depth_1:** 1

**Box_object_1:** [350, 700, 674, 980]

**Caption_object_2(isolated):** A group of hikers sitting and relaxing, with sticks, faces softly lit.

**Relative_depth_2:** 2

**Box_object_2:** [150, 350, 874, 850]

**Compositional caption:** A warm campfire glows brightly in the evening, and a group of hikers sits closely around it, their faces softly illuminated. The hikers hold marshmallow sticks and enjoy stories together beneath the night sky.

**Background caption:** a starry night outdoors

<figure><img src="image_20.png" /><figcaption>Figure 21: LLM planning for object prior generation, with final generated image.</figcaption></figure>
## G 3D metric evaluation with LLM

Figure 22 presents the detailed instructions given to the LLM for evaluating 3D-spatial relations.

<figure><img src="image_21.png" /><figcaption>Figure 22: LLM instructions for evaluating 3D-spatial relations.</figcaption></figure>