CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step
Authors: Zheyuan Liu
Munan Ning
Qihui Zhang
Shuo Yang
Zhongrui Wang
Yiwei Yang
Xianzhe Xu
Yibing Song
Weihua Chen
Fan Wang
Li Yuan
Venue: NeurIPS 2025 poster ()
Keywords: MLLM
Diffusion
CoT
3D Layout
Image Generation
Primary area: applications
TL;DR: CoT-Diff tightly couples MLLM reasoning with diffusion to enable step-by-step 3D layout planning and dynamic spatial control within a single generation process.

Abstract: Current text-to-image (T2I) generation models struggle to align spatial composition with the input text, especially in complex scenes. 
Even layout-based approaches yield suboptimal spatial control, as their generation process is decoupled from layout planning, making it difficult to refine the layout during synthesis.
We present CoT-Diff, a framework that brings step-by-step CoT-style reasoning into T2I generation by tightly integrating Multimodal Large Language Model (MLLM)-driven 3D layout planning with the diffusion process.
CoT-Diff enables layout-aware reasoning inline within a single diffusion round: at each denoising step, the MLLM evaluates intermediate predictions, dynamically updates the 3D scene layout, and continuously guides the generation process. 
The updated layout is converted into semantic conditions and depth maps, which are fused into the diffusion model via a condition-aware attention mechanism, enabling precise spatial control and semantic injection. 
Experiments on 3D Scene benchmarks show that CoT-Diff significantly improves spatial alignment and compositional fidelity, and outperforms the state-of-the-art method by 34.7% in complex scene spatial accuracy, thereby validating the effectiveness of this entangled generation paradigm.

CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step
Zheyuan Liu
Munan Ning!
Qihui Zhang
Shuo Yang'
Zhongrui Wang'
Yiwei Yang
Xianzhe Xu
Yibing Song?3
Weihua Chen 2.3
Fan Wang'
Li Yuan
Shenzhen Graduate School, Peking University
HHupan Lab
DDMO Academy, Alibaba Group
*Shanghai Jiao Tong University
FLUX
RPG
EliGen
FLUX
RPG
EliGen
FLUX
RPG
EliGen
CoT-Diff (Ours]
CoT-Diff (Ours)
CoT-Diff (Ours)
Prompt: Six ceramic in different colors arranged circle beautiful wooden bothed warm glow aftenoon sunlight.
On smooth table, apple rests at backlett cylindrical glass, with glass obscuring the right the apple.
Prompt:
Prompt: the fronhleft, child the backright stands. The northern lights bloze viridly across the sky above.
snowy
Multiple Objects
Precise Layouts
Spatial Relation
Figure I: Comparison of generated images across three challenging spatial scenarios: multiple-object configuration (left), precise layout (middle), and complex spatial relations (right), CoT-Diff (ours) achieves significantly better spatial alignment than FLUX [5]. RPG [47], and EliGen [48], faithfully following 3D-aware instructions in the prompt (highlighted in red).
Abstract
Current text-to-image (T2I) generation models struggle to align spatial composition with the input text, especially in complex scenes. Even anyout-based approaches yield suboptimal spatial control, as their generation process is decoupled from layout planning, making it difficult to refine the layout during synthesis. We present CoT-Diff, a framework that brings step-by-step CoT-style reasoning into T2I generation by tightly integrating Multimodal Large Language Model (MLLM)- driven 3D layout planning with the diffusion process. CoT-Diff enables layout- aware reasoning inline within single diffusion round: at each denoising step, the MILM evaluates intermediate predictions, dynamically updates the 3D scene layout, and continuously guides the generation process. The updated layout is
* Equal contribution Corresponding author (1iuzy2233, munanning)@@maill. com kugang, cwh@al 1 babaa-ina.........a com maanileceeepuu.co edmppku..duu....m edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

converted into semantic conditions and depth maps, which are fused into the diffusion model via condition-aware attention mechanism, enabling precise spatial control and semantic injection. Experiments on 3D Scene benchmarks show that CoT--Diff' significantly improves spatial alignment and compositional fidelity, and outperforms the state-of-the-art method by 34.7% in complex scene spatial accuracy, validating the effectiveness of this entangled generation paradigm.
Introduction
Diffusion models have shown remarkable capabilities in generating high-quality and diverse images from textual descriptions [15, 11, 22]. However, these models lack explicit structural reasoning abilities [51, 45, 52]. When tasked with complex scenes involving multiple objects, precise layouts, and spatial relations, current models still face significant limitations in controllability, thereby limiting their applicability to structured generation tasks.
Recent works have introduced layout-aware conditions into the diffusion process, incorporating points, scribbles, 2D bounding boxes semantic masks to enhance entity-level spatial control [49, 27, 21, 50, 40]. Yet. most of these methods [23, 43] adopt a decoupled paradigm where layout planning is conducted before generation, without any feedback from the image synthesis process. Such static pipelines lack the ability to refine spatial arrangements, and thus often struggle to model fine-grained structure in complex scenes. Some recent 3D-aware approaches [4, 10, 39] support explicit object placement in space, but typically rely on manually crafted layouts and exhibit limited generalization in automated generation settings. These limitations highlight the need for a unified framework that can reason about 3D structure from text and dynamically control layout during generation a goal we aim to achieve in this work.
We present CoT-Diff, a 3D-aware text-to-image framework for stepwise, fine- fine-grained spatial control over complex visual scenes shown in Figure 1. Unlike prior approaches that perform layout planning as a static pre-process, CoT--Diff tightly entangles a multimodal large language model (MILMM with a diffusion model, enabling inline scene reasoning and generation within single sampling trajectory. Given a text prompt, CoT--Diff first uses the MI.LMM to plan an initial 3D) scene layout including entity descriptions, spatial positions, and size estimates. Then, at each denoising step, the intermediate predicted image is passed back to the MLLM, which re-evaluates alignment with the input text and adjusts the layout plan accordingly, modifying selected entities' attributes.
The updated layout is immediately transformed into two types of spatial conditions: a semantic layout, encoded from the global prompt and the entity local prompts, and depth map, rendered from the relative depth of each object given the scene geometry and camera view. These spatial priors are encoded and injected into the latent features of the pretrained diffusion model guide the generation process with precise semantic and geometric constraints. In contrast to iterative editing methods that regenerate images after layout revision, CoT-Diff performs all reasoning and control within a single diffusion round, avoiding redundant denoising while maintaining stepwise structural consistency. To support multi-source conditioning, we introduce separate semantic and depth control branches, implemented via lightweight Low-Rank Adaptation (LoRA) modules [16], and propose a condition-awan attention mechanism that spatially disentangles different modalities and injects control signals only into relevant image regions.
To support structured layout supervision for training, construct automatically annotated 3D layout dataset based on ÉliGen [48] and LooseControl [4]. Building upon the original annotations of global prompts, per-entity descriptions, and 2D bounding boxes, we incorporate monocular depth estimation and segmentation models to recover entity-level depth and generate 3D bounding boxes via geometrie fitting. To validate CoT--Diff, present new benchmark, dubbed 3DSceneBench. consisting of diverse and complex compositions of spatial relationship. On 3DSceneBench and two existing T2I benchmarks [34, 17], CoT-Diff outperforms state-of-the-art diffusion baselines, including FLUX and RPG [47]. by up to 34.7% in terms of complex scene spatial accuracy.
This paper makes the following contributions:
We propose CoT-Diff, a novel framework that tightly couples multimodal large language models (MLLMs) with diffusion models for stepwise 3D-awar image generation.

---

We introduce an inline layout reasoning mechanism, where MLLM dynamically updates the 3D layout at each denoising step, enabling CoT-style spatial control within a single diffusion round. We design disentangled spatial control modules that convert each layout plan into semantic masks and depth maps, which are selectively injected via condition-awan attention mechanism. We build an automatically annotated 3D-aaaree layout dataset, enabling entity-level layout supervi- sion and structured evaluation of spatial consistency.
2 Related Work
2.1 Text-to-Image Diffusion Models
Recent advances in diffusion models have greatly improved the quality and diversity of text-to-image (T2I) generation [15, 37]. Latent Diffusion Models (LDMs)) [35, 31] further enhance efficiency by operating in a compressed latent space. Modern systems like SD3 [11], Hunyuan-dit [22]. and FLUX [5] adopt transformer- saasseerme--beeed backbones such as DiT [30] and leverage pretrained text encoders (e.g., CLIP [32], T5 [33]) map text into rich visual content. However, these models treat text as static global condition and lack explicit control over spatial composition. When dealing with complex scenes containing multiple entities and spatial relationships [49, 27], they often fail to reason about relative positioning, depth, or occlusion. This limitation has led to efforts incorporating layout or cructurr-based conditions into the generation pipeline.
2.2 Layout-guided Image Generation
To improve spatial controllability, layout-guided methods introduce object-level structure into T2I generation. Training- -bsed approaches [21, 40, 50] inject layout or mask annotations into attention modules during training. EliGen [48] pushes this further by using fine-grained entity-level prompts. Other methods avoid retraining by designing plug-and-play mechanisms. For example, MultiDiffu- sion [3] applies region-speciff denoising followed by fusion; RPG [47] and RAG-Diffusion [9] use lssize-and-concatenaa schemes to combine regional latents. While these methods improve control, they mainly operate in 2D and lack true 3D spatial reasoning. More recent works explore 3D-aware generation [4, 10], but often rely manually defined layouts and still lag behind top-performing 2D models in visual quality. These limitations motivate automated 3D planning and controllable generation directly from text.
2.3 MLLM-GGrunded Diffusion
Large Language Models (LLMs) exhibit strong reasoning capabilities through techniques such as chain-of-thought prompting [42] and reinforcement fine-tuning [13]. Their multimodal counterparts (MLLMs) [25, 53, 36] extend this ability to visual-linguistic understanding, enabling structured scene interpretation from natural language. LayoutGPT [12] and Layout [38] use MLLMs to convert captions into 2D or 3D layouts, which can support downstream image generation. RPG [47] incor- porates layout planning into diffusion synthesis using regional generation. Despite these advances, existing MLLM-guided diffusion systems [41, 24] focus primarily on static layout generation and 2D spatial arrangements. They lack iterative reasoning performance dense or physically plausible scenes. contrast, Plan2Pix uses MILMs for dynamic 3D planning and integrates feedback loop that adjusts layouts based on generation outcomes, enabling more accurate and semantically aligned control over complex scenes.
full 3D understanding, which limits their
3 Method
3.1 Overview of CoT-Diff Framework
As illustrated in Figure 2, CoT-Diff enables step-by-step coupling between MLLM and diffusion model, allowing layout reasoning and image generation to proceed tandem. At each step, the updated 3D layout is converted into semantic masks and depth maps, which encode spatial priors and are injected into the diffusion model via LoRA modules and a condition-awar attention mechanism for disentangled and precise control.
3

---

MLLM
COT
Step
Step 2
GOT
COT
Step N
3d Layout
Feedback 2
Feedback N
Prompt
3D Luyont
A studious woman is engrossed in her research, her meticulously organized
Prompt
Prompt
desk positioned directly
Feculback Category
Guidance
Guidance
in front of her in the
Roll back
Continue
quiet alcove of the library.
Image
Image
Image
FLUX
Denaise
Step
Denoise
Step
Denoise
8
Step N
Overview of the CoT-Diff framework. Given a prompt, an MLLM first plans a 3D scene layout and then collaborates with the diffusion model in a step-by-step manner. At each denoising step, the MILM inspects intermediate predictions, refines the layout through CoT--style reasoning, and provides updated guidance the diffusion model.
Figure 2:
3.2 MLLM-guided 3D Scene Planning with Stepwise optimization
We propose a CoT-lized reasoning mechanism, where a MLLM collaborates with the diffusion model to perform stepwise planning and optimization. Unlike previous approaches that decouple layout from generation, our framework tightly integrates MLLM-driven layout reasoning into the sampling trajectory, enabling dynamic updates the 3D plan during generation.
Initial 3D Scene Planning. We use a MLLM to parse the input prompt and generate a structured 3D scene plan. The model identifies key entities, estimates their physical attributes, and determines their spatial arrangement the scene, The MLLM first identifies & salient entities from the prompt, forming entity set:
= (e
For each entity Ej E, the MILMM generates local text prompt py describing its appearance and attributes. It also predicts a plausible 3D size size, = (1,,w, and a spatial coordinate pos, (my Uj. 2z)- The overall output is represented
(MLLM(e5lP) - (Fjz size;,oos,).
(2)
This planning is based on entity semantics and spatial relationships implied in the prompt, guided by the commonsense knowledge embedded in the MILM. The final 3D scene plan is:
SP (p, (p)) size,, pos,)
Stepwise Feedback-base Refinement. While the initial 3D layout provides strong spatial guidance, discrepancies may still arise during generation. To address this, CoT-Diff introduces an iterative optimization loop driven by MLLM feedback. At each timestep f, the model predicts clean image from noisy latent 2, via:
201: 2t t V2-
The MILM evaluates čoje comparing against both the input prompt and the current plan SP. misalignment detected, the MLLM proposes refined attributes for selected entities (e.g., size, or pos,), producing an updated plan:
Sp' Refine(SP. Zoje: p);
(5)

---

Labeling Pipeline
[b] 3D Layout Control
DiT
Grounding
Prompt
TS Text
Depth
Prompt
Rabbit
Encoder
Self Attention
Deptt Mtap
robot rabbit driving yellow car city street.
Segment
Projection
Encoder
Projection
Point
(d)
Attention Map
[c] Condition Injection Lora
P
P3
X
C
P
P2
X
Cp
Tokens
P
QKV
Lordel
QKV
X
QKV
C
Features
Figure 3: Illustration of 3D layout conditioning and condition-awar attention. (a) 3D bounding boxes are automatically labeled from input images using depth estimation, segmentation, and projection. (b) Text and spatial inputs are encoded into semantic and geometric conditions via T5 and VAE encoders. (c) Condition Injection LoRA activates moddlittyspeeiin branches during QKV projection. (d) learned attention mask enforces condition-wise separation and spatially localized injection.
which is used to reguide the denoise process at timestep f. This rediic--valluate-refne cycle progressively enhances spatial and semantic consistency. Evaluation continues for a maximum steps or until the MLLM deems the layout correct.
Unlike prior methods that rely on multi-round generation, our framework performs planning and opti- mization within a single diffusion process, enabling efficient and interpretable 3-awwrre generation.
3.3
3D Layout Control
Given the 3D scene plan SP from Section 3.2, we extract two types of conditional signals: (i) semantic layout and (ii) depth maps. These jointly injected into the diffusion model via condition-awart attention mechanism, enabling precise control over the spatial and semantic structure.
Semantic Layout Condition. This module focuses on injecting global and entity-specific semantics into corresponding regions of the image. As illustrated top-left of Figure 3(b), we extract global embedding P and local embeddings {Pr} from the global prompt and local prompts (p,) using T5 encoder, and train a Semantic LoRA module, denoted LoRAst, control this injection. The model is optimized using the following conditional flow matching loss:
Lst. E.z,c [1]e C, P.P. Pt) 3z
(6)
where Uy is the velocity field predicted by the model augmented by LoRAst.
Depth Map Condition. This module focuses extracting depth information to guide spatial structure and object positioning, as illustrated on the bottom-lef of Figure 3(b). Starting from the 3D scene plan SP, we obtain the of 3D bounding boxes {By} and render corresponding depth map D. This depth map is then encoded using the DiT VAE to produce the latent depth condition Cp- We inject Cp into the diffusion model via dedicated Depth LoRA module, LoRApM, which optimized using the following conditional flow matching loss:
LDM EL,e [2,,, Cp, Pemb) ue {2(
5

---

where is the velocity field predicted by the model augmented by LoRAst and LoRADM. 2e the target velocity field, and Pams is the all text prompt embedding.
Condition-Aware Attention Mechanism. To integrate semantic and depth conditions without messeinterfer,,ce,, design Condition Attention mechanism that selectively controls how different condition types attend to the image tokens, ensuring spatial alignment while avoiding modality entanglement. We begin by constructing unified token sequence by concatenating the semantic tokens, image tokens, and depth tokens, as illustrated in Figure 3(c):
S = PI.... Pky CD,
where P is the global prompt token, { are local prompt tokens for each entity, Cp is the depth condition token, and X denotes the image latent tokens.
To guide attention computation, we construct binary attention mask M {0, 1)15)xx x|S| that controls which tokens are allowed to attend to one another, as illustrated in Figure 3(d). The mask is designed to satisfy three principles: (i) tokens from different condition sources mutually isolated to prevent cross-condition interference, i.e.
b)a = O a b E (Pi...., Pk, Ca), a b;
99)
(ii) all tokens are allowed to self-attend i.c., M (a, a) for all a E and (iii) interactions between condition tokens and image latents are modulated by spatial masks. Specifically, global conditions Po and C2 attend to all latents:
M(P,, X) = M(X, Po) = M(C,, X) = M(X, = 1,
(10)
while each local prompt P3 is restricted to its associated region:
M(P,, X) = M(X, P3) = patchify(my),
(11)
where my is a 2D mask derived by projecting the 3D bounding box Bj onto the image plane. With this mask M. the attention output computed as:
(QK
Attn Q, K, V) Softmax
log
(12)
Vile
This formulation allows the model to attend over spatially relevant regions for each condition while avoiding semantic-depth entanglement and cross-entity confusion.
3.4 3D-Aware Dataset Construction Pipeline
As shown in Figure (a)) construct 3D-aware dataset which automatically generates 3D bounding boxes for each object. This dataset includes global prompts, local prompts, and 3D bounding boxes.
We generate depth maps using a monocular depth estimator (Depth Pro) [6], and apply the Segment Anything Model (SAM) [20] with 2D bounding boxes extract segmentation masks. For each object, we extract its depth region and back-project it into 3D point cloud: C {xx,, y, dz(x, u)) (x, y) E m.). where d, denotes the depth values. We then an oriented 3D bounding box with minimal volume: ming Volume (B) CC).
Finally, we re-project the 3D bounding boxes onto the image space and render depth maps, ensuring accurate and consistent 3D representations.
Experiments
4.1 Experimental Setup
Implementation Details. Our method implemented using Gemini 2.5 Pro as the default multimodal language model and FLUX.1-schnee [5] as the diffusion model. We perform all inferences with 20 denoising steps and use 5 different random seeds for fairness. The LoRA scale set to 1. All models are implemented in PyTorch and executed on NVIDIA A 1100 GPUs.
Baselines. We compare our method with two categories of existing approaches: (1) Pre-trained T2I diffusion models, including SD1.5 [35], SDXL [31], PixArt [9], and FLUX-schnell [5]: (2) Layout-controlle methods, including RPG [47]. EliGen [48], and Loose Control (LC) [4], which incorporate external layout representations to guide spatial placement of objects. All baselines are evaluated under consistent settings using their official or publicly available implementations.

---

Front
Behind
Front Left
Front Right
Back Left
Back Right
Multi Rela
Complex
PixArt
Flux
Loose Control
RPG
EliGen
Col-Diff (Ours)
chair in desk in the
behind
fhonr left ol'a dog in desert
bicycle the front right a the street
horse a bay on beach
bench the back right bird on the
rigtt coms right of sola operinet
the makk bouch lighlly chevel
snowy
landscupe
Figure 4: Qualitative comparison of CoT-Diff and baselines across spatial relation categories.
Datasets. We evaluate CoT-Diff with
Table Dataset statistics. %Complex
dataset, 3DSceneBench, which we design to validate the generation ability in complex 3D scenes, and two existing datasets, DVMP [34] and 121-CompBench [17, 18], for general compositional image generation. As shown in Ta- ble 1, BDSceneBench includes more diverse and spatially grounded relationships compared existing benchmarks. Further evaluation details can be found in the Appendix G.
is the rate of spatial complex prompts evaluated by GPT4. (See Appendix F).
Datascts
3DSecneBencc DVMP T21-CompBench
# Prompts Complex
800
200
2400
95.3
23.0
27.6
1. Benc:: We design the 3DSceneBench dataset to cover wide range of realistic 3D spatial relationships. All prompts are categorized into two major types based on relational complexity: Basic Relation and Hard Relation. The Basic Relation group includes six canonical spatial configurations: Front, Behind, Front Left, Front Right, Back Left, and Back Right (e.g., "a [ in front of (object2) in the [scene]"). The Hard Relation group contains more challenging compositions, including Multi-Relation (e.g., "a [objectl] [relation1] ) (object2), a {object3] (relation2) (object4) in the scene") and Complex prompts to reflect complex 3D spatial logic. We generate 100 prompts for each of the eight relation types, resulting in total of 800 prompts. All prompts manually validated to ensure semantic clarity and spatial correctness. Constructing process of 3DSceneBench detailed Appendix
DVMP: This dataset is built by randomly pairing 38 objects with 26 attributes (including 13 colors), covering both single-object and multi-objeet prompts, with 100 examples for each setting.
3. T2I-CompBench: This benchmark contains multi-object prompts with relatively common positions, serving as a baseline for evaluating spatial and semantic consistency.
4.2 Main Results of CoT-Diff
3DSceneBench. We first present a qualitative comparison as shown Figure 4. Traditional diffusion models (SD1.5, SDXL, PixArt, FLUX) perform poorly due the absence of explicit layout
7

---

Table 2: Text-to-image alignment performances of CoT-DDiff and other baselines the 3DSceneBench dataset. The best values are in blue and the second best values are in green
Basic Rela
Hard Rela
Models
Front
Rehind
Front Lefi
Front Right
Back Left
Back Right
Multi Rela
Complex
SD1.5
32.8
36.8
26.5
28.6
34.1
33.7
21.8
34.0
SDXL
39.3
43.9
36.7
35.6
35.4
40.2
27.7
34,9
PixArt
39.4
43.7
33.8
32.2
35.4
36.4
27.0
37.65
Flux
48.0
45.9
40.7
40.8
33.3
37.4
35.2
40.5
RPG
40.5
37.1
39.8
38.8
42.1
EliGen
43.6
42.9
48.4
52.5
46.7
39.9
40.6
LC
25.3
28.5
26.9
29.0
35.5
27.0
12.7
23.7
Col-Diff
54.9
55.2
66.4
69.0
64.2
64.5
48.7
50.8
Table 3: T2I alignment performance of CoT-Diff and baselines on DVMP and 121-CompBench.
DVMP
T2I-CompBench Spatial
Models
Single
Multi
Color
Shape
Texture
Non-Spatial
Complex
SD1.5
65.0
37.8
37.5
38.8
44.1
09.5
31.2
30.8
SDXL
76.8
59.0
58.8
46.9
53.0
21.3
31.2
32.4
Art
73.5
44.0
66.9
49.3
64.8
20.6
32.0
34.3
Flux
66.8
72.5
74.1
57.2
69.2
28.6
31.3
37.0
RPG
74.0
30.0
64.1
45.8
56.6
48.8
30.2
44.3
EliGen
78.7
78.5
72.8
58.7
66.8
53.6
31.2
43,9
LC
51.6
45.7
27.9
29.1
26.4
26.7
30.2
28.1
CoT-Diff
80.9
78.8
78.1
61.1
69.1
55.8
31.3
50.0
guidance. While layout-guided methods like RPG and EliGen incorporate positional control, they still demonstrate limited capability in modeling complex depth relationships. contrast, CoT-Diff accurately synthesizes the specified spatial relationships by tightly coupling layout reasoning and generation, dynamically refining the scene plan, and injecting spatial guidance at each denoising step.
For quantitive experiment, Table 2 shows the spatial scene alignment accuracy of CoT-Diff compared to the baselines on the 3DSceneBench dataset. Overall, CoT-Diff consistently outperforms all baselines across all spatial relationship categories. Numerically, CoT-Diff achieves the highest alignment scores in both basic and complex relational categories, with improvements ranging from +10.2% to +22.4% over the best baselines.
DVMP and T2I-Compeench. Table 3 summarizes the T2I alignment performance of CoT-Diff and baseline models on DVMP and 121-CompBench. On DVMP, CoT-Diff achieves the highest scores across all attribute-binding categories-
Shape, and Texture-ouperfomning the strongest baseline (EliGen) by 2.3% 5.3%. This confirms that semantic-aware generation in CoT-Diff strengthens consistency between visual attributes and textual descriptions. contrast, traditional diffusion models such SD1.5, SDXL, and PixArt show clearly inferior performance due to the lack of layout or semantic control. On T2I-CompBench. CoT-Diff also leads in most categories, with notable improvements in Spatial (+2.2%) and Complex (+6.7%) over EliGen. While EliGen incorporates 2D layout planning, its performance degrades in cases involving deeper relational reasoning. In contrast, CoT- demonstrates generation guided by semantic planning. Overall, CoT-Diff excels not only in 3D-structured scenes but also generalizes well to traditional T2I alignment tasks.
robust behavior by leveraging structure-aware
4.3 Ablation Study of CoT-Diff
Component Analysis. We conduct an ablation study to evaluate the contribution of each core component in CoT-Diff, starting from the FLUX base model. As shown in Table 4, adding semantic layout guidance significantly improves multi relationship (+4.9) and complex scene (+3.2) accuracy by providing better object placement control. Introducing depth control further enhances performance, especially in multi relationship (+1.6) and complex (+4.8) settings, by resolving occlusion and ensuring accurate object ordering, Finally, the full CoT-Diff model, with optimized settings, achieves

---

Table Ablation study of CoT-Diff. We progressively add layout, depth, and optimization compo
nents top of the FLUX base.
Model Variant
Front
Behind
Front Left
Pront Right
Back Left
Back Right
Multi Rela
Complex
FLUX
48.0
45.9
40.7
40.8
33.3
37.4
35.2
40.5
Semantic Layout
44.5
48.0
56.9
58.0
51.9
50.6
40.1
43.7
Depth Map
51.3
51.2
62.2
61.4
59,9
56.7
41.7
48.5
Optim (Full CoT-Dit)
54,9
55.2
66.4
69.0
64.2
64.5
48.7
50.8
Table 5: CoT-DDiff with different MILMs.
Table 6: User Study.
Models
Front Behind Front Left
Front Right
Back Left
Back Right
Method
3D Layout Accuracy
Visual Quality
Flux
48.0
45.9
FLUX
2.71
3.56
CoT-Diffowss
48.3
49.1
39.2
61.4
58.3
58.2
RPG
3.57
3.75
CoT-Diffarr-oo
53.2
53.3
67.8
62.9
62.7
EliGen
3.65
3.81
CoT-Diffoeudi
54.9
55.2
66.4
69.0
64.2
64.5
CoT-Diff (Ours)
4.08
4.12
the highest scores across all categories, highlighting the complementary benefits of semantic layout, depth, and 3D layout optimization.
Robustness across Different MLLMs. We further evaluate CoT-Diff with several different MLLMs. including Qwen2.5-VL [2]. GPT-4o [1] and Gemini 2.5 Pro. as detailed in Table 5. All tested MLLMs significantly improve spatial consistency over the FLUX baseline, but CoT-Diff when paired with Gemini 2.5
Pro consistently achieves the best results, particularly in handling front and behind relations. This result indicates that Gemini 2.5 Pro offers superior semantic planning capabilities for 3D layout generation, ultimately leading more accurate spatial alignment.
User Study. We conducted a user study comparing CoT-Diff against four baselines using 200 complex prompts from 3DSceneBench. Six evaluators rated 3D Layout Accuracy and Visual Quality on a 1-5 scale. As shown in Table 6, CoT-Diff significantly outperformed all baselines, achieving the highest scores in both Layout Accuracy (4.08) and Visual Quality (4.12). This demonstrates our method's ability to enhance spatial control without the fidelity degradation.
4.4
3D Layout Consistency
Neares:
Prompt: a vase and a chair in the room
Purthest
Consktreey
20.
0.7
EliGen
0.6
Bee((=]]) CUT-DiT(Oann)
0.3
CoT-Diff (Ours)
Depth Difference (A)
Figure 5: 3D consistency under depth variation. (Left) CoT-Diff accurately adjusts spatial placement across relative depths, while EliGen struggles maintain correct object scale and distance. (Right) CoT-Diff consistently outperforms EliGen in 3D alignment across varying depth gaps.
This experiment evaluates how well different models maintain spatial consistency when object depth varies. We select two central objects from 3DSceneBench and construct base 3D layout. One object is shifted along the camera axis by multiple of A, creating modified layouts. Each generated image is compared to the bround-truth layout by computing depth- based consistency score. See Appendix H for details on setup and metric. Figure presents both qualitative (left) and quantitative (right) results across depth differences. EliGen performs well at close range, but its consistency drops sharply as the depth difference increases. In contrast, CoT-Diff maintains stable consistency across all ranges, demonstrating stronger adherence to 3D layouts.

---

Table Inference time and success rate of different methods.
Table 8: Inference time across MLLM steps.
MLLM slep
Planning Time
Feethack Time
Generation Time
Whole
Planning Time
Feedback Generation Time
Whole Time
Time
Method
Success
Time
hase(O)
23.4
19.3
19.5
62.2
Flux
28.1
28.1
41.6
RPG
19.8
8.5
28.3
49.6
23.4
58.8
21.4
102.5
EliGen
5.2
55.7
60.9
53.2
125.0
Image-CoT CoI-Ditf
114.4
114.4
61.2
23.4
99.2
23.3
145.9
23.4
56.8
21.2
101.4
75.4
ada (2.9)
23.4
56.8
21.2
101.4
4.5
Runtime Efficiency and Spatial Success Rate
To evaluate the efficiency and effectiveness of CoT-DDff in complex scenes, randomly sample 50 real prompts from 3DSceneBench and compare different methods terms of runtime efficiency and spatial success rate. For efficiency, as shown in Table 7, while CoT-Diff's iterative feedback time adds overhead compared to simpler layout-based methods, its total inference time remains comparable to other CoT frameworks like Image-CoT [14]. This trade-off enables significantly improved spatial consistency. CoT-Difff achieves a 75.4% success rate, substantially outperforming baselines. In contrast, FLUX lacks layout modeling, while RPG and EliGen show limited spatial precision.
Furthermore, we evaluated the inference time across different MLLM optimization steps Table We found our adaptive early-stopping strategy (ada) achieves high performance in just 2.9 steps average, striking an effective balance between generation quality and computational cost.
Conclusion
We propose CoT-Diff, 3D-aware image generation framework that tightly couples MLLM reasoning with diffusion. It performs layout planning and generation jointly within single diffusion round, leading to improved spatial structure and semantic alignment. By combining semantic and depth condition with a condition-ane awaae attention mechanism, CoT-Diff enables precise and disentangled multimodal control. Considering the limmitations in Appendix K, we aim to integrate layout reasoning directly into the generative model for end-to-end CoT-lized generation in the future.
Acknowledgement
This work was supported in part by the Natural Science Foundation of China (No. 62332002, 62202014, 62425101), and the Alibaba Innovative Research Program of Alibaba DAMO Academy.
References
[1] Achiam, S. Adler, Agarwal, L. Ahmad, Akkaya, L. Aleman, Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] S. Bai, K. Chen, X. Liu, J. Wang. W. Ge, Song. K. Dang. P. Wang. S. Wang. J. Tang, al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502. 13923, 2025.
[3] O. Bar-Tal, L. Yariv, Y. Lipman, and T. Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. 2023.
[4] S. F. Bhat, N. Mitra, and Wonka. Loosecontrol: Lifting controlnct for generalized depth conditioning. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
BlackForestLabs. Flux: A powerful tool for text generation. hutps://blackforettaaee ai/, 2024. Accessed May 2025.
[6] A. Bochkovskii, A. Delaunoy, H. Germain, M. Santos, Y. Zhou. S. R. Richter, and V. Koltun. Depth pro: Sharp monocular metric depth in less than second. arXiv preprint arXiv:2410.02073, 2024.
[7] D. Chen, R. Chen, Zhang, Y. Wang, Y. Liu, H. Zhou, Q. Zhang. Y. Wan, P. Zhou, and L. Sun. Milm-as-a- judge: Assessing multimodal llm-as aam-as---judge a-judge with language benchmark. In Forty-first International Conference Machine Learning, 2024.
10

---

[8] X. Chen, Z. Wu, X. Liu. Z. Pan, W. Liu, Z. Xie, X. Yu, and C. Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint (rXi::5001178111, 2025.
[9] 7. Chen, Y. Li, H. Wang, Chen, Z. Jiang, Li, Q. Wang, J. Yang, and Y. Tai. Region-aware text-to-image generation via hard binding and soft refinement. arXiv preprint arXiv:241 1XXt::41110665555 1,06558, 2024.
[10] A. Eldesokey and P. Wonka. Guild-a-scene, Interactive 3d layout control for diffusion basod image generation. arXiv preprint 2rXiv:2408. 14819, 2024.
[11] P. Esser, Kulal, A. Blattmann, R. Entezari, Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel. et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[12] Feng. W. Zhu, T.-j. Fu, V. Jampani. A. Akula, X. He, Basu, X. Wang, and W. Y. Wang. Layoulgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 36: 36:18223 18225-22511 18250, 2023.
[13] D. Guo, D. Yang, H. Zhang. J. Song. R. Zhang, R. Xu, Q. Zhu, S. Ma, Wang, Bi, et al. Deepseek-r1: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501112945. 2025.
[14] Z. Guo, R. Zhang, C. Tong, Z. Zhao, Huang. H. Zhang, M. Zhang. J. Liu, S. Zhang. P. Gao, et al. Can we generate images with cot? let's verify and reinforce image generation step by step. arXiv preprint arXiv:2501.13926 2025.
[15] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840-6851, 2020.
[16] E. J. Hu, Shen, P. Wallis, adaptation of large language models, ICLR, 1,2)3, 2022.
Allen-Zhu, Y. Li, Wang. L. Wang, W. Chen, et al. Lora: Low-rank
[17] K. Huang. K. Sun, E. Xie, Z. Li, and X. Liu. T2i-compbench: comprehensive benchmark for open- world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36: 78723-78747, 2023.
[18] K. Huang, C. Duan, K. Sun, F. Xie, Li, and X. Liu. T2icomphencch+:: An enhanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[19] D. Jiang, Z. Guo, R. Zhang, Z. Zong, H. Li, L. Zhuo, S. Yan, P.-A. Heng, and H. Li. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot. arXiv preprint arXiv:2505.0077 2025.
[20] A. Kirillov, Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, Xiao, S. Whitchead, C. Berg, W.-Y. Lo, al. Segment anything. Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
[21] Y. Li, H. Liu, Q Wa, Mu, Yang, J. Gao, C. Li, and Y. J. Lee. Gligen: Open-set grounded text-to-image image generation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 22511-22521, 2023.
[22] Z. Li, J. Zhang, Q. Lin, J. Xiong, Long, X. Deng, Y. Zhang. X. Liu, M. Huang, Z. Xiao, et al. Hunyuan- dit: A powerful preprint ar arXi:22455.087
diffusion transformer with fine-grained chinese understanding. arXiv 2024.
[23] L. Lian, B. Li. A. Yala, and T. Darrell. Lim groundr diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305. 2023.
[24] M. Liu, L. Zhang, Y. Tian, X. Qu, L. Liu, and T. Liu. Draw like artist: Complex scene generation with diffusion model via composition, painting, and retouching. arXiv preprint arXiv:2408. 13858, 2024.
[25] Z. Liu, Z. Sun, Y. Zang, X. Dong, Y. Cao, H. Duan, D. Lin, and J. Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025.
[26] K. Mishchenko and A. Defazio. Prodigy: An expeditiously adaptive parameter-free leamer. arXiv preprint 200. 2023.
[27] C. Mou, X. Wang. L. Xie, Y. Wu, J. Zhang, Z. Qi, and Y. Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to- texaaee- diffusion models. In Proceedings of the AAAI conference on artificiall intelligence, volume 38, pages 4296-499 4304,430,2000 2024.

---

[28] M. Ning, B. Zhu, Y. Xie, B. Lin. J. Cui, L. Yuan, D. Chen, and L. Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arX1v:231.116103 2023.
[29] D. Park. S. Kim, T. Moon, M. Kim, K. Lee. and J. Cho. Rare-to-frequent Unlocking compositional generation power of diffusion models on rare concepts with Ilm guidance. arXiv preprint 27X2v:2410.22336. 2024.
[30] W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[31] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
[32] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[33] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1667, 2020.
[34] R. Rassin, E. Hirsch, D. Glickman, S. Ravfogel, Y. Goldberg, and G. Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Information Processing Systems, 36:3536-359,, 2023.
[35] R. Rombach, A. Blattmann, D. Lorenz, P. Esser. and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 10674-10685 IEEE, 2022.
[36] H. Shen, P. Liu, J. Li, C. Fang, Y. Ma, J. Liao, Q. Shen, 7. Zhang, K. Zhan, Q. Zhang, et al. Vlm-rl: A stable and generalizable r1-style large vision-language model. arXiv preprint arX1v:2504.0715,, 2025.
[37] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arX1v:20100002502 2020.
[38] F.Y. Sun, W. Liu, S. Gu, D. Lim, G. Bhat, F. Tombari. M. Li, N. Haber, and J. Wu. Layoutvlm: Differentiable optimization of 3d layout via vision-language models. arXiv preprint arXiv:2412.02193. 2024.
[39] Q Wang, Y. Luo, X. Shi. X. Jia. H. Lu, T. Xue, X. Wang, P. Wan, D. Zhang. and K. Gai. Cinemaster: A 3d- aware and controllable framework for cinematic text-to-video generation. arXiv preprint arXiv:2502.08633 2025.
[40] X. Wang, T. Darrell, Rambhatla, R. Girdhar, and I. Misra. Instancediffusion: Instance-level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition, pages 6232-6242, 2024.
[41] Z. Wang, A. Li, Z. Li, and X. Liu. Genartist: Multimodal I1m as an agent for unified image generation and editing. Advances in Neural Information Processing Systems, 37: 1777444--128395 2024.
[42] Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi. Q. V. Le, D. Zhou, et al. Chain-of thouhht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-246 244337, 2022.
[43] T.H.. Wu, L. Lian, J. E. Gonzalez, B. Li, and T. Darrell. Self-correcting llm-controlled diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6327-6336, 2024.
[44] Y. Wu, Z. Zhang, J. Chen, H. Tang, Li, Y. Fang, L. Zhu, E. Xie, H. Yin, L. Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.0444 2024.
[45] J. Xie, Z. Yang, Y. Yang, et al. 3dis: Depth-driven decoupled instance synthesis for text- text-to-image generation. In The Thirteenth International Conference on Learning Representations.
[46] L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth anything: Unleashing the power of ungee-ccae unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371-10383 2024.
12

---

[47] L. Yang. Z. Yu, C. Meng, M. Xu, S. Ermon, and B. Cui. Mastering text-lo-image diffusion: Recaptioning. planning, and generating with multimodal llms. In Forty-first International Conference Machine Learning, 2024.
[48] H. Zhang, Z. Duan, X. Wang, Y. Chen, and Y. Zhang. Eligen: Entity-level controlled image generation with regional attention. arXiv preprint arXiv:2501.011 00099977.17 2025.
[49] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control lext-to-image diffusion models. In Proceedings of IEEE/CVF international conference on computer vision, pages 3836-3847, 2023.
[50] D. Zhou, Y. Li, F. Ma, X. Zhang, and Y. Yang. Mige: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 6818 6818-6828, 2024.
[51] Zhou. Li, Z. Yang.
Yang. Dreamrenderer: Taming multi- instance attribute control in large-scale text-to-image models. arXiv preprint arXiv:2503.12885 2025.
[52] Zhou, J. Xie, Yang, and Yang. 3dis-flux: simple and efficient multi-instance generation with dit rendering, arXiv preprint arXiv:2501.0513/ 2025.
[53] H. Zhou, X. Li, Wang, M. Cheng. T. Zhou, and C.-J. Hsieh. RI-zero's" aha moment" in visual reasoning on a 2b non-sft model. arXiv preprint arXiv:2503.05132, 2025.
[54] X. Zhou, V. Koltun, Krähenbühl. Simple multi-dataset detection. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7571-7580, 2022.
13

---

A LLM Instruction for CoT-Diff
Key Identity Parsing Prompt Template
You are tasked with identifying and extracting all the real object names from a caption.
An object name refers to any tangible or physical entity mentioned in the caption. Ensure not to include any adjectives or single-word descriptions that do not refer specific object, such as "background.
Please follow these instructions:
Identify all object names in the caption in the order they appear. Maintain the exact wording of each object name as it is in the caption, including case consistency.
Output the object names in Python list format.
For example, consider the following caption:
<In-context Examples>
Now, given the following caption, extract the object names in the same format:
<caption>
MI.uM-u 3D Scene Planning Prompt Template
As 3D scene layout planner, generate quantitative 3D layout (size, position) for specified entities based on a text caption.
Input:
1. A text caption describing the scene.
A list of important entity names in the scene.
Output: JSON object with two keys:
scene_ paaameepaameterss and entity _aayout.
peene_parameter: Describe the overall scene.
scene_size (meters): Approximate scale of the main subject area.
camera_pitch_ (degrees): Vertical camera angle (positive = downward).
entity_ A list of objects, each including:
-entity_nmm name: Name the entity.
size: [length, width, height] in meters. Should be large enough to be visible in the scene (each dimension scene_size/10).
possiiion [X. Y, Z] in meters, centered around the ground origin (Y 0). Enforce
explicit spatial relationships in the caption.
Coordinate System: Right- handed. Origin (0, 0, 0) is the ground center. +X right, +Y up, +Z = into the scene.
14

---

Prompt Template: Iterative 3D Layout Optimization Assistant
System Role: You AI Layout Optimization Assistant. Your core mission is to iteratively
refine 3D JSON layouts through multi-turn dialogue with the user.
Key Principles:
Entity Focus: Evaluate and modify only the entity_list -iist items for each turn.
2. Viewer's Perspective: Interpret all spatial terms (e.g., "left", "right") from the viewer of the generated_inen _imgge.
3. Iterative Learning: Improve layout step-by-step based on prior adjustments.
4. Adhere to Task Definition: Strictly follow user-provided format and criteria. Consider past actions and outcomes to inform new proposals.
5. Historical Context:
Task:
Iteratively optimize the 3D JSON layout to align the generated _mmage with the text_caption, improving the clarity and spatial correctness entities the antity_list
Per-Iteration Inputs:
cext_caption: (string) natural language description of the scene.
entity_list: (list of strings) entities optimize.
current_layour (JSON) 3D layout with size = [X len, Z width, Y height] and position [X, 2]. = is ground.
generated_image image: the image rendered from the current layout.
Step-by-Step Process:
Step 1: Parse Inputs
Receive and acknowledge all inputs.
Step 2: Evaluate Alignment (for entity_list _1ist)
2.1 Discernibility:: Is each entity clearly visible?
2.2 Verifiability: Are their described attributes verifiable?
2.3 Spatial Accuracy: Are spatial relations correct from viewer's perspective?
2.4 Determine isaligned: Set to true if 2.1-2.3 pass; else false.
Step Diagnose Misalignment (if isal i ggnd = false)
3.1 Identify which entities failed which checks.
3.2 Classify each as Incorrect or Insufficient.
3.3 Refer to previous adjustments and compare changes.
Step 4: Revise Layout
4.1 Strategize updates to size, position, or orientation of problematic entities.
4.2 Adjust other entities only if they cause conflicts.
4.3 Ensure layout format valid.
Step Generate Output
5.1 Text: Explain isaligned result and edits made.
5.2 JSON:
"isaligned";
<true/false>, "optimized_ layout":
<layout_object)
}
User Prompt Format:
text_ca :att_ccption <caption>
antity_list. <entities>
current current_layout: <layout>
generated_ image: <image>
15

---

Table 9: Performance across different optimization steps.
"ada (2.9)' is our adaptive strategy.
Basic Rela Front Right
Hard Rela
Steps
Front
Behind
Front Left
Back Left
Back Right
Multi Rela
Complex
base(O)
51.3
51.2
62.2
61.4
59.9
56.7
41.7
48.5
I
53.3
51.9
63.4
66.0
61.2
59.8
45.8
48.4
2
57.6
53.6
66.6
67.1
62.8
63.3
47.9
50.4
3
55.2
54.5
67.5
69.5
62.2
64.9
47.6
49.7
4
56.0
54.5
68.3
69.9
61.9
64.9
48.8
51.4
5
56.2
56.4
67.9
71.2
64.4
66.1
51.0
51.8
ada (2.9)
54.9
55.3
66.4
69.0
64.1
64.5
48.7
50.8
Table 10: Comparison with Unified and Image CoT models.
Basic Rela Front Right
Hard Rela Multi Rela
Steps
Front
Behind
Front Left
Back Left
Back Right
Complex
Janus-Pro
35.2
36.5
29.7
24.9
24.9
22.9
19.6
39.0
VILA-U
37.2
38.3
28.8
31.5
29.2
32.4
21.3
29.5
Image-CoT
48.2
49.7
56.1
61.2
56.7
57.6
46.6
T2I-R1
47.0
46.8
47.2
45.3
50.4
51.1
35.5
41.5
CoT-Diff
54.9
55.2
66.4
69.0
64.2
64.5
48.7
50.8
B Training Details
We employed FLUX.1 dev as the pre-trained DiT. For each LoRA training, we utilize 8 A100 GPUs(80GB), a batch size of per GPU. We employ the Prodigy optimizer [26] with safeguard warmup and bias correction enabled, setting the weight decay to 0.01 following OminiControl. For semantic LoRA LoRASL- we train the model 5000 iterations base on EliGen LoRA. For depth LoRA LoRADL, we train the model 30000 iterations.
C Performance across Different Optimization Steps
To validate our adaptive strategy, we evaluated the performance at different fixed optimization steps. As shown Table 9, performance consistently improves as the number of steps increases, and typically saturates around 3 steps. Our adaptive method ("ada (2.9)) achieves considerable performance in just 2.9 steps on average, striking an effective balance between performance and efficiency.
D Comparison with Unified models and Image CoT models
We added a comparison with Janus-Pro [8]. VILA-U [44], Image-CoTi [14], and T21-R1 [19]. The results, shown in Table 10, demonstrate CoT-Diff's superior performance spatial reasoning and text-image alignment. This suggests that current unified ceneration-understtading models still lack explicit reasoning capabilities compared to our CoT-Diff framework. Furthermore, existing Image- CoT methods primarily implement CoT at the semantic token level, whereas CoT-Diff introduces explicit 3D spatial reasoning, using an MLLM to generate and refine physically interpretable 3D layout precise guidance for the diffusion model.
E
Details for Constructing 3DSceneBench
Overview. 3DSceneBench aims to evaluate the T2I model's compositional generation ability for complex spatial relationship prompts across basic- and hard-relations. For basic-relations prompts, we categorize planar spatial relationships in the basic orientation in six cases: (1) front, (2) back, (3)
16

---

front left, (4) front right, (5) back left and (6) back right. These cases represent mixture of horizontal and depth attributes defining the relative positions of typically two objects. The hard-relations prompts are designed to challenge models with increased compositional complexity, categorized into two main types based on how basic relations are combined or extended: m1)ulii combining two basic-relation prompts; (2)complex, expanding a basic-relation prompt into complex scene description.
Table 11: Object Categories, Associated Scenes, and Possible Objects.
Object Category
Associated Scenes
Possible Objects
in the desert. in the jungle, on the road, on the beach
dog. mouse, sheep, cat, cow, chicken, turtle, giraffe, pig. butterfly, horse, bird, rabbit, frog, fish
Animals
bed, desk, key, chair, vase, candle, cup, phone, computer, bowl, sofa, balloon, plate, refrigerator, bag, painting. suitcase, table, couch, clock, book, lamp, television
in the room, in the studio, in the apartment, in the library
Indoor
in the desert, on the street, on the road, snowy landscape
car, motorcycle, backpack, bench, train, airplane, bicycle
Outdoor
Person
All
Woman, man, boy, girl
Complex prompt generation. We choose 50 diverse objects from the MS-COCO dataset. These objects are categorized into four types based their nature: indoor, outdoor, animal, and person. To provide contextual variation, four distinct scene contexts are defined for all categories except "person". Prompt generation is structured around basic spatial relationships and their compositions. For basic-relations prompts, we employed a template of the form "a (objectl) }/ [relation] a (object2) (scene)". Using this template with selected objects and defined scenes, we generated 200 prompts for each of the six basic-relation categories (front, back, front left, front right, back left. back right). Multi-relation prompts (200 in total) are created by combining pairs of the generated basic-relation prompts. Complex-relatin selected basic-relation prompts into descriptions of more elaborate scenes. The final dataset undergo human filtering process, similar to methods used in other benchmarks [28, 29], where human annotators review the generated prompts for suitability and quality before final inclusion.
prompts (200 in total) are generated by leveraging GPT-4o to expand
F GPT Instruction for Calculating % Complex
To measure whether a prompt contains complex spatial relationships, following the approach of [29], we ask GPT4 with the yes or no binary question using the following instructions. You are assistant to evaluate if the text prompt contains complex spatial relationships. Evaluate if complex sputial relationships or NO, without any reasoning. " Formally, the % complex of each test dataset Crest is calculated as % CComplex(Cess
contuined in the texi prompt: PROMPT, The answer format should be YES
1/|Cesl Eeecu.. 1(GPT (o) == Yes), where GPT compoce (c) is the binary answer of complex from GPT.
G Details for Evaluation
We adopt different evaluation protocols for each benchmark. For 3DSceneBench, we use UniDet to evaluate spatial layout consistency. For DVMP, we ask GPT4o with a detailed score rubric [7], following the approach of [29]. For T2I-CompBench, we follow the official evaluation scripts released [17].
GPT-based Evaluation. For DVMP dataset, we leverage GPT-4o to evaluate the image-text alignment between the prompt and the generated image. The evaluation is based on a scoring scale from 1 to 5, where score of represents perfect match between the text and the image, and a score of 1 indicates that the generated image completely fails to capture any aspect of the given prompt.
17

---

Table 12 presents the complete prompt with full scoring rubric. We convert the original score scale {1, 2, 3, 4, 5) {0, 25, 50, 75, 100), which is reflected in the reported results.
Table 12: Full LLM instruction for evaluation.
You my assistant to evaluate the correspondence of the image given text prompt. Focus the objects in the image and their attributes (such color, shape, texture), spatial layout, and action relationships. According to the image and your previous answer, evaluate how well the image aligns with the text prompt: [PROMPT]
Give score from 0 to 5, according to the criteria:
5: image perfectly matches the content of the text prompt, with no discrepancies. image portrayed most of the content of the text prompt but with minor discrepancies. image depicted some elements in the text prompt, but ignored some key parts or details. image depicted few elements in the text prompt. and ignored many key parts or details.
1: image failed to convey the full scope in the text prompt.
Provide your score and explanation (within 20 words) in the following format: ### SCORE: score
#* EXPLANATION: explanation
UniDet- based Spatial Relationship Evaluation. For 3DSceneBench evaluation, we decompose complex spatial relationship (c.g.. "front left") into its constituent one-dimensional components: a horizontal relationship ("left" or "right") and depth relationship ("front" or "back").
As a prerequisite, we utilize the UniDet [54] model to detect relevant objects the generated image and obtain their bounding boxes and positional information.
Following detection, we evaluate the presence and accuracy of each individual decomposed relation- ship based on metric similar to the UniDet-based approach described in 121-CompBench [17].
For horizontal relationships ("left" or "right"), we compare the bounding box center coordinates of the involved objects and get a horizontal score. For depth relationships ("front" or "back"), we leverage depth information, typically obtained via depth estimation alongside detection and get a depth score.
The final score for a complex relationship is computed as the average of the evaluation scores of its constituent oinedinnlli relationships.
H 3D Consistency Metric
The 3D layout consistency experiment evaluates depth consistency under controlled 3D layout variations. A synthetic scene is configured with two objects selected from the 3DSceneBench, initially placed centrally. Layout modification involves shifting one object along the camera axis by distance d, defined as multiple of predefined unit distance and sampled across 12 discrete values to cover a range of changes.
Following image generation for each modified layout, monocular depth estimation model [46] is applied to the generated image to obtain depth information. From the resulting depth map, the depth difference d2 between the centers of the two objects is measured. The 3D consistency score is then computed by comparing this measured depth difference dg to the original 3D shift distance d1, using the following formula:
3D Consistency = Id d2l
[d] + [dz
This score quantifies how well the depth relationship perceived in the generated image aligns with the intended depth change in the 3D layout, with higher values indicating better 3D consistency.
I Further Visualization
Figure 6 displays images generated by CoT-DDif on 3DSceneBench. We randomly selected 8 prompts from Bench and generated images using 5 different random seeds. Overall, the generated images show strong alignment with the input prompts while maintaining naturalness and high quality.
18

---

J More Visualization Results of challenging spatial scenarios
Figure shows more visualization examples of challenging spatial scenarios. For each scenario, display the 3D layout planned by CoT-Diff alongside five results generated using different random seeds, demonstrating the precise spatial control capabilities of our method.
K
Limitations
Although CoT-Diff demonstrates strong spatial control in complex scenes, it has two primary limita- tions. First, the reliance on a MLLM for per-step reasoning introduces non-negligible inference cost. Second, layout planning is currently fully dependent the MILM, without any spatial reasoning capacity integrated into the generative model itself. Future work could explore endowing the generator with internal reasoning ability to enable more efficient and unified control.
L Boarder impacts
CoT-Diff enhances spatial alignment and composition fidelity in text-to-image generation, benefiting creative and educational upplications. However, risks remain, including misuse for fake content and bias inherited from MLLMs. We encourage responsible deployment, bias auditing, and transparency future development.
19

---

Prompt: A mouse in front frog on the beach.
Prompt: A plate bahind of bowl the library.
Prompt: backpack ion
Prompt: right apartment.
Prompt: left chicken
Prampt: phone back right cup
Prompt: candle back right a computer, the back the desert.
Prompt: elegant holding fresh Howers brightens the studio, while
wailing right.
Figure 6: CoT-Diff visualizations on 3DSceneBench: 3D layout depth maps (left) and corresponding generated images (right), from random prompts with random seeds each.
20

---

Prompt: A mouse is chasing cat sunlit forest path.
Prompt: Six ceramic in different colors are arranged in circle wooden desk, bathed in the worm glow of afteroonn sunlight.
Prompt: On smooth table, an apple sits the backleft of cylindrical glass. Glass obscures right half of apple.
Prompt: Three apples and two pears on table, altematinn one in front and one behind each other
Prompt: On beautiful white tabletop, clear glass vase Riled with bright sunflowers pasitioned to the back right of classic white teapot. their right, large, slightly antique simple against softly lit,
Prompt: cosy night scene snowy forest. Warm yellow lights glow cobin sitting center. front-laft main entrance, a child busy building snowman under the falling snow. Near lorge snow covered pine tree to the backright of the cabin, deer stands watching quietly. Smake gently rises from the chimney, and the the northern lights shimmer above the treetops.
Figure 7: More visualization results of challenging spatial scenarios.
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction align with the paper's contributions and results, clearly stating the contributions, assumptions, and the scope of the work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Appendix K.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification: We provide the full set of theoretical results, definitions, and formulas in Sec- tion 3, including all necessary assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 4.1 provides all the information needed to fully reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The complete code, datasets, and experimental setup required to reproduce the main results will be included in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips...
onPoonooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nipss cc/public//peeccccccccccccc /gwuides//////eeeeeeo /CoodSuumdiiiic ss ssionsolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4.1 and Appendix B provides complete details on the training and test settings and other necessary information.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Our experimental results include statistical tests for mean value of 5 random inferences.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 4.1 provide the necessary details on the compute resources required for reproducing the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https : /ttp///eeerppp.eeeee cc/publicc/thiessuuueeee
Answer: [Yes]
Justification: Our research fully conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the positive societal impacts of our work in Appendix L.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We have not released any data models with high misuse risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators and original owners of all assets (e.g., code, data, models) used in this paper have been properly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. con/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All assets introduced in the paper well documented, and the documen- tation is provided alongside the assets in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Our experiments do not involve crowdsourcing or research with human sub-
jects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our experiments do not involve crowdsourcing research with human sub- jects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We use MLLM for Chain-of-Thouggh (CoT) generation in our methodology. A detailed description of this can be found in Section 3.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: hotea://nneeee //nurrurps.... for what should or should not be described.
28


Reviews:
[{"review_id": "b2sYvFUKBy", "reviewer": "Reviewer_XeQc", "summary": "Current text-to-image models often fail to accurately depict complex scenes with specific spatial layouts. The CoT-Diff framework addresses this by integrating a Multimodal Large Language Model (MLLM) with a diffusion model to enable step-by-step, \"Chain-of-Thought\" style reasoning during image creation. At each step of the generation process, the MLLM assesses the intermediate image, dynamically refines a 3D scene plan, and guides the diffusion model. This updated 3D layout is translated into semantic and depth map conditions, which are injected into the model via a condition-aware attention mechanism for precise control. To test this method, the researchers developed a new dataset, 3DSceneBench, which features prompts with complex 3D spatial relationships. Experiments demonstrate that CoT-Diff significantly improves spatial accuracy and compositional fidelity, outperforming state-of-the-art baselines by up to 34.7% in complex scenes.", "questions": "1. While Table 6 indicates a higher success rate for a \"reasonable inference time,\" could you provide a more direct, quantitative comparison of the wall-clock time and computational cost (e.g., in GPU hours or API costs) per image for CoT-Diff versus the baselines like FLUX and EliGen? Furthermore, have you explored the trade-off between performance and efficiency by reducing the frequency of MLLM feedback (e.g., invoking the MLLM every N steps instead of every step)? Is continuous feedback essential, or can similar results be achieved with a more sparse feedback schedule?\n2. How does CoT-Diff perform when paired with smaller, open-source MLLMs (e.g., Qwen family)? Does the performance degrade gracefully, or does the entire framework collapse without a state-of-the-art reasoning engine? \n3. Could you provide a qualitative analysis of the system's failure modes? For instance, what happens if the MLLM makes an incorrect assessment of an intermediate image or misinterprets a spatial relationship early in the process? Does the system typically recover, or does the error propagate and lead to a completely failed generation?", "limitations": "Please refer to the aforementioned weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper's primary strength is its core idea of \"CoT-lized Diffusion,\" a well-engineered technical approach that tightly integrates MLLM-driven reasoning within the diffusion sampling loop. Unlike prior works that use an LLM for a one-time layout plan before generation begins, CoT-Diff creates a dynamic feedback system. The MLLM inspects intermediate images at each denoising step and refines the 3D layout on the fly. This represents a significant conceptual shift from static, decoupled pipelines to an entangled, step-by-step generation paradigm, which is a novel and valuable contribution to the field.\n2. The paper's claims are supported by an exceptionally thorough evaluation. The experimental comparison against a strong suite of baselines, including both standard diffusion models and other layout-guided methods, shows state-of-the-art performance, with a particularly impressive 34.7% improvement in complex scene accuracy. \n\nWeaknesses:\n1. The framework's main drawback is its significant computational overhead. The core mechanism requires invoking a powerful MLLM at each denoising step to evaluate the image and refine the layout. While the paper claims \"reasonable inference time\", this iterative reasoning process is inherently much slower and more expensive than methods that perform layout planning only once. The authors acknowledge this limitation in its Appendix, but its severity makes the method impractical for many real-world applications and raises serious questions about scalability.\n2. The system's \"reasoning\" capability is entirely outsourced to a large, proprietary MLLM. This creates several issues. First, it may harm the reproducibility because the performance is dependent on a black-box API that can change over time. Second, it limits the research contribution by not integrating the reasoning capacity directly into the generative model itself, a point the authors concede is future work. The success of CoT-Diff is therefore heavily contingent on the quality of an external, pre-existing model, rather than being a fully self-contained advancement.\n3. The framework operates as a multi-step cascade, where an error at any stage can potentially corrupt the entire output. If the MLLM initially misinterprets the spatial relationships in the prompt or incorrectly evaluates an intermediate image, the subsequent layout refinements may be misguided. While the results are impressive, the paper does not sufficiently analyze the system's failure modes or its robustness to such errors. A single flawed \"thought\" in the chain could lead the generation process astray, and it is unclear if the system can reliably recover.", "was_revised": true, "final_justification": "The rebuttal responses almost addressed my concerns, so I remain my positive score.", "rebuttal": "We sincerely thank you for your positive and encouraging feedback. We are glad that you recognize the novelty of our CoT-lized Diffusion framework and its contribution to integrating MLLM-driven reasoning into the diffusion process. Based on your suggestions, we have incorporated several additional experiments into the paper to further showcase the effectiveness of our approach.\n\n## Weakness\n\n> **Weakness 1**: The framework's main drawback is its significant computational overhead. The core mechanism requires invoking a powerful MLLM at each denoising step to evaluate the image and refine the layout.\n\n**Answer:**\nWe thank the reviewer for their concerns regarding computational overhead. CoT-Diff's MLLM is not invoked at every denoising step; its primary cost is concentrated in the initial \"thinking\" phase for layout refinement. \n\nInspired by this, we designed an adaptive early-stopping strategy: the MLLM ceases evaluation once the layout is deemed correct (max 5 steps), averaging just 2.9 steps. This effectively balances performance and efficiency, as shown in the second time cost table.\n\n| MLLM step | Planning Time | Feedback Time | Generation Time | Whole Time |\n|-----------|---------------|---------------|-----------------|------------|\n| base(0)   | 23.4          | -             | 19.0            | 43.8       |\n| 1         | 23.4          | 19.3          | 19.5            | 62.2       |\n| 2         | 23.4          | 38.9          | 20.3            | 82.6       |\n| 3         | 23.4          | 58.8          | 21.4            | 102.5      |\n| 4         | 23.4          | 79.1          | 22.5            | 125.0      |\n| 5         | 23.4          | 99.2          | 23.3            | 145.9      |\n| ada (2.9) | 23.4          | 56.8          | 21.2            | 101.4      |\n\nWe argue this front-loaded computational cost is a reasonable trade-off for the significant improvements in generation quality and spatial accuracy. CoT-Diff's value in layout-critical applications like game asset creation, virtual staging, and film pre-visualization justifies this expense.\n\n\n> **Weakness 2**: The system's \"reasoning\" capability is entirely outsourced to a large, proprietary MLLM.\n\n**Answer:**\nWe acknowledge CoT-Diff's reliance on external MLLMs, potentially impacting reproducibility. However, our core contribution lies in designing a novel framework that effectively bridges MLLM's high-level cognitive reasoning with diffusion models' pixel synthesis. Our key innovation demonstrates how to activate and dynamically inject powerful spatial and logical reasoning from external MLLMs into the diffusion process in a step-by-step, corrective manner, representing a significant advance in controllable generation. Furthermore, our framework is model-agnostic, and while we showcased maximum potential with a proprietary model, CoT-Diff can readily adapt to powerful open-source MLLMs (see Q2 response), solidifying its reproducibility.\n\n\n> **Weakness 3**: Lacking a sufficiently analysis for the system's failure modes or its robustness to such errors.\n\n**Answer:**\nWhile CoT-Diff operates as a multi-step cascade with theoretical error propagation risks, its core iterative feedback mechanism is inherently designed for self-correction. \n\nEven if an error occurs at step t, the MLLM re-evaluates and corrects it at step t+1. A prime example is Figure 2 in our main paper, where the MLLM's initial flawed layout (Image 1) is detected and rectified in the subsequent loop, leading to a corrected generation (Image 2) and a successful final output. The system is designed to recover from its own intermediate mistakes.\n\nWe provide a detailed qualitative analysis and case study in our Q3 response explicitly demonstrating the system's recovery process, and experimental results empirically validate that increasing MLLM evaluation and correction steps consistently improves performance. This indicates CoT-Diff is not a brittle cascade but a robust mechanism leveraging its iterative nature for error correction and progressive refinement.\n\n\n## Question\n\n> **Question 1**:\nCould you quantify CoT-Diff's inference time and computational cost against baselines (e.g., FLUX, EliGen)? Have you explored the trade-off between performance and efficiency by reducing MLLM feedback frequency, and is continuous feedback essential?\n\n**Answer:**\nWe will include a detailed quantitative comparison of inference time and cost against baselines in the revised manuscript. The results are presented in the table below. Our results show that while CoT-Diff introduces overhead, our adaptive strategy keeps it manageable. The baseline methods FLUX, RPG, and Eligen use 50 inference steps, and our CoT-Diff use 20 steps. The search number of Image-CoT is 20.\n\n| Method    | Planning Time | Feedback Time | Generation Time | Whole Time |\n|-----------|---------------|---------------|-----------------|------------|\n| FLux      | -             | -             | 28.1            | 28.1       |\n| RPG       | 19.8          | -             | 8.5             | 28.3       |\n| Eligen    | 5.2           | -             | 55.7            | 60.9       |\n| Image-CoT | -             | -             | 114.4           | 114.4      |\n| CoT-Diff  | 23.4          | 56.8          | 21.2            | 101.4      |\n\nWe believe that continuous feedback, especially early on, is crucial for timely and accurate layout correction. This is because the initial denoising steps establish the foundational structure of the image, where any modifications have a far more significant and decisive impact on the final output.\nA sparse feedback schedule (e.g., every N steps) risks missing critical windows for correction, potentially leading to errors that cannot be fixed in later stages. Therefore, our adaptive approach, which focuses intense guidance at the beginning and stops when the layout is stable, strikes a vital balance between ensuring high-quality results and managing computational cost.\n\n> **Question 2**:\nHow does CoT-Diff perform when paired with smaller, open-source MLLMs (e.g., Qwen family)?\n\n**Answer:**\nWe thank the reviewer for this important question regarding the framework's robustness with different MLLMs. We compared CoT-Diff's performance when guided by proprietary Gemini-2.5-Pro versus open-source Qwen2.5-VL-72B-Instruct, with results summarized below:\n\n| Model  | Front | Behind | Front Left | Front Right | Back Left | Back Right | Multi | Complex |\n|--------|-------|--------|------------|-------------|-----------|------------|-------|---------|\n| gemini | 54.9  | 55.3   | 66.4       | 69.0        | 64.1      | 64.5       | 48.7  | 50.8    |\n| qwen   | 48.3  | 49.1   | 59.2       | 61.4        | 58.3      | 58.2       | 42.2  | 49.0    |\n\nThe results clearly indicate that the framework's performance degrades gracefully when switched to the open-source MLLM; critically, the system does not collapse and still outperforms other baseline methods. This demonstrates our framework's inherent value and robustness, supporting our W2 response that our approach is not fundamentally tied to a single proprietary MLLM and ensures reproducibility.\n\n> **Question 3**:\nCould you provide a qualitative analysis of the system's failure modes?\n\n**Answer:**\n\nWe thank the reviewer for their question on system failure modes. CoT-Diff's core strength lies in its inherent self-correction capability.\n\n1) **Qualitative Analysis:** Our framework is inherently designed to recover from initial errors. Figure 2 in the main text serves as a prime case study of this self-correction mechanism. \n\n\tIn that example:\n\n\t(1) The MLLM initially generates a flawed 3D layout, misinterpreting the spatial relationship.\n\n\t(2) This leads to an incorrect intermediate generation (Image 1), where the relationship between woman and desk is wrong.\n\n\t(3) Crucially, in the next feedback loop, the MLLM assesses Image 1, detects the discrepancy between the generated image and the text prompt, and rectifies its own initial layout plan.\n\n\t(4) This corrected layout then guides the diffusion model to produce a spatially accurate intermediate image (Image 2), leading to a successful final output (Image N).\n\nWhile the system is robust, it can occasionally fail if the MLLM repeatedly makes incorrect assessments or if the diffusion model fails to follow the guidance. We will add a detailed qualitative analysis of such failure cases to the revised manuscript.\n\n2) **Quantitative Validation:** \nTo validate this, we evaluated performance at varying optimization steps. As the table below shows, performance consistently improves with more steps, saturating around 3 steps.\n\n| Step         | Front | Behind | Front Left | Front Right | Back Left | Back Right | Multi | Complex |\n|--------------|-------|--------|------------|-------------|-----------|------------|-------|---------|\n| base(0)      | 51.3  | 51.2   | 62.2       | 61.4        | 59.9      | 56.7       | 41.7  | 48.5    |\n| 1            | 53.3  | 51.9   | 63.4       | 66.0        | 61.2      | 59.8       | 45.8  | 48.4    |\n| 2            | 57.6  | 53.6   | 66.6       | 67.1        | 62.8      | 63.3       | 47.9  | 50.4    |\n| 3            | 55.2  | 54.5   | 67.5       | 69.5        | 62.2      | 64.9       | 47.6  | 49.7    |\n| 4            | 56.0  | 54.5   | 68.3       | 69.9        | 61.9      | 64.9       | 48.8  | 51.4    |\n| 5            | 56.2  | 56.4   | 67.9       | 71.2        | 64.4      | 66.1       | 51.0  | 51.8    |\n| ada (2.9)    | 54.9  | 55.3   | 66.4       | 69.0        | 64.1      | 64.5       | 48.7  | 50.8    |\n\nThis trend indicates that the iterative process is effective at correcting errors and that the system reliably recovers and improves with more \"thinking\" time."}, {"review_id": "VCrG3Fkxvg", "reviewer": "Reviewer_UhfL", "summary": "This paper presents CoT-Diff, a 3D-aware text-to-image generation pipeline. It uses a multimodal LLM to plan scene layouts and provide feedback at each diffusion step. Local prompt details and depth cues are injected via a condition-aware attention mechanism. The method is evaluated on three benchmarks including the newly introduced 3DSceneBench, and DVMP [30] and T2I-CompBench [14, 15].", "questions": "- How does performance change if you omit the custom attention mask?\n- In Section 4.5 you mention comparing runtime efficiency, but the table omits these numbers. How does CoT-Diff’s inference time compare to other methods?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- Well-written and easy to follow.\n- Visual outputs are noticeably improved over prior methods.\n- 3DSceneBench fills an important gap for evaluating 3D compositionality.\n\n**Weaknesses**\n- The authors do not compare against unified understanding-and-generation models such as Janus-Pro [*1] and ViLA-U [*2], which often performs well at text-image alignment. Including these baselines would provide useful insight.\n- As the authors discussed, CoT-Diff will cause high computational overhead: using GPT-4o or Gemini 2.5 Pro at each step is time-consuming.\n- The pipeline relies on intermediate images for feedback, but no visualizations of these images. The early steps are usually very blurry, leaving it unclear how useful they are for the MLLM.\n- The proposed condition-aware attention mechanism closely resembles the one in EliGen, yet this similarity is not discussed.\n\n[*1] Chen, Xiaokang, et al. “Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling.” arXiv:2501.17811 (2025).\n\n[*2] Wu, Yecheng, et al. “ViLA-U: A Unified Foundation Model Integrating Visual Understanding and Generation.” ICLR 2025.", "was_revised": true, "final_justification": "After reading the rebuttal and the reviews from other reviewers, and considering the discussion phase, most of my concerns have been addressed. The paper proposes a 3D-aware text-to-image generation pipeline that achieves state-of-the-art performance with relatively fast speed compared to other CoT works. Therefore, I recommend acceptance.", "rebuttal": "We sincerely thank you for the valuable feedback. Our main contribution is a CoT-lized framework that activates the powerful reasoning abilities of MLLMs to guide diffusion, effectively addressing the limitations of generative models in spatial reasoning and the rigidity of one-time layout planning.\n\n\n## Weakness\n\n> **Weakness 1**: Lack of comparison with unified understanding-and-generation models.\n\n**Answer:**\nWe have added comparisons with Janus-Pro[1], VILA-U[2], Image-CoT[3], and T2I-R1[4]. The results demonstrate CoT-Diff’s superior performance in spatial reasoning and text-image alignment, suggesting that current unified generation-understanding models still lack explicit reasoning capabilities compared to our CoT-lized framework.\n\n| Method      | Front | Behind | Front Left | Front Right | Back Left | Back Right | multi | complex |\n| :---------- | :---- | :----- | :--------- | :---------- | :-------- | :--------- | :---- | :------ |\n| janus-pro   | 35.2  | 36.5   | 29.7       | 24.9        | 24.9      | 22.9       | 19.6  | 39.0    |\n| vila-u      | 37.2  | 38.3   | 28.8       | 31.5        | 29.2      | 32.4       | 21.3  | 29.5    |\n| Image-CoT     | 48.2  | 49.7   | 56.1       | 61.2        | 56.7      | 57.6       | 39.7  | 46.6    |\n| t2i-r1      | 47.0  | 46.8   | 47.2       | 45.3        | 50.4      | 51.1       | 35.5  | 41.5    |\n| **CoT-Diff**    | **54.9**  | **55.2**   | **66.4**       | **69.0**        | **64.2**      | **64.5**       | **48.7**  | **50.8**    |\n\n> **Weakness 2**: As the authors discussed, CoT-Diff will cause high computational overhead: using GPT-4o or Gemini 2.5 Pro at each step is time-consuming.\n\nWe thank the reviewer for this crucial point on computational overhead. We acknowledge that using a powerful MLLM introduces a time cost. To provide full transparency, we have conducted a detailed quantitative analysis.\n1. Comparison with Existing Methods\nFirst, we compare CoT-Diff with both layout-based and other CoT-based methods. The baseline methods FLUX, RPG, and Eligen use 50 inference steps, and our CoT-Diff use 20 steps. The search number of Image-CoT is 20.\n\n| Method    | Planning Time | Feedback Time | Generation Time | Whole Time |\n|-----------|---------------|---------------|-----------------|------------|\n| FLux      | -             | -             | 28.1            | 28.1       |\n| RPG       | 19.8          | -             | 8.5             | 28.3       |\n| Eligen    | 5.2           | -             | 55.7            | 60.9       |\n| Image-CoT | -             | -             | 114.4           | 114.4      |\n| CoT-Diff  | 23.4          | 56.8          | 21.2            | 101.4      |\n\nThe data shows that while CoT-Diff's Feedback Time adds to the overhead compared to simpler layout-based methods, its total inference time is comparable to other CoT-based frameworks like Image-CoT. This suggests the observed overhead is characteristic of the complexity required for step-by-step generation.\n\n2. Adaptive Strategy for Efficiency\nMore importantly, we have designed an adaptive early-stopping strategy to explicitly manage this cost without sacrificing quality. The MLLM guidance is not fixed but applied dynamically until the layout is deemed correct (up to a maximum of 5 steps).\nThe table below shows how runtime scales with the number of MLLM feedback steps:\n\n| MLLM step | Planning Time | Feedback Time | Generation Time | Whole Time |\n|-----------|---------------|---------------|-----------------|------------|\n| base      | 23.4          | -             | 19.0            | 43.8       |\n| 1         | 23.4          | 19.3          | 19.5            | 62.2       |\n| 2         | 23.4          | 38.9          | 20.3            | 82.6       |\n| 3         | 23.4          | 58.8          | 21.4            | 102.5      |\n| 4         | 23.4          | 79.1          | 22.5            | 125.0      |\n| 5         | 23.4          | 99.2          | 23.3            | 145.9      |\n| ada (2.9) | 23.4          | 56.8          | 21.2            | 101.4      |\n\nOur experiments show that this adaptive strategy achieves considerable performance in just 2.9 steps on average, striking an effective balance between generation quality and computational cost.\n\n> **Weakness 3**: The pipeline relies on intermediate images for feedback, but no visualizations of these images. The early steps are usually very blurry, leaving it unclear how useful they are for the MLLM.\n\n**Answer:**\nWe use FLUX-schnell as the diffusion backbone, which is a distilled version of FLUX-dev. \nThanks to its improved denoising capability, even the early-stage intermediate images are relatively clean and informative. \nAs shown in Figure 2 (Image 1 and Image 2), the visual quality at each step is sufficient for the MLLM to perform reliable evaluations and provide meaningful feedback.\n\n> **Weakness 4**: The proposed condition-aware attention mechanism closely resembles the one in EliGen, yet this similarity is not discussed.\n\n\n**Answer:**\nWe thank the reviewer for pointing out the similarity between CoT-Diff's condition-aware attention mechanism and EliGen. We acknowledge that both methods indeed share common ground in their core mechanisms: both leverage global and local textual prompts to guide image content generation, and both employ similar isolation strategies by masking regions corresponding to different prompts (e.g., P1 and P2) during attention calculation to prevent semantic interference.\n\nHowever, CoT-Diff extends upon this foundation in two key aspects, endowing it with more powerful conditional control capabilities:\n\n1) Introduction of Multimodal Conditions: Beyond textual prompts (such as p, p1, p2), CoT-Diff additionally incorporates a Depth Condition as a novel geometric input. This allows the model to simultaneously utilize both the semantic information from text and the precise spatial-geometric information from depth maps, enabling richer and more refined conditional control.\n\n2) Extended Attention Mechanism for Heterogeneous Condition Fusion: To effectively fuse these two inherently heterogeneous conditional information types—text and depth—and to prevent potential conflicts and interference between them, we specifically extended the attention mechanism. By applying custom-designed attention masks, we ensure that information flow between different modal conditions (i.e., textual conditions and depth conditions) remains independent, thereby guaranteeing the precision and autonomy of each guiding information type.\n\n\n## Question\n\n> **Question 1**:\nHow does performance change if the custom attention mask is omitted?\n\n**Answer:**\nWe have demonstrated the performance of the model without optimization. It can be observed that omitting the custom attention mask leads to a significant performance degradation in dimensions other than \"front\" and \"behind\". This is because the absence of the mask causes conditions from different objects to act on the same region, leading to issues such as attribute leakage and confusion.\n\n| Dataset        | Front | Behind | Front Left | Front Right | Back Left | Back Right | Multi | Complex |\n|----------------|-------|--------|------------|-------------|-----------|------------|-------|---------|\n| w/o attn       | 48.8  | 50.9   | 36.9       | 41.3        | 39.9      | 36.9       | 32.0  | 46.3    |\n| + attn         | 51.3  | 51.2   | 62.2       | 61.4        | 59.9      | 56.7       | 41.7  | 48.5    |\n| + optim (Full) | 54.9  | 55.3   | 66.4       | 69.0        | 64.1      | 64.5       | 48.7  | 50.8    |\n\n\n\n> **Question 2**:\nSection 4.5 mentions runtime efficiency comparison, but the table lacks these numbers; how does CoT-Diff’s inference time compare to other methods?\n\n**Answer:**\nAs stated in W2, we adopt a quantitative comparison. The result shows CoT-Diff’s runtime is acceptable when compared against layout-based methods and is comparable to other CoT-based frameworks.\n\n\n[1] Chen, Xiaokang, et al. \"Janus-pro: Unified multimodal understanding and generation with data and model scaling.\" arXiv preprint arXiv:2501.17811 (2025).\n\n[2] Wu, Yecheng, et al. \"Vila-u: a unified foundation model integrating visual understanding and generation.\" arXiv preprint arXiv:2409.04429 (2024).\n\n[3] Guo, Ziyu, et al. \"Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step.\" arXiv preprint arXiv:2501.13926 (2025). \n\n[4] Jiang, Dongzhi, et al. \"T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot.\" arXiv preprint arXiv:2505.00703 (2025)."}, {"review_id": "Yw8ErSTztJ", "reviewer": "Reviewer_QBNx", "summary": "The key contribution of this paper is the application of Chain-of-Thought (CoT) abilities from large multimodal models to Text-to-Image (T2I) tasks, which improves spatial layout and 3D perception. To assess these improvements, the authors have designed the 3DSceneBench benchmark for evaluating structure-based reasoning. The paper presents a strong motivation for this research.", "questions": "1.\tIs the design of some prompts problematic? For example, in Figure 4 with the prompt, \"a car behind a bench on a snowy landscape,\" the spatial structure is correct in the results from both CoT-Diff and EliGen. The perceived correctness in both is due to different viewing angles, as the prompt only specifies the front-behind relationship without left-right positioning. How should such cases be judged during the actual labeling process, and could this ambiguity affect the other evaluation metrics?\n2.\tThe authors do not state the resolution of the generated images. Could the image size have an impact on the results?", "limitations": "1.\tThe evaluation section of this paper is somewhat insufficient.\na)\t In the \"Main Results of CoT-Diff\" section, the authors state that traditional diffusion models perform poorly in spatial tasks due to a lack of layout guidance. However, the LC method, which incorporates layout guidance, shows relatively the worst performance in the metrics. The authors do not provide a more detailed analysis of this counterintuitive result.\nb)\t In the results analysis, the authors generally describe performance improvements but do not offer a detailed breakdown of individual results. For instance, in the \"Ablation Study of CoT-Diff,\" the performance on the \"Front\" category decreases after adding \"+Semantic Layout.\" The authors should briefly explain the potential reasons for this decline.\nc)\tAn evaluation of the number of inference steps is missing. The authors have not analyzed how many steps are typically required for the model to generate images with well-composed spatial structures. As a major limitation cited in the paper is the high inference cost, such an evaluation is crucial.\nd)\tThere have been other works this year that have also applied CoT to image generation (as listed below). The authors should cite or compare their work against these papers in the final version, clarifying the key differences and advantages of their proposed method. \n[1] Guo, Ziyu, et al. \"Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step.\" arXiv preprint arXiv:2501.13926 (2025).\n[2] Jiang, Dongzhi, et al. \"T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot.\" arXiv preprint arXiv:2505.00703 (2025).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros: \n1.\tThis work is well-motivated and novel. The understanding of 3D structures in images is crucial for tasks such as image generation and content comprehension. Furthermore, the paper introduces an innovative, CoT-based approach to enhance the model's understanding of spatial structures in T2I tasks.\n2.\tThe contributions are substantial. In addition to proposing the Cot-Diff model, the authors have also designed a sound evaluation framework, 3DSceneBench.\n3.\tThe paper is concisely written and easy to follow, clearly articulating the primary work presented.\n\nWeaknesses:\n\n1. The proposed CoT-Diff framework requires invoking a large MLLM (GPT-4o or Gemini) at each denoising step, resulting in substantial inference cost. Although the authors mention this in the appendix and describe inference time as “reasonable,” the iterative nature of the reasoning process raises concerns regarding scalability and real-world applicability.\n2. While the method is framed as a CoT approach, its reasoning capabilities are primarily restricted to spatial layout understanding. Broader reasoning capabilities (temporal, causal, or functional reasoning) are not explored.\n3. Errors in early stages (misinterpretation of spatial prompts or poor initial layout) could propagate and negatively affect the final output. The paper does not sufficiently analyze failure modes or the model’s ability to recover from such errors.", "was_revised": true, "final_justification": "After considering the rebuttal and discussions, I believe the authors have made a strong effort to address the key concerns, and my overall impression of the paper has improved.", "rebuttal": "We sincerely thank you for your recognition of our work. Based on your suggestions, we have added further analysis, clarified key experimental details, and included comparisons with recent CoT-based methods to strengthen the paper.\n\n\n\n## Question\n\n> **Question 1**:\nIs the design of some prompts problematic? For example, in Figure 4 with the prompt, \"a car behind a bench on a snowy landscape,\" the spatial structure is correct in the results from both CoT-Diff and EliGen. The perceived correctness in both is due to different viewing angles, as the prompt only specifies the front-behind relationship without left-right positioning. How should such cases be judged during the actual labeling process, and could this ambiguity affect the other evaluation metrics?\n\n**Answer:**\nThank you for raising this question. In our evaluation, spatial relations such as \"behind\" are defined with respect to the camera (observer) viewpoint, which provides a consistent and objective reference frame. Without this convention, viewpoint rotation could arbitrarily flip spatial relations (e.g., “behind” becoming “front” or “right”), making evaluation unreliable. This camera-centric assumption is also adopted by established benchmarks such as T2I-CompBench[1].\n\n\n\n> **Question 2**:\nCould the image size have an impact on the results?\n\n**Answer:**\nOur current implementation generates images at a resolution of 512×512. Since our method relies on accurate 3D layout modeling, low-resolution settings (e.g., 128×128) may introduce errors in rendered depth, leading to hallucinations. However, we observe that at 512×512 or higher resolutions, the depth maps are sufficiently accurate to guide high-quality image generation.\n\n\n## Limitation\n\n> **Limitation 1 a)**: \nThe LC method incorporates layout guidance, yet it performs worse than traditional diffusion models in the reported metrics. A more detailed analysis is needed to explain this counterintuitive result.\n\n**Answer:**\nThe seemingly counterintuitive performance of LC arises from several limitations in its design:\n\n1) Backbone Choice: LC uses Stable Diffusion 1.5 as its backbone, which has relatively limited image quality and spatial understanding capabilities compared to more advanced models (e.g., FLUX). This constrains the upper bound of its spatial reasoning performance.\n\n2) Conditioning Mechanism: Although LC accepts 3D layout input, it does so through a sparse-depth LoRA module trained on top of a ControlNet pretrained for dense depth. The sparse layout signal is thus weakly injected into the model through an indirect path (LoRA → ControlNet → SD), which reduces its effectiveness in enforcing correct spatial structure. Moreover, this training approach is also known to be challenging to converge and can even degrade the backbone's inherent generative capabilities.\n\n3) Lack of Semantic Disentanglement: LC lacks a semantic module like the Semantic LoRA in CoT-Diff, making it more prone to object confusion when generating scenes with multiple entities. For example, in Fig. 4 (column 2), both the car and the bench are generated as benches, failing to represent the intended spatial relationship (“car behind a bench”).\n\nTogether, these factors explain why LC, despite incorporating layout input, underperforms in spatially grounded generation tasks.\n\n> **Limitation 1 b)**: \nThe results analysis lacks fine-grained interpretation. For example, in the ablation study, the \"Front\" category shows a performance drop after adding the \"+Semantic Layout\" module. The authors should briefly explain this decline.\n\n**Answer:**\nThank you for your meticulous review of our ablation study. You have correctly identified the slight performance dip in the \"Front\" category after adding \"+Semantic Layout,\" which is an excellent question. We believe this phenomenon is not an anomaly and can be attributed to the following factors:\n\n1) Conflict between Semantic and Geometric Cues: The Semantic Layout module injects entity-level semantic information into specific image regions (as described in lines 143–144). However, in the absence of corresponding depth maps to enforce geometric structure, this strong semantic bias may conflict with the model’s pre-existing handling of simple spatial relations such as “front.” This can temporarily interfere with the model’s generation patterns and lead to slight performance drops.\n\n2) Importance of Depth Information: This decline further confirms the necessity and effectiveness of adding the Depth Map module. As shown in Table 4, once depth information is introduced on top of the semantic layout, the model not only resolves occlusions and object ordering issues but also significantly improves performance in the “Front” category (from 44.5 to 51.3). This highlights the complementary nature of semantic and geometric guidance: while semantics guide content, depth provides spatial anchoring in 3D space.\n\n3) The characteristics of the dataset: Our training data is constructed based on annotations from EliGen, and a similar pattern was observed in EliGen's results (as seen in Table 2 of our main text), where front/behind performance also dipped when only semantic layouts were introduced. Therefore, the decline we observe may be partially inherited from the limitations of the source dataset.\n\n> **Limitation 1 c)**: \nThe paper lacks an analysis of how many inference steps are needed to produce spatially coherent images, which is important given that high inference cost is a stated limitation.\n\n**Answer:**\nWe evaluates the performance with a fixed number of MLLM optimization steps at varying total steps. It can be observed that as the number of steps increases, the system's performance progressively improves, tending to converge after 3 steps.\n\nAccordingly, we designed an adaptive early-stopping strategy: once the model determines the layout is correct, it ceases further evaluation, with a maximum of 5 evaluation steps. This strategy balances performance and time overhead, achieving considerable performance with an average of 2.9 steps.\n\n| Step         | Front | Behind | Front Left | Front Right | Back Left | Back Right | Multi | Complex |\n|--------------|-------|--------|------------|-------------|-----------|------------|-------|---------|\n| base(0)      | 51.3  | 51.2   | 62.2       | 61.4        | 59.9      | 56.7       | 41.7  | 48.5    |\n| 1            | 53.3  | 51.9   | 63.4       | 66.0        | 61.2      | 59.8       | 45.8  | 48.4    |\n| 2            | 57.6  | 53.6   | 66.6       | 67.1        | 62.8      | 63.3       | 47.9  | 50.4    |\n| 3            | 55.2  | 54.5   | 67.5       | 69.5        | 62.2      | 64.9       | 47.6  | 49.7    |\n| 4            | 56.0  | 54.5   | 68.3       | 69.9        | 61.9      | 64.9       | 48.8  | 51.4    |\n| 5            | 56.2  | 56.4   | 67.9       | 71.2        | 64.4      | 66.1       | 51.0  | 51.8    |\n| ada (2.9)    | 54.9  | 55.3   | 66.4       | 69.0        | 64.1      | 64.5       | 48.7  | 50.8    |\n\n\n> **Limitation 1 d)**: \nRecent works have also explored applying CoT to image generation. The paper should cite and compare with these methods, clarifying the key differences and advantages of the proposed approach.\n\n**Answer:**\nThank you for bringing these relevant recent works to our attention. We have studied them and added experimental comparisons.\n\nKey Differences and Advantages:\nThe core difference between our CoT-Diff and these methods lies in the fundamental mechanism of the \"Chain-of-Thought.\" Other methods (e.g., Image-CoT[2], T2I-R1[3]) implement CoT at the semantic or token level, refining text prompts or internal features iteratively. In contrast, CoT-Diff introduces explicit 3D spatial reasoning, using an MLLM to generate and refine a physically interpretable 3D layout, which serves as precise guidance for the diffusion model. This structured approach excels at handling complex spatial relationships and ambiguities.\n\nExperimental Comparison:\nWe evaluated these methods on our 3DSceneBench benchmark. The results clearly show CoT-Diff's superior performance in spatial reasoning, particularly in challenging categories like \"Front/Back Left/Right,\" \"Multi Relation,\" and \"Complex.\"\n\nWe will include these works in our \"Related Work\" section and add the comparison to the \"Experiments\" section in the final version to demonstrate the novelty and effectiveness of our method.\n\n| Method      | Front | Behind | Front Left | Front Right | Back Left | Back Right | multi | complex |\n| :---------- | :---- | :----- | :--------- | :---------- | :-------- | :--------- | :---- | :------ |\n| janus-pro   | 35.2  | 36.5   | 29.7       | 24.9        | 24.9      | 22.9       | 19.6  | 39.0    |\n| vila-u      | 37.2  | 38.3   | 28.8       | 31.5        | 29.2      | 32.4       | 21.3  | 29.5    |\n| Image-CoT | 48.2  | 49.7   | 56.1       | 61.2        | 56.7      | 57.6       | 39.7  | 46.6    |\n| t2i-r1      | 47.0  | 46.8   | 47.2       | 45.3        | 50.4      | 51.1       | 35.5  | 41.5    |\n| **CoT-Diff**    | **54.9**  | **55.2**   | **66.4**       | **69.0**        | **64.2**      | **64.5**       | **48.7**  | **50.8**    |\n\n[1] Huang, Kaiyi, et al. \"T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.\" Advances in Neural Information Processing Systems 36 (2023): 78723-78747.\n\n[2] Guo, Ziyu, et al. \"Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step.\" arXiv preprint arXiv:2501.13926 (2025). \n\n[3] Jiang, Dongzhi, et al. \"T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot.\" arXiv preprint arXiv:2505.00703 (2025)."}, {"review_id": "GcqUhbw9Uj", "reviewer": "Reviewer_sj9U", "summary": "This paper proposes CoT-lized Diffusion, a pipeline that improves spatial and semantic consistency in text-to-image generation by incorporating step-by-step layout reasoning. The core idea is to use GPT-4 to decompose a long or complex prompt into sub-goals, each guiding a 3D layout. A 3D scene planner integrates these sub-layouts into a unified structure, which is then fed into a conditional diffusion model (FLUX.1-schnell) to generate the final image. The method is motivated by the need to address ambiguities in natural language descriptions, especially those involving spatial relations. Experimental results show that the layout accuracy and fidelity to prompt details is better than the baselines.", "questions": "- Overclaiming CoT: Could the authors clarify in what way their method performs reasoning beyond spatial layout decomposition? Do you claim any general CoT behavior or just layout disambiguation?\n- Backbone bias: Since FLUX.1-schnell is state-of-the-art, please clarify the choice of baseline. Have you tested the method on a weaker or standard backbone like SDXL?\n- Qualitative correctness: In Fig. 4’s last column, the key action (“pointing”) is not present. Could you discuss failure cases or limitations of action grounding?", "limitations": "Partially. The paper does not explicitly address limitations such as over-reliance on 3D layout accuracy, lack of end-to-end differentiability, or the societal risks of generation failures (e.g., hallucinated actions or biases in spatial understanding). A more complete discussion would improve transparency.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The main strengths are listed below.\n- (1) The main issue of this paper aiming at is clear, i.e., prompt ambiguity, especially for spatial layouts.\n- (2) The proposed modular architecture cleanly separates prompt decomposition, layout generation, and image synthesis by leveraging GPT-4 to structure prompts into layout subgoals is an adaptation of CoT-style reasoning to the vision domain.\n- (3) This paper uses 3D layout as intermediate grounding which offer better controllability and interpretability.\n\nHowever, several weaknesses are listed below.\n- (1) The method addresses only spatial disambiguation rather than broader forms of reasoning (e.g., temporal, causal, or functional). The \"CoT\" framing may be misleading as it only affects layout.\n- (2) This paper lacks of user study or human evaluation, despite the task involving semantics and perception. The evaluation relies solely on LLM scorer and visual examples.\n- (3) Some details are missing in formulation. For instance, vt in Eq. 4 is undefined, which is important. Moreover, the loss optimization procedure is vaguely described.\n- (4) FLUX.1-schnell is a powerful backbone. It is unclear whether the baselines use the same backbone or use others, e.g., SDXL. RPG is training-free, which can be applied to the same backbone.\n- (5) Qualitative failure in core claim: As seen in Fig. 4 (last column), the image for “pointing at a distant sailboat” fails to depict the pointing action. This suggests the pipeline may not faithfully capture verb-level intent or fine-grained semantics.", "was_revised": true, "final_justification": "This paper presents a technically solid and well-motivated framework for step-by-step spatial reasoning in text-to-image diffusion models. The experiments are thorough and show consistent improvements over strong baselines. The authors have provided detailed clarifications in the rebuttal, addressing the main concerns. Overall, the contribution is significant and timely. I recommend acceptance.", "rebuttal": "Thank you for your thoughtful and encouraging feedback. We have clarified the CoT scope, addressed technical details, and discussed limitations as suggested.\n\n\n## Weakness\n\n> **Weakness 1**: The method addresses only spatial disambiguation rather than broader forms of reasoning (e.g., temporal, causal, or functional). The \"CoT\" framing may be misleading as it only affects layout.\n\n**Answer:**\nThank you for your insightful comment. We framed our method as \"CoT-lized\" because it adopts the step-by-step reasoning paradigm from LLMs, using an iterative \"plan-evaluate-refine\" loop to guide generation. This describes the process of our method, not just its application domain. \nFurthermore, our framework can indeed handle broader inferential tasks, leveraging the MLLM's knowledge. For instance, given the prompt \"Einstein's favorite musical instrument\", our MLLM reasons that it is a violin and rewrites the prompt into a generatable form like \"a beautiful violin.\" This demonstrating its capacity for factual and conceptual reasoning. We will clarify this point in our revised manuscript. Thank you for your valuable feedback.\n\n> **Weakness 2**: This paper lacks of user study or human evaluation, despite the task involving semantics and perception. The evaluation relies solely on LLM scorer and visual examples.\n\n**Answer:**\nThank you for the valuable suggestion. In response, we have conducted a human evaluation to assess our method's semantic and perceptual performance. We asked six human evaluators to rate images generated for 200 complex prompts from 3DSceneBench. They scored our method (CoT-Diff) against four key baselines on a 1-5 scale across two criteria: 3D Layout Accuracy and Visual Quality.\n\n| Method        | 3D Layout Accuracy | Visual Quality |\n|---------------|--------------------|----------------|\n| FLUX          | 2.71               | 3.56           |\n| RPG           | 3.57               | 3.75           |\n| EliGen        | 3.65               | 3.81           |\n| Image-CoT     | 3.83               | 2.95           |\n| CoT-Diff (Ours) | 4.08               | 4.12           |\n\nAs shown in Table, CoT-Diff significantly outperforming all baselines. Notably, our method achieves the highest scores in both Layout Accuracy (4.08) and Visual Quality (4.12), demonstrating a unique ability to enhance complex spatial control without the degradation in image fidelity seen in methods like Image-CoT.\n\n> **Weakness 3**: Some details are missing in formulation. For instance, vt in Eq. 4 is undefined, which is important. Moreover, the loss optimization procedure is vaguely described.\n\n**Answer:**\nWe thank the reviewer for pointing out these omissions and apologize for the lack of clarity.\n\nTo address your first point, **v_t** in Equation (4) represents **the velocity field predicted by our FLUX model at timestep t**. Formally, $v_t = v_Θ(z_t, t, C)$, where $C$ denotes the combined conditional inputs (e.g., depth map and text embeddings) and $Θ$ represents the model's parameters.\n\nRegarding the loss optimization procedure, we train our two LoRA modules (Semantic and Depth) independently using a **flow-matching loss**. The specific loss functions are as follows:\n\n1. **For the Semantic LoRA module (L_SL):**\n$L_{SL} = E_{t,z,\\epsilon}[||v_{\\theta}(z, t, P, P_1, ..., P_k) - u_t(z|\\epsilon)||^2]. $\nHere, $v_θ$ is the velocity field predicted by the Semantic LoRA-augmented model, $P, P1, ..., Pk$ are the global and local prompt embeddings, and $u_t(z|ε)$ is the target velocity field.\n2. **For the Depth LoRA module (L_DM):**\n$L_{DM} = E_{t,z,\\epsilon}[||v_{\\Phi}(z, t, C_D, P_{emb}) - u_t(z|\\epsilon)||^2]. $\nHere, $v_Φ$ is the velocity field predicted by the Depth LoRA-augmented model, $C_D$ is the depth condition, and $P_emb$ is the prompt embedding.\n\nWe will update Section 3 with these precise formulations to improve clarity and reproducibility.\n\n> **Weakness 4**: FLUX.1-schnell is a powerful backbone. It is unclear whether the baselines use the same backbone or use others, e.g., SDXL. RPG is training-free, which can be applied to the same backbone.\n\n**Answer:**\nThank you for this crucial point on experimental fairness.\n\nYou are correct. In our initial experiments, we aimed to showcase each method at its best by using their officially recommended backbones (e.g., RPG with SDXL, EliGen with FLUX.1-dev). However, for the most direct comparison, **we highlight our results against EliGen in Tables 2 and 3 of our main text, a training-based method that also utilizes a FLUX backbone**. As shown in Tables 2 and 3 of our main text, CoT-Diff already demonstrates a significant performance advantage. This provides compelling evidence that our framework's superiority will hold when compared against **a training-free method like RPG** on the same architecture.\n\nFor the sake of complete academic rigor, we agree that an explicit experiment on the FLUX.1-schnell backbone is essential for isolating the specific advantages of our CoT-Diff framework. Unfortunately, adapting RPG from its native SDXL (U-Net architecture) to FLUX (DiT architecture) is a non-trivial engineering task due to significant architectural disparities. This adaptation is already in progress, but its complexity prevented us from finalizing the results within the brief rebuttal period.\n\nTherefore, we commit to including this crucial experiment in the camera-ready version of the paper. We will adapt RPG to the FLUX.1-schnell backbone for a direct and fair comparison. This will further solidify the superiority of our CoT-Diff framework.\n\n> **Weakness 5**: Qualitative failure in core claim: As seen in Fig. 4 (last column), the image for “pointing at a distant sailboat” fails to depict the pointing action. This suggests the pipeline may not faithfully capture verb-level intent or fine-grained semantics.\n\n**Answer:**\nWe acknowledge the limitation shown in this example. This is likely due to two factors: \n1) Coarse Control Granularity: our method emphasizes controlling the 3D layout of full objects, but the hand is only a partial component of the human body. Guiding such fine-grained regions (e.g., hand gestures) via whole-body 3D bounding boxes is inherently difficult, and precise control at this level remains a challenging open problem; \n2) Backbone Limitations: the backbone itself has limited capacity for faithfully capturing verb-level semantics—none of the compared methods successfully rendered the pointing action, indicating a general limitation in current diffusion models for action grounding.\n\n## Question\n\n> **Question 1**:\nOverclaiming CoT: Could the authors clarify in what way their method performs reasoning beyond spatial layout decomposition? Do you claim any general CoT behavior or just layout disambiguation?\n\n**Answer:**\nAs stated in W1, our main claim is the CoT-style generation paradigm, where MLLMs guide the diffusion process step by step, rather than claiming general-purpose CoT reasoning.\n\n\n\n> **Question 2**:\nBackbone bias: Have you tested the method on a weaker or standard backbone like SDXL?\n\n**Answer:**\nThank you for this question regarding the generalizability.\n\nOur initial experiments used FLUX.1-schnell to show our framework's maximum potential. We agree that testing on SDXL is crucial for demonstrating generalizability. \n\nHowever, implementing and training our entire framework on SDXL is a significantly more demanding task than adapting a training-free method like RPG (as discussed in our response to W4). This training process is already underway, but given the substantial computational resources and time required, we were unable to obtain fully converged results and complete the evaluation within the brief rebuttal period.\n\nWe note that prior work (e.g., MIGC [1]) has successfully applied semantic and 2D layout control to the SDXL backbone. This suggests that our 3D-aware approach would also be effective on SDXL. We commit to including this analysis in a future version of our paper.\n\n\n\n> **Question 3**:\nQualitative correctness: Could you discuss failure cases or limitations of action grounding?\n\n**Answer:**\nAs stated in W5, our pipeline faces challenges in fine-grained action grounding such as \"pointing,\" due to both the coarse 3D control granularity and limitations of the diffusion backbone.\n\n\n\n## Limitation\n\n> **Limitation 1**: \nThe paper does not explicitly address limitations like over-reliance on 3D layout accuracy, lack of end-to-end differentiability, or societal risks from generation failures (e.g., hallucinations or spatial understanding biases).\n\n**Answer:**\nIndeed, our method exhibits a degree of reliance on the accuracy of upstream 3D layouts. If these input 3D layout representations themselves contain errors or lack robustness, the quality and spatial precision of the final generated images may be impacted, even with CoT-Diff's effective utilization. Furthermore, the iterative reasoning process of the MLLM in the current framework is not end-to-end differentiable, which limits the potential for large-scale holistic optimization and deeper learning. More broadly, as the reviewer mentioned, all large generative models face risks of hallucination and bias, and our model is no exception. This could include generating \"hallucinated\" content inconsistent with semantics, or exhibiting biases in spatial understanding derived from training data.\n\nHowever, CoT-Diff's strength lies in its explicit 3D layouts and iterative MLLM refinement, allowing for improved spatial understanding and self-correction during generation, thus mitigating spatially related hallucinations and error propagation. Future work will focus on improving the robustness of the 3D layout generation, exploring differentiable reasoning frameworks, and actively addressing biases and societal risks in models.\n\n[1] Zhou, Dewei, et al. \"Migc: Multi-instance generation controller for text-to-image synthesis.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024."}]

Author rebuttal:

Meta-review: Reviewers agree that this is a significant step forward in the field. The rebuttal and post-rebuttal discussion clarified initial questions. Congratulations, this submission is accepted.
Decision: Accept (poster)