ToddlerDiffusion: Interactive Structured Image Generation with Cascaded Schrödinger Bridge
Authors: Eslam Mohamed BAKR
Liangbing Zhao
Vincent Tao Hu
Matthieu Cord
Patrick Perez
Mohamed Elhoseiny
Venue: ICLR 2025 Poster (2025)
Keywords: Generative AI
Diffusion models
Cascaded models
Interpretability
Image Editing
Primary area: generative models
TL;DR: Our approach decomposes the generation process into more straightforward, interpretable stages, for instance, first generating contours, then a palette, and finally, a detailed colored image.

Abstract: Diffusion models break down the challenging task of generating data from high-dimensional distributions into a series of easier denoising steps. Inspired by this paradigm, we propose a novel approach that extends the diffusion framework into modality space, decomposing the complex task of RGB image generation into simpler, interpretable stages. Our method, termed {\papernameAbbrev}, cascades modality-specific models, each responsible for generating an intermediate representation, such as contours, palettes, and detailed textures, ultimately culminating in a high-quality RGB image.
Instead of relying on the naive LDM concatenation conditioning mechanism to connect the different stages together, we employ Schr\"odinger Bridge to determine the optimal transport between different modalities.
Although employing a cascaded pipeline introduces more stages, which could lead to a more complex architecture, each stage is meticulously formulated for efficiency and accuracy, surpassing Stable-Diffusion (LDM) performance.
Modality composition not only enhances overall performance but enables emerging proprieties such as consistent editing, interaction capabilities, high-level interpretability, and faster convergence and sampling rate. 
Extensive experiments on diverse datasets, including LSUN-Churches, ImageNet, CelebHQ, and LAION-Art, demonstrate the efficacy of our approach, consistently outperforming state-of-the-art methods.
For instance, {\papernameAbbrev} achieves notable efficiency, matching LDM performance on LSUN-Churches while operating 2$\times$ faster with a 3$\times$ smaller architecture.
The project website is available at:
\href{https://toddlerdiffusion.github.io/website/}{$https://toddlerdiffusion.github.io/website/$}

Published as a conference paper at ICLR 2025
TODDLERDIFFUSION: INTERACTIVE STRUCTURED IM- AGE GENERATION WITH CASCADED SCHRÖDINGER BRIDGE
Eslam AbdeIrahman 1, Lianghing Zhao ' Vincent Tao Hu
Matthieu Cord Patrick Perez 2, MMohamed Elhoseiny
KAUST LLMU Valco Al Kyyuta
Speed Up
Editing
Real Image Editing.
Efficiency
Convergence
Robustness
Interactivity
Original SE|6-1
SE(K-8)
Origioll
stik-s)
Performing on har compared to LDM while heing 2x faster, using 3x smaller architecture. 2) Interactivity: Producing intermediate outputs, e.g.. sketch and palette. 3) Editing: Offering free, robust editing capabilities even for real images. 4) Robustness: Our method is robust against input condition perturbations, c.g., sketch sparsity.
Figure I: overview ToddlerDiffusion capabilities: Speed- Up:
ABSTRACT
Diffusion models break down the challenging task of generating data from high- dimensional distributions into series of easier denoising steps. Inspired by this paradigm, we propose a novel approach that extends the diffusion framework into modality space, decomposing the complex task of RGB image generation into simpler, interpretable stages. Our method, termed ToddlerDiffusion, cascades modelity-specific models, each responsible for generating an intermediate represen- tation, such as contours, palettes, and detailed textures, ultimately culminating in a high-quality RGB image. Instead of relying on the naive LDM concatenation condi- tioning mechanism to connect the different stages together, we employ Schrödinger Bridge to determine the optimal transport between different modalities. Although employing cascaded pipeline introduces more stages, which could lead to more complex architecture, each stage is meticulously formulated for efficiency and accuracy, surpassing Stable-Diffusion (LDM) performance. Modality composition not only enhances overall performance but enables emerging proprieties such as consistent editing, interaction capabilities, high-level interpretability, and faster vergence and sampling rate. Extensive experiments on diverse datasets, including LSÜN-Churches, ImageNet, CelebHQ, and LAION-Art, demonstrate the efficacy of our approach, consistently outperforming state-of-the-an methods. For instance, ToddlerDiffusion achieves notable efficiency, matching LDM performance on while operating 2x faster with a x smaller architecture. The project website available at: httpps: //toddlerdi f fusiion gith.hh.thhhh..... io website/

---

Published as a conference paper at ICLR 2025
Xo
Xt
XT
Figure 2: Trajectory comparison. We compare our approach (Top) against LDM (bottom) in terms of trajectory. Instead of starting from pure noise such as LDM, our approach, ToddlerDiffusion, leverages the intermediate stages achieve steady and shorter trajectory.
INTRODUCTION
In recent years, diffusion models (Sohl-Dickstein et al. 2015 Ho et al. 2020: Song et al. 2020b)
have made significant strides across diverse domains, revolutionizing image synthesis and related tasks by transforming noisy, unstructured data into coherent representations through incremental diffusion steps (Ho et al. 2020: Preechakul et al. 2022 Dhariwal & Nichol 2021] Nichol Dhariwal
2021). Their versatility extends beyond image generation to tasks such image denoising (Gong ct al. 2023] Kulikov et al.] 2023] Zhu et al., 2023), inpainting (Lugmayr et al. 2022 Alt et al.
2022), super-resolution (Li et al. 2022 Gao 2023), ) and applications in 3D content creation 2023 Poole et al. 2022). data augmentation (Carlini et al.
(Anciukevičius et al. 2023 Qian et al.
2023 Li al.] [023c] Voetman et al. (002)) medical imaging (Chung al.. 2022 Kazerouni
et al. 2022 Wu et al. 2022 Wolleb et al. | [2022), anomaly detection, and more. Diffusion models,
such as Denoising Diffusion Probabilistic Models (DDPMs) (Ho al. 2020). draw inspiration from the diffusion process in physics, breaking down the intricate task of generating data from nigh-dimensional distributions into sequence of simpler denoising steps. this framework, instead of generating an image in a single step, the task is reformulated as a gradual denoising process, where a noisy image is iteratively refined until the final, detailed image emerges. This step-by-step approach leverages the power of decomposition: by solving series of simpler tasks, we can ultimately solve the original, more challenging problem.
Inspired by this philosophy, we propose ToddlerDiffusion, novel method extending this decom- position principle beyond just noise removal. Instead of tackling the image generation problem in one go, decompose into modality-speciffe stages. Each stage focuses generating different aspect or modality of the image, such as abstract contours or sketches, followed by color palettes, and finally, the detailed RGB image. This approach mimics the learning process of a child developing artistic skills (Stevens) 2012, Marr 2010 1974). where the child first learns to draw basic outlines, then adds colors, and finally fills in the details. We use the term "ToddlerDiffusion" reflect this gradual, step-by-step learning and generation process.
Despite the intuitive appeal of task decomposition, it introduces significant challenges. More stages mean more parameters, longer training times, and potentially slower sampling speeds. However, our hypothesis is that by addressing simpler tasks at each stage, we can leverage smaller architectures tailored to each modality, resulting in a more compact overall model compared to traditional single- stage architectures. Moreover, multi-stage frameworks often suffer from error accumulation (Ho) et al., 2022 Huang et al., 2023), where inaccuracies in earlier stages can propagate and amplify through subsequent stages. However, we showed that by using simple techniques such as condition truncation dropout, can significantly mitigate the error accumulation, Appendix A.7 Additionally, learning these intermediate representations, e.g., counters, explicitly or implicitly is challenging. where the first requires paired data and the latter requires special complicated designs (Singh et al.) 2019 Li et al.) 202I). To tackle the guidance challenge, we leverage pseudo GT and show that this noisy GT is enough to learn robust intermediate modalities.
2

---

Published as a conference paper at ICLR 2025
To implement this framework, considered two approaches. The first is the standard Latent Diffusion Model (LDM) (Rombach et al.] 2022) conditioning mechanism, using techniques such as cross-attention and concatenation to link stages. Given focus spatial modalities like sketches, concatenation seemed more appropriate. However, this method proved insufficient for achieving our hypothesis due to its weak conditioning, which results in poor information transfer between stages. We instead adopt a more principled approach by employing the Schrödinger Bridge (Wang et al. 202T Li al. 2023a), method from stochastic optimal transport theory, to determine the optimal path between modalities at each stage. Unlike simple concatenation, which starts each stage from state of pure noise, the Schrödinger Bridge enables us to begin each stage from meaningful signal with a higher Signal-to-Noiss Ratio (SNR) (Kingma et al. (021), illustrated in Figure 2 This leads to more effective and stable progression through the generation process, significantly reducing the denoising steps required during training and testing.
This structured, multi-stage approach not only allows for more efficient and stable generation process but also enables a range of emerging capabilities. ToddlerDiffusion supports consistent editing of both generated and real images and facilitates user interaction both conditional and unconditional settings, shown in Figure [ Moreover, while sampling efficiency not our primary objective, our approach yields faster sampling and training convergence due to the shorter trajectories between stages (Figure 2). resulting in fewer denoising steps and less computational cost (Figure [ For instance, IDM is trained using 1K steps; in contrast, ToddllrDiiffusion could be trained using only 10 steps with minimal impact on generation fidelity. To the best of our knowledge, ToddlerDiffusion is the first work to successfully integrate and achieve vancemmetss across multiple domains, including controllability, consistent editing, sampling and training efficiency, and interactivity, within a unitied framework, as shown in Figure T Our extensive experiments on datasets such as (Yu et al. 2015). ImageNet, and CelebA-HQ (Karras al.) (017)) demonstrate the effectiveness of this approach, consistently outperforming existing methods. ToddlerDiffusion not only redefines the image generation process but also paves the way for future research into more structured, interpretable, and efficient generative models.
Our contributions can be succinctly summarized as follows:
We introduce cascaded structured image generation pipeline, denoted as ToddlerDiffusion, that systematically generates a chain of interpretable stages leading to the final image. Leveraging the Schrödinger Bridge enables us to train the model from scratch using minimal number of steps: 10 steps only, in addition to, performing on the bar compared to
LDM while being 2x faster, using 3x smaller architecture.
ModdlerDiffusion provides robust editing and interaction capabilities for free; without requiring any manual annotations.
Introduce a novel formulation for sketch generation that achieves better performance than LDM while using 41 x smaller architecture.
2
TODDLERDIFFUSION
We introduce ToddlerDiffusion which strategically decomposes the RGB image generation task into more straightforward and manageable subtasks, such as sketch/edge and palette modalities. In Section 2.1 we detail the formulation each stage. By breaking down the generation process into simpler components, our model not only allows for more compact architecture but also enables dynamic user interaction with the system, as illustrated in Figure I Notably, these advancements are achieved without relying manual ground truth annotations, instead utilizing human-free guidance for enhanced efficiency and practical applicability (Section 2.2). Lastly, we demonstrate how our method facilitates robust, consistent edits across stages (Section 2.4
2.1 TODDLER GROWTH
The development of our method is inspired by child's growth, where the learning process in their brains decomposed into stages and developed step-by-step (Stevens) 2012, Marr 2010 1974). By analogy, decompose the 2D image synthesis into different, more straightforward stages. We employ the sketch and palette as intermediate stages, resulting three cascaded stages approach. However, our approach is generic and seamlessly we can add or remove intermediate steps, based
3

---

Published as a conference paper at ICLR 2025
on the application needs. Mainly, the pipeline can be clustered into two main modules: 1) The unconditional module, that generates the first modality, e.g. sketches, based on pure noise. 2) The conditional module, that condition the previous modality to generate the next intermediate modality, e.g., palette, or the final RGB image. Both modules can be implemented using Latent Diffusion Models (LDM)), where unconditional LDM will start from noise to generate sketches, then by employing the concatenation conditioning mechanism we can condition on our spatial intermediate modalities; sketches and palettes. We instead adopt more principled approach by employing the Schrödinger Bridge to determine the optimal path between modalities at each stage. Unlike simple concatenation, which starts each stage from a state of pure noise, the Schrödinger Bridge enables us to begin each stage from meaningful signal with higher Signal-to-Noise Ratio (SNR).
Schrödinger Bridge. The Schrödinger Bridge problem arises from intriguing connection between statistical physics and probability theory. At its core, it addresses the task of finding the most likely stochastic process that evolves between two probability distributions over time, Po and pr. governed by Brownian motion, while minimizing the control cost. Mathematically, it could be written as: P argmin (L(P W), where P is the probability path that minimizes the Kullback-Leibler (KL) PEP divergence with respect to a reference Brownian motion W, and subject to marginal constrains; Po = Po and Pr = pr. The process can be described as stochastic differential equation (SDE):
dx. = vtt, X.) dt + a dW,,
where X, represents the state at time I, v(1,, X.) is the optimal drift that minimizes the KL divergence and dW, is the Wiener process (Brownian motion).
Conditional Module. By analogy, the two probability distributions pr and po represent the distribu- tions of features from simpler (sketch) and more complex (RGB) modalities, respectively. this context, the goal of the Schrödinger Bridge is to find the optimal stochastic process that transforms the simpler modality (e.g., sketch) into the more complex one (e.g.. RGB image) while minimiz- ing divergence measure like KL divergence while respecting the boundaries. Given condition E "I x3 that could be sketch, palette, or any other modality. we define the forward process as an image-to-image translation and formulate the problem using the discrete version of schrödinger bridge as follows:
Te == (1 ax)ar + auy' + oiee,
oo az oi,
where i the stage number, xy and y are the boundaries of the bridge, of controls the noise term, and a₂ blends the two domains. In the SDE, the term "u(t. X,)" is the deterministic drift, while "dW," " represents the diffusion term. In contrast, we follow the DDPM formulation, which works in the discrete space. Therefore, by analogy, the terms "a,z0 + (1 ar)y" and "o,e, represent the drift and diffusion terms, respectively. of represents uncertainties, following a sphere formulation that constructs a bridge between the two domains. Whereas, at the edge of the bridge, we are fully confident we are in one of the distributions pr and po. thus of = O. In contrast, the uncertainties increase while moving away from the edges. reaching their maximum value in the middle between the two domains, forming the bridge shape. Eq. 2 forms the skeleton for 2nd and 3rd stages, where:
2 Fo (2 1
=i=2 2 (Palette)
a = F. (apaa :8=3 2 (Palette) y'=
20
:i=3 3 (Detailed Imuge)
= F
:i= 3 (Detailed Image)
where is the stage number, Fp and F, are palette and sketch functions that generate the pseudo-GTT respectively, detailed in Section 2.2
Unconditional Module: 1s Stage (Sketch). This stage aims generate abstract contours zá E Hx starting from just pure noise or label/text conditions. In both cases, we name this module unconditional as it does not condition on previous modality be- cause it will generate the first modality, the sketch. One possible solution is to utilize the original LDM, which is not aligned with the sketch nature, as shown in Figure 3 case A. More specifically, we are starting from a pure noise distribution Xy and want to learn the inverse mapping q (x,-1 xx) to be able to sam- ple real data zy- The issue relies upon the vast gap between the two domains: xy and zn- Following (Kingma al. 2021). this can be interpreted as
Figure 3: Comparison between different formulations for the 1
stage. This depicts the forward process for each formulation.
4

---

Published as a conference paper at ICLR 2025
signal-to-noise ratio (SNR), where (t Consequently, at the beginning (t TT, ar yields to O, therefore SNR(7)=0. On the contrary, to fill this gap, we propose a unique formulation tailored for sketch generation that adapts Eq. 2 Specifically, we introduced two changes to Eq. 2 First, we replaced the Gaussian distribution e, with discretized version, the Bernoulli distribution, as shown in case B, Figure 3 However, this is not optimized enough for the sparsity nature of the sketch, where more than 80% of the sketch is black pixels. Thus, we set our initial condition to y = =F((z)), where Fa is a dropout function that takes the GT sketch z8 and generates more sparse version. To align our design with the sketch nature, as shown in Figure 3 part C, we changed of to follow a linear trend instead of the bridge shape by setting the variance peak æT. We have to make sure to set the variance peak to small number due to the sparsity of the sketch. This could be interpreted as adding brighter points on black canvas
as control points during the progressive steps (I' - 0) while converging to form the contours, as shown in Figure 3 part C. More details about the sparsity level importance are shown in Section A.6
2.2 TODDLER GUIDANCE
A crucial factor for the success of our framework, ToddlerDiffusion, is obtaining accurate and human-free guidance for the intermediate stages. Two modules, F, and Fp. are employed to generate contours/skethes and palettes, respectively.
Sketch. The first module, =/ FF (z serves sketch edge predictor, where o0 # x3 is the ground-truth RGB image, and zn E /H is the generated sketch. For instance, F, could be PidiNet (Su et al. 202T) or Edter (Pu et al. 2022) for sketch generation, or Canny (Canny 1986) and Laplacian [Wang] 2007) edge detectors.
Palette. The palette can be obtained without human intervention by pixelating the ground- RGB image z = Frp (2,5))), where = E #R x3 is the palette image, and Fp will be the pixelation function. However, we introduce more realistic way have high-quality palettes for free by first generating the segments from the RGB image using FastSAM (Zhao et al.) (023b)). Then, a simple color detector module utilized to get the dominant color, e.g., the median color per segment. This design choice is further analyzed in Section A.6, More details are mentioned in the Appendix.
2.3 TRAINING OBJECTIVE & REVERSE PROCESS
Training Formulation. We adapt the conventional diffusion models' learning objective (Ho et al. 2020; Sool-DDckstein et al., 2015). i.c., Variational Lower Bound (ELBO), to include our new condition y, whereas, each marginal distribution has to be conditioned on y, as follows:
LELSo (DKL w'llee(ere))) Dru (o(#:-1 ('lloooee- (e., log (o
(4)
where Pa is a function approximator intended to predict from xy,, Using Bayes'rule, rulle, we can formulate the reverse process as follows:
(ae-1 | ue. wb. v0) = =N (i--is (2., 2b. of(1)
(5)
of 1
d - oz)²
1
a
21 + au-1(1
Yab (ai-1 2i
(6)
1-a
07(1 a.-1)
I a1-1 of
(1 -1-00 az)² of a: 1)²
The derivation for Eq. 6 and Eq. 7 is detailed in Appendix A.
The overall training and sampling respectively.
pipelines are summarized in Appendix A.5 Algorithms D and
Training Objectives. Previous work (Rombach et al., 20022 showed that learning directly the noise e enhances the performance compared to predicting the zy directly. This is intuitive as predicting the difference X I₁ is easier than predicting the original complicated Zn. However, in our case, this is not valid due to the conditioned domain y, which makes predicting the difference much harder, as shown in Eq.
I Xa au
ro) + d:ér
(8)
Accordingly, directly predict the Zo. Please refer to Section A.6 for a detailed analysis.
5

---

Published as a conference paper at ICLR 2025
2.4 TODDLER CONSISTENCY
Our novel framework, ToddlerDiffusion. inherently provides interactivity, allowing to ob- serve and engage with the intermediate outputs of various modalities. This interactivity further strengthened by ensuring consistency, erucial element for effective control and editing capabilities.
Generated Images Editing. To ensure consistency during editing, we store the noise e and the intermediate modalities used in generating the initial reference image, creating memory that will help us achieve outstanding consistency. Throughout the editing session, the memory, that was constructed while generating the reference image, remains fixed until the user resets it by generating a new reference image. F [0]emnnttratess our framework's s editing capabilities, highlighting the strong consistency it achieves. Additional editing examples can be found our demo.
Real Images Editing, Real image editing is much more challenging as we cannot access intermediate modalities and noise. Therefore, given real image 2 first, we run Fx and (x to get the corresponding sketch vn and palette x/i respectively. Then, we utilize the recent DDPM inversion technique (Huberman-Spiee) Spiegellgaas et al.] 2024) to store the noise ft. More specifically, to get the noise, we have first to run our forward process, Eq. 2 using random noise E, and store the corresponding intermediate states Z₁. Then, run reverse process, Eq. 6 to try to recover the original image. During the reverse process, the model will predict وo (Tz, Tb. y²) using Eq. 6 Then, our goal is that given the predicted id,, the predicted conditions y and the stored intermediate states Tt-1+ 1, we want to estimate the noise 6, Following these two steps, we construct our memory, which will be used similarly to edit real images.
EXPERIMENTAL RESULTS
To probe the effectiveness of our proposed framework, we first compare our approach against LDM when trained from scratch on unconditional datasets, LSUN-Churches and CelebHQ, and conditional ones, ImageNet. Then, we show that our approach can leverage from the high-quality SD-v1.5 weights, enabling the model be conditional on new modalities seamlessly without adding additional modules (Zhang et al. 2023 Zhao et al. 2024) or adapters (Mou et al. (2024).
3.1 TODDLERDIFFUSION FROM SCRATCII
In this section, first we will holistically compare our method, ToddlerDiffusion, against LDM regard- ing generation quality and efficiency, e.g., convergence rate and number of denoising steps. Then, we will compare our conditioning ability with ControlNet and SDEdit. Generally, ToddlerDiffusion consists of three cascaded stages: sketch, palette, and RGB generation. However, in the following section, we will show the versatility of our method by omitting the 1st or the 2nd stages, enabling our method to start from a reference sketch or palette, respectively, instead of generated ones.
Unconditional Generation. We evaluated our framework, ToddlerDiffusion, against LDM on two unconditional datasets, i.e., LSUN-Churchn (Yu et al.) (0155) and Celeb datasets (Karras et al. 2017). For a fair comparison, and as we notice challenges in replicating LDM results]] we have reproduced LDM results. As shown in Table ! we boost the LDM performance by FID point on the two datasets, using 3x smaller architecture and 2x faster sampling, shown in Table
Efficient Architecture. While the primary focus of this paper is not on designing an efficient architecture, our method inherently benefits from architectural efficiency due to the shorter trajectory between stages, as illustrated in Figure 2 The key insight is that DofdlerDiffusion simplifies each generation task by breaking it into modality-specific stages (e.g., from sketch palette). This approach allows us to leverage significantly smaller networks at each stage because solving the problem is easier, where the distributions of the domains, such as sketches and palettes, are much closer than the distribution between pure noise and RGB images. As a result, the neural network at each stage can be more lightweight. Furthermore, as shown Table 2 we operate directly in the image space for the first two stages (i.e., noise to sketch and sketch to palette). This eliminates the need for complex encoder-decoder architectures, such as VQGAN, typically used to generate
See issues
32.
262
90
142
30 and
138
the LDM GitHub repository.

---

Published as a conference paper at ICLR 2025
Table 1: Benchmark results on three datasets; Celebl K
Karras
Table 2: A detailed unalysis of our method complexities comngrred to LDM on the CelebHQ dataset Karras et al. (2017). We report the training and sampling throughput Kataset[Karras operating on the pixel space. Therefore, the VQGAN enconder and decoder used (N/A). The training time reported till convergence, ie, 600
2017)_ LSUN Churches Yu et 2015 ImageNet
Dong al. 2009)
each stage. For etliciency, the lirst two stages are
Datasel
Method
Epuchs
FID
1
KID L
Prec.
Recall
spochs, using 4 A100.
CelebH1Q
LDM
Curs
600
600
8.15
7.10
0.013
0.009
0.61
0.52
0.41
0.47
VOGAN Puram
UNet
Train Time
Time
Churches
LDM
Ours
250
250
6.19
7.30
0.009
0.005
0.71
0.59
0.39
0.44
11 Neose Skench
Sketch Piless
NIA
N7A
1,3M
1.5M
1hr
1hr
85
85
ImageNet
LDM
350
350
8,55
7.8
0.015
0.010
0.51
0.58
0.40
0.32
Ours Stgges)
Paletee RGB
L.M
55.3M
1048M
263M
101M
26hr
24hr
38
2.27ips
1.21 fps
2.3
Oue Tabein))
Figure 4: "Trining convergence comparison between DiffusiDiffusion and LDM,
Higure Trimuning denoising steps abla- tion study during training and sampling.
Figure 6: Training steps ablation study on LSUN-Churches datasct.
detailed images from latent representations. Combining these factors, ToddlerDiffusion achieves 3x reduction in network size despite cascading across three stages.
Faster Convergence. In addition to outperforming LDM across multiple datasets and metrics, as shown in Table ToddlerDiffusion exhibits a significantly faster convergence rate. We analyzed the convergence behavior by reporting the FID score every 50 epochs, as shown in Figure 4 Remarkably, our method achieves the same performance after just 50 epochs as LDM does after 1,50 epochs. Furthermore, after 150 epochs, ToddlerDiffusion matches the performance LDM achieves after 500 epochs. This substantial improvement in convergence speed highlights the strength of our approach in efficiently navigating the feature space between stages. It's important to note that faster convergence was not an explicit target of our method but rather an emergent property of the pipeline's ' design. Other techniques tailored explicitly for improving convergence speed, e.g., (Hang et al. 2023), can be considered orthogonal and complementary to our method.
explictf
Trimming Denoising Steps. Another unique characteristic of TooddeerDiffusiin its ability to reduce the number of denoising steps during both training and sampling without significantly impacting performance. As shown in Figure 5 our framework maintains consistent and robust results when the number of steps is reduced from 1000 to 100. In contrast, the performance of LDM degrades drastically under the same conditions. ToddleDDiffssion can trim denoising steps while maintaining a good Signal-to-Noise Ratio (SNR), particularly for larger time steps, further discussed in Section 2.1 Additionally, Figure higghlighss another intriguing capability of our method: models trained with fewer denoising steps outperform those trained with larger step counts when later using techniques like DDIM (Song et al., 2020a) to reduce sampling steps. For example, as seen in Figure 6 the blue curve (trained with 100 steps) achieves an FID score of around 60 when using only ten steps for sampling. However, if we initially train the model with just ten steps, as represented by the purple curve, the performance improves significantly, achieving an FID score of 15. This demonstrates the efficiency and adaptability of ToddlerDiffusion, even when minimizing the denoising process.
Class-Label Conditional Generation. To holistically evaluate the performance of Toddler we tested it on the widely-used ImageNet dataset, where the condition type is a class label. Due to limited resources, we conducted our experiments on ImageNet- 100, a subset of ImageNet containing 100 classes instead of the full 1,000. As shown in Table [ Difflioniifuusion performs better than LDM on key metrics such as FID, KID, Precision, and Recall.
Controllability. We evaluated our editing capabilities against two notable methods: SDEdit (Meng) et al. 2021) and ControlNet (Zhang et al., 2023). As illustrated in Figure 7 SDEdit struggles when conditioned binary sketches, which is expected given that it is training- approach designed to operate with conditions closer to the target domain (i.c., RGB images). Consequently, compared to ControlNet, based method that adds copy of the encoder network, our approach
7

---

Published as a conference paper at ICLR 2025
Conchnl
Skotch
Sketch Palette
Figure 7: Comparison between editing capabilities and SDEdit (Meng
2021
Figure Sketch robusinss amalysis.
demonstrates superior faithfulness to the sketch condition, consistency across the generated images, and overall realism. Starting from the converged model train on the CelebHQ dataset for 600 epochs, we train our method and ControlNet for and additional 50 epochs with the sketch as a condition. Despite ControlNet's larger model size, our method achieves these improvements without adding additional parameters. Quantitatively, we achieved an FID score of 7, outperforming SDEdit and ControlNet, which achieved 15 and 9, respectively.
3.2 ROBUSTNESS ANALYSIS
Label Robustness. Beyond these quality comparisons, we also conducted a robustness analysis to assess our model's behavior when faced with inconsistencies between condi- tions. Specifically, we deliberately provided incorrect class labels for the same sketch. This mismatched the sketch (geometry) and the desired class label (style). Despite the inconsistent conditions, as illustrated by the off-diagonal images in Figure 9 our model consistently adheres to the geometry defined by the sketch while attempting to match the style of the misassigned category, demonstrating the robustness of our approach. Successful cases are highlighted in yel- low, and even in failure cases (highlighted in red), our model avoids catastrophic errors, never hallucinating by completely ignoring either the geometry or the style. More examples can be seen in Appendix [B.3]
Platpggin
Spotted
Too Trop
Sketch Robustness. To further validate the robustness of ToddlerDiffusion, we designed an aggressive robust- ness test aimed at evaluating the model's ability to handle sketch perturbations. Specifically, we generated three variants of the input sketches using the Structured Edge (SE) detector (Dollár & Zitnick 2014), varying the kernel size K to introduce different detail and noise levels. As shown in Figure g our model consistently demonstrates strong robustness despite these perturbations being absent during training. The left part of Figure 8 depicts our model's output using the 2nd stage only (Sketch to RGB), while the right part demonstrates the output when using both the 2nd and 3rd stages. Even with significant variations in the sketches, TodflerDiffusion maintains high fidelity to both the geometry and style, underscoring its ability adapt to unseen input distortions without compromising output quality.
Figure 9: Our results ImageNet dataset. lirst col- umn depicts the input sketch. The diagonal (green) represents the generated outpot that follows the sketch and the GT label. The off-diagonal images show the output given the sketch and inconsistent label. These adversarial labels show our model's iss robustness, where the successful cases in yellow, and the failure ones are in red.
3.3 EDITING CAPABILITIES
Generated Images Editing. In Figure 10 we showcase the editing capabilities starting from gener- ated sketch and RGB image (A). Our method allows for the removal of artifacts or unwanted elements, highlighted in red (B), the addition of new content, shown in yellow (C-D), and modifications to existing content, marked in green (E-F), simply by manipulating the underlying sketch. These consis- tent edits are supported in unconditional and conditional generation scenarios. While conditional generation naturally lends itself to such control, ToddlerDiffusion extends this capability to unconditional settings and other conditional models like Text-to- In these cases, refer to this property as "interactivityy" allowing users to seamlessly edit through intermediate modalities (sketch and palette) regardless of the original input format, whether class-label or text.
8

---

Published as a conference paper at ICLR 2025
Figure 10:
Controllability ahility our framework, Todill:Diffasson, Starting from the generated sketch and RGB image (A), we can remove artifacts undesired parts red (B), add new yellow (C-D), and edit the existing content green (E-F) by manipulating the sketch.
Real Images Editing. As explained Section 2.4 we ensure consistent edits for real images by storing the noise used during the generation process for later edits. The primary challenge with real image editing is that we don't ton't have access to the noise values at each denoising step, as our model didn't generate the image. To address this, employ DIDMM-invession (Huberman-Spiegelge et al. (2024) to reconstruct the intermediate noise. As illustrated in Figure [ our model achieves consistent edits real reference images. In Appendix B.22 compare against SEED-X (Ge et al. 2024). and our model achieves more consistent edits, as shown.
3.4 INCORPORATING
INTO STABLE DIFFUSION
In this section, we will answer an important question: "Can our method be integrated seamlessly into the existing powerful Stable-Diffie Diffusson architecture?" This question is crucial as follows: 1) Despite our approach's 's great capabilities and performance, leveraging the available powerful weights trained on large-scale datasets such SD is still important. 2) Our approach follows different forward and reverse processes, Eq. 2 and Eq. 6 respectively, which could make it challenging to leverage weights that are trained using different training objectives.
Integration Recipe. This section will detail the recipe for successfully integrating SD weights into approach. initially adopted To the training objective instead of the original SD objective, and employed Low-Rank Adaptation (LoRA) for fine-tuning. However, we observed that this approach does not converge easily due to the significant disparity between the pre-trained SD model's training objective and the new objective. Consequently, we opted to fine-tune the entire model. Remarkably, after just 10k iterations, the model generated meaningful images that aligned perfectly with the input conditions, such as sketches or palettes. More qualitative results can be seen in Figure 24 in Appendix B.6
Controllability. We convert our method to "sketch2RGB" by omitting the 1st stage to show our controllability capabilities. By doing this, we can compare our model against the vast tailored sketch-based models built top of SD-v1.5. However, these methods have been trained for a long time on large-scale datasets. Thus, for a fair comparison, we retrain them using CelebHQ datasets for 5 epochs starting from SD-v1.5 weights. As demonstrated in Appendix B.5 despite this is not main focus, our method outperforms conditioning conditioning methods without adding additional modules (Zhang al. 2023 Zhao et al. 2024) adapters (Mou et al. 2024).
4 DISCUSSION AND FUTURE WORK
In this section, we will discuss some possible future work that shows the versatility of ToddlerDiffu- sion and its potential to drive innovation across image, video, and 3D tasks. Detailed future work is discussed in Appendix A.9
Distill the knowledge from substage features for perception tasks. The intermediate outputs from Toddler Diffusion (c.g., the sketch stage) can be reused strong geometric priors for perception tasks by distilling the knowledge into perception networks to enforce geometric or perceptual constraints, making them structure-aware and geometry- aligned (Zhao et al. 2023a).

---

Published as a conference paper at ICLR 2025
Reusing Diffusion Features as High- and Low-Frequency Representations, Toddler Diffusion's staged outputs inherently capture distinct frequency information, with the sketch stage emphasizing low-frequency global structures and geometry and the palette and RGB stages capturing high- frequency details like textures. These features can be repurposed into a dual-branch network (Bi et al.) Wei al. 2024): one branch focusing
www-freuenccy representations for tasks requiring global context (e.g., semantic segmentation, object localization) and the other specializing in high- freuencyy details for tasks like texture classification, instance segmentation, or boundary refinement, Integrating this knowledge into VLMs can improve performance and generalization for perception tasks.
Addressing Limitations in T2I Models. significant challenge for text-to-image (T2I) models lies in handling complex prompts involving counting or spatial relations (Bakr et al. 2023 Ghosh al. 2024). Fixing these issues in the sketch or abstract stage will be easier and simpler due to its lightweight. In addition, the subsequent stages (e.g., sketch-to-RGii will remain unaffected and reusable. This approach allows targeted improvements without retraining/fnetuningg the entire pipeline, significantly reducing computational costs.
Modular Prompt Handling for T21 Models. Toddler Diffusion can simplify complex text prompts by decomposing them into modular components mapped to specific stages. For example, detailed prompt can be split into geometry- focused instructions for the sketch stage and color-specific details for the sketch-te to--tlette stage. In addition, can add more stages and specific prompts such as depth, e.g., objı closer than obj2. This enables a more focused and interpretable generation, particularly for long, detailed prompts with intricate requirements.
Given a complex prompt: "A bustling city street during sunset with a red double-decker bus on the right, a blue car parked on the left, people walking on the sidewalk, and sunlight casting long shadows on the ground." " will be split into:
For instance:
Structure-Focused prompt: "A city street with double-decker bus the right, car parked on the left, and people walking on the sidewalk." "
Palette-Focuset prompt: "The double-decker bus is red, the car is blue, the sky shows sunset with shadows on the ground."
Video Generation in Abstract Domains. Generating videos directly in the RGB domain is com- putationally expensive and prone to inconsistencies. Toddler Diffusion can split the problem into geometry consistency and stylistic consistency. Specifically, it first generates videos in an abstract domain (e.g., sketches), ensuring geometric consistency across frames. Then, these sketches can be refined into RGB frames using another stage, Sketch-to-RGii conditioned on reference frame for color and style consistency. This two-step process simplifies the complex task of maintaining temporal and stylistic coherence in video generation.
5 CONCLUSION
Toddle:Diffusion introduces a paradigm shift in image generation by breaking down the complex task into more manageable, modality-speciifc stages. Inspired by the natural learning process of a child, our framework elegantly mirrors how human artists progress from rough sketches to fully detailed images. This structured approach allows to simplify each stage of the generation process, enabling the use of smaller architectures and achieving efficient, high-quality results. Despite the potential challenges introduced by multi-stage frameworks, such increased parameter counts and error accumulation, ToddlerDiffusion overcomes these obstacles through the strategic use of the Schrödinger Bridge, which ensures smooth transitions between modalities. By maintaining a high Signal-to-Noise Ratio (SNR) at each stage, we enable stable, effective progression, minimizing the number of denoising steps required during both training and sampling. This results in faster convergence, lower computational cost, and enhanced sampling efficiency. Moreover, ToddlerDiffusion provides level of control and interaction that unprecedented in current diffusion-based models, offering consistent editing capabilities across both generated and real images. Through extensive experiments on multiple datasets, we have shown that ToddlerDiffusion outperforms existing state-of-the- art models like LDM in terms of quality, interactivity, and speed. The flexibility and versatility of our approach make it applicable to a wide range of tasks, and its emergent properties, such as faster convergence and improved editing control, further solidify its potential for real-world applications.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Tobias Alt, Pascal Peter, and Joachim Weickert. Learning sparse masks for diffusion-based image inpainting. In Iberian Conference on Pattern Recognition and Image Analysis, pp. 528-539. Springer, 2022.
Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
27608-12611 2023.
Eslam Mohamed Bakr, Pengzhan Sun, Xiaogian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20041-20053, 2023.
Alberto Baldrati, Davide Morelli, Marcella Cornia, Marco Bertini, and Rita Cucchiara. Multimodal- conditioned latent diffusion models for fashion image editing. arXiv preprint WXXv::2403.14820 2024.
Qi Bi. Jingjun Yi, Hao Zheng, Haolan Zhan, Yawen Huang, Wei Ji. Yuexiang Li, and Yefeng Zheng. Learning frequency-adage adapted vision foundation model for domain generalized semantic segmentation. The Thirty-eighth Annual Conference on Neural Information Processing Systems.
John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, 09x679-698, 1986.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, 2023.
Hyungjin Chung, Eun Sun Lee, and Jong Chul Ye. Mr image denoising and super-resolution using regularized reverse diffusion. IEEE Transactions on Medical Imaging, 42(4):922 2944944,,944 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. leee, 2009.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Piotr Dollár and C Lawrence Zitnick. Fast edge detection using structured forests. IEEE transactions on pattern analysis and machine intelligence, (7)8811558-1 1570, 2014.
Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10021-10030, 2023.
Songwei Ge, Vedanuj Goswami, Lawrence Zitnick, and Devi Parikh. Creative sketch generation. arXiv preprint arX1:20011.10033 2020.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granulan comprehension and generation. arXiv preprint arXiv:2404.1 143966 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
11

---

Published as a conference paper at ICLR 2025
Kuang Gong, Keith Johnson, Georges El Fakhri, Quanzheng Li, and Tinsu Pan. Pet image denoising based on denoising diffusion probabilistic model. European Journal of Nuclear Medicine and Molecular Imaging, pp. 1-11, 2023.
Tiankai Hang. Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. arXiv preprint Stt::2303.005566 2023,
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 68500, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans.
Cascaded difffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-2281, 2022.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint 20X1::2302.09778. 2023.
Inbar Huberman- Huberman-Spicgellae Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12469-12478, 2024.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint urXiv:1710. 10196, 2017.
Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, llker Hacihaliloglu, and Dorit Merhof. Diffusion models for medical image analysis: A compre- hensive survey. arXiv preprini (tt::2//1.0780// 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21701 2021.
Juil Koo, Seungwoo Yoo, Minh Hieu Nguyen, and Minhyuk Sung. Salad: Part-level latent diffu- sion for 3d shape generation and manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14441-14451, 2023.
Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. Sinddm: A single image denoising diffusion model. In International Conference on Machine Learning, pp. 17920-17930. PMLR, 2023.
Bo Li, Kaitao Xue, Bin Liu, and Yu-Kun Lai. Bbdm: Image-to-image translation with brownian bridge diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1952-1961, 2023a.
Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47-59, 2022.
Yuheng Li, Krishna Kumar Singh, Yang Xue, and Yong Jae Lee. Partgan: decomposition for image generation and segmentation. In British Machine Vision Conference (BMVC), 2021.
part
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511-22521, 2023b.
Zheng Li, Yuxuan Li, Penghai Zhao, Renjie Song, Xiang Li, and Jian Yang. Is synthetic data from diffusion models ready for knowledge distillation? arXiv preprint arXiv:2305. 2023c.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
David Marr. On the purpose of low-level vision. 1974.
12

---

Published as a conference paper at ICLR 2025
David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.00 01073, 2021.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4296-4304, 2024.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, PMLR, 2021.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot taage-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 1400:228,,448 2022.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
Mengyang Pu, Yaping Huang, Yuming Liu, Qingji Guan, and Haibin Ling. Edter: Edge detection with transformer. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 1402-1412, 2022.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. / 17843, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Finegan: Unsupervised hierarchical disentanglement for fine-grained object generation and discovery. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6490-6499, 2019.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning, pp. 2256-2266 PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1rXiv:2011.13445: 2020b.
Kent Stevens. The vision of david marr. Perception,
2012
Zhuo Su, Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian, Matti Pietikäinen, and Li Liu. Pixel difference networks for efficient edge detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5117-5127, 2021.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-am Plug-and-play diffusion features for text-driven saage-to-imagee translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1921-1930, 2023.
13

---

Published as a conference paper at ICLR 2025
Roy Voetman, Maya Aghaci, and Klaas Dijkstra. The big data myth: Using diffusion models for dataset generation to train deep detection models. arXiv preprint arXiv:2306.09776, 2023.
Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang. Deep generative learning via schrödinger bridge. In International Conference on Machine Learning, pp. 10794-10804, PMLR, 2021.
Xin Wang. Laplacian operator-based edge detectors. IEEE transactions on pattern analysis and machine intelligence, 29(5):886-85) 890, 2007.
Zhixiang Wei, Lin Chen, Jin, Xiaoxiao Ma, Tianle Liu, Pengyang Ling, Ben Wang, Huaian Chen, and Jinjin Zheng. Stronger fewer & superior: Harnessing vision foundation models for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer 2024.
Vision and Pattern Recognition, pp.
Julia Wolleb, Florentin Bieder, Robin Sandkühler, and Philippe Cattin. Diffusion models for medical anomaly detection. In International Conference on Medical image computing and computer-assisted intervention, pp. 35-45. Springer, 2022.
Junde Wu, Huihui Fang, Yu Zhang, Yehui Yang, and Yanwu Xu. Medsegdiff: Medical image segmentation with diffusion probabilistic model. arXiv preprint arXiv:2211.00611 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 550. 03365, 2015.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text- to-image diffusion models for visual perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5729557725557 2023a.
Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything, 2023b.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for toxt-to-image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5802-5810, 10, 2019.
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiczhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and play image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1219-1229. 2023.
14

---

Published as a conference paper at ICLR 2025
A TODDLERDIFFUSION METHOD (MORE DETAILS)
A.1
DERIVATIONS
From the forward process, have:
(1 - a:)a0 + anl + JGEt
(9)
It-1 = (1 a=-1)a0 lao + 14 + 01-16-1
(10)
From equation 10
we get:
0t-1
0t-1 Et-1
20
Q4-1 T1-1
(11)
ar-1
1 0t-1
Inserting equation into equation 9 we get:
1 a1
(1 - az)ay-
g 1201 - 04-1
7t-1
y
- 01-1
(1 (4)00-1
+04
6-- 1 + O26t
(12)
1 04-1
The reverse process follows: a ( Fe. v.. = N (#1-1:
To. (o. ,0,1)
(13)
Using Bayes' rule:
g( 1 20))
q (x+-1 1 It, No, ko. = g( 1t- mo)
(14)
g( zo
Plugging equation equation 10 and equation 12 into equation
14 yields:
of 1 at (x, To. v) of
It + (1 ac-1 (1
of=1 (1 ax)
(1-a 11 )20
a-
of -
a1) o/
+(01-1 - 01 (1 04-1) 1) o7
)y
(15)
6,- 0. 0.11 (1 a;)²
(16)
a
Finally, we can apply the DDIM non- Non-Markovian forward processes. In return, only the zo coefficient will be affected.
A.2 ARCHITECTURE DETAILS
Figure [I depicts the details of our architecture. We adopt the LDM architecture, which consists of two main blocks: 1) VQGAN Encoder-Decoder to convert the image into low-cost latent space. 2) Unet that responsible for predicting ø4-1- Figure 1T shows that the condition y is encoded into Ty However, it could be optionally concatenated to Ta. In addition, shown in orange and green the and a, respectively. Consequently, [2 depicts an abstract flow for the sampling mode, where show how our approach works starting from random noise till generating the final RGB image.
A.3 1e STAGE: ABSTRACT STRUCTURE
Sketch FID. A discrepancy exists between the reported conventional FID (RGB-FID), trained on ImageNet (Deng et al. 2009). and qualitative results, as illustrated in Figure [3 This discrepancy (Ge al. 2020) may arise from differences between the training data (RGB images) and the evaluation
15

---

Published as a conference paper at ICLR 2025
Friwarrd Ptrenst
Porword Preseia
Inser Space
(Sptneesl)
Reveree Frocees
An overview of proposed architecture, dubbed ToddlerDiffusion. The left block demon- strates the first stage, which generates a sketch unconditionally. Due to our efficient formulation, this stage operates in the image space 64 64 resolution. The right module depicts the third stage, which generates an RGB image given sketch only or both sketch and palette.
Figure 11:
[256, 256, 3]
[64, 64, 1]
[64, 64, 1]
[64, 64, 3]
Sketch Unet
Sketch
Palette to RGB
VQ-GAN Decader
Palette Unet
Y=xT
XT X
Xo=XA
rub
Xo
Figure 12: An abstract flow for the sampling mode, where we show how our approach works starting from random noise till generating the final RGB image.
3
Mettee
Usrt Ponomoters
Vinz (MincSac)
Sampliaa (1P5)
Noko Schookiker
Stradh P17
RCB F171
Clars
19M
51
Brdgy
110.
Dan
19
818
51
Eag
013
156
Dan 19
Benchmarking results for the 1d stage on the CelebA HQ dataset (Karras et al. 2017). The training time is calculated per epoch using 4 NVIDIA RTX A6000. The sampling time calculated per frame using one NVIDIA RTX A6000 with batch size equals 32.
Figure 13:
data (binary images). To bridge this gap, we introduce Sketch-FID by re-training the inception model (Szegedy et al. 2015) sketch version of the ImageNet dataset. We generate sketches for ImageNet RGB images using PidiNet (Su et al.) 2021) and train the inception model on the conventional classification task.
Noise Scheduler. In the 1d stage, where our starting point is a black image (Section 2.1 designing an appropriate noise scheduler is crucial. The bridge noise scheduler is intuitively unsuitable, climinates randomness by adding no noise at both edges. fixing the starting point to a black image. This hypothesis supported by empirical results Figure [3] row b. where the model outputs random patterns. We explored linear and logarithmic schedulers, finding the linear schedule superior, yielding Sketch-FID scores of 15.19 and 18.47, respectively (Figure 13 rows c-d).
Ours vs. LDM. In Section 2.1 we propose an alternative way to generate sketches by leveraging LDM (Rombach et al. 2022), shown Figure 3 row However, this approach deviates from the nature of sketching. Our proposed formulation, aligned with the topology of sketches (Figure row C), resulted in significant improvements over L.DM in both model complexity and performance, depicted in Figure I3 Our formulation (Section 22 allows direct operation on the image space (64 x 64) and compression of the Unet to a tiny variant without sacrificing performance. Despite the aggressive compression, Sketch-FID scores 15.19 and 49, using 41x smaller network.
performance is significantly better than LDM, with respective
Sketch Intensity. Controlling the intensity is crucial. Thus, conducted further analysis, shown in Table that probes for best performance, the intensity should follow the data distribution. We refer the intensity the ratio between the white pixels to the black ones in the sketch. The reported FID
16

---

Published as a conference paper at ICLR 2025
Table 3: Sketch Intensity Analysis.
Datweet/nntensty
31%
CelchHQ
7.4
7.9
9.2
10.5
12.7
Lson-Churches
6.9
8.3
10.1
Figure 14: Palette generation pipeline. First, we employ FasSAM (Zhaa el [1221 segment the image. Then, the dominant color for each segment.
Figure 1S: Ablation study for different inpur's types for the 2nd
Table 4: Comparison the LDM concalenation mecha- against the schridinger bridge (SB) using CelchHQ dataset.
Table S: Training objective ablation
stage on Churches Yu
2015
study Celebit() dataset (Karras 2017
Edges
M.5
LDM
Ours
51
Riden Phimt-
2nd Stage
FID
4) Skrrch-ae Phivtto-
Sketch i Paletio22 7.1
0 SAM (Calored)
13.3
15.1
8.2
scores in Table are the overall FII using all stages, and the intensity numbers are the initial intensity at = The GT intensity for CelebHQ and Chun-Churches are 19.2% and 23.5%, respectively.
A.4
2nd STAGE: PALETTE
introduce realistic way have high-quality palettes for free by first generating the segments from the RGB image using FastSAM (Zhao et al.] (203b)). Then, a simple color detector module is utilized get the dominant color, e.g., the median color per segment.
As depicted in Figure 14|
A.5 TRAINING AND SAMPLING ALGORITHMS
Algorithm Training Pipeline
for 1.... N For 1..... E for (urj
b. Train each stage sepurately
p. Train for E epochs
v²)
Loop on the dataset 'D Sampling the nosse Sampling step
e U(I....,
ag by applying Fq
based the stage Forward process
ora (1 0.l of
| pa ( v\ll
Simplified objective
Algorithm 2 Sampling Pipeline
zinter y [(H, (,
Initialize the list intermediatc output for each stage image Sampling the noise once
Initialize
A.C. 1)
For
L.......A.
Sequential Sampling
5:
2T =
Run forward process once Progressive Sampling D Reverse process
for T.......
Get 2e-1 by applying Bin Eq
Pe( ite)
D. Update the condition for the next stage Store the intermediate outputs each stage
Zinter append((r)))))
10. return Finter
A.6 ABLATIONS
1t Stage Representation. In Section 2.2 we explore the versatility of the 3rd stage (detailed image) by examining six input modalities, detailed in Figure [5] Comparing different contours representations, namely Edges (using Laplacian edge detector (Wang, 2007)), Sketch (utilizing PidiNet (Su et al. 2021)), and SAM-Edges (generated by SAM followed by Laplacian edge detector), we find that Sketch outperforms Edges, as edges tend to be noisier. However, SAM-Edges provides more detailed contours, yielding superior results. Notably, feeding SAM-Colored leads to significant performance degradation, likely due color discrepancies, observed in SAM in Figure [S] While SAM-EEges achieves optimal results, its computational intensity renders it impractical. Sketch and Edges are computationally inexpensive. Furthermore, Sketch's 's sparse and user-friendly
contrast,
17

---

Published as a conference paper at ICLR 2025
nature makes it more suitable for editing, facilitating interpretation and modification than dense, noisy edges. Consequently, we adopt Sketch as the input modality for subsequent experiments.
Palette Effect. Adding more guidance will offer more editing abilities to the pipeline and enhance the performance. As shown in Figure [3] rows b and d. when we incorporate the palette into the contours, i.e., edges and sketch, the performance improved by almost and 1.5, respectively. In addition, Palette-2 outperforms Palette-1 by FID score. A more detailed analysis of fusing the stages together and how mitigate the error accumulation can be found in Appendix A.7
Concatenation v.s. Schrödinger Bridge. We formulate the generation problem as image-to-image translation. For instance, generating palette given sketch generating the final RGB image given overlaid palette and sketch. One possible approach is to utilize the conditioning mechanisms offered by LDM (Rombach et al. 2022). c.g.. cross-attention and concatenation. However, these mechanisms are inefficient as their SNR tends to 0 as t-> In contrast, our formulation follows the Schrödinger bridge, which directly incorporates the condition y into 2, during the forward process. Accordingly, the SNR>> 0 at ( = T. which allows reducing training steps from IK to 50 (Figure [ and achieving faster convergence (Figure. 4. Additionally, Schrödinger Bridge and concatenation. As shown in Table, f the schrödinger bridge (SB) combined with the concatenation mechanism achieves the best results; however, SB alone is better than using the naive concatenation.
quantitative comparison between
Training Objectives. Previous work (Rombach et al. 2022) showed that learning the noise directly enhances the performance compared predicting the To- As, intuitively, predicting the difference X 24 Fo is easier than predicting the original complicated zo- However, this is not valid in our case due the new term y. which makes predicting the difference much harder, discussed in Section 2. Table[5]hhow the same conclusion reported in (Rombach et al. (2022), that predicting the difference leads better performance than Fo. where the FID score is 8.7 and 15.1, respectively. On the contrary, our method achieves almost the same performance regardless of the training objective, which is expected, as predicting the difference involves predicting zg. Eq.
A.7 HOW TO FUSE DIFFERENT STAGES EFFICIENTLY?
Tahle 7: Ablation study of the sketch mugmentation effect on the overall performance after fusing the abstract and the detailed stages.
Table 6: Systematic search for the hest stopping step for the condition truncation.
Cutont
DeVries Taylor 2013
Dropout Percentage
20 Siage
Ovemal RID
Pereesagee
FID.
Metrichups.co (e)
0
10
20
40
60
120
160
700
4)
8.6
16,
yMd stage HDD
61
7.4
7.9
86
999
10.4
10.8
11.2
b)
5-10
5:20
9.59
13.94
Overall FID
13.6
10.7
9.5
10.9
11.2
13.5
15.7
18.9
() 10:20
20-40
9.37
13.98
() 20-30
50-70
9.80
13.76
30-40
The reported FID scores for the 1st and the 2nd stages, in Figure 13 and Figure 15 respectively, for each stage separately. In other words, in Figure 15| row c. the 2nd stage achieves an 8.6 FID score when GT sketch is fed, which is obtained from PidiNet (Su et al.; (2021). However, when fed the generated sketch from the 1sr stage. the performance drastically dropped from 8.6 to 16.1 (almost doubled), as shown in Table 7 row a. due the domain gap between the generated and the GT sketches. To fill this gap. we explored two types of augmentations: 1) Cutout and Dropout augmentation. 2) Condition truncation augmentation.
Cutout and Dropout Augmentation. First, we explored the straightforward augmentation types, such as Cutout (DeVries & Taylor 2017) and Dropout augmentations. For the Cutout, we apply a kernel to randomly blackout patches in the sketch. Additionally, regarding the dropout augmentation, we randomly convert white pixels to black pixels, interpreted as dropping some white points from the sketch. As shown in Table 7 generally, augmentation leads to drop in the 2nd stage performance while helping fill the gap in the overall performance, as the FID score decreased from 16 to almost 14. However, shown in rows b-d, gradually increasing the amount of the applied augmentation does not help much, as the performance remains almost the same. However, the overall accuracy, i.e., FID, drops significantly from 14 to 18 when aggressive augmentation is applied.
Condition Truncation Augmentation. As shown in Table / the conventional augmentation tech- niques do not help much. Accordingly, explored another augmentation variant tailored for the
18

---

Published as a conference paper at ICLR 2025
diffusion models, ie., condition truncation (Ho et al. 2022). During training the 2nd stage, we apply a random Gaussian noise to the fed condition, i.e., the sketch. So now the 2nd stage is trained on noisy sketches instead of pure ones, which makes it more robust to the variations between the real sketches and the generated ones from the 1 stage. Typically, we progressively generate the sketch in T time steps; T -> O. However, This added noise could be interpreted generation (1 stage) at a particular step T + s. Consequently, we search for it during sampling to determine which step works best for the overall performance, shown in Table 6 In other words, the 2nd stage is trained on a wide range of perturbed sketches, e.g., pure sketch (s = 0), noisy one (0 s < T). and even pure noise (s T). The 1x stage trained for 200 steps, so 200 means we omit the sketch and feed pure noise. In contrast, s = 0 indicates that we feed the generated sketch as it to the 2nd stage. As shown in Table 6 following this truncation mechanism, the fusion the two stages is performed more efficiently, where the overall performance drops from 16.1 to 10.6.
stop the sketch
A.8 CONDITIONING MECHANISM ERROR VULNERABILITY
In this section, we will compare the two conditioning mechanisms, Concatenation (Concat) and Schrödinger Bridge (SB), in terms of the error vulnerability.
While SB+Concat shows superior performance in Table for conditional generation when the input condition is the ground truth (GT) its performance diminishes in the unconditional setup.
In the unconditional setup, y is generated from the previous stage rather than being a perfect GT. This introduces errors into the generated y, leading to error accumulation, as discussed in Appendix A.7 Consequently, the model's performance suffers when relying on concatenation for conditioning. This is intuitive as y is fed to the network in each denoising step f. Thus, the network emphasizes the y lot, making it more vulnerable to error accumulation.
To address this, we use only the Schrödinger Bridge (SB) as the conditioning mechanism for unconditional generation because of its demonstrated robustness against accumulated errors.
To this end, we tested a compromise approach by applying concatenation for 50% of the steps and then dropping the condition from concatenation by feeding zeros. The experiment is conducted on the LSUN-Churcher dataset. Table 9 shows that this trick improves performance compared to concatenation in all steps. However, still performs worse than using SB alone on generated sketches.
Accordingly, the choice of y-feeding strategy depends the setup:
Unconditional generation: Use only the Schrüdinger Bridge, as it exhibits superior robust- ness against noise in the input condition.
Conditional generation: Use a combination of Concatenation and Schrödinger Bridge, as shown in Table for optimal results.
Table 8: Hummn evaluation for real image editing capabilities between method and SEED-X 100 randomly sampled images from ColebHQ dataset.
Table 9: Comparison between different conditioning mechanisms regarding errrr vulnerability on the LSUN dataset.
Method SB
FID
Method
FID
6.6
SEED-X
2.6
SB+Concat ( (100% steps)
10.1
Ours
4.3
SB 3B+Concat (50% steps)
8.9
A.9
FUTURE WORK
In this section, we will discuss some possible future work that shows the versatility of ToddlerDiffu- sion and its potential to drive innovation across image, video, and 3D tasks.
Distill the knowledge from substage features for perception tasks. The intermediate outputs from Toddler Diffusion (e.g., the sketch stage) be reused strong geometric priors for perception tasks by distilling the knowledge into perception networks to enforce geometric or perceptual constraints, making them structure-aware and geometry- guided aligned (Zhao et al. 2023a).
Reusing Diffusion Features as High- and low-Frequency Representations. Toddler Diffusion's 's staged outputs inherently capture distinct frequency information, with the sketch stage emphasizing
19

---

Published as a conference paper at ICLR 2025
low-frequency global structures and geometry and the palette and RGB stages capturing high- frequency details like textures. These features can be repurposed into a dual-branch network (Bi et al.; Wei et al. 2024): one branch focusing on low- wwwfrequency representations for tasks requiring global context (e.g., semantic segmentation, object localization) and the other specializing in high-frequency details for tasks like texture classification, instance segmentation, boundary refinement. Integrating this knowledge into VLMs can improve performance and generalization for perception tasks.
Addressing Limitations in T2I Models. significant challenge for text-to-image (T21) models lies in handling complex prompts involving counting or spatial relations (Bakr et al. [023]] Ghosh al. 2024). Fixing these issues in the sketch or abstract stage will be easier and simpler due to its lightweight. In addition, the subsequent stages (e.g., sketch-to to-RGB) will remain unaffected and reusable. This approach allows targeted improvements without retraining/fnetuningg the entire pipeline, significantly reducing computational costs.
Modular Prompt Handling for T2I Models. Toddler Diffusion can simplify complex text prompts by decomposing them into modular components mapped to specific stages. For example, detailed prompt can be split into geometry-focused instructions for the sketch stage and color-specific details for the pretch-to-palette stage. In addition, we can add more stages and specific prompts such as depth, e.g., obj1 closer than obj2. This enables a more focused and interpretable generation, particularly for long, detailed prompts with intricate requirements.
Given a complex prompt: "A bustling city street during sunset with a red double-decker bus the right, a blue car parked on the left, people walking on the sidewalk, and sunlight casting long shadows on the ground." " will be split into:
For instance:
Prructure-Focusee prompt: "A city street with double-decker bus the right, car parked on the left, and people walking on the sidewalk." "
Palette-Focused prompt: "The double-decker bus is red, the car is blue, the sky shows sunset with shadows on the ground."
Video Generation in Abstract Domains. Generating videos directly in the RGB domain is com- putationally expensive and prone to inconsistencies. Toddler Diffusion can split the problem into geometry consistency and stylistic consistency. Specifically, it first generates videos in an abstract domain (e.g., sketches), ensuring geometric consistency across frames. Then, these sketches can be refined into RGB frames using another stage, Sketch-to- -orch- conditioned on reference frame for color and style consistency. This two-step process simplifies the complex task of maintaining temporal and stylistic coherence in video generation.
Spatial Decompositionnality An intriguing future direction is to adapt Toddler Diffusion to cascade generation across spatial dimensions instead of modalities:
Object-Level Cascading: Generate foreground objects first, followed by backgrounds, enabling fine-grained control for game development or storyboarding applications.
Part-Level Cascading:
Decompose objects into parts (c.g.. wheels, body, interior) for industrics like automotive design and industrial manufacturing, where precise control at the component level is critical.
Enhancing 3D Generation. Toddler Diffusion can streamline 3D generation by reusing the sketch stage for geometry, followed by a depth estimation stage, and finally, lifting 2D geometry into full 3D representations. This staged approach is particularly valuable for AR/VR scene creation, autonomous driving simulations, etc. Additionally, we can first generate the layout, followed by another stage that generates the meshes (Koo et al. 2023).
Integration with NeRFs. The sketch stage can provide strong structural priors for Neural Radiance Fields (NeRFs), improving 3D scene reconstruction from sparse views. This integration could accelerate NeRF optimization and enhance generalization in applications like scene editing and AR/VR reconstruction, where structural integrity is critical.
Intrinsic Image Decomposition. Toddler Diffusion's staged approach can be extended tasks like intrinsic image decomposition, where images are disentangled into reflectance, illumination, and shading components. This decomposition enables applications like relighting, material editing, and realistic AR/VR rendering, where physically consistent outputs are crucial.
20

---

Published as a conference paper at ICLR 2025
B MORE QUALITATIVE RESULTS
B.1 DEMO
Please refer to our demo which probes our controllability capabilities performing consistent edits.
Our demo demonstrates lot of use cases, including:
(0:00 -> 0:20) [Ist stage] Starting from noise generates a sketch by running only the Ist stage, which is unconditional generation.
(0:20 - 0:24) [2nd stage] Starting from a generated sketch generates palette. This one considered unconditional generation noise generates the sketch unconditionally. (0:24 -> 0:32) [3rd stage] Starting from generated palette generates an RGB image. This
one also considered unconditional generation.
(0:32 -> 0:44) [Conditional Generation] Starting from the GT sketch, we generate an RGB
image.
(0:44 -> 1:15) [Editing] We show our sketch-based editing capabilities.
(1:15 -> 1:46) [Editing] We show our palette-based editing capabilities.
B.2 REAL IMAGE EDITING
we ensure consistent edits for real images by storing the noise used during the generation process for later edits. The primary challenge with real image editing is that don't have access to the noise values at each denoising step, as our model didn't 't generate the image. To address this, we employ DDPM-inversion (Fuberman-Spiegelglia intermediate noise. As illustrated in Figure T6), we compare our model against SEED-X (Ge et al. 2024). and our model achieves consistent edits on real reference images better than SEED-X. XX despite being trained on much fewer data compared to SEED-X and using smaller architecture. Additionally, SEED-X fails to perform the edits correctly and keeps the rest of the image unedited, such as the rest of the face or the background. On the contrary, our method shows great faithfulness to the original image while performing the edits depicted the sketch. We have created a small test-set for editing where we manually designed 100 edits on CelebHQ and generated the corresponding edits for both our method and SEED-X. Then, we asked annotators (4 annotators) to give a score based on the aestheticism of the generated image and the adherence to the edit command. As shown in Table 8 the annotators scores show great preferences to our method over SEED-X where our score 4.3 while SEED-X is only 2.6
As explained in Section 2.4
al.) 2024) reconstruct the
21

---

Published as a conference paper at ICLR 2025
Input
Inversion
Sketch
Edited Sketch
Output
Ours
Input
Text Condition
Output
Close the man's mouse
SEEDA X
Put wrinkles on his forehead
Remove the bangs from the woman's hair
Figure 16:
Comparison between our approach, ToddlerDifiusion and SEED-X terms of the real images capabilities.
Ge al. 2024 in
B.3 CLASS-LABEL ROBUSTNESS
Beyond these quality comparisons, we also conducted a robustness analysis to assess our model's behavior when faced with inconsistencies between conditions. Specifically, we deliberately provided incorrect class labels for the same sketch. This mismatched the sketch (geometry) and the desired class label (style). Despite the inconsistent conditions, illustrated by the off-diagonal images in Figure [7] our model consistently adheres to the geometry defined by the sketch while attempting to match the style of the misassigned category, demonstrating the robustness of our approach. Successful
22

---

Published as a conference paper at ICLR 2025
cases highlighted in yellow, and even in failure cases (highlighted in red), model avoids catastrophic errors, never hallucinating by completely ignoring either the geometry or the style.
Tarantula
Ptarmigan
Spotted
Macaw
Kite Bird
Cacatua Galerita
Tree Frog
Salamander
Figure 17: Our results on the ImageNet dataset. The first column depicts the input sketch. The
diagonal (green) represents the generated output that follows the sketch and the GT label. The off-diagonal images show the output given the sketch and inconsistent label. These adversarial labels show our model's 's robustness, where the successful cases are yellow, and the failure ones are red.
B.4 EDITING CAPABILITIES
Our approach introduces interpretable generation pipeline by decomposing the complex RGB generation task into series of interpretable stages, inspired by the human generation system (Stevens) 2012 Marr 2010 1974). Unlike traditional models that generate the complete image in one complex stage. we break it into N simpler stages, starting with abstract contours, then an abstract palette, and concluding with the detailed RGB image. This decomposition not only enhances interpretability but also facilitates dynamic user interaction, offering unprecedented editing capabilities for unconditional generation, shown in Figure 18 Figure 19 Figure 20) and Figure 21
23

---

Published as a conference paper at ICLR 2025
A
E
Sketch
Sample
Sample le 2
Sampau 3
Figure 18: Controllability ability of our framework, ToddlerDiffusion. Starting from generated sketch and RGB image (A), we can remove artifacts or undesired parts, in red, (B), add a new content, in yellow, (C-D), and edit the existing content, in green, (E-F) by manipulating the sketch.
24

---

Published as a conference paper at ICLR 2025
Sketch
Sample
Sampln
Sample 3
Sample le 4
Sample
Figure 19: Controllability ability of our framework, DofuuirD Starting from generated sketch and RGB image (A), we can remove artifacts or undesired parts, red, (B), add a new content, in yellow, (C-D), and edit the existing content, in green, (E-F) by manipulating the sketch.
25

---

Published as a conference paper at ICLR 2025
Sketch
olla
Sample
Sample 2
Sample 3
hooterstt
Sample Temple-- 4
Sample 5
Figure 20: Controllability ability of our framework, TodulerDiffusion. Starting from generated sketch and RGB image (A), we can remove artifacts or undesired parts, red, (B), add a new content, in yellow, (C-D), and edit the existing content, in green, (E-F) by manipulating the sketch.
26

---

Published as a conference paper at ICLR 2025
Sketch
Semple 1
Sampau peeple
Sample 3
Sample le 4
Sample
Figure 21: Controllability ability of our framework, TodulerDiifusion. Starting from generated sketch and RGB image (A), we can remove artifacts or undesired parts, red, (B), add a new content, in yellow, (C-D), and edit the existing content, in green, (E-F) by manipulating the sketch.
B.5 SKETCH CONDITIONING (SD-v1.5)
We convert method "sketch2RGB" by omitting the 1et stage show controllability capabilities. By doing this, we can compare our model against the vast tailored sketch-based models built on top of SD-v1.5. However, these methods have been trained for a long time on large-scale datasets. Thus, for fair comparison, we retrain them using CelebHQ datasets for 5 epochs starting from SD-v1.5 weights. demonstrated in Figure 22 despite this not our main focus, our method outperforms conditioning methods without adding additional modules (Zhang et al. (0233 Zhao
ct al. 2024) or adapters (Mou et al. 2024). We compare training- free (Tumanyan et al. 2023 Parman al. 2023 and training-based (Li et al.) 2023b Zhang et al. 2023 Zhao et al. 2024) methods. The performance is worse than expected for the trained-base based methods due to two reasons: 1) We train
all the models for only 5 epochs; thus, some of this method, in contrast to us, needs more epochs to converge. 2) We apply augmentations to the input sketch, which shows that some of these methods vulnerable to sketch perturbations.
27

---

Published as a conference paper at ICLR 2025
Condition
3
PnP
Pix- Zeroshot
Control
Net
Control
GLIGEN
Figure 22: Comparison between our approach, TofdlsoDi and training-free (Tumanyan et al.
2023
Parmar al. 2023) and training-based (Li et al. 2023b Zhang al. 2023 Zhao al. 2024
sketch conditioning methods.
B.6 TEXT + SKETCH CONDITIONING (SD-V1.5)
We conducted another experiments to condition the model both text and sketch. To this end, we have extended the CelehHQ dataset by adding detailed text description for each image using Llava-Next v1.6. As Llava generates long descriptions, We train the model starting from the SDv1.5 weights for only five epochs. As shown in Figure 23 we feed the same sketch with different prompts, where the only difference between them the gender, then the model generates the RGB images that follow both the sketch and the text conditioning.
CelebHQ:
ask GPT-4o to summarize them.
Todeler Diffusion
The tusan tace, who i5 males le has short, dark hair and is wearing his eyes hight
Gender Editing
Diffusion
The image siowe e clase-op of lace, who fisulee, rt,
Light calerr
Figure 23: Text and Sketch conditioning capabilities of our proposed method.
28

---

Published as a conference paper at ICLR 2025
LionArt: Furthermore, conducted more challenging experiment condition the model on both text and sketch on a more challenging and diverse dataset, LionArt. To this end, we randomly sample 1 M images from LionArt and train model for only I epoch starting from SDv1.5 weights. As shown in Figure 24, our model generates RGB images that adheres to both conditions; the text prompt and the input sketch.
Maqui Pluot Raspberry Smoothie
2021 Indian Scout Scout Bobber - ABS at Ford Lauderda le
beautiful nature Norway natural landscape aerial footage lovatnet lake Live Acti on
Rainier Cherries with White Bowel, 5x7 oil on linen panel. Original fine art by Carolina Elizabeth
Picture of 1959 MG MGA Offered by E&R Classics PGST
A French press and mug filled with coffee
Hand Painted Pitcher, Pastel Flowers, Made in Portugal, Short and Wide
man wearing black cap with white patterns and light gray shirt, looking slightly to the side
Figure 24: Qualitative results for our Text-conditional generation on LionArt dataset.
29

---

Published as a conference paper at ICLR 2025
B.7 TEXT CONDITIONING
To show the versatility of approach, conducted experiments show capabilities in generating sketches from text on two different datasets: CUB200 and Dress Code dataset. The Caltech-UCSD Birds-200-2011 I (CUB-200-201) 1) dataset is the most widely-used dataset for fine- grained visual categorization task. It contains 11 11,788 images of 200 subcategories belonging to birds. Due to its importance, DM-GAN extends the dataset to include text description for each image. There is no GT paired sketches for each image, thus, run PidiNet generate sketch for each RGB image, Additionally, we show our capabilities on the Dress Code dataset, which is more than 3x larger than publicly available datasets for image-based virtual try-on and features high-resolution paired images (1024x768) with front-view, full-body reference models. Like CUB200, the Dress Code data is unimodal dataset that only contains RGB images. Fortunately, Ti-MGD extends the dataset to be a multi-modal dataset by having image-text pairs.
In this setup have two conditioning mechanisms: 1) The sketch y, which is handled by the forward process (Eq.2). 2) cross-attention layer to handle the text condition.
We have trained the 1s stage for IK epochs it is very small, only 5 M parameters, and the dataset scale is very small, thus the IK epochs takes less than 12 hours using single A A100 GPU. Then, we train the 2nd stage, 141 Million parameters, for only 200 epochs. In contrast, LDM takes longer converges as shown in Figure 4, thus we train it for 400 epochs.
we have achieved better accuracies on both datasets compared LDM despite being training for half the epochs. For qualitative results, please refer to Figures 25 and
As shown in Table 10 and Table I1]
26
Table Comparison between our approach 1.DM using eext conditioning multi-modal datsset, Dress Code (Baldrati et
Table 11: Comparison between our approach and LDM using text
conditioning multi modial dataset, CUR2200 (Zhu 2019.
Method
FID
Method
FID
LDM
12.2
LDM
10.7
Ours
10.8
Ours
9.1
30

---

Published as a conference paper at ICLR 2025
A-line print, long print dress, printed brown dress
Long floral print dress, red florat print, v-necked printed wrap dress
Black one-shoulder flunce,, ruffle dress, cold shoulder black dress
Pink pleat-free front pants, Pink putl- pants, Pink shorts
Blue lang sleeves, blue Knee length
Beige stripes, cream long striped dress
Long so black sleeveless, spaghetti stp black maxi dress
Black zip dress, black zip nock neck sweater dress, sleeve short black dress
Gray lace, silver grey short dress, white enbellished mesh shift
Body con tube dress, long brown halter dress, sleeveless ribbed dress
Red knee lenght shorts, rise, red pleat trouser
Asymnctric neckline, black white long dress
Blue sheer flounce dress, larre sleeves, long blue dress
Knee length dress, long Llevved black and white
Blue and has long sleeves, blue Long-s Leeve midi dress, Long Leeved blue dress
One shoulder, red maxi dress, solid red long
Figure 25: Qualitative results for our Text-conditional generation Dress Code dataset (Baldrati) al. 2024). The first column the input text prompt, second column the generated sketch, last column is the generated RGB image.
31

---

Published as a conference paper at ICLR 2025
This blue and black bird blue breast and small bill
small, dark- feathered bird perched on tree trunk
A dark-colored bird perched on thin branch, with clear blue sky
Light gray gull with orange
Black bird with g gosss,, sleek
A small bird with gray and white plunage
p plumage, with
pointed beak,
slight open, and
lonn tail feathers
A small blue bird with sharp beak, standing on natural surf
snall bird with dark gray, white, and red plunage, featuring dark cap.
small Ovenbird with olive-green upperparts, and distinctive eye- ring
slightly downward curved bill which has white point coming off it
The bird submerges in water and large with an even greyish black coloring
white qull with yellow beak, black legs, and grayish wings
striking red- headed woodpecker with a white belly, black-and-whit patterned wings
brown bird with a sleek body, Flosting water.
A glassy blackbird with iridescent feathers reflecting shades of green and blue
A dark-colored flight, captured over gray water
Figure 26: Qualitative results for our Text-conditional generation on CUB200 dataset (Zhu et al. 2019p. The first column input text prompt, second column is the generated sketch, last column the generated RGB image.
B.8 FREE-HAND SKETCH CONDITIONAL GENERATION SAMPLES
Figure 27 depicts more qualitative results for conditional generation pipeline, where first generate the sketch using drawing tools and then use the generated sketch to generate the RGB image. Despite the sketch being too bad and sparse, model still generates high-quality images.
32

---

Published as a conference paper at ICLR 2025
Successful
Successful Cases
Failure Cases
Cases
Figure 27: Qualitative results for our unconditional generation.
33

---

Published as a conference paper at ICLR 2025
B.9 QUALITATIVE RESULTS FOR OUR SKETCH-CONDITTINAL GENERATION
In addition to our unconditional generation, our framework can be used as a conditional generation pipeline, where the user can omit the first stage and directly fed a reference sketch. As shown in Figure 28 given the reference sketch, we run only the second stage, which responsible for generating the RGB image given a sketch. Compared generated images in Figure 28 are much better as the reference sketches are much better than the generated ones. This observation aliened with our discussion regarding the gap between the two stages. Therefore, improving the quality of the generated sketches will directly reflect on the overall images quality.
the generated images in Figure 27 the
Successful
Successful
Failure Cases
Cases
Cases
Figure 28: Qualitative results for our sketch-conditional generation.
34

---

Published as a conference paper at ICLR 2025
C
FAILURE CASES
The last column of Figure 27 and Figure 28 depicts failure cases, where the generated RGB is following the sketch but generates unrealistic image. Intuitively, the failure cases are more apparent in the unconditional generation due to the generated sketches quality.
35


Reviews:
[{"review_id": "e77bqu64u8", "reviewer": "Reviewer_xMYw", "summary": "This work proposes ToddlerDiffusion, a cascaded sequence of relatively small diffusion models, to gradually generate different aspects of images. Those different diffusion models are connected by the Schrödinger Bridge to improve conditioning consistency.", "questions": "- Could you please provide more details about how to incorporate ToddlerDiffusion into Stable Diffusion? For instance, what text prompts are used? Additionally, is Stable Diffusion fine-tuned on CelebHQ datasets for five epochs using x_0 prediction and the Schrödinger Bridge conditioning mechanism similar to ToddlerDiffusion? Is the network architecture of Stable Diffusion modified to include sketch conditions, as demonstrated in Figure 12? If so, then how are the parameters initialized in the modified components?\n\n- Why do the numbers of LDM parameters differ between Table 2 (263M) and Figure 13 (187M)?\n\n- Why are the schedulers for $\\alpha$ and $\\sigma$ different in Figure 12 (left) and (right)? Why does $\\alpha$ increase from $T$ to $0$ in the left block, but decrease from $T$ to $0$ in the right block? What does the presented $y$ represent in the first stage (Figure 12 left block) that unconditionally generates the sketch?\n\n- Why is SB+Concat not utilized in ToddlerDiffusion, while only SB is used, despite the former demonstrating superior performance in Table 3?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **The proposed cascaded framework is reasonable and effective**. With fewer network parameters and training time, ToddlerDiffusion can outperform the standard LDM in both generation quality and conditioning consistency.\n\n2. **Thorough experiments and ablation studies are conducted** in the main content and appendix to demonstrate the versatility of the proposed framework and validate the effectiveness of several design choices.", "weaknesses": "1. **Presentation of this work requires thorough improvements.**.\n- The authors should use `\\citep` for most cases in the manuscript, which places the authors' names and the year in parentheses. The current use of `\\cite` makes the manuscript cluttered and difficult to read.\n- The manuscripts contain many incoherent parts. For instance, in Lines 474-479, the authors state: \"we used x_0 ... and employed LoRA fine-tuning for efficiency. **But** these two variants are crucial to make it work\". The use of the transitional word \"but\" in this context is confusing. Later, the authors assert that \"we **must** fine-tune the entire model instead of using LoRA\". Then why is LoRA fine-tuning described as crucial if it is not the recommended approach?\n- Notations are inconsistent in the manuscript. For example, in Equation (2), $x_t=\\alpha_t x_0^i+(1-\\alpha_t)y^i+\\sigma_t^2 \\epsilon_t$, whereas in Equation (6)~(16), $x_t=(1-\\alpha_t) x_0+\\alpha_t y+\\sigma_t \\epsilon_t$.\n- What is the meaning of \"$:\\sigma_t^2=\\alpha_t-\\alpha_t^2$\" in Equation (2)?\n\n2. **Many statements in this work are incorrect or overclaimed**.\n- In the abstract: \"surpassing Stable-Diffusion (LDM) performance\". However, all experiments conducted in the main content are compared with the LDM trained on small datasets, which are NOT Stable Diffusion (v1/v2/v3).\n- The training objective of LDM or Stable Diffusion is the noise $\\epsilon$. However, $\\epsilon$ is NOT $x_t-x_0$ in LDM as stated in Line 266 and Table 4.\n- In Line 507-509, by the SNR definition in this work, SNR does not tend to 0 as $t \\rightarrow T$, as the concatenated condition $y$ also provides some signals to a certain extent similar to the proposed framework even at $t=T$. SNR should NOT be the reason that distinguishes between different conditioning methods (Concatenation v.s. Schrödinger Bridge).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTHLqsLxLn", "reviewer": "Reviewer_dsQk", "summary": "This paper introduces a generative framework comprising three stages: noise to sketch, sketch to palette, and palette to RGB images. The paper uses the Schrodinger bridge I2SB [1] to train these stages. This framework takes less training time and samples efficiently. Moreover, the architecture is smaller and has better training convergence. This framework could be used for editing and control generation.\n\n[1]: I2SB: Image-to-Image Schrödinger Bridge", "questions": "My biggest concern is about the novelty of this work since the framework could be considered as many conditional generative models. Secondly, I am not sure about the ability to extend this framework to text-to-image generation; this could limit the applicability of the proposed method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. I particularly like the idea of the \"toddler,\" which is motivated by how the child grows up and learns everything.\n2. This training framework is more training/sampling efficient than unconditional diffusion generation. It could be used for editing and control generation since it comprises many conditional generative problems trained by Schrodinger Bridge.\n3. The experiment is well-organized.", "weaknesses": "1. The methodology seems simple since it decomposes the generative problem into subsequent conditional generative problems. Each problem can be solved directly using LDM or Schrodinger Bridge, and the paper uses Schrodinger Bridge to solve them like I2SB. Therefore, this paper's novelty is limited since it is just like you apply I2SB directly to solve conditional generative problems.\n2. The paper mentions extending the method to Stable Diffusion but omits stage 1. This is a meaningful editing and controllable generation problem. However, for the text-to-image generation, I doubt the ability to train stage 1 mapping from (noise and text) to sketch using the Schrodinger Bridge. The reason behind this is that text is a completely different modality with RGB, sketch, and palette. Therefore, mapping from (noise and text) to sketch is not easily solved by unconditional Schrodinger Bridge. Further, the architecture should be larger and specifically designed for that task. This is an interesting question that should be discussed in the paper to answer the framework's scalability. The author could try to train such (noise and text) to sketch models on small text-to-image datasets such as CUB-200 or COCO dataset. [2] (see this work for small text-to-image problem)\n3. The paper should include citations for [1] since their methodology is similar to the paper method. I2SB uses the Schrodinger bridge for image-image problems, which are the same problem as your decomposed conditional generative problems. The only difference is they solve image superresolution/inpainting/deblurring instead of sketch-to-palette and palette-to-image.\n4. Minor: Several typos at Eq. 3 (the right-hand equation label should be sketch and palette)\n\n[1]: I2SB: Image-to-Image Schrödinger Bridge\n[2]: Vector Quantized Diffusion Model for Text-to-Image Synthesis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oS8RFT1MhO", "reviewer": "Reviewer_rf6t", "summary": "The paper introduces Toddler diffusion, a method to generate images like the thought process of a child. The paper decomposes the image generation process into three stages first on a high level sketch generation, then a colour palette generation and finally generating a complete RGB image. To enable learning from one modality to another, the authors leverage Schrödinger Bridges. Such a form a training enables capabilities of seamless editing into the model. Moreover also enables faster sampling compared to conventional LDMs. The authors perform comparisons with conventional LDMs to illustrate the effectiveness of the method.", "questions": "1. Minor spelling mistake Ln 46  emerging proprieties \n\n2. Regarding the problem prescribed in section 2.1 Ln 216,217 : [1] provides some suggestions on how to schedule the noise process to achieve zero terminal SNR and fix the inference schedule. I refer the paper here for the reference of the authors\n\n   [1] Lin, S., Liu, B., Li, J. and Yang, X., 2024. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp. 5404-5411).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The proposed framework of a multistep generation process is novel to the best of my knowledge. The method is motivated by an inspiring thought process based on the workflow of how humans think to create a picture.\n2. The disentangled training process also enables enhanced controllability, stability and faster training and inference process. Moreover such a generation process also gets benefits of added diversity because of the compositional effect in each stage\n3. The performance boost over naive LDMs is substantial while also enabling zero shot capabilities to the the model.\n4. The method achieves better performance than LDMs across diverse datasets while showing better training and inference efficiency.", "weaknesses": "1. The training process of toddler diffusion is not clear from the main manuscript. Are different U-Nets leveraged from different modality transformations. I would advise the authors to include a flowchart diagram of signal flow in the main manuscript or alternatively add a subsection detailing this aspect.\n2. Toddler diffusion utilizes Schrödinger bridges and an SDE perspective for the training and sampling process. However Rectified Flows have shown to achieve much better performance utilizing ODEs, which also improves further distillation performances. Can the authors contrast on the benefits of training and sampling using diffusion  Schrödinger bridges and rectified flows.\n3. Toddler diffusion requires annotated data in different modalities to enable conditional generation. Hence I believe the comparisons provided by the authors by comparing with Naive LDMs are not fair, for a fairer evaluation. I believe the authors need to train a diffusion model which can generate sketch from noise, then 2 conditional models for the translation tasks. Could the authors please provide a comparison with LDMs trained in the proposed manner or alternatively comment on how the current experiments are valid comparisons\n4. Can the authors also include performance of Toddler diffusion compared to pixel space diffusion architectures and diffusion transformers for the same datasets. Some valid methods would be SiT[1], DiT[2], Pixel space models[3,4]\n\n   [1] Ma, N., Goldstein, M., Albergo, M.S., Boffi, N.M., Vanden-Eijnden, E. and Xie, S., 2024. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740.\n\n   [2] Peebles, W. and Xie, S., 2023. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4195-4205).\n\n   [3] Kingma, D. and Gao, R., 2024. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36.\n\n   [4] Kynkäänniemi, T., Aittala, M., Karras, T., Laine, S., Aila, T. and Lehtinen, J., 2024. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GvQPPJHBqE", "reviewer": "Reviewer_byq3", "summary": "This paper proposes a new approach to decompose the diffusion framework into simpler and interpretable stages. The new framework called ToddlerDiffusion is a cascaded model that introduces two additional intermediate representations, sketch and palette. Different modality-specific models are connected to obtain high-quality image outputs with detailed textures. Instead of naive LDM, the Schrödinger Bridge is leveraged to determine the optimal transformation between modalities. The new design achieves better overall performance with faster sampling and smaller architecture compared with LDM, and also enables editing and interaction at intermediate steps.", "questions": "1.\tGiven ToddlerDiffusion is a smaller framework compared with LDM, showing it converges faster and has better performance with the same number of epochs is not sufficient to prove it’s a better model. Table 2 claims the training converges at epoch 400 (not mentioning on which dataset), but in Table 1 none of the datapoints was reported at epoch 400. Showing that ToddlerDiffusion outperforms when both training procedures converge would be a stronger evidence.\n2.\tPlease provide more details about the experiments, such as but not limited to: at which resolutions were the three stages trained and evaluated? How was controlnet/SDEdit configured when the editing capacities was compared? Is there any quantitative results supporting it’s editing capacity?\n3.\tIt’s understandable that The generation quality of ToddlerDiffusion is inferior to that of the SOTA model (SDXL, Flux.1, etc.) given limited data and training. However, The use cases for low-resolution unconditional or label-conditioned generation are very limited. Would the first two stages be able to handle more complicated conditions at higher resolutions as in latest models? If so, would the framework still be efficient?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe framework natively offers decent capability in editing and interpretability without additional adapters or fine-tuning.\n2.\tBy decomposing the image generation into three simpler sub-tasks, the framework has fewer parameters, converges faster, and is more robust against a smaller number of sampling steps.\n3.\tThe framework shows its capacity to achieve better performance on given datasets comparing with naive LDM.", "weaknesses": "1.\tThere is a large gap between the performance of the baseline model (LDM/SD1.5) and the current state-of-the-art models.\n2.\tThe experiments showed the model’s capability in limited training data and resources, however, the value of proving a smaller model converges faster is limited. There’s lack of the evidence about its performance compared with baselines given sufficient data and training. \n3.\tThe description of the experiments lacks many details that reduced the robustness of the conclusions. For example, the experiments comparing editing capabilities with controlnet did not mention which base model was used by controlnet or at which resolution was the experiment performed, makes the fairness of the experiment questionable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)