ToddlerDiffusion: Interactive Structured Image Generation with Cascaded Schrödinger Bridge

## Abstract

Abstract Diffusion models break down the challenging task of generating data from high-dimensional distributions into a series of easier denoising steps. Inspired by this paradigm, we propose a novel approach that extends the diffusion framework into modality space, decomposing the complex task of RGB image generation into simpler, interpretable stages. Our method, termed ToddlerDiffusion, cascades modality-specific models, each responsible for generating an intermediate representation, such as contours, palettes, and detailed textures, ultimately culminating in a high-quality RGB image.
Instead of relying on the naive LDM concatenation conditioning mechanism to connect the different stages together, we employ Schrödinger Bridge to determine the optimal transport between different modalities.
Although employing a cascaded pipeline introduces more stages, which could lead to a more complex architecture, each stage is meticulously formulated for efficiency and accuracy, surpassing Stable-Diffusion (LDM) performance.
Modality composition not only enhances overall performance but enables emerging proprieties such as consistent editing, interaction capabilities, high-level interpretability, and faster convergence and sampling rate.
Extensive experiments on diverse datasets, including LSUN-Churches, ImageNet, CelebHQ, and LAION-Art, demonstrate the efficacy of our approach, consistently outperforming state-of-the-art methods.
For instance, ToddlerDiffusion achieves notable efficiency, matching LDM performance on LSUN-Churches while operating 2 × \times × faster with a 3 × \times × smaller architecture.
The project website is available at: h t t p s : / / t o d d l e r d i f f u s i o n . g i t h u b . i o / w e b s i t e / https://toddlerdiffusion.github.io/website/ italic_h italic_t italic_t italic_p italic_s : / / italic_t italic_o italic_d italic_d italic_l italic_e italic_r italic_d italic_i italic_f italic_f italic_u italic_s italic_i italic_o italic_n . italic_g italic_i italic_t italic_h italic_u italic_b . italic_i italic_o / italic_w italic_e italic_b italic_s italic_i italic_t italic_e /

## 1 Introduction

In recent years, diffusion models Sohl-Dickstein et al. (); Ho et al. (); Song et al. () have made significant strides across diverse domains, revolutionizing image synthesis and related tasks by transforming noisy, unstructured data into coherent representations through incremental diffusion steps Ho et al. (); Preechakul et al. (); Dhariwal & Nichol (); Nichol & Dhariwal ().
Their versatility extends beyond image generation to tasks such as image denoising Gong et al. (); Kulikov et al. (); Zhu et al. (), inpainting Lugmayr et al. (); Alt et al. (), super-resolution Li et al. (); Gao et al. (), and applications in 3D content creation Anciukevičius et al. (); Qian et al. (); Poole et al. (), data augmentation Carlini et al. (); Li et al. (); Voetman et al. (), medical imaging Chung et al. (); Kazerouni et al. (); Wu et al. (); Wolleb et al. (), anomaly detection, and more.
Diffusion models, such as Denoising Diffusion Probabilistic Models (DDPMs) Ho et al. (), draw inspiration from the diffusion process in physics, breaking down the intricate task of generating data from high-dimensional distributions into a sequence of simpler denoising steps.
In this framework, instead of generating an image in a single step, the task is reformulated as a gradual denoising process, where a noisy image is iteratively refined until the final, detailed image emerges. This step-by-step approach leverages the power of decomposition: by solving a series of simpler tasks, we can ultimately solve the original, more challenging problem.

Figure: Figure 1: Trajectory comparison. We compare our approach (Top) against LDM (bottom) in terms of trajectory. Instead of starting from pure noise such as LDM, our approach, ToddlerDiffusion, leverages the intermediate stages to achieve a more steady and shorter trajectory.
Refer to caption: x1.png

Inspired by this philosophy, we propose ToddlerDiffusion, a novel method extending this decomposition principle beyond just noise removal.
Instead of tackling the image generation problem in one go, we decompose it into modality-specific stages.
Each stage focuses on generating a different aspect or modality of the image, such as abstract contours or sketches, followed by color palettes, and finally, the detailed RGB image.
This approach mimics the learning process of a child developing artistic skills Stevens (); Marr (; ), where the child first learns to draw basic outlines, then adds colors, and finally fills in the details.
We use the term “ToddlerDiffusion” to reflect this gradual, step-by-step learning and generation process.

Despite the intuitive appeal of task decomposition, it introduces significant challenges.
More stages mean more parameters, longer training times, and potentially slower sampling speeds.
However, our hypothesis is that by addressing simpler tasks at each stage, we can leverage smaller architectures tailored to each modality, resulting in a more compact overall model compared to traditional single-stage architectures.
Moreover, multi-stage frameworks often suffer from error accumulation Ho et al. (); Huang et al. (), where inaccuracies in earlier stages can propagate and amplify through subsequent stages.
However, we showed that by using simple techniques such as condition truncation dropout, we can significantly mitigate the error accumulation, Appendix A.6
Additionally, learning these intermediate representations, e.g., counters, explicitly or implicitly is challenging, where the first requires paired data and the latter requires special complicated designs Singh et al. (); Li et al. ().
To tackle the guidance challenge, we leverage pseudo GT and show that this noisy GT is enough to learn robust intermediate modalities.

To implement this framework, we considered two approaches.
The first is the standard Latent Diffusion Model (LDM) Rombach et al. () conditioning mechanism, using techniques such as cross-attention and concatenation to link stages.
Given our focus on spatial modalities like sketches, concatenation seemed more appropriate.
However, this method proved insufficient for achieving our hypothesis due to its weak conditioning, which results in poor information transfer between stages.
We instead adopt a more principled approach by employing the Schrödinger Bridge Wang et al. (); Li et al. (), a method from stochastic optimal transport theory, to determine the optimal path between modalities at each stage.
Unlike simple concatenation, which starts each stage from a state of pure noise, the Schrödinger Bridge enables us to begin each stage from a meaningful signal with a higher Signal-to-Noise Ratio (SNR) Kingma et al. (), as illustrated in Figure 1.
This leads to more effective and stable progression through the generation process, significantly reducing the denoising steps required during training and testing.

This structured, multi-stage approach not only allows for a more efficient and stable generation process but also enables a range of emerging capabilities. ToddlerDiffusion supports consistent editing of both generated and real images and facilitates user interaction in both conditional and unconditional settings, as shown in Figure LABEL:fig_teaser_fic.
Moreover, while sampling efficiency was not our primary objective, our approach yields faster sampling and training convergence due to the shorter trajectories between stages (Figure 1), resulting in fewer denoising steps and less computational cost (Figure LABEL:fig_teaser_fic).
For instance, LDM is trained using 1k steps; in contrast, ToddlerDiffusion could be trained using only 10 steps with minimal impact on generation fidelity.
To the best of our knowledge, ToddlerDiffusion is the first work to successfully integrate and achieve advancements across multiple domains, including controllability, consistent editing, sampling and training efficiency, and interactivity, within a unified framework, as shown in Figure LABEL:fig_teaser_fic.
Our extensive experiments on datasets such as LSUN-Churches Yu et al. (), ImageNet, and CelebA-HQ Karras et al. () demonstrate the effectiveness of this approach, consistently outperforming existing methods.
ToddlerDiffusion not only redefines the image generation process but also paves the way for future research into more structured, interpretable, and efficient generative models.

Our contributions can be succinctly summarized as follows:

- •
We introduce a cascaded structured image generation pipeline, denoted as ToddlerDiffusion, that systematically generates a chain of interpretable stages leading to the final image.
- •
Leveraging the Schrödinger Bridge enables us to train the model from scratch using a minimal number of steps: 10 steps only, in addition to, performing on the bar compared to LDM while being 2$\times$ faster, using 3$\times$ smaller architecture.
- •
ToddlerDiffusion provides robust editing and interaction capabilities for free; without requiring any manual annotations.
- •
Introduce a novel formulation for sketch generation that achieves better performance than LDM while using 41$\times$ smaller architecture.

## 2 ToddlerDiffusion

We introduce ToddlerDiffusion, which strategically decomposes the RGB image generation task into more straightforward and manageable subtasks, such as sketch/edge and palette modalities.
In Section 2.1, we detail the formulation of each stage. By breaking down the generation process into simpler components, our model not only allows for a more compact architecture but also enables dynamic user interaction with the system, as illustrated in Figure LABEL:fig_teaser_fic.
Notably, these advancements are achieved without relying on manual ground truth annotations, instead utilizing human-free guidance for enhanced efficiency and practical applicability (Section 2.2).
Lastly, we demonstrate how our method facilitates robust, consistent edits across stages (Section 2.4).

### 2.1 Toddler Growth

The development of our method is inspired by child’s growth, where the learning process in their brains is decomposed into stages and developed step-by-step Stevens (); Marr (; ).
By analogy, we decompose the 2D image synthesis into different, more straightforward stages.
We employ the sketch and palette as intermediate stages, resulting three cascaded stages approach.
However, our approach is generic and seamlessly we can add or remove intermediate steps, based on the application needs.
Mainly, the pipeline can be clustered into two main modules:
1) The unconditional module, that generates the first modality, e.g., sketches, based on pure noise.
2) The conditional module, that condition on the previous modality to generate the next intermediate modality, e.g., palette, or the final RGB image.
Both modules can be implemented using Latent Diffusion Models (LDM), where unconditional LDM will start from noise to generate sketches, then by employing the concatenation conditioning mechanism we can condition on our spatial intermediate modalities; sketches and palettes.
We instead adopt a more principled approach by employing the Schrödinger Bridge to determine the optimal path between modalities at each stage.
Unlike simple concatenation, which starts each stage from a state of pure noise, the Schrödinger Bridge enables us to begin each stage from a meaningful signal with a higher Signal-to-Noise Ratio (SNR).

Schrödinger Bridge.
The Schrödinger Bridge problem arises from an intriguing connection between statistical physics and probability theory.
At its core, it addresses the task of finding the most likely stochastic process that evolves between two probability distributions over time, $\rho_{0}$ and $\rho_{T}$, governed by Brownian motion, while minimizing the control cost.
Mathematically, it could be written as:
$P^{*}=\underset{P\in\mathcal{P}}{\text{argmin}}\,\text{KL}(P\,||\,W)$, where $P$ is the probability path that minimizes the Kullback-Leibler (KL) divergence with respect to a reference Brownian motion $W$, and subject to marginal constrains; $P_{0}=\rho_{0}$ and $P_{T}=\rho_{T}$.
The process can be described as a stochastic differential equation (SDE):

$$ $dX_{t}=v(t,X_{t})\,dt+\sigma\,dW_{t},$ (1) $$

where $X_{t}$ represents the state at time $t$, $v(t,X_{t})$ is the optimal drift that minimizes the KL divergence and $dW_{t}$ is the Wiener process (Brownian motion).

Conditional Module.
By analogy, the two probability distributions $\rho_{T}$ and $\rho_{0}$ represent the distributions of features from simpler (sketch) and more complex (RGB) modalities, respectively.
In this context, the goal of the Schrödinger Bridge is to find the optimal stochastic process that transforms the simpler modality (e.g., sketch) into the more complex one (e.g., RGB image) while minimizing a divergence measure like KL divergence while respecting the boundaries.
Given a condition $y\in\mathbb{R}^{H\times W\times 3}$, that could be sketch, palette, or any other modality, we define the forward process as an image-to-image translation and formulate the problem using the discrete version of schrödinger bridge as follows:

$$ $x_{t}=\alpha_{t}x^{i}_{0}+(1-\alpha_{t})y^{i}+\sigma^{2}_{t}\epsilon_{t},\quad \quad:\sigma^{2}_{t}=\alpha_{t}-\alpha^{2}_{t},$ (2) $$

where $i$ is the stage number, $x_{0}$ and $y$ are the boundaries of the bridge, $\sigma^{2}_{t}$ controls the noise term, and $\alpha_{t}$ blends the two domains.
In the SDE, the term “$v(t,X_{t})$” is the deterministic drift, while “$dW_{t}$” represents the diffusion term.
In contrast, we follow the DDPM formulation, which works in the discrete space.
Therefore, by analogy, the terms “$\alpha_{t}x_{0}+(1-\alpha_{t})y$” and “$\sigma^{2}_{t}\epsilon_{t}$” represent the drift and diffusion terms, respectively.
$\sigma^{2}_{t}$ represents uncertainties, following a sphere formulation that constructs a bridge between the two domains.
Whereas, at the edge of the bridge, we are fully confident we are in one of the distributions $\rho_{T}$ and $\rho_{0}$, thus $\sigma^{2}_{t}=0$.
In contrast, the uncertainties increase while moving away from the edges, reaching their maximum value in the middle between the two domains, forming the bridge shape.
Eq. 2 forms the skeleton for $2^{nd}$ and $3^{rd}$ stages, where:

$$ $x^{i}_{0}=\begin{cases}x^{p}_{0}=\mathcal{F}_{p}(x^{rgb}_{0})&:\dot{\imath}=2 \quad(\text{Palette})\\ x^{rgb}_{0}&:\dot{\imath}=3\quad(\text{Detailed Image})\end{cases}\quad\quad \quad\quad y^{i}=\begin{cases}x^{s}_{0}=\mathcal{F}_{s}(x^{rgb}_{0})&:\dot{ \imath}=2\quad(\text{Palette})\\ x^{p}_{0}=\mathcal{F}_{p}(x^{rgb}_{0})&:\dot{\imath}=3\quad(\text{Detailed Image})\end{cases}$ (3) $$

where $i$ is the stage number, $\mathcal{F}_{p}$ and $\mathcal{F}_{s}$ are palette and sketch functions that generate the pseudo-GT, respectively, detailed in Section 2.2.

Unconditional Module: $1^{st}$ Stage (Sketch).
This stage aims to generate abstract contours $x^{s}_{0}\in\mathbb{R}^{H\times W}$ starting from just pure noise or label/text conditions.

Figure: Figure 2: Comparison between different formulations for the $1^{st}$ stage. This depicts the forward process for each formulation.
Refer to caption: x2.png

In both cases, we name this module unconditional as it does not condition on a previous modality because it will generate the first modality, the sketch.
One possible solution is to utilize the original LDM, which is not aligned with the sketch nature, as shown in Figure 2, case A.
More specifically, we are starting from a pure noise distribution $\mathbf{x}_{T}$ and want to learn the inverse mapping $q\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t}\right)$ to be able to sample real data $x^{s}_{0}$.
The issue relies upon the vast gap between the two domains: $x_{T}$ and $x_{0}$.
Following Kingma et al. (), this can be interpreted as signal-to-noise ratio (SNR), where SNR($t$)=$\frac{\alpha_{t}}{\sigma^{2}_{t}}$.
Consequently, at the beginning ($t=T$), $\alpha_{T}$ yields to 0, therefore SNR($T$)=0.
On the contrary, to fill this gap, we propose a unique formulation tailored for sketch generation that adapts Eq. 2.
Specifically, we introduced two changes to Eq. 2.
First, we replaced the Gaussian distribution $\epsilon_{t}$ with a discretized version, the Bernoulli distribution, as shown in case B, Figure 2.
However, this is not optimized enough for the sparsity nature of the sketch, where more than 80% of the sketch is black pixels.
Thus, we set our initial condition to $y=\mathcal{F}_{d}(x^{s}_{0})$, where $\mathcal{F}_{d}$ is a dropout function that takes the GT sketch $x^{s}_{0}$ and generates more sparse version.
To align our design with the sketch nature, as shown in Figure 2, part C, we changed $\sigma^{2}_{t}$ to follow a linear trend instead of the bridge shape by setting the variance peak at $x_{T}$.
We have to make sure to set the variance peak to a small number due to the sparsity of the sketch.
This could be interpreted as adding brighter points on a black canvas to act as control points during the progressive steps ($T\rightarrow 0$) while converging to form the contours, as shown in Figure 2, part C.
More details about the sparsity level importance are shown in Section 3.5.

### 2.2 Toddler Guidance

A crucial factor for the success of our framework, ToddlerDiffusion, is obtaining accurate and human-free guidance for the intermediate stages.
Two modules, $\mathcal{F}_{s}$ and $\mathcal{F}_{p}$, are employed to generate contours/sketches and palettes, respectively.

Sketch.
The first module, $x^{s}_{0}=\mathcal{F}_{s}(x^{rgb}_{0})$, serves as a sketch or edge predictor, where $x^{rgb}_{0}\in\mathbb{R}^{H\times W\times 3}$ is the ground-truth RGB image, and $x^{s}_{0}\in\mathbb{R}^{H\times W}$ is the generated sketch.
For instance, $\mathcal{F}_{s}$ could be PidiNet Su et al. () or Edter Pu et al. () for sketch generation, or Canny Canny () and Laplacian Wang () edge detectors.

Palette.
The palette can be obtained without human intervention by pixelating the ground-truth RGB image
$x^{p}_{0}=\mathcal{F}_{p}(x^{rgb}_{0})$,
where $x^{p}_{0}\in\mathbb{R}^{H\times W\times 3}$ is the palette image, and $\mathcal{F}_{p}$ will be the pixelation function.
However, we introduce a more realistic way to have high-quality palettes for free by first generating the segments from the RGB image using FastSAM Zhao et al. (). Then, a simple color detector module is utilized to get the dominant color, e.g., the median color per segment.
This design choice is further analyzed in Section 3.5.
More details are mentioned in the Appendix.

### 2.3 Training Objective & Reverse Process

Training Formulation.
We adapt the conventional diffusion models’ learning objective Ho et al. (); Sohl-Dickstein et al. (), i.e., Variational Lower Bound (ELBO), to include our new condition $y$, whereas, each marginal distribution has to be conditioned on $y$, as follows:

$$ $\mathcal{L}_{ELBO}=-\mathbb{E}_{q}\big{(}D_{KL}(q(x_{T}|x^{i}_{0},y^{i})||p(x_ {T}|y^{i}))+\sum^{T}_{t=2}D_{KL}(q(x_{t-1}|x_{t},x^{i}_{0},y^{i})||p_{\theta}( x_{t-1}|x_{t},y^{i}))-\log p_{\theta}(x^{i}_{0}|x_{1},y^{i})\big{)},$ (4) $$

where $p_{\theta}$ is a function approximator intended to predict from $x_{t}$.
Using Bayes’rule, we can formulate the reverse process as follows:

$$ $q\left(x_{t-1}\mid x_{t},x^{i}_{0},y^{i}\right)=\mathcal{N}\left(x_{t-1}; \tilde{\mu}\left(x_{t},x^{i}_{0},y^{i}\right),\tilde{\sigma}^{2}_{t}\mathbf{I}\right)$ (5) $$

$$ $\tilde{\mu}_{t}(x_{t},x^{i}_{0},y^{i})=\frac{\sigma^{2}_{t-1}}{\sigma^{2}_{t}} \frac{1-\alpha_{t}}{1-\alpha_{t-1}}x_{t}+(1-\alpha_{t-1}(1-\frac{\sigma^{2}_{t -1}(1-\alpha_{t})^{2}}{\sigma^{2}_{t}(1-\alpha_{t-1})^{2}}))x^{i}_{0}+(\alpha_ {t-1}-\alpha_{t}\frac{1-\alpha_{t}}{1-\alpha_{t-1}}\frac{\sigma^{2}_{t-1}}{ \sigma^{2}_{t}})y^{i}$ (6) $$

$$ $\displaystyle\tilde{\sigma}^{2}_{t}=\sigma^{2}_{t-1}-\frac{\sigma^{4}_{t-1}}{ \sigma^{2}_{t}}\frac{(1-\alpha_{t})^{2}}{(1-\alpha_{t-1})^{2}}$ (7) $$

The derivation for Eq. 6 and Eq. 7 is detailed in Appendix A.1.
The overall training and sampling pipelines are summarized in Appendix A.5 Algorithms 1 and 2, respectively.

Training Objectives.
Previous work Rombach et al. () showed that learning directly the noise $\epsilon$ enhances the performance compared to predicting the $x_{0}$ directly.
This is intuitive as predicting the difference $\epsilon_{t}=x_{t}-x_{0}$ is easier than predicting the original complicated $x_{0}$.
However, in our case, this is not valid due to the conditioned domain $y$, which makes predicting the difference much harder, as shown in Eq. 8

$$ $x_{t}-x_{0}=\alpha_{t}(y-x_{0})+\sigma_{t}\epsilon_{t}$ (8) $$

Accordingly, we directly predict the $x_{0}$. Please refer to Section 3.5 for a detailed analysis.

### 2.4 Toddler Consistency

Our novel framework, ToddlerDiffusion, inherently provides interactivity, allowing users to observe and engage with the intermediate outputs of various modalities. This interactivity is further strengthened by ensuring consistency, a crucial element for effective control and editing capabilities.

Generated Images Editing.
To ensure consistency during editing, we store the noise $\epsilon_{t}$ and the intermediate modalities used in generating the initial reference image, creating a memory that will help us achieve outstanding consistency.
Throughout the editing session, the memory, that was constructed while generating the reference image, remains fixed until the user resets it by generating a new reference image.
Figure 9 demonstrates our framework’s editing capabilities, highlighting the strong consistency it achieves.
Additional editing examples can be found in our demo.

Real Images Editing.
Real image editing is much more challenging as we cannot access intermediate modalities and noise.
Therefore, given a real image $x^{rgb}_{0}$, first, we run $\mathcal{F}_{s}(x^{rgb}_{0})$ and $\mathcal{F}_{p}(x^{rgb}_{0})$ to get the corresponding sketch $x^{s}_{0}$ and palette $x^{p}_{0}$, respectively.
Then, we utilize the recent DDPM inversion technique Huberman-Spiegelglas et al. () to store the noise $\epsilon_{t}$.
More specifically, to get the noise, we have first to run our forward process, Eq. 2, using random noise $\epsilon_{t}$ and store the corresponding intermediate states $x_{t}$.
Then, run our reverse process, Eq. 6, to try to recover the original image.
During the reverse process, the model will predict $\tilde{\mu}_{t}(x_{t},x^{i}_{0},y^{i})$ using Eq. 6.
Then, our goal is that given the predicted $\tilde{\mu}_{t}$, the predicted conditions $y$ and the stored intermediate states $x_{t-1}$, we want to estimate the noise $\epsilon_{t}$
Following these two steps, we construct our memory, which will be used similarly to edit real images.

## 3 Experimental Results

To probe the effectiveness of our proposed framework, we first compare our approach against LDM when trained from scratch on unconditional datasets, LSUN-Churches and CelebHQ, and conditional ones, ImageNet.
Then, we show that our approach can leverage from the high-quality SD-v1.5 weights, enabling the model to be conditional on new modalities seamlessly without adding additional modules Zhang et al. (); Zhao et al. () or adapters Mou et al. ().

### 3.1 ToddlerDiffusion from Scratch

In this section, first we will holistically compare our method, ToddlerDiffusion, against LDM regarding generation quality and efficiency, e.g., convergence rate and number of denoising steps.
Then, we will compare our conditioning ability with ControlNet and SDEdit.
Generally, ToddlerDiffusion consists of three cascaded stages: sketch, palette, and RGB generation.
However, in the following section, we will show the versatility of our method by omitting the $1^{st}$ or the $2^{nd}$ stages, enabling our method to start from a reference sketch or palette, respectively, instead of generated ones.

Unconditional Generation.
We evaluated our framework, ToddlerDiffusion, against LDM on two unconditional datasets, i.e., LSUN-Churches Yu et al. () and Celeb datasets Karras et al. ().
For a fair comparison, and as we notice challenges in replicating LDM results(^1^11See issues [325](https://github.com/CompVis/latent-diffusion/issues/325), [262](https://github.com/CompVis/latent-diffusion/issues/262), [90](https://github.com/CompVis/latent-diffusion/issues/90), [142](https://github.com/CompVis/latent-diffusion/issues/142), [30](https://github.com/CompVis/latent-diffusion/issues/30), and [138](https://github.com/CompVis/latent-diffusion/issues/138) in the LDM GitHub repository.), we have reproduced LDM results.
As shown in Table 1, we boost the LDM performance by 1 FID point on the two datasets, using 3x smaller architecture and 2x faster sampling, as shown in Table 2.

Efficient Architecture.

**Table 1: Benchmark results on three datasets; CelebHQ Karras et al. (), LSUN-Churches Yu et al. () and ImageNet Deng et al. ().**
| Dataset | Method | Epochs | FID $\downarrow$ | KID $\downarrow$ | Prec. $\uparrow$ | Recall $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| CelebHQ | LDM | 600 | 8.15 | 0.013 | 0.52 | 0.41 |
| Ours | 600 | 7.10 | 0.009 | 0.61 | 0.47 |  |
| Churches | LDM | 250 | 7.30 | 0.009 | 0.59 | 0.39 |
| Ours | 250 | 6.19 | 0.005 | 0.71 | 0.44 |  |
| ImageNet | LDM | 350 | 8.55 | 0.015 | 0.51 | 0.32 |
| Ours | 350 | 7.8 | 0.010 | 0.58 | 0.40 |  |

While the primary focus of this paper is not on designing an efficient architecture, our method inherently benefits from architectural efficiency due to the shorter trajectory between stages, as illustrated in Figure 1.
The key insight is that ToddlerDiffusion simplifies each generation task by breaking it into modality-specific stages (e.g., from sketch to palette).
This approach allows us to leverage significantly smaller networks at each stage because solving the problem is easier, where the distributions of the domains, such as sketches and palettes, are much closer than the distribution between pure noise and RGB images.
As a result, the neural network at each stage can be more lightweight. Furthermore, as shown in Table 2, we operate directly in the image space for the first two stages (i.e., noise to sketch and sketch to palette).
This eliminates the need for complex encoder-decoder architectures, such as VQGAN, typically used to generate detailed images from latent representations.
Combining these factors, ToddlerDiffusion achieves a 3x reduction in network size despite cascading across three stages.

Faster Convergence.

Figure: Figure 3: Training convergence comparison between ToddlerDiffusion and LDM.
Refer to caption: extracted/5903868/Figures/figures/ConvergenceRate.png

In addition to outperforming LDM across multiple datasets and metrics, as shown in Table 1, ToddlerDiffusion exhibits a significantly faster convergence rate.
We analyzed the convergence behavior by reporting the FID score every 50 epochs, as shown in Figure 3.
Remarkably, our method achieves the same performance after just 50 epochs as LDM does after 150 epochs.
Furthermore, after 150 epochs, ToddlerDiffusion matches the performance LDM achieves after 500 epochs. This substantial improvement in convergence speed highlights the strength of our approach in efficiently navigating the feature space between stages.
It’s important to note that faster convergence was not an explicit target of our method but rather an emergent property of the pipeline’s design.
Other techniques tailored explicitly for improving convergence speed, e.g., Hang et al. (), can be considered orthogonal and complementary to our method.

Trimming Denoising Steps.
Another unique characteristic of ToddlerDiffusion is its ability to reduce the number of denoising steps during both training and sampling without significantly impacting performance.
As shown in Figure 4, our framework maintains consistent and robust results when the number of steps is reduced from 1000 to 100.
In contrast, the performance of LDM degrades drastically under the same conditions. ToddlerDiffusion can trim denoising steps while maintaining a good Signal-to-Noise Ratio (SNR), particularly for larger time steps, as further discussed in Section 2.1.
Additionally, Figure 5 highlights another intriguing capability of our method: models trained with fewer denoising steps outperform those trained with larger step counts when later using techniques like DDIM Song et al. () to reduce sampling steps.
For example, as seen in Figure 5, the blue curve (trained with 100 steps) achieves an FID score of around 60 when using only ten steps for sampling.
However, if we initially train the model with just ten steps, as represented by the purple curve, the performance improves significantly, achieving an FID score of 15.
This demonstrates the efficiency and adaptability of ToddlerDiffusion, even when minimizing the denoising process.

Class-Label Conditional Generation.
To holistically evaluate the performance of ToddlerDiffusion, we tested it on the widely-used ImageNet dataset, where the condition type is a class label.
Due to limited resources, we conducted our experiments on ImageNet-100, a subset of ImageNet containing 100 classes instead of the full 1,000.
As shown in Table 1, ToddlerDiffusion performs better than LDM on key metrics such as FID, KID, Precision, and Recall.

Controllability.

Figure: Figure 6: Comparison between our editing capabilities and SDEdit Meng et al. ().
Refer to caption: x3.png

We evaluated our editing capabilities against two notable methods: SDEdit Meng et al. () and ControlNet Zhang et al. (). As illustrated in Figure 6, SDEdit struggles when conditioned on binary sketches, which is expected given that it is a training-free approach designed to operate with conditions closer to the target domain (i.e., RGB images).
Consequently, compared to ControlNet, a training-based method that adds an additional copy of the encoder network, our approach demonstrates superior faithfulness to the sketch condition, consistency across the generated images, and overall realism.
Despite ControlNet’s larger model size, our method achieves these improvements without adding any additional parameters.
Quantitatively, we achieved an FID score of 7, outperforming both SDEdit (15) and ControlNet, which achieved 15 and 9, respectively.

### 3.2 Robustness Analysis

Label Robustness.
Beyond these quality comparisons, we also conducted a robustness analysis to assess our model’s behavior when faced with inconsistencies between conditions.
Specifically, we deliberately provided incorrect class labels for the same sketch.

Figure: Figure 8: Our results on the ImageNet dataset. The first column depicts the input sketch. The diagonal (green) represents the generated output that follows the sketch and the GT label. The off-diagonal images show the output given the sketch and inconsistent label. These adversarial labels show our model’s robustness, where the successful cases are in yellow, and the failure ones are in red.
Refer to caption: x5.png

This mismatched the sketch (geometry) and the desired class label (style). Despite the inconsistent conditions, as illustrated by the off-diagonal images in Figure 8, our model consistently adheres to the geometry defined by the sketch while attempting to match the style of the misassigned category, demonstrating the robustness of our approach. Successful cases are highlighted in yellow, and even in failure cases (highlighted in red), our model avoids catastrophic errors, never hallucinating by completely ignoring either the geometry or the style.
More examples can be seen in Appendix B.3.

Sketch Robustness.
To further validate the robustness of ToddlerDiffusion, we designed an aggressive robustness test aimed at evaluating the model’s ability to handle sketch perturbations.
Specifically, we generated three variants of the input sketches using the Structured Edge (SE) detector Dollár & Zitnick (), varying the kernel size $K$ to introduce different detail and noise levels.
As shown in Figure 7, our model consistently demonstrates strong robustness despite these perturbations being absent during training.
The left part of Figure 7 depicts our model’s output using the $2^{nd}$ stage only (Sketch to RGB), while the right part demonstrates the output when using both the $2^{nd}$ and $3^{rd}$ stages.
Even with significant variations in the sketches, ToddlerDiffusion maintains high fidelity to both the geometry and style, underscoring its ability to adapt to unseen input distortions without compromising output quality.

### 3.3 Editing Capabilities

Figure: Figure 9: Controllability ability of our framework, ToddlerDiffusion. Starting from the generated sketch and RGB image (A), we can remove artifacts or undesired parts in red (B), add new content in yellow (C-D), and edit the existing content in green (E-F) by manipulating the sketch.
Refer to caption: x6.png

As discussed in Section 2.4, our approach provides consistent editing capabilities for both generated and real images.

Editing Generated Images.
In Figure 9, we showcase the editing capabilities starting from a generated sketch and RGB image (A). Our method allows for the removal of artifacts or unwanted elements, highlighted in red (B), the addition of new content, shown in yellow (C-D), and modifications to existing content, marked in green (E-F), simply by manipulating the underlying sketch.
These consistent edits are supported in unconditional and conditional generation scenarios. While sketch-based conditional generation naturally lends itself to such control, ToddlerDiffusion extends this capability to unconditional settings and other conditional models like Text-to-Image. In these cases, we refer to this property as "interactivity," allowing users to seamlessly edit through intermediate modalities (sketch and palette) regardless of the original input format, whether class-label or text.

Editing Real Images.
As explained in Section 2.4, we ensure consistent edits for real images by storing the noise used during the generation process for later edits. The primary challenge with real image editing is that we don’t have access to the noise values at each denoising step, as our model didn’t generate the image. To address this, we employ DDPM-inversion Huberman-Spiegelglas et al. () to reconstruct the intermediate noise.
As illustrated in Figure LABEL:fig_teaser_fic, our model achieves consistent edits on real reference images.
In Appendix B.2, we compare against SEED-X Ge et al. (), and our model achieves more consistent edits, as shown.

### 3.4 Incorporating ToddlerDiffusion into Stable Diffusion

In this section, we will answer an important question: “Can our method be integrated seamlessly into the existing powerful Stable-Diffusion architecture?
This question is crucial as follows:
1) Despite our approach’s great capabilities and performance, leveraging the available powerful weights trained on large-scale datasets such as SD is still important.
2) Our approach follows different forward and reverse processes, Eq. 2 and Eq. 6, respectively, which could make it challenging to leverage weights that are trained using different training objectives.

Integration Recipe.
This section will detail the recipe for successfully integrating SD weights into our approach.
First, we used $x_{0}$ as our default training objective and employed LoRA fine-tuning for efficiency.
But these two variants are crucial to make it work.
The original SD is trained to predict the difference, not $x_{0}$.
Thus, we must first try to match its conventional training objective and replace ours by predicting the difference.
However, this will not solve the gap, as our new training objective still includes $x_{0}$ prediction.
Due to this considerable gap in our formulation compared to SD, we must fine-tune the entire model instead of using LoRA, using a small learning rate, e.g., 1e-6.

Controllability.
We convert our method to a “sketch2RGB” by omitting the $1^{st}$ stage to show our controllability capabilities.
By doing this, we can compare our model against the vast tailored sketch-based models built on top of SD-v1.5.
However, these methods have been trained for a long time on large-scale datasets.
Thus, for a fair comparison, we retrain them using CelebHQ datasets for 5 epochs starting from SD-v1.5 weights.
As demonstrated in Appendix B.5, despite this is not our main focus, our method outperforms conditioning conditioning methods without adding additional modules Zhang et al. (); Zhao et al. () or adapters Mou et al. ().

### 3.5 Ablations

Figure: Figure 10: Ablation study for different input’s types for the $2^{nd}$ stage on LSUN-Churches dataset Yu et al. ().
Refer to caption: x7.png

$1^{st}$ Stage Representation.
In Section 2.2, we explore the versatility of the $3^{\text{rd}}$ stage (detailed image) by examining six input modalities, detailed in Figure 10. Comparing different contours representations, namely Edges (using Laplacian edge detector Wang ()), Sketch (utilizing PidiNet Su et al. ()), and SAM-Edges (generated by SAM followed by Laplacian edge detector), we find that Sketch outperforms Edges, as edges tend to be noisier.
However, SAM-Edges provides more detailed contours, yielding superior results.
Notably, feeding SAM-Colored leads to significant performance degradation, likely due to color discrepancies, as observed in SAM in Figure 10.
While SAM-Edges achieves optimal results, its computational intensity renders it impractical.
In contrast, Sketch and Edges are computationally inexpensive.
Furthermore, Sketch’s sparse and user-friendly nature makes it more suitable for editing, facilitating interpretation and modification than dense, noisy edges.
Consequently, we adopt Sketch as the input modality for subsequent experiments.

Palette Effect.
Adding more guidance will offer more editing abilities to the pipeline and enhance the performance.
As shown in Figure 10, rows b and d, when we incorporate the palette into the contours, i.e., edges and sketch, the performance improved by almost 1 and 1.5, respectively.
In addition, Palette-2 outperforms Palette-1 by 1 FID score.
A more detailed analysis of fusing the stages together and how to mitigate the error accumulation can be found in Appendix A.6.

Concatenation v.s. Schrödinger Bridge.
We formulate the generation problem as image-to-image translation.
For instance, generating a palette given sketch or generating the final RGB image given overlaid palette and sketch.
One possible approach is to utilize the conditioning mechanisms offered by LDM Rombach et al. (), e.g., cross-attention and concatenation.
However, these mechanisms are inefficient as their SNR tends to 0 as $t\!\rightarrow\!T$.
In contrast, our formulation follows the Schrödinger bridge, which directly incorporates the condition $y$ into $x_{t}$ during the forward process.
Accordingly, the SNR$\gg\!0$ at $t=T$, which allows reducing training steps from 1K to 50 (Figure 4) and achieving faster convergence (Figure. 3).
Additionally, Table 3 demonstrates a quantitative comparison between Schrödinger Bridge and concatenation.
As shown in Table, 3, the schrödinger bridge (SB) combined with the concatenation mechanism achieves the best results; however, SB alone is better than using the naive concatenation.

Training Objectives.
Previous work Rombach et al. () showed that learning directly the noise $\epsilon$ enhances the performance compared to predicting the $x_{0}$.
As, intuitively, predicting the difference $\epsilon_{t}=x_{t}-x_{0}$ is easier than predicting the original complicated $x_{0}$.
However, this is not valid in our case due to the new term $y$, which makes predicting the difference much harder, as discussed in Section 2.1.
Table 4 shows the same conclusion reported in Rombach et al. (), that predicting the difference $\epsilon$ leads to better performance than $x_{0}$, where the FID score is 8.7 and 15.1, respectively.
On the contrary, our method achieves almost the same performance regardless of the training objective, which is expected, as predicting the difference involves predicting $x_{0}$, Eq. 8.

## 4 Conclusion

ToddlerDiffusion introduces a paradigm shift in image generation by breaking down the complex task into more manageable, modality-specific stages.
Inspired by the natural learning process of a child, our framework elegantly mirrors how human artists progress from rough sketches to fully detailed images.
This structured approach allows us to simplify each stage of the generation process, enabling the use of smaller architectures and achieving efficient, high-quality results.
Despite the potential challenges introduced by multi-stage frameworks, such as increased parameter counts and error accumulation, ToddlerDiffusion overcomes these obstacles through the strategic use of the Schrödinger Bridge, which ensures smooth transitions between modalities.
By maintaining a high Signal-to-Noise Ratio (SNR) at each stage, we enable stable, effective progression, minimizing the number of denoising steps required during both training and sampling.
This results in faster convergence, lower computational cost, and enhanced sampling efficiency.
Moreover, ToddlerDiffusion provides a level of control and interaction that is unprecedented in current diffusion-based models, offering consistent editing capabilities across both generated and real images.
Through extensive experiments on multiple datasets, we have shown that ToddlerDiffusion outperforms existing state-of-the-art models like LDM in terms of quality, interactivity, and speed.
The flexibility and versatility of our approach make it applicable to a wide range of tasks, and its emergent properties, such as faster convergence and improved editing control, further solidify its potential for real-world applications.

## Appendix A ToddlerDiffusion Method (More Details)

### A.1 Derivations

$$ $\displaystyle x_{t}=(1-\alpha_{t})x_{0}+\alpha_{t}y+\sigma_{t}\epsilon_{t}$ (9) $\displaystyle x_{t-1}=(1-\alpha_{t-1})x_{0}+\alpha_{t-1}y+\sigma_{t-1}\epsilon _{t-1}$ (10) $$

### A.2 Architecture Details

Figure 11 depicts the details of our architecture.
We adopt the LDM architecture, which consists of two main blocks:
1) VQGAN Encoder-Decoder to convert the image into low-cost latent space.
2) Unet that responsible for predicting $x_{t-1}$.
Figure 11 shows that the condition $y$ is encoded into $x_{t}$. However, it could be optionally concatenated to $x_{t}$.
In addition, shown in orange and green are the $\sigma^{2}$ and $\alpha$, respectively.

Figure: Figure 11: An overview of proposed architecture, dubbed ToddlerDiffusion. The first block demonstrates the first stage, which generates a sketch unconditionally. Due to our efficient formulation, this stage operates in the image space on $64\times 64$ resolution. The bottom module depicts the third stage, which generates an RGB image given a sketch only or both sketch and palette.
Refer to caption: x8.png

### A.3 1 s ⁢ t superscript 1 𝑠 𝑡 1^{st} 1 start_POSTSUPERSCRIPT italic_s italic_t end_POSTSUPERSCRIPT Stage: Abstract Structure

Figure: Figure 12: Benchmarking results for the $1^{st}stage$ on the CelebA-HQ dataset Karras et al. (). The training time is calculated per epoch using 4 NVIDIA RTX A6000. The sampling time is calculated per frame using one NVIDIA RTX A6000 with a batch size equals 32.
Refer to caption: x9.png

Sketch FID.
A discrepancy exists between the reported conventional FID (RGB-FID), trained on ImageNet Deng et al. (), and qualitative results, as illustrated in Figure 12. This discrepancy Ge et al. () may arise from differences between the training data (RGB images) and the evaluation data (binary images). To bridge this gap, we introduce Sketch-FID by re-training the inception model Szegedy et al. () on a sketch version of the ImageNet dataset. We generate sketches for ImageNet RGB images using PidiNet Su et al. () and train the inception model on the conventional classification task.

Noise Scheduler.
In the $1^{st}$ stage, where our starting point $y$ is a black image (Section 2.1), designing an appropriate noise scheduler is crucial. The bridge noise scheduler is intuitively unsuitable, as it eliminates randomness by adding no noise at both edges, fixing the starting point to a black image. This hypothesis is supported by empirical results in Figure 12, row b, where the model outputs random patterns. We explored linear and logarithmic schedulers, finding the linear schedule superior, yielding Sketch-FID scores of 15.19 and 18.47, respectively (Figure 12, rows c-d).

Ours vs. LDM.
In Section 2.1, we propose an alternative way to generate sketches by leveraging LDM Rombach et al. (), as shown in Figure 2, row A. However, this approach deviates from the nature of sketching. Our proposed formulation, aligned with the topology of sketches (Figure 2, row C), resulted in significant improvements over LDM in both model complexity and performance, as depicted in Figure 12. Our formulation (Section 2.1) allows direct operation on the image space ($64\times 64$) and compression of the Unet to a tiny variant without sacrificing performance. Despite the aggressive compression, our performance is significantly better than LDM, with respective Sketch-FID scores of 15.19 and 49, using a 41x smaller network.

Sketch Intensity.

**Table 5: Sketch Intensity Analysis.**
| Dataset/Intensity | 19% | 24% | 31% | 37% | 43% |
| --- | --- | --- | --- | --- | --- |
| CelebHQ | 7.4 | 7.9 | 9.2 | 10.5 | 12.7 |
| Lsun-Churches | 7.8 | 6.9 | 8.3 | 8.9 | 10.1 |

Controlling the intensity is crucial. Thus, we conducted further analysis, as shown in Table A.3, that probes for best performance, the intensity should follow the data distribution.
We refer to the intensity as the ratio between the white pixels to the black ones in the sketch.
The reported FID scores in Table A.3 are the overall FID using all stages, and the intensity numbers are the initial intensity at $t=T$.
The GT intensity for CelebHQ and Lsun-Churches are 19.2% and 23.5%, respectively.

### A.4 2 n ⁢ d superscript 2 𝑛 𝑑 2^{nd} 2 start_POSTSUPERSCRIPT italic_n italic_d end_POSTSUPERSCRIPT Stage: Palette

As depicted in Figure 13, we introduce a more realistic way to have high-quality palettes for free by first generating the segments from the RGB image using FastSAM Zhao et al. (). Then, a simple color detector module is utilized to get the dominant color, e.g., the median color per segment.

### A.5 Training and Sampling Algorithms

Figure: Algorithm 1 Training Pipeline

### A.6 How to Fuse different Stages Efficiently?

Figure: Table 6: Systematic search for the best stopping step $s$ for the condition truncation.

The reported FID scores for the $1^{st}$ and the $2^{nd}$ stages, in Figure 12 and Figure 10, respectively, are for each stage separately.
In other words, in Figure 10, row c, the $2^{nd}$ stage achieves an 8.6 FID score when a GT sketch is fed, which is obtained from PidiNet Su et al. ().
However, when we fed the generated sketch from the $1^{st}$ stage, the performance drastically dropped from 8.6 to 16.1 (almost doubled), as shown in Table 7, row a, due to the domain gap between the generated and the GT sketches.
To fill this gap, we explored two types of augmentations:
1) Cutout and Dropout augmentation.
2) Condition truncation augmentation.

Cutout and Dropout Augmentation.
First, we explored the straightforward augmentation types, such as Cutout DeVries & Taylor () and Dropout augmentations.
For the Cutout DeVries & Taylor (), we apply a kernel to randomly blackout patches in the sketch.
Additionally, regarding the Dropout augmentation, we randomly convert white pixels to black pixels, interpreted as dropping some white points from the sketch.
As shown in Table 7, generally, augmentation leads to a drop in the $2^{nd}$ stage performance while helping fill the gap in the overall performance, as the FID score decreased from 16 to almost 14.
However, as shown in rows b-d, increasing the amount of the applied augmentation gradually does not help much, as the performance remains almost the same.
However, the overall accuracy, i.e., FID, drops significantly from 14 to 18 when an aggressive augmentation is applied.

Condition Truncation Augmentation.
As shown in Table 7, the conventional augmentation techniques do not help much.
Accordingly, we explored another augmentation variant tailored for the diffusion models, i.e., condition truncation Ho et al. ().
During training the $2^{nd}$ stage, we apply a random Gaussian noise to the fed condition, i.e., the sketch.
So now the $2^{nd}$ stage is trained on noisy sketches instead of pure ones, which makes it more robust to the variations between the real sketches and the generated ones from the $1^{st}$ stage.
Typically, we progressively generate the sketch in $T$ time steps; $T\rightarrow 0$.
However, This added noise could be interpreted as we stop the sketch generation ($1^{st}$ stage) at a particular step $s$; $T\rightarrow s$.
Consequently, we search for it during sampling to determine which step $s$ works best for the overall performance, as shown in Table 6.
In other words, the $2^{nd}$ stage is trained on a wide range of perturbed sketches, e.g., pure sketch ($s=0$), noisy one ($0<s<T$), and even pure noise ($s=T$).
The $1^{st}$ stage is trained for 200 steps, so $s=200$ means we omit the sketch and feed pure noise. In contrast, $s=0$ indicates that we feed the generated sketch as it is to the $2^{n}d$ stage.
As shown in Table 6, following this truncation mechanism, the fusion of the two stages is performed more efficiently, where the overall performance drops from 16.1 to 10.6.
Algorithm 2 is adapted in the supplementary materials to incorporate the condition truncation mechanism.

## Appendix B More qualitative results

### B.1 Demo

Please refer to our [$demo$](https://toddlerdiffusion.github.io/website/), which probes our controllability capabilities in performing consistent edits.

Our demo demonstrates a lot of use cases, including:

- •
(0:00 –> 0:20) [1st stage] Starting from noise generates a sketch by running only the 1st stage, which is unconditional generation.
- •
(0:20 –> 0:24) [2nd stage] Starting from a generated sketch generates a palette. This one is considered an unconditional generation as noise generates the sketch unconditionally.
- •
(0:24 –> 0:32) [3rd stage] Starting from a generated palette generates an RGB image. This one is also considered an unconditional generation.
- •
(0:32 –> 0:44) [Conditional Generation] Starting from the GT sketch, we generate an RGB image.
- •
(0:44 –> 1:15) [Editing] We show our sketch-based editing capabilities.
- •
(1:15 –> 1:46) [Editing] We show our palette-based editing capabilities.

### B.2 Real Image Editing

As explained in Section 2.4, we ensure consistent edits for real images by storing the noise used during the generation process for later edits. The primary challenge with real image editing is that we don’t have access to the noise values at each denoising step, as our model didn’t generate the image. To address this, we employ DDPM-inversion Huberman-Spiegelglas et al. () to reconstruct the intermediate noise.
As illustrated in Figure 14, we compare our model against SEED-X Ge et al. (), and our model achieves consistent edits on real reference images better than SEED-X, despite being trained on much fewer data compared to SEED-X and using smaller architecture.
Additionally, SEED-X fails to perform the edits correctly and keeps the rest of the image unedited, such as the rest of the face or the background.
On the contrary, our method shows great faithfulness to the original image while performing the edits depicted in the sketch.

Figure: Figure 14: Comparison between our approach, ToddlerDiffusion and SEED-X Ge et al. () in terms of the real images capabilities.
Refer to caption: x11.png

### B.3 Class-Label Robustness

Beyond these quality comparisons, we also conducted a robustness analysis to assess our model’s behavior when faced with inconsistencies between conditions.
Specifically, we deliberately provided incorrect class labels for the same sketch.
This mismatched the sketch (geometry) and the desired class label (style). Despite the inconsistent conditions, as illustrated by the off-diagonal images in Figure 15, our model consistently adheres to the geometry defined by the sketch while attempting to match the style of the misassigned category, demonstrating the robustness of our approach. Successful cases are highlighted in yellow, and even in failure cases (highlighted in red), our model avoids catastrophic errors, never hallucinating by completely ignoring either the geometry or the style.

Figure: Figure 15: Our results on the ImageNet dataset. The first column depicts the input sketch. The diagonal (green) represents the generated output that follows the sketch and the GT label. The off-diagonal images show the output given the sketch and inconsistent label. These adversarial labels show our model’s robustness, where the successful cases are in yellow, and the failure ones are in red.
Refer to caption: x12.png

### B.4 Editing Capabilities

Our approach introduces an interpretable generation pipeline by decomposing the complex RGB generation task into a series of interpretable stages, inspired by the human generation system Stevens (); Marr (; ).
Unlike traditional models that generate the complete image in one complex stage, we break it into $N$ simpler stages, starting with abstract contours, then an abstract palette, and concluding with the detailed RGB image. This decomposition not only enhances interpretability but also facilitates dynamic user interaction, offering unprecedented editing capabilities for unconditional generation, as shown in Figure 16, Figure 17, Figure 18, and Figure 19.

Figure: Figure 16: Controllability ability of our framework, ToddlerDiffusion. Starting from generated sketch and RGB image (A), we can remove artifacts or undesired parts, in red, (B), add a new content, in yellow, (C-D), and edit the existing content, in green, (E-F) by manipulating the sketch.
Refer to caption: x13.png

Figure: Figure 17: Controllability ability of our framework, ToddlerDiffusion. Starting from generated sketch and RGB image (A), we can remove artifacts or undesired parts, in red, (B), add a new content, in yellow, (C-D), and edit the existing content, in green, (E-F) by manipulating the sketch.
Refer to caption: x14.png

Figure: Figure 18: Controllability ability of our framework, ToddlerDiffusion. Starting from generated sketch and RGB image (A), we can remove artifacts or undesired parts, in red, (B), add a new content, in yellow, (C-D), and edit the existing content, in green, (E-F) by manipulating the sketch.
Refer to caption: x15.png

Figure: Figure 19: Controllability ability of our framework, ToddlerDiffusion. Starting from generated sketch and RGB image (A), we can remove artifacts or undesired parts, in red, (B), add a new content, in yellow, (C-D), and edit the existing content, in green, (E-F) by manipulating the sketch.
Refer to caption: x16.png

### B.5 Sketch Conditioning (SD-v1.5)

We convert our method to a “sketch2RGB” by omitting the $1^{st}$ stage to show our controllability capabilities.
By doing this, we can compare our model against the vast tailored sketch-based models built on top of SD-v1.5.
However, these methods have been trained for a long time on large-scale datasets.
Thus, for a fair comparison, we retrain them using CelebHQ datasets for 5 epochs starting from SD-v1.5 weights.
As demonstrated in Figure 20, despite this is not our main focus, our method outperforms conditioning methods without adding additional modules Zhang et al. (); Zhao et al. () or adapters Mou et al. ().
We compare training-free Tumanyan et al. (); Parmar et al. () and training-based Li et al. (); Zhang et al. (); Zhao et al. () methods.
The performance is worse than expected for the trained-based methods due to two reasons:
1) We train all the models for only 5 epochs; thus, some of this method, in contrast to us, needs more epochs to converge.
2) We apply augmentations to the input sketch, which shows that some of these methods are vulnerable to sketch perturbations.

Figure: Figure 20: Comparison between our approach, ToddlerDiffusion and training-free Tumanyan et al. (); Parmar et al. () and training-based Li et al. (); Zhang et al. (); Zhao et al. () sketch conditioning methods.
Refer to caption: x17.png

### B.6 Free-Hand Sketch Conditional Generation Samples

Figure 21 depicts more qualitative results for our conditional generation pipeline, where we first generate the sketch using drawing tools and then use the generated sketch to generate the RGB image.
Despite the sketch being too bad and sparse, our model still generates high-quality images.

Figure: Figure 21: Qualitative results for our unconditional generation.
Refer to caption: x18.png

### B.7 Qualitative results for our sketch-conditional generation

In addition to our unconditional generation, our framework can be used as a conditional generation pipeline, where the user can omit the first stage and directly fed a reference sketch.
As shown in Figure 22, given the reference sketch, we run only the second stage, which is responsible for generating the RGB image given a sketch.
Compared to the generated images in Figure 21, the generated images in Figure 22 are much better as the reference sketches are much better than the generated ones.
This observation is aliened with our discussion regarding the gap between the two stages.
Therefore, improving the quality of the generated sketches will directly reflect on the overall images quality.

Figure: Figure 22: Qualitative results for our sketch-conditional generation.
Refer to caption: x19.png

## Appendix C Failure cases

The last column of Figure 21 and Figure 22 depicts failure cases, where the generated RGB is following the sketch but generates unrealistic image.
Intuitively, the failure cases are more apparent in the unconditional generation due to the generated sketches quality.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: - Could you please provide more details about how to incorporate ToddlerDiffusion into Stable Diffusion? For instance, what text prompts are used? Additionally, is Stable Diffusion fine-tuned on CelebHQ datasets for five epochs using x_0 prediction and the Schrödinger Bridge conditioning mechanism similar to ToddlerDiffusion? Is the network architecture of Stable Diffusion modified to include sketch conditions, as demonstrated in Figure 12? If so, then how are the parameters initialized in the modified components?

- Why do the numbers of LDM parameters differ between Table 2 (263M) and Figure 13 (187M)?

- Why are the schedulers for $\alpha$ and $\sigma$ different in Figure 12 (left) and (right)? Why does $\alpha$ increase from $T$ to $0$ in the left block, but decrease from $T$ to $0$ in the right block? What does the presented $y$ represent in the first stage (Figure 12 left block) that unconditionally generates the sketch?

- Why is SB+Concat not utilized in ToddlerDiffusion, while only SB is used, despite the former demonstrating superior performance in Table 3?
soundness: 3
strengths: 1. **The proposed cascaded framework is reasonable and effective**. With fewer network parameters and training time, ToddlerDiffusion can outperform the standard LDM in both generation quality and conditioning consistency.

2. **Thorough experiments and ablation studies are conducted** in the main content and appendix to demonstrate the versatility of the proposed framework and validate the effectiveness of several design choices.
summary: This work proposes ToddlerDiffusion, a cascaded sequence of relatively small diffusion models, to gradually generate different aspects of images. Those different diffusion models are connected by the Schrödinger Bridge to improve conditioning consistency.
weaknesses: 1. **Presentation of this work requires thorough improvements.**.
- The authors should use `\citep` for most cases in the manuscript, which places the authors' names and the year in parentheses. The current use of `\cite` makes the manuscript cluttered and difficult to read.
- The manuscripts contain many incoherent parts. For instance, in Lines 474-479, the authors state: "we used x_0 ... and employed LoRA fine-tuning for efficiency. **But** these two variants are crucial to make it work". The use of the transitional word "but" in this context is confusing. Later, the authors assert that "we **must** fine-tune the entire model instead of using LoRA". Then why is LoRA fine-tuning described as crucial if it is not the recommended approach?
- Notations are inconsistent in the manuscript. For example, in Equation (2), $x_t=\alpha_t x_0^i+(1-\alpha_t)y^i+\sigma_t^2 \epsilon_t$, whereas in Equation (6)~(16), $x_t=(1-\alpha_t) x_0+\alpha_t y+\sigma_t \epsilon_t$.
- What is the meaning of "$:\sigma_t^2=\alpha_t-\alpha_t^2$" in Equation (2)?

2. **Many statements in this work are incorrect or overclaimed**.
- In the abstract: "surpassing Stable-Diffusion (LDM) performance". However, all experiments conducted in the main content are compared with the LDM trained on small datasets, which are NOT Stable Diffusion (v1/v2/v3).
- The training objective of LDM or Stable Diffusion is the noise $\epsilon$. However, $\epsilon$ is NOT $x_t-x_0$ in LDM as stated in Line 266 and Table 4.
- In Line 507-509, by the SNR definition in this work, SNR does not tend to 0 as $t \rightarrow T$, as the concatenated condition $y$ also provides some signals to a certain extent similar to the proposed framework even at $t=T$. SNR should NOT be the reason that distinguishes between different conditioning methods (Concatenation v.s. Schrödinger Bridge).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: My biggest concern is about the novelty of this work since the framework could be considered as many conditional generative models. Secondly, I am not sure about the ability to extend this framework to text-to-image generation; this could limit the applicability of the proposed method.
soundness: 3
strengths: 1. The paper is well-written and easy to follow. I particularly like the idea of the "toddler," which is motivated by how the child grows up and learns everything.
2. This training framework is more training/sampling efficient than unconditional diffusion generation. It could be used for editing and control generation since it comprises many conditional generative problems trained by Schrodinger Bridge.
3. The experiment is well-organized.
summary: This paper introduces a generative framework comprising three stages: noise to sketch, sketch to palette, and palette to RGB images. The paper uses the Schrodinger bridge I2SB [1] to train these stages. This framework takes less training time and samples efficiently. Moreover, the architecture is smaller and has better training convergence. This framework could be used for editing and control generation.

[1]: I2SB: Image-to-Image Schrödinger Bridge
weaknesses: 1. The methodology seems simple since it decomposes the generative problem into subsequent conditional generative problems. Each problem can be solved directly using LDM or Schrodinger Bridge, and the paper uses Schrodinger Bridge to solve them like I2SB. Therefore, this paper's novelty is limited since it is just like you apply I2SB directly to solve conditional generative problems.
2. The paper mentions extending the method to Stable Diffusion but omits stage 1. This is a meaningful editing and controllable generation problem. However, for the text-to-image generation, I doubt the ability to train stage 1 mapping from (noise and text) to sketch using the Schrodinger Bridge. The reason behind this is that text is a completely different modality with RGB, sketch, and palette. Therefore, mapping from (noise and text) to sketch is not easily solved by unconditional Schrodinger Bridge. Further, the architecture should be larger and specifically designed for that task. This is an interesting question that should be discussed in the paper to answer the framework's scalability. The author could try to train such (noise and text) to sketch models on small text-to-image datasets such as CUB-200 or COCO dataset. [2] (see this work for small text-to-image problem)
3. The paper should include citations for [1] since their methodology is similar to the paper method. I2SB uses the Schrodinger bridge for image-image problems, which are the same problem as your decomposed conditional generative problems. The only difference is they solve image superresolution/inpainting/deblurring instead of sketch-to-palette and palette-to-image.
4. Minor: Several typos at Eq. 3 (the right-hand equation label should be sketch and palette)

[1]: I2SB: Image-to-Image Schrödinger Bridge
[2]: Vector Quantized Diffusion Model for Text-to-Image Synthesis

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: 1. Minor spelling mistake Ln 46  emerging proprieties 

2. Regarding the problem prescribed in section 2.1 Ln 216,217 : [1] provides some suggestions on how to schedule the noise process to achieve zero terminal SNR and fix the inference schedule. I refer the paper here for the reference of the authors

   [1] Lin, S., Liu, B., Li, J. and Yang, X., 2024. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp. 5404-5411).
soundness: 3
strengths: 1. The proposed framework of a multistep generation process is novel to the best of my knowledge. The method is motivated by an inspiring thought process based on the workflow of how humans think to create a picture.
2. The disentangled training process also enables enhanced controllability, stability and faster training and inference process. Moreover such a generation process also gets benefits of added diversity because of the compositional effect in each stage
3. The performance boost over naive LDMs is substantial while also enabling zero shot capabilities to the the model.
4. The method achieves better performance than LDMs across diverse datasets while showing better training and inference efficiency.
summary: The paper introduces Toddler diffusion, a method to generate images like the thought process of a child. The paper decomposes the image generation process into three stages first on a high level sketch generation, then a colour palette generation and finally generating a complete RGB image. To enable learning from one modality to another, the authors leverage Schrödinger Bridges. Such a form a training enables capabilities of seamless editing into the model. Moreover also enables faster sampling compared to conventional LDMs. The authors perform comparisons with conventional LDMs to illustrate the effectiveness of the method.
weaknesses: 1. The training process of toddler diffusion is not clear from the main manuscript. Are different U-Nets leveraged from different modality transformations. I would advise the authors to include a flowchart diagram of signal flow in the main manuscript or alternatively add a subsection detailing this aspect.
2. Toddler diffusion utilizes Schrödinger bridges and an SDE perspective for the training and sampling process. However Rectified Flows have shown to achieve much better performance utilizing ODEs, which also improves further distillation performances. Can the authors contrast on the benefits of training and sampling using diffusion  Schrödinger bridges and rectified flows.
3. Toddler diffusion requires annotated data in different modalities to enable conditional generation. Hence I believe the comparisons provided by the authors by comparing with Naive LDMs are not fair, for a fairer evaluation. I believe the authors need to train a diffusion model which can generate sketch from noise, then 2 conditional models for the translation tasks. Could the authors please provide a comparison with LDMs trained in the proposed manner or alternatively comment on how the current experiments are valid comparisons
4. Can the authors also include performance of Toddler diffusion compared to pixel space diffusion architectures and diffusion transformers for the same datasets. Some valid methods would be SiT[1], DiT[2], Pixel space models[3,4]

   [1] Ma, N., Goldstein, M., Albergo, M.S., Boffi, N.M., Vanden-Eijnden, E. and Xie, S., 2024. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740.

   [2] Peebles, W. and Xie, S., 2023. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4195-4205).

   [3] Kingma, D. and Gao, R., 2024. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36.

   [4] Kynkäänniemi, T., Aittala, M., Karras, T., Laine, S., Aila, T. and Lehtinen, J., 2024. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1.	Given ToddlerDiffusion is a smaller framework compared with LDM, showing it converges faster and has better performance with the same number of epochs is not sufficient to prove it’s a better model. Table 2 claims the training converges at epoch 400 (not mentioning on which dataset), but in Table 1 none of the datapoints was reported at epoch 400. Showing that ToddlerDiffusion outperforms when both training procedures converge would be a stronger evidence.
2.	Please provide more details about the experiments, such as but not limited to: at which resolutions were the three stages trained and evaluated? How was controlnet/SDEdit configured when the editing capacities was compared? Is there any quantitative results supporting it’s editing capacity?
3.	It’s understandable that The generation quality of ToddlerDiffusion is inferior to that of the SOTA model (SDXL, Flux.1, etc.) given limited data and training. However, The use cases for low-resolution unconditional or label-conditioned generation are very limited. Would the first two stages be able to handle more complicated conditions at higher resolutions as in latest models? If so, would the framework still be efficient?
soundness: 3
strengths: 1.	The framework natively offers decent capability in editing and interpretability without additional adapters or fine-tuning.
2.	By decomposing the image generation into three simpler sub-tasks, the framework has fewer parameters, converges faster, and is more robust against a smaller number of sampling steps.
3.	The framework shows its capacity to achieve better performance on given datasets comparing with naive LDM.
summary: This paper proposes a new approach to decompose the diffusion framework into simpler and interpretable stages. The new framework called ToddlerDiffusion is a cascaded model that introduces two additional intermediate representations, sketch and palette. Different modality-specific models are connected to obtain high-quality image outputs with detailed textures. Instead of naive LDM, the Schrödinger Bridge is leveraged to determine the optimal transformation between modalities. The new design achieves better overall performance with faster sampling and smaller architecture compared with LDM, and also enables editing and interaction at intermediate steps.
weaknesses: 1.	There is a large gap between the performance of the baseline model (LDM/SD1.5) and the current state-of-the-art models.
2.	The experiments showed the model’s capability in limited training data and resources, however, the value of proving a smaller model converges faster is limited. There’s lack of the evidence about its performance compared with baselines given sufficient data and training. 
3.	The description of the experiments lacks many details that reduced the robustness of the conclusions. For example, the experiments comparing editing capabilities with controlnet did not mention which base model was used by controlnet or at which resolution was the experiment performed, makes the fairness of the experiment questionable.

Meta-review: The submission proposes ToddlerDiffusion, a cascaded sequence of diffusion models operating on different modalities such as contours, palettes, and textures, to finally generate an RGB image.
Instead of using concatenation of different modalities as conditioning, they use the Schrödinger Bridge for consistency.
The submission received final ratings of 6, 6, 8, 6. 
The ACs did not find enough reason to overturn the positive consensus and recommend acceptance.

Decision: Accept (Poster)