DISCO: DISCrete nOise for Conditional Control in Text-to-Image Diffusion Models
Authors: Longquan Dai
Wu Ming
Dejiao Xue
He Wang
Jinhui Tang
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Models
Conditional Control
Primary area: deep_learning

Abstract: A major challenge in using diffusion models is aligning outputs with user-defined conditions. Existing conditional generation methods fall into two major categories: classifier-based guidance, which requires differentiable target models and gradient-based correction; and classifier-free guidance, which embeds conditions directly into the diffusion model but demands expensive joint training and architectural coupling. In this work, we introduce a third paradigm: DISCrete nOise (DISCO) guidance, which replaces the continuous conditional correction term with a finite codebook of discrete noise vectors sampled from a Gaussian prior. Conditional generation is reformulated as a code selection task, and we train  prediction network to choose the optimal code given the intermediate diffusion state and the conditioning input. Our approach is differentiability-free, and training-efficient,   avoiding the gradient computation and architectural redundancy of prior methods. Empirical results demonstrate that DISCO achieves competitive controllability while substantially reducing resource demands, positioning it as a scalable and effective alternative for conditional diffusion generation.

DISCO: DISCrete nOise for Conditional Control in Text-to-Image Diffusion Models
Longquan Dai, Wu Ming, Dejiao Xue, He Wang, and Jinhui Tang" Nanjing University of Science and Technology, Nanjing, China {dailongquan, wuming, xuedejiao, wanghe, tangj innuinnhiiii (anuuij@jjj @njjsts conjustee edut.edd..cmm cn
Abstract
A major challenge in using diffusion models is aligning outputs with user-defined conditions. Existing conditional generation methods fall into two major categories: classifier-based guidance, which requires differentiable target models and gradient- based correction; and classifier-free guidance, which embeds conditions directly into the diffusion model but demands expensive joint training and architectural coupling. In this work, we introduce a third paradigm: DISCrete nOise (DISCO) guidance, which replaces the continuous conditional correction codebook of discrete noise vectors sampled from Gaussian prior. Conditional generation is reformulated as code selection task, and we train prediction net- work to choose the optimal code given the intermediate diffusion state and the conditioning input. Our approach differentiabilltyyfees and training-efficiemn avoiding the gradient computation and architectural redundancy of prior methods. Empirical results demonstrate that DISCO achieves competitive controllability while substantially reducing resource demands, positioning as scalable and effective alternative for conditional diffusion generation. Code is available at
with finite
Introduction
Diffusion models 4 g have achieved impressive success across diverse domains [ 25 47 51]
scaling to billions of samples (27) and establishing themselves as foundational generative models. They are particularly effective for conditional generation tasks [
13 32 46) and support
wide range downstream applications [26] 38 39
30
40 B1. Consequently, the challenge of
conditional ceneration-taalloinnn outputs to satisfy user-defined constraints such as labels, attributes, or spatiotemporal cues is becoming increasingly critical.
Conditional generation in diffusion models typically follows one of two paradigms: classifier-based guidance 10) and classifier-free guidance [5]. Both require training dedicated model tailored to each conditioning signal. More recently, vraining-free guidance, a subclass of classifier-based methods, has emerged. These approaches leverage off-the-shelf, differentiable target predictors- as classifiers, loss functions, or energy functions to steer generation without additional training. The predictor evaluates sample quality and provides gradients to guide the diffusion process toward desired outputs.
Classifier-based guidance [10] trains a noise- conditionniiiionall classifier to estimate the class probability Po(c)z;) Training-fre -ree guidance [3 32
44 46) builds on this idea from an energy-based perspec- tive [66]. using pretrained target networks such as depth or segmentation models-to approximate Po po(c)2;) without training dedicated classifier. While these predictors still require pretraining, their decoupling from the diffusion model reduces overall computational cost. A key limitation, however, is that these approaches require differentiable predictors, guidance relies on computing the gradient z, log p(z, c) to apply conditional corrections during sampling.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Classifier- free guidance 15] removes the need for an external classifier by interpolating between the score estimates of a conditional diffusion model and its jointly trained unconditional counterpart to form a conditional correction term. This approach relies on conditioning mechanisms [48] that adapt an unconditional model into a conditional one. A major advantage of classifier-free guidance is its ability to generate outputs more faithfully aligned with the conditioning input [18). However, this comes at the cost of increased training complexity: the conditional branch modifies features throughout the diffusion model, requiring the full model to be loaded during training. In contrast, classifier-basee methods allow the conditional network to be trained independently.
In this paper, we propose a third approach: DISCrete nOise (DISCO) guidance, which overcomes key limitations of existing methods. Both classifier-based and classifier-free methods aim to estimate a conditional correction term (zz, c). which transforms the zero-mean Gaussian noise used in unconditional denoising into a non-7ero-meat noise for conditional denoising. Our insight is that, under certain conditions, this non-zero-mean noise can be effectively approximated using a finite codebook of discrete vectors, pre-sampled from a Gaussian distribution. This reframes conditional generation as code selection problem: given the current diffusion state and conditioning input the model selects an optimal code from the codebook. To realize this, we introduce a discrete noise prediction network that learns to perform this selection efficiently and effectively. In summary, the contribution of our DISCO are that:
Differentiability-Free:
Our discrete noise prediction network directly selects the optimal code to substitute for the non-zero-mean Gaussian noise, avoiding the need to compute gradients of the target predictor as required classifier-based guidance.
Training Efficiency:
Training the discrete noise prediction network does not require loading or fine-tuning the diffusion model, significantly reducing computational overhead. This is because both inputs 2, and c-as well as the discrete code indices can be precomputed.
Alignment Improved: Our method explicitly trains the prediction network to select control noise, providing direct supervision for controllability. In contrast, Mlassifier-basee guidance relies on auxiliary classification loss, which does not directly align with the diffusion process.
2 Related Work
Classifier-based guidance has primarily focused on training-free free approaches, where an independently trained classifier used to guide the diffusion process during inference without modifying the diffusion model itself. In contrast, classifier-free guidance relies on learned conditioning, typically by integrating an auxiliary conditional network into the diffusion model during training. In this section, we briefly review both categories to contextualize our proposed DISCO method, which introduces third paradigm for conditional generation that is distinct from both classifier-based and classifier-free approaches.
Classifier-based guidance was introduced by Dhariwal and Nichol [10], who trained a time- dependent classifier to approximate the posterior log Pt (c)z,) using pre-trained diffusion model. Later works estimate the gradient Vz, Iog,: 22) at inference using pre-trained predictors. Chung et al. [6] applied Tweedie's 's formula to linear inverse problems, which was extended to broader conditional generation by Chung et al. 77. Yu al. [46), Bansal al. M. and Wang et al. [37] Song et al. (22 reduced bias via multiple samples from noisy Gaussian prior. Zhu et al. [52] introduced manifold projection, later refined by He et al. [13 with an autoencoder enforcing guidance in the tangent space. Yang et al. assumptions. Despite these advances, classifier-based guidance fails when the target predictor is non-differentiable.
[3 further improved this by mitigating manifold drift without strong
Classifier-free guidance extends the conditional processing capabilities of diffusion models by modifying the architecture to directly incorporate conditioning information. ControlNet 48] proposes to utilize the trainable copy of the UNet encoder in the diffusion model to encode extra condition signals into latent representations and then apply zero convolution to inject into the backbone of the UNet diffusion modal. However, ControlNet is a single-modality framework and requires a separate model for each modality. To address this, unified -et models [17 [8 20 29 are proposed to handle diverse control signals by inject multi-modality into original diffusion model. Despite their flexibility, these models are typically iestrce-intensive to train.
22 49

---

3 Preliminaries
In this section, we delve into two fundamental theorems that theoretically guarantee the practicality of our discrete noise guidance approach. Theorem 1. Let n. 2 denote the ambient dimension, and let 0 = Z(pe, c) be the angle between
the mean vector I and random vector e NN 1). Then, v cos² 0 ( (TeTl follows the non-central beta distribution:
(p.e)
v Beta (ax, 3; X). Let 2F1 (a, by G; 2) be a Gaussian hopergeomerric (1)
where a = 2' == n-1 and | 2
The expectation of cos" 0 given by
a a+ a - a+B+A 2F1 2P1 (1. 1:a 3 ++1 1TT) a+
Elu]
This theorem can be leveraged assess whether a vector is statistically consistent with being drawn from N ((p,, I). Specifically, one can evaluate its cumulative probability under the above distribution p P(u cos² Z(1a, E)), where v Beta (ar, 3; A). A large p-value suggests that the observed angle is likely under the assumed distribution. Thus, we have the following definition:
Definition 1. Under the setup described above, consider that vector e is likely to be sampled
from N(,, I) if
(p..)) (|alll-lel)) Elu]. Equivalently, if the angle between μ and satisfies Liu, C) <
arccos Elu)) we regard e as being drawn from N (u,
Let K 2 be the number of independent random samples. We consider independent random vectors CKK = =(e(1), e(2) . e(k) sampled from (0, I) in the n--iieensiona space, and let μ RR" be a vector. For each i = ,,.... K, 0, = L(u, e (()) is defiend as the angle between and e'. Thus e/- cos2 0, be independent sampled from the non-central beta distribution (T).
The maximum of }u;} is given by VKK = maxisisk Uj. As K - co, the typical maximum of the cosine square similarity vK is value by such that: P(UK x bx) - 1. When the sample size K becomes large, the typical maximum refers to the value that the maximum is most likely to take. It is exactly the expected value, nor is concerned with rare extreme fluctuations; rather, it describes where the maximum tends to be most of the time. We use it to describe how similar the random set CKK is to ya and have the following theorem to estimate the typical maximum Veg.
Theorem Under the setup described above, let {u be K samples drawn from the non-central beta distribution Beta (a, 3; A). When n is moderately large and K is not too small, the typical maximum Veg is approximately the same:
2 log K
vek
n
This theorem reveals that in high nigh-dimensional spaces with large n, any two vectors are nearly orthogonal the typical maximum of approaches 1. To increase the similarity between two vectors, we must decrease the dimension n and increase the number of samples K. Since each 2, measures the cosine square similarity between (K) and μ., the typical maximum ve describes how the set can approach μ.. Thus, under the setup described above, the definition in the following is reasonable.
Definition If vex 0.15nE[(v), we say the set CK can provide a sample for the distribution (p, I) and say argmin cos? L(u, (()) as the sample.
DISCO
In this section, we introduce DISCO, a method for conditional generation using diffusion models. We begin by showing that conditional generation can be achieved by sampling noise from a non-zen non-zero- mean Gaussian distribution. Next, we demonstrate that this sampling process can be replaced by selecting discrete codes from a predefined codebook. This insight implies that accurately predicting the correct discrete code based on the input condition enables effective conditional generation. To realize this, we describe how to train a network that predicts these discrete codes from intermediate diffusion states and the conditioning input.
3

---

4.1 Non-zero-mean Gaussian Distribution for Conditional Generation
To derive conditional diffusion models, Dhariwal and Nichol [0] introduced classifier guidance. which modifies the score cø(z,, c) = -o vs, log p(zz|e) by adding the gradient of the log- ag-likelihood
from a classifier model po (c) which is proportional to exp(E(z), c)), as shown below:
Eo(z,, co (2t, c) wo V log po (c) zz) co(21, c) + - Vz, log El (Zz, c)
(4)
Conditional Correction (s)
where w a parameter controlling the strength of the classifier guidance. The modified score (z c) is then used in place of cØ( (z c) when sampling from the diffusion model, resulting in approximate samples from the distribution: (2 (c)(c) x po (clc)))(c(/22)) This approach effectively increases the probability of data where the classifier (a(c)z) assigns high likelihood to the correct conditions.
Since classifier guidance relies on gradients from an image classifier, Ho and Salimans [5] aim to remove the need for the classifier and introduce classifier-free guidance. This approach modifies cg(=,; e) in way that produces the same effect as classifier guidance but without using classifier. They train an unconditional denoising diffusion model pe (z), parameterized by cø(z,. c 2), along with the conditional model po (z)cc. parameterized by Eø ( c), where Ø denotes the absence of the class identifier c. Sampling is then carried out using the following linear combination of the conditional and unconditional score estimates:
En(2,. c) = co(z): c) + w(e c) co( R))
(5)
Conditional Correction A (z).c)
The DDPM denoising step
16 for conditional generation thus becomes
1
1 - a4 Eg(Z(,,) c) + pie', é' e'~N(0,1I)
21-1
2
a
v
(6)
1
1- at
E0(21: e) PIE, e~N(u(Z), c),
2t
a
VI āi
where pl c)
VaivI-a A c) and the last term is sampled from non-zero-mean
1-a,
Gaussian distirbution N' (p(zz, I)
4.2 Discrete Noise for Conditional Generation
Equation (6) shows that in conditional DDPM sampling, noise is drawn from a non-zero-mean Gaussian distribution. We propose replacing this noise with code selected from a fixed codebook CK e1)) ,..,,, (K)), where each code e(k) is independently sampled from (O, I) and remains unchanged during training and inference. The core idea is that appropriate code from this set can serve as an effective discrete approximation of the non-zero-men yean Gaussia noise, enabling simple codebook lookup in place of continuous sampling.
Theorem provides a criterion for selecting a good substitute for Gaussian noise. It shows that the squared cosine similarity between and 6 ~~ (N,,, I) follows a non-central beta distribution. there exists a code elt) E C,K whose squared cosine similarity with fu(z,, c) exceeds the expected value E(u] of this distribution, then e can be considered a valid substitute under Definition 1
Theorem 2 evaluates whether the codebook CK can provide such substitute. It defines the typical maximum similarity Vex between μu uzt, c) and the codes in CK as in Equation (3). If vex > Blu]. the codebook is deemed sufficient to approximate samples from the target non-zero-ment Gaussian distribution N (z, e), I).
Since the latent space of SD 1.5 has size 4096 = 4 x 64 x 64, the dimensionality of the noise in Equation (6) is also 4096. According to Theorem 2 in such digh-dimmnsional spaces, predefined codebook C,K cannot provide a good approximation to samples from a non-zero-menn Gaussian distribution, as most vectors tend be nearly orthogonal. To address this, we divide into 256 blocks [55 each of size 4 x 4 x 4, so that e cat((E;)). and each block has dimensionality 64. In this configuration, the expected value of Beta(0.5. 31.5; X) 0.0278, with A || 1 x 256 based on empirical statistics. For predefined codebook 1096 with 4096

---

"A close-up a capybara standing sandy or rocky terrain."
"A coastal scene sunset, featuring lighthouse situated on sea."
"A young is playing the trumpet outdoors in an urban setting.
"A close-up of fluffy gray kitten sitting an orange blanket." "
TFG
EMControl
ControlNet
AnyControl
Figure I: Conditional Generation with Discrete Noise. The first two rows present the text prompt and corresponding conditioning input. The third row shows the outputs generated using continuous noise. The final row displays the results generated using discrete noise. The results demonstrate that our method maintains generation quality without degradation.
vectors sampled from 64-dimeesional standard Gaussian, the typical maximum squared cosine similarity is 0.263. Since this exceeds the expected similarity from the non-zero-mean distribution, we conclude that C4000 is sufficient to approximate samples from (zz c), I). where || 16,
and fu(z,, c) = cat( c (z,, This enables us to reformulate Equation (6) as follows:
1
1 - at
ea (2), e) picat((E:)),
2t-1
2t
Vot
v1 āt
E, = (Ri),
k; = argmax S5$4096 cos² ((pe(ZE,,))
To validate our approach, we replace the non-zer men-men--mea mean Gaussian noise used in TFG [44], EMCon- trol (37), ControlNet [48], and AnyControl 34 with proposed discrete noise. The evaluated algorithms are categorized into two types: classifier-based guidance methods (e.g.. TFG and EMCon- trol) and classifier-free guidance methods (e.g.. ControlNet and AnyControl). We apply Equations (4) and (5) to compute the conditional correction, followed by Equation (7) to perform conditional gener- ation with discrete noise. The results are presented in Figure I where the third row shows outputs generated using the original methods, and the fourth row displays the results with our approach. The comparable visual quality confirms the effectiveness of our substitution strategy.
4.3 Conditional Control by Discrete Noise Guidance
Both classifier guidance and classifier-free guidance estimate the condition term A A e) based on the intermediate state 2, and the condition In the previous section, discretized the noise ~ (Nu(,, c), I) and validated its effecctiveness. Building on this, we propose training network that directly predicts the discrete noise code from and 2;, thereby enabling explicit control over the generation process, Notably, this approach differs fundamentally from prior methods, as it predicts the discrete control noise rather than the condition term A A(z, c) itself.
4.3.1 Discrete Noise Predication
In this subsection, we describe how the discrete noise indices {k; (k,),20
256 is predicted from the interme- diate features 2, cat( (z,,, 1256 (25)) and the conditioning input c, where each 21,; is x block of the intermediate state z,. We adopt the Diffusion Transformer (DiT) [2] our backbone, which
integrates time-step conditioning into standard encoder-only transformer (36] and employs rotary
positional embeddings (33 for spatial encoding.
5

---

"A gray kitten and a golden retriever puppy sit together"
"An car parked under a tree with large green leaves." "
"A winding path leading to small cottages.
Sketch
Canny Edge
M-LSD
"A young woman is standing outside a cozy café"
"Two young women standing back-to back-to-back"
"A serene and cozy indoor scene featuring a cat."
HED "A church with twin towers stands by calm river."
Skeleton
Location
"A small, fluffy bird with warm brown plumage"
"A minimalist and serene interior scene.
Depth
Normal
Segmentation
Figure 2: Conditional Generation with Diverse Conditions via Discrete Noise. Discrete noise enables seamless integration of various conditions into diffusion models. To demonstrate its versatility, we showcase ten diverse applications: Sketch, Canny Edge, MLSD, HED, Skeleton, Location, Depth, Normal, and Segmentation.
We adopt BERT-style training scheme p for our DiT model. Specifically, formulate the training as discrete noise index completion task: a subset of the input discrete noise indices is randomly masked, and the model trained to recover the missing indices { hee using the condition the intermediate state 2z, and the unmasked indices {ke hee where M denotes the set of masked positions. Unlike traditional Transformers that operate solely on discrete token sequences, our model uses hybrid input that combines discrete indices with continuous visual features.
The hybrid inputs DiT consist of condition c, the intermediate state 2,, and unmasked discrete noise indices [ke For each masked index, we replace its original codebook index with special index [M ASK Conditioned on the unmasked indices, intermediate features {zr, 1 and condition the Transformer is trained to estimate the likelihood the indices the masked positions:
256 P((ki)ie (iks (2.))=110 c)
II P(k Plks (2u,)tm1 c) iEM
(8)
The model is optimized by minimizing the softmax cross-entropy loss between the predicted proba- bilities and the ground-truth indices. During the inference time, the model starts with the intermediate features 2, cat( 2t. 1256 and the conditioning input and the discrete noise indices all filled with [MASK] The predicted discrete noise indices are generated by the wybrid-transformer
4.3.2 Training Dataset Preparation
Training the discrete noise prediction network requires data pairs (zt,i, k:) for given condition A straightforward approach is to use intermediate states cat( (at((2,,) (=11) and corresponding discrete noise indices { 250 generated by existing conditional method such as FreeDoM [6). However, this method inherits the limitations of the underlying generation model, potentially compromising
data quality. Instead, we adopt aeconstruction-based strategy. Given c, we define the energy function in Equation (4) as E E(2,,c) = Z - ell2, where Z = (z, VI- dien (2,, This formulation frames conditional denoising as reconstructing 2g from noisy input 2,. By applying Equation 7 during reconstruction, we obtain training pairs (z,,; k;). Structural cues (e.g., depth or
edge maps) extracted from the reconstructed image 20 are then used conditioning inputs.

---

Table Quantitative Comparison for Controllable Generation Under Different Conditions. The best results are highlighted in bold. We use FID, CLIP, and CLIP-Ac to evaluate generation quality, and MSE, SSIM, mloU, and mAP to assess controllability. dash "_" indicates that the method does not provide a publicly available model for evaluation.
Method
Depth
Canny
HED
M-LSD
Segmentation
Normal
Skeleton
Location
Sketch
UGD
25. 1851
FreeControl
24.9604
17,9107
17.4471
30.8189
28.9317
34.7962
FID
ControlNet
19,8064
18,9211
16.1678
21.5897
20.3947
29.5817
26.1679
29.6951
24.0166
T21-Adapter
20.2817
24.8351
24.9437
AnyControl
18.6452
19.3719
19.3647
25.7867
DISCO
18.0431
21.3836
15.7542
22.0951
21.1360
27.4390
25.5083
28.8875
23.6784
UGD
0.2916
FreeControl
0.3017
0.3214
0,3039
0,3045
0.3043
0,3016
CLIP
ControlNet
0.3061
0,3085
0.3008
0.2915
0.2997
0.2987
0,3011
0.2917
0.3023
T21-Adapter
0,2990
0.3045
0,3111
AnyControl
0.3063
0.3032
0.3069
0.3089
DISCO 0,2952
0,3329
0.3035
0.2951
0.3189
0.2943
0.2920
0.2968
0.3106
UGD
5.3704
CLIP-ac f
FreeControl
5.0129
5,0010
5,0048
5,0520
5,0704
5,4050
ControlNet
5.0973
5.1213
5.1683
5.2763
5.3920
5,2437
5.2956
5,2737
5,0048
T21-Adapler
5,3498
5,1650
5,4802
AnyControl
5.1342
5.0485
4.9362
4.8970
DISCO
5.4225
5.4264
5,4575
5.4341
5.3905
5,4284
5,4794
5,3977
5,4730
MSFJ
SSIMt
SSIMT
SSIMI
mloU1
MSEJ
mAPt
mAPt
SSIMT
Controllability
UGD
0.6843
FreeControl
99.3874
0.4679
0.6159
0.7005
0,3569
0.5262
ControlNet
87.3678
0.4754
0.4836
0.7447
0.4428
0,3572
0,4332
0.3092
0,5262
T21-Adapter
89.8904
0.4786
0.5283
AnyControl
88.9634
0,5083
0.3382
0.3764
DISCO
86.6756
0.4412
0.4752
0.7793
0.3916
0.3443
0.4043
0,2563
0.6118
Experiment
This section presents a comprehensive quantitative and qualitative evaluation of our method, em- phasizing its training efficiency and enhanced alignment performance. Additionally, we include an analysis of how dimensionality and codebook size influence the final results.
5.1 Training Details & Efficiency
Unlike ControlNet-style methods 34 45 48
our discrete noise prediction network does not require
loading frozen Stable Diffusion backbone into GPU memory, which significantly improves training efficiency. Our training process completes in 3 days on a single NVIDIA RTX 3090 (24GB). In contrast, ControlNet [48.. IP-Adapter [45). and AnyControl (34 require an A6000 GPU (48GB) and 3+ days for training, according to our experiments.
We train the discrete noise prediction network for 35 epochs with a batch size of 32, using the Adam optimizer with a learning rate of 1 x 10-5 During training, both input images and condition maps are resized to 512 512. We utilize approximately 118,00 images from the COC02017 dataset [19 across all conditioning types. For human pose estimation, we use a subset of about 6,500 images specifically selected from the "person" category.
5.2 Conditional Generation
DISCO effectively integrates diverse conditioning inputs directly into the image generation process of diffusion models, enabling precise and flexible control over outputs. To demonstrate this capa- bility, we present ten single-condition examples Figure 2 covering wide range of modalities, including Canny edge 22] MLSD 12], HED 41. Skeleton [3. Location [24]. Depth 42 Nor-
mal 355,, and Segmentation (5) These examples highlight the versatility and robustness of DISCO in
accommodating broad spectrum of visual conditions.
7

---

no
afficial model
no afficial model
no
no official model
official model
no official model
no official model
official model
official model
Mo official model
official
model
Condition
UGD
FreeCtrl
ControlNet
T21-Adapter
AnyCont
Ours
Figure 3: Qualitative Comparison for Controllable Generation under Different Conditions: We compare method with five other approaches. Among them, UGD and FreeCtrl belong to the classifier-based guidance category, while ControlNet, T2I-Adapter, and AnyControl are part of the classifier-free guidance category. The results demonstrate that method is competitive with state-of-the--ar techniques.
5.3 Qualitative and Quantitative Comparison
In this section, present both qualitative and quantitative comparisons demonstrate the control- lability of DISCO. It is important to note that not all bascline methods provide publicly available implementations for handling all types of conditions, such as Depth, Canny, HED, M-LSD, Segmen- tation, Normal, Skeleton, Location, and Sketch. For conditions that are not supported by given method, we leave the corresponding entries blank.
For thorough quantitative evaluation, we adopt several metries, including FID 14] CLIP Score 23 and CLIP Aesthetic Score [28 to assess generation quality. To measure condition controllability, we use SSIM, mAP, MSE, mloU. As shown Table our method consistently outperforms existing approaches across various conditions. These results demonstrate that DISCO effectively handles diverse conditioning inputs while producing high-quality, coherent outputs aligned with both textual prompts and constraints. Overall, DISCO achieves superior performance most metrics, validating its advantage over prior methods.
Figure 3 shows examples of DISCO handling various conditions, demonstrating its ability to generate high-quality results that accurately follow the spatial constraints imposed by the conditioned inputs. The prompts for the six rows are: "a stone hut perched on rugged, rocky outcrop", "a chipmunk perched on weathered concrete surface". young woman sitting on wooden dock by the water", and "a small, curious kitten sitting on wooden windowsill". As illustrated, the generated outputs effectively fulfill the requirements of both the textual prompts and visual conditions.
majestic lion with full mane is captured in raw", "a

---

Table Quantitative comparison with sfate-of-the-ar methods on multiple conditioning modalities. Higher values indicate better performance. DISCO achieves the best results across all conditions.
Depth
Canny
HED
MLSD
Seg
Normal
Skeleton
Location
Sketch
UGD
0.12
FreeControl
0.10
0.13
0.30
0.11
0.34
0.11
ControlNet
0.21
0.28
0.33
0.42
0.23
0.27
0.14
0.37
0.41
T21-Adapter
0.22
0.11
0.25
AnyControl
0.18
0.17
0.24
0.19
DISCO (Ours)
0.29
0.31
0.37
0.58
0.30
0.39
0.31
0,63
0.59
Condition
16
K 256
K = 1024
K 4096
65536
K
131072
Condition
4
16
64
256
1024
2048
Figure 4: Conditional Generation Quality with Different Dimensionalities and Codebook Sizes K. In the first row, we fix = 64 and vary K to examine its effect on conditional generation, using the prompt "stormtrooper behind lectern." In the second row, fix K 4096 and vary with the prompt "a fluffy white Samoyed dog sitting gracefully on a lush, green lawn."
5.4 User Study
In this section, we conducted a user study to further validate the effectiveness of our method, the results of which are summarized in Table For each conditioning type, randomly selected 50 images. The outputs from different methods were presented to the participants in a random order to avoid positional bias. Participants were then asked to select the best result based on three criteria: realism, consistency with the text prompt, and consistency with the input condition.
Each participant answered three formal questions per condition, plus one validation question, resulting in a total of 28 questions. Responses that failed the validation question were excluded from the subsequent analysis. To compute the final aggregate score, we employed weighted scoring scheme across the three criteria: realism (30%). ), text alignment (30%), and condition alignment (40%).
We invited volunteers from our research group, which comprises nearly 100 students, to participate in the study. total, we collected questionnaires. After filtering out responses that failed the validation check, 72 were considered valid for the final evaluation.As shown in Table 2 our method DISCO achieves the best performance across all nine control conditions, significantly outperforming existing approaches.
5.5 Study for Dimensionality and Codebook Size
DISCO is built on the assumption that predetined codebook C.K. constructed from a zero-mean Gaussian distribution, can effectively approximate samples from a non-zero-ment Gaussian distribu- tion. This assumption relies on the existence of least one code vector in the codebook that is closely aligned in direction with the target mean-i.e -i.... the angle between the selected code vector and the mean of the non-zero-mean Gaussian distribution is small to act as reliable proxy for conditional generation. Theorem 2 guarantees that this assumption holds when the dimension n is moderately large and the codebook size K is not too small (i.e., n 64, K 4096). Here, we analyze how varying the values of n and K impacts the performance of conditional generation.

---

In the first row of Figure 4 we fix and vary K. When K is small, the generated outputs fail to align with the conditional input, as the typical maximum of cosine square similarity Vog is close meaning the codebook cannot approximate samples from the non-zero-mean Gaussian. When K is very large, Veg approaches causing the selected code vector to converge with the mean, reducing randomness in sampling and thereby decreasing output quality. In the second row of Figure E we fix K and vary n. With small n, Vex approaches I, leading to degraded quality. With large 7, Vex approaches 0, causing the codebook to fail in aligning with the target mean and making conditional control ineffective. We thus conclude that our method is robust for conditional generation only when and K within appropriate ranges, and not at their extreme values.
The configuration (n 64 4 x 4 K 4096) is not the only viable option for discrete noise control. Alternatives such as (n 16 = x 2 4, K = 32) and (n = 256 = 8 x 8 4, K = 65536) are also feasible, as defined in Definition [ However, we exclude these due training efficiency concerns. For example, = 16 requires the discrete noise prediction network to predict 1024 indices, increasing DiT's computational cost by 16x compared the 256 predictions. Conversely, larger codebook size of K 65536 significantly raises memory and computational demands relative to K 4096. Thus, we adopt (n 64, 4096) as a balanced and efficient choice.
setting, which requires only
6 Conclusion
In this paper, we present two key findings: (1) conditional generation in diffusion models can refor- mulated sampling noise from non-zero-mean Gaussian distribution; and (2) this distribution can be approximated by selecting noise vectors from a predefined codebook, constructed from standard as discrete proxies for conditional guidance. Based this discrete
Gaussian samples, which conditional generation perspective, we propose a novel approach- discrete noise prediction wwhich differs fundamentally from both classifier-based and classifier free guidance. Specifically, DiT to transform the input (z,,,, c) into noise index h; indicating the selected entry in the codebook. Experimental results demonstrate the superiority and adaptability of DISCO.
employ
7 Acknowledgment
This work supported by the National Natural Science Foundation of China (62372237.62332000 and the Major Science and Technology Projects in Jiangsu Province under Grant BG2024042.
10

---

References
[1] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal Guidance for Diffusion Models. In International Conference on Learning Representations, 2024.
[2] John Canny, A Computational Approach to Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1986.
[3] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh, OpenPose: Realtime Multi-Person
2D Pose Estimation Using Part Affinity Fields, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[4] Stanley Chan. Tutorial Diffusion Models for Imaging and Vision. Foundations and Trends in Computer Graphics and Vision, 2024.
[5] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention Mask Transformer for Universal Image Segmentation. In Conference on Computer Vision and Pattern Recognition, 2022.
[6] Hyungjin Chung. Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving Diffusion Models for Inverse Problems using Manifold Constraints. In Advances in Neural Information Processing Systems, 2022.
[7] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion Posterior Sampling for General Noisy Inverse Problems. In International Conference on Learning Representations, 2023.
[8] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion Models in Vision: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[9] Jacob Devlin, Ming-Wei, Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding. In Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthesis. In Advances in Neural Information Processing Systems, 2021.
[11] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling Rectified Flow Transformers for High- Resoluton Image Synthesis. In International Conference on Machine Learning, 2024.
[12] Geonmo Gu, Byungsoo Ko, SeoungHyun Go, Sung-Hyun Lee, Jingeun Lee, and Minchul Shin. Towards Light- and Real-Time Line Segment Detection. In AAAI Conference on Artificial Intelligence, 2022.
[13) Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J. Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold Preserving Guided Diffusion. In International Conference on Learning Representa- tions, 2024.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems, 2017.
[15] Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applications, 2021.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, 2020.

---

[17] Minghui Hu, Jianbin Zheng, Daqing Liu, Chuanxia Zheng, Chaoyue Wang, Dacheng Tao, and Tat-Jen Cham. Cocktail: Mixing Multi-Modality Control for Text- next-Conditional Image Generation. In Advances in Neural Information Processing Systems, 2023.
[18] Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Con Improving Conditional Controls with Efficient Consistency Feedback. In European Conference on Computer Vision, 2024.
[19] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision, 2014.
[20] Chong Mou, Xintao Wang, Liangbin Xie. Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models. AAAI Conference on Artificial Intelligence, 2024.
[21] William Peebles and Saining Xie. Scalable Diffusion Models with Transformers. In Interna- lional Conference on Computer Vision, 2023.
[22] Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang. Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, Stefano Ermon, Yun Fu, and Ran Xu. UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild. In Advances in Neural Information Processing Systems, 2023.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning, 2021.
[24] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You Only Look Once: Uni- fied, Real-Tima Object Detection. In Conference on Computer Vision and Pattern Recognition, 2016.
[25] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- Resolution Image Synthesis With Latent Diffusion Models. In Conference on Computer Vision and Pattern Recognition, 2022.
[26] Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models. In Advances in Neural Information Processing Systems, 2023.
[27] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5B: An Open Large-Scale Scale Dataset for Training Next Generation image-Text Models. In Advances in Neural Information Processing Systems, 2022.
[28] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models. Advances in Neural Information Processing Systems, 2022.
[29] Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu Edit: Precise Image Editing via Recognition and Generation Tasks. In Conference on Computer Vision and Pattern Recognition, 2024.
[30] Zhenning Shi, Haoshuai Zheng, Chen Xu, Changsheng Dong, Bin Pan, Xueshuo Xie, Along He, Tao Li, and Huazhu Fu. Resfusion: Denoising Diffusion Probabilistic Models for Image Restoration Based on Prior Residual Noise. In Advances in Neural Information Processing Systems, 2024.
[31] Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency. In International Conference Learning Representations, 2024.
12

---

[32] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. LoiseG Diffusion Models for Plug-am aad--ndd Control- lable Generation. In International Conference on Machine Learning, 2023.
[33] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomputing. 2024.
[34] Yanan Sun, Yanchen Liu, Yinhao Tang, Wenjie Pei, and Kai Chen. AnyControl: Create Your Artwork with Versatile Control on Text-to-lmage Generation. In European Conference on Computer Vision, 2024.
[35] Igor Vasiljevic, Nick Kolkin, Shanyi Zhang. Ruotian Luo, Haochen Wang, Falcon Z. Dai, Andrea Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R Walter, et al. DIODE: A Dense Indoor and Outdoor DEpth Dataset. arXiv, 2019.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, 2017.
[37] He Wang. Longquan Dai, and Jinhui Tang. EMControl: Adding Conditional Control to Text- to-Image Diffusion Models via Expectation-Maximizatioo In AAAI Conference on Artificial Intelligence, 2025.
[38] Yufei Wang, Wenhan Yang, Xinyuan Chen, Yaohui Wang, Lanqing Guo, Lap-Pui Chau, Ziwei Liu, Yu Qiao, Alex C. Kot, and Bihan Wen. SinSR: Diffusion-Bait -ased Image Rusor-Resolution in a Single Step. In Conference on Computer Vision and Pattern Recognition, 2024.
[39] Rongyuan Wu, Tao Yang, Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lei Zhang. SeeSR: Towards Semantics-Aware Real-World Image Super- Resplr-neooltion. In Conference on Computer Vision and Pattern Recognition, 2024.
[40] Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool. DifMRR Efficient Diffusion Model for Image Restoration. In International Conference on Computer Vision, 2023.
[41] Saining Xie and Zhuowen Tu. Holistically-Neste Edge Detection. In International Conference on Computer Vision, 2015.
[42] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao Depth anything: Unleashing the power of large-scale unlabeled data. In Conference on Computer Vision and Pattern Recognition, 2024.
[43] Lingxiao Yang, Shutong Ding, Yifan Cai, Jingyi Yu, Jingya Wang, and Ye Shi. Guidance with Spherical Gaussian Constraint for Conditional Diffusion. In International Conference on Machine Learning, 2024.
[44] Haotian Ye, Haowei Lin, Jiaqi Han, Minkai Xu, Sheng Liu, Yitao Liang, Jianzhu Ma. James Y Zou, and Stefano Ermon. TFG: Unified Training- Guidance for Diffusion Models. In Advances in Neural Information Processing Systems, 2024.
[45] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ad-Aettter: Text Compatible Image Prompt Adapter for Tmxteto-Image Diffusion Models. arXiv, 2023.
[46] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. FreeDoM: Training- Free Energy-Guides Conditional Diffusion Model. In International Conference on Computer Vision, 2023.
[47] Lvmin Zhang and Maneesh Agrawala. Packing Input Frame Context in Next-Frame Prediction Models for Video Generation. arXiv, 2025.
[48] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding Conditional Control Text-to-Image Diffusion Models. In International Conference Computer Vision, 2023.
13

---

[49] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K. Wong. Uni-ControlNet: All-in-One Control Text-to-Image Diffusion Models. In Advances in Neural Information Processing Systems, 2023.
[50] Jianbin Zheng, Daqing Liu, Chaoyue Wang, Minghui Hu, Zuopeng Yang, Changxing Ding, and Transformer for Composed Multimodal Conditional Image Synthesis. International Journal Computer Vision, 2024.
Dacheng Tao. MMoT: Mfituureoo-MModlliyy
[51] Yupeng Zhou, Daquan Zhou, Ming- Ming Cheng, Jiashi Feng, and Qibin Hou. StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation. In Advances in Neural Information Processing Systems, 2024.
[52] Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising Diffusion Models for Plug-and-PPl Image Restoration. In Conference on Computer Vision and Pattern Recognition Workshops, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state DISCO's contributions and scope. Claims align with theoretical (Section 3 and experimental results (Section 5).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [No]
Justification:
The paper does not include the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We provide the proof in the supplementary material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We offer an exhaustive discussion of our DISCO approach in Section where the theoretical underpinnings and procedural details are laid out. Additionally, experieental setups and their configurations are elaborated upon in Section [5] providing clarity on how the method's 's effectiveness is empirically validated, enabling others to replicate the results with the described methodology and datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
16

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Our intention is to make the code publicly available as open source once our paper is accepted. However, preparing the code for open source involves extensive work to facilitate the reproduction of our results by the readers. Currently, our focus entirely on the preparation of the paper, which leaves us with limited time to concurrently undertake the necessary steps to ready the code for open source distribution.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //n.ps.cc/public//////cccc Sewpudddee/oeeebbliminn
www.iiisssiooiiiooioooociiasmiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
we detail our experimental setup, including data splits, hyper- parameters, and their selection process. Upon acceptance of the paper, we will release the code, which will include all the detailed settings necessary for replication of our results.
Justification: In Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We consider that error bars essential for comparative analysis, as the focus is on qualitative and quantitative metrics that sufficiently validate the method's performance.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our training and inference processes were carried out on a 3090 GPU, as specified Section including details the type of compute resources used.
Guidelines:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't ' make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://harrrppp...cc//l/l/lllt
Answer: [Yes]
Justification:
We have meticulously adhered to the NeurlPS Code of Ethics all aspects of work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
18

---

Answer: [NA]
Justification: We introduce a conditional generation method for Stable Diffusion. Therefore, both potential positive societal impacts and negative societal impacts of the work performed are inherently addressed by the Stable Diffusion framework,
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: We propose conditional generation approach for Stable Diffusion, with all required safeguards naturally incorporated within the Stable Diffusion framework.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We utilize publicly available datasets, such as COCO2017, and cite the original sources in the paper, respecting their licenses and terms of use.
Guidelines:
19

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The methodology of this research does not involve large language models (LLMs) as important, original, or non-standard elements. The DISCO method relies on diffusion models without LLM usage in its methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standard components.
Please refer to our LLM policy (https://neure) httpr/pppp..//ooeeeeeeee/e//222 for what should or should not be described.
21


Reviews:
[{"review_id": "JIXVGyVvrI", "reviewer": "Reviewer_qShU", "summary": "The authors propose DISCO, an alternative paradigm to conditional T2I diffusion model generation compared to the current classifier/optimization-based guidance and classifier-free guidance methods. Particularly, the authors frame conditional generation as a “code selection task” from a discrete noise codebook, where the trained control network directly predicts the indices of said codes which are then used as guidance during denoising.", "questions": "1. What is the peak VRAM usage of training DISCO (vs. ControlNet, IP-Adapter, etc.)? That is often also an important bottleneck for training conditional generation models/adapters. Moreover, what is the exact speed comparison of DISCO vs. the baseline training-based methods? The RTX 3090 and A6000 are not exactly comparable regarding speed as they are not the same GPU.\n2. To clarify, would the training dataset preparation process require backpropagating through the base T2I model (and VAE)? Or does DISCO only need to do forward passes through them? (Addressing weakness 1 would help with clarifying this.)\n3. The paper frames DISCO in terms of DDPM sampling, but will DISCO (like ControlNet and other training-based baselines) work with alternative/newer sampling methods like DPS and flow-matching trained models, especially since many SOTA T2I models are flow-matching models?\n4. Since DISCO limits the selection of the guidance noise to a discrete codebook, would it potentially limit the diversity of the output images for higher resolution and more expressive T2I models (i.e., newer models after SD 1.5), especially considering training efficiency?\n\nAddressing question 2 (and thus addressing weakness 1) and providing discussions on question 3 and 4 could potentially raise my score. Also addressing weakness 2 (user study) can push me to raise my score too, though I am not expecting the authors to be able to conduct user study in the short time frame during the rebuttal.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n1. DISCO is a conditional generation method which is trained without having to backpropagate through the base T2I diffusion model, resulting in faster (and presumably more VRAM-efficient) training process, while showcasing comparable performance to existing conditional generation baselines.\n2. DISCO seems to be model architecture agnostic, meaning it can be applied to any diffusion model.\n3. The design of the discrete codebook is well-motivated.\n\n**Weaknesses**\n1. The training dataset preparation section (4.3.2) is a bit confusing and can benefit from a bit more detail. Particularly, explicitly writing out the algorithm for obtaining training data pairs $(\\boldsymbol{z}_{t, i}, k_i)$ from condition $\\boldsymbol{c}$ and clean image $\\boldsymbol{z}_0$ would be very helpful, especially since often it’s confusing whether $\\boldsymbol{c}$ refers to the text condition or image/spatial condition (e.g., depth or edge maps)—the former is usually an input into T2I models, and the latter is not.\n2. DISCO is not evaluated with a user study, especially since FID and the condition-alignment metrics are often not entirely indicative of human preference.", "was_revised": true, "final_justification": "The authors have addressed most of my questions and concerns, and I especially appreciate the addition of the user study which better aligns with human preference. I also want to thank the authors for clarifying how the training dataset is obtained and pointing me to NoiseCtrl to contextualize the motivation of the paper. I urge the authors to incorporate the rebuttal into the final version of the paper, and I will keep my final rating as Accept.", "rebuttal": "We greatly appreciate the reviewers' hard work and thank them for their valuable comments. We will address each concern in the following sections.\n\n---\n**Q: The training dataset preparation section (4.3.2) is a bit confusing and can benefit from a bit more detail.**\n\nA:  We apologize for the confusion in Section 4.3.2 and appreciate the opportunity to clarify.\n\nRecently, we found the CVPR 2025 paper *“NoiseCtrl: A Sampling-Algorithm-Agnostic Conditional Generation Method for Diffusion Models”* to offer helpful insights that align with our approach. That work shows that conditional generation in diffusion models can be effectively achieved by controlling the noise added at each DDPM step. This reframing allows the conditional generation process to be interpreted as learning to generate appropriate control noise.\n\nBuilding on this idea, our method aims to generate such control noise based on the given condition. Inspired by VQGAN, which employs a two-stage structure to generate natural images, we discretize the control noise using a codebook and model its distribution with a Transformer. More clearly, we design our training process in two stages: (1) tokenizing the control noise and (2) predicting the discretized noise tokens.\n\n1. **Noise Tokenization:** For each image in the training dataset, we identify the specific control noise $z\\_{t, i}$—selected from a predefined codebook—that, when injected into the DDPM sampling process, successfully reconstructs the original image. This control noise is discretized by our codebook.\n2. **Noise Predicting :** With the tokenized noise, we train a transformer to model the distribution of these discrete noise tokens conditioned on the input modality (e.g., pose, sketch, etc.).\n\nAt inference time, the transformer generates the control noise tokens based on the given conditioning signal. These tokens are decoded into actual noise, which is then injected into the diffusion model to guide image generation.\n\nWe will revise Section 4.3.2 in the final version to more clearly explain this two-stage dataset preparation and training procedure.\n\n---\n**Q: To clarify, would the training dataset preparation process require backpropagating through the base T2I model (and VAE)? Or does DISCO only need to do forward passes through them? (Addressing weakness 1 would help with clarifying this.).**\n\nA:  Our training dataset preparation **does not require backpropagation** through the base T2I model or the VAE. As noted earlier, our process is similar to VQGAN in that we only perform **forward passes** to extract the necessary representations. Backpropagation through the base model is **not involved** during this stage.\n\n---\n**Q: What is the peak VRAM usage of training DISCO (vs. ControlNet, IP-Adapter, etc.)? That is often also an important bottleneck for training conditional generation models/adapters. Moreover, what is the exact speed comparison of DISCO vs. the baseline training-based methods? The RTX 3090 and A6000 are not exactly comparable regarding speed as they are not the same GPU.**\n\nA:  A key advantage of DISCO lies in its significantly reduced training cost—especially in terms of GPU memory and compute time. Traditional conditional generation methods such as ControlNet, IP-Adapter, and AnyControl require an auxiliary control network appended to the diffusion model. During training, the base diffusion model must still run in the forward pass and receive gradient updates in the backward pass, resulting in substantial computational overhead and memory consumption. These methods often require high-end GPUs with 48GB of memory.\n\nIn contrast, DISCO avoids these costs by decoupling the conditional component from the diffusion model entirely. The transformer in DISCO is trained without executing the diffusion model in either the forward or backward pass, which enables much more efficient training.\n\nSince DISCO does not require loading the full Stable Diffusion (SD) model into GPU memory during training, it achieves substantial savings in both memory usage and training time. The table below summarizes the peak VRAM usage and total GPU hours, with all measurements conducted on the same hardware (NVIDIA A6000) to ensure a fair comparison:\n\n| Method    | ControlNet | T2I-Adapter | DISCO |\n| --------- | ---------- | ----------- | ----- |\n| GPU Hours | 500        | 290         | 65    |\n| Peak VRAM | 42 GB      | 31 GB       | 20 GB |\n\nThese results demonstrate that DISCO achieves significant reductions in both memory usage and training time compared to existing conditional generation methods.\n\n---\n**Q: The paper frames DISCO in terms of DDPM sampling, but will DISCO (like ControlNet and other training-based baselines) work with alternative/newer sampling methods like DPS and flow-matching trained models, especially since many SOTA T2I models are flow-matching models?**\n\nA: We appreciate the reviewer’s observation. This is indeed a valid concern, and we regret not including a discussion of this issue in the original submission due to space constraints.\n\nRecent works such as “Closing the ODE–SDE Gap in Score-Based Diffusion Models through the Fokker–Planck Equation” and “Flow-GRPO: Training Flow Matching Models via Online RL” propose effective techniques for converting deterministic ODE-based sampling processes into equivalent SDEs. These methods preserve the marginal distributions across time steps and enable reinterpretation of DDIM and Flow Matching trajectories as stochastic processes.\n\nBy leveraging these ODE-to-SDE conversion techniques, our method—originally designed for stochastic DDPM samplers—can be naturally extended to support DDIM and Flow Matching frameworks as well.\n\n---\n**Q: Since DISCO limits the selection of the guidance noise to a discrete codebook, would it potentially limit the diversity of the output images for higher resolution and more expressive T2I models (i.e., newer models after SD 1.5), especially considering training efficiency?**\n\nA:  This is an excellent question. Indeed, using a small codebook would limit the model's capacity and could result in poor image quality or even generation failure, let alone capturing diverse outputs.\n\nBased on our experiments, a codebook size of 4096 entries strikes a good balance between expressiveness and efficiency—it performs well in generating high-quality and diverse images, even with SD 1.5.\n\nHowever, for users seeking even higher fidelity or working with more expressive or higher-resolution models (e.g., SDXL or beyond), a larger codebook may be beneficial. That said, increasing the codebook size requires the transformer to have a larger output vocabulary, which in turn increases both training and inference costs—not just in terms of memory, but also latency and compute.\n\nIn summary, while the discretization inherently introduces a trade-off between diversity and efficiency, our design allows flexibility: users can select an appropriate codebook size depending on the desired quality-performance balance.\n\n---\n**Q:  DISCO is not evaluated with a user study, especially since FID and the condition-alignment metrics are often not entirely indicative of human preference.**\n\nA:  Thank you for your suggestion regarding the user study. To further validate the effectiveness of our method, we conducted a user study, the results of which are summarized in the table below. For each conditioning type, we randomly selected 50 images. The outputs from different methods were presented in random order to avoid positional bias. Participants were asked to choose the best result based on three criteria: realism, consistency with the text prompt, and consistency with the input condition.\n\nEach participant answered three formal questions per condition, plus one validation question, resulting in a total of 28 questions. Responses that failed the validation question were excluded from analysis. To compute the aggregate score, we applied a weighted scoring scheme across the three criteria: realism (30%), text alignment (30%), and condition alignment (40%).\n\nWe invited volunteers from our research group of nearly 100 students to participate in the study. In total, we collected 76 questionnaires, of which 72 were considered valid after filtering out responses that failed the validation check.\n\n|                  | Depth | Canny |  HED | MLSD |  Seg | Normal | Skeleton | Location | Sketch |\n| ---------------- | :---: | :---: | :--: | :--: | :--: | :----: | :------: | :------: | :----: |\n| **UGD**          |   –   |   –   |   –  |   –  | 0.12 |    –   |     –    |     –    |    –   |\n| **FreeControl**  |  0.10 |  0.13 | 0.30 |   –  | 0.11 |  0.34  |   0.11   |     –    |    –   |\n| **ControlNet**   |  0.21 |  0.28 | 0.33 | 0.42 | 0.23 |  0.27  |   0.14   |   0.37   |  0.41  |\n| **T2I-Adapter**  |  0.22 |  0.11 |   –  |   –  |   –  |    –   |   0.25   |     –    |    –   |\n| **AnyControl**   |  0.18 |  0.17 |   –  |   –  | 0.24 |    –   |   0.19   |     –    |    –   |\n| **DISCO (Ours)** |  0.29 |  0.31 | 0.37 | 0.58 | 0.30 |  0.39  |   0.31   |   0.63   |  0.59  |\n\nThe table reports the proportion of participants who preferred each method for a given condition. A dash (–) indicates that no official model was released for that conditioning type. As shown, our method consistently outperforms others across most conditions, demonstrating its superior controllability and perceptual quality in the eyes of human users."}, {"review_id": "FPY5HE10AJ", "reviewer": "Reviewer_RMAQ", "summary": "The paper proposes a new conditional generation method for diffusion models: DISCO. The key idea is to predict discrete noise from a pre-defined notebook under the given condition. The paper gives a theoretical framework to support its design. It also conducts empirical studies first to demonstrate selecting best discrete noise from a pre-defined notebook can generate high-quality images. After that, the authors introduce the actual method for training a DiT to predict indices of discrete noise from a predefined notebook. They also solve the challenge of data collection by viewing each conditional denoise step as predicting the original clean image. The paper compare DISCO's performance against a various set of methods such as ControlNet and show it is superior.", "questions": "See weakness. The most important questions are point 4,5,6. The paper is in general having high quality and I would advocate acceptance if these points are addressed. Other points are minor.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea is quite novel and interesting.\n2. The logic flow is clear, and the paper is in general well-written.\n3. The paper compares its method with a variety of baselines using comprehensive metrics, showing strong and convincing performance.\n\nWeakness:\n1. The paper has some obvious typos and needs polishing, such as L220 and L227.\n2. The paper lacks some definitions before using a symbol such as $c$, the conditional variable.\n3. The preliminary section is usually for introducing backgrounds, yet the paper introduces its theoretical contributions there. Maybe consider changing the title.\n4. Lack of experiment details. (1) What is the evaluation dataset used in Table 1? (2) The paper shows conditional generation results under multiple different modalities, does each modality require a different model trained? (3) when comparing with other methods, is every method using the same network structure?\n5. The scope of the method is unclear. During introduction, the paper discusses existing classifier-free and classifier-based generation methods. In method introduction, it also seems a very general conditional method. However, during experiment, only results on text+additional modality is shown. Will the model still work well if there is only text modality as the condition, and is it better than classifier-free and classifier-based method?\n6. The evaluation of efficiency is not solid. Efficiency is one of the key strength of the method as claimed by the authors, however, their discussion on the efficiency is vague (L227-L231). For example, it is not clear how much is the difference between ''3 days'' (the proposed method's training time) and ''3+ days'' (others). Besides, the requirement of training on one 24 GB GPU vs. one 48 GB GPU is not significant.", "was_revised": true, "final_justification": "I'm still concerned about point 5, the scope of the method. If the method is generalizable to text-only conditioning, the authors compared its performance with only multi-modal conditioning methods; if the scope aims at multi-modal conditioning methods only, the writing lacks some clarity and needs to be improved. Hence, I decided to maintain my score at borderline accept.", "rebuttal": "We greatly appreciate the reviewers' hard work and thank them for their valuable comments. We will address each concern in the following sections.\n\n---\n**Q: Lack of experiment details. (1) What is the evaluation dataset used in Table 1? (2) The paper shows conditional generation results under multiple different modalities, does each modality require a different model trained? (3) when comparing with other methods, is every method using the same network structure?**\n\nA: We are sorry for missing these details. \n\n1. All modalities—except for human pose—were evaluated on the full MS-COCO dataset. For OpenPose-based conditioning, we followed a common practice and evaluated the model only on the person category subset of COCO, as is widely adopted in prior work on pose-conditional generation.\n2. For simplicity, we train a separate model for each conditioning modality. This allows us to focus on modality-specific performance and avoid confounding factors during training. However, we acknowledge that training a unified model capable of handling multiple modalities is an important direction, and we are actively exploring this extension of our transformer-based approach.\n3.  All models (across different modalities) use the exact same architecture and hyperparameters; only the training data differs according to the target conditioning signal. Similarly, when comparing our method to baseline approaches, we adopt their publicly available official implementations and configurations.\n\n---\n**Q: The scope of the method is unclear. During introduction, the paper discusses existing classifier-free and classifier-based generation methods. In method introduction, it also seems a very general conditional method. However, during experiment, only results on text+additional modality is shown. Will the model still work well if there is only text modality as the condition, and is it better than classifier-free and classifier-based method?**\n\nA:  Thank you for raising this important point. We apologize for not making this aspect more explicit in the paper.\n\nOur current focus is on multi-modal conditioning—i.e., using text plus additional modalities—because existing models like Stable Diffusion already handle text-to-image generation effectively. Models such as ControlNet, IP-Adapter, and AnyControl are likewise built on top of Stable Diffusion and focus on incorporating additional non-text modalities (e.g., sketches, depth, poses).\n\nOur DISCO framework is similarly designed to enhance generation by incorporating additional modalities on top of text prompts. However, in principle, DISCO can also be applied when only text is available. In that case, we could treat the initial noise predicted by a text-guided model as the control signal to tokenize and refine with DISCO. \n\nIn summary, our method is general in scope and can, in principle, be adapted to the text-only setting. However, our current experiments focus on multi-modal conditioning beyond text alone, leveraging the Stable Diffusion model to handle the text modality.\n\n---\nQ: **The evaluation of efficiency is not solid. Efficiency is one of the key strength of the method as claimed by the authors, however, their discussion on the efficiency is vague (L227-L231). For example, it is not clear how much is the difference between ''3 days'' (the proposed method's training time) and ''3+ days'' (others). Besides, the requirement of training on one 24 GB GPU vs. one 48 GB GPU is not significant.**\n\nA: One key advantage of our approach is its significantly reduced training cost, especially in terms of GPU memory usage. Traditional conditional generation methods such as ControlNet, IP-Adapter, and AnyControl require an additional control network appended to the diffusion model. During training, the base diffusion model must still be executed in the forward pass, and gradients must propagate through it in the backward pass to optimize the control network. This results in substantial computational overhead and high memory consumption, typically necessitating GPUs with 48GB of memory. In contrast, training the transformer component in DISCO does not involve executing the diffusion model during either the forward or backward pass. This decoupling enables much more efficient training. Specifically, DISCO can be trained on a single NVIDIA RTX 3090 (24GB) in under 3 days as shown in the following table. According to our experiments, ControlNet, IP-Adapter, and AnyControl require A6000 GPUs (48GB) and also about more than 3 days to train.  We also note that the computational capability of the A6000 (38.7 TFLOPS) is higher than that of the RTX 3090 (35.6 TFLOPS).\n\n| Method    | ControlNet | T2I-Adapter | DISCO |\n| --------- | ---------- | ----------- | ----- |\n| GPU Hours | 500        | 290         | 65    |\n| Peak VRAM | 42 GB      | 31 GB       | 20 GB |\n\n---\nQ: **The paper has some obvious typos and needs polishing, such as L220 and L227.**\n\nA: Thank you for pointing this out. We have corrected these typos in our LaTeX source code.\n\n---\nQ: **The paper lacks some definitions before using a symbol such as $c$, the conditional variable.**\n\nA: Thank you for pointing this out. We have added brief definitions to clarify the meaning of symbols such as $c$, the conditional variable.\n\n---\nQ: **The preliminary section is usually for introducing backgrounds, yet the paper introduces its theoretical contributions there. Maybe consider changing the title.**\n\nA: Thank you for the suggestion. We agree that the current content goes beyond standard background material. Accordingly, we have revised the section title to better reflect its content."}, {"review_id": "zxOt0ohZOj", "reviewer": "Reviewer_BpCV", "summary": "The paper proposes a method for conditional guidance. Instead of classifier-guidance / classifier-free guidance, DISCO learns an auxiliary model for sampling the noise associated with conditional correction.\n\nThe auxiliary model predicts the conditional correction from a set of random codes given the noisy image and the conditioning. It can be trained separately from the diffusion model. The paper also explains in detail and theoretically justifies how to construct the codebook of possible noises (randomly generated and unchanged throughout).\n\nThe experimental results showcase strong performance across models and across different types of conditioning (depth, pose, skeleton, sketch etc.). It also includes ablations on the codebook sizes.", "questions": "The inference cost question above.\n\nHow does unmasking work? I understand that during training, the codebook is partially unmasked, at generation time, it starts with fully masked codes. Are all the codes unmasked in one step, or is it gradual? What is the algorithm?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe method is very flexible. It does not require a differentiable classifier model to train and most importantly, the auxiliary model can be trained independently from the diffusion model. This is key in that it enables applying a new type of conditioning to an existing diffusion model without costly pre-training or finetuning.\n\nThe paper explains how the size of the codebook is chosen, including theoretical and empirical justification. It is welcome to see such a level of detail.\n\nLimitations:\n\nThe method shows a consistent improvement in CLIP-ac, but the improvement over baselines in other metrics is less consistent.\n\nThe computational cost of the method is unclear. How does the inference cost compare to CFG (that requires two forward passes in the diffusion model)?\n\nThe method does not work for the recently popularized DDIM and FlowMatching style samplers, since they do not sample additive noise in the refinement step. Given that most recent diffusion models follow these paradigms, the impact of this work is lessened.\n\nTypos: \n\nLine 53 summery\n\nLine 227 uunlike", "was_revised": true, "final_justification": "My two questions are answered.\n\nRegarding my third point, while I understand the author's argument, more evidence is needed to back the claim that one can simply convert the ODE to an SDE, apply DISCO and see the benefits.\n\nI am keeping my score.", "rebuttal": "We greatly appreciate the reviewers' hard work and thank them for their valuable comments. We will address each concern in the following sections.\n\n---\n**Q: The computational cost of the method is unclear. How does the inference cost compare to CFG (that requires two forward passes in the diffusion model)?**\n\nA: We apologize for not clarifying this earlier. The Classifier-Free Guidance (CFG) method requires two forward passes: one for conditional generation and one for unconditional generation. The final output is essentially a linear interpolation of the results from these two forward passes.\n\nOur control mechanism differs fundamentally from CFG. Recently, we came across a CVPR 2025 paper titled \"NoiseCtrl: A Sampling-Algorithm-Agnostic Conditional Generation Method for Diffusion Models,\" which offers useful insights for categorizing our approach. In that work, the authors demonstrate that conditional generation can be achieved by directly controlling the noise added at each step of the DDPM sampling process.\n\nInspired by this perspective, our method treats the controlled noise signals as images and tokenizes them. For conditional generation, we follow a strategy similar to VQGAN, which uses a tokenizer to encode images into discrete tokens and a transformer to model these token sequences, which are then decoded back into RGB images. Similarly, we tokenize the control noise and train a transformer to predict discrete tokens, which are then interpreted as noise values injected into the DDPM steps—enabling conditional generation.\n\nOur method also involves two forward passes—one through the transformer and one through the diffusion model. However, since the computational cost of the transformer is significantly lower than that of the diffusion model, our approach is generally more efficient than CFG in terms of inference time. For example, in our experiments, ControlNet requires 6 seconds to generate an image, whereas our method only takes 4 seconds.\n\n---\n**Q: How does unmasking work? I understand that during training, the codebook is partially unmasked, at generation time, it starts with fully masked codes. Are all the codes unmasked in one step, or is it gradual? What is the algorithm?**\n\nA: We apologize for not clearly explaining this in the paper. Before addressing the unmasking process, we would like to briefly introduce the core idea behind our method. Recently, we found the CVPR 2025 paper “NoiseCtrl: A Sampling-Algorithm-Agnostic Conditional Generation Method for Diffusion Models” to offer valuable insights that closely align with our approach. That work shows that conditional generation in diffusion models can be effectively achieved by controlling the noise added at each DDPM step. This reframing allows the conditional generation process to be interpreted as learning to generate appropriate control noise.\n\nBuilding on this idea, our method aims to generate such control noise based on the given condition. Inspired by VQGAN, which employs a two-stage structure to generate natural images, we discretize the control noise using a codebook and model its distribution with a Transformer. During training, we randomly mask some of the tokenized noise vectors and train the Transformer to predict the masked parts, similar to masked language modeling.\n\nHowever, unlike VQGAN's inference strategy, which generates tokens sequentially in an autoregressive manner, our inference begins with a sequence where all tokens are initially masked. The Transformer then predicts all tokens in parallel, conditioned on the input. In other words, there is no gradual or step-by-step unmasking — the entire sequence is inferred in one forward pass.\n\n---\n**Q: The method does not work for the recently popularized DDIM and FlowMatching style samplers, since they do not sample additive noise in the refinement step. Given that most recent diffusion models follow these paradigms, the impact of this work is lessened.**\n\nA:  We appreciate the reviewer’s observation. This is indeed a valid concern, and we regret not including a discussion on this topic in the original submission due to space limitations.\n\nHowever, this limitation is not fundamental. Recent works such as \"Closing the ODE–SDE Gap in Score-Based Diffusion Models through the Fokker–Planck Equation\" and \"Flow-GRPO: Training Flow Matching Models via Online RL\" propose effective ODE-to-SDE conversion techniques. These approaches allow a deterministic sampling trajectory (as in DDIM or Flow Matching) to be reinterpreted as a stochastic process that preserves the same marginal distributions across time steps.\n\nBy applying these ODE-to-SDE conversion methods, our technique—originally designed for stochastic DDPM samplers—can be naturally extended to handle DDIM and FlowMatching frameworks."}, {"review_id": "hk3YPbtXk6", "reviewer": "Reviewer_gWpC", "summary": "This paper proposes a discrete noise guidance for Conditional Text-to-Image Diffusion Models.", "questions": "See Weaknesses above.", "limitations": "No, please discuss the trade-off between the expression capability and the efficiency if the problem exists.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1. The derivation and proof are reasonable and support the design of discrete noise guidance.\n\nWeaknesses:\n1. The novelty concern: code book is a widely used trick which can be dated back to generative models such as VQVAE.\n2. Theoretical gap: There may be a trade-off between the expression capability and the efficiency for the discrete noises. For example, the number of discrete noises can be regarded as the rank of a matrix, which represents the style space of images. Only a large number of discrete noises can cover enough image style elements while requiring more computational costs. Line 175 and equation 7 may reveal a huge cost to get a proper combination of discrete noises. Furthermore, Section 5.1 does not provide an impressive improvement on the efficiency. More analysis about training costs and inference costs are required since the efficiency is claimed as a main contribution in this paper.\n3. Marginal controllability: In table 1, the improvement of this work is limited. And there are even large gap to baseline methods on metrics such as SSIM, mIoU, and mAP.\n\nI am willing to increase my score if the authors provide additional sufficiently convincing explanations and experiments.", "was_revised": true, "final_justification": "I will keep my score unchanged.", "rebuttal": "We greatly appreciate the reviewers' hard work and thank them for their valuable comments. We will address each concern in the following sections.\n\n---\n**Q: The novelty concern: code book is a widely used trick which can be dated back to generative models such as VQVAE.**\n\nA: Using a codebook for conditional generation is the central contribution of our work. To the best of our knowledge, the concept of a codebook was originally introduced by Robert M. Gray in the early 1980s for data compression. VQ-VAE was one of the first works to adapt this idea for image tokenization in generative models.\n\nIn contrast to existing conditional diffusion models—which predominantly inject continuous signals for guidance—we are the first to propose a *discrete* approach to conditional generation by applying a codebook to the *tokenized noise* space. Moreover, while prior works typically rely on learned codebooks, our method uses randomly sampled noise vectors to construct the codebook. This represents a fundamentally different design philosophy and provides a lightweight and effective alternative for conditional control.\n\n---\n**Q: Theoretical gap: There may be a trade-off between the expression capability and the efficiency for the discrete noises. For example, the number of discrete noises can be regarded as the rank of a matrix, which represents the style space of images. Only a large number of discrete noises can cover enough image style elements while requiring more computational costs. Line 175 and equation 7 may reveal a huge cost to get a proper combination of discrete noises. Furthermore, Section 5.1 does not provide an impressive improvement on the efficiency. More analysis about training costs and inference costs are required since the efficiency is claimed as a main contribution in this paper.**\n\nA: We apologize for the confusion on this point. Recently, we came across a CVPR 2025 paper titled \"NoiseCtrl: A Sampling-Algorithm-Agnostic Conditional Generation Method for Diffusion Models,\" which provides helpful insights for classifying our approach. In that work, the authors demonstrate that conditional generation can be achieved by controlling the noise added during each DDPM step.\n\nInspired by this perspective, our paper treats the controlled noise signals as images and tokenizes them. For conditional generation, we adopt a strategy similar to VQGAN, where a tokenizer is used to encode images into discrete tokens, and a transformer is trained to model these token sequences, which are then decoded back into RGB images. Similarly, in our method, we tokenize the control noise and use a transformer to generate discrete tokens. These tokens are then interpreted as the noise added during the DDPM sampling steps, enabling conditional generation.\n\nA trade-off between expressiveness and efficiency exists in our tokenizer method—as in VQVAE and VQGAN—because all of these approaches use a finite set of discrete tokens to represent inherently continuous features. This is analogous to how floating-point numbers are used in computing to approximate real numbers in mathematics. In our experiments, we found that a codebook with 4096 entries is sufficient to effectively represent the controlled noise.\n\nDue to the similarity in their generation pipelines, the runtime cost of discrete token generation in DISCO is nearly identical to that of VQGAN. The main differences lie in the transformer architecture and the codebook size: both DISCO and VQGAN adopt BERT-like transformers, but VQGAN uses a codebook with 1024 entries, while DISCO utilizes a larger codebook with 4096 entries. Therefore, the operations described in Line 175 and Equation (7) do not incur significant additional computational cost.\n\n\n\nOne key advantage of our approach is its significantly reduced training cost, especially in terms of GPU memory usage. Traditional conditional generation methods such as ControlNet, IP-Adapter, and AnyControl require an additional control network appended to the diffusion model. During training, the base diffusion model must still be executed in the forward pass, and gradients must propagate through it in the backward pass to optimize the control network. This results in substantial computational overhead and high memory consumption, typically necessitating GPUs with 48GB of memory. In contrast, training the transformer component in DISCO does not involve executing the diffusion model during either the forward or backward pass. This decoupling enables much more efficient training. Specifically, DISCO can be trained on a single NVIDIA RTX 3090 (24GB) in under 3 days. According to our experiments, ControlNet, IP-Adapter, and AnyControl require A6000 GPUs (48GB) and also take over 3 days to train.\n\n---\n**Q: Marginal controllability: In table 1, the improvement of this work is limited. And there are even large gap to baseline methods on metrics such as SSIM, mIoU, and mAP.**\n\nA:  We appreciate the reviewer’s observation regarding performance metrics. The relatively modest results reported in Table 1 are primarily due to limited time for thorough tuning of the generation network prior to submission. Since then, we have continued refining the model over the past two months, and have observed significant performance gains across multiple metrics.\n\nThat said, we are currently unsure whether updating the table post-submission is permissible. If the reviewers find it helpful and if doing so does not violate the review guidelines, we would be glad to provide the updated results during the discussion phase.\n\nRegardless of the current numbers, we believe Table 1 already highlights the potential of DISCO. To the best of our knowledge, DISCO is the first approach to leverage discrete tokens for conditional generation within diffusion models. This opens a promising new direction for integrating large language models to control the generation process directly. We are actively exploring this line of work and are encouraged by the early results."}]

Author rebuttal:

Meta-review: The paper introduces a novel technique for conditional generation in diffusion models based on a codebook, which allows for decoupling conditional training from the diffusion model entirely. Theoretical justifications are included and appreciated.

**Strengths**: Training without backpropagation through the diffusion model, which results in reduced GPU memory usage and training time (20GB vs. 48GB RAM, 65 GPU-hours vs. 290–500 GPU-hours in baselines).

**Weaknesses**: **Codebook novelty**: Codebooks are well-known techniques (e.g., VQVAE, DALLE). However,  their use for control conditioning is novel. **Scope**: Could be  generalized to text-only conditioning; not only **multimodal** settings. **LImited improvments**: Improvements on metrics like **SSIM** and **mIoU** are modest or inconsistent across baselines. 


**Final justification:**  I find the method novel. While results are not significantly better, its training decoupling from diffusion models is a notable contribution, offering real-world benefits in terms of efficiency and scalability. 

**Discussion summary:** 
**Reviewer qShU**: Initially concerned about user study absence and unclear dataset prep. Post-rebuttal, new user study, and clarifications; maintained **Accept**. **Reviewer RMAQ**: Raised issues about experimental scope and efficiency evaluation. Authors clarified the multi-modal focus and presented comparative hardware-controlled efficiency data. Score increased from borderline to **Accept**. **Reviewer BpCV**: Appreciated flexibility and clarity but mentioned incompatibility with modern samplers. Authors addressed this issue in the ODE-SDE discussion, but the reviewer maintained the score at **Borderline Accept**. **Reviewer gWpC**: Maintained **Reject**, citing limited novelty, efficiency doubts, and weak metric gains. Authors responded with theoretical and empirical support, but this did not sway the reviewer.
Decision: Accept (poster)