Noise Consistency Training: A Native Approach for One-step Generator in Learning Additional Controls
Authors: Yihong Luo
Shuchen Xue
Tianyang Hu
Jing Tang
Venue: NeurIPS 2025 poster ()
Keywords: One-step Controllable Generation
Primary area: deep_learning

Abstract: The pursuit of efficient and controllable high-quality content generation stands as a pivotal challenge in artificial intelligence-generated content (AIGC).
While one-step generators, refined through diffusion distillation techniques, offer excellent generation quality and computational efficiency, adapting them to new control conditions—such as structural constraints, semantic guidelines, or external inputs—poses a significant challenge. 
Conventional approaches often necessitate computationally expensive modifications to the base model and subsequent diffusion distillation. 
This paper introduces Noise Consistency Training (NCT), a novel and lightweight approach to directly integrate new control signals into pre-trained one-step generators without requiring access to original training images or retraining the base diffusion model. 
NCT operates by introducing an adapter module and employs a noise consistency loss in the noise space of the generator. 
This loss aligns the adapted model's generation behavior across noises that are conditionally dependent to varying degrees, implicitly guiding it to adhere to the new control. 
This training objective can be interpreted as aligning the adapted generator with the intractable conditional distribution defined by a discriminative model and the one-step generator from moment-matching perspectives. 
NCT is modular, data-efficient, and easily deployable, relying only on the pre-trained one-step generator and a control signal model. Extensive experiments demonstrate that NCT achieves state-of-the-art controllable generation in a single forward pass, surpassing existing multi-step and distillation-based methods in both generation quality and computational efficiency.

Noise Consistency Training: A Native Approach for One-Step Generator in Learning Additional Controls
Yihong Luo'; Shuchen Xue²; Tianyang Hu Jing 411g HKUST PUCAS *CUHK (SZ) "HKUST
Abstract
The pursuit of efficient and controllable high-quality content generation remains central challenge in artificial intelligence-generade content (AIGC). While one- step generators, enabled by diffusion distillation techniques, offer excellent gen- eration quality and computational efficiency, adapting them to new control condi- tions such as structural constraints, semantic guidelines, or external inputs- poses a significant challenge. Conventional approaches often necessitate computationally expensive modifications to the base model and subsequent diffusion distillation. This paper introduces Noise Consistency Training (NCT), novel and lightweight approach to directly integrate new control signals into pre-trained one-step gen- erators without requiring access to original training images or retraining the base diffusion model. NCT operates by introducing an adapter module and employs noise consistency loss in the noise space of the generator. This loss aligns the adapted model's s generation behavior across noises that are conditionally dependent to varying degrees, implicitly guiding it to adhere to the new control. Theoretically, this training objective can be understood as minimizing the distributional distance between the adapted generator and the conditional distribution induced by the new conditions. NCT modular, data-efficient, and easily deployable, relying only on the pre-trained one-step generator and control signal model. Extensive experi- ments demonstrate that NCT achieves ttate-of-the-arr controllable generation in a single forward pass, surpassing existing multi-step and distillation-based methods in both generation quality and computational efficiency.
tions-such
1 Introduction
The pursuit of high-quality, efficient, and controllable generation has become a central theme in the udvancement of artificial intelligence-generate content (AIGC). The ability to create diverse and realistic content is crucial for a wide range of applications, from art and entertainment to scientific visualization and data augmentation. Recent breakthroughs in diffusion models and their distillation techniques have led to the development of highly capable one-step generators 2 3
These models offer compelling combination of generation quality and computational efficiency, significantly reducing the cost of content creation. Methods such as Consistency Training [6) and Inductive Moment Matching 7 have further expanded the landscape of native few-step or even one-step generative models, providing new tools and perspectives for efficient generation.
However, AIGC applications continue evolve, new scenarios constantly emerging that demand models to adapt to novel conditions and controls. These conditions can take many forms, encompassing structural constraints (c.g., generating an image with specific edge arrangements), semantic guidelines (c.g., creating an image that adheres to a particular artistic style), and external
"Core contribution.
'Corresponding authors: Tianyang Hu and Jing Tang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

factors such as user preferences or additional sensory inputs (e.g., generating an image based on depth map). Integrating such controls effectively and efficiently is a critical challenge.
The conventional approach to incorporating controls into diffusion models often involves modifying the base model architecture and subsequently performing diffusion distillation to obtain one-step student model [8]. This process, while effectíve, can be computationally expensive and time-intensive, requiring significant resources and development time. A more efficient alternative would be to extend the distillation pipeline to accommodate new controls directly, potentially bypassing the need for extensive retraining of the base diffusion model [9]. However, even extending the distillation pipeline can still be heavy undertaking, adding complexity and computational overhead. Therefore, the question of how directly endow one-step generators with new controls lightweight and efficient manner remains significant challenge.
In this paper, we answer this question by proposing Noise Consisteney Training (NCT) simple yet powerful approach that enables pre-trained one-step generator to incorporate new conditioning signals without requiring access training images or retraining the base model. NCT achieves this by introducing an adapter module that operates in the noise space of the pre-trained generator: Specifically, we define noise-space consistency loss that aligns the generation behavior of the adapted model across different noise levels, implicitly guiding it to satisfy the new control signal, Besides, we employ a boundary loss ensuring that when given a condition already associated with input noise, the generation should remain the same one-step uncontrollable generation. This can ensure the distribution of the adapter generator remains in the image domain rather than collapsing. Theoretically, we demonstrate in Section that that this training objective can be understood as matching the adapted generator to the intractable conditional induced by a discriminative control model when the boundary loss is satisfied, effectively injecting the desired conditioning behavior.
Our method is highly modular, data-efficient, and easy deploy, requiring only the pre-trained one- step generator and a control signal model, without the need for full-scale diffusion retraining or access to the original training data. Extensive experiments across various control scenarios demonstrate that NCT achieves state-of-the-ar controllable generation in a single forward pass, outperforming existing multi-step and distillation-ban based methods in both quality and computational efficiency.
Preliminary
Diffusion Models (DMs). DMs 1 operate via forward diffusion process that incrementally adds Gaussian noise data over T timesteps. This process is defined as q(x, /x) & N NXx;;; aşx, I),, where aj and 0g are hyperparameters dictating the noise schedule. The diffused samples are ob- tained via x2 a,x oee, with /N(O, I). The diffusion network, ca is trained by denoising: Ex,e,ellea (x,, xx, 1) - <Il.. Once trained, generating samples from DMs typically involves iteratively solving the corresponding diffusion stochastic differential equations (SDEs) probability flow ordinary differential equations (PF-ODEs), a process that requires multiple evaluation steps.
NontrolNet. Among other approaches for injecting conditions 100 12 13 14. ControlNet [5]
has emerged as a prominent and effective technique for augmenting pre-trained DMs with additional conditional controls. Given pre-trained diffusion model Ep. ControlNet introduces auxiliary network, parameterized by ø. This network is trained by minimizing a conditional denoising loss L(Ø) to inject the desired controls:
((d) = Ex,c,lec (x,,(x;, c)
After training, ControlNet enables the integration of new controls into the pre-trained diffusion models.
Maximum Mean Diserepancy. Maximum Mean Discrepancy (MMD 16) between distribution p(x), q(y) is an integral probability metric 17
MMD (p, q) = [x))) - Ey (v))))
where (-) is kernel function.
Diffusion Distillation. While significant advancements have been made in training-free acceleration methods for DMs [88 19
21 22]. diffusion distillation remains key strategy for achieving high-quality generation very few steps. Broadly, these distillation methods follow two primary
20
paradigms: 1) Trajectory distillation 23
24
25
26
27 28), which seeks to replicate the teacher

---

Variance-Presservee Forward Diflusion Process
Condition e
TTainabble
c Frozen
Condition
p(cpx
Adapter d
Generator
Generator
coupled with c
Boundary loss
Condition
Condition
Adapter +
Adapler +
Generator
Generator
stop
gradient
less coupled with
Ztu even coupled with
Noise Consistency loss
Framework description of our proposed NCT. We note that we deliberately added some structural features to the noise to enhance readability, rather than faithfully rendering Gaussian noise.
Figure 1:
model's ODE trajectories on an cestannc-bb-instaate basis. These methods can encounter difficulties with precise instance-level matching. 2) Distribution matching, often realized via score distillation [5] which aims to align the output distributions of the student and teacher models using divergence metrics. Our work utilizes pre-trained one-step generator, which itself is product of diffusion distillation; however, the training of our proposed NCT method does not inherently require diffusion distillation.
29 4
Additional Controls for One-step Diffusion. The distillation of multi-step DMs into one-step gener- ators, particularly through score distillation, is an established research avenue 3 5 29
However, the challenge of efficiently incorporating new controls into these pre-trained one-step generators less explored. CCM [30], for example, integrates consistency training with ControlNet, demonstrating reasonable performance with four generation steps. contrast, our work aims to surpass standard ControlNet performance in most cases using merely single step. Many successful score distillation rely on initializing the one-step student model with the weights of the B explored learning controlled one-step generators via score distillation,
techniques 3 5 31
teacher model. SDXS but their framework requires both the teacher model and the generated "fake" scores to possess ControlNet compatible with the specific condition being injected. JDM 99] minimizes tractable upper bound of the joint KL divergence, which can teach a controllable student with an uncontrollable teacher. Generally, prior works are built specific distillation techniques for adapting controls to one-step models. We argue that given an already proficient pre- trained one-step generator, performing an additional distillation for adding However, how to develop native technique for one-step generators remains unexplored. Our work takes the first step in designing native approach for one-step generators to add controls to one-step generators without requiring any diffusion distillation.
controls is computationally expensive and unnecessary.
3 Method
Problem Setup. Let z E R" be latent variable following a standard Gaussian density p(z). We have a pre-trained generator fo R R that maps a data sample x fz(z). The distribution of these generated samples has a density Po (x),, providing a high-quality approximation of the data distribution, such that (x( (x).). For any x, there specifying the likelihood of condition given x. Our goal is to directly incorporate additional control с for pre-trained one-step generator with additional trainable parameters & (e.g. ControlNet).
conditional probability density p(c[x)
More specifically, we aim to train a conditional generator fa,ç(z, c) that, when given latent code sampled from standard Gaussian distribution and an independently sampled condition e, produces sample such that the joint distribution of (x, c) matches p(x, c) (e)))x(c(x).

---

3.1 Failure modes of Naive Approaches for Adding Controls
Given pre-trained diffusion model Eg (X,, l), the adapters for injecting new conditions can trained by minimizing a denoising loss [1] 10] Hence, a natural idea for injecting new conditions into the pre-trained one-step generator is also adapting the denoising loss for training as follows:
mingd(fo,d c), x), z = aTx + OTE, C p(c|x),
where d(:;) .) is a distance metric and T denotes the terminal timestep. This approach can potentially inject new conditions into the one-step generator fe. similar to existing adapter approaches for DMs. However, it fails to generate high-quality images - the resulting images are blurry, which is due to the high variance of the optimized objective. Specifically, its optimal solution is achieved at fa.o (z. c) = E[x|z, c), which is an average of every potential image.
To reduce the variance, one may consider performing denoising loss over coupled puirs (z, x, c), where z ~ (O, I), is the condition corresponding to the generated samples x = fo(z). However, such an approach is unable to perform conditional generation given random z. This is because the model is only exposed to instances of z strongly associated with e (i.c., c ~ focl.f(z)) during its training. and never encountered random pairings of с and z.
High variance in denoising loss is also a key factor hindering fast sampling in diffusion models. Several methods have been proposed to accelerate the sampling of diffusion models, with optimization objectives typically characterized by low variance properties [26] 24 33). Among these, consistency models [26 [7] stand out promising approach instead of optimizing direct denoising loss, they optimize the distance between denoising results of highly-noisy samples and lowly-noisy samples:
min L(a) = (xt.,)); (x:..) (x),
a
where sg(-) denotes the stop-gradient operator and go denotes the desired consistency models. Similar to denoising loss, consistency loss can also force networks to use conditions; thus, can be used to train adapters to inject new conditions (30]. However, the consistency approach cannot be adapted to the one-step generator since it requires defining the loss over multiple noisy-level images, while the one-step generator only takes random noise input.
3.2
Our Approach: Noise Consistency Training
To directly inject condition to one-step generator, we propose Noise Consistency Training, which diffuses noise to decouple from the condition and operates the consistency training in noise space. Specifically, we diffuse an initial noise ~ N(0, 7) to multiple levels 2, via diffusion as follows:
Z vI o2 + OjE,
(5)
where e ~ NO, 7). This ensures that 2, also follows the standard Gaussian distribution, thus it can be transformed to the high- qush-qualty image by the pre-trained one-step generator fe.
To inject new conditions fe. we apply an adapter with parameter which transforms fe(-) that only takes random noise as input to (-. that can take an additional condition e as input. We sample coupled pairs (z, c) from po(z, c), where p(z, c) pg (c|z), and po(clz) A po(c)fo(2)). By the (z, c) pairs, we can perform Noise Consistency Loss as follows:
min EaingEp(x)ptee pp(x)((l((()) Eyter, (x),((e, (1x) E. E.-. (), (Z.....))) Ex,c/2, d(fo, [fo (Ze,, <), sg( wfg.fff..o (Z)..,, c))), #Simplified Notation
(6)
where Zr.. V 11 oï, ot., and Zr.-1 / 11 of, Ot.,, The defined diffusion process can gradually diffuse the coupled pairs (z, c) to independent uncoupled pairs (zy, c). By minimizing the distance between predictions given "less-coupled" pairs and "more-coupled" pairs, we can force the network to utilize the condition. Once trained, the consistency is ensured in the noise space. expected that the adapter & can be trained for injecting new conditions c, while keeping the high-quality generation capability in one-step. Since the optimized objective has low variance and the generator fo can produce high-quality images, the adapter just need learn how to adapt to the conditions c.

---

Lemma 1.
Define p(zolc)
p(ao)p(c/folt) and p(zzlc) [ (o)p(z))))dd The forward p(c)
diffiusion process defines an interpolation for the joint distribution p(z,,c) - p(zzlo)p(c) between p(zo. c) p(c]fo(zo)) (zz)))( and p(zr, c) p(zr )p(c).
The above Lemma provides formal justification our noise diffusion process interpolation between the coupled pairs (z, c) to independent pairs (zT, c). The proof can be found in Section A
We define the So oinn" oty 6,c) induced distribution be po.00 The proposed noise consistency loss is a practical estimation of the following loss:
Lemma 2.
N-1
(P.0 P00.0.ss))
R=0
under specific hyper-paramette choices (e.g., set particle samples to 1).
See proof in the Section A The above Lemma] 2 builds the connection between our noise consistency training and conditional distribution matching. Technically speaking, using larger particle numbers can further reduce training variance. However, in practice, we found that directly using single particle achieves similar performance and is more computationally feasible. More investigations on the effect of particle numbers can be found in Section [ This work serves proof of concept that we can design an approach native to one-step generator learning new controls, we leave other exploration for further reducing variance in future work.
Boundary Loss A core difference between NCT and CM lies the model's behavior when reaching boundaries. Specifically, for CM, when the input reaches the boundary the model only needs to degenerate into an identity mapping outputting Xg, which can be easily satisfied through reparam- eterization g to stabilize the training. NCT, however, is fundamentally different when the input reaches the boundary z, the model cannot simply degenerate into an identity mapping, but needs to map to high-quality clean images. This means this boundary is non-trivial - the network needs to learn to map z to corresponding images. Simply reparameterizing Se.s cannot fully stabilize training. To satisfy this boundary condition and stabilize the training, we propose setting the clean image corresponding to as fo (z) and implementing the following boundary loss:
min difo c), fz))) z ~~ N(O, e~ (z))
(8)
By minimizing this loss, we can ensure the boundary conditions hold and constrain the generator's output to be close to the data distribution. Intuitively, this loss is easy to understand: when the generator receives the same noise and conditions corresponding invariant. With the help of this loss, we can constrain the generator's output to stay near the data distribution otherwise, if we only minimize the noise consistency loss, the model might find unwanted shortcut solutions.
fe, its generation should be
Theorem 1. Consider a parameter set é that satisfies the following two conditions:
1. Boundary Condition: The parameters o ensure the boundary loss is zero:
Eptchptel (d(fa)/ddffff.. (2, fo fo(z)) = 0,
2. Consistency Condition: The parameters & also satisfy:
N-1
L(Ø)= = Σ (pD 00
Then fe.s maps independent p(z)p(c) to the target joint distribution pe (x))((((
See proof in the Appendix. Theorem provides theoretical insight for optimization objective, which is an empirical version for practice.
Overall Optimization We observed that the noise consistency loss is only meaningful when boundary conditions are satisfied or nearly satisfied; otherwise, the generator fo,s can easily find undesirable shortcut solutions, thus we suggest using constrained optimization form as follows:
5

---

Algorithm Noise Consistency Training
Require: Pre-trained One-Step Generator fa. Adapter ø, total iterations N
Ensure: Optimized adapter for injecting new condition.
1: for i + to N do
2: Sample noise z from standard Gaussian distribution;
3: Sample noise from standard Gaussian distribution;
4: Sample with initialized noise z from frozen generator fo, i.e., x = fo(z).
5: Sample condition corresponding to x by p(c[x).
6: # Primal Step:
7: ## Diffuse Noise via Preserrvd Diffusion
8:
#tu11 + at 11 2 + otuie é and Et, + 0r2y 2++20000 +
9:
## Compute Noise Consistency Loss
10:
Lcon t
11:
## Compute Boundary loss
12:
Lbound t- d(fff x).
13:
## Compute Total Loss and Update
14:
Dtotal f Leon + ALbound
15: Update Φ using VoLtotal
16:
# Dual Step:
Update À according Eq. 12
18:
end for
Definition (Noise Consistency Training). Given fixed margin &, the general optimization can be transformed into the following:
min Ez,c/z, (o) d(Se. (z,,, c), (Ze, ,.))))
s.t. Lbound(0) El fo, fo(z)) <E
where 2, N),,1), e~ p(clfe(z)), Zt / 11 of + or, e and Ztnt1 C.
11 11-00 o
The constrained optimization problem presented in Definition D hard to optimize directly. We therefore reformulate corresponding saddle-point problem:
max min { (Ø) + (o))und
} A20 0.
(10)
Concrete Algorithm To efficiently optimize this saddle-poing problem, employ the primal- dual algorithm tailored for the saddle-point problem, which alternates between updating the primal variables & and the dual variable A. Specifically, in the primal step, for given dual variable X, the algorithm minimizes the corresponding empirical Lagrangian with respect to & under a given dual variable X, i.e.,
Øk+1 arg min [eou (0x) + ALbound (bx)
(11)
In practice, this update for & is performed using stochastic gradient descent. Subsequently, in the dual step, we update the dual variable À as follows:
A++1 max {hx 17 - ())
(12)
where 7) is the learning rate for the dual update. Algorithm
D provides the pscudo-code for primal-dual optimization of the adapter parameters 6. In contrast to the direct application of stochastic gradient descent in Eq. (10), the primal-dual algorithm dynamically adjusts X. This avoids extra earram-prreeter tuning and provide an carly-stopping condition (e.g., À = 0). Additionally, convergence is guaranteed under sufficiently long training and adequately small step size 34

---

Table I: Comparison of machine metrics of different methods for Canny, HED, Depth and 8x Super Resolution tasks. The mark denotes our reimplementation with the same one-step generator as used in NCT.
Canny
HED Consistencyy
Depth Comiistency.
8x Super Resolution Consistancy.
FIDL
Consistency.
FID
FID!
FID
Consistencyl
ControlNet
50
14.48
0.113
19.21
0.101
18:25
0.093
11.93
0.065
15.22
0.093
DI CostrolNei
22.74
0.141
28.01
0.113
22.19
0.097
15.57
0.126
22.21
0.119
14.35
16.75
0.055
16.71
13:23
15.26
0,855
NCT (Ouns)
13467
0.110
14.96
0.060
16.45
0.088
12.17
0.053
14.31
0.078
DM - DM's
DI + DM's ControlNct, NFE
DM DM's ControlNet, 50 NFE
DI + DM's ControlNet, NFE
Control Signal
Ours, NFE
Control Signal
Ours,
ControlNct, 50 NFE
NFE
Figure 2: Qualitative comparisons controllable generation across different control signals against competing methods.
Experiments
4.1 Controllable Generation
Experimental Setup. All models are trained on an internally collected dataset. The one-step generator was initialized using weights from Stable Diffusion 1.5 [35.. Subsequently, the one-step generator was pre-trained using the Diff-Instruct 3 The ControlNet was initialized following the procedure outlined in its original publication 15 An Exponential Moving Average (EMA) with decay rate of 0.9999 was applied to the ControlNet parameters, denoted as Φ.
To evaluate the performance of our proposed method in one-step controllable generation, employed four distinct conditioning signals: Canny edges (36), HED (Holistically-Ner Edge Detection) boundaries 377, depth maps, and lower-resolution images.
Evaluation Metric. Image quality was assessed using the Fréchet Inception Distance (FID) [38 Specifically, the FID score was computed by comparing images generated by the base diffusion model without controls against images generated with the incorporation of the aforementioned conditional inputs. The consistency metric for measuring controllability is quantified between the conditioning input and the condition extracted from the generated image hxx). as formulated below:
Consistency = ||hxx) c|l1,
(13)
where h(-) represents the function used extract the conditioning information (e.g., Canny edge detector, depth estimator) from generated image x, and c is the target conditional input. Furthermore, assess computational efficiency,
report the NFE required to generate single image.
Quantitative Results. We conduct comprehensive evaluations, benchmarking our proposed approach against three established bascline methods: (1) the standard diffusion model with ControlNet: (2)
7

---

Image Prompt
Text
wearing a hat
with sunglasses
Image Prompt
Text
cat
with red clothing
Image Prompt
No Text
red hair
floerss on head
Image Prompt
No Text
playing guitur
rumming
Image Prompt
Condition
Samples
Image Prompt
Condition
Samples
Image Prompt
Condition
Samples
Figure 3: Visual samples of image-reference generations. The samples are generated by our NCT with INFE.
pre- tre-trained one-step generator integrated with the DM's ControlNet: and (3) a crafted ControlNet specifically trained for one-step generator trained via JDM distillation [9. Notably, the JDM approach necessitates an additional, computationally intensive distillation phase to incorporate control mechanisms. This step is redundant given that the one-step generator has already undergone distillation process. In contrast, our method is tailored for one-step generators, obviating the need for further distillation and thereby enhancing computational efficiency. The quantitative results, presented in Table ! assess both image fidelity (FID) and adherence conditional inputs across diverse control tasks. Our proposed method achieves a remarkable reduction in the number of function evaluations (NFEs) from 50 to I, while concurrently maintaining surpassing the performance metrics of the baselines. Specifically, our approach demonstrates superior FID scores and stronger consistency measures across various conditioning tasks, signifying enhanced image quality and more precise alignment with control conditions. These findings collectively establish that our method achieves superior trade-off between computational efficiency and sample quality in controlled image generation. delivers state-of-the-art performance with substantially reduced computational overhead and more streamlined training pipeline.
Qualitative Comparison. qualitative comparison of our method against baselines is presented in Fig. 2, comparing standard ControlNet and NI+ControlNet which does not require additional distillation. Visual results reveal that while the standard DM's ControlNet can impart high-level control one-step generators, this integration frequently results in discernible degradation of image quality. In stark contrast, our approach, which involves customized training for adding new controls to one-step generator, consistently produces images of significantly higher fidelity. These visual results substantiate the efficacy of our proposed methodology, suggesting its capability to implicitly learn the conditional distribution p(x(e) through novel noise consistency training.
4.2 Image Prompted Generation
Experiment Setting. The pre-trained one-step generator remains consistent with that employed in the prior experiments. We employ the IP-Adapter (39] architecture to serve the adapter for injecting image prompts. Following IP-Adapter, we use OpenCLIP ViT-H/14 as the image encoder..
Quantitative Comparison. Our method is quantitatively benchmarked against the original IP- Adapter. Following IP-Adapter (39], we generate four images conditioned on each image prompt, for every sample in the COCO-2017-5k dataset [40]. Alignment with the image condition is assessed using two established metrics: 1) CLIP-I: The cosine similarity between the CLIP image embeddings of the generated images and the respective image prompt: 2) CLIP-T: The CLIP Score measuring

---

Method
NFE
Clip-Tt
Clip-I1
Control
wio noise
w/o boundary
Signal
consislency loss loss
NCT (Ours)
IP-Adatter'
100
0.588
0.828
JDM
0,585
0,826
Ours
0.593
0.821
Table 2: Comparison of machine metries of different methods regarding image- prompted generation. The mark de-
notes that the result is taken from the official report.
Method
FIDI
Con.I
Ours
13.67
0.110
w/o noise consistency loss
20.56
0.165
Figure 4: Both boundary loss and noise consistency loss are crucial to our NCT. Without Boundary loss, the model's 's distribution collapses. Without noise consistency loss, the model ignores the injected condition.
w/o boundary loss w/o primal-dual
216.93
0.113
14.13
0.117
Table 3: Ablation study on proposed components in our NCT.
the similarity between the generated images and the captions corresponding the image prompts. The quantitative results, summarized in Table reveal that our Noise Consistency Training (NCT) method achieves performance comparable the original IP-Adapter (which necessitates 100 NFEs) on both CLIP-I and CLIP-T metrics. Crucially, NCT attains this level of performance with only single NFE, signifying an approximate 100-fold improvement in computational efficiency.
Multi-modal Prompts. Our investigations indicate that NCT can concurrently process both image and textual prompts. Fig. 3 illustrates generation outcomes achieved through the of such multimodal inputs. As demonstrated, the integration of supplementary text prompts facilitates the generation of more diverse visual outputs. This allows for capabilities such as attribute modification and scene alteration based on textual descriptions, relative to the content of the primary image prompt.
Structure Control. We observe that NCT permits the test-time compatibility of adapters designed for image prompting with those designed for controllable generation. This enables the generation of images based on image prompts while jointly incorporating additional structural conditional controls, as shown in Fig. 3 Such test-time time compatibility underscores the inherent flexibility and potential of NCT for training distinct adapters for one-step generator, which can subsequently be combined effectively during the inference stage.
4.3 Ablation Study
The Effect of Noise Consistency Loss. The noise consistency loss is crucial to force adapter & to learn condition c. Without the loss, it can be seen that the consistency metric degrades severely, and the generated samples do not follow the condition at all. This is because the adapter Φ is trained on fully-coupled (z, c) pairs, allowing it find find a shorteut solution that directly ignores the learnable parameters to satisfy the boundary loss.
The Effect Boundary Loss.
The boundary loss can constrain the output of the generator fe.e in the image domain. Without the loss, although the generator can still learns some conditions, its generated samples entirely collapse as indicated by the FID and visual samples.
The Effect of Primal-Dual. We use primal-dual since it is crafted for solving the constrained problem, while it owns theoretical guarantees and dynamically balances the noise consistency loss and boundary loss. We empirically validate its effectiveness, it can be seen that without primal-dual, the performance slightly degrades regarding both fidelity and condition alignment.
Conclusion
This paper addressed the critical challenge of efficiently incorporating new controls into pre-trained one-step generative models, a key bottleneck in the rapidly evolving field of AIGC. We introduced Noise Consistency Training (NCT), novel and lightweight approach that empowers existing one-step

---

generators with new conditioning capabilities without the need for retraining the base diffusion model or additional diffusion distillation or accessing the original training dataset. By operating in the noise space and leveraging carefully formulated noise-space consistency loss, NCT effectively aligns the adapted generator with the desired control signals. Our proposed NCT framework offers significant advantages in terms of modularity, data efficiency, and ease of deployment. The experimental results across diverse control scenarios robustly demonstrate that NCT achieves state-of-the-a performance in controllable, single-step generation. It surpasses existing multi-step and distillation-base methods in both the quality of the generated content and computational efficiency.
Acknowledgments
Jing Tang's s work is partially supported by National Key R&D Program of China under Grant No. 2024YFA1012700 and No. FF0725000,, by the National Natural Science Foundation of China (NSFC) under Grant No. 62402410 and No. U22B2060, by Guangdong Provincial Project (No. 20230N10X025), by Guangdong Basic and Applied Basic Research Foundation under Grant No. 2023A1515110131, by Guangzhou Municipal Science and Technology Bureau under Grant No. 2024A0414454, by Guangzhou Municipal Education Bureau (No. 2024312263), and by Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628) and Guangzhou Municipal Key Laboratory of Financial Technology Cutting-Edge Research (No. 2024A0310630).
References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[2] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[3] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2023.
[4] Yihong Luo, Xiaolong Chen, Xinghua Qu, Tianyang Hu, and Jing Tang. You only sample once: Taming one-step text-to-image synthesis by self-cooperative diffusion gans, 2024.
[5] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18822 2023.
[6] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023.
[7] Linqi Zhou, Stefano Ermon, and Jiaming Song. Inductive moment matching. arXiv preprint arXiv:2503.07565 2025.
[8] Xuanwu Yin Yuda Song. Zehao Sun. Sdxs: Real-time one-step latent diffusion models with image conditions. arxiv, 2024.
[9] Yihong Luo, Tianyang Hu, Yifan Song, Jiacheng Sun, Zhenguo Li, and Jing Tang. Adding additional control to one-step diffusion with joint distribution matching. arXiv preprint arXiv:2503.0666 06652, 2025.
[10] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08455 08455, 2023.
[11] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12594 2022.
[12] Arpit Bansal, Mong-Min Chu, Avi Schwarzschild, Roni Sengupta, Micah Goldblum, Jonas Geip- ing, and Tom Goldstein. Universal guidance for diffusion models. In The Twelfth International Conference Learning Representations, 2024.
10

---

[13] Jiajun Ma, Tianyang Hu, Wenjia Wang, and Jiacheng Sun. Elucidating the design space of Mlassifiergguide diffusion generation. arXiv preprint axX1:2310.113112222. 2023.
[14] Yihong Luo, Tianyang Hu, Weijian Luo, Kenji Kawaguchi, and Jing Tang. Reward- instruct: reward-centric approach to fast photo-realistic image generation. arXiv preprint arXiv:2503. 13070, 2025.
[15] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 3836-3847, 2023.
[16] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[17] Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in applied probability, 29(2):429 1997.
[18] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 2022.
[19] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36:49842 49869, 2023.
[20] Shuchen Xue, Zhaoqiang Liu, Fei Chen, Shifeng Zhang, Tianyang Hu, Enze Xie, and Zhenguo Li. Accelerating diffusion sampling with optimized time steps. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8292-8301, 2024.
[21] Chenyang Si, Ziqi Huang. Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4733-47453 2024.
[22] Jiajun Ma. Shuchen Xue, Tianyang Hu, Wenjia Wang, Zhaoqiang Liu, Zhenguo Li, Zhi-Ming Ma, and Kenji Kawaguchi. The surprising effectiveness of skip-tuning in diffusion sampling. arXiv preprini 240. 2024.
[23] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388 2021.
[24] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14297-14306, June 2023.
[25] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
[26] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
[27] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations, 2024.
[28] Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and-play accelerator. arXiv preprini arXiv:2405.07510, 2024.
[29] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In International Conference Machine Learning, 2024.
11

---

[30] Jie Xiao, Kai Zhu, Han Zhang, Zhiheng Liu, Yujun Shen, Zhantao Yang, Ruili Feng, Yu Liu, Xueyang Fu, and Zun Zha. CCM: Real-time controllable visual content creation using text-to-image consistency models. In Forty-first International Conference on Machine Learning, 2024.
[31] Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T. Freeman. Improved distribution matching distillation for fast image synthesis, 2024.
[32] Yihong Luo, Tianyang Hu, Jiacheng Sun, Yujun Cai, and Jing Tang. Learning few-step diffusion models by trajectory distribution matching, 2025.
[33] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. arXiv preprint arXiv:2309.063880 2023.
[34] Luiz FO Chamon, Santiago Paternain, Miguel
and Alejandro Ribeiro. Con- strained learning with non-convex losses. arXiv:2/03.05/144 2021.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[36] John Canny, A computational approach to edge detection. PAMI, 1986.
[37] Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In ICCV, 2015.
[38] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[39] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023.
[40] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
12

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our work proposes NCT, which can directly add new controls to one-step generators without additional diffusion distillation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
13

---

Justification:
We discuss the limitations in the Appendix C
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Necessary information provided in main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
14

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The experiments are conducted in internally collected datasets. We have described how to form the training data and provided pseudo-code of our method.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/publcc/eee.c wttp://ddd/e/oeeuummieeeeeee ssi ssionyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
15

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We have specified them in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: error bar is not reported, since it is too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reported them in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
16

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.. http://cccccppcccccccictccaeeea
Answer:
[Yes]
Justification:
We follow the Code of Ethics from all the perspectives stated.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed this in Appendix D
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: We discussed the Appendix
Guidelines:
The answer NA means that the paper poses such risks.
17

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing assets used in this paper have been properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets in the submission phase.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
18

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy /ettps://neurr cc/ //ooofferencee/2225555..... for what should or should be described.
19

---

A Theoretical Foundation of Noise Consistency Training
This section establishes the theoretical foundation: begins with definitions and the mathematical setup (A.1), then introduces key lemmas (A.2) that collectively build the necessary mathematical foundation by defining critical distribution relationships and an input interpolation path for formulating the conditions and proving the main theorem (A.3). Specifically, Lemma] [ Lemma 2 and Theorem presented in the main paper are proved Lemma A.3 Remark and Theorem A. respectively in this section.
A.1 Definition and Setup
Latent Distribution: The latent distribution I is the standard Gaussian measure
( B((R))). The measure I has density 7(2) w.r.t dz, so dl = (2)dz. is
probability measure: JEm 72))dz
Implicit Generator: f8 m + R is measurable function.
Data Distribution: Po (RR", B(B")) is the push-forward P. = I It has density p(x)
w.r.t. dx, so dPo (x) = pa (x) Since I' is probability measure, Pe is also probability
measure: JR- Po (x) 1.
Condition: Let (C, Bc, (4c) be measure space for the conditions. Be o-algebra on C, and Me is a reference measure (e.g., Lebesgue measure if C R (such as Canny Edge), or counting measure if C is discrete) (such as class labels). For each x E R" p(- x) is probability measure (C, Be). We assume it has density p(c|x) with respect to pe. Thus, for any x E R": ((pccx)dueel
1.
Combined Map: T fe id Rm x C - R" C, T(z, c) (fe(z), c). Since fo and id are measurable, B(R") & Be.
measurable with respect to the product o-algebras ((R")) & Be and
Implicit Generator with Condition: fa.p R. x C - R be measurable function.
Combined Map with Condition: We define a new map To T x C + R x as (z(2,c) c) (2,,6(2,c).. c),
Marginal Condition Density: We define the marginal probability density p(c) of the condition c as:
p(c) / p=(x)p(c/x)dx = -f ((=)p(clfo()))dz
This a probability density with respect to μe. i..., Se p(c)duc(c) = 1. Initial Coupled Latent-Condition Distribution: Density ((2,c) = (z)p((c(f((z)) w.r.t.
(c)
Independent Latent-Condition Coupling: We define the probability measure the input space (I x C, (RR") Be) by its density with respect to the reference measure dz (c)c
dp(z,c) c) = (z))p(c)dz duc(c)
Here, ((2) the density of the standard Gaussian measure T on T.., The measure p corresponds to sampling z ~ independently from sampling p(c).
Target Data-Condition Distribution 77: Density (x, c) pe(x)p(()x) w.r.t. dxdue(().
MMD (Maximum Mean Diserepancy): MMDD2(P, (P,QQ. Q) is metric between probability
distributions P and Q. For characteristic kernel, MMD
(P.Q) MMM2(P..Q)
= P=Q.
A.2 Lemmas
Lemma A.1. Let I' be the standard Gaussian measure on I with density 7(z) with respect to the Lebesgue measure dz. Let fo 1 - R be a measurable function, and let Pg = so be the forwaforward measure on R2, assumed to have density po(x) with respect to the Lebesgue measure dx. Let (C, Be, HC) be a measure space for conditions, and let p(c]x) be conditional probability density on C with respect pe for each R", such that fe (((cl))dpcce = 1.
20

---

Define the measure on I x C by its density with respect to dzdpe(c):
dz c) =(())(ellfo(2))dddddc((
Define the map T = Jo RD x C R x C by ( c) = (fo(z),c).
Then the push-forward measure T v on R xC has the density (x())p(clxx) with respect to dxd That is,
(fo id) # (7()) (zdduccc() p=(x)p(c(x)xxxxxc(
Proof. We want to show T 7/. By definition of equality of measures, it suffices show that for any bounded, measurable test function + I x R:
[ xr
((x, ((x,c)d(Txu)(xxc)
fx-xc ((x,c)dd)(x,
We start with the left-han left-hand side (LHS). Using the change of variables formula for push-forward measures:
LHS
F(x, ((xxc)d((Tv))xx.c)
R
((((2cc))du(z,c)
(Change of Variables)
Rm
[ Jemxc xC
(Substitute T and density of v)
7(2) | Bm
dz (Fubini's Theorem)
The application of Fubini's theorem is justified because & is bounded, y(z) Z 0, p(c)fo(z)) 2 0, and is finite (probability) measure.
Let's define an auxiliary function g R as:
g(y) I o(y, c)p(elydddec)(
Since & is bounded (say | M) and Se p(cly)duc(c) 1, g(y) is also bounded (1g(y) S M). If Φ is B(R") 2 Be-muuuurable and p(cly) defines a measurable transition kernel, then is B(R")- measurable.
Substituting into our integral expression:
LHS [ 8m
((fo(z)))(zddz
Now, recall the definition of the push-forward measure Po I. For any bounded, measurable function h : I + R:
[ h(x)dP(xx) / h(folz))d((z)
In terms of densities:
M())poxx)xx / ((folz))
Applying this identity with h = g:
f g(fa(z))(())dz / (xx))o(x)dd)
So.
LHS g(x))o(xx)dx
Now, substitute back the definition of g(x):
LHS = { [ 2 (((xxxx)(ccx)ddcc()) po (x)dx
21

---

Applying Fubini's Theorem again (justified as before):
/ ((x, b(xxc)no((2pplekxxxxxeeee
LHS
JR"X
This is precisely the integral with respect to the target measure n)
- Jaax ((x,c)dn(x,c)
LHS
Since have shown that (T)(T = Φdn for all bounded, measurable test functions +, the measures must be equal:
Tgv =
Lemma A.2 (Boundary Loss). Let fo.o Ih x C + R be a measurable function. Let v be the measure T C with density > (Z))((((f(zz)) w.E. dzdpe(c). Let d be distance metric IR.. If the boundary loss
(dac)ruld([oobbl So(z)) =
then:
The push-forward measure nh (To) is equal the target measure 1j Tœv. This means the joint distribution po.a c) induced by fa.s is po (()c(x) i.e., 17 = 1)-
Proof. The condition is (x,,,))> (else))w(((food (((fo,o(z, c), fo()] = Since d(a, b) 0 for any a, b E R and d(a, b) 0 if and only if a = 6, the expectation of this non-negative quantity being zero implies that the integrand must be zero v-almost everywhere. That is,
d(fe.o(z.c) fo(z)) for.e (z,c)
This implics
f0, fa(z)) for v-a.e (z,c)
We want to show that 1/ (To)# #2 equal to 1) = Twv. Recall the definitions of the maps:
T(z,c) c) = (f)(z).c)
Tø c) (fe.s (z, c)
Since (2,) = fz( for v-ae. (z,c), it follows directly that the maps To and T are equal p-almost everywhere:
To(2,c) = (Jo.o = (fo(z),c) = C(z,c) for v-a.e (z,c)
two measurable maps T and To are equal v-a.e., their push-forward measures T&x and (Tx) H are identical. Let V RRDh C - IR be any bounded, measurable test function.
[ J v(x, ((xxce)((TT))))((xe)
V(Tolz,c))duez.c)
RM
((((z,c))dr(zz.c) (since T6 T v-a.e. and V is bounded)
Rm xc
v(x,c)d([=r((x.c)
Since this holds for all bounded measurable V, we have (To) Twv. From Lemma 1, we know Ty 7) where r) has density po (x))(((c(x) with respect to (x) Therefore, ns = (Tx) #
11
Lemma A.3 (Interpolation Joint Latent-Condition Distributions (Lemmal) i in main paper)). Let (z) be the density of the standard Gaussian measure Ip. Let fo Rh" R be measurable function. Let p(c)x) be a conditional probability density on (with respect to a reference measure (c) for each R Define the marginal condition density p(c) as:
p(c) = =fo 7(2))p(c) (zd)))z
22

---

Assume p(c) > foror pe-almost every in the support of interest. Define the conditional latent density Pduma (Zo (c) as:
Pdotol (zolc) ((z0)p(c((00zzo)
p(c)
Consider a time-dependenn process for E [0, 1] where z, generated Zo ~ Paoor(-1o by: Ze 0z + ore,
where e ~ N(0, Im) independent of Zo and c.
The coefficients o₁, 0 R satisfy:
Okn 10
. 0j 0, 01 =1
* is monotonically decreasing, dN is monotonically increasing.
ozZ0, of Let (z(ze/z)) = 0170 Im) be the density of given Zg. Define the conditional density
Pr [z(z(c) as:
Pt (z|c) = = =ff- q:(2) (oal)z)))aae Prinaa (Zol )d)o
JEm
And the joint density cz) on R x C (with respect to dzdjec()) as: pı(z,c) = p((zc))((c)
Then,
1. At t 0, the joint density is po(z, c) (22)((c]5((z))
2. At t = 1, the joint density is pi(z, c) = -/))p((c)
The joint density at time is given by (z pz(cc)pp(c) Substituting the definition of pr(z(c):
Proof.
c) p(c) [ N(2; of Im Pamm (z0 2ole))za
Now, substitute the definition of Paamaa (zolc) f(mo)pl
p(c)
(e c) = p(c) / N(z 1 (())) (zo)) dzo
p(c)
Rm
Assuming p(c) + 0 (for pe-aee c), we can cancel p(c):
(z m.
N(z; aizo,
At O, we have ag 1 and dp The Gaussian density N an mg becomes ((2: zo, O I This is interpreted as the Dirac delta function 5(z - zp). So,
po(z,c)= = 8(z (o))(2)))(((ff(())))))
= ==(z)p(c)ff(() (by the sifting property of the Dirac delta)
This matches the first target distribution.
At = 1, we have 0n = 0 and aR = 1. The Gaussian density N(z; 0:100 mm) becomes N(2; 0 2o. 12 (2; 0, I. By definition, (2 0, Imm (2). So,
pi (z(z,o) c) / (2))
=7(z) / ((z))((c (()
The integral fg.. ((z))p(clff((z)))dd is. by definition, p(c). Therefore, (z c) (z))p(c)
This matches the second target distribution.
Thus, the process defines an interpolation for the joint density (z( c) between po(2,c) (z))p(c).f(z)) and pı (z, c) (z))p(c).
23

---

A.3 Main Theorem and Proof
Definition 1 (Interpolation Distribution Sequence (from Lemma 3)). A sequence of time points 0 = to < 11 <... tN = L. For each the we have a latent-condition distribution Vt. (density Pt, c)) such that Vta = vo and Vix = p.
Theorem A.I. Assume the distributions v); vo: p and the interpolation sequence {ut, IN Ik= as defined above. Let So m - " be pre-trained generator, and Jo.o R x C - IR be conditional generator with a single set of trainable parameters dh The map T is defined as Ta(2,c) (fe,ol,,c),c) c).
Consider the following two conditions:
1. Boundary Condition: The parameters & ensure the boundary loss is zero:
E(r.c)rio [d(fo. fol)))
where d(.,-) is a distance metric on RR. By the Boundary Loss Lemma, this implies uTo) =17. 17.
2. Consistency Condition: The parameters o also satisfy:
N-1
Liotal
((T))((T))##H)))) (To)
K=D
If such a parameter set & exists and satisfies both conditions above, then fo. (when its input is distributed according to generates the target data-conding distribution ny:
(To) =1
That is, if c) (i.e., z 7(-) and independently pl-)), then (fa,ozz,c), n (i.e., its density is po (x)p(()))).
Proof. Let è be parameter that satisfies the two conditions stated in the theorem.
The first condition is E(z,et>er [d(So,o (2, (2, fo())] = 0. Recall that Vto is the distribution with density ((x)p(c)fo(z)) According to the Boundary Loss Lemma (Lemma 2), this zero loss implies that the push-forward measure (Tp) (T)) is equal the target distribution 7/. So, (T²) tto = 7)-
The second condition Σk MMDD ((To) (Tp) y v O. Since MMD 2 (P.Q) 2 0 for any probability distributions P. Q. for the sum of non-negative terms to be zero, each individual term the sum must be zero. Therefore, for each E (0, 1,..., N 1}:
MMD2( Mre vv)) =
Assuming MMD is based on characteristic kernel, MMD (P,Q) (P,D2(P,,)) for each (0, 1, .......... 1}:
0 if and only if P = Q. Thus,
(To) = (To)
The result from step 2 implies chain equalities for the push-forward measures generated by T from the sequence of input distributions V4,:
(To) Mtx (To)
=To)
(To) (To)
So, have (To) (To) From the first condition, we established that (Tx) Vto 7). Substituting this into the equality chain:
(T 4 Vix 17

---

From Lemma A.3 we know that Vtx (which corresponds to pz c) at = ty = 1) is the independent Iutent-condition distribution p. The density of p pplz.c) 7())p()). Substituting Vty p:
(T #
This is the desired conclusion. If (z, c) is sampled from (meaning z ~ 7(-) independently of с ~ p(-) and then transformed by To (i.e., forming (fa,o(z, c)), the resulting distribution the larget data-condition distribution (which has density pe (x)
Remark 1 (Lemma 2 in main paper), Specificully, when we take N (particle number) in MMD loss, and take we have some specific kernel choice:
k(z, y) -1 yl1? although it is not a proper positive definite kernel required by MMD. we find it works well in practice
k(æ, v)
VI y/1² + c is conditionally positive definite kernel.
Then the summed MMD Loss
( Σ
MMD? (1tg) (To)
km=
can be implemented in a practical way:
Liotal (0)
Σ (ereell fa(e)) Bnw, 2 0 c)2 So (an on o w,,,,),,
where d is loss or pseudo-hubie huber loss, other kernel-induced losses also work.
B Experiment Details
One-step generator We adopt Diff-Instruct 3 for pre-training the one-step generator. We adopt the Adam W optimizer. The 31 to be and the 33 is set to 0.95. The learning rate for the generator is 2e 6, the learning rate for fake score is 1e 5. We apply gradient norm clipping with a value of 1.0 for both the generator and fake score. We use batch size of 256.
Controllable Generation We use Contorlnet's 's architecture [15 for training. We adopt the A optimizer with 31 0.9, 32 = 0.95, and the learning rate of le 5. We use batch size of 128.
Image-prompted Geneartion We IP-adapter's architecture [39 for training. We adopt the AdamW W optimizer with 31 0.9, 8n = 0.95, and the learning rate of le We use batch size of 128. We use probability of 0.05 to drop text during training.
C
Limitations
Our model shares common challenges with other controllable text-to-image diffuusio models, particu- larly regarding fairness considerations and precise detail handling. We plan to explore these ongoing challenges in the generation domain in future works, improve the model's performance in text synthesis, fairness, and fine- -raingd control.
D Broader Impacts
This work presents NCT, a method that can inject new controls into pre-trained one-step generators. From positive perspective, this academic contribution has potential for widespread industrial adop- tion, where its computational efficiency could reduce energy consumption and provide environmental advantages. Conversely, malicious of such rapid generation technologies could facilitate the faster production of harmful content. While our focus remains on scientific advancement, we are committed mitigating risks through measures such removing harmful material from training datasets.
25

---

E Safeguards
The NCT is trained on an internally curated dataset that has undergone rigorous human and machine- based filtering to exclude harmful or violent content.
26


Reviews:
[{"review_id": "gHJTfNbPRw", "reviewer": "Reviewer_MrHE", "summary": "This paper introduces Noise Consistency Training (NCT), a novel method for integrating new control signals into pre-trained one-step generative models. The core contribution is to achieve this without the original training images or retraining the base diffusion model. The methodology involves training an adapter module with two losses: a noise consistency loss, which enforces that the generator's outputs remain consistent for noise vectors sampled along a diffusion trajectory, and a boundary loss, which acts as a regularizer to ensure the adapted model's output distribution does not collapse. By enforcing this consistency objective, NCT theoretically works to minimize the distributional distance between the adapted generator and the target conditional distribution. The authors demonstrate through extensive experiments that this approach achieves state-of-the-art controllable generation in a single forward pass, claiming superiority in both image quality and computational efficiency over existing multi-step and full distillation-based methods.", "questions": "1. I suggest adding something indicating noise injection in Figure 1. Its current version is not very intuitive to understand, and all 4 noises look exactly the same while there are 3 levels.\n2. Is the primal-dual algorithm really necessary? Have you tried sweeping different values of the dual weight? Related question: What is the computational overhead of the primal-dual algorithm, compared to just primal ($\\lambda$ is a hyperparameter)?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n1. The proposed method, to the best of my knowledge, is novel, and it provides a practical, data-efficient, and modular solution to an increasingly relevant problem.\n2. The idea of \"noising\" and \"denoising\" the initial Gaussian noise is interesting.\n3. The experimental results seem quite promising, and the figures are all well-made.\n\n**Weakness**\n1. I feel the writing of the paper can be improved. For example, in the main text of the paper, it is not clearly explained the notion of $p(c|x)$, which throws me off in the experiment section. I was confused about how the model learns the new control signals when the alignment is with the original model. I hope I understand it correctly, and we assume we have online access to turning samples into the underlying conditions. This brings me to point 2.\n2. What if the control signal is data-driven? Specifically, if we assume that we have access to paired data of conditions and samples but no model outputting conditions given samples, we could still do ControlNet then distillation, but the proposed method here will not work, correct?\n3. Technically, this is not a weakness, but the need for an additional boundary loss may be concerning.", "was_revised": true, "final_justification": "I do not have more questions, and I think it is an interesting work. Since I am not an expert on training diffusion models to obey control, I give a borderline assessment with low confidence.", "rebuttal": "We sincerely thank you for your time and effort in reviewing our paper. We address each comment below.\n\n> It is not clearly explained the notion of $p(c|x)$. Do we assume we have online access to turning samples into the underlying condition?\n\nThe $p(c|x)$ is indeed for sampling the underlying condition $c$ regarding $x$. This is for constructing the triplet $(\\epsilon,x,c)$ for conducting NCT's training, where the diffusing process in noise space is performed on $(\\epsilon,c)$ and the boundary is defined by $(\\epsilon,x)$.\n\n> What if the control signal is data-driven? Specifically, if we assume that we have access to paired data of conditions and samples but no model outputting conditions given samples, we could still do ControlNet then distillation, but the proposed method here will not work, correct?\n\nThanks for the great question.\nWe note that the training of NCT requires triplets $(\\epsilon,x,c)$.\nHence, our method can be extended to the data-driven case by a minor modification: train an encoder to invert $x$ to noise $\\epsilon$, then formulate the triplet $(\\epsilon,x,c)$ for performing NCT's training. We note that training the encoder may be expensive and difficult for diffusion models, but it is easy for a one-step generator. In particular, the encoder can be trained by minimizing reconstruction loss: $ \\mathrm{Min} _ { \\mathrm{Enc} \\in \\mathcal{F}} | | f_\\theta(\\mathrm{Enc}(x)) - x | | _2^2$. \nWe train an encoder initialized from DMs in just 5,000 iterations. We report NCT trained by data-driven control signal as follows:\n\n| Method                      | FID↓  | Con.↓ |\n| --------------------------- | ----- | ----- |\n| Ours w/ data-driven signals | 14.03 | 0.108 |\n| Ours                        | 13.67 | 0.110 |\n\nIt can be seen that the data-driven NCT can also achieve promising results.\n\n> I suggest adding something indicating noise injection in Figure 1. Its current version is not very intuitive to understand, and all 4 noises look exactly the same while there are 3 levels.\n\nGreat suggestion. We will improve the Figure 1 in the revision. Specifically, we will add some structural/semantic features to the noise to enhance readability, rather than faithfully rendering standard Gaussian noise.\n\n> Is the primal-dual algorithm really necessary? Have you tried sweeping different values of the dual weight? Related question: What is the computational overhead of the primal-dual algorithm, compared to just primal ( $\\lambda$ is a hyperparameter)?\n\nWithout primal-dual can also work reasonably well as shown in Table 3. The major advantage of primal-dual is that it can enable the dynamic tuning of $\\lambda$. Notably, it can allow $\\lambda$ to be zero, when the boundary is nearly satisfied. This can not be achieved by any fixed $\\lambda$ value (except zero).\n\nThe computational overhead is negligible, since we just need to compute $\\lambda_{t+1} = \\mathrm{max}[{\\lambda_t + \\eta (L_{con} - \\xi),0}] $ without additional network forward and backward, which is highly efficient."}, {"review_id": "54ItBPG5aM", "reviewer": "Reviewer_1vDf", "summary": "This paper addresses the significant challenge of efficiently integrating new control signals into pre-trained one-step generation models. Existing approaches have had the disadvantage of high computational costs, requiring modifications to the base model and performing additional diffusion distillation. To solve these problems, the authors propose Noise Consistency Training (NCT), a novel and lightweight approach. NCT operates by adding adapter modules to pre-trained one-step generators without requiring retraining of the original training data or the base diffusion model. The key idea is to define a 'noise consistency loss' in the generator's noise space. This loss function ensures that the adapted model's generation results are consistent across noises with different conditional dependencies, thereby implicitly learning new control signals. Additionally, they use a boundary loss alongside this to stabilize training and prevent generated images from deviating significantly from the data distribution. The authors claim to experimentally demonstrate that NCT is modular and data-efficient, achieving state-of-the-art controllable generation with only a single forward pass, surpassing existing multi-step and distillation-based methods in both quality and efficiency.", "questions": "1. On a Fairer Comparison with ControlNet: The proposed NCT methodology appears to be applicable not only to one-step generators but also to standard multi-step diffusion models. To provide a more direct and fair comparison, could you apply NCT on a standard diffusion model (e.g., Stable Diffusion 1.5) and benchmark it against the original ControlNet using the exact same base model and evaluation settings (e.g., 50 NFEs)? The current comparisons in Table 1 are challenging to interpret on a one-to-one basis, as they compare models with different base architectures and step counts. This new experiment would help isolate the performance of the training methodology itself.\n\n2. On the Lack of Qualitative Comparisons: The paper relies heavily on quantitative metrics but lacks crucial qualitative comparisons against the most relevant competing methods. While quantitative scores are important, they do not fully capture perceptual quality, the subtlety of control adherence, or potential failure modes. Could you please provide side-by-side visual comparisons of your method's outputs against those from other one-step controllable generation models, such as JDM? Such qualitative evidence is necessary to substantiate the claim that your method achieves state-of-the-art performance.\n\n3. On the Generalizability of NCT: The experiments currently focus on structural control signals, such as Canny edges and depth maps. Have you explored the efficacy of NCT with non-structural or stylistic control signals, for instance, controlling for a specific texture or an artistic style? Could you elaborate on how the core assumption of NCT—that diffusing noise progressively decouples it from the condition—is expected to hold for features that are less spatially robust? For such controls, even minor perturbations in the noise space might lead to significant semantic deviations, potentially challenging the consistency objective.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n**Originality and Significance** : This paper addresses a highly important and practical problem in the field of Artificial Intelligence Generated Content (AIGC). Utilizing efficient one-step generators, born from the distillation of diffusion models, with additional controls is essential for many applications. The proposed Noise Consistency Training (NCT) method is original in that it performs consistency learning directly in the noise space, unlike conventional consistency models that operate in the image space. This approach offers significant academic contributions by presenting a new avenue for adding controls directly to pre-trained models, thereby avoiding expensive retraining or distillation processes.\n\n\n\n\n\n**Clarity** : The paper is well-written and clear overall. The problem definition, proposed method, and experimental design are logically well-connected. In particular, the core idea of NCT—progressively decoupling the condition by diffusing the noise and optimizing for consistency between generation results from different noise levels—is intuitive and easy to grasp. The process of introducing a primal-dual algorithm to solve the constrained optimization problem is also clearly explained.\n\n\n\n\n\n\n**Quality (Empirical Results)** : The proposed method demonstrates strong performance both quantitatively and qualitatively. In Table 1, NCT achieves comparable or superior FID (image quality) and consistency scores with just 1 NFE (Number of Function Evaluations), compared to the standard ControlNet which requires 50 NFEs. This signifies a dramatic improvement in computational efficiency while maintaining high control quality. The qualitative comparison in Figure 2 also shows that while a conventional approach (DI+ControlNet) suffers from image quality degradation when integrating control signals, NCT successfully incorporates the control while maintaining high fidelity.\n\n\n\n\n\n## Weaknesses\n\n**Quality (Scope and Generalizability)** : The paper's experiments are limited to structural controls, such as Canny edges, HED, and depth maps. The core assumption of NCT—that diffusing the noise progressively decouples it from the condition —may not hold for spatially less robust features like texture, style, or other fine-grained attributes. For instance, when using an artistic style as a control signal, minor changes in the noise could lead to significant stylistic shifts, potentially making the consistency training ineffective. Although the paper claims general applicability, the experimental evidence provided is insufficient to fully support this.\n\n**Inappropriate Baseline Selection** : A direct comparison with JDM is omitted in Table 2. While the authors argue that JDM is inefficient due to its need for additional distillation, a direct comparison of the final model's performance (in terms of FID and Consistency) is essential to properly situate NCT, even acknowledging the efficiency difference. Furthermore, the paper compares against IP-Adapter in Table 2 but not in Table 1, giving the impression that baselines were selectively chosen for each task, potentially to present the results more favorably.\n\n**Insufficient Ablation Study** : In Section 3.1, the paper claims that a direct denoising approach yields \"blurry images,\" and while Table 3 shows a very high FID for the \"w/o noise consistency loss\" ablation, it fails to provide a visual example, unlike the other ablations in Figure 4. This makes the claim less convincing, as it is difficult to assess the degree and nature of the \"blurriness\" based solely on the FID score.\n\n\n\n**Question on the Generality of the Method** : A key question is why this method is presented as specific to one-step/consistency-based models. The proposed training scheme seems general. Could the NCT framework be used as an alternative training objective for adapters on standard, multi-step diffusion models like ControlNet? A discussion on this point would help clarify the fundamental contributions and limitations of the proposed loss functions.", "was_revised": true, "final_justification": "The author well addressed the concerns. So my intention is clear accept.", "rebuttal": "We sincerely thank you for your time and effort in reviewing our paper. We address each comment below.\n\n> Insufficient Ablation Study : In Section 3.1, the paper claims that a direct denoising approach yields \"blurry images,\" and while Table 3 shows a very high FID for the \"w/o noise consistency loss\" ablation, it fails to provide a visual example, unlike the other ablations in Figure 4. This makes the claim less convincing, as it is difficult to assess the degree and nature of the \"blurriness\" based solely on the FID score.\n\nThe results for the 'w/o noise consistency' variant are presented in both Figure 4 and Table 3. \nThis variant is distinct from direct denoising, for which we report the results below:\n\n| Method              | FID↓  | Con.↓ |\n| ------------------- | ----- | ----- |\n| Ours                | 13.67 | 0.110 |\n| w/ direct denoising | 37.49 | 0.115 |\n\nIt can be seen that direct denoising shows much worse FID due to its blurry samples. For visual samples, we will provide them in the revision. We cannot provide it in the rebuttal stage due to the policy. However, we refer to Figure 3 of JDM, which shows results of direct denosing with an additional distillation loss (Diff-instruct loss). It can be seen that the direct denoising approach still produces blurry samples even with an additional distillation loss. \n\nIt is worth noting that the reason why the variant (w/o noise consistency loss) exhibits a worse FID score compared to NCT is that the FID calculation is performed between the real images and the samples generated under the conditions corresponding to the same real images. \nControllable generation with $c\\sim p(c|x)$ allows for a distribution closer to that of the real image $x$. \n\n\n\n> The paper compares against IP-Adapter in Table 2 but not in Table 1\n\nThis is because the IP-Adapter in Table 2 and ControlNet in Table 1 focus on different tasks. ControlNet is concerned with controllable generation, aiming to preserve the details of the given conditions. In contrast, the IP-Adapter focuses on image-prompted generation, where the main goal is to inject the semantics and style of a reference image. This makes it unsuitable to evaluate them using the same benchmark.\n\n> Compare JDM regarding image-prompted generation\n\nGreat suggestion. We compare JDM regarding image-prompted generation below:\n\n| Method      | NFE↓ | Clip-T↑ | Clip-I↑ |\n| ----------- | ---- | ------- | ------- |\n| IP-Adapter | 100  | 0.588   | 0.828   |\n| JDM         | 1    | 0.585   | 0.826   |\n| Ours        | 1    | 0.593   | 0.821   |\n\nIt can be seen that NCT achieves competitive results compared to JDM, which requires additional expensive distillation loss.\n\n> On a Fairer Comparison with ControlNet: The proposed NCT methodology appears to be applicable not only to one-step generators but also to standard multi-step diffusion models. To provide a more direct and fair comparison, could you apply NCT on a standard diffusion model (e.g., Stable Diffusion 1.5) and benchmark it against the original ControlNet using the exact same base model and evaluation settings (e.g., 50 NFEs)? The current comparisons in Table 1 are challenging to interpret on a one-to-one basis, as they compare models with different base architectures and step counts. This new experiment would help isolate the performance of the training methodology itself.\n\n**We first clarify that the network architecture (Controlnet + UNet) adopted by NCT is identical to that of standard ControlNet.**\nThe difference is that NCT trains a ControlNet for a one-step generator distilled from a pre-trained diffusion model, whereas standard ControlNet trains a ControlNet for the diffusion model itself. \n***NCT actually faces a greater challenge because it needs to inject the condition in one-step generation.***\nHence, we believe the comparison is fair for baselines and NCT's advantage is convincing. \n\nNCT is tailored for one-step generators, and extending it to standard diffusion models presents some difficulties. \nThe reason is: NCT training requires access to the model's predicted $x_0$ and its corresponding condition $c$. This is not feasible for diffusion models because their one-step predictions are blurry, and $p(c|x)$ is trained on clean images, making it difficult to obtain the accurate corresponding conditions needed for training NCT. Exploring the extension of NCT to standard diffusion models would be interesting future work.\n\n> Could you please provide side-by-side visual comparisons of your method's outputs against those from other one-step controllable generation models, such as JDM?\n\nThanks for the great suggestion. We will provide the visual samples in revision. But limited to the policy of rebuttal, we cannot provide the samples in the rebuttal stage. Sorry for this.\n\n> On the Generalizability of NCT: The experiments currently focus on structural control signals, such as Canny edges and depth maps. Have you explored the efficacy of NCT with non-structural or stylistic control signals, for instance, controlling for a specific texture or an artistic style? Could you elaborate on how the core assumption of NCT—that diffusing noise progressively decouples it from the condition—is expected to hold for features that are less spatially robust? For such controls, even minor perturbations in the noise space might lead to significant semantic deviations, potentially challenging the consistency objective.\n\nWe highlight that, theoretically, using NCT to inject conditions for one-step generation does not depend on what the condition is. In particular, in Theorem 1, we show that our optimization objective can learn a conditional generator $f_{\\theta,\\phi}(x,c)$, and that the validity of this theorem is independent of the type of the condition $c$.\n\nEmpirically, for the generation of injected image styles, we already have a visualized image based on IP-Adapter in our main paper (Figure 3). In the second row of Figure 3, we can effectively use Van Gogh's paintings as an image prompt to inject style, which can effectively infuse the model-generated images with the Van Gogh style."}, {"review_id": "dVPwILRn3P", "reviewer": "Reviewer_X1xZ", "summary": "The paper introduces Noise Consistency Training (NCT), a lightweight procedure for adding new control signals to a pre-trained one-step generator without re-running diffusion distillation or accessing the original training set. NCT diffuses latent noise rather than images, forming pairs of latents with different noise levels; an adapter is trained with a noise-space consistency loss that pushes its outputs for the two latents to match, plus a boundary loss that keeps the adapter anchored to the original generator.  The proposed method is highly modular, data-efficient, and easy to deploy.", "questions": "See \"weaknesses\".", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1. The paper introduces Noise Consistency Training (NCT), a conceptually simple yet novel approach that enables controllable image generation using a one-step generator. It trains a lightweight adapter using a consistency loss in the latent noise space, effectively sidestepping the need for teacher models or expensive multi-step diffusion processes.\n\n2. Empirical results demonstrate that NCT achieves a significant reduction in computational cost (from 50 NFEs to just 1 NFE) while maintaining or even improving generation fidelity and control alignment, across multiple control types.\n\n3. The paper includes ablation experiments showing that both the boundary loss and noise consistency loss are necessary to achieve good performance, demonstrating the value of these components in training stable and condition-sensitive models.\n\nWeakness: \n1. The paper is generally not well written and can be difficult to follow, especially for readers who are not already familiar with the  consistency models. For instance, the presentation of consistency models and Equation (4), which defines the consistency loss, lacks clarity. Could the authors provide a clearer explanation of why this objective can force inject conditioning signals, and definition of the stop-gradient operator?\n Moreover, the paper suffers from inconsistent and overloaded notation: time steps $t_n$, $t_k$  are used ambiguously as subscripts for both $x$ and $z$. and the definitions of diffusion parameters in later equations (e.g., (5), (6)) are inconsistent with those used earlier.\n\n\n2. The training procedure assumes sampling from $p(c \\mid f_\\theta(z))$, and the paper appears to treat this step as a black-box oracle without further justification or clarification. However, for conditions like Canny, HED, or depth maps, such conditional distributions are not readily defined or tractable to sample from to me. Could the authors elaborate on how this sampling is performed in practice?\n\n\n3. Lemma 1 makes its claim without stating the necessary assumptions in the main text. While the appendix provides a specific parameter scheduling under which the result holds ($\\alpha_0 = 1, \\sigma_0 = 0$ and $\\alpha_1 = 0, \\sigma_1 = 1$), this renders the lemma rather trivial. However, the main paper does not clearly state this parameter scheduling. \n\n\n4. The paper repeatedly asserts that the proposed noise consistency loss yields lower variance, yet it offers neither theoretical justification  nor empirical evidence to support this claim. I believe this is true, but could the authors give a more precise explanation?", "was_revised": true, "final_justification": "We appreciate the authors' effort in the rebuttal stage. However, my primary concern lies in the writing quality and mathematical rigor. For example, Lemma 1 does not explicitly state all the assumptions used in its proof. In addition, the paper has some unclear aspects in notation and presentation, which I feel are not minor and may require substantial revision beyond the scope of the rebuttal. Therefore, I will keep my score (3), but I will lower my confidence from 4 to 3.", "rebuttal": "We sincerely thank you for your time and effort in reviewing our paper. We address each comment below.\n\n> The paper can be difficult to follow for readers who are not already familiar with the consistency models.  For instance, the presentation of consistency models and Equation (4), which defines the consistency loss, lacks clarity.\n\nThank you for your valuable feedback. \nWe will expand the preliminary section to include a comprehensive introduction to consistency models, ensuring readers unfamiliar with this framework can follow our work more easily.\nBesides, while consistency models serve as inspiration for our noise consistency training, we want to clarify that our approach is fundamentally self-contained. The key differences include: 1) our noise consistency training operates in a different space and uses a distinct formulation; 2) the theoretical foundations and practical implementation differ significantly from standard consistency models.\n\n> Could the authors provide a clearer explanation of why consistency training can force inject conditioning signals, and definition of the stop-gradient operator? \n\nFor consistency training with condition $c$, it tries to make $g(x_{t_{n+1}}, c)$ approximate $g(x_{t_{n}}, c)$, where $t_{n+1} > t_n$ and  $g$ is the desired consistency model, taking noisy samples with condition and predicts clean samples. \nIntuitively, the network can better predict clean samples given less noisy samples $x_{t_{n}}$, hence the $g(x_{t_{n}}, c)$ can be used as the target for learning $g(x_{t_{n+1}}, c)$.\nIn this case, the information in $x_{t_{n+1}}$ about the image is more corrupted than $x_{t_{n}}$, which can force the network to utilize the information from condition $c$ to minimize the loss.\n\nThe stop-gradient operator treats a variable as a constant, preventing gradients from backpropagating through it.\n\n> Moreover, the paper suffers from inconsistent and overloaded notation timesteps $t_n, t_k$, are used ambiguously as subscripts for both $x$ and $z$. \n\nSorry for the confusion. We will clean up the notation in our revision.\n\n> The definitions of diffusion parameters in later equations (e.g., (5), (6)) are inconsistent with those used earlier.\n\nWe deliberately use different symbols to distinguish between different models. Specifically, Eq. 4 is a standard consistency model ($g_\\alpha$), while Eq. 6 involves a one-step generator ($f_{\\theta}$) and its adapted version ($f_{\\theta,\\phi}$). \n\n> The definition of $p(c|x)$\n\nWe model the distribution $p(c|x)$ as a Dirac delta distribution: $p(c|x) \\triangleq \\delta(c - h(x))$, where $h(\\cdot)$ represents a deterministic discriminative model that maps inputs to conditions. The Dirac delta distribution can be sampled easily by $h(x)$, but it is hard to estimate the density. This is well-suited for our NCT, since NCT just requires samples from $p(c|x)$ for training.\n\n> Lemma 1 makes its claim without stating the necessary assumptions in the main text. While the appendix provides a specific parameter scheduling under which the result holds this renders the lemma rather trivial. However, the main paper does not clearly state this parameter scheduling.\n\nWe are sorry for the confusion on the scheduler. \nWe note that our schedule is specified by $z_t = \\sigma_t z + \\sqrt{1-\\sigma_t}\\epsilon$. And we acknowledge a typo in Eq. 5, where the $z_T=\\sigma_t z + \\sqrt{1-\\sigma_t}\\epsilon$ should be $z_t=\\sigma_t z + \\sqrt{1-\\sigma_t}\\epsilon$.\nBesides, $z_T$ denotes the terminal diffused noise (i.e., $\\sigma$ = 1), while $z_0$ denotes the initial noise (i.e., $\\sigma$ = 0). This is a widely used notation in diffusion literature, and we will clarify this in the revision. \n\nRegarding Lemma 1, it is given for \\textit{formally stating} that our diffusion process can obtain independent noise-condition pairs, which is crucial for enabling our approach to learn conditional generation. \nAlthough the approach is straightforward, it provides a formal ground for subsequent analysis. \n\n> The paper repeatedly asserts that the proposed noise consistency loss yields lower variance, yet it offers neither theoretical justification nor empirical evidence to support this claim. I believe this is true, but could the authors give a more precise explanation?\n\nThe lower variance is compared to the direct denoising objective. For the direct denoising objective, the target samples follow conditional distribution $p(x|x_T)$ which is highly multimodal. Since when $x_T$ is pure noise, there are infinitely many possible clean images $x$ that could have generated it. \nIn contrast, for noise consistency loss, the target samples follows $q(x|z_{t_{n}},c) = \\int p(z_{t_{n}}|z_{t_{n+1}})p(x|z_{t_n},c)$, where $p(x|z_{t_n},c)\\triangleq \\delta(x-f(z_{t_n},c))$. \nAnd $p(z_{t_{n}}|z_{t_{n+1}})$ has small variance, since the timestep gap between $z_{t_{n+1}}$ and $z_{t_{n}}$ is chosen to be small. \nBased on the above intuition, we believe our method produces lower variance.\n\nTo substantiate this claim, we conducted an empirical analysis measuring the variance of the optimization objectives:\n\n| Method           | Variance |\n| ---------------- | -------- |\n| Direct denoising | 0.956    |\n| NCT              | 0.013    |\n\nThe results clearly demonstrate that the noise consistency loss exhibits significantly lower variance compared to direct denoising, supporting our assertion."}, {"review_id": "w8dm7FctLU", "reviewer": "Reviewer_92Eo", "summary": "This paper introduces Noise Consistency Training (NCT), a lightweight method to adapt pre-trained one-step generators for controllable content generation without retraining the base model or requiring original training data. NCT employs an adapter module and a noise consistency loss to align generation behavior with new control signals. Essentially, the proposed method combines the Consistency model and ControlNet. Authors evaluated NCT based on pretrained models such as SD on COCO benchmarks.", "questions": "As in weaknesses", "limitations": "No analysis of when or why NCT might fail is presented.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strengths**\n\n1.  **Practical Significance & Well-Defined Problem:** The paper tackles a highly relevant and practical challenge in AIGC: efficiently adapting powerful one-step generators to diverse control signals without costly retraining or distillation. The focus on computational efficiency and deployability addresses a key bottleneck in real-world applications.\n2.  **Modularity and Data Efficiency:** The proposed NCT framework is commendably lightweight and modular. Its reliance solely on the pre-trained generator and a control signal model, without needing original training data or full model retraining, enhances its practical utility and reduces deployment barriers.\n\n### **Weaknesses**\n\n1.  **Technical Writing and Presentation:**\n    *   **Formula 1 (ControlNet Representation):** The overly simplistic reduction of ControlNet's complex architecture to a single parameter `phi` is misleading and does not accurately reflect its known structure or contributions. This requires substantial revision and proper explanation.\n    *   **Figure 1:** The figure lacks meaningful discussion or analysis within the main text. Its relevance to NCT's mechanism or results is unclear, making it appear disconnected from the core narrative. A detailed caption and textual integration are essential.\n    *   **Formula 3 (Definition of `d`):** The variable `d` is used without a clear definition. Its meaning and role in the loss function must be explicitly stated.\n    *   **Formula 6 (Notation & Errors):**\n        *   The presence of `#Simplify Notation` is unacceptable in a final manuscript and suggests unrevised drafting artifacts.\n        *   The chaining of three expectation operators (`E`) is unnecessarily complex and hinders readability without a clear justification. Simplification or splitting is advised.\n        *   The subscript `z,c | z,ε` appears erroneous and logically inconsistent (`z` appears conditioned on itself?). The intended conditional variables must be corrected and precisely defined.\n    *   **Formula 13 (Norm Symbol):** The norm symbol (`|| · ||_1`) lacks specification (e.g., L1, L2). The choice of norm can impact the loss behavior and results; this must be clarified.\n\n\n2.  **Limited Novelty & Insufficient Contextualization (Major Concern):**\n    *   The core innovation, NCT, appears heavily reliant on applying a consistency loss principle (well-established in consistency models) to the specific context of adapting ControlNet-like structures for one-step generators. While the *combination* is novel and practically useful, the paper fails to sufficiently differentiate the *fundamental contribution* beyond this synergistic application.\n    *   The discussion lacks depth in contrasting NCT with highly relevant alternative approaches for efficient adaptation or conditional generation, particularly:\n        *   **Shortcut Learning/Modeling:** How does NCT compare conceptually and empirically to methods designed for fast adaptation with minimal changes?\n        *   **Meanflow / Other Recent Efficient Generators:** The absence of comparison or discussion regarding very recent advances like Meanflow, which also target efficient high-quality generation (potentially including controllability aspects), weakens the claim of state-of-the-art positioning and novelty. A thorough related work update is needed.\n    *   **Risk:** The work risks being perceived primarily as a competent engineering solution (\"experimental report\") applying known concepts (consistency, adapters) to a new task, rather than introducing a fundamentally new principle or significant theoretical advancement.\n\n3.  **Superficial and Problematic Theoretical Analysis:**\n    *   **Derivative Nature:** Lemmas and theorems strongly resemble those from the foundational consistency model literature without clear novel theoretical extension or significant adaptation specific to NCT's conditional control objective. The paper does not convincingly establish what *new theoretical insight* NCT provides.\n    *   **Lack of Cohesive Narrative:** The theoretical components feel bolted on rather than integral. There's a disconnect between the listed lemmas/theorems and the core arguments or practical results of the paper. A stronger narrative linking theory to the proposed method and its empirical success is missing.\n    *   **Confusing Claims:** Key arguments lack supporting evidence or rigorous justification:\n        *   Line 123: Attributing blurry images solely to \"high variance of the optimized objective\" is asserted without analysis, proof, or citation. What variance? Compared to what?\n        *   Line 130: The claim about the model's inability to perform conditional generation given random `z` due to training on coupled pairs `(z, c)` is stated dogmatically. No theoretical argument or experimental evidence (e.g., showing failure cases) is provided.\n    \n\n4.  **Insufficient Experimental Scope and Depth:**\n    *   **Limited Resolution:** Heavy reliance on low-resolution image datasets significantly diminishes the claim of \"high-quality\" generation and practical relevance. Modern AIGC demands robust performance at higher resolutions. Results on standard high-resolution benchmarks are crucial.\n    *   **Narrow Condition Scope:** Experiments focus on a limited set of control signals. The paper does not demonstrate generality across diverse, complex, or semantically rich conditions. How broadly applicable is NCT really?\n    *   **Lack of Ablation Studies:** The current ablation studies are overly simple. Critical components lack ablation:\n        *   How essential is the specific adapter architecture?\n        *   What is the impact of the noise consistency loss vs. simpler losses?\n        *   How does performance scale with the amount of adaptation data?\n    \n[1] One Step Diffusion via Shortcut Models\n[2] Mean Flows for One-step Generative Modeling", "was_revised": true, "final_justification": "Thanks for the detailed reply. It resolves some of my concerns but not all of them. The notation problems and the concerns about empirical results are solved. However, the resemblance with Consistency model and ControlNet still constrain the contribution of this work. After thorough consideration, I decide to raise my score to 3 and not against for acceptance.", "rebuttal": "We sincerely thank you for your time and effort in reviewing our paper. We address each comment below.\n\n> The overly simplistic reduction of ControlNet's architecture to a single parameter does not accurately reflect its structure/contributions.\n\n**Our NCT is agnostic to the specific structure of the ControlNet or adapter**. \nThe notation $\\phi$ is used as a general representation of the adapter's learnable parameters, not as a reduction of its complexity. Whether the adapter is a standard ControlNet, a lightweight adapter like IP-Adapter, or any other control module, our training methodology applies uniformly.\n**Our work is not in modifying ControlNet's architecture, but in developing an effective training strategy for *any* control adapter when applied to one-step generators.** We chose this general notation precisely because our method works across different adapter architectures without requiring architecture-specific modifications.\n\n> Figure 1 lacks discussion or analysis in main text. Its relevance to NCT's mechanism or results is unclear.\n\nFigure 1 is the overall framework description of NCT with a clear caption and contents. \nWe will provide a brief introduction of Figure 1 in the method section to make it clearer. \n\n> The chaining of three expectation operators is unnecessarily complex and hinders readability.\n\nWe have simplified the notation in the second line of Eq. (6).\n\n> The presence of \\#Simplify Notation is unacceptable and suggests unrevised drafting artifacts.\n\nThe \"\\#Simplify Notation\" is to clearly indicate that we are simplifying notation in Line 2 from Line 1 of Eq. (6).\n\n> The subscript $z,c|z,\\epsilon$ appears erroneous and logically inconsistent (z appears conditioned on itself?).\n\nThis denotes three variables rather than one variable, where each is separated by \",\".\n\n> The symbol ($|| \\cdot ||_1$) lacks specification (e.g., L1, L2).\n\nWe note that the symbol $||\\cdot||_1$ exactly denotes the L1 norm.\n\n> The core innovation, NCT, appears heavily reliant on applying a consistency loss principle (well-established in consistency models) to the specific context of adapting ControlNet-like structures for one-step generators. While the combination is novel and practically useful, the paper fails to sufficiently differentiate the fundamental contribution beyond this synergistic application.\n\n**NCT is not a combination of Consistency Training (CT) and ControlNet.**\nThere is a fundamental and significant difference between Noise Consistency Training (NCT) and CT. NCT \"diffuses\" noise in the noise space and \"denoises\" the noise to learn adapters for adding controls to one-step generators, while CT adds noise to data in the data space and denoises the noisy data to learn few-step generator.\n\n> The discussion lacks depth in contrasting NCT with relevant alternative approaches for efficient adaptation or conditional generation (Shortcut Models and Meanflows).\n\n**NCT is orthogonal to the specific method used for learning the one-step generator**. Since NCT focuses on injecting conditions into a well-trained, one-step generator, which is agnostic to the method used to train the one-step generator.\nFurthermore, current approaches like Shortcut Models and Meanflows do not prioritize controllable generation, nor have they demonstrated the capability for high-quality, one-step text-to-image synthesis. In contrast, NCT is capable of achieving high-quality, controllable text-to-image generation in one step.\n\n> Lemmas\\&theorems strongly resemble those from the foundational consistency model literature without clear significant adaptation specific to NCT's conditional control objective. The paper does not convincingly establish what new theoretical insight NCT provides.\n\nWe respectfully argue that NCT is not a derivative work but a novel approach with a fundamental distinct theoretical and algorithmic foundation.\n\n**The core difference is that NCT enforces consistency in the noise space, while Consistency Models (CMs) operate in the data space**. These Lemmas\\&theorems are necessary reformulation, not a minor adaptation. In particular, NCT \"diffuses\" noise in the noise space and \"denoises\" the noise, while CMs adds noise to data in the data space and denoises the noisy data. NCT is designed for pre-trained one-step generators to add controls, which lack the multi-step PF-ODE trajectory originating from a clean data point  that CMs fundamentally rely on. \n\nThis algorithmic shift necessitates a different theoretical underpinning. While CM theory is rooted in approximating ODE solutions, NCT's theory is based on distribution matching. Our noise diffusion process creates an interpolation path from a coupled to an independent latent-condition distribution. The NCT objective, including the crucial boundary loss, forces the model's output distribution to remain invariant along this path. \n\n> There's a disconnect between the listed lemmas/theorems and the core arguments or practical results of the paper. A stronger narrative linking theory to the proposed method and its empirical success is missing.\n\nWe will revise the manuscript to make the connection between our theory and practice more explicit. The narrative is as follows:\n\nOur main theorem establishes the ideal objective. It proves that a generator can learn the true conditional posterior if it satisfies two abstract conditions: a Boundary Condition and a Consistency Condition.\n\nOur proposed method is a direct, practical implementation of this theory. The theoretical Boundary Condition is directly optimized by our Boundary Loss ($L_{bound}$).\nThe theoretical Consistency Condition (formalized as an MMD objective in Lemma is empirically approximated by our Noise Consistency Loss ($L_{con}$).\n\nThe strong empirical results are the validation of this framework. They demonstrate that our practical losses effectively serve as proxies for the theoretical ideals, successfully guiding the model to the desired conditional distribution. Our ablation study, which shows complete model failure when either loss is removed, further reinforces that both theoretical pillars (and their practical counterparts) are essential.\n\n> Attributing blurry images solely to \"high variance of the optimized objective\" is asserted without analysis, proof, or citation. What variance? Compared to what?\n\nThe \"high variance\" refers to the variance in the conditional distribution $p(x|x_T)$ when optimizing the direct denoising objective. When $x_T$ is pure noise, there are infinitely many possible clean images $x$ that could have generated it, making $p(x|x_T)$ highly multimodal. The optimal solution for minmizing the direct denoising loss under an L2 loss is $\\mathbb{E}[x|x_T]$, which averages over all possible images in the dataset, inherently producing blurry results.\n\nTo substantiate this claim, we conducted an empirical analysis measuring the variance of the optimization objectives:\n\n| Method           | Variance |\n| ---------------- | -------- |\n| Direct denoising | 0.956|\n| NCT| 0.013|\n\nThe results clearly demonstrate that the noise consistency loss exhibits significantly lower variance compared to direct denoising, supporting our assertion.\n\n> The claim about the model's inability to perform conditional generation given random z due to training on coupled pairs (z, c) is stated dogmatically. No theoretical argument or experimental evidence is provided.\n\n**We have ablated this variant in our main paper, showing that it is unable to handle the condition**. In particular, training on coupled pairs (z, c) is equal to our ablated variant \"w/o noise consistency training\". Results are in Figure 3 and Table 4, indicating that this approach is unable to inject conditions.\n\n> Limited Resolution\n\nOur experiments are conducted over 512 resolution, which is already relatively high-resolution and widely adopted in most prior works in the line of distillation [3, 5, 9] and also the original multi-step ControlNet [12].\n\n> Narrow Condition Scope.\n\n**Our experiments focused on controllable generation (based on ControlNet) and Image-prompted generation (IP-Adapter). We believe this demonstrates NCT's broad applicability rather than limiting it.**\n\nOur experimental design deliberately selected these two architectures because they represent fundamentally different conditioning paradigms: ControlNet exemplifies spatial/structural control (edges, depth, HED, low-resolution) while IP-Adapter represents semantic/style control through image prompts. Together, they span the spectrum from low-level geometric constraints to high-level semantic guidance.\n\nMoreover, ControlNet and IP-Adapter serve as foundational architectures for numerous downstream applications. We believe that NCT's experiments have covered a sufficient number of conditional scenarios.\n\n> How essential is the specific adapter architecture?\n\n**Our NCT is agonistic to specific adapter architecture, since NCT is an algorithm in training adapters to add new controls for one-step generators.** We apply the same adapter architecture among methods in our experiments. NCT can effectively train adapters for one-step generator in adding new controls, as indicated by the success on both ControlNet and IP-Adapter architecture.\n\n> What is the impact of the noise consistency loss vs. simpler losses?\n\nWe have ablated that simply use boundary loss without noise consistency loss in Table 3 and Figure 4. It can be seen that without noise consistency loss, it is unable to inject conditions. Additionally, we conduct an extra ablation study on simply using direct denoising loss:\n| Method                     | FID↓  | Con.↓ |\n| -------------------------- | ----- | ----- |\n| Ours| 13.67 | 0.110 |\n| w/o noise consistency loss | 20.56 | 0.165 |\n| w/ simple direct denoising | 37.49 | 0.115 |\nThe results show that the simple denoising loss leads to inferior generation, while our full approach can ensure high-quality generation and condition alignment."}]

Author rebuttal:

Meta-review: This paper introduces Noise Consistency Training (NCT) -- a method to incorporate additional control to a pre-trained one step diffusion generator using an additional trainable model such as ControlNet. The core idea is to optimize a consistency type loss on the one step trainable model outputs on two noisy versions of the initial coupled noise $z$ and condition $c$. 

Reviewers appreciated the method’s novelty to a relevant and timely problem, the practical and interesting solution with "noising the initial noise" approach, and the solid results especially for a one step method. Reviewers’ concerns included exposition issues; questions about the availability of samples $p(c|x)$, which the authors provided a remedy in rebuttal by inversion of the generation to find noise matching an image and condition pair; and concerns about experiments including baseline selection and limited score, which the authors also addressed during rebuttal to the satisfaction of the relevant reviewer. 
The negative reviewers maintained their weak-reject scores after discussion but lowered their confidence with one of the reviewers explicitly not objecting to acceptance. The second negative reviewer main concern was writing quality which didn't seem to be a strong consensus concern. Overall, a weak accept recommendation here.
Decision: Accept (poster)