A Noise is Worth Diffusion Guidance
Authors: Donghoon Ahn
Jiwon Kang
Sanghyun Lee
Jaewon Min
Minjae Kim
Wooseok Jang
Hyoungwon Cho
Sayak Paul
SeonHwa Kim
Eunju Cha
Kyong Hwan Jin
Seungryong Kim
Venue: ICLR 2026 Poster (2026)
Keywords: Classifier Free Guidance
Diffusion Guidance
Guidance Distillation
Text to Image Synthesis
Primary area: generative models
TL;DR: We train a noise-refining network that refines Gaussian initial noise to encode an initial layout, enabling high-quality samples without guidance (e.g., classifier-free guidance) during denoising.

Abstract: Diffusion models have demonstrated remarkable image generation capabilities, but their performance heavily relies on sampling guidance such as classifier-free guidance (CFG). While sampling guidance significantly enhances image quality, it requires two forward passes at every denoising step, leading to substantial computational overhead. Existing approaches mitigate this cost through distillation, training a student network to learn the guided predictions. In contrast, we take a distinct approach by refining the initial Gaussian noise, a critical yet under-explored factor in the diffusion-based generation pipelines. 
We introduce a noise refinement framework, NoiseRefine, where a refining network is trained to minimize the difference between images generated by unguided sampling from the refined noise and those produced by guided sampling from the input Gaussian noise.
This simple approach demonstrates that images from the refined noise alleviate artifacts and mitigate structural collapse, achieving significantly higher quality than those generated from pure Gaussian noise without modifying the diffusion model, thereby preserving its prior knowledge and compatibility with finetuned or timestep distilled variants. 
Beyond its practical benefits, we provide an in-depth analysis of refined noise, offering insights into its role in the denoising process and its interaction with guidance. Our findings suggest that structured noise initialization is key to efficient and high-fidelity image synthesis. 
Project page: https://cvlab-kaist.github.io/NoiseRefine/

Published as a conference paper at ICLR 2026
A NOISE IS WORTH DIFFUSION GUIDANCE
Donghoon Ahn
Jiwon Kang²*
Sanghyun Lee
Jaewon Min
Wooseok Jang?
Minjae Kim?
Hyungwon Cho?
Sayak Paul
SeonHwa Kim
Eunju Cha
Kyong Hwan Jin
Seungryong Kim
UC Berkeley
KAAIS
Korrea University
*Sookmyung Women's 's University
Hugging Face
[Standard Gaussian Noise Space N(O,
[Distilled Noise Space]
(b) Ad residual noise signals
Quality
Quality
Images qunernted from random noise without guidance
Images generated from refined without guidance
Figure 1: Effectiveness of NoiseRefine. Diffusion models often fail to generate high-quality images without guidance, such as classifier-free guidance (CFG) Ho & Salimans (2022), which doubles the inference cost. In this paper, propose NoiseRefine, novel approach to improve image quality by learning to map Gaussian noise space to guidance-distilled noise space. Images are generated using the same seed and prompt.
ABSTRACT
Diffusion models have demonstrated remarkable image generation capabilities, but their performance heavily relies on sampling guidance such as classifier- free guidance (CFG). While sampling guidance significantly enhances image quality, it requires two forward passes at every denoising step, leading to substantial com- putational overhead. Existing approaches mitigate this cost through distillation, training a student network to learn the guided predictions. In contrast, we take distinct approach by refining the initial Gaussian noise, critical yet under- explored factor in the diffusion-based generation pipelines. We introduce noise refinement framework, NoiseRefine, where a refining network is trained min- imize the difference between images generated by unguided sampling from the refined noise and those produced by guided sampling from the input Gaussian noise. This simple approach demonstrates that images from the refined noise alle- viate artifacts and mitigate structural collapse, achieving significantly higher qual- ity than those generated from pure Gaussian noise without modifying the diffusion model, thereby preserving its prior knowledge and compatibility with finetuned or timestep distilled variants. Beyond its practical benefits, analysis of refined noise, offering insights into its role in the denoising process and its interaction with guidance. Our findings suggest that structured noise initializa- tion is key to efficient and high-fidelity image synthesis. Project page: cvlab- wwt.gggitu...oNiiieeee..
provide in-
Equal contribution.
Equally contributed as second authors.
Corresponning authors.

---

Published as a conference paper at ICLR 2026
Initial nosse (0.
Generated imagee Guidance x
Galde
XW
Diffinion sampling guidance
Denoising process wi' guidance
XY
Dittision sumpling guidunce
learn?
resitual connect
Inversion w/o guidance
Noise Refining Nerwork fi
XN
Diffisionn sampling
x Guide
w/o guidunce
xg
Guide
Backproppggtion
Refined noise
Denoising wio guidance
Generated image Guidance 9g
By
(a) Mapping between noise spaces
(b) Overview of training framework
Figure 2: Motivation and training framework of NoiseRefine. (a) Starting from an initial noise JT, unguided sampling often produces low-quality images, necessitating sampling guidance such as CFG. In contrast, the inversion noise zeGuide obtained by inverting guidance-generann generatdd images from the same Zy, can yield high-quality results even without guidance. This raises our central question: we learn map 2T into ? (b) Learning with reconstruction loss between ær and 27 Guidee may be suboptimal due to errors during inversion. Instead, our model Iearns to refine 2y into frr, with the objective of matching unguided image in from refined noise fr with guidance-generater image 2 Guide from initial noise TT.
INTRODUCTION
In recent years, text-to-image (T2I) diffusion models (Rombach et al., 2022; Esser et al., 2024; Podell al., 2023), which generate images conditioned on text prompts, have achieved remarkable advancements. These models produce visually appealing images that are both realistic and well- aligned with human perception. central factor behind their effectiveness the use of sampling guidance techniques (Dhariwal & Nichol, 2021; Ahn et al., 2024; Hong et al., 2023; Hong, 2024), most notably classificr-free guidance (CFG) (Ho & Salimans, 2022). While indispensable for high- quality synthesis, these methods require evaluating additional prediction (unconditional prediction in the case of CFG) at every denoising step, effectively doubling inference cost.
A common strategy mitigate this overhead is guidance distillation, where student net- work (Meng et al., 2023) or an adapter (Hsiao et al., 2024) is trained to approximate the guided predictions of the original model. However, such approaches often require modifications to the denoising network, which is prone to catastrophic forgetting (Kirkpatrick et al., 2017), and poten- tially incompatible with complementary techniques such as domain-specifie fine-tuning (Ruiz al., 2023) or timestep distillation Yin et al. (2024); Lin et al. (2024); Salimans & Ho (2022); Sauer et al. (2024b).
Recently, a growing line of work has explored the role of initial noise, suggesting that it can influ- ence the final image structure to some extent (Singh et al., 2022; Wu & De la Torre, 2022; Mao et al., 2023a: Ban et al., 2024; Xu et al., 2024a; Qi et al., 2024; Guo et al., 2024; Eyring et al., 2024; Zhou al., 2024; Mannering et al., 2025: Ma et al., 2025). Inspired by this, we ask: Instead of distilling guidance into the denoising network, can we distill it into noise? Diffusion inversion methods (Song et al., 2020a; Garibi et al., 2024) provide important clue. Ideally, a perfect inversion method would reconstruct a given image without requiring any guidance. Under this idealized as- sumption, a straightforward way to obtain the "guidance- noise" target is to start from an initial Gaussian noise, generate high-quality image using guidance, and then apply inversion to compute its corresponding noise. This resulting "inversion noise" should, principle, reproduce a similar image without guidance. This conceptual idea is illustrated in Fig. 2(a).
2

---

Published as a conference paper at ICLR 2026
However, show that directly learning the mapping from Gaussian noise to inversion noise is sub- optimal due to the accumulated reconstruction errors introduced by the inversion process (Sec. 3.2). To overcome this limitation, we shift the objective from the noise space to the image space and propose NoiseRefine, a novel method that refines Gaussian noise into informative and structured noise, enabling high-quality generation without guidance. As illustrated in Fig. 2 (b), lightweight transformation network maps arbitrary Gaussian noise into the refined noise space, trained so that unguided samples closely match guided counterparts generated from the same seed. At inference, single forward pass through this network substantially improves unguided generation quality, pro- ducing significantly more plausible images even without guidance, while preserving the original diffusion pipeline intact, in a prompt-learnin- manner (Zhou et al., 2022a) that avoids catas- trophic forgetting from model fine-tuning (Kirkpatrick et al., 2017).
Beyond eliminating guidance, NoiseRefine offers several advantages. First, since it operates solely on the noise input, it can be directly applied to fine-tuned models in various domains (e.g- Anime) without retraining the refining network. Second, it remains fully compatible with timestep- distillation techniques (Meng et al., 2023; Hsiao et al., 2024; Zhou et al., 2025). Together, these properties make NoiseRefine plug-an aod--nnd-layy solution for enhancing base, fine-tuned, and timestep distilled models. We validate our approach on both class-conditional and widely used -inage diffusion models.
Our contributions can be summarized as follows:
Noise refinement for guidance-free generation: To the best of our knowledge, this work the first to explore refining initial noise in diffusion pipelines to achieve high-quality image generation without diffusion guidance.
Preserving the diffusion pipeline: Our method does not modify the original diffusion model or pipeline, which ensures compatibility with LoRA modules in the original pipeline, generalizes well to fine-tuned models, and seamlessly integrates with existing
disttiilliitiiittoonnn techniques. Thorough analysis of refined noise in diffusion models: We provide a detailed study on the role of refined noise in the denoising process, offering insights into their impact on generation quality.
2 RELATED WORK
Classifier Guidance (CG) (Mao et al., 2023a) enhances fidelity by leveraging trained classifier gradients, albeit at the cost of diversity. CFG (Ho & Salimans, 2022) models an implicit classifier to achieve similar effects. Ahn et al. (Ahn et al., 2024) and Karras et al. (Karras et al., 2024) further generalize those guidance methods by intentionally generating lower-quality samples to guide the process toward improved outputs and other guidance techniques (Hong al., 2023; Sadat et al., 2024; Hong, 2024) generate 'perturbed' samples in various ways. While effective, these methods double computational and memory costs by requiring degraded sample generation each step, which is essential to their operation.
Diffusion guidance.
Distillation of diffusion models. Diffusion models are costly at inference due to guidance and it- erative denoising. A line of work distills teacher models into lighter students (Salimans Ho, 2022; Meng et al., 2023; Sauer et al., 2024b; Lin et al., 2024; Sauer et al., 2024a), targeting fewer steps (timestep distillation) (Salimans & Ho, 2022; Sauer et al., 2024b;b;a) or cheaper guidance (guid- ance distillation) (Meng et al., 2023), with extensions via adapters (Hsiao et al., 2024) prompt distillation (Zhou et al., 2025). In contrast, while existing guidance distillation approaches trans- fer guidance signals into the student network, distill guidance directly into the initial noise of diffusion models, making our method fully compatible with timestep distillation.
Noise optimization. Recent studies have explored improving the initial noise through optimiza- tion, reinitialization, or task-specific refinements (Samuel al., 2024; Eyring et al., 2024; Mao al., 2023b;a; Karunratanakul et al., 2024; Zhou et al., 2024; Guo et al., 2024; Mannering et al., 2025; Ma et al., 2025). Approaches include reward-model optimization (Eyring et al., 2024), bootstrap sam- pling for rare concepts (Samuel et al., 2024), patch databases for layout control (Mao et al., 2023b), iterative disentanglement (Guo et al., 2024), and systematic noise search (Ma et al., 2025). Recent work (Zhou et al., 2024) trains a noise transformation network using CFG and inversion, optimizing it in noise space with inversion noise as the supervision target. In contrast, we train the network with an image-space loss and focus guidance-free generation rather than human preference alignment.
3

---

Published as a conference paper at ICLR 2026
3 METHOD
In Sec 3.1, we analyze the differences between the original initial noise and the inversion noise ob- tained via during the inversion and propose to learn in the image space rather than the noise space. Finally, Sec 3.3 presents our complete training framework, incorporating a multi-step score distillation loss that mitigates the cost of backpropagation.
followed by inversion. In Sec. 3.2, we address the errors introduced
3.1 DIFFERENCE BETWEEN INITIAL NOISE AND INVERSION NOISE
When an image generated with guidance is inverted back to its noise using an inversion method (Song et al., 2020a), the recovered "inversion noise" tends to reproduce similar image even without guidance, as described in Fig. (a). We investigate the relationship between the initial noise generated image, as their differences may explain the gap in the quality of denoised outputs. To this end, we sample Gaussian noise TT ~ NN0 I), Denoise generate a guided image 29 Guide = Denoise Guide (IT, c) using a text-to-image diffusion model (Rom- bach al., 2022) with CFG and/or other guidance methods (Ho & Salimans, 2022; Ahn al., 2024; Hong, 2024), and then apply an inversion method (Song et al., 2020a; Garibi et al., 2024; Meiri et al., 2023) to obtain the corresponding inversion noise 21 GGidd == Inversion(=.guiã)) Both a0 Guide and ai depend on the condition c, which we omit for notational simplicity. This procedure yields paired samples (er,ayuue)
TT and the inversion noise 21 Guide of guidance-generated
Guide
Guudd) for subsequent analysis. Notation definitions are provided in Appendix B.
We generate 10K {IT, Giidde pairs via the aforementioned process with randomly selected prompts from the MS-COCO dataset (Lin al., 2014) and Stable Diffusion 2.1 (Rombach et al., 2022). Comparing the pixel-wise absolute differ- ences between FT and x Guide to those between random noise instances, Fig. 3 (a) shows that the differences in {IT, 27 Guide] pairs are signif- icantly smaller than those of 'Random' pairs. These differences are primarily concentrated in low-frequency components in the frequency do- main, as shown in Fig. (b), which plots the mag- nitude differences between noises. This indicates that the discrepancy be- tween the initial noise and inversion noise is structured rather than random. Such structured, low-frequency deviations imply the existence of a consistent and potentially learnable mapping be- tween the two noise distributions. If this mapping can be effectively modeled, it may provide pathway to generating higher-quality samples without explicit guidance during sampling.
De. Random
Magnituee Diflerance
2.5-
2.0
1.0
05-
0.
Absolute ofference
Figure 3: Analysis of the relationship between JT and Guuudde (a) Histogram of pixel-wise absolute differences. Blue: pairs of Gaussian noise and corresponding inversion noise; Or- ange: pairs of random Gaussian noise. (b) Mag- nitude difference of Fourier components, show- ing that æт and J.Culdde mainly differ in low- frequency regions.
3.2 LEARNING IN IMAGE SPACE RATHER THAN NOISE SPACE
Original
Reconstructed
Mitigating inversion error. A straightforward approach would be to directly learn a mapping from the initial noise to the cor- responding inversion noise. While this approach is conceptually feasible, inversion methods (Song et al., 2020a; Meiri et al., 2023; Garibi et al., 2024) have inherent limitations: they rely on approx- imations, and the ideal (true) inversion noise Iy noise that would perfectly reconstruct the guided image, is not di- rectly accessible in practice due to approximation errors in existing inversion methods. As a result, training on approximated inversion noise corrupted by inversion errors may limit performance (Fig. 4).
Guidef defined as the
Figure 4: Inversion error. The right image is recon- structed from the inversion noise of the left one. 50 in- version steps were used.
In practice, we trained two noise refining networks on 10K guided images and 10K inverted noises from a fixed prompt "a photo of a corgi riding a skateboard." As shown in Fig. 5 top row, directly learning this mapping produces blurry reconstructions.
4

---

Published as a conference paper at ICLR 2026
To sidestep this issue, optimize the objective in the image space rather than directly in the noise space. The key idea is to reduce the distance between images generated with and without guid- ance, d(zg. 20 (iinde)) rather than directly reducing the distance between their corresponding noises, d(ær, x. Here denotes a distance metric, instantiated as the 1.2 distance. We formally state this relationship in Proposition and provide a proof in Appendix B.2.
Proposition Let FT be an initial noise, and suppose that zo is the image obtained through de- noising. Assuming Lipschitz continuity of the denoising mapping Eg with respect to the distance metric d, for every xr there exists constant K. >> O such that the following holds:
d(rr, TT
Guidety < kd( 200 20
Guide
This result suggests optimizing in the image space, since reducing the image discrepancy also bounds the discrepancy in the corresponding noise. In the following sections, we detail how to train the refining network, our architectural choice for the refining network, and how to mitigate the costly backpropagation through full denoising steps.
3.3 TRAINING FRAMEWORK
Fig. (b) illustrates the training framework. Starting from Gaussian noise JT and prompt c, a diffusion model generates guided image zo
Guide using N' denoising steps with guidance. Any diffusion guidance (Ho & Salimans, 2022; Ahn al., 2024; Hong al., 2023; Sadat et al., 2024; Hong, 2024; Karras et al., 2024) or their combination can be applied for distillation.
Our model, noise refining network ga(+), refines the initial noise FT into the refined noise êr, which is then fed into the same diffusion pipeline to generate an image åo using N denoising steps without guidance. The training objective minimizes the L.2 distance d(āg. 26 Guide between the unguided out- put ån and the guided target zo Guide Through this process, noise refining network learns to transform initial noise into refined noise, capturing the benefits of guidance without modifying the diffusion model. At inference time, we sample Gaussian noise, apply single refinement step once before denoising, and then perform the standard unguided denoising process starting from the refined noise. Note that both noise refining network and the original model also receive prompt c, though omitted for simplicity.
Noise refining network. Although the architec- ture of noise refining network (+) can be chosen flexibly, we adopt lightweight LoRA (Hu et al., 2021) on pretrained diffusion models. This allows noise refining network to effectively leverage the dif- fusion model's rich knowledge of text and image information while enabling parameter-efficien fine- tuning and faster convergence. Moreover, instead of loading new refining network, LoRA can be at- tached for refinement and then detached, reducing GPU memory usage and seamlessly integrating with the original diffusion pipeline.
Table 1: Quantitative comparison between noise space loss vs. image space loss (ours).
Method
PickScore
HPSx2
AES
CLIPScore
space
-2,269
18.39
Oues
0.258
Noise Space
(Ours) Image Space
To better capture the structured differences between Gaussian and refined noise observed in Fig. 3, we introduce a residual connection in (.).), as shown in Fig. 2 (b), allowing the network to predict only the correction rather than the full refined noise.
Figure 5: Sample images from models ing noise space loss and image space loss.
Mitigating the cost of backpropagation on mul- tiple denoising steps. Widely used foundational diffusion models, such as the Stable Diffusion family, typically require 20-30 denoising steps to produce high-quality results. Although it is possible to naively apply our method, doing so would incur high computational costs due to backpropagation through the denoising network up to N times, along with substantial GPU memory usage, making training inefficient. These constraints have been a major bottleneck, limiting many prior noise optimization methods (Eyring et al., 2024; Kim et al., 2024) to one- or few-step diffusion models (Lin et al., 2024; Sauer et al., 2024b).
5

---

Published as a conference paper at ICLR 2026
To circumvent the backpropagation costs of the full-step diffusion model, propose novel ap- proach, "multistep score distillation (MSD)", where we detach gradients through the denoising net- work during backpropagation, inspired by score distillation sampling (Poole et al., 2022).
Specifically, the typical denoising process is:
Di Dr(go(2T)),
where Dx(2) represents single denoising step:
Dz(x) - axTs + betg (x);
where de and b, coefficients derived from the DDIM sampler (Song et al., 2020a) and for- mally defined in Appendix B. Then, the loss Denoise, defined as the L2 loss between the denoised image and the target image "0 is given by
Demoise (go (xT), = d (... (g( 2 Guide)
(3)
where d represents the L2 distance.
In MSD, we perform the typical denoising process but detach the gradients the denoising network ca at each step. Specifically:
Loss (MSL))
( = d (...Fr((o((T))
Guide
where
Optimize steps
Fz = a, + SG (2 (z)).
(5) Figure 6: Comparison of optimization re- sults. Orange: full-gradient MSE optimiza- tion; Blue: MSD loss optimization.
SG(-) denotes the stop-gradient (detach) operation.
We conduct a pilot experiment to examine the effect of detaching gradients in the denoising process. Specifically, we optimize initial Gaussian noise JT using the MSD loss LMSD and the full-step gradient loss Densise to make the denoised image close the given target 20 Guide and compare the results in Fig. 6. As shown, detaching gradients leads to faster convergence and sharper images while significantly reducing computational costs. This improvement arises because omitting the de- noiser Jacobian avoids unstable multi-step backpropagation and helps mitigate gradient explosion vanishing that occurs when repeatedly backpropagating through the same denoiser, similar to long- horizon instability in recurrent networks. We further illustrate this stabilization effect in Fig. 35-36 in Appendix B.3.
In our training framework, the noise refining network (-(-) is trained to minimize LMSD (5)(()))))) with respect to the refining network parameters 6. We validate our approach, demonstrating that MSD closely approximates learning with the full-gradient loss Denoise Io (#T) ) 9). This is formal- ized in the following proposition, with detailed proof and empirical evidence in Appendix B.2-B.3.
Proposition 2. By approximating the gradients through multistep score distillation (MSD) using detached gradients at each step, we approximate the full-gradient objective with a mild assumption. In conclusion, the two gradients can be approximated as follows:
V LDeoniee Vo 0) =o RV LMsD (TT), B),
(6)
where & E (0, 1) is constant.
4 EXPERIMENTS
In this section, to show the effectiveness and efficiency of the noise refining network, we present extensive qualitative and quantitative results. Following this, demonstrate the advantages of our method which stem from preserving the diffusion pipeline intact, such as its generalizability to other fine-tuned diffusion models and compatibility with time-step distillation methods.

---

Published as a conference paper at ICLR 2026
(ours) Refined noise w/o guidance
(ours) Refined noise w/o guidance
Random noise w/o guidance
Random noise w/ guidance
Random noise w/o guidance
Random noise w/ guidance
M
goldfich
free frog
"majestic cat mountiin rop"
"A steampunk airship"
"A man stonding in a room next to metal and red pole."
"A mon in purple shirt and tie and purple hair."
Figure 7: Qualitative results. Samples generated (left) from Gaussian noise without guidance, (middle) from Gaussian noise with sampling guidance (Ho Salimans, 2022; Ahn et al., 2024), and (right) from refined noise without guidance. The first row present results from SiT-XL/2, the next row from SD and the final row from SDXL.
4.1 SETUP
Training setup. To evaluate the effectiveness of NoiseR we train the noise refining network on three models with distinct conditions, objectives, and architectures: a class-conditional flow- matching model (SiT-XL/2 (Ma et al., 2024)) and two text-to-image (T21) diffusion models (Stable Diffusion (SD) 2.1 (Rombach et al., 2022) and SDXL (Podell et al., 2023)). For T2I models, prompts are sampled from MS COCO (Lin et al., 2014) and Pick-a-Pic (Kirstain et al., 2023) for training. Notably, method does not require paired image datasets. The refining network is trained with classifier-free guidance (CFG) (Ho & Salimans, 2022) on the class-conditional model, and with both CPG and perturbed-attention guidance (PAG) (Ahn al., 2024) the T21 models. Further implementation details are provided in Appendix D.
Evaluation setup. For T2I models, we generate 30K images for evaluation from 30K unique prompts randomly sampled from the MS COCO 2014 validation set, disjoint from the training split. For SIT-XI//2, we use 50K samples with ImageNet conditions. All qualitative examples are drawn from these sets. For the main experiments with SiT-XL/2, we use the Euler sampler with 20 denois- ing steps. For SDXL and SD 2.1, we use the DDIM sampler with 20 denoising steps.
4.2 QUALITATIVE AND QUANTITATIVE EVALUATION
Qualitative comparison. Fig. presents representative samples. Without guidance, Gaussian noise yields spatially incoherent images (1st, 4th columns), while refined noise produces consistently higher-quality results with plausible structure (3rd, 6th columns). This underscores the critical role of the initial noise and demonstrates that our refining network distills guidance signals into spatially informed noise, enabling consistent generations. Additional results on SiT-XL/2, SD 2.1, and SDXL are provided in Appendix (Figs. 40-45). For completeness, we also provide results on SiT using the original evaluation protocol (250-step 2nd-order Heun sampler) in Appendix A.
evaluate image fidelity and diversity, we compute Fréchet Incep- tion Distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016) as shown in Tab. For each model, we compare four settings: (1) unguided sampling from Gaussian noise (our baseline), (2) guided sampling from Gaussian noise, (3) quidance-distille sampling (Meng et al., 2023) from Gaussian noise, and (4) unguided sampling from refined noise with noise refin- ing network. Refined noise consistently improves FID and IS over Gaussian noise, and achieves
Quantitative comparison.

---

Published as a conference paper at ICLR 2026
Gravinn
Cuzasia toie
Adiand ooo
Relind
Grasnaz
Xafiedd
Anime
Clay
Generalizability fine-tuned model
Goneralizability on timestep-distilled model
Figure 8: Generalizability and compatibility of refined noise. (a) Results on fine-tuned models
(animation and clay object domains) comparing Gaussian vs. refined noise. (b) Results on timestep- distilled models (SD-Turbo), showing that refined noise improves structural coherence and quality over Gaussian noise.
quality close to guided or guidance-distilles sampling with only a single refinement step. Details on implementation of each method are provided in Appendix D.
Training cost. Since our proposed method involves guided sampling during training, it introduces additional computational overhead compared to guidance distillation Meng et al. (2023). For a fair comparison, we provide de- tailed training cost analysis in Appendix A.7, and report the performance of both methods over training steps in Fig. 26 (Appendix A.8).
Table Quantitative comparison of image quality. 30K prompts from MS-COCO (Lin et al., 2014) validation dataset were used for evalua- tion. Guidance Distil. indicates guidance distil- lation (Meng et al., 2023).
Initial
Sumpling Guidunce
1
Gaussian
18.43
40.00
User study. We conducted a user study to evaluate prompt adherence and image qual- ity by comparing guided samples from Gaus- sian noise with unguided samples from refined noise. shown Tab. 3, participants pre- ferred refined-noise samples and guided sam- ples at similar rates. Additional details and comparisons with unguided Gaussian noise are provided in Appendix E.2.
SIT
Gaussian
Gaussian
X (Gtidonce Distil.)
14.20
12.12
63.99
58.90
Refined (Ouns)
X
10,80
50.59
Gaussian
42,71
20.86
SIX2.1
Gaussian
Gaussian
x Guiddmce Distil.)
16.19
19.09
37.95
33.45
Refined (Ours)
X
14.62
34.90
Gaussian
X
63.28
17.61
Gaussian
21.20
34,60
SDXL
Gaussian
(Guidunce Distil))
18.57
Refined (Ours)
x
26.22
27.63
User study on image quality and prompt adherence.
Table
Ablation studies.
We provide additional ab- lations Appendix C, including network ar- chitecture, number of denoising steps N, and other factors. We also report the results of SiT- XL/2 Ma et al. (2024) using the Heun sampler with 125 denoising steps, following the original SiT paper, in Appendix A.13.
Gaussian Noise
Refined Noise (Ours)
Metric
w/ Guidance
wo Guidance
Image Quality
46.04% 48.24%
53.96% 51.76%
Prompt Adherence
4.3 ADVANTAGES OF NOISE REFINEMENT
In this subsection, we highlight the advantages of noise refining for guidance- generation. This approach preserves the diffusion pipeline, including the denoising network, maintaining the model's integrity. Our method can be viewed as a form of prompt learning (Zhou et al., 2022a), which prevents catastrophic forgetting (Kirkpatrick et al., 2017). Further discussion is available in A.
Generalizability on differrnt domains. Guidance distillation (Meng et al., 2023) can remove the need for guidance in base model. Yet, applying it to fine-tuned models necessitates a separate distillation step for each variant, making the process computationally expensive. noise refining network, trained on the base model, can be directly applied to fine-tuned models, enabling efficient adaptation across multiple domains. We present this by transferring our noise refining network, trained on Stable Diffusion 2. to fine-tuned model in the animation and clay object domain. Fig. (a) shows that our model effectively refines noise, eliminating the need for guidance across different domains. We also provide quantitative results for this zero-shot transfer of the noise refining network in Tab, 4, showing performance comparable to guided generation. Additional results are in Appendix E.3.
contrast, our
8

---

Published as a conference paper at ICLR 2026
Table Quantitative results generalization finetuned models across different domains.
Initial
Guidance
PickScore
ImugeReward
Aesthetic
CLIPScore
Goussion
X
17.95
0.18
-1.69
5.32
18.31
Clay
Gaussian
/
19.17
Refined (Ours)
X
18.82
0.21
-0.95
5.39
23.61
Gaussion
X
16.47
0.17
-1.59
22.37
Anime
Gaussian
5
17.68
0.24
0.04
5.56
30.04
Refined (Ours)
Compatibility with timestep distillation models. Our method integrates seamlessly with existing timestep distillation approaches (Luo et al., 2023a; Sauer et al., 2024a; Xu et al., 2024b; Luo et al., 2023b; Yin al., 2024; Lin al., 2024; Salimans Ho, 2022) without requiring additional training, since it preserves the diffusion pipeline unchanged. We apply refined noise to SD-Turbo (Luo et al., 2023a) and evaluate its performance. Qualitative results are shown in Fig. 8 (b), and quantitative comparisons are reported in Table 5.
Table 5: Quantitative results of refined noise on timestep-distilled model (SD-Turbo).
Noise
Denoising Step
PickScore
HPSv2
AES
ImageReward
CLIPScore
FID 1
IS 1
Gaussian
Step Step Step
21.38
0.270
5.47
0.04
30.97
27.18
34.31
Guussiun
21.75
0.295
5.62
30.24
Refined
21.92
0.300
5.51
0.43
31.19
24.94
38.07
Compared to generation starting from Gaussian noise, our approach improves structural coherence and overall quality, highlighting the role of structured initial noise even in few-step models. More- over, single-step inference with refined noise generally outperforms two-step inference from Gaus- sian noise in terms of numerical metrics.
DISCUSSION
In this section, we analyze what noise refining network learns and identify components in refined noise that support better generation quality.
Low-frequency components aid denoising. Analysis of the refining network's output shows that primarily adds low-magnitude, low-frequency signals. In Fig. 9 (a). the difference between Gaus- sian and refined noise is concentrated in small values, unlike the difference between two Gaussian samples. Moreover, (b) indicates that noise refining network naturally produces www-freuency lay- outs without explicit constraints. This observation is consistent with Fig. 3, where the gap between inversion noise and Gaussian noise also lies mainly the oow-frquency range.
As illustrated in Fig. 10, these components are condition-dependden and serve initial layout, shaping object structures early in denoising and improving coherence. To further examine their role, we performed frequency decomposition (separating low- and high-frequency contributions; Appendix A.2) and cross-prompt experiments (testing robustness under mismatched prompts; Ap- pendix A.3), which highlight the critical importance of low- wowfrequency signals.
85
15-
62x
0..
Aboolts dTeeees
Tadial Teeguery
(a)Allolu Diffetence Hiatorum
(3) Pregptscy-Miggiittee
Figure 9: Analysis of initial noise Pr refined noise îr. (a) Histogram of ab- solute differences (vs. pairs of random
Figure 10: Analysis of initial noise Fr tined noise âτ. The top row visualizes the absolute noise difference, and the bottom row shows the cor- responding generated images. The added signal acts as a coarse structural layout for generation.
Gaussian noise). (b) Fourier magnitude differences, showing variation mainly in low frequencies.

---

Published as a conference paper at ICLR 2026
Random noise w/o guidance
Random noise w/ guidance
Refined noise w/o guidance (ours)
Figure 11: Refined noise enables coherent trajectories. From left right, To predictions shown as goes from T to O. Refined noise yields a consistent trajectory by providing initial layout.
Truining Dataset
Generated image w/ sume prompt (Ours)
0.75
"a pizza covered greens on plate
"A white plate with brownie and white frosting"
Figure 13: Controlling the strength of noise refinement. Numbers in the top-left corner in- dicate the scaling factor of 9ø-
Nearest generated images from training images. From 10 samples per training image, the top-3 nearest neighbors are shown, indicating novelty beyond the training set.
Figure 12:
Consistent trajectory. The third row of Fig. 11 shows that starting from refined noise, model quickly forms plausible layouts in early steps, enabling it to focus on adding details during denois- ing. In contrast, the first row shows that Gaussian noise fails to establish a coherent structure early, leading to misplaced details and leaving ambiguous regions untouched throughout denoising. We also analyze the corresponding cross-attention maps in Fig. 20 of Appendix.
Diversity and novelty. Although refined noise provides initial layout, results remain diverse across seeds, with IS (Salimans et al., 2016) scores surpassing those from Gaussian noise (Tab. 2). neigest retrieval (Fig. 12) confirms that the outputs are not simple replicas of training data but genuinely novel samples.
Controllability. The strength of guidance can be adjusted in two ways. In the training free case, scaling the output of ga. the residual between Gaussian and refined noise, controls the coherence of image structure (Fig. 13), analogous to tuning the guidance scale. In the training-based case, the model can be conditioned
additional guidance-scale embedding, with results Appendix A.4.
Comparison to related works. Our goal is to distill diffusion guidance into the noise space, which differs from prior work noise reinitialization, search, optimization. For completeness, we dis- cuss related approaches and their objectives, and provide comparison experiments Appendix A.9.
CONCLUSION
In this work, we propose NoiseRefine, a method that replaces costly guidance in diffusion sam- pling with single noise refinement step. Our approach preserves the original diffusion pipeline, prevents catastrophic forgetting, and enables seamless integration with existing timestep distillation techniques (Meng et al., 2023; Sauer et al., 2024b) enhance image quality and coherence. Ad- ditionally, we analyze the properties of refined noise and its role in denoising, providing insights into the influence of noise diffusion models. We believe our work paves the way for leveraging expressive noise space in a training-based manner.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
We provide detailed explanations and proofs of the theoretical results in Appendix B, and further describe the architecture, implementation, and experimental details in Appendix D. We will also release our code and model checkpoints to ensure reproducibility.
ACKNOWLEDGMENTS
This research supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) RKS-2019-11190075, RS-20244 00509279, RS-2025-I121206
RS-2025-02214479, RS-2024-00457882, RS- 2025-254418 838, RS-2025-25441888 RS-2025-02214477) RS-2025-02217255) and the Culture,
Sports, and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism RRS-2024-00345022
35-2022-00033066 RS-2023- Korea (RS-2024-003465977 We also gratefully acknowledge the provision of GPU compute resources by Hugging Face for this research.
00222280,
and National Research Foundation
REFERENCES
Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-atttenioon guidance. arXiv preprint arXiv:2403.1733 2024.
Yuanhao Ban, Ruochen Wang, Tianyi Zhou, Boqing Gong, Cho-Jui Hsieh, and Minhao Cheng. The crystal ball hypothesis in diffusion models: Anticipating object positions from initial noise. arXiv preprint arXiv:2406.01970 2024.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or, Attend-and-ee excite: Attention-baal basedd semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis, Advances in neural information processing systems, 34:8780-8799 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. Reno: Enhancing one-step text-to-image models through soward-based noise optimization. arXiv preprint arXiv:2406.044: 04 2024.
Daniel Garibi, Or Patashnik, Andrey Voynov, Hadar Averbuch-Elor, and Daniel Cohen-Or. Renoise: Real image inversion through iterative noising. arXiv preprint arXiv:2403. 14602, 2024.
Daniel Geng, Inbum Park, and Andrew Owens. Factorized diffusion: Perceptual illusions by noise decomposition. In European Conference on Computer Vision, pp. 366-384. Springer, 2025.
Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting text-to-image diffusion models via initial noise optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9380 9389, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207. 12598, 2022.
11

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6883 2020.
Susung Hong. Smoothed energy guidance: Guiding diffusion models with reduced energy curvature of attention. arXiv preprint arXiv:2408.00760, 2024.
Susung Hong. Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7462-7471, 2023.
Yi-Ting Hsiao, Siavash Khodadadeh, Kevin Duarte, Wei-An Lin, Hui Qu, Mingi Kwon, and Ratheesh Kalarot. Blug-and-play diffusion distillation. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Patlern Recognition, pp. 13743-13752, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1.Xiv:2006.09685, 2021.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi,, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. arXiv preprint arXiv:2406.02507 2024.
Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakorn, and Siyu Tang. Optimizing diffusion noise serve universal motion priors. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1334-1345, 2024.
Jeeyung Kim, Ze Wang, and Qiang Qiu. Model-agnostic human preference inversion in diffusion models. arXiv preprint arXiv:2404.0087,, 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Baaibssk--aaiiskaa,..a et al. Overcom- ing catastrophic forgetting in neural networks. Pruceedings of the national academy of sciences, 114(13):3521-3322 2017.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion
distillation. arXiv preprint arXiv:2402.13922, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Eision-ECCV 2014: 1 13th European Conference, Zurich, Switzerland. September 6-12, 2014, Proceedings, Part V 13. pp. 740-755. Springer, 2014.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX1v:2211.01095. 2022.
Jian Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprint arXiv:2310004377 2023a.
Simian Luo, Yigin Tan, Suraj Patil, Daniel Gu, Patrick Platen, Apolinário Passos, Longbo Huang. Jian Li, and Hang Zhao Lem-lora: A universal stable-diffusion acceleration module. arXiv preprint 231 05556, 2023b.
12

---

Published as a conference paper at ICLR 2026
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- and Sain- ing Xie. Sit: Exploring flow and diffusion-basee generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23 4340.. Springer. 2024.
Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, al. sifference-tmme scaling for diffusion models beyond scaling denoising steps. arXiv preprint urXiv:2501.09736 2025.
Harvey Mannering, Zhiwu Huang, and Adam Prugel-Bennett. Noise-level diffusion guidance: Well begun is half done. arXiv preprint arXiv:2509. 13936, 2025.
Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. Guided image synthesis via initial image editing in diffusion model. Proceedings of the st ACM International Conference on Multimedia, pp- 5321-5329, 2023a.
Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. Semantic-driven initial image construction for guided image synthesis in diffusion model. arXiv preprint arXiv:23/2.08872, 2023b.
Barak Meiri, Dvir Samuel, Nir Darshan, Gal Chechik, Shai Avidan, and Rami Ben-Ari. Fixed-point inversion for text-to-image diffusion models. arXiv preprint arXiv:2312.125400 2023.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint 2rXiv:2307.0/992 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. /4988, 2022.
Zipeng Qi. Lichen Bai, Haoyi Xiong, et al. Not all noises are created equally: Diffusion noise selection and optimization. arXiv preprint arXiv:2407. 14041, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763, PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-lo-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 22500- 22510, 2023.
Seyedmorteza Sadat, Manuel Kansy, Otmar Hilliges, and Romann Weber. No training, no prob- lem: Rethinking classifier-free guidance for diffusion models. arXiv preprint arXiv2240..02687, 2024.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 0rXiv:2202.00512, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
13

---

Published as a conference paper at ICLR 2026
Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. Generating images of rare concepts using pre-trained diffusion models. Proceedings of the AAAI Conference on 2024.
Artificial Intelligence, volume 38, pp.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. In SIG- GRAPH Asia 2024 Conference Papers, pp. 1-11, 2024a.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation. In European Conference Computer Vision, pp. 87-103. Springer, 2024b.
Christoph Schuhmann. Improved aesthetic predictor, 2022.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.1555 15980, 2020.
Vedant Singh, Surgan Jandial, Ayush Chopra, Siddharth Ramesh, Balaji Krishnamurthy, and Vi- neeth N Balasubramanian. On conditioning the input noise for controlled image generation with diffusion models. arXiv preprint arXiv:2205.03855 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv
preprint arXiv:2010.02502, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011..34550 2020b.
Chen Henry Wu and Fernando De Torre. Making text-to-image diffusion models zero-shot image- to-image editors by inferring" random seeds". In NeurIPS 2022 Workshop Score-Based Meth- ods, 2022.
Katherine Xu, Lingzhi Zhang, and Jianbo Shi. Good seed makes a good crop: Discovering secret seeds in text-to-image diffusion models. arXiv preprint arXiv:2405.14828 2024a.
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024b.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6613-6623, 2024.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 168 16-16225, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2348, 2022b.
Zhenyu Zhou, Defang Chen, Can Wang, Chun Chen, and Siwei Lyu. Dice: Distilling classifier-free guidance into text embeddings. arXiv preprint 2502:2550..03726, 2025.
Zikai Zhou, Shitong Shao, Lichen Bai, Zhiqiang Xu, Bo Han, and Zeke Xie. Golden noise for diffuuson models: A learning framework. arXiv preprint arXiv:2411.09502 2024.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
In the Appendix, we provide discussions including the in-depth analysis on refined noise and its impact denoising process (Section A), clarify the notations and formulations related diffusion models used in the main paper and provide the proofs for our propositions (Section B), more abla- tion studies regarding noise refining network (Section C), implementation details and experimental settings (Section D), additional results including qualitative results, comparison with other methods, user study (Section E).
CONTENTS
A Additional Discussions
A.1
Effectiveness of prompt learning
16
A.2
Why does refined noise help denoising?
16
A.3
Impact of initial noise and prompt the generated image
20
A.4
Controllability
22
A.5
Guided sampling with refined noise
22
A.6 Inference time analysis
23
Training cost analysis
23
A.8 Comparison training efficiency with guidance distillation
23
A.9 Comparison with other noise optimizationnrefinmment works
24
A.10 Robustness to the number of denoising steps and samplers
25
A.11 Analysis of DDIM Inversion.
28
A.12 Comparisons with direct training with CFG-generated data.
29
A.13 Comparison Under the Same Settings as the SiT Paper
30
Theoretical Background
31
B.I Preliminaries..
31
B.2 Derivations
32
B.3 Diffusion model Jacobian approximation
34
More Ablation Studies
38
C.I. Noise refining network Number denoising steps
38
D Implementation and Experimental Details
39
D.1 Implementation details
39
D.2
Experimental details
40
E Additional Results
41
E.1 Additional qualitative results
41
User study
................................
41
Generalization other domains
42
F LLM Usage Disclosure
49
15

---

Published as a conference paper at ICLR 2026
A ADDITIONAL DISCUSSIONS
In this section, discuss the advantages of training noise refining network Is for quidance-fres generation (Sec. A.1). In addition, we present our hypothesis on why refined noise eliminates the need for guidance methods, explaining step by step (Sec. A.2). We further analyze the impact of initial noise and prompt on the generated image (Sec. A.3).
A.1 EFFECTIVENESS OF PROMPT LEARNING
Why learning noise mapping beneficial? A useful perspective comes from the success of prompt learning in large-scale models. Models such as CLIP (Radford et al., 2021), trained on web-scale datasets with billions of parameters, are difficult to fine-tune due to their sheer size and the risk of disturbing well-learned representations (Zhou et al., 2022b). Instead, prompt learning, which optimizes input prompts rather than model parameters, has emerged as an effective alternative (Zhou al., 2022b;a; Jiang et al., 2020; Shin et al., 2020). In particular, conditional prompt learning methods like CoCoOp (Zhou et al., 2022a) generate prompts based on different inputs. Similarly, in approach, noise prompts learned based on Gaussian noise zr and the text prompt c, allowing for more efficient guidance.
In this context, restricting training the noise space rather than modifying the entire denoising pipeline offers several advantages. As illustrated in Fig. 9 and Fig. 10, key low-frequency compo- nents in the noise space encode structural information such image layout. This enables efficient learning with a relatively small dataset, without requiring modifications to the entire model. By contrast, full fine-tuning often leads excessive computational costs and the risk of overfitting.
More importantly, unlike guidance distillation methods such as (Meng et al., 2023), our approach preserves the original model and prevents catastrophic forgetting (Kirkpatrick et al., 2017). This ensures that pretrained modules, such as DreamBooth (Ruiz et al., 2023) or LoRA (Hu et al., 2021), remain fully compatible. Fig. 14 illustrates this effect: when applying the Miranda Kerr LoRA, guidance distillation (Meng al., 2023) alters identity characteristics, whereas our method preserves original sample's s identity while improving image quality. This demonstrates that our method maintains the integrity of the representation space, while guidance distillation compromises it.
Gaussian noise
Refined noise
Guidance Distill
Figure 14: Compatibility of LoRA for each method. Results of applying the 'Miranda Kerr'
LoRA, trained on 2.1. Distilled model exhibits different identity with 'Miandaa Kerr'.
A.2 WHY DOES REFINED NOISE HELP DENOISING?
To identify which refined noise components contribute to guidance-free generation, we first de- compose the refined noise into multiple frequency components. this study, we utilize two- dimensional Fourier transform to break down both the refined noise and the initial noise into their respective frequency components. Each frequency component is represented by a frequency band, denoted as (a.b), b), which corresponds to the frequency range from to b. Note that although explored other decomposition methods, such as dividing the noise into patches, they did not yield interpretable results.
16

---

Published as a conference paper at ICLR 2026
(a) denoised images according cutoff band
(0, 0.1)
(0.1, 0.2)
(0.9, 1)
(b) denoised images according the cutoff radius
Figure 15: Visualization of denoised images according to the cutoff band. Both refined and initial noise were transformed into the frequency domain using Fourier transforms. The frequency domain of the initial noise, normalized such that the maximum radius is 1. (a) The frequency divided into intervals of 0.1. For each interval, the corresponding frequency components were replaced with those from the refined noise, followed by denoising. The results show that only when the (0, 0.1) frequency band was replaced does an image generated by the refined noise emerge. (b) Visualization of denoised images by incrementally increasing the cutoff radius from 0 in steps of 0.01 and replacing the corresponding components of the initial noise with refined noise. The results demonstrate that images denoised using refined noise are obtained starting at a cutoff radius of 0.03.
200
175
150-
Nocs
100
59
25
0.4
Figure 16: Visualization of the norm based on the frequency-fittrred radius of refined noise. This visualization demonstrates the increase in norm as the cutoff radius in the frequency domain is expanded. The refined noise was transformed into the frequency domain using Fourier transform, and the norm corresponding each cutoff radius was calculated and plotted.
Low-frequency components matter. Using 2D Fourier transforms. we transform both refined and initial noise into the frequency domain. The initial and refined noise frequeecy domain is normalized into (0, 1). We synthesize new noise signal by replacing specific frequency bands of the initial noise with the corresponding bands from the refined noise. Fig. 15 (a) presents the generated images corresponding to different frequency bands, demonstrating that the low-frequency components of the refined noise predominantly influence generation process. In Fig. 15 (b), images are generated by varying the band length within the low-frequency region. The results indicate that, despite the low magnitude of the low-frequency components, which can be confirmed through Fig. 16, they are sufficient reconstruct the image effectively.
Diffusion models can generate images using only low-frequency components. In Fig. 17, we examine how well diffusion models can denoise when specific frequency bands of refined noise are retained, and the values of the remaining bands are set to zero (using ideal high/low pass filters). The top row shows the results of applying 2D Fourier transform to the refined noise, normalizing the FFT frequency domain into (0, 1). and sequentially retaining lower frequency bands, such as (0, 0), (0. 0.1), (0, 0.2). .... (0, 1), while setting the remaining bands to zero. These noise inputs then denoised without CFG (Ho & Salimans, 2022). The figure demonstrates that the diffusion model begins forming recognizable corgi shape even when only the lower 50% of frequency bands
17

---

Published as a conference paper at ICLR 2026
Frequency band to keep, otherwise 2ero (0, 0.1)
(0.0)
(6, 0.2)
(0.0.3)
(0, 0.4)
(0,9.5)
(0,06)
(0,0.7)
1.00
High Low fleg
(0.9.
Figure 17: Denoised images using only low(top) high(bottom) frequency components. Diffu- sion models can generate the overall structure of the image using only the low- cowffrequency bands of the refined noise. We use DDIM (Song et al., 2020a) with 20 steps for denoising without CFG, and the prompt was "a photo of corgi".
of the refined noise are present. In contrast, noise containing only high-frequency bands fails to generate coherent images.
Frequency keep,
(0.0)
(0, (.1))
(0,0.3) 0.20
(0, 0.3)
(0, 0.4)
(0,
(0.0.6)
(0,0.7)
(0. 0.81
10, 0.91
(0. 1.0)
High Low
()
()
(0.2, 1)
(0.4,
(0.5,
(0.7, 1)
(0.8,
(0.9, 1)
Figure 18: Denoised images using only low (top) high (bottom) frequency components with reinitialization. We use DDIM (Song et al., 2020a) with 20 steps for denoising without CFG, and the prompt was "a photo of corgi".
High-frequency components contribute details. Here, we use the same noise decomposition process of refined noise Fig. 17 but following (Geng et al., 2025), reinitialize the frequency components that were set to zero with corresponding components from standard Gaussian noise, then denoise again. The results, shown in Fig. 18, indicate that when all frequency components are present, the diffusion model can generate clear and complete images. Randomly reinitialized high-frreuenne components appear add details onto the structure formed by the low- www-freuenccy components. While refined noise retaining only the lower 10%-20% of frequencies can still re- construct the original image when the rest reinitialized, noise retaining only the high- feggufrry components fails to do so. This suggests that low-frequency components alone carry the significant information needed for image generation.
In Fig. 19, each row visualizes images generated with only the lower 5%, 10%, 20%, and 30% (from the top rows to last rows) frequency components of the refined noise, while the bottom row shows images generated with only the upper 5%, 10%, 20%. and 30% frequency components. These results confirm that www-frquency components encode the overall layout and structure, whereas high-free frggefregeency components lack meaningful information.
From these observations, we infer that the poor quality of unguided diffusion model outputs is due to their failure to form appropriate low-frequency components during denoising. High-frequency details added on poorly formed layouts result in artifacts that are perceived as unnatural.
How do guidance methods form plausible initial layouts? As highlighted (Ahn et al., 2024), classifier-free guidance (CFG) (Ho & Salimans, 2022) enhances the difference between conditional and unconditional predictions at each step, amplifying "signals that can only be generated with the condition" (e.g.. features like the eyes nose of corgi in "a photo of corgi"). This effectively strengthens salient features corresponding to low- www-freuencc components the early denoising steps. From this, deduce that guidance methods (Ahn et al., 2024; Ho & Salimans, 2022: Hong al.,
18

---

Published as a conference paper at ICLR 2026
Seat
00, 0.05
06.55
0..0
10, 0.303
0.3033
(a) Low-frequency components refined noisc
(b) High-frequency components of refined noise
Figure 19: Different denoised images using only low(a) / high(b) frequency components for different seeds. Here we use 8 different seeds. From the top rows, it visualizes 8 images using only the lower (a) higher (b) 5%, 10%, 20%, and 30% (from the top the last rows) frequency components of the refined noise.
2023) add appropriate low-frequency components during inference, aiding the formation of high- quality layouts.
Denoising Direction
Random noise w/o guidance
Random noise w/ guidance
Refined noise w/o guidance
(ours)
Figure 20: Visualization of 11th layer cross attention map. Token corresponding to 'cat' is used for visualization among the prompt photo of a cat'. For each case, first row shows zo prediction at each timestep and second row shows cross attention map at the timestep. When guidance is not used, failure create meaningful attention map across all timestep is notable, leading to completely refined noise is used, meaningful cross attention
broken generation. However when guidance map is observed, leading to successful generation.
How does noise refining network form low-frequency layouts? Interestingly. noise refining net- work naturally forms low-frequency layouts even though our training framework does not explicitly enforce learning them as can be seen in Fig. 9. To understand this, analyze cross-attention maps across denoising steps. Fig. 20 visualizes these maps at different timesteps. Gaussian noise fails to form meaningful atts-ttt maps in early steps due to its near-zero signal-tt tonal-to-ooiee ratio (SNR), which is expected. However, this failure persists in later steps, indicating an inability to form well- aligned layouts (Fig. 20 first row).
19

---

Published as a conference paper at ICLR 2026
Several studies (Chefer et al., 2023; Guo et al., 2024; Mao al., 2023b) has shown that reducing noisy artifacts in cross-attention maps and aligning them with object regions during inference im- proves performance. This suggests that the failure of cross-attention maps align is key reason for the diffusion model's inability to create coherent layouts. When using CFG (Ho Salimans, 2022) (second row) refined noise (third row), the cross-attention maps align well with the prompt, resulting in better outputs. Notably, tross-attention maps for refined noise exhibit accurate object shapes from the very first step, implying that the diffusion model can form plausible layouts from the beginning of the denoising process. This is further supported by zo predictions of Fig. 20 at each denoising step.
Implications for guidance-free generation. Without guidance methods noise refiners aiding the formation of low-frequency layouts, diffusion models fail create plausible initial layouts. Ran- dom low-frequence components lead to artifacts that perceived unnatural. An interesting enue for future research would be identifying why diffusion models struggle to form low- www-freuueccy components without guidance and developing training techniques to eliminate the need for guidance during the training stage.
A.3 IMPACT OF AN INITIAL NOISE AND PROMPT ON THE GENERATED IMAGE
Nuise refining prompt P₁ "u photo of a lion in the wild"
Denoising
T
T
(a) Gaussian noise,
"a photo lion the wild"
(b) Refined noise, P2 "a photo of a lion in the wild"
(c) Refined noise, Pg
(d) Refined noise, P₂ "a photo of a tiger in the wild"
(e) Refined noise, P2 "a laptop computer on desk"
(e) Refined noise, P2 "a laptop computer on a desk", with CFG
Figure 21: Visualization of denoised image using different prompt for noise refinement eg and denoising 9ø-
20

---

Published as a conference paper at ICLR 2026
We previously demonstrated how refined noise affects initial layouts and how guidance and refined noise contribute to forming these layouts effectively. In this section, we investigate how the 'layout' and the prompt influence the final generated image during the denoising process. Specifically, explore what happens when the prompt used to generate the initial layout (P1, one of the inputs noise refining network gø) differs from the prompt used during denoising (P2, one of the inputs to the denoising network Eø in the Guidance- Free T2I Pipeline shown in Fig. 38), Does the model prioritize one prompt over the other? Or does attempt harmonize both? We investigate this question through the results shown in Fig. 21.
Fig. 21 (a) visualizes the predicted zo term in Eq. 15 during the denoising process when no layout is provided (starting from Gaussian noise). The lefumost image corresponds to the predicted 2i at t = T, and subsequent images are visualized every three steps. Due to the noisy and ambiguous nature of the initial layout of Gaussian noise, the diffusion model fails to form a coherent lion layout from the initial structure. Instead, it partially adds features such as fur, mane, nose, mouth, resulting in poor perceptual quality.
contrast, (b) shows that in the case of Pi = P2. refined noise effectively forms the lion layout from the beginning. The diffusion model accurately places the overall lion shape, including its mane, eyes, nose, and mouth, in appropriate positions during the denoising process.
(c) shows the results when the denoising prompt P2 is set to an empty prompt (null prompt). Despite this, the model successfully generates feline animal based solely unconditional generation, as the layout sufficiently captures the overall structure of the object. This can be interpreted as the information embedded in the refined noise.
(d) demonstrates the case where the denoising prompt P2 is set to a prompt similar to the initial layout prompt ("a photo of tiger in the wild"). When similar prompt is used, the image retains the layout provided by the refined noise while also adhering to the prompt.
In (e), Py is set to entirely independent prompt ("a laptop computer on desk"). Here, the model fails to generate a coherent image corresponding to the layout or the prompt. The diffusion model attempts to form laptop the existing lion or feline layout but fails to align with the laptop prompt, leading to failure.
Finally, (f) shows that applying CFG (Ho & Salimans, 2022) in the settings of (c) allows the diffusion model to disregard the initial layout and generate a laptop. This partially explains why CFG consistently produces high-quality images. Randomly generated initial noise is unlikely to align with the prompt (as shown in (a)), and CFG helps the model ignore such initial noise and generate images consistent with the given prompt.
a 0.0
Interpolated Gaussian noise
(b) Refined uoise
Figure 22: Images from interpolated refined Gaussian noise.
Interpolation between refined noise. To evaluate whether noise refining network effectively learss noise mapping, we follow (Song et al., 2020a;b) to perform spherical interpolation on ini- tial noise samples, generating multiple interpolated noises. We then refine each interpolated noise using noise refining network and verify that the refined noises effectively interpolate natural images. In Fig. 22, (a) shows the images denoised by the diffusion model without any guidance method,
21

---

Published as a conference paper at ICLR 2026
0,0
9.0
0.0
(ours)
(ours)
Refined nosse
Refined noise
gridence
gridunce
Random noisr
Random noise gudancee
guidance
"A pootoo a car deanted, Hi, realiate."
"A pbooo of not
(ours) Refined noist
(ours) Refined noise guidance
guidance
Raidom
Random
wi guidance
wi guidance
of Seautifal, noday Iwaee"
Beantid
Figure 23: Qualitative results of the training-basse approach for controlling guidance strength. Numbers above each column indicate the guidance scale provided as input to the model. Our results show that NoiseRefine effectively learns the controllability of classifier-fres free guidance, preserving the expected variations in image characteristics as the guidance scale changes.
starting from spherical interpolations of two random Gaussian noises. Specifically, each interpo- lated noise is obtained by performing slerp(æT, 2T,,, a) for various interpolation ratios a, where slerp performs spherical interpolation between two Gaussian noise at ratio of a.
Fig. 22 (b) shows the results of denoising the refined versions these interpolated noises without guidance. The results demonstrate that the refined noises effectively interpolate between the two images. This indicates that noise refining network does not simply memorize specific low- www-frequency signals while ignoring the input noise. Instead, it effectively learns mapping from Gaussian noise space to guidance-free noise space where semantic interpolation between guidance-free images is possible.
A.4 CONTROLLABILITY
The guidance strength can be controlled by scaling the output of the noise refining network, which provides a training-free control mechanism. We further show on SD2. that controllability can also be achieved through a training-based approach by incorporating the guidance scale an additional input to the refiner, following (Meng et al., 2023; Luo et al., 2023a). The model architecture and dataset remain unchanged, except for adding small linear projection layer for the guidance scale. Specifically, following the conditioning design in (Meng et al., 2023), we inject the guidance-scale conditioning branch.
embedding into the existing timestep embedding rather than introducing The scale is encoded with a sinusoidal embedding and projected through linear layer to match the timestep-embedding dimension, keeping the architectural change minimal and the training pipeline nearly identical. The model is trained for one epoch. Fig. 23 shows qualitative results, and Fig. 24 reports quantitative results.
A.5 GUIDED SAMPLING WITH REFINED NOISE
Our noise refining network improves image quality not only in unguided sampling but also when guidance is applied. Unlike random Gaussian noise, refined noise (Fig. 10) preserves structural cues and provides consistent "initial layout", reducing artifacts such extra limbs and enhancing overall coherence (Fig. 25).
We also provide quantitative results in Tab. 6, using MS-COCU prompts under the same settings as the main guantitative results in Tab. 2, following the evaluation configurations detailed in Ap- pendix 2.2.2. This shows gains in prompt alignment and human preference.
22

---

Published as a conference paper at ICLR 2026
45
40
30-
23
Guidance Scale
Figure 24: Quantitative results of training-basee approach to control guidance strength. The metries were computed using 5K prompts from the MS COCO 2014 validation set (Lin al., 2014).
Gaussian noise
Refined noise
Gausslac noise
Refined moio:
t'zas
SDXL
Figure 25: Effectiveness of refined noise in guided sampling.
A.6 INFERENCE TIME ANALYSIS
We report comparison results of inference time for each method in Tab. 7. Inference time is com- puted by averaging time per image across 30K images generated with the inference step of 20 and batch size on RTX 3090.
A.7 TRAINING COST ANALYSIS
For training cost analysis based on NFE (Number of Function Evaluation), we assume unit cost for Refine (R), Backpropagation (B), Denoising (D), and VAE (V) operations. Following Appendix D, we denote N' as the number of denoising steps used during inference (including dataset generation) and N as number of denoising steps used during training. In our SD2. setting, we used M 50K samples, E = 6 epochs, N' 20, and N = 10.
Based on these parameters, the total training cost of our method amounts to 5.65M steps, which is only 0.054% of the original training cost of SD2.1 (Tab. 8).
A.8 COMPARISON OF TRAINING EFFICIENCY WITH GUIDANCE DISTILLATION
Although our primary aim is not develop a more efficient guidance distillation procedure but rather to explore whether diffusion guidance can be distilled into noise instead of the network, we nevertheless provide a training-efficiency comparison with conventional guidance distillation for reference. To this end, present the FID curves a guidance-distilled model on SiT-XL/2 in Fig. 26.
steps for training noise refining network and
Note that, aside from the dataset generation phase, training noise refining network imposes higher per-step computational cost: our method applies loss after denoising for N steps, whereas standard guidance distillation computes its loss after single denoising step. Thus, in theory, the GPU cost per step differs by approximately a factor N. Because the two models were trained on different hardware environments, we present step FID eurves rather than wall-clock comparisons.
23

---

Published as a conference paper at ICLR 2026
Model
Noise
PickScore
HPSv2
AES 1
IR
CLIPScore
FID 1
1S
Gaussian
21.70
0.280
5.530
0.294
30.72
18.74
32.55
SD 2.1
Refined
21.93
0.324
5.602
0.448
30,99
34.37
Gaussion
0.280
5.706
0.717
30.77
21.02
34.60
SDXL
Refined
22.48
0.289
5.720
0.977
31.36
22.34
35.23
Table 6: Quality improvement of refined noise in guided sampling.
Model
Initial Noise
Guidance
Inference Time
Gaussian
1.357s
SD2.1
Refined
x
1.504s
Gaussian
/
2.58%
Gaussian
3.218s
SDXL.
Refined
X
3.323s
Gaussian
5.525s
Table 7: Quantitative comparison image quality and computational cost. 30K prompts from MS-COCO (Lin et al., 2014) validation dataset were used for evaluation.
As shown in the figure, training noise refining network converges somewhat more slowly, but reaches the same FID to guidance distillation at around 2K steps
A.9 COMPARISON WITH OTHER NOISE OPTMIEZTIIN/REEIEEEEEY WORKS
Our primary goal is to learning noise space where diffusion guidance is distilled. This objective fundamentally differs from prior work on noise optimization refinement. As result, direct com- parisons are not entirely fair. Several other studies pursue distinct objectives, such as layout syn- thesis (Mao al., 2023b), rare concept generation (Samuel al., 2024), or prompt alignment (Guo et al., 2024). Also, method of (Eyring et al., 2024) is restricted to timestep distilled (one-step) diffu- sion models, where comparisons with multi-step models are infeasible due to memory constraints. Nevertheless, approach is related in terms of improved noise initialization, and thus partial com- parisons can still informative. Thus, we present some comparisons to provide useful insights for the research community.
A.9.1 INITNO
We compare image quality with (Guo al., 2024) using subset of the Attend-and And-Exxite prompt dataset and Stable Diffusion 2.1, optimizing only the initial noise. Since there are no corresponding ground truth images for the Attend-and- Exn-Fxeite prompts, we evaluate both human preference metrics and prompt alignment scores. As shown in Tab. 9, our method outperforms in all metrics, especially in the setting guidance is not used.
A.9.2 PAHI
PAHI (Kim et al., 2024) exists under the category of noise manipulation. To the best of our knowl- edge, this work is the first in its focus learning the noise space itself, rather than optimizing or selecting. Therefore, we compare our proposed approach with this methodology PAHI (Prompt Adaptive Human preference Inversion) (Kim et al., 2024) this section.
There are several key differences between the two approaches. First, the tasks being addressed are distinct. While PAHI (Kim et al., 2024) aims at generating outputs aligned with human preferences, our objective is to replace conventional guidance mechanisms entirely. Second, our method offers much greater flexibility. PAHI (Kim et al., 2024) assumes that sampling from certain (i, E) instead of standard normal Gaussian distribution is more beneficial and predict y and Σ. However, this assumption lacks a strong theoretical foundation. In contrast, our approach aims to learn a gaussian-frr -free noise space without imposing such constraints. Additionally, while PAHI (Kim al., 2024) is limited to few-step models due to the computational overhead of backpropagation, our approach leverages MSD loss, enabling the use of full-step models without modification.
24

---

Published as a conference paper at ICLR 2026
Stage
Formula
# of NFE
Dataset Generation
(2N' D 1.).V)).MM (1 N + B)- M.
2,050K
Post-Training
3,600K
Total Training Cost
Dataset Generation Post- fost-Training
5,650K
Original SD2.1 I Training
Batch Size (2048) Total Step (1.69M) (1 D. D B)
10.383B
Table 8: Training cost analysis based on NFE (Number of Function Evaluation).
NoiseRefine
Guidance Distil
17
16
15
14
13
12
1000
2000
3800
4000
5000
6000
Figure 26: Comparison of training efficiency with guidance distillation SIT-XL/2 (Ma et al., 2024).
Although the official code for PAHI (Kim al., 2024) is unavailable, we adhere to the guidelines presented in their paper as possible and compare with our method. Specifically, we compare noise refining network with the setup that samples noise from N(μ, Σ) where μ and is predicted by MLP for given prompt. Both models are trained with filtered 20K MS COcO(Lin et al., 2014) dataset for 25K steps using two RTX 3090 GPUs. Example qualitative results of employing MLP presented in Fig. 27, and quantitative comparisons noise refining network outperforms the other setup by a significant margin, showing the effectiveness of our proposed method.
shown in Tab.10. Across both evaluations,
Method
FID
MLP (Kim 2024)
217.30
estimating Gaussian parameters
Figure 27: Qualitative results when employing shallow 2-layer MLP for estimating Gaussian parameters, as proposed by (Kim et al., 2024). The results are significantly blurry, indicating that the simple approach of predicting and E under the assumption that the optimal noise lies within (M, Σ) performs poorly.
Noise refining network
13.74
Table 10:
Quantitative results when em- ploying a shallow 2-layer MLP for esti- mating Gaussian parameters, proposed by (Kim et al., 2024).
A.10 ROBUSTNESS TO THE NUMBER OF DENOISING STEPS AND SAMPLERS
Since noise refining network is trained with fixed sampler (DDIM (Song al., 2020a)) and denois- ing steps (10), concerns arise regarding its performance when using different samplers or denoising steps. To examine the impact of varying samplers and denoising steps, conduct experiments com-
25

---

Published as a conference paper at ICLR 2026
Guidance
Initial Noise
PickScore
HPPv2
AES
ImageReward
CLIPScore
Gaussian
19.78
0.174
5.073
-1.644
24.95
InitNo
19.80
0.176
5.071
-1.666
25.02
Refined (Ours)
21.14
0.241
5,389
-0,307
30.31
Gaussian
21.67
0.260
5.525
0.368
32.25
InitNo
21.68
0.261
5.524
0,376
32.26
Refined (Ours)
21.83
0.276
5,571
0,533
32.51
Table 9; Comparison with different noise initialization methods under guided (top) and un- guided (bottom) settings.
paring qualitative results across diverse configurations. For comparison, we select DPM++ SDE (Lu et al., 2022), DPM++ 2M (Lu et al., 2022), and EDM (Karras al., 2022), using the prompt "a photo of cat". The results, presented in Fig. 28, show that our refined noise consistently produces reliable outputs regardless of the denoising timestep or sampler. This demonstrates the robustness noise refining network across diverse samplers and denoising step configurations.
26

---

Published as a conference paper at ICLR 2026
DDIM
DPM+ SDE
DPM+ 2M
EDM
(n) 10 Steps
DDIM
DPM SDF
DPM++ 2M
EDM
20 Steps
DDIM
DPM+ SDE
DPM++ 2M
EDM
(0) 50 Steps
Figure 28: Inference results refined noise in various denoising steps and sampler set- tings. (a), (b), and (c) present inference results employing different samplers at denoising steps of 10, 20, and 50, respectively. The consistency observed across these results highlights the robustness of our refined noise to variations in both denoising steps and samplers.
27

---

Published as a conference paper at ICLR 2026
A.11 ANALYSIS OF DDIM INVERSION
Following Tab.2 in the DDIM (Song et al., 2020a), we evaluate reconstruction quality under different numbers of DDIM inversion steps. Using 100 COCO (Lin et al., 2014) prompts, first generate images with CFG (Ho & Salimans, 2022) scale 7.0 and PAG (Ahn et al., 2024) scale 3.0, and then perform inversion and reconstruction using the same number of DDIM steps. Qualitative and quantitative results are reported in Fig. 29 and Tab. 11, respectively.
The results show that simply increasing the number of inversion steps does not necessarily improve reconstruction quality. Instead, performance stabilizes within a moderate range, typically around 50-200 steps. This observation is consistent with prior findings of ReNoise (Garibi et al., 2024), whose Fig. 8 also indicates that more steps of DDIM Inversion do not always lead to better recon- structions.
Due to the inherent approximation nature of DDIM inversion, even within this favorable step range, perfect reconstructions remain challenging, Fine-grant sane-grained details are often lost compared the orig- inal images, highlighting the intrinsic limitations of the DDIM inversion.
DDIM Inversion Step
10
20
50
199
200
500
999
Avg. Error Avg. PSNR
85727.84 84
58266. 11
24523.32
20565.77
25935.9
46407.59
63493.84
12.16
15.82
23.24
24.83
22.93
17.71
14.84
Table 11: Quantitative reconstruction error analysis across different DDIM inversion steps.
Original
10 Step
20 Step
50 Step
100 Step
200 Step
500 Step
1000 Step
Figure 29: Qualitative reconstruction error analysis across different DDIM inversion steps.
28

---

Published as a conference paper at ICLR 2026
A.12 COMPARISONS WITH DIRECT TRAINING WITH
DATA.
An alternative to our approach is to directly train or fine-tune the diffusion model on CFG-generated images, effectively learning to map the initial noise to the guided output xh_ To explore this, we fine-tune the SiT-XL/2 model on 10K images generated using classifier-free guidance (CFG) with guidance scale 4.0 and 20-step Euler sampling the same dataset used in Tab. 2 (SiT- XL/2 row). The results are shown in Tab. 12.
Model
Initial Noise
Sampling Guidance
Training Dataset
FID |
Gaussian
x
Original
18.43
40.00
SIT-XL/2
Gaussian
x
CFG-Generated
12.31
58.59
Gaussian
x (Guidance Distil.)
Original CFG-Generated
11.90
59.14
Refined (Ours)
X
10.42
50.39
Table 12: Comparison with direct mapping from noise to guided images.
Interestingly, observe that this direct mapping yields noticeable quality improvements. How- ever, its performance remains slightly worse than both guidance distillation and our proposed noise refining method in terms of FID. Furthermore, both guidance distillation and fine-tuning on CFG- generated images share the risk of catastrophic forgetting and lack flexibility across domains or sam- pler configurations. In contrast, NoiseRefine preserves the pretrained diffusion model throughout the denoising process. It modifies only the initial noise and thus remains compatible with domain- finetuned backbones, few-step or single-step samplers, and alternative denoising strategies without retraining the base model (see Sec. 4.3, Appendix 10). This moodel-preserving property is key design goal of our method: to distill guidance into the initial condition without altering the diffusion pipeline. If one allows modifying the denoising model itself, existing guidance distillation methods already provide a more direct and stable solution than fine-tuning on generated samples.
Nevertheless, these results raise several interesting research questions. What specific properties of CFG-generated images reduce the need for guidance at inference time? Can such properties be characterized and used identify or filter natural images that are inherently more amenable to guidance-free free generation? Exploring these questions in depth is an interesting direction for future research.
29

---

Published as a conference paper at ICLR 2026
A.13 COMPARISON UNDER THE SAME SETTINGS AS THE SIT PAPER
In the main experiment, we reported SiT-XL/2 results using a CFG scale of 4.0, as higher guidance scales often yield slightly worse FID but produce qualitatively superior samples.
For clearer comparison with the original SiT results, we use a CFG scale of 1.5, expand the eval- uation to the full IK ImageNet classes, and adopt the evaluation protocol of the SiT paper (Ma et al., 2024). Specifically, we follow their setup by using the second-order Heun sampler with 250 NFE (equivalently, 125 denoising steps) for all procedures, including dataset generation, training, and evaluation. Due to limited computational budget, this comparison experiment was trained for epochs, whereas the main experiment used 8 epochs.
We report quantitative results in Tab. 13 and qualitative results in Fig.30. Both evaluations show that NoiseRefine produces refined noise that achieves image quality comparable to that of CFG samples under the SiT evaluation setting.
Model
Initial Noise
Sampling Guidance
FID
IS f
Gaussian Gaussian
x
9.35
126.06
SiT-XL/2
/
2.15
258.09
Refined (Ours)
4.50
173.48
Table 13: Quantitative results of STT-XL/2 with settings of the SiT paper (Ma et al., 2024). The reference values used for comparison were sourced from the original paper.
(ours) Refined noise w/o guidance
(ours)
(ours)
Random noise
Random noise w/ guidance
Random noise w/o guidanee
Random noise guidance
Refined noise guidance
Random noise guidance
Random noise Refined noise wio guidance
guidance
guidance
Figure 30: Qualitative results of SiT-XL/2 with settings of the SiT paper (Ma et al., 2024).
30

---

Published as a conference paper at ICLR 2026
B THEORETICAL BACKGROUND
B. PRELIMINARIES
Denoising Diffusion Probabilistic Models (DDPM). DDPM Ho et al. (2020) defines forward process that derives by adding Gaussian noise to the image - according to the variance sched- ule, and a reverse process that samples 2 from It, both as a Markovian chain. The forward process is defined
(01) N Iti ac-1 21-1: (1 at
(1-a01) )
an))
(0, No a 2o. (1 II)
(8)
with noise rate timestep 1 a where a denotes noise scaling factors up time step The reverse process is defined below.
po(2:_1laz) = N (x.),
(9)
To reparameterize the equation using
Tt = / a + -1- age for N),,1),
(10)
and ea, which is function approximator for predicting e from st, the inference process becomes
o1 ai-1 co (ax)
(1)
Tt-1
Ft
+ 0zz,
(11)
Or
VI
Where 2 IO, and of denotes the variance of Gaussian trainsitions ,he objective of DDPM is defined as
siiple (0) Er, [|le
66 (=x))1
(12)
where the loss between the actual noise added during training and the noise prediction 6g( () for uniformly sampled t {I,...,T] TT.
Denoising Diffusion Implicit Models (DDIM). DDIM Song et al. (2020a) consider the following inference distributions:
T
Le ( (r) = go (2T mr)
Il
dn (2+
20).
(13)
2=
with mean function as below.
go (01-1111 zo) .V (wv(VVV (VaL-1z0+ + v a: 07
Et V V0r
0,,
(14)
04
Distinctively from DDPM, the forward process is Non-Markovian since each æ, could depend on both -z-1 and To. Reparameterizing with eg, we can sample _t-1 from v, through an equation:
11 - vI
at cB (x))
Va--1
+ a:-1 of 5 (xt) (xx) + 0,60
Va
(15)
prodicted zo
a.Tt + beeg (x),
(1)
where 6 AN(O.T) I) and voi-i/ b /1 a:1 av a+
The objective of DDIM is the same as that of DDPM:
(OD = Er, ||e Eo (x:))
(2)
(16)
31

---

Published as a conference paper at ICLR 2026
Denoising and inversion process. We denote the denoising process Denoise(<r). When using the DDIM sampler (Song et al., 2020a), the denoising process is defined as:
Denoise(2r) = D1 (... Dr(e)(Er))).
(17)
where each step D, is given by:
De(x) 0,34 + beeh (xr).
(8)
(18)
The guided denoising process, denoted Denoise Guide (xr, c), follows the same steps Eq. 17, but replaces (x) with guided seores, such as the classifier-fre guided score Eg Salimans, 2022), the perturbed-attention guided score <x)((r)) (Ahn et al., 2024), or a combination of both (c (EFGG.PAG((z)) (zz)). These guided scores are defined in Eqs. 31 and 32.
CRG (x; c) (Ho &
(ar)
While we utilize the DDIM scheduler in this work, any other diffusion scheduler (Ho et al., 2020; Song et al., 2020a; Karras et al., 2022) can be used by appropriately modifying a and 6;-
For the inversion process Inversion (In, c). we follow the method in (Garibi et al., 2024) obtain the initial noise which can be denoised back to the given image zo without employing any guidance methods (Ho& & Salimans, 2022; Ahn al., 2024) during inversion.
B.2 DERIVATIONS
Proposition 1. Let æt be initial noise, and suppose that zo is the image obtained through de- noising. Assuming Lipschitz continuity with distance metric d, for every ær, there exists a constant 0 such that the following holds:
d(x,, 27 Guidel) < kd((z (o, 20 Guide
proofs. The Lipschitz condition is expressed as follows: (0) (v)) s L,d(x,
d(cy
(a), ())))()) (x),,
(19)
where Le is constant dependent on t, x and y are arbitrary inputs to (t) DDIM step in terms of æt be expressed follows:
Ft
Ct-1
Ft.
V
Ote (1
oc))
c
(L)
(xI)-
(20)
Ott
Eq. 20 can be expressed in terms of x; get the following equation,
Guidet which is denoised from 27
Guidet
With those equations,
Guidet
0t-1
Guidot) + --00--
at at-1 ac)
(e (00 (xxe c (t) Guidet)
Tt-1 2t-1
(x:
It
Ctt
Oz
at-1
Guidet
(1) 7e(E6 (xx) - (1² Guidet )),
(x's
It
Ct
where "t invariance, the equation can be expressed as follows with Eq. 19:
(V as)/ar VI-ai-1
If the distance metric d have translation
Goidet)
at
i L (((r)))))))))))))) Cuiidtt)
(21)
V 04-1
Recursively organizing Eq. 21 for = T.T 1,.... 1, it can be expressed as follows:
T (1 tk.))
Guidet) S d(rr, BT
OT d( 20 Guidei).
(22)
au
Since or close to O, using d(xp To Guidel) is sufficient to directly learn 2T small enough.
Guidet if d( 20
Guidet is
32

---

Published as a conference paper at ICLR 2026
By approximating the gradients through Multistep Score Distillation (MSD) using detached gradients at each step, we approximate the full-gradient objective with a mild assumption. In conclusion, the two gradients can be approximated follows:
Proposition 2.
V De (( 0) kV &V&Ds go (x)):
(23)
where l E (0, 1) is constant.
Warmup. We begin by recalling the typical denoising process in DDIM sampling:
D1 D2 .... Dr(rr),
(24)
where Dx(x)) arr bee (x)) denotes single denoising step, and a, by are DDIM-derived coefficients (Song et al., 2020a).
(4)
The final generated image To. obtained by applying the full denoising trajectory the refined noise, is:
Ho = Di o.... (r(a) where Do (XT) is the refined noise output from the refining network Dø-
(25)
From Eq. 15, the denoised image from the full DDIM denoising process Ty is:
To
V oT
000 9o (TT)
Σ
0
(t) 7 (Iz).
(26)
0x-1
This expression is derived by recursively applying the DDIM update rule (Eq. 20) from IT to zn. unfolding the full denoising trajectory step by step. The result is a closed-form expression for the cu- mulative DDIM trajectory, expressing &0 as a function of the refined noise (IT) and intermediate model predictions.
We define the denoising loss as follows:
Demoise (go (xT); = d (To, 20
Guide)
(27)
where -(::-) denotes the L,2 distance.
proofs. Since the only difference between the two losses is the stop gradient in the diffusion model and all other components are identical, it suffices, by the chain rule, to show that the gradient of Fi Fi(F((... (g(((((T)) with respect to ó is proportional to the gradient of (enoise(gs(zT)) with respect to ø. The derivation proceeds as follows:
T
Denoise((ge (xT) V
Og go (o( oT
(1) meb
(xx))
ar-1
(28)
T
ap
deg (1)
(xx) Ox,
(ge(xT)
OT
I
Te
04-1 Oz.
(go()r)
OΦ
(1) As detailed in B.3, the term de (zx)) can be approximated being proportional to the iden- tity matrix. Additionally, the term (rg)) can be expressed in terms of dc," (xxx)/xxx. Then, each component of δel" (ex)/Ox; can be approximated by the identity matrix. Consequently, () (fxx)/ara (dzx/ 00gg (xT)) becomes proportional to the identity matrix. Denoting the propor- tionality constant as 1)- =
(x.)
Or. Sga(er)
Eq. 28 is simplified as follows:
T
Okn
ag (xT)
Eq. 28
YeMl
ds
at-1
on
(gg(ET) aó
V
VoT
Yo7
V oT
(29)
[4-1
t=]
OT
(r(g)(aT))). (.F(((((..PP(Gaa(())))
V (oF((F2(...
/a
33

---

Published as a conference paper at ICLR 2026
B.3 DIFFUSION MODEL JACOBIAN APPROXIMATION
Why is the approximation possible? In this subsection, we present experimental results demon- strating that the Jacobian of the diffusion model ei with respect to the input dt can be approximated as proportional to the identity matrix. Fig. 31 illustrates the Jacobian Oe 0e /0x We observe that the Jacobian behaves like the identity matrix regardless of timestep, except when t is significantly small. To quantify this observation, we plot the distributions of the Jacobian's 's diagonal and off-diagonal elements across timesteps in Fig. 32 (log scale). The off-diagonal elements are consistently much smaller and concentrated near zero, while the diagonal elements remain significantly larger, con- firming the strong diagonal dominance XL/2 Peebles & Xie (2023), where observe similar pattern: the off-diagonal values stay close to zero across timesteps. whereas the diagonal values remain substantially larger. This demonstrates that the identity matrix--lke Jacobian structure holds not only for Stable Diffusion 2.1 but also for transformer- -basede models such as DiT.
the Jacobian. Fig. 33 shows the same analysis for DiT-
In such cases, the deviation does not affect our primary analysis. According to the results of Propo- sition 1, the timestep-dependen constant you "tt multiplied to each Jacobian term is expressed as follows:
I az a'.
at-1
(30)
0rp-1
at-1
This value can numerically determined based on the scheduling, and the case of DDIM (Song et al., 2020a), it is presented in Fig. 34. The graph shows that the constant decreases toward zero as approaches
Fl
101
201
t301
401
501
601
7011
801
t 901
Figure 31: Visualization of Jacobian of denoising network. Starting from T 1000, we performed denoising over 10 steps and plotted the Jacobian heatmap at each timestep. We extracted 500 500 section from the full Jacobian matrix for visualization. Each plot demonstrates that the Jacobian close to the identity matrix.
34

---

Published as a conference paper at ICLR 2026
Jacobian Value Distribution (Log Scale)
Diagoral Off oitgoiagonal
Absolute Value (log scale)
10
10-4
10
Timestop
Figure 32: Distribution of Jacobian elements in Stable Diffusion 2.1 (log scale). Scatter plot of diagonal and off-diagonal Jacobian magnitudes across timesteps. The off-diagonal elements remain close to zero, while the diagonal elements are significantly larger, demonstrating strong diagonal dominance of the Jacobian.
Jacobian Value Distribution (Log
Diagoral
10a
Off-diagonal
Absolute Value (log scale)
10-2
900
800
700
600
500
400
300
250
200
100
Timestap
Figure 33: Distribution of Jacobian elements in DIT/XL-2 (log scale). Violin plot of diagonal and off-diagonal Jacobian values across timesteps for the DiT/XL-2 Peebles Xie (2023). Despite its transformer structure and larger receptive field, the off-diagonal values remain close zero, showing an identity-like Jacobian similar to Stable Diffusion.
1.6
1.4
1.2
Constant Value
1.0
0.8
0.6
0.4
0.2
0.0
200
400
600
800
1000
Timesteps
Figure 34: Visualization of constant values over timesteps. Visualization of the time-dependent constant value vou-r corresponding to Eq. 30 across different timesteps. The results numerically demonstrate that for small timesteps, where the Jacobian deviates from the identity matrix. the multiplied constant values are sufficiently close zero.
_4__
35

---

Published as a conference paper at ICLR 2026
Why does the approximation enhance performance? The above analysis explains why the Ja- cobian can be approximated by the identity matrix, but does not address why this approximation empirically improves optimization, yielding faster convergence and higher-quality results. Although the Jacobian "close enough" to identity, it is not perfectly identity, especially at small timesteps. When full gradients are backpropagated through multiple denoising steps, these small off off-diagonal components accumulate across steps and induce optimization instability, similar to exploding and vanishing gradients recurrent networks.
To illustrate this effect, we perform a simple toy experiment in which we directly optimize an initial Gaussian noise zт so that its denoised output &g matches given target image. Fig. 35 shows the MSE loss | - Zuageell and the gradient norm during optimization. Using the full-step gradient leads unstable behavior: the loss fails to converge, and the gradient norm becomes large and highly erratic. We also visualize the gradient norm | at each denoising step (Fig. 36 top), and observe that for some iterations the gradients become progressively larger as approaches 1, clear evidence of gradient explosion.
By skipping the Jacobian 0e// at each step, MSD avoids this long-horizon accumulation and yields a far more stable optimization process. As shown in Fig. 36 (bottom), the gradient norms remain well-behaved and stable throughout optimization. This behavior aligns with prior findings in score distillation sampling, and further demonstrates that gradient skipping acts effective regularizer that prevents instability arising from multi-step backpropagation.
MSE Loss Comparison
Gradient Norm Compartson
Stedinn
Cracknt
14"
50
100
12
13744
neration
Figure 35: Optimization instability of full-step gradients. We optimize the initial noise so that the denoised output dg matches a target image and plot the MSE loss (left) and gradient norm (right) over iterations. The full-step gradient exhibits unstable dynamics, with the loss failing to converge and the gradient norm becoming large and erratic. In contrast, the MSD approximation maintains stable gradients and converges reliably, demonstrating that skipping the Jacobian 0c//0m effectively prevents long-horizon instability.
36

---

Published as a conference paper at ICLR 2026
Full step gradient norm
961
861
761
661
Timesteps
861
481
381
201
Iteraiio
851
Stop gradient norm
Figure 36: Gradient explosion across denoising steps when using full-step gradients. We visual- ize the gradient norm |0C/0x,1| 2 at each denoising step during optimization. With full-step back- propagation, the gradients become progressively larger t approaches 1, and in several iterations the norm reaches values close to I, indicating clear gradient explosion. In contrast, when apply- ing MSD (gradient skipping), the gradients remain small and stable. This confirms that multi-step Jacobian accumulation is the primary source of instability, and that skipping the Jacobian de /Oz effectively prevents this issue.
37

---

Published as a conference paper at ICLR 2026
C MORE ABLATION STUDIES
C.I NOISE REFINING NETWORK
To effectively leverage pretrained knowledge, we attach LoRA layers to the original model when training noise refining network. To evaluate the effectiveness of LoRA (Hu et al., 2021), we conduct an ablation by training the refining network using the same original Stable Diffusion 2.1 UNet architecture, but from scratch. We use the filtered MS COCO dataset from both datasets and train models for 25K steps on two RTX 3090 GPUs, keeping all other experimental configurations identical. As shown in Tab. 14 and Fig. 37, the LoRA-based approach achieves faster convergence and significantly lower FID at the same iteration, demonstrating its efficiency training. These results indicate that leveraging pretrained knowledge leads superior performance compared to training from scratch.
Model
FID
From scratch 37.87
Pretrained (Ours)
13.74
Table 14: Quantitative comparison with noise refining network using pretrained UNet + LoRA and UNet trained from scratch.
Figure 37: Qualitative comparison with noise refin- ing network (top) and UNet trained from scratch (bottom).
Parameter
FID (Heusel al., 2017) 1.
(Salimans et al., 2016) 1
13.74
30.80
steps
10
13.36
32.81
Table 15: Ablation study on the number of denoising steps.
C.2 NUMBER OF DENOISING STEPS
We analyze the impact of denoising steps by comparing N = and N 10. reporting FID (Heusel et al., 2017) and IS (Salimans et al., 2016) in Tab. 15. The results show improved performance with more steps, but high step counts (N 10) incur prohibitive backpropagation costs, highlighting the need for MSD to mitigate computational overhead.
38

---

Published as a conference paper at ICLR 2026
D IMPLEMENTATION AND EXPERIMENTAL DETAILS
D.1 IMPLEMENTATION DETAILS
Initial noise z-N((O,))
Generated image w/ Guidance x
Text prompt
Guidance T2I Pipeline
Text prompt e (P-)
Noise Refining
LORA
residual connaction
NolseRefiner de
Text prompt (Pa)
Stop
x N
gradient
(xy)
0
8
Backpropagation
Suidance-Fie noise 8y
Guidance-Free T2I Pipeline
Generated image Guidance X
Figure 38: Training framework with annotations. We provide an annotated illustration of the training framework to clarify the notation in the following discussion.
More details of our framework. Most experiments are conducted with text-to-image diffusion models (Rombach et al., 2022; Podell al., 2023), so provide implementation details for these models here. Although our framework NoiseRefine can be generalized from pixel-level to latent- level diffusion models, in our experiments use MSE loss latent space for d(z (aide 20))
We provide our training framework in Fig. 38. It consists of three parts: Guidance T2I Pipeline takes Gaussian noise æт -O I) and condition (text prompt) as inputs and generates an image Guide with guidance methods (Ho & Salimans, 2022; Ahn et al., 2024; Hong et al., 2023; Sadat 2 al., 2024; Hong, 2024: Karras et al., 2024). The noise refining network Jo refines Gaussian noise Guidance-Frit Free T2I Pipeline takes refined noise रा = go (IT) and condition (text prompt) and generates an image åo without guidance. For Guidance T2I Pipeline, with the denoising network 6p, we can use the guided score < ( c) for CFG (Ho & Salimans, 2022) or PAG c) for PAG (Ahn et al., 2024) in denoising process as below:
CFG (Iz, c) eo co c) w(Ea(Es, <o(x)))
(31)
&
PAG (xx) co(x)) + s(co(x:) (o(xx))
(32)
where w and denote the guidance scale of CFG (Ho & Salimans, 2022) and PAG (Ahn et al., 2024), and is for the condition. Note that the perturbed score éo from perturbing the forward process of the denoising network (Ahn et al., 2024). With the denoising step N' = 20, we can get the guided image 26 Guide Our noise refining network refines Gaussian noise 2T with Jo timestep t = T, which is from the reverse step of DDIM (Song et al., 2020a) in Eq. 15. The output of noise refining network dø denoted as दτ = (FT) and becomes the input of Guidance-Free T2I Pipeline. In this pipeline, ãe is denoised into ao without guidance using N denoising steps.
Architecture details. For noise refining network 9ø. we use Stable Diffusion 2.1 (Rombach et al., 2022) with LoRA (Hu et al., 2021) rank of 128, applied to all attention, convolutional, and feed-
39

---

Published as a conference paper at ICLR 2026
forward layers. We use DDIM (Song et al., 2020a) scheduler with the same settings the pre- trained model. For noise refinement, we use an input timestep T 999, and the default denoising step N is set to 10. In Stable Diffusion XL (Lin et al., 2024), the same configs of Stable Diffusion 2.1 except LoRA rank which set to 256.
D.2 EXPERIMENTAL DETAILS
D.2.1 TRAINING SETUP
Note that our model requires only text prompts for training, eliminating the need for real images, as leverage self-generated images from the model we aim to train using guidance methods.
For Stable Diffusion 2.1 (Rombach et al., 2022), we train our model on 20K MS COCO prompts, 30K Pick-a-Pic prompts, using CFG scale 7.0 and PAG scale 3.0 for all generated images.
For Stable Diffusion XL (Podell et al., 2023), we train our model on 55K MS COCO prompts, 36K Pick-a-Pic prompts, and 90K LAION prompts, using the same CFG and PAG scales as for Stable Diffusion 2.1.
For SiT-XL/2 (Ma al., 2024), train model 100 classes of ImageNet (Krizhevsky al., 2012) (class to class 100) using CFG scale 4.0. Total dataset consists of 100K images, 1K images for each class. For dataset generation, employ Fuler sampler with 20 denoising steps.
For SD2.1 and SDXL, we generated images for all datasets with guidance and retained only the top-N samples ranked by AES (Schuhmann, 2022) scores, where N denotes the reported dataset size. For SiT-XL/2, no filtering was applied.
For the ablation study the number of denoising steps, primarily use SD2.
D.2.2 EVALUATION SETUP
The datasets used are described in Sec. 4.1. For guided sampling, use the same guidance scale as in the training of the noise refining network across all models. For guidance-distilled sampling with distilled denoising network (Meng al., 2023), since no official implementation is available, we follow Eq. 3 in their paper for reimplementation. The same dataset and guidance scale are used for training the distilled model.
For all experiments using SiT-X1//2 except A. 13, we generate 50K samples using random initial noise and the Euler sampler with 20 denoising steps.
40

---

Published as a conference paper at ICLR 2026
E ADDITIONAL RESULTS
E ADDITIONAL QUALITATIVE RESULTS
We present our additional qualitative results of SD2.1 on Fig. 41, 42, 43, 44 and results of SDXL on Fig. 45, Results show that the performance of using refined noise by noise refining network is comparable that of using guidance on random Gaussian noise. All the results are selected from images used in Tab. 2.
E.2 USER STUDY
Gaussian noise vs refined noise. conducted user study to evaluate prompt adherence and im- age quality by comparing images generated from random Gaussian noise and our refined noise. The images are generated using randomly sampled MS COCO validation prompts, as shown in Tab. 2. The results are presented in Tab. 16. The study demonstrates that method outperformed the baseline in all human evaluation criteria. A total of 26 participants anonymously evaluated 20 pairs of images, each pair consisting of an image generated using initial Gaussian noise and our refined noise from noise refining network. The percentage was calculated by dividing the total number of selections for each option by the total number of responses, following the same methodology in Tab. 16.
Participants were provided with the following instructions for each pair images:
1. Which image has better overall quality? (left/right)
2. Which image more faithfully reflects the given prompt? (left/right)
Metric
Gaussian Noise
Refined Noise (Ours)
Image Quality Prompt Adherence
3.08%
96.92 %
6. 4.73%
93.27 %
Table 16: User study on the image quality and prompt adherence of generated images.
Guided sampling vs refined noise. Tab. 17 shows the results of user study, confirming noise re- fining network's 's comparable to results starting from Gaussian initial noise without guidance. 45 participants compared image pairs generated with guidance and our method (refined noise with- out guidance), using generated images for evaluation in Tab. 2, and evaluated visual appealing and prompt alignment. The instructions for the survey are the same as the above.
Metric
Gaussian Noise + Guided Sampling Refined Noise (Ours)
Image Quality
46.04%
53.96%
Prompt Adherence
48.24%
51.76%
Table 17: User study on the image quality and prompt adherence of generated images.
41

---

Published as a conference paper at ICLR 2026
E.3 GENERALIZATION ON OTHER DOMAINS
Fig. 39 presents additional qualitative results across different domains, including anime and clay. The prompts used for generation provided in Tab. 18.
Cocoian Doso
Aefbad Noms
Goassion poire
Refioed Noioo
Cocation soion
Reflaed Noiso
Grassion noles
Reffand Neiio
Ganavan Iteliadd bocee
Gra Relied Keies
Figure 39: Additional qualtitative results of generalization on other domains.
(masterpiece, best quality, ultra-detailed, best shadow), (detailed background,dark fantasy), (beauti- ful detailed face), high contrast, (best illumination, an extremely delicate and beautiful), (cinematic light), colorful, hyper detail, dramatic light, intricate details, (1 girl, solo,black hair, sharp face,low teintails,red eyes)
(masterpiece, best quality), Igirl, long hair, red hair, solo, dress, red eyes, looking viewer, long sleeves, standing, building, white dress, gloves, hair ornament, black jacket, smile, floating hair, dutch angle, closed mouth, looking away, outdoors
imasterpiece.bes, quality), 1girl, solo, black skirt, blue eyes, electric guitar, guitar, headphones, holding, holding plectrum, instrument, long hair, music, one side up, pink hair, playing guiter, pleated skirt, black shirt, indoors
(masterpiece. best quality, ultra detailed:1.3), perfect composition, anime, illustration 4k, (extremely detailed, hyper detailed), raw, hdr, 8k textures, extreme detail, hight detailed skin texture, high sharpness, Igirl, (detailed eyes: petite, on the street, in public, night street, night lights (masterpiece, best quality, ultra detailed:1.33 1.33. beautiful, anime-style female character with long flowing hair, wearing flowing summer dress, standing in a field of flowers at sunset, soft pastel colors, detailed facial features
Igirl, aqua eyes, baseball cap, blonde hair, closed mouth, earrings, green background, hat, hoop earrings, jewelry, looking at viewer, shirt, short hair, simple background, solo, upper body, yellow shirt, (waifu, anime, exceptional, best aesthetic, new, newest, best quality, masterpiece, extremely detailed:1.2 1.2)
clayitization, portrait of black cat with piercing green eyes, Ultra-detailed, 3d, octane render, intricate details
clayitization, a photo of a cheese burger, ultra detailed, 3d, octane render, intricate details clayitization, colorful tropical bird perched on branch, ultra detailed, 3d render, smooth clay textures, vibrant palette, 3d, octane render, soft lighting, realistic textures
clayitization, stylish pair of sneakers, detailed textures, 3d, vibrant colors, realistic clay appearance, octane render
clayitization, classic red Vespa scooter, highly detailed, glossy clay finish, 3d model, studio lighting, octane render
clayitization, a photo of the Biffel Tower, ultra detailed, intricate architectural details, 3d, octane render
Table 18: Example prompts used for domain generalization experiment (Fig 39).
42

---

Published as a conference paper at ICLR 2026
(ours) Refined noise w/o guidance
(ours)
Random noise w/o guidance
Random noise w/ guidance
Random noise w/o guidance
Random noise w/ guidance
Refined noise w/ guidance
goldfish
cock
green lizard
junco, snowbird
sulphur-crested cockatoo
goldfinch
great grey owl
mud turtle
partridge
lorikeet
toucan
drake
tiger shark
ostrich
Figure 40: Additional qualitative results on SIT-XL/2.
43

---

Published as a conference paper at ICLR 2026
(ours) Refined noise w/o guidance
(ours) Refined noise w/o guidance
Random noise w/o guidance
Random noise w/ guidance
Random noise w/o guidance
Random noise w/ guidance
"Two cars on the street." "
"a bedroom with a night stand near the bed"
"A large motor vehicle carrying passengers by road, rypically one serving the public on fexed mate and for fare."
woman sunglasses and hat standing by plant.
"A brightly painted tomple with ornate structures and dramatic lighting inspired by Mayan and Islamic archirecture."
"A bathroom is shown with toilet, sink and mirror:"
"opal gun"
"A close-up of grains and pastries on a table.
"The stainless steel refrigerator is being moved into the newly constructed home.""
"A small vase with a few flowers is in the snow."
"An image of Malta, covered in Palm mees, highly detailed and realistic"
"Poto of guy having with chubby young redhead, POV"
"Some luggage sets the living room ready to go."
"beautiful summer landscape, an udtrofine detailed painting, intricate pasta waves, made of noodles, poper quilling, inspired by van Gogh
Figure 41: Additional qualitative results on SD2.1.
44

---

Published as a conference paper at ICLR 2026
(ours) Refined noise w/o guidance
(ours)
Random noise w/o guidance
Random noise w/ guidance
Random noise w/o guidance
Random noise w/ guidance
Refined noise w/o guidance
"Traditional library with floor-to-ceiling bookcases"
large hawk flying through a purple and orange sky.' "
"Navy blue wall livingroom with dusty pink curtains"
"a boy a blue shirt eating some food"
"fur away abondoned pickup truck, overgrown"
"A blue plate topped with rice and stew."
"A painting depicting foothpath Indian summer with epic evening sky sunset and low shunder clouds,"
"A bench sitting on top of a sandy beach next to the ocean."
"A watercolor portroit Luke Ruedo Studios and Dovid Downton.
"a man that is on a surfboard on some water"
"A painting depleting snowy winter scene featuring small house hill, and dreamy cloudy sky.""
"An orange colored sandwich."
"A lighted birthday cake with chunks of walnuts.
"A yellow and black bus cruising through the rainforest.
Figure 42: Additional qualitative results on SD2.1.
45

---

Published as a conference paper at ICLR 2026
(ours) Refined noise w/o guidance
(ours) Refined noise w/o guidance
Random noise w/o guidance
Random noise w/ guidance
Random noise w/o guidance
Random noise w/ guidance
"logo of a blue elephant, flat modern vector icon"
"A beautifil ancient Chinese chivalrous woman'
"Alight blue hatred anime girl ocean themed anime is opon antenna twintails"
"A sail boat entering majestic fjord landscape winter"
"<pixel art gray French bulldog"
"An oil painting of a bowl fruit"
"A silhouette of a dog looking stars"
"portrait sir borzoi dog wearing royal uniform and crown"
"A snowy Chicago street during Christmas art by Ludwig Fahrenkrog
"a low light photo of city night"
"Gothic cathedral a stormy night"
"Face shaped out of old rusty technology" "
"Photo of a black panther"
"An stylized entrance rocky cave"
Figure 43: Additional qualitative results on SD2.1.
46

---

Published as a conference paper at ICLR 2026
(ours) Refined noise w/o guidance
(ours) Refined noise w/o guidance
Random noise w/o guidance
Random noise w/ guidance
Random noise w/o guidance
Random noise w/ guidance
"A black display with lights in the background.
"A feast of meat, potatos, and veggies plate"
"A red fire hydrant the side of street."
"photograph of man when he was much younger"
"A subway train the boarding platform."
"A white sandy beach has chair and straw umbrella."
"A woman holding plate with slice of cake." "
"A cute kitten hiding in something on a chair."
large boat sitting the middle of a body of water."
"A grey motorcycle dirt road next building.
bird perched on branch by some leaves."
"A image of bathroom scenic from window.
"A building a gold clock inside of "
"A white Volkswagen beetle parked lush grass field."
Figure 44: Additional qualitative results on SD2.1.
47

---

Published as a conference paper at ICLR 2026
(ours) Refined noise w/o guidance
(ours) Refined noise w/o guidance
Random noise w/o guidance
Random noise w/ guidance
Random noise w/o guidance
Random noise w/ guidance
"A brown dog on bed sitting under the covers"
"A large dog sitting in the mud outside"
"A bench in park with trees in the background"
"A man smiling wearing a white shirt and bow
"a mouse sits in front of a keyboard and monitor"
"Open faced sandwich being somcone table."
"A bottle half glass of dark liquid" "
"A tennis player in yellow shirt with striped shorts"
"A polar bear balances blue ball." "M
"A woman in a kitchen preparing sandwich with tomatoes.
"A dog walking down the middle of steee next to store lined sidewalk:
"A small sandwich sitting on white china plate.
"An Olympic skier who is racing down mountain"
"A gray cat sitting top of chair near table
Figure 45: Additional qualitative results on SDXL.
48

---

Published as a conference paper at ICLR 2026
F LLM USAGE DISCLOSURE
During the preparation of this paper, the authors made limited use of large language models (LLMs) for polishing the writing, grammar refinement and LaTeX formatting. LLMs were not used for generating research ideas, designing or conducting experiments, analyzing results, or formulating conclusions. All scientific content and contributions are entirely the responsibility of the authors, and any LLM-assisted text was carefully reviewed and revised before inclusion.
49


Reviews:
[{"review_id": "OmKzAqm3AQ", "reviewer": "Reviewer_86wZ", "summary": "The paper aims to address the reliance of Diffusion models on sampling guidance (e.g., CFG), which effectively doubles inference cost. It proposes to learn a (external) noise refining network $g_\\phi$ to modify the initial Gaussian noise $x_T$, for the correspondence with the effects of CFG. It also uses methods including training in image space for better performance and using Multistep Score Distillation (MSD) for efficient training, with rounded analysis and sound experiments.\n\nTotally, this work is simple in the method but interesting in the perspective. The core idea (noise refinement for guidance) is actually like a special (but clever) variant of guidance distillation approaches, and some comparisons are unfair or not comprehensive. But it does give an insightful perspective from the initial noise, with convincing (at least for me) motivation, not bad results, and the potential for wider application in various scenarios. \n\nSome questions need to be clarified and solved, but overall, this paper is well-written, easy to follow, rich in content, and deserves a positive rating.", "questions": "1. The Study/Evaluation of Controllability: The NoiseRefine is used to replace the role of CFG, but the study of the controllability seems not to be sufficient. Can the details of the training-based approach of controllability be further illustrated? Curious about how the guidance scale $w$ is incorporated as an additional input for training, and how to guarantee the range of $w$ (e.g., show the num of samples of different $w$). Also, have you done contrast experiments of different fixed $w$ of CFG during the training of NoiseRefine, and quantitatively compared the FID/CLIP score?\n2. Cost and Performance Comparison with Guidance Distillation: Given that training NoiseRefine also requires running guided sampling, how does its total training budget (e.g., in GPU-hours) compare directly to training a standard guidance distillation model [1] to achieve similar performance?\n3. The Architectural Choice for $g_\\phi$: Is implementing $g_\\phi$ as a LoRA attached to $\\epsilon_\\theta$ a limitation? Have you experimented with a completely separate, smaller network for $g_\\phi$? How would this impact performance and the leveraging of \"pretrained knowledge\"?\n4. The Jacobian Approximation in MSD: The proof of Proposition 2 critically depends on the assumption that $\\partial\\epsilon_\\theta^{(t)} / \\partial x_t \\propto I$. Does this assumption still hold for Transformer architectures (e.g., DiT, SiT) where attention mechanisms might induce more complex, long-range Jacobian structures? Could you provide a quantitative error bound or an ablation study (e.g., detaching gradients for only a subset of steps) to more rigorously validate this approximation? This theoretical/experimental verification may not be so easy to do, so it is OK as the current results are great, just a comment. But if you can provide an intuitive understanding, or if the validity of this Jacobian Approximation can be verified, it would be really helpful to the research community.\n\n\n[1] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14297–14306, 2023.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Totally, it is well-written and easy to follow.\n2. The overall motivation is clear. Optimizing the guidance to accelerate the inference of Diffusion models is useful.\n3. The perspective from the initial noise is quite interesting. The idea of distilling guidance information into the initial noise, rather than into the denoising network itself, is also insightful and provides a fresh perspective on tackling the guidance overhead problem (with potential and compatibility for wider application in various scenarios).\n4. Sound experiments and rounded analysis make the results convincing. The method significantly outperforms the unguided baseline, with an acceptable gap to CFG.", "weaknesses": "1. A Special Form of Distillation: While the paper claims their method is \"orthogonal\" to guidance distillation, it is more accurately a clever variant rather than a new paradigm. The framework fits the knowledge distillation paradigm, the difference from guidance distillation is the locus of learning (the noise refiner $g_\\phi$ v.s. the denoising network). This positioning overstates the fundamental novelty, more discussion/comparison/clarification should be made.\n2. Loss of Critical Controllability: The method encodes the effect of a *fixed* guidance scale (e.g., CFG scale 7.0) into the network, thereby losing the native, zero-cost flexibility of the CFG scale $w$ at inference time. The workarounds proposed (scaling the residual or conditioning the training) are either heuristics with unverified behavior or add complexity to the training process, undermining the method's simplicity. Especially, the presented results are not intuitive to understand (Fig. 13; Fig. 23): Changing $w$ seems to have no effect on the \"richness\" of the image at all, but merely affects the degree of distortion, failing to reach the original function of $w$ in the CFG.\n3. The Theoretical Soundness of MSD: The entire framework's viability hinges on Multistep Score Distillation (MSD) for gradient approximation. However, its theoretical justification (Proposition 2) relies on a strong assumption ($\\partial\\epsilon_\\theta^{(t)} / \\partial x_t \\propto I$). This assumption is empirically/visually supported in the appendix but lacks rigorous quantitative evidence.\n4. Additional Complexity and Overhead: The training process requires running the expensive guided sampling to generate the teacher signal $x_0^{Guide}$ for every training sample. As for the inference overhead, unlike standard LoRA (which can be merged into the base model's weights to eliminate inference overhead), the noise refining network $g_\\phi$ implemented with LoRA cannot be merged, meaning an extra model forward pass is also always required at inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cYZvcAh3Qw", "reviewer": "Reviewer_ScWm", "summary": "This paper propose a novel and orthogonal approach that \"distills\" the effect of guidance into the initial noise vector rather than into the diffusion model itself. The core idea is to train a lightweight \"noise refining network\" (implemented as a parameter-efficient LoRA on the diffusion model's UNet) that maps a standard Gaussian noise vector ($x_T$) to a \"refined\" noise vector ($\\hat{x}_T$). Specifically, this paper propose Multistep Score Distillation (MSD), a technique that stops the gradient during backpropagation through the denoising network at each step.", "questions": "Can we directly learn the mapping from the initial XT to X0_guide using a bridge model initialized from the original diffusion network? For example, DDBM or I2ISB. What are the advantages of MSD compared to this approach?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This idea is novel and plug-and-play.\n2. It's orthogonal to other acceleration methods.", "weaknesses": "1. I'd like to know if optimizing Eq4 is difficult and costly, since it requires go through the network many times.\n2. The result after refinement was still not good enough, significantly worse than the result generated by CFG.\n3. For me, it's difficult to understand why noise-to-noise mapping has generalization properties. The paper lacks discussion and analysis on this aspect. For example, are the generated results of training samples has better quality than validation samples? For refined noise, can it be combined with CFG to obtain better results? If so, how does it compare to the results obtained using CFG with initial noise?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rVHBZLIIYT", "reviewer": "Reviewer_a4W6", "summary": "This paper proposes _NoiseRefine_, a guidance distillation method that learns to refine the initial Gaussian noise into one that can generate high-quality images without classifier-free guidance. When performing inversion _without_ guidance on an image generated _with_ guidance, the resulting \"inversion noise\" is generally non-Gaussian, yet it can still recreate the image without guidance. Based on this observation, the authors propose training a lightweight refinement network—a LoRA of the pretrained diffusion model—to learn the mapping between the original Gaussian noise and the inversion noise. This effectively distills guidance into the noise.\n\nTo achieve this, the method introduces three main contributions:\n- **Image-space supervision**: Training is supervised in image space, which is shown to be more robust to inversion errors.\n- **Residual connection**: Instead of learning the full mapping, the refinement network is parameterized to learn the residual.\n- **Backpropagation**: Similar to score distillation sampling (SDS), the gradient is not propagated through the network during training. This has been shown to lead to faster convergence and higher-quality results.\n\nTo validate the approach, the paper ablates the method on several conditional image diffusion models and thoroughly analyzes what the refinement network learns, which seems to be primarily adding low-frequency structures to the noise. Additionally, the authors compare their method with other guidance distillation approaches and argue that distilling guidance into the noise is more efficient and generalizes better to other fine-tuned models.", "questions": "Again, not many questions, as the exhaustive appendix addressed most of my interrogations already.\n\n- The user study is purely done against no guidance in the main paper, and against with guidance in the appendix. Have the authors tried to compare the method with other guidance distillation methods?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper tackles an important topic in the diffusion community, which is related to guided generation. The main strength of this paper is in the idea it proposes. To the best of my knowledge, this is the first work to look into guidance distillation into the noise input, which seems like a simple yet brilliant idea. \n\nThe execution of the method is also in itself very well done: the authors validate each contribution that make the method work using small experiments. The manuscript is easy to read and the method clearly explained. \n\nFurthermore, the appendix is very exhaustive and provides many additional details and experiments that makes the submission very strong.", "weaknesses": "No major weaknesses, the appendix addresses many of my concerns already.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "klkwlKfnV4", "reviewer": "Reviewer_AzoV", "summary": "This paper introduces NoiseRefine, a new method to eliminate the computational overhead of classifier-free guidance (CFG) during diffusion model inference. Instead of applying guidance at each sampling step, NoiseRefine employs a refining network to perform a one-time transformation on the initial noise vector. This network, created by fine-tuning a pre-trained diffusion model with LoRA, maps random noise to a \"guidance noise\". By using this refined noise as the starting point, an off-the-shelf diffusion model can generate high-quality, guided images without the need for iterative guidance, effectively halving the inference cost.", "questions": "All my concerns and questions are listed above.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a new approach to guidance distillation. The core idea of refining the initial noise vector instead of performing iterative guidance is interesting. The work is well-motivated by the clear and practical goal of reducing the inference cost of classifier-free guidance, a significant bottleneck in modern diffusion models.", "weaknesses": "Though the idea of adopting noise refinement in distilling guidance is interesting, the manuscript in its current form has several major issues related to contradictory motivation, methodological clarity, and experimental evaluation that weaken its conclusions. My primary concerns are detailed below.\n\n### Contradictory Motivation and Methodological Justification.  \nThe introduction motivates the method by showing that \"guidance noise\" (inverted from a high-quality guided image) can be successfully mapped back to a similar high-quality image by the diffusion model (Figure 2). This suggests that such a guidance noise target is stable and learnable. However, in Section 3.2, the authors argue against directly learning this mapping. They claim that inversion errors make this approach infeasible, leading to low-quality reconstructions (Figure 4). This directly contradicts the initial motivation. If the inversion error is as severe as Figure 4 suggests, the premise illustrated in Figure 2, which underpins the entire approach, is undermined. Could the authors please clarify this discrepancy?  \n\n### Ambiguity in the Proposed Loss Function and Training.   \nThe discussion around the loss functions ($L_{denoise}$ v.s. $L_{MSD}$) is confusing and raises questions.  \n**a. Training Cost**: The paper claims that training with the full-gradient loss $L_{denoise}$ is significantly more expensive than with the proposed score-matching approximation $L_{MSD}$. Could the authors elaborate on this? Since the diffusion model's weights are frozen during the refiner's training, the primary cost should be the forward/backward passes of the refining network itself. It's not immediately clear why the diffusion model incurs expensive training cost.  \n**b. Approximation Outperforming the Target**: In Figure 6, the proposed $L_{MSD}$ loss not only converges to a lower value but also produces visually superior results compared to the full-gradient objective. This is counter-intuitive. If $L_{MSD}$ is a theoretical approximation, why does it empirically outperform the objective it is meant to approximate? This suggests either the approximation is not just an approximation but introduces a beneficial regularization, or there is an issue with the implementation or interpretation of the $L_{denoise}$ objective.  \n\n### Experimental Evaluation Weaknesses\n1. Missing cost analysis in the main paper.  \nThe paper is motivated by reducing the high cost of guidance sampling and claims the commonly adopted guidance distillation is computationally expensive, yet it fails to demonstrate the efficiency gain or provide a quantitative comparison of its own training costs against related methods. Please provide a direct comparison of the training cost (e.g., in GPU hours) of NoiseRefine against the key guidance distillation baselines mentioned.  The inference process is not clearly described. Is the noise refined once before the sampling process begins, or at every step? The primary contribution is claimed to be efficiency, so a detailed breakdown of the inference speed-up (e.g., wall-clock time, FLOPs) compared to the standard CFG baseline is essential.  \n2.  Under-reported and puzzling baseline results.  \n**a. CFG baseline**: The CFG performance for SiT-XL/2 in Table 2 is substantially worse than what is reported in the original paper. This weakens the claimed improvements of NoiseRefine.  \n**b. Approximation surpassing full method**: It is highly unexpected that NoiseRefine, which is trained to approximate CFG, would achieve better FID scores than full CFG itself. This result is counter-intuitive and may point to a suboptimal implementation of the CFG baseline or an unfair comparison setup.  \n3. Incomplete baselines and flawed user study.  \n**a. Missing baselines**: For conditional image generation, the comparison is incomplete. Methods like auto-guidance, which also aim to reduce guidance costs with a lightweight model, are highly relevant and should be included as a baseline.\n**b. User study design**: The user study is flawed. It compares images generated without any guidance to images generated with NoiseRefine. A meaningful comparison must be between the standard guidance method (CFG) and the proposed method (NoiseRefine) to evaluate if the approximation preserves perceptual quality.\n4. Questionable application and generalizability.  \n**a. Experiments on SD-Turbo**: The experiments on SD-Turbo are conceptually questionable. NoiseRefine is designed to alleviate the cost of iterative guidance. In a one-step model like SD-Turbo, there is no iterative cost to reduce. Instead, NoiseRefine adds the overhead of an extra network pass, making it inherently less efficient than the native one-step model. Furthermore, could the authors explain the surprising result in Table 4 where 2-step sampling yields a worse FID than 1-step sampling?  \n**b. Generalizability**: The claim of superior generalizability in Figure 8 is supported only by limited qualitative examples. To make this claim convincing, quantitative metrics evaluating performance on out-of-distribution prompts or styles are needed.  \n\nWhile the core idea of refining initial noise is interesting, the manuscript requires a major revision. Addressing the concerns regarding baseline performance, cost analysis, and experimental design is critical to substantiating the contributions of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)