Score Distillation Beyond Acceleration: Generative Modeling from Corrupted Data
Authors: Yasi Zhang
Tianyu Chen
Zhendong Wang
Ying Nian Wu
Mingyuan Zhou
Oscar Leong
Venue: ICLR 2026 Poster (2026)
Keywords: Generative model
diffusion distillation
Primary area: generative models

Abstract: Learning generative models directly from corrupted observations is a long-standing challenge across natural and scientific domains. We introduce **Restoration Score Distillation (RSD)**, a unified framework for learning high-fidelity, one-step generative models using **only** degraded data of the form
$
y = \mathcal{A}(x) + \sigma \varepsilon, x\sim p_X,\ \varepsilon\sim \mathcal{N}(0,I_m),
$
where the mapping $\mathcal{A}$ may be the identity or a non-invertible corruption operator (e.g., blur, masking, subsampling, Fourier acquisition). RSD first pretrains a corruption-aware diffusion teacher on the observed measurements, then *distills* it into an efficient one-step generator whose samples are statistically closer to the clean distribution $p_X$. The framework subsumes identity corruption (denoising task) as a special case of our general formulation.

Empirically, RSD consistently reduces Fr\'echet Inception Distance (FID) relative to corruption-aware diffusion teachers across noisy generation (CIFAR-10, FFHQ, CelebA-HQ, AFHQ-v2), image restoration (Gaussian deblurring, random inpainting, super-resolution, and mixtures with additive noise), and multi-coil MRI—*without access to any clean images*. The distilled generator inherits one-step sampling efficiency, yielding up to $30\times$ speedups over multi-step diffusion while surpassing the teachers after substantially fewer training iterations. These results establish score distillation as a practical tool for generative modeling from corrupted data, *not merely for acceleration*. We provide theoretical support for the use of distillation in enhancing generation quality in the Appendix. The code is available at https://github.com/TianyuCodings/RSD.

Published as a conference paper at ICLR 2026
SCORE DISTILLATION BEYOND ACCELERATION: GEN- ERATIVE MODELING FROM CORRUPTED DATA
Yasi Zhang*'/, Tianyu Chen Zhendong Wang?, Ying Nian Wu',
Mingyuan Zhou 2.3 Oscar Leong "Equal contribution TEqual advising
'University of California, Los Angeles UUniversity of Texas at Austin
Microsoft AI Superintelligence I Work done during an internship in Microsoft
cianyuchen@utesn edu Correspondonce to @eaeminzzanggaeee waa.anguuulaaa.......
ABSTRACT
Learning generative models directly from corrupted observations is long-standing challenge across natural and scientific domains. We introduce Restoration Score Distillation (RSD), unified framework for learning high-fidelity, one-step genera- tive models using only degraded data of the form (0,,, Im), where the mapping may be the identity or a non-invertible corruption operator (c.g., blur, masking, subsampling, Fourier acquisition). RSD first pretrains a corruption-aware diffusion teacher on the observed measurements, then distills it into an efficient one-step generator whose samples are statistically closer to the clean distribution px. The framework subsumes identity corruption (denoising task) special case of our general formulation.
A(x) + oré, x ~ Px, ~
Empirically, RSD consistently reduces Fréchet Inception Distance (FID) rela- tive to corruption-aware diffusion teachers across noisy generation (CIFAR-10 10, FFHQ, CH,, AFHQ-v2), image restoration (Gaussian deblurring, ran- dom inpainting, super-resolution, and mixtures with additive noise), and multi-coil MRI-without access to any clean images. The distilled generator inherits one- step sampling efficiency, yielding up to 30x speedups over multi-step diffusion while surpassing the teachers after substantially fewer training iterations. These results establish score distillation as practical tool for generative modeling from corrupted data, not merely for acceleration. We provide theoretical support for the use of distillation in enhancing generation quality in the Appendix. The code is www available at https: / htt.ttaaw..coo/aaaaaaooooo/////o
INTRODUCTION
Learning from corrupted data is central to many scientific and engineering domains where clean observations are scarce or costly, including astronomy Roddier (1988); Lin et al. (2024), medical imaging Reed al. (202T): Palal al. (202T). and seismology Nolet (2008; Rawlinson et al.) (2014). For instance, fully sampled MRI acquisitions are time-consuming and uncomfortable for patients Knoll et al. (2020); ; Zhontar al. 2008)), motivating methods that recover the structure of the underlying clean distribution from corrupted measurements alone.
Problem Statement. We study generative modeling from corrupted observations. Let x E Rd be drawn from an unknown clean distribution px. We observe only
A(x) + oc, C0.
where A Rd - I. is (known) non-invertible corruption operator and is the noise level. The operator may be identity (i.e., denoising). a deterministic linear map (blur, downsampling). a random mask (inpainting). or Fourier-domain undersampling pattern (MRI). Our goal is to learn a generator
whose samples follow px using only dataset of N corrupted datapoints (1(0) N of the form Eq
Background and limitations. Diffusion models also known as score-based generative models Song and Ermon (2019): 9) Song et al. 222 achieve
et al. 2001 ; Ho et al. 2020).

---

Published as a conference paper at ICLR 2026
Problem Statement
Our Method: Restoration Score Distillation
How to leern from corrupted data?
Phase 1: diffusion pretraining
Flexible coruution-sware
step generator
Clean Disribution
Comppttd
Corruption Pipeline
A
teacher dillusion
Comuption
Notio
generatirr
fake ision
Random Inpainting
Gaussion Deblurinng
(c) Super Resolution
Masked
Downsampled
RSD
RSD
FSD
FID: 275.04
FID: 4.44
FID: 72.83
FID: 52.71
FID: 28.94
RID: 12.98
Figure 1: Overview of Restoration Score Distillation (RSD) and some qualitative results across diverse operators: Gaussian deblurring, random inpainting. and super-resolution. Additional examples appear in Appendix A and
state- taate-of-the-rne the-art results nigh-dimensional image synthesis Dhariwal and Nichol (2021 y Ho et al. (2022); Ramesh al. (2022); Rombach et al. (2022); PPeelle and Xie (2023) Zheng
(2024) )
Zhang et al. (2025); ; Chang et (2025): Chen et al. (2025). When only measurements are available,
corruption-awes aware training adapts diffusion objectives the forward operator: Ambient Diffusion for masking Daras et al. (2023b), Ambient Tweedie for additive noise Daras et al. (2025), and Fourier-space variants for ARli et (2025). These methods, however, inherit the sampling cost of multi-step reverse processes. EM-Diffusion Bai et al. (2024) offers broader operator coverage but requires a few clean images, computationally expensive, and can struggle under severe corruption.
Distillation beyond acceleration. Score distillation transfers a pretrained diffusion teacher into
one-step generator while largely preserving fidelity (Poole al. 2022 Wang al. 2024b Luo
et al. 2024 Yin et al. 2024b; Zhou et al., 2024 Xie et al., [2224] Xu et al. 2025 Yin et al. 2024a). Recent reports indicate that distilled generators can even outperform their diffusion teach- ers (Zhou et al. (2024), though gains under clean-data training are typically modest. In contrast, under corrupted-data-on/ training, we observe substantially larger improvements over the teacher (Sec. underscoring distillation's 's particular advantage in challenging regimes.
Restoration Score Distillation (RSD). We introduce RSD, unified framework for learning high- fidelity one-step generators directly and only from corrupted observations, with denoising (AA=l) as special case. RSD proceeds in two stages: (i) corruption-aware diffusion pretraining, where a teacher is trained on measurements using an objective matched to the forward operator A (Sec. 3.1) and (ii) score distillation, which transfers the teacher into single-step generator while explicitly respecting the measurement operator during training (Sec. 3.2). Concretely, synthesize measurements by applying the same corruption pipeline to generator outputs and then align the induced generator scores with the teacher's scores under divergence (c.g., Fisher KL). The distillation phase, which includes a corrptioo-respectinn procedure, consistently improves generation quality upon the diffusion teacher across both denoising and more general operators (Fig.)
Contributions. Our work makes three primary contributions. Unified framework for diverse corruptions: We propose RSD, unified approach that learns generators directly from diverse corrupted measurements y A(x) ) toe. This formulation encompasses denoising (AA = /) as well as more general operators, including blur, downsampling, random masking, and Fourier undersampling, under both noisy and noiseless regimes, and achieves of-te-of-the-art performance across these settings. Modular training: Our training pipeline is organized into phases. Phase I accommodates variety corruption-awe techniques, such as standard diffusion, diffusion for denoising, random inpainting, and masked Fourier-space (F.S.) transformations. Phase distills the teacher model into
2

---

Published as a conference paper at ICLR 2026
Table Summary of corruption-ave -ware diffusion objectives used for pretraining. Our framework can be seamlessly integrated with existing advanced corruption-awes aware diffusion objectives.
Suitable Secnario
Algorithan
Operator
Domain
Pretrain Objective
Notation
Noiscless Corputtinn
2la
A(x)
Image
Cip (x ese.t) v
corrupted data
Nonky Comuption
A(x)
Image
E 2i folu O fj vil
data moise level, 6 max(s,, 0)
Random Ispainting
2
Mr
Image
Cu 3 (A My., a1i
A7: further cormipted mask
Misked F.S.
6
MFF
Fourier
Ar BlIA// R.,4) -xli)
Gr: further cormpted y.. defined in Apps H.4
one-step generator while retaining the corruption pipeline used during training. This modularity makes it straightforward to plug in new forward operators or training objectives. We also provide theoretical analysis that explains why the distillation phase can enhance generation quality. Extensive experiments: We conduct comprehensive evaluations on natural- image benchmarks (CIFAR-10) 10, CelebA-HQ, FFHQ, AFHQ-v2), restoration tasks (denoising with a E {0.1, 0.2, 0.4], Gaussian deblurring, random inpainting with p E (0.6, 0.8, 0.9), and 2 super-resolution), and multi-coil MRI with acceleration factors R E {4, 6, 8}. Across all settings, RSD consistently improves FID over corruption-awan diffusion teachers while offering substantial speedups via one-step generation (Sec. J. Additional ablations (Sec. 4.5) on unknown corruption types and different data size further underscore the robustness of our framework. Moreover, we demonstrate that the learned clean-image prior (the generator) can be directly leveraged for downstream conditional inverse problems, achieving good performance (Sec. 4.7)
2
BACKGROUND
2.1 DIFFUSION MODELS
Diffusion models Sick-DDckstein et al. (2015); Ho et al. (2220), also known as score-based generative models Song and Ermon (2019); Song et al. (2021b). consist of forward process that gradually perturbs data with noise and reverse process that denoises this signal to recover the data distribution noise levels N(axx, of 7), with marginals q (I We adopt a variance-explodingg (VE) process Song and Ermon 2019) by setting o₁ 1, yielding the simple form 3y
Px (x). Specifically, the forward process defines family of conditional distributions
t E (0, 1]. given by ge (3xx 2))
o where e ~ .V (O, To model the reverse denoising process, one typically trains a time-dependent denoising autoencoder (DAE) (- d x 10, 1] - IRd Vincent (201 1b, parameterized by a neural network, to approximate the posterior mean Eiz xr). This Is achieved by minimizing the following standard diffusion loss:
(d (2o = E. [A(I)Ifa 1) xx|2]
where = x + 0,E and 1 denotes the dataset. One can also apply the loss to the corrupted data { Ji=1 to directly learn the distribution py (v). For clarity, we omit the diffusion training
schedule X(t), p(oz) and the noise term E (O, Im);; full details are deferred to Appendix H. A The corresponding objectives are summarized in Table [ with additional discussion below.
2.2 DIFFUSION MODELS FOR CORRUPTIONS
Recent advances have extended diffusion models to address diverse forms of data corruption. We present several variants of the diffusion loss tailored to different corruption settings, including objectives for noisy data, random inpainting in the image domain and Fourier domain.
Diffusion for Noisy Corruptions. Daras et al. (2025) generalizes score matching to noisy observa- tions y = 2 + de, E ~ (0, I). Here, a >> is a known noise level in the measurements, One can incorporate the known corruption by minimizing the following diffusion for noisy corruptions
0, 0²
x²
Ex(0: (1 = Ey....
A(t)
fol,
+
U1- 0
(3)
o
where 5, max 0z J.......11 y og, and p(or)) and A(t) arise from the diffusion training schedule. This formula models the distribution of the x, := x + o,E where o, 2 a. Full details are deferred to Appendix H.2 the training algorithm in Algorithm 3 and the sampling algorithm in Algorithm
3

---

Published as a conference paper at ICLR 2026
Diffusion for Image-Space Random Masked Corruptions. Daras et al. (2023b) learn px (a) from randomly inpainted measurements by incorporating mask into the diffusion objective. Given
observations y Mr for binary mask M, the teacher is trained with random random-inpainting inpainting loss
=E (g.M), Mr, o..e [IM (fo(M, My, t) - -) 112]
(o
(4)
where y/ y ore is from diffusion schedule, (O 1), and M is secondary mask that further erases pixels based on M. Details of the diffusion schedule A(t), p(oz), and the full training procedure are provided in Appendix H.3
Algorithm(5
Diffusion for Fourier-Space Random Masked Corruptions. (Aali ct al. 2025) extends diffusion models handle frequency-dome domain measurements with random masking, which are used in scientific imaging (e.g., MRI). Specifically, observations take the form y = MFF,, where F denotes the Fourier transform operator and M a sampling mask. Formal definitions of the measurement process and the corresponding training algorithm provided in Appendix H..4 and Algorithm
These specialized frameworks enable generative modeling from realistic scenarios such as noisy, missing data, or frequency-domain degradation. All objective functions are summarized in Table
2.3 SCORE DISTILLATION FOR GENERATIVE MODELING
Score distillation compresses multi-step diffusion models into efficient one-step generators. Originally proposed for text-to-3D generation Poole et al. (2022); Wang et al. (2024b) and later extended to image synthesis Luo et al. (2024); Yin et al. (2024b) Zhou et al. (2024) Xie et al. (2024), it transfers knowledge from a pretrained diffusion teacher fo to a generator Go :
R To bridge the two, a
fake diffusion model f approximates the distribution induced by Ga(-) across diffusion noise levels.
Training encourages consistency between fg. and fø over time:
Lassin = E.g,, (( B Go(s)) [II,s(Fs))) - So (x,,,11)) 1)1
(5)
where Jy Tig aFe and the loss corresponds Fisher divergence, following SiD Zhou al. (2024). This divergence of true and fake distribution can be replaced by KL divergence; ablation results are reported in Section 4.5 and Appendix D.5 Intuitively, the fixed teacher fo represents the true data distribution, while Js captures the generator's ss induced distribution. Updating Go to minimize (assi(()) aligns generator samples with the true data. Notably, distilled generators can even outperform their teachers: for instance, Zhou et al. (2024) reports that on FFHQ the teacher achieves an FID of 2.39, whereas its distilled generator attains 1.55. our setting, we find that such improvements are further amplified when distillation is performed under corrupted-date training. Details of the distillation training schedules are deferred to Appendix
RESTORATION SCORE DISTILLATION
Problem Statement. Suppose we are given a finite corrupted dataset of size N, denoted by W Each corrupted observation is generated as ( = A(x(1)) + (1) where is a known noise level (with extensions available for the unknown-o setting, see Section 4.5). and e(i) itd. (0, Crucially, the clean data 1, is never accessible. In certain scenarios, such as random inpainting, the corruption operator A may vary across samples, drawn from a common distribution. In this case, each corrupted observation takes the form y (0) (z)(0) + (1)
RSD two-phase framework for learning generative models solely from corrupted observations. It decouples the process into: (1) flexible corruption-awar diffusion pretraining stage, and (2) a distillation stage that compresses the pretrained model into a single-step generator and further improves generation quality. The overall procedure is summarized in Algorithm [ We summarize different types of pretrained diffusion models in Table and then describe how they can be seamlessly integrated into RSD to further enhance performance. Notably, even when suitable corruption-awan pretrained model is unavailable, employing a standard diffusion model (Tab T to learn the corrupted data distribution still yields substantial improvements after distillation (Tab
4

---

Published as a conference paper at ICLR 2026
3.1 PHASE I: FLEXIBLE CORRUPTION-AWARE PRETRAINING
The first phase trains a teacher diffusion model fa(-, () directly on corrupted observations = A(x) + te, where A is a potentially non-invertible corruption operator. A straightforward approach is to train on corrupted data using the standard diffusion loss 2). In practice, we find that this naive strategy already yields strong performance. To further improve performance across diverse corruption operators, we incorporate existing corruption-awan diffusion training methods, as summarized in Table T Our choice of pretraining objectives for different corruption scenarios is as follows: 1) Standard Diffusion is effective under noiseless corruptions, directly modeling the distribution of y = A(x). 2) Noisy Corruption Daras et al. (2025) targets additive noise settings, learning the distribution of A(x) from noisy observations y A(x) + oe, with denoising as a special case when
A I. Random Inpainting Daras et al. (2023b) addresses learning from partial observations Mr. where M is random inpainting mask, and learns the distribution of x. 4) Masked Aali et al. (2025) extends inpainting to Fourier-doman corruptions, well-suited for frequency- degradation such accelerated MRI, learning from masked Fourier observations M.s..
3.2 PHASE II: ONE-STEP GENERATOR DISTILLATION
After pretraining, we distill the teacher diffusion model fa(:, () into a one-step generator Gø that maps latent noise ~ (o, directly to clean samples. To facilitate this distillation, we introduce an auxiliary fake diffusion model fø (-, t), initialized from fo and trained on corrupted output of Gz(z), which also initialized from fa- Note that this is common practice in the distillation literature to facilitate and stabilize training. In this case, Go, for fo share the same network structure; see Appendix[foo for further discussion. By encouraging fy to match the teacher's time-dependent dynamics (e.g., score fields parameterized by fa), we align the behavior of Go with the data distribution learned by fa. thereby narrowing the gap between the generated and clean data distributions.
We adopt the SiD Zhou et al. (2024) framework for distillation and optimize the distillation loss Ldisun (0) (Eq. [ An ablation of loss choices is provided in Sec. [4.5 Beyond standard score distillation, we further corrupt the generated samples Fg Go(z) into y using the same corruption pipeline as the training data (Algorithm) [ Line 6). These corrupted samples are then used to train a fake diffusion model ()) In Algorithm] 1 Lines 2 and 7 allow any of the pretraining objectives summarized in Table [ (or new objectives tailored to specific corruptions), but the same objective must be used for both lines; mixing objectives destabilizes training and can lead to divergence. To complement the empirical results in Section[4] [ Appendix [F] offers a theoretical analysis establishing conditions under which distillation yields improved sample quality.
Algorithm 1 Restoration Score Distillation (RSD)
procedure RESTORATION SCORE DISTILLATION( (1) M
2: fo +DIFFUIONTRALNNNNGE. ( Obj)
> Diffusion training with Obj e: Tab.
3: Initialize fo < fe. Go +-1 f
4: for j 1..... do
5:
Zg +- Go(z),
La)
Generate fake clean images
6:
Corrupt Tg same way as observation If corruption contains noise, choose an Objfor J J.p that is denoising-Cen
A( stopgrad(zz)) + T ~ (00 Ia)
7:
8:
fo + DIFFUSION TRAIIINNTTAIIINN((P" Ti=1, ) Obj)
& Same Obj as above
9:
# Do NOT inject additional noise during distillation when updating 0
10:
ye + (z), 2 ~ ~N(00.1 Ia)
11:
Update by distillation loss (0) with generated xy ye
Bq.
12:
end for
13:
end procedure
EXPERIMENTS
We conduct comprehensive experiments to evaluate the effectiveness. flexibility, and generality of our framework, RSD, across both natural and scientific imaging domains. We first demonstrate its performance through a 2D toy task and qualitative visualizations, providing intuition into the
5

---

Published as a conference paper at ICLR 2026
(a) Noisy dataset 0.05
(b) Teacher-Full
(c) Teacher-Truncater
(d) RSD (ours one-step)
Figure 2: A toy example of learning from noisy dataset with = 0.05.
underlying mechanism. Then consider four challenging image restoration tasks denis-dnnoiiing. Gaussian deblurring, random impainting, and super-resolution on CIFAR-10 (32 x 32). FFHQ (64 64), and CelebA-HQ (64 64) Liu al. (2015); Karras et al. (2017). under both noiseless (a = 0.0) and noisy (a = 0.2) regimes. We further show that RSD can be distilled directly from pretrained teacher models trained with corruption-an aware objectives, including diffusion for denoising Daras et al. (2025), diffusion for random inpainting Daras et al. (2023a), and diffusion for Fourier-space inpainting Aali al. (2025). highlighting the framework's modularity and flexibility. Sec. 4.2 benchmarks denoising across multiple datasets and noise levels. Sec. 4.3 evaluates general corruption operators both noiseless and noisy settings, including a sweep over random -inaanniing missing rates p {0.6, 0.8, 0.9).
To assess real-world applicability, we evaluate RSD on multi-coil Magnetic Resonance Imaging (MRI) using the FastMRI dataset Zbontar et al. (018); Knoll al. (2020), a setting where fully sampled ground truth is often unavailable. Across tasks and corruption regimes, RSD consistently surpasses teacher diffusion models in generative quality measured by FID Heusel et al. (2017) without access to clean data (Sec. 4.4
Finally, RSD exhibits strong distillation-data efficiency (Sec. 4.6 it quickly surpasses the
teacher-and offers substantial inference-time gains from its one-step design (up to 30x; see Sec. (4.6). Our ablations (Sec. 4.5) show that the framework accommodates unknown noise levels a and is robust with respect to the training-date size; we also examine alternative distillation losses in an ablation study. Separately, Sec. Bec. demonstrates that the trained generator achieves superior performance on downstream conditional inverse problems, indicating broad applicability.
4.1
2D ToY EXAMPLE
In Fig. 2 the observation is noisy spiral dataset with 0.05. The issue with teacher diffusion models trained on noisy datasets, i.e., (b) Teacher-Full and (c) Teacher-Trunne Truncated (Daras) al. (2024), Algorithm 4 is that they force the approximating distribution spread its probability mass across all regions, making the learned density overly diffuse. In contrast, (d) RSD excels at denoising the original dataset, producing narrow, concentrated, and sharp approximation. This effect occurs without providing the clean data itself during training.
4.2 DENOISING AS SPECIAL CASE OF RESTORATION SCORE DISTILLATION
We begin with the denoising special case I + de where the for- ward operator A = I. We compare RSD against three groups of base- lines (Table 2. (1) Teacher diffu- sion models. A teacher trained with the noisy corruption loss in Eq. 3 serves strong generative baseline. adopt the two sampling sched- Teacher-
Table 3: RSD vs. teachers on CIFAR- CIFAR-1FFFQ//AFHO- RSD (distilled) consistently surpasses teacher diffusion (Full, Truncaled) across datasets and noise levels. (FID).
Methods Data noise
CIFAR-10
FFHQ
AFHQ-v2
a 0-0.1
a 0.2
0 0.4
5
5 0.2
Observation Teacher-Full Teacher-Truncant
73.74
127.22
205.52
110.83
51.51
25.55
60.73
124.28
41.52
17.93
ules Daras et al. (2024) - Full and Teacher- Truncated (Algo- rithm and additionally evaluate the Teacher-consistency variant Daras et al. 2025). There may exist better sampling schedules for denoising; however, the best of our knowledge, the schedules adopt achieve strongest
7.55
12.21
22.12
14.67
9.82
RSD (Distilled)
3.98
4.77
21.63
6.29
5.42

---

Published as a conference paper at ICLR 2026
Table 2: Denoising results on CIFAR-10 and CelebA-HQ
0.2. Rows with a=0.0 are clean-data upper bounds. Few-shot methods use 50 clean images. Baseline numbers are from the original papers or Bai et al. (2024); Lu et al. (025). The distilled student (RSD, one-step) improves over teacher models Full/Truncated//Conissssy and surpasses all few-shot baselines.
CIFAR-10 (32x32)
CelehA-HO (64x 64)
Methods
Type
FID
Type
FID
DDPM Ho al.
2020
0.0
Full-Shot
4.04
0.0
Full Full-Shot
3.26
DDIM Song
2021
0.0
Full- Shll-Shot
4.16
0.0
Full Bull-Shot
6.53
EDM
Karras al
2014
0.0
Full-Shot
1.97
SURE-Score Auli al.
2023
0.2
Few-Shot
132.61
Ambient Difluston Daras el al.
2023a
0.2
Few-Shot
114.13
EM-Diffusion al.
2024
Few-Shol
86.47
TifedieDiff Darus
2024
0.2
Few-Shot
65.21
0.2
Few-Shot Few-Shot 6.49
58.52
SFBD et 2025
0.2
Few-Shoi
13.53
0.2
TweedieDiff Darus al.
2024
0.2
Zero-Shot
167.23
0.2
Zero-Shot 246.95
Teacher-Full] Duras al.
2025
0.2
Zero-Shot
60.73
0.2
Zero-Shot 61.14
Teacher-Truneteed Daras et al.
2025
0.2
Zero-Shot
12.21
0.2
Zero-Shot 13.90
Woacher-Conssseeneyy Daras 2025
0.2
Zero-Shot
11.93
0.2
Zero-Shot 12.97
RSD (Ours, One-Step)
0.2
Zero-Shot
4.77
0.2
Zero-Shot 6.48
Table 4: CelebA-HO restoration under noiseless/noisy settings. Baselines are taken from original papers when available, otherwise reproduced. EM-Diffusion uses 50 clean images for initialization, while RSD uses none. Best results are highlighted.
Methods
Gaussian Deblurring Random Inpainting (p 0.9)
Super-Resolttion (x2)
Observation
72.83
396.14
23.94
Teacher Diffusion
94.40
25.53
23.28
EM- EM-Diffusion (Few-Shor)
56.69
104.68
58.99
RSD (Ours, One-Step)
31,90
16.86
12.99
Observation
264.37
419.92
200.04
Teacher Diffusion
99.19
319.34
23.92
EM- MM--iffusion (Fow)Shorr
51.33
165. 60
$7.31
RSD (Ours, One-Step)
76.98
79.48
22.00
performance reported in prior work. (2) Few-shot methods. EM-Diffusion Bai et al. (2024) alter- nates DPS-based reconstructions (E-step) with model refinement (M-step), while SFBD
al.)
(2025) casts the problem as density deconvolution; both use 50 clean images for initialization. (3) Clean-data diffusion (upper bound). DDPM Ho et al. (2020), DDIM Song et al. (2021a), and
KaMas et al.) (2022) are trained on clean data (a=0) and serve as upper bounds for any method trained purely on corrupted observations. Across CIFAR- 10 and CelebA-HQ at a=0.2, RSD (one- step) outperforms all zero-shot and few-shot baselines and improves
its teachers (Table. 2).
For FFHQ and AFHQ-v2 where prior few-shot results are not reported, RSD also surpasses teacher models (Table 3 Finally, sweep noise levels {0.1, 0.2, 0.4) CIFAR-10 and observe consistent gains over teachers. Quality examples are provided in Appendix A and K
4.3 GENERAL CORRUPTIONS MIXED WITH NOISE
We now move beyond denoising to the general measurement model y A(x) + where A may be
non-invertible. Using CelebA-HQ as a running example, we study both noiseless (a = 0.0) and noisy (a 0.2) regimes and instantiate A as (i) Gaussian deblurring, (ii) random inpainting, and (iii) 2x super-resolution. Aggregate results across tasks and noise levels are reported in Table 4 For random inpainting, further sweep the missing rate p (0.6, 0.8. 0.9): see Table 5 Across settings, RSD-trainne without clean mmages-consistemt competitive with, exceeds, few-shot methods that rely on clean initialization. Qualitative examples are provided in Appendix A and
improves over its teacher diffusion models and is
4.4 BEYOND NATURAL IMAGES: MULTI-COIL MAGNETIC RESONANCE IMAGING (MRI)
7

---

Published as a conference paper at ICLR 2026
Table 5: CelebA-HQ random inpainting vs. missing rate Teacher diffusion is trained as in Daras et al. (203bb)
Table 6: FID across acceleration levels in multi-coil MRI. Teacher diffusion is trained as in Aali al. (0244..
Method
p 0.6
p 0.8
p 0.9
Method
R 2
R =
R
6
R 8
Observation
275.04
383.82
396.14
LI-EDM
18.55
27. 64
$1.43
102.98
Teacher Diffusion
6.08
11.19
25.53
Teacher Diffusion
30.34
32.31
31.50
48.15
RSD
4.44
7.10
16.86
RSD
12.95
10.71
14.64
22.51
Observvttio
(ours)
Ground
R=
X
A
R-6
R-8
qualitative reconstructions from RSD across acceleration levels R. See Tab. 6 for corresponding FID trends.
Figure 3: Multi-coil MRT:
Table 7: Comparison of RSD with teacher diffusion and
We next apply RSD to a practical medical-imaging setting where fully sampled data are often unavailable due to time and cost constraints Knoll et al. (2020); Zbontar et al. (2018); Tibrewala et al. (202)); Desai et al.
baselines on CIFAR-10. Metries for DDPM and Rectified Flow are from 34
Method
FID.
IST
Prec.
Rec.
KID
Teacher
0.2
12.21
8,31
0.59
0.41
.0059
(2022). This case study demonstrates: (i) the flexibility of RSD to integrate with advanced corruption-awas sion techniques, (ii) robustness across acceleration factors R. and (iii)
RSD
0.4
21.63
7.93
0.53
0.38
.0127
RSD
0.2
4.77
9.16
0.65
0.56
.0025
RSD
0.1
3.98
9.34
0.64
0.57
.0015
diffu-
DDPM
0.0
3.21
9.46
N/A
0.57
N/A
RF (ODE)
0.0
2.58
9.60
N/A
0.57
N/A
tension from real- to complex-valued signals, & E Cd. Table reports FID versus acceleration
R E (2, 4, 6, comparing RSD, teacher trained via Fourier-Space Ambient Diffusion Aali et al. (2025), ) and a baseline L1-EDM that trains EDM Karras et al. (022) on L.-WWvelet reconstruc- tions Lustig al. (2077). Across all acceleration levels, RSD improves over the teacher; notably, also outperforms LI-EDM at low acceleration (Rt=2), suggesting that distillation-ban regularization more effective than handcrafted LI priors in the wavelet domain. Qualitative examples are shown in Fig. 3 with full algorithmic details in Appendix H.4 and
4.5 ABLATIONS: DATA SCALE, DISTILLATION Loss CHOICE, AND MORE
Data scale. We next vary the training set size
Table 8: Data-size ablation on CelebA-HO.
on CelebA-HQ to test data efficiency. RSD maintains strong performance even with 10% of the data and improves as more data become available, outperforming both Teacher-Full and Teacher- Truncated across settings. The results shown in Table g
Data Size
Teacher-Full
Teacher-Truie
RSD
10%
62.25
14.36
10.53
50%
56.09
17.19
9.76
100%
61.14
13.90
6.48
8

---

Published as a conference paper at ICLR 2026
Comprehensive data quality metrics. While FID is the primary metric for assessing data quality in image generation, we incorporate additional metrics for a more holistic evaluation, as detailed in Table / The results demonstrate that our distilled RSD generator achieves both superior density modeling (low FID) and robust mode coverage (high Recall), even when compared to generative models trained on clean datasets.
Choice of distillation loss. We compare several popular distillation losses used compress diffusion teachers into one-step generators, including SDS Poole et al.] (2022). DMD Yin et al. (2024b), and SiD Zhou et al. (2024). Unless otherwise noted, we use the default hyperparameters from the original papers (and public repos) without task-specific tuning. Under these settings, SiD consistently yields the strongest FID across our datasets, while SDS and DMD underperform. We report the full per-dataset and per-o breakdown in Table [8 (Appendix D.5). We emphasize that we did not perform a hyperparameter sweep; consequently, better-tuned configurations of SDS or DMD may close the gap SiD. Empirically, we find SiD's default hyperparameters at different setting is stable choice in our RSD pipeline.
More ablations can be found in the Sec. D of the Appendix.
4.6 SAMPLING AND TRAINING EFFICIENCY FOR DISTILLATION
After distillation, RSD attains markedly higher inference throughput than its diffusion teacher. On CIFAR-10 at a 0.2, producing 50,000 samples drops from 10 minutes (teacher) to 20 seconds (generator). a ~30 reduction in wall-clock time. Training is like- wise efficient: the Phase // distillation stage surpasses the teacher's FID in roughly 4 hours, compared to 48 hours for Phase I teacher pretraining. Thus, once the teacher is available, running RSD adds only a minor computational budget yet yields substantial quality and speed gains. Figure 4 summarizes both sampling and training efficiency. All wall-clock times were measured on 8x RTX A6000 (32 GB). Additional quantitative statistics and efficiency examples provided in Appendix Cand Table 12
Tertiggee prreeer
Toathr
Past Phane
Cumulative Training Wall Time (hours)
Figure 4: Efficiency of RSD (CIFAR-10, = 0.2).
4.7 AFTER DISTILLATION: SOLVING CONDITIONAL INVERSE PROBLEMS
Table 9: Conditional inverse problem results of denoising on CIFARIO at a = 0.2. Results for the baselines are taken from [5]. We follow [5] and
Because our framework yields a high-quality clean image generator-whine whhch naturally serves as a prior in conditional inverse solvers Chung et al. (2022), Zhang al. (2024); Zhu al.
sample 250 test images and compute the average PSNR and LPIPS.
2024
Zhang and Leong (2025): Hertrich et al.
evaluate its util- ity on downstream inverse problems. We report denoising task performance in Table where RSD substantially outperforms prior methods and achieves performance comparable to few- shot approaches such as EM-Diffusion. In our implementation, we solve min, A(G
2025 direct extension
Method
Type
PSNR
LPIPS!
Observations
18.05
0.047
DPS w/ clean prior 12
Full-Shot
25.91
0.010
SURE-Score
[
Few-Shot
22.42
0.138
AmbientDiffusion
Few-Shot
21.37
0.033
EM-Diflusion
Few-Shot
23.16
0.022
Noise2Self
Shot
21.32
0.227
y for 1000 steps using Adam Kingma 2014)
RSD (ours)
Zero-Shot
24.11
0.025
with a learning rate of 0.05. Exploring alter- native strategies for inverse problems with one-
step generators is an exciting direction. Additional results are provided in Appendix D.3
4.8 MODEL SELECTION CRITERION UNDER CORRUPTION: PROXIMAL FID
Prior baselines Daras et al. 2023a): Bai al. 2024) report FID scores yet do not specify how to perform model selection under corruption during training. To address this gap, we introduce Proximal FID, model-selection metric tailored to corrupted-date regimes. Concretely, we generate 50k clean samples (x)() 50k from the current generator, corrupt them match the training noise

---

Published as a conference paper at ICLR 2026
and operator- yielding {A(x(9) + (() 50k
i==" and compute FID against the corrupted training set "y/")= As shown on CIFAR-10 10 in Fig. 5 and Tab. 10. Proximal FID tracks the true FID closely throughout distillation and attains near-optimal true FID across datasets. More results can be found in the Appendix. Taken together, these results support Proximal FID as a practical and reliable proxy for model selection when clean data are unavailable.
Table 10: Model selection by Proximal FID. Best true FID obtained by selecting checkpoints via Proximal FID vs. selecting via true FID (ora- cle). The red (+:) indicates the gap to the best achievable true FID during training.
Dataset
Proximal FTD (selected)
Best true FID (oracle)
Figure 5: CIFAR-10 (denoising, σ = 0.2): Proximal FID closely tracks true FID and lects a near-optimal checkpoint.
CIFAR-10
(+0.45)
4.76
FIHQ
6.12 (+0.04)
6.08
CelebA-HQ
6.90 (+0.54)
6.36
APHQ-12
5.45 (+0.06)
5,39
4.9 WIIY SCORE DISTILLATION BEYOND ACCELERATION?
Score distillation is traditionally viewed as means of accelerating sampling by compressing a multi-step diffusion process into single forward pass. However, our paper aims reveal a more fundamental conceptual shift: In corrupted or low-quality data regimes, score distillation serves as a principled mechanism for improving the sample quality of teacher diffusion models.
Diffusion models are trained using the ELBO objective for maximum likelihood, which minimizes the forward KL divergence, mode-covering objective. As a result, teacher diffusions are incentivized to place probability mass on all plausible locations suggested by noisy data, even if these regions do not correspond the true clean distribution.
Intuitive Understanding:
In contrast, the reverse distillation objective (Eq. 5) takes its expectation over the student generator's 's distribution, which naturally induces mode-seeking behavior: the generator does not need to cover all regions where the teacher assigns non-zero mass. Instead, it is encouraged to place probability only where will actually sample from. Importantly, we stress that encouraging mode seeking does not imply mode collapse. This is also evidenced by the FID and Recall in Tab. 7 qualitative results in Fig. and extensive qualitative examples in Appendix K This change in objective allows the one-step generator to concentrate its probability mass the high-densiry, well-supported regions implied by the teacher's ' score field, while discarding diffuse or low-density regions that the teacher includes.
Theoretical Support: Detailed analysis is provided in the appendix. Sec. F.1 first focuses linear Gaussian setting, providing a direct quantitative analysis and error bounds (Eq. 9). These findings are extended to general linear corruption in Sec. F22 Sec. derivve performance bounds for the distilled generator under a more general setting. We posit that a deeper theoretical analysis of the surprising effects of score distillation under corruption is promising research direction.
CONCLUSION
In this work, we introduced Restoration Score Distillation (RSD) to learn clean data distribution from broad class of corruption types. Our empirical results on natural images and scientific MRI datasets show consistent improvements over existing baselines. Moreover, beyond standard diffusion objectives, the RSD framework is compatible with several corruption-wware training techniques, enabling flexible integration with recent advances in diffusion modeling. Together, our contributions highlight the potential of score distillation powerful mechanism for robust generative learning in real-world settings where clean data are searce or unavailable. detailed discussion of limitations is provided in Appendix E
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work develops methods for training generative models from corrupted data without requiring access clean ground truth. The primary applications target are scientific and medical imaging domains where clean acquisitions are expensive or infeasible. Our framework does not involve human subjects, personal data, or harmful content, and thus poses minimal ethical risks. Nevertheless, as with all generative models, there is potential for misuse in creating synthetic content; to mitigate this, we emphasize the intended use our approach scientific and restoration contexts.
REPRODUCIBILITY STATEMENT
We provide complete algorithmic and training details in the Appendix. To ensure reproducibility, we will release all code and model checkpoints upon acceptance of this manuseript.
ACKNOWLEDGMENTS
Ying Nian Wu supported in part by NSF DMS-2415226. DARPA W912CG25CA007, and research gifts from Amazon and Qualcomm. Tianyu Chen and Mingyuan Zhou acknowledge the support of NSF-IIS 2212418 and NIH-R37 CA271 We gratefully acknowledge Microsoft for providing the computing resources used in this project.
REFERENCES
A. Aali, M. Arvinte, S. Kumar, and I. Tamir. Solving inverse problems with score-based generative priors learned from noisy data. 2023 57th Asilomar Conference on Signals, Systems, and Computers, pages 837-843. IEEE, 2023.
A. Aali, G. Daras, B. Levac, S. Kumar, A. G. Dimakis, and I. Tamir. Ambient diffusion posterior sampling: Solving inverse problems with diffusion models trained on corrupted data. arXiv preprint M.XV:22033.0828.. 2024.
A. Aali, G. Daras, B. Levac, S. Kumar, A. Dimakis, and Tamir. Ambient diffusion poste- rior sampling: Solving inverse problems with diffusion models trained on corrupted data. In The Thirteenth International Conference on Learning Representations, 2025. URL https 7openreview net. forett/foumm?1d1mm foreeoffouuTT10eeeMMMERE
K. Akiyama, A. Alberdi, W. Alef, K. Asada, Azulay, A.-K. Baezko, Ball, M. Baloković, J. Barrett, D. Bintley, al. First m87 event horizon telescope results. iv. imaging the central supermassive black hole. The Astrophysical Journal Letters, 875(1):L4, 2019.
W. Bai, Y. Wang, W. Chen, and H. Sun. An expectation -axiatiioommaimmziiiiii algorithm for training clean diffusion models from corrupted observations. In The Thirty Annual Conference on Neural Information Processing Systems, 2024. URL https: http:ss//pperevvic. ew httparrwwwwwwweeeeeeeeeeeeeeeeee formm? id= JURBh4V9N4]
J. Batson and I.. Royer. Noise2self: Blind denoising by self-supervision. In International conference machine learning, pages 524-533. PMLR, 2019.
Y. Chang, Y. Zhang, Z. Fang, Y. Wu, Y. Bisk, and F. Gao. Skews the phenomenon space hinder generalization in text-to-image generation. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol, editors, Computer Vision - ECCV 2024, pages 422-439, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031--3021--
M. Chen, K. Huang, Zhao, and M. Wang. Score approximation, estimation and distribution recovery of diffusion models on oiwndinnsiional data. In International Conference on Machine Learning, pages 4672-4712, PMLR, 2023.
S. Chen, V. Kontonis, and K. Shah. Learning general gaussian mixtures with efficient score matching. arXiv preprint arXiv:2404. /0044044/889,,.. 2024a.
11

---

Published as a conference paper at ICLR 2026
T. Chen, Z. Wang, and M. Zhou. Enhancing and accelerating diffusion-based inverse problem solving through measurements optimization. arXiv preprint arXiv:2412.0391,, 2024b.
T. Chen, Y. Zhang, Z. Zhang, P. Yu, S. Wang, Z. Wang, K. Lin, X. Wang. Z. Yang, L. Li, et al. Edival-agent: An object-centric framework for automated, fine-grained evaluation of multi-turn editing. arXiv preprint arXiv:2509. 13399, 2025.
H. Chung, J. Kim, M. T. Mccann, M. L.. Klasky, and J. C. Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209. 14687, 2022.
H. Cui, F. Krzakala, E. Vanden-Eijnden, and L. Zdeborová. Analysis of learning a flow-based generative model from limited sample complexity. arXiv preprint arXiv:2310.03575, 2023.
K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image processing, 16(8):2080-2095, 2007.
G. Daras, K. Shah, Y. Dagan, A. Gollakota, A. Dimakis, and A. Klivans. Ambient diffusion: Learning clean distributions from corrupted data. Advances in Neural Information Processing Systems, 36: 288-313, 2023a.
G. Daras, K. Shah, Y. Dagan, A. Gollakota, A. Dimakis, and A. Klivans. Ambient diffusion: Learning clean distributions from corrupted data. Advances in Neural Information Processing Systems, 36: 288-313, 2023b.
G. Daras, Dimakis, and C. C. Daskalakis. Consistent diffusion meets tweedie: Training exact ambient diffusion models with noisy data. In Forty-first International Conference on Machine Learning, 2024. URL https: oppnrevieee net www.mrorrrroorrrere Porumnid---1V
G. Daras, Y. Cherapanamjeri, and C. C. Daskalakis. How much is a noisy image worth? data scaling laws for ambient diffusion. In The Thirteenth International Conference Learning Representations, 2025. URL https: opw.revien.eeeeeeeeoeeeee. /peneeee// net
A. D. Desai, A. M. Schmidt, E. B. Rubin, C. M. Sandino, M. S. Black, V. Mazzoli, K. J. Stevens, R. Boutin, C. Ré, G. E. Gold, al. Skm-tea: A dataset for accelerated mri reconstruction with dense image labels for quantitative clinical evaluation. arXiv preprint AXiv:2203306822 2022.
P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. Advances in neural information processing systems, 34:8780-888 8794, 2021.
D. L. Douoho and I. M. Johnstone. Ideal spatial adaptation by wavelet shrinkage. biometrika, 81(3): 425-455, 1994.
S. Foucart and H. Rauhut. An invitation compressive sensing. In A mathematical introduction compressive sensing, pages 1-39. Springer, 2013.
Hertrich, H. S. Wong, A. Denker, S. Ducotterd, Z. Fang, M. Haltmeier, Ž. Kereta, E. Kobler, O. Leong. M. S. Salchi, et al. Learning regularization functionals for inverse problems: A comparative study. arXiv preprint arXiv:2510.01755, 2025.
M. Heusel, H. Ramsauer, Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
N. J. Higham. Accuracy and stability of numerical algorithms. SIAM, 2002.
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Ho, C. Saharia, W. Chan, J. Fleet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33 2022.
A. Jalal, M. Arvinte, G. Daras, E. Price, A. G. Dimakis, and J. Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34: -4954, 2021.
12

---

Published as a conference paper at ICLR 2026
T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 2017.
T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-basee generative models. Advances in neural information processing systems, 35:26565-26577 2022.
D. P. Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,694 2014.
F. Knoll, J. Zbontar, A. Sriram, M. J. Muckley, M. Bruno, A. Defazio, M. Parente, K. J. Geras, J. Katsnelson, H. Chandarana, et al. fastmri: A publicly available raw k-space and dicom dataset of knee images for accelerated image reconstruction using machine learning. Radiology: Artificial Intelligence, 2(1):e190007, 2020.
Y. Y. Lin, A. F. Gao, and K. L. Bouman. Imaging an evolving black hole by leveraging shared structure. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2475-2477 IEEE, 2024.
X. Liu, C. Gong, and Q. Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 0XXi:2209900003, 2022.
Z. Liu, Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
H. Lu, Q. Wu, and Y. Yu. Stochastic forward-backwartd deconvolution: Training diffusion models with finite noisy datasets. arXiv preprint arXiv:2502.00544. 2025.
W. Luo, T. Hu, S. Zhang, J. Sun, Z. Li, and Z. Zhang. Diff-Instruet: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36:76525-76546, 2023.
W. Luo, T. Hu, S. Zhang. J. Sun, Z. Li, and Z. Zhang. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
M. Lustig, D. Donoho, and M. Pauly. Sparse mri: The application of compressed sensing for rapid imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 18(6):112--119) 2007.
L. Mirsky. trace inequality of john von neumann. Monatshefte für mathematik, 79(4)3303-306 1975.
G. Nolet. breviary of seismic tomography. A breviary of seismic tomography, 2008
W. Peebles and Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-42000 2023.
B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Drcamfusion: Text-to-3d using 2d diffusion. arXiv preprint ar 2rXiv:2209. 14988888,,.4 2022.
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
N. Rawlinson, A. Fichtner, M. Sambridge, and M. K. Young. Seismic tomography and the assessment of uncertainty. Advances in geophysics, 55:1-76, 2014.
A. W. Reed, Kim, Anirudh, ct reconstruction from limited views with implicit neural representations and parametric motion fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2258 2288-226,,222 2021.
Mohan, K. Champley, J. Kang, and S. Jayasuriya. Dynamic
F. Roddier. Interferometric imaging in optical astronomy. Physics Reports, 170(2):97 1988.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 10684-10699 2022.
13

---

Published as a conference paper at ICLR 2026
O. Ronneberger, P. Fischer, and T. Brox. U-net:
Convolutional networks for biomedical image segmentation. Medical image computing and computer assisted cirvention-MMCCAN 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning, pages 2256-2260 2265. PMLR, 2015.
Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In International Confer- ence on Learning Representations, 2021a.
StigiarCHLP
Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Y. Song, J. Sohl- Sock-Diccst,in, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. ccsse-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b. URL
net/ www.uoodddddiiii 2RRHS
R. Tibrewala, T. Dutt, A. Tong, L. Ginocchio, M. B. Keerthivasan, S. H. Baete, S. Chopra, Y. W. Lui, D. K. Sodickson, H. Chandarana, al. Fastmri prostate: A publicly available, biparametric mri dataset to advance machine learning for prostate cancer imaging. arXiv preprint arXiv:2304.09254, 2023.
Vincent. connection between score matching and denoising autoencoders. Neural computation, 23())16661-167, 2011.
B. Wang and J. J. Vastola. The unreasonable effectiveness of gaussian score approximation for diffusion models and its applications. arXiv preprint arXiv:24/2.09776, 2024.
P. Wang, H. Zhang. Z. Zhang, S. Chen, Y. Ma, and Q Qu. Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409.02426, 2024a.
Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024b.
S. Xie, 7. Xiao, D. P. Kingma, T. Hou, Y. N. Wu, K. P. Murphy, T. Salimans, B. Poole, and R. Gao. EM distillation for one-step diffusion models. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL ptttps://openreveee www.ew.eeefooooo forum? id=rafvvthux)
Y. Xu, W. Nic, and A. Vahdat. One-step diffusion models with f-divergence distribution matching. arXiv preprint ar a50iv:2002.1 256,25502.,581,, 2025.
T. Yin, M. Gharbi, T. Park, R. Zhang, F. Shechtman, F. Durand, and B. Freeman. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 17:47455-47493 2024a.
T. Yin, M. Gharbi, R. Zhang, F. Shechtman, F. Durand, W. T. Freeman, and T. Park. One-step diffuuion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 131-6623, 2024b.
J. Zbontar, F. Knoll, A. Sriram, T. Murrell, Z. Huang, M. J. Muckley, A. Defazio, R. Stern, P. Johnson, M. Bruno, et al. fastmri: An open dataset and benchmarks for accelerated mri. arXiv preprint arXiv: 1.08839. 2018.
K. Zhang, W. Zuo. Y. Chen. D. Meng, and L. Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, 26(7):3142-3155 2017.
K. Zhang, W. Zuo, and L. Zhang. Fidnet: Toward a fast and flexible solution for enn-based image denoising. IEEE Transactions on Image Processing, 27(9):4008 4420446668662,,,,, 2018.
14

---

Published as a conference paper at ICLR 2026
Y. Zhang and O. Leong. Learning seffereence-f-coomee regularizers for inverse problems: A flexible framework with theoretical guarantees. arXiv preprint arXiv:2502.00240, 2025.
Y. Zhang, P. Yu, Y. Zhu, Y. Chang, F. Gao, Y. N. Wu, and O. Leong. Flow priors for linear inverse problems via iterative corrupted trajectory matching. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
forum?
id= 1USSI9
Y. Zhang. P. Yu, and N. Wu. Object-conditioned energy- beaee attention map alignment in text- lo-image diffusion models. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol, editors, Computer Vision ECCV 2024, pages 55-71, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-7294611
H. Zheng, Z. Wang, Yuan, G. Ning, P. He, Q. You, H. Yang, and M. Zhou. Learning stackable and skippable LEGO bricks for efficient, reconfigurable, and variable-resolution diffusion modeling. In The Twelfth International Conference on Learning Representations, 2024. URL https http://wwwe///oummiiiiieeooom 7 //pperevview net wwr.mmrrmmmmmmmmmmmmmmmmm (tm.ummmmiiggmmmdwxi11111
H. Zheng, W. Chu, B. Zhang, Z. Wu, A. Wang, B. T. Feng, C. Zou, Y. Sun, N. Kovachki, Z. E. Ross, et al. Inversebench: Benchmarking plug-and-play diffusion priors for inverse problems in physical sciences. arXiv preprini arXiv:2503. 1043, 2025.
M. Zhou, H. Zheng, Z. Wang, M. Yin, and H. Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024.
Y. Zhu, Z. Dou, H. Zheng, Zhang, Y. N. Wu, and R. Gao. Think twice before you Improving inverse problem solving with meme. arXiv preprint arXiv:2409.088: 08551, 2024.
15

---

Published as a conference paper at ICLR 2026
Qualitative Snapshots of Generated Results
16
Proximal FID
18
Training and Inference Efficiency
18
Additional Experiments
19
D.I More Metrics
19
Unknown
20
D33 More Conditional Inverse Problem Results
21
D.4
Random Noise Ablation
21
D.5
Choice of Distillation Loss
21
Discussions and Limitations
22
Theoretical Results and Proofs
23
Quantitative analysis in linear setting
23
Extensions of the theory in Section
28
General guarantees for distillation
35
Visualization for Overestimated Data Noise Level
39
Phase Variants of Pretraining Details
Standard Diffusion
39
H.2
Diffusion for Denoising
40
H.3
Diffusion for Random Inpainting Task
41
H
Diffusion Training for Fourier Space Inpainting
41
Phase II: Distillation
42
Implementation Details
42
Additional Qualitative Results
A QUALITATIVE SNAPSHOTS OF GENERATED RESULTS
In this section, present visual examples highlighting the quality of outputs produced by model across various tasks and corruption settings. A full version of qualitative examples are in Appendix
K
16

---

Published as a conference paper at ICLR 2026
Training Dataset
Our jone-tlpp genenation)
Triiing Datesee
Oura (one-dtep gereration)
FID: 130.33
FID: 8.28
FID: 51.51
FID. 5.42
Figure 6: Qualitative results for the Denoising task. Each pair shows the corrupted input and the generation output from our RSD at = 0.2. The left two panels are from FFHQ, while the right two from AFHQ-v2.
Dataset 0)
(b) RSD (ours)
(c) Dataset (a 0.2)
(d) RSD (ours)
Figure 7: Qualitative results for the Gaussian blur task. Each pair shows the corrupted input and the generation output from our RSD.
(a) Dataset 0)
(b) RSD (ours)
(c) Dataset (o 0.2)
(d) RSD (ours)
Figure 8: Qualitative results for the Super Resolution task. Each pair shows the corrupted input and the generation output from our RSD.
(a) Dataset (a 0)
(b) RSD (ours)
(c) Dataset (a 0.2)
(d) RSD (ours)
Figure 9: Qualitative results for the Random Inpainting task with missing probability (p = 0.9). Each pair shows the corrupted input and the generation output from RSD.
17

---

Published as a conference paper at ICLR 2026
(a) Dataset (p 0.6)
(b) RSD (ours)
(c) Dataset (p 0.8)
(d) RSD (ours)
Figure 10: Qualitative results for the Random Inpainting task with different missing probability
(p = 0.6, p 0.8). Each pair shows the corrupted input and the generation output from our RSD.
B PROXIMAL FID
When only corrupted data are available, model selection must proceed without clean references. In standard image generation, FID is the de facto criterion, but in setting computing FID against clean ground-truth images is infeasible.
Prior baselines Ambientt Diffusion Daras al. (2023a) and EM-Diffusion Bai al. 2024 report
FID scores yet do not specify how to perform model selection under corruption during training. To address this gap, introduce Proximal FID, medel-selection metric tailored to corrupted-data regimes. Concretely, generate 50k clean samples (a((9) from the current generator, corrupt them to match the training noise and operator- mmerator-yielding yielding
{A(x (0) + (6) 50k
fi=1'
and compute FID against the corrupted training set {- As shown on CIFAR-I0 in Fig. 5 Proximal FID tracks the true FID closely throughout distillation. Quantitatively, Table 10 shows that the model chosen by Proximal FID attains near-optimal true FID across datasets (e.g., 6.12 vs. best 6.08 FFHQ) for denoising task, and Table I1 general corruption task. We further visualize the dynamics on FFHQ, Cel.bA-HQ, and AFHQ- v2 in Fig. 1T and extend the analysis to multiple corruption operators in Fig. 12 Taken together, these results support Proximal FID practical and reliable proxy for model selection when clean data are unavailable.
PPHO
CokeA-HO
APHO
Ro
Rosiinal selecting prodeaa
Frcernal
Beut
selecting arodmall
Butte ooried
1007
290
Murber wwa
(a) FFH 0.2)
(b) CelebA-IIQ 0.2)
AFHQ- (a 0.2)
Figure 11: Across datasets: True FID vs. Proximal FID during distillation on FFHQ, CelebA-HQ. and AFHQ-v2 (denoising. 0 0.2). In all cases, Proximal FID reliably identifies checkpoints with near-optimal true FID.
C TRAINING AND INFERENCE EFFICIENCY
We use the denoising setting with a = 0.2 as a running example to quantify efficiency. Our approach improves not only accuracy but also end-to-end efficiency in both training and inference. All experiments were conducted on Linux system with 8xNVIDIA RTX A6000 GPUs unless otherwise stated.
Training. The additional distillation phase introduces only a minor overhead: the FID of the one-step generator rapidly decreases and surpasses the Teacher (Teacher-Trrunatee within 4 hours.
18

---

Published as a conference paper at ICLR 2026
Random Impairting p O
Super Resolution
(O moume
34.34
19425
2013
23.03
(a) Gaussian deblurring (a = 0.0)
(b) Random inpainting
(c) Super-resolution (x2, 0.0)
Figure 12: Across corruption operators: True FID Proximal FID during training for Gaussian deblurring, random inpainting, and super-resolution. Proximal FID consistently tracks the true FID and supports reliable model selection across operators.
Table 11:
Comparison between the best true FID and the FID of the model selected by proximal FID for the general corruption task.
Noise (o)
Task
Best FID)
FII) Selected by Proximal FID
Gaussian Deblurring
31.90
35.68
0.0
Random Inpainting (p 0.9)
16.79
20.59
Super Resolution (x2)
12.99
14.66
Gaussian Deblurring
76.98
88.29
0 0.2
Random Inpainting (p = 0.9)
79.48
83.98
Super Resolution (x2)
22.00
27.42
Table 12: Training and inference efficiency of our method. During training, the additional distillation phase introduces only minor overhead, as FID decreases rapidly and surpasses the teacher diffusion model, Teacher-Trun Truncated, within just hours. For inference, our one-step generator enables the generation of 50k images in only 20 seconds, achieving 30x speedup.
Distillation Time Achieve the Same FID
Time Generate 50k Images
Datasets
Pretraining Time
Teacher-Full
Teacher- Truncated
Best
Diffusion
RSD
CIFAR-10
minutes
-3 hours
-3 days
10 minutes
seconds
FFHQ
-2 days
56 minutes
<3 hours
~9 hours
CclebA-HQ
34 minutes
-2 hours
-13 hours
minutes
seconds
AFTIQ-v2
80 minutes
-3 hours
~13 hours
Representative wall-clock times across datasets are summarized in Table employ early stopping when the validation FID begins diverge.
12
During distillation, we
Inference. Our one-step generator produces 50k images in ~20s on 4xNVIDIA RTX A6000 GPUs with batch size 1024, compared to ~~10 min for the diffusion teacher -yielding a 30x speedup. Inference wall-clock measurements are reported in the rightmost columns Table 12
D ADDITIONAL EXPERIMENTS
D.
MORE METRICS
The detailed results show that our distilled RSD generator achieves both strong density modeling (low FID) and robust mode coverage (high Recall).
19

---

Published as a conference paper at ICLR 2026
Table 13: Comparison of RSD with teacher diffusion and baselines on CIFAR-10. * *Metries for
DDPM and Rectified Flow (ODE) are copied from the Rectified Flow Liu et al. 2022 paper's
Table
Method
FID (L)
(t)
Precision (t)
Recall (t)
KID (L)
Teacher
0.2
12.21
8.312
0.595
0.416
0.00593
RSD
0.4
21.63
7.934
0.536
0.384
0.01270
RSD
0.2
4.77
9.165
0.650
0.564
0.00252
RSD
0.1
3.98
9.346
0.643
0.578
0.00157
DDPM
0.0
3.21
9.46
N/A
0.57
N/A
Rectified Flow (ODE)
0.0
2.58
9.60
N/A
0.57
N/A
Mismatched Corruption Levels
175
underest mation
overastimation
diffusian
125
75
50
100
200
300
Figure 13: Mismatched Corruption Levels during Distillation. Our method remains robust even when the assumed corruption level is mismatched during distillation.
D.2 UNKNOWN a.
When unavailable training time, we adopt simple strategy: (1) estimate per-image noise level using the off-the she- estimator implemented via sk Kimage...... restorat ietorratioon. est imate_msee.. s sima Donoho and Johnstone (1994); (2) for reliability, select the maximum estimated over small calibration set. Concretely, we sample 100 noisy CIFAR-10 10 images, compute ò per image across 100 independent trials, and use the trial-wise maximum as the working As shown Table [4 this deliberately slightly overestimales the noise, a regime where RSD remains stable (see experiments in Table I5) and, in line with blind denoising results Zhang et al. 20018 2017); Dabov et al. (2007). helps avoid umder-regularization.
Table 14: Unknown noise: true Ø vs. ē on CIFAR-10.
Table 15: Misspecification study CIFAR-10. RSD robust to slight overestimation when the true = 0.2.
95% CI
Teacher-Full
Teacher- Trune.
RSD
0.05
0.07
[0.073. 0.074]
0.15 0.15 (under)
80.60
49,78
103.55
0.10
0.12
[0.119, 0.121]
0.20 (true)
60.73
12.21
4.77
0.22
10.215,0.200 0.216]
0.30
0.31
0.313)
è 0.25 (over)
42.99
88.11
16.07
Effect of misspecified o. To isolate the impact of noise misspecification, we fix the ground-truth level at 0.2 on CIFAR-10 and evaluate three settings: (i) underestimation. à = 0.15: (ii) correct, 0.20; and (iii) overestimation, Phase pretraining (noisy corruption objective) and Phase II distillation. As shown in Table 15 and Fig. 13 RSD attains its best accuracy under correct specification and remains competitive under mild overestimation, whereas underestimation is substantially more harmful harnful-echoing observations in blind denoising Daboy et al.] (2007); Zhang et al. (2018) 2017). Thus, when or unknown, slight overestimation robust practical choice, consistent with the strategy shown in Table 14 A
0.25. The same is used consistently in both
20

---

Published as a conference paper at ICLR 2026
Table 16: x2 super-resolution on CelebA-HQ (100 images). We compare a pseudo-inverse baseline (AT y), EM-Diffusion (few-shot) with DPS, Teacher Diffusion with DPS, and our RSD prior with latent optimization (Eq. 66p. Higher PSNR/SSIM and lower LPIPS/FID are better.
Method
PSNR t
SSIM 1
LPIPS 4
Prior FID
ATy
24.065
0.816
0.125
23.94
EM--iffusion (few-shot, DPS)
25.678
0.839
0,056
58.99
Teacher Diffusion (DPS)
23.810
0.778
0.056
23.28
RSD (ours, latent opt.)
27.803
0.909
0.047
12.99
complementary 2D toy example in Appendix corroborates this conclusion. We further include an ablation where the noise level is drawn from a distribution, i.c., o ~ p(o). Our method remains robust under this setting well; see Appendix D.4 for details.
D.3 MORE CONDITIONAL INVERSE PROBLEM RESULTS
Our primary goal is to learn strong generative prior solely from corrupted data. Once such prior obtained via RSD, agnostic the forward operator, enabling conditional generation under arbitrary measurement models. A natural approach with one-step generator is to solve
min ||A(Go(=)) vl|
(6)
which enforces data consistency through the measurement process.
Setup. Beyond the denoising task discussed in Section 4.5 we consider an additional conditional inverse problem. Specifically, evaluate a conditional inverse setting where A corresponds to a x2 down-sampling operator. We conduct experiments 100 CelebA-HQ images and compare RSD against EM-Diffusion (few-shot) and the Teacher Diffusion prior. For fair comparison, RSD optimizes Eq. 6 for 1000 steps using Adam with learning rate of 0.1, while both EM-Diffusion and Teacher Diffusion adopt DPS Chung et al. (2022), a classical solver for diffusion priors, with 1000 diffusion sampling steps. LPIPS is computed using AlexNet.
the generator trained on the x2 super-resolie superrresolution task and
Results. Table reports PSNR, SSIM, LPIPS, and the prior's 's FID. RSD attains the best perfor- mance across all metrics (e.g., PSNR 27.803, LPIPS = 0.047), and learned prior achieves strong FID of 12.99, highlighting the benefit of high-quality priors for conditional generation.
D.4 RANDOM NOISE ABLATION
We conducted an additional experiment in a more challenging setting where each sample is corrupted with different noise level. Specifically, we adopt the corruption model:
y = x + ae, o p(a)
where p(o) Uniform[0.155 0.25]. This naturally introduces variability across samples both in terms of noise level and corruption behavior.
To accommodate this setup, we made minimal modification Algorithm D replacing
Deeplacig Lines 6 from the fixed-noise setting (ğ = Tg + ac) to the sample-dependent corruption (y = 2g + ac., ~ p(o)).
With this simple change, RSD can be applied directly heterogeneous corruption settings.
We tested this on the CIFAR-I0 dataset, and the results are reported below in Table 17
D.5 CHOICE OF DISTILLATION Loss
In our experiments (Sec. 4 the distillation phase adopts the SiD loss (Eq. 5) by default. Other distillation objectives also applicable-e.. e.g., KL-based variants such SDSS Poole al. (2022).
21

---

Published as a conference paper at ICLR 2026
Table 17: Comparison of FID scores. Lower is better.
Method
FID
Teacher-Full
50.29
Teacher- Trunccted (a 0.25)
16.21
Teacher- Trunccted (a = 0.15)
11.80
RSD (ours)
5.67
Table 18: Ablation of distillation losses on CIFAR-10 denoising (a E (0.1,0.2,0.4) FID)
Method
0.1
0 = 0.2
= 0.4
SDS
>> 200
200
200
DMD
12.52 +0.04
7.48
+0.06
30.09
=0.23
SiD
3,98 10.04
10.03
21.63
10.03
DMD Yin et al. (024b) (also referred to as Diff-Instruct Luo et al. (2023 or VSD Wang et al. (2024b)), and SiD Zhou al. (2024). For completeness, report their generator-level results below and defer implementation details (e.g., time-scheduling and hyperparameters) to the original papers.
We ablate this design choice on the CIFAR-10 denoising task; results are summarized in Table 18
Notes. All distillation variants use the default hyperparameters from their official repositories; we did not tune berraarameeers. The relatively weaker performance D-SDS and D-DMD Table may therefore reflect suboptimal default settings for the corrupted-data regime rather than intrinsic limitations the losses. Empirically, SiD robust under defaults and already yields strong results, hence our choice to use SiD for all main experiments (Sec. 4. Importantly, the distillation stage in our framework modular and can be replaced by more advanced objectives; future work may close the gap even surpass Sil)-via principled hyperparameter tuning and improved losses.
E DISCUSSIONS AND LIMITATIONS
We demonstrate how to use the distilled generator as a lcarned prior for inverse problems in Section 4.7 and Appendix D.3] Concretely, given measurements A(x) + noise, we recover plausible x by optimizing over the latent so that the synthesized sample matches the observations, e.g.,
Solving inverse problems.
min |A(Ga(=)) - v (optionally with regularization or priors on 2).
broader treatment- iiciineentincluding including principled conditioning, data-consistency guidance, and plug-and- play/score-basee solvers-is a promising direction Chung al. (2022) Zhang et al. (2024); Zhu et al. (2024); Zhang and Leong (2025); Chen et al. (2024b), with applications in scientific and engineering domains where reconstructing clean signals from measurements critical.
Applications in scientific discovery. Our approach is particularly well-suited for scientific discov- cry, where clean observational data are often scarce fundamentally unobtainable. Extending our method to datasets across diverse scientific domains is promising avenue for future research. For instance, ground-trute black hole images are inherently unobservable, yet large collections of cor- rupted telescope measurements are available, as demonstrated by the Event Horizon Telescope (EHT) observations Akiyymma al. (2019). EHT relies on Very Long Baseline Interferometry (VLBI), where the measurement process modeled as a 2D Fourier transform of the sky brightness distribution. Specifically, the forward model can be expressed as
// [(x, (xx,y)e y)e 2n((ux+uy) drdy.
V(u, v)
as in Eq. 2 of Akiyama et al. Vt ((.,)) gigle :(0) o I (2) + ka.,)))
2019), or in practice as
described in Eq. 16 of Zheng al. 20055
22

---

Published as a conference paper at ICLR 2026
Unknown corruption operator. We first note that in many scientific applications such as backhole imaging and multi-coil MRI, the corruption operator is known. settings where the true corruption operator is unknown, off-the-shelf estimators
provide usable approximations of the corruption process, as illustrated in Section 4.5 Crucially, our method does not require an explicit dlosed-form specification of the operator: it only assumes access to its forward propagation (i.e., the ability to apply the corruption). This black-box requirement confers a key advantage- our approach can recover salient properties of the underlying data distribution without explicit knowledge of how the corruption is parameterized.
F THEORETICAL RESULTS AND PROOFS
We provide theoretical insights to support our empirical results. In Section we first focus on a simpler, linear Gaussian setting where we explicitly analyze the optimization landscape of score distillation given noisy samples, yielding quantitative error bounds and characterizations of global minimizers. Then, we offer several extensions of this theory in Section
F.2 including handling linear corruption and multiple noise levels. Finally, in Section F.3 we provide a more general analysis asking when can the distilled student achieve a strictly smaller Fisher divergence to the clean distribution than the teacher. This regime will operate under distributional assumptions the data and corruption along with capacity and optimization assumptions on the teacher and generator.
F.1 QUANTITATIVE ANALYSIS IN A LINEAR SETTING
To begin, we first analyze the performance of score distillation a stylized setting where the underlying data distribution is Gaussian. In particular, we will assume that our data follows low-rank linear model for our analysis.
Assumption (Linear Low-Rank Rank Data Distribution). Suppose our underlying data distribution is given by a low-rank linear model x = Ez ~ px and 2 ~ N00, I,), where E E Raxr with < d and with orthonormal columns (i.e., , E E I,).
Assumption iis equivalent px NO, EE*). For fixed corruption noise level 0, consider the setting we only have access to the noisy distribution In other words, PY,o == (O, EET + o² La). In our setting we assume that we have perfectly learned the noisy score:
+ are, where x ~ px and e ~ (O,
Assumption 2 (Perfect Score Estimation). Suppose we can estimate the score function of corrupted data y perfectly:
V VlogPy.a = (EE" + +o²la) '') -'x.
Our goal is to distill this distribution into a distribution PG, == (Go) Ia)) given by the push- forward of (0, Ia) by a generative network Go Rd + RRd To model U-Net Ronneberger et al. (2015) style architecture with bottleneck structure, we assume Go satisfies the following low-rank linear structure detailed in Assumption
Assumption (Low-Rank Linear Generator). Assume the generator is a low-rank linear mapping, where Gy is parameterized by 0 (U, where U, E Boxr with r <, d and has the form:
Co(z):=UVTZ
Note that Ge induces degenerate low-rank Gaussian distribution PG, := UOTUUT) Con- sider a bounded noise schedule (0.) S [omin, mmax] for some 0 < Jmin Omax < 00 and perturbed data points It = x + deC where e NN Ia) and x PG.- Then It PG. := (0 UVTVUT + Ta). To distill the noisy distribution, we minimize the score-based loss (or Fisher divergence) as in Zhou et al. (2024):
C()) = En..Uni((,,))
|1100,0 (xx) log (a.)
Here, Sp,0. (=) == -1=-(EEE (EEE + (a² + Ix. Note this objective similar to Eq. 5 but with the real score in place of the fake score. This is also considered the idealized distillation loss (see Eq.
23

---

Published as a conference paper at ICLR 2026
(8) in Zhou al. (2024)). In Theorem 1 we show that minimizing Eq. 79..7vver certain family of non non-degenerate parameters finds a distilled distribution with smaller Wasserstein-2 distance to the underlying clean distribution.
Theorem 1. Fix a > 0. Under Assumptions and consider the family of parameters 0 = =U,, V)
such that
0 0 (--)
03.
For any bounded noise schedule (o,) S moin max) the global minimizers of L over O denoted
by 0 = (U*, ); satisfy the following:
U EQ some orthogonal matrix Q and (V²) = =1+
(8)
For any such 0r the induced generator distribution pG = =N (1 + o²) satisfies
W (pGag pP) (P - (d- < W (pv,..Px).
Discussion. This result shows that global minimizers of the distillation loss a family of "non-degenerate" parameters induces a distribution close to the ground truth. Moreover, we can precisely quantify the distance the underlying distribution due the fact that all distributions are now Gaussians and the Wasserstein-2 distance has a closed form. We further explore what the unconstrained minimizers are Theorem B for the rank-one case.
Regarding our assumptions, we note that making either Gaussian or potentially more complex Gaussian mixture model assumptions common the generative modeling literature Chen al.) (2023); Cui et al.) (2023); Wang and Vastola (2024). We further note that this result focuses on setting where the underlying generator has low-rank structure. While it is common to make simplifying assumptions on the network architecture to understand score-based models Chen et al. (2023 2024a), there also recent work Wang al. (024a) that has shown when trained on data of low intrinsic dimensionality, score-based models can exhibit low-rank structures. Empirically, we find that netuol--etwor--aaaee bassed distilled models can find such low-dimensional structures through noisy data. An interesting future direction of this work is to understand the influence of neural- sawork-bassd parameterizations of the score function along with analyzing the fake score setting.
ChodelsCChen
Before dive into the proof, we provide the following lemmas.
Lemma 1. [Generalized Woodbury Matrix Identity Higham (2022))
Given invertible square matrix A E Raxn along with matrices U Ra and V E RR define the perturbed matrix: B = A+ UV. / If (I& + +VA V-- ) is invertible, then the inverse of B given by:
B- - A U + -V-
Lemma 2. The Wasserstein-2 distance between two mean-zero Gaussians (0 11) and N E2) whose covariance matrices commute, i.e., ΣnΣ² = ΣzE,, is given by
d
(1) E2)
Σ ( +EZ) - 2 (
i=1
Lemma 3 Mirsky 1975)). Suppose A and B are d d complex matrices with singular values (A) > 02 (2( > ... (A(A) A)= > 0 and 01 (B) 2 (2(B) > ... oa(B) > 0, respectively. Then
[tr(AB)
Σ (A)o
Lemma 4. Let E E Rdxr with r < d have orthonormal columns and E E RRxx be symmetric positive definite. Then
argmax UrEE = (EQ orthogonal). UT=I,
Proof of Lemma any feasible U.
Observe that by the von Neumann trace inequality (Lemma 3 we have that for
(EEE"UUUU)) = tr(U" EE < U EE" & Σ A(E
24

---

Published as a conference paper at ICLR 2026
where the last line used the fact that \U UETU) & 1 for each i [r]. Hence, to maximize
++ tr(EE"UEU") over {U : U = 1,). we want U to satisfy tr(EE" =
We claim that this occurs and only if U = EQ for some orthogonal Q. U = EQ, then
(U*)"EEEU = Q" EEE EQ I so
-
tr = -
Σ (x))
For the other direction, suppose U maximizes the objective. Then
((("")"EE'U"'>= =t()) = tr ((UU)) -,) I,)E) =0
Set Q = ETU" Note that the eigenvalues of Q"Q are bounded by so Q² I, is negative definnefe while Σ is positive definite. But if tr (Q'(QQ' Q I,)E) O, by positive definiteness of E,
we must have QTQ Q - I, = 0, i.e. Q"Q== = I,. This means Q is orthogonal. Since Q is orthogonal and Q E
U EQ, desired.
Lemma 5. Fix
and consider noise schedule or 0 for (0.1) such that (ot) C
[omin maxx for some O < mmin < 0max <0..
co. Define the function fo : (0, 00) - IR by
u [o2 of +12
u
fo == (ew-nif(0..)
oj of)
Then fo is strictly convex and has unique minimizer u² = o² + I which is the unique solution to the equation
1 [o2 - 1)3 Be-Uni((O,.)
(n-Uni((),1)
(u- + of
Proof of Lemma First, note that the conditions on o, ensure that all of the following expectations are finite. By direct calculation, we have the derivatives of fo are
1 (u) = E, (02 of
- E.
and 1 ("(()) = E, (o?
u)²
Hence (u) > O for all u > so fo is strictly convex. To find its minimizer u², ,setting the derivative equal to yields w" must satisfy
Eg
0o - 1)2 E,
(o + )²
Note that the point =² = 1 + 0² clearly satisfies the critical point equation. Uniqueness follows due strict convexity.
F.1.1 PROOF OF THEOREM 1I
We break down the proof of Theorem into three key steps. First, we show that minimizing the objective (Eq. 7 is equivalent to minimizing simpler objective. Then, we show that we can derive exact analytical expressions for the global minimizers of this simpler objective, which are then global minimizers of the original score-based loss. Finally, we will directly compute the Wasserstein distance between our learned distilled distribution to the clean distribution and compare this to the noisy distribution.
Reduction of objective function: For o, > 0, define Po. N' "'O,UV"VU" + o2 and $0.0. (z) = -EEE" + (a² + dd) 1z. For the proof, we will assume our parameters =
(U, V) E θ so that UTU = I, and V V > We consider minimizing the loss
[()) = Ex-Unit() Br [|| || (xx) - logga (Ix [12]
25

---

Published as a conference paper at ICLR 2026
For E (0, 1), consider the inner expectation of the loss
() == Euirpa ||50.or (xt) - V log (x1 11]]
For notational convenience, set Egg == EET + (o² + oa and Eg, == UV + la Then 30,0r (x) ;= ->ig,' and V log P (x) == -2 x. First, recall that for 2, Pč, and any matrix 2.
Ey Ex-re [|Ex+ [3] = | 1|?-- Using this, we can compute the loss as follows:
(0) Emp ||(2 1
|1 - Ez.t |
11/2111 >|
tr (Fer (E Σ (Z Eg,1
tr (Ear(Zx) - Egh ( - Eggt))
= tr (Eog - ( - -Φ/
= tr (Zo,2 - Eo -ot 2 Mar + -ي.T
= tr (g)) - tr EggFoof -يo - tr (Σ + tr (I
= -r (2g))0,) - 2tr (Σai)) + tr -
=: Co, + (2,) + tr [Σ,
Using Lemma 2 it is straightforward to see that
1
1
Edl 0² o Ia oz oī + 1)
EE and
0 = o, 1- (1 -1 UT
Hence the third term in L, is gy
tr( tr(Ep)) = tr (o, 2 or AU ((VTV)-+ + o +1,2 21,) UT) Co, -o.-0, Ar ((ITV)-++ o0, 21,
"1)
where we used the cyelic property of the trace and U I, in the last equality. For the second term, let 83? == a² + of and To, == 3
Then we have by direct computation,
tr (on) tr (13, Ia - EE,
To,EE" (UV +otla)
= -t (T3 1a - 23, 2 EE, =E, (UVT +of1a))
=tr VU - o,, प. - (2o,- -20 FF"UVTVU")
-23 of o
+ (3e, - 23, (8.)) VVT = =C. 3r (r((v V) + (22. - 2,, 23-2%o.) Vo.. uV"EE"
=: 1,
where we used the cyclic property of trace and orthogonality of U in the final line. Combining the above displays, we get that there exists a constant Co.a. == Ca. + Co, + Caa such that
(0) Co.o. + (2P10++) +12
2
.U(BEEUVTVV")
31 + 1)
+ + - o, Air (((VV)) + o (.21,)=1 1)
Co.. + Bt(U, + Rv(V)
26

---

Published as a conference paper at ICLR 2026
where we have defined the quantities
V) = B7 +12 (3 +1) (VTEE" and
R+(V)== 3r tr - tt to, 21)
Recalling the definition of [()). we have that
((0) Unif((() [[] E(-Uni((O,)) [Co,, + BU + R)),
Hence we have the equivalence
argmin [(0) argmin Ee-Unit().) [B] Un (V))) lee
dce
Form of minimizers: We use the shorthand notation E+H- = En First, note that can first minimize [,[B,UU, V) over feasible U. But note that
E, [B V) = E.
2 B1 + 1)
((EEE
(3, 1)2
since for any 6, +1 (EETUVVVUT) Taking Σ VTV in Lemma 4 we have that the minimizer of E, (U, V)] is
[p] +1) Hence minimizing E, [U, V)] is equivalent to maximizing
given by
U EQ for some orthogonal Q.
Moreover, the proof Lemma 4 shows that ((EEETVVVV( - This gives
2
Eu[Bi(U*, V)] E.
(B708 +12 1)2
B1(3? + 1) Vr(v")).
In summary, we now must minimize the following with respect to invertible V:
E,[B.(U +] + [ E,
B2 + 1) + tr(vTT)
B2 + 1)2
31
Eg or *tr
((1 + o, II)) -11
2,,
32 +1 -1
(r(v" - Es 10-4t1 ((121))++0:221, [
E
34
E,
B3
2
tr(vTV)- - -Ee 1
-0 o, (,, [1]
87 +
E, (8) + 1)2
o tr(vTV) E, oo (1 [
o,,,,,)
where in the second equality, completed the square.
We now claim that EE, [B, (U", V)] + E, (R)) solely depends on the eigenvalues of VTV. In particular, for invertible V. note that VTV 0 it admits the decomposition VTV QAQT where Q" Q QQ" = I, and a diagonal matrix with positive entries AG = VVVVV) > Hence tr(VVV) tr(@AQ") tr(Q" QA) tr(A) Ei= (VVVV)). Likewise, have using
27

---

Published as a conference paper at ICLR 2026
the orthogonality of Q that for any 0,
tr ((IVVV)- + -1, 21,)==) -'(((0Q))~++ 0 21,)
=tr -tr((Q-100 3t +02 ) -11
=tr -tr((QAA) + 11, ( ( (t
-ir -I((Q(A)+ (( Q²)
tr ((A-++ + -
X(VTV)- +
V((VV)) 62
(v(VVV) +
i=1
In sum, the final objective is a particular function of the eigenvalues of VTV:
AV(VV))
A((VV))
E, B, =
Σ
E
(3 1)²
(VVVVV
X((VVV)
X(VVV)
Et
(02 (02+0l +12 (((A((VVV + of)
fo (V((TV))
In Lemma 5 we show that the function F> fo (u) is strictly convex (0, 00) with unique minimizer at 1 + o². Thus V + B(U V) + R(V) ) for invertible V is minimized when the gram matrix of V has equal eigenvalues >c((Vz))Vv)) are the same, by the Spectral Theorem, we must have that (V = (1 + IT)
1 o² for all i E [r]. Since all of its eigenvalues
Wasserstein bound: We now show the Wasserstein error bound. Note that 0 (U", VVa) induces the distribution PGag defined by
x Go (z), 2 ~ NO, Ia) - x ~ pGo; == =
Then by Lemma
we have
W2(PY.,,Px) r 0² -1 の² +
W2 ,P) -r 1 -1+02+1-2/1++7+
This gives
W2 PP) px) W2 px) rd- (2
F.2 EXTENSIONS OF THE THEORY IN SECTION
We now discuss three extensions of Theorem T we allow for additional corruption in y. 2) characterize the full optimization landscape in the rank-one case, and 3) analyze the global minimizers when we may have varying noise levels the training data.
F.2.1 ADDITIONAL MEASUREMENT CORRUPTION
We will now consider the case when the data is not simply noisy, but also exhibits more general
corruption. For a fixed corruption noise level a > O, consider the setting we only have access to the noisy distribution y A(x)) ac, where ~ (O I,m) and A(x) Ax with A R is linear corruption operator. By our assumption on px (see Assumption [ and the noise, PY, := (0, AEET AT + I²I)). In order to get rid of error in estimating score, we assume that we have perfectly learned the noisy score:
28

---

Published as a conference paper at ICLR 2026
Assumption 4 (Perfect Score Estimation with A). Suppose A E Rmxd with rank(A) and we can estimate the score function of corrupted data y perfectly:
a (x) == V Py (x) = - AT I
Our goal is to match the corrupted noise distribution's score with the score of pc, under Assumption corrupted by A over series of noise schedules (ot), which is given by
Pe (y) = NN AUV A" + of Im).
To distill the noisy distribution, we minimize the score- boore-based loss (or Fisher divergence) as in Zhou al. (2024):
( == 1) [1 [1 - V log (v1)1
(10)
We show that we can characterize the global minimizers of this loss, which correspond noise- dependent scaling of the true eigenspace of px plus perturbations in the kernel of A. If we penalize the norm of our solution, we can nearly recover the true data distribution Wasserstein-2 distance up to the noise in our measurements. We consider the rank-r = 1 case for simplicity.
Theorem 2. Fix a > O and consider e E R2d with unit norm and Ae f Under Assumptions /
and and any bounded noise schedule (at) S [o m the of global minimizers of the loss 10 under the parameterization u Y f(u) = (u, u/ [ull]) given by
+.
e + ker(A).
Aell
I E Imi(A"), we have that 0. = (u())) with the minimum norm solution u, E argmin ||/|| satisfies
(2x.XxPa..) (V++ o²
2
| Aeel 10-111))
Discussion. Theorem 2 aims to provide quantitative bound of the Wasserstein distance between the distribution learned via distillation and the target clean distribution. To do this, we characterize the global minimizers of the loss, which correspond to scalings of the true principal component e plus perturbations in the kernel of A. The scaling depends on an effective signal-to-noise ratio SNR = | /a² Note that terms involving perturbations in the kernel of A are expected since the corruption compresses the data, leaving many plausible images that could give rise to the same measurements (a fundamental part of the ill-posedness in inverse problems). Furthermore, we show that if we penalize the norm of our parameter, the Wasserstein distance simplifies into a more interpretable quantity. A more general bound for all elements in the set of global minimizers is shown at the end of the proof. The intuition for the Theorem is that if we encourage finding a "simple" model (i.e., one with low-norm), the Wasserstein distance decreases and is effectively inversely proportional to the SN R. The distance between our learned distilled distribution and the true distribution goes to zero whenever 1) the noise goes to zero or the signal strength increases. This result recovers the rank-1 version of Theorem 2 when A = 1, showing that the learned distilled distribution's 's Wasserstein distance to the true distribution is less than the noisy distribution's 's Wasserstein distance the true distribution subtracted by a factor of (d - 1--))ee 1) showing clear improvement in distribution learning. Finally, we note that the condition sensing literature on stable recovery via the construction of dual certificates Foucart and Rauhut (2013).
Im (A²)) we use akin to assumptions in the compressed
To prove the Theorem, we first show that the objective under the parameterization 0 (u, //]ll simplifies into a form that we directly analyze.
Lemma 6. Consider the setting of Theorem 2 For i [0, let at of, ce at + 0², and
2e.t||AAe Vh [elfpp Then the objective L()) with 0 = (u, (u,u//lu) w/|lu]]) satisfies the following: there
exists a constant C' independent of E Rdd such that
(()) C I(u)
29

---

Published as a conference paper at ICLR 2026
where
Aull (a + ||
T(u) = Unif(00)
b All Te(e=AA A² Au)²
Proof of Lemma 0
Consider the loss function (()) under the parameterization (u, u/ u//lul)) for
+ 0:
|| E(o.A)
[(0) := En- yl
where
Exe,A = Aee
"eefAA A + cl and E,,0,1 == Aun A az amlm-
Using a similar reduction in the proof of Theorem we get that the above loss equals
(()) En-UUirfoo. [Co,]] Eu-Umir(o.1] tr (
tr (Ird.A)
where Ca.t is constant that depends only time and and on 0. Note that using the Woodbury matrix identity in Lemma]
we get
2a + ||AE|
LeeA (Aee" AT
+ AA = =c I
c7(c + |ee Aee AT
which implies
2c, + ||Ae||
2-2 Le, 21 =
(c, + |e||a)] Ace T AT)
Auu AT
20 + ||ee|
+ c am at (c + | 12)2 Ace AT
q Aun AT - Aeee T A Aun AT + o amm Aee AT.
Also, we have that
Aun A
F., = (Auu² AT + 02 a.Tm) ar I as (a, + ||u
Thus computing the trace, taking expectation, and collecting terms that only involve u, see that there exists a constant C' depending on (at), a, A, and e (independent of u) such that
tr(Auu" A²')
((0) C + E, for A²) E, En]n]tr(Aed A Auxu" A²) E,
ar(a, ||Au[(?)
||ul|
C' + E.l. |||Au| Eelml(e ATeTAAAul Au)² EE, an ||uul)))
=: C L(u).
Proof of Theorem
Note that Lemma 6 shows that we have the simpler formula
(()) C L(u)
where
1 |uul 11(6= A= Au)² ae + || 1
Au
L(u) E,
c
For notational simplicity, let G A, g(u) Tu, h(u) T (g( g/(a, + +)). and 7/- = (2c, + [[Ael/))((((cc + ||| |
Then our objective results in
I(u) : - E, [n])((u))² Ee(de(o(u))) =: (g((4) ijh(u) 2 (g())(
30

---

Published as a conference paper at ICLR 2026
We claim that any global minimizer of L is of the form u, = q for some constant A, to be defined and q E ker(A)) First, note that we can lower bound L(u) by another function v (g(u)) as follows: since G is PSD, note that we have the generalized Cauchy Schwarz inequality:
h(u)² (e" Gu)² (e" Ge)"G)) Gu) g(e)g(u).
Moreover, this holds with equality if and only if Au and Ae are collinear, i.e., Au = AAe or equivalently u = de for E ker(A). Then we get the lower bound
L(w) = ög(w) ih(u)² (x())((())) > (č - AellAel|))((v
(x(oi(((u))] =: V(((u))
where we have defined the new function (g) = (2 Ae 12 }gg (x( for g 0. Since g(u)= 2 for every u, we have that
L(u) min (g). 920
We first analyze the minimizers of ve and then construct u. such that L(u.) (g.). First let (2 (2-1) Aell which is given by
(2c, + | |
A E, [ 2 - Th [ee| = E,
C7
Gla + Ae 12)2
We claim that A > 0. Indeed, note that for any > 0,
g(c, + g) g(2c + g)g
g(2c, + g) + G c
(o: + g]² G + (2e + +gg
c7 + (20, + g)g
(a. + g)²
Applying this to A with g = ||Ac|| 12 yields
(2c, + | de|| ||] Aell?
A = E
c
cg + ||]22)
(2c, + || Ael
E,
1
c
(ce + ||)2
1
E,
1- 1 +
(cr + ||ee
E,
>0
(a: ||42
as desired. This gives
(((g) Ag (x(r
Observe that
0
9
= L''' (a + azg)2 dt A E2
af
1
(g) =
0g J d; + agg
(ar + 9)
and
V²(g) = - D/OgE, + +)) -1 = 2E, (at + g) 3 0 4g.
Hence the function T is strictly convex. Moreover, sign analysis reveals that
1 a3
1
V0 A - F. 1/4 R2, (c + |
a3 <0
(02 + at + 2)
while for > d² + || we have
(g) = E,
(o2 + + ||e||z) (a, + g)2 >0
so there must exist unique root g- such that V'(g)) = 0. In fact, go = o² A|el| achieves V V'(g.) O. Finally, have that
L(u)= min (g) (g.)) A(o2 ||Ae|| E0 + ||AE|3)].
220
31

---

Published as a conference paper at ICLR 2026
We claim that g. achieves L(u,) (g.).
Indeed, note that g(u.)
(u.)² ( Gu, g-/llAe||ee Ge g-/1|Ae||2 ||Aee| 9+ and
M(u.)² (e" Gu.)²
|| Ae Ge) 9- Aell?
9+
Hence
M(.)) (g(umz) (ij(uu) (o
Ög. n1. 11 Ael2
R. (9-))
(e - Ail (2 - [x(0r(g+)]
Ag. E. lobe (um))
(gg)).
The only other cases include h. = u, + v for u E ker(A) since these are precisely the equality cases of the generalized Cauchy- -Shhwarz inequality. Note that for such h.,
g(h,) = Gh, = (u, + b)" G(u, + v) = ( + 2v² Gu, + u² Gv = 9+ and h(u, + v) = e² G(u, + v) = h(u.). These cases are also global minimizers, which gives us the total set of globally optimal solutions:
σ ||Ae|
+. = +
of + ker
|
Finally, we consider the Wasserstein distance bound. Note that for any u F O, we have that
(uni") 1/200 "c" (ooc"""("")") (uuu") 1/2
(2 cc""),N((0, un²")) = tr ee" + +wi"
un² |uul| ee uni
1/2
1 + |/ul| 2tr
(ef u)²
1/27
1 ||u]|3 2tr
||ul12 uuee""
1e vl
1 ||ul12 2tr
| uw
+1 ||]l| - 2 2/e u/.
In our case, note that u. = =q Age where q E ker(A) (A) and X |1+02/1
Acl2 > Since
e E Im(A"), e" ==-=00 0. Hence le"uu.| = X, and ||u.|22 ||g|| + X² where we used ||ll = 1 both
equalities. Using the previous result and these properties of u.. see that for any u. E O..
WV (e ). u.4)))
l|q// X 22, ||a/|2 + (X, 1)2.
The minimum norm element of θ, is precisely given by either A.e or -e,.. This because e and
are orthogonal for any E ker(A), so for any E O., ||u. \2|le|12 + |//l2 > X² | Taking q = minimizes the norm of u, E O..
F.2.2 STRICT SADDLE PROPERTY
natural question whether the objective landscape well-behaved for first-order optimization algorithms. We now analyze the rank-one case and show that the objective landscape has a strict saddle property, namely that all critical points are either global minimizers or strict saddles, i.e., points for which the gradient is zero, but the Hessian exhibits negative eigenvalue, hence descent direction.
Theorem 3. Consider the setting of Theorem 1 with and E = E Rd with unit norm. Then the objective C under the parameterization ((u) = (u, //|lu]]) satisfies the following: the set of critical points for F is precisely given by
Ω = IVI+o', - uu where Ω fu _Rd (u, E) 0, ||u|| = =o..
32

---

Published as a conference paper at ICLR 2026
Each point in saddle, meaning that for any u E f2', the Hessian [2)((u) is a strictly negative eigenvalue. Hence all local minima global minima.
+ is a global minimizer. Moreover, each point in Q2' is strict
Proof of Theorem
By Lemma
6 we have the simpler formula
((0) = C' L(u)
where
||
L(u): == E,
2 ||||3 mm(e"" u)² (r + ||u1
We will use the notation aj oî. Ct ai + 0², and ?i (2a + | + ||e(l>))) for I ~ Unif 00, 1). Furthermore, let = E, [c] 21, = [m and (1 = (a) for 0. Further
setting s = u² u. and h(u) ex u our objective can be written
[uu) = ēu T u ij(c u)² E( u)]-
An elementary calculation shows that (g(gg) (a+tg)² and (9) -2( + 9) for 0.
Note that
VL(u)= = 2 [e(((((lu])))) ije jeTue)== =
and only if
(é E, [.(o((1lll)))) ñje" "je'ue.
ue.
Note that E (+VI+ ode] satisfies this condition. On the other hand, consider decomposition
Be + 8 where s 1 e. Set F(r) = ē E, (r)) Then u = Be + s a critical point if and only
B(F + ||| rie + F(B² + [| 0.
Suppose 3 = 0. Then this above equation requires F [[[[>|]2) 8 = 0. If + O, then we need ||>||
since F(o²)) = 0. If 3 # O, then we require F(8 + || = rj > 0 so s 0 which means
F(B²) = ij. This indeed satisfied by 32 = 1 0², ie., B = +vI + 0². Hence the critical points are
{ U Uu: Le, ||/|| = aJ.
Note that the Hessian is
L 4F + +2F([[u]))] 2ñee""
Note that for critical points u such that a 1 e and ||w|| a, we have
V²I(u)e = 4F 4F"(([u|l))ue e + 2F( [ull]]]]]])]))) je- - 2ñee = 2(F((²)
2ije
so e is an eigenvector of (²I(u) with eigenvalue 22i < O, which is strictly negative. Hence e is descent direction of the objective L(u) is strict saddle point. If =+1 ofe, then
L2I(u) = = 4(1+0²) +2F -4 = (4(1 + F() 2n) cE +2A
Note that for directions orthogonal to v Lee,
V²L(u)u = 242 with 27j > 0.
Along e, we have
V'L(w)e= (2n + 4(1 F - (1+))e
where 4(1 + x²) F2) > 0 so L²L(u) is fact positive definite at u = - + d=e. Hence such points global minimizers while other points strict saddles.
33

---

Published as a conference paper at ICLR 2026
F.2.3 MULTIPLE NOISE SCALES
possible extend Theorem the setting in which one has access to a dataset of varying noise levels p(o) (as in the experiments discussed in Appendix D.4). An illustrative example the case when we have noisy images y I a2 where comes from finite set of noise levels (oI....,o , J. This can be modeled as a ~ p(a) = Σx= Tkóaa where T 7 0, Ek=1 Th = 1 and Ok > for each & E [K]. We will minimize the following objective that also considers noise at different scales:
[( = Ba-p(a) ((0)= == Bo-p(o) U-nii(()) B [1(o.o((12)
V Vlogr,, ( (+:)l
We prove that, when the noise levels follow a general distribution a ~ p(o). we can also characterize minimizers to the above loss. In particular, the scaling of the covariance now depends on the p(o) and (ot). We will show through an example for finite set of noise levels how the distilled generator can outperform any of the noisy teachers.
Theorem 4. Consider the same setting Theorem I Then for any distribution p(o) supported on (0, oo), there exists a unique X > 0 that depends on (ar) and p(a) such that the global minimizers of C over O denoted by 97, == (U*, V²), ). satisfy the following:
U EQ for some orthogonal matrix and (V2)² ='
(11)
In particular, X is the unique solution to the equation
(X* + o;)- Eplor [(o2 + o +1) =
Hence the learned distilled distribution is given by PGag = NO0, X EE
Proof. The proof of Theorem turns out to be very similar to the proof of Theorem with a particular modification of Lemma 5 needed find the precise form of the minimizer. In particular, one can show using similar arguments that the optimal U is given by U F.Q for orthogonal Q. To find the form of V, one needs minimize the following function over feasible V. which only depends on the cigenvalues of V V:
X((VVV) (o2 +o + 1)
X(VVVV (V(V(VVVV) + +f)
Σ ((T(VV = Ep(o).
where F(u) == E Io (w)) and fo is the function defined in Lemma
5 We claim that F is strictly convex and has unique minimizer A Note that for any >> O, derivatives are
1
F'())= = Ep(o),
E, (o? + 4
and
(a" + of
+
1)
F") = Ep(o).
>
[0] + uj}
Moreover, note that + (uu) := Ep(o). [00 + uu 2] is strictly decreasing with lim (u))
and lim (u)) = E (a E Epoo) [(02 + a² + 1) 1 Note the last strict inequality follows by strict monotonicity of sb. Hence there must exist a unique X E (0. oo) that minimizes F satisfying F'(A*) i.e., Eppe)). [(a2 of + 1) invertible VTV is minimized when the gram matrix of V has equal eigenvalues >=((V)) =
E, [(>* --) Thus the above function for
for all i E [r]. Since all of its eigenvalues are the same, by the Spectral Theorem, we must have that VV* I,.
Example with a finite number of noise levels: Consider the case when p(o) = Ek=l Txoog for ΣK == Th = 1. Then X is the unique solution to
=K
Tk > 0 with Te 2 0 and
K
T (0 + 0f + 1) = E, [X* + +
34

---

Published as a conference paper at ICLR 2026
Since the right-hand side is strictly decreasing with respect to X", one show that we always have + min
x" < 1 +o max mmin min kk, Omax = max OR-
The precise value of X interestingly depends now on the noise schedule ). In particular, when the noise schedule o, has much smaller values, the right-hand side increases, requiring X to decrease to satisfy the equation. Likewise, when o, focuses on larger noise levels, the right-hand side goes down, requiring a larger X for the equation to be satisfied.
One can also give mathematical condition on when the distilled distribution is closer in Wasserstein distance to each of the noisy distributions to the ground-truth In particular, consider the low-rank regime < d. Then if X satisfies
d - 1 V (1 -1)
d
X
min ke [K]
o ~ 1 + -0 min
min
then
W3 (PGo3 Px) < min (PY......xx
kE[K]
i..,, the distilled generator is closer the ground-truth distribution than every noisy teacher. Note that the difference between max and min is not too large and the data is low-rank enough d > r, then this can be satisfied. For example, if the largest noise level is less than a multiple of the smallest noise level Fmax V d/r- mmins then this would ensure X + 2 suppose d/r 10. Then as long as mmax < 30min the above condition would be satisfied.
L As an example,
F.3 GENERAL GUARANTEES FOR DISTILLATION
Following up on our linear analysis, we aim derive more general guarantees showing that, under capacity and optimization assumptions the teacher and generator, can give bounds on the distilled generator's performance. This will for more general corruptions and then we will state result more tailored to denoising after where improves upon the noisy distribution.
give conditions when the distilled model
To state results, we set the notation. Let F(pll9) = V V log [[] denote the Fisher divergence between and q. For density p. we let dp p(z)dx pdz. Let (pllg)
[p//- dlg denote the chi-square divergence between p and q. For notational simplicity, we S denote an inequality up to absolute constants so that S b if there exists an absolute constant C 0
such that a & Cb. We let y = Ar + are with o > O and define py == Tpx == A.px * (0, o² Im)) We will use the notation
3t = x + oec, Ut oge'
with marginals PX,,.PY, For the generator Go with clean law pe and parameters A E θ, its measurement law is pa.y := [ppe] with noise-convolved marginals Po,Y. We consider distilling by minimizing
(i Laissi == Es (i 1) - fe (ūr 1)
where fo is the teacher and f is the fake diffusion model. For each let sy, and so,y. denote the scores of py, and pa,y., respectively. Throughout, we will assume sufficient regularity of the densities so that all scores V log p and gradients Vp are well-defined, including those induced by po for 0 E O Moreover, put
(v, t) = fo t) SY,(V) ()( = fv(ū.t) t) s0.Y.
2. == sup Epy. [1 (a(,),)))
1)1,, L²,1 sup (Epy. 11 (v,, 1)1)2) 1/2
We will assume we have optimized our parameter 0 to point Define the local fake dafe-diffusion error at 0:
(0) = sup E.p
11 1).
?
Finally, let (0)) denote the local density ratio
( = sup wx wr (v) = På.x.
35

---

Published as a conference paper at ICLR 2026
We will make the following assumptions on the data and learned parameters: Assumption 5 (Data distribution). Suppose that the corrupted data distribution py, satisfies uniform Poincaré-lie like inequality in the sense that there exists a X > 0 such that
Vt,p E (po.y.leee.. /140//Y..NI/Pppp /
(pllpv.d) & 25'
Assumption (Capacity and local near-optimality). There exists a O such that po y = py and satisfies the following for some Eopt > 0:
(d) Cdisti((0)) + Eopt.
Moreover, the fake diffusion network parameters v satisfies Et ( 6) (ve, t)l
Eg.py,
We note that this can be relaxed to an upper bound up to a constant.
We will aim to prove the following:
Theorem 5. Under Assumptions 5 and we have that the following holds:
1. (general bound) the learned distilled distribution in measurement space satisfies
2
A(0) Ao
A Eoo Ao
E. B.F(Pay,,lly.... & 60.4
+ =
+ =22 + =0) + Copt
2. (measurement injectivity) if the measurement operator T p * N(0 T satisfies FF, (Px,c) & R FyF(Pgy (llpy,)) for some distilled distribution in image space satisfies
0, then we have that the learned
2
A Ao
A(Ò) Ao
E, E.F(lo,llpx.d SA
V
45A
+
=2 + = + Eopt
0.4
Discussion. This result guarantees a bound on the Fisher divergence between the distilled genera- tor's measurement distribution and the true measurement distribution. The key idea is that minimizing the distillation loss encourages the generator to learn an image distribution whose induced measure- ments close the true measurements. Lemma more explicitly connects the distillation loss the reverse Fisher divergence between the measurement distributions. The second key component the second bound, which shows that if the corruption operator satisfies injectivity property over the data, then we can transfer this bound to the distilled distribution in image space. Hence distillation has the potential to succeed when 1) the distilled generator learns create images such that, when corrupted further, look like the measurements and 2) the corruption operator is stable or injective over our distributions. We show in the following Corollary that in the instructive case of denoising, the corruption operator is stable and we can give a condition on when distillation can improve over the noisy distribution.
Corollary (Improvement in denoising). Under the setting of Theorem when = we have that the measurement injectivity condition holds for some & > 0 that depends the noise schedule and for E6,1 (0), Copt sufficiently small, have that the distilled distribution improves upon the noisy distribution
E, E, (px.)) < E.F(pYallxx...
To prove these results, we require a number of technical lemmas. The first a change of measure result that will be useful in transferring expectations.
Lemma 7. For fixed I, let eo(u, 1) = [v 1). Then we have that
E Epy Eolu. 1) < 2 + e x (poo..ellpx))
36

---

Published as a conference paper at ICLR 2026
Proof of Lemmal 7 Suppose we set = PY.t, W....P. p po,Y. and dp/dq. Recall that for density the induced T. (g) norm is given by |1 = dq.. Then we have via an application of Cauchy-Schward that
/ (w(y) - 1)e t)dg
(v, - Egco(y,t)]==
< I 1 |
(xlla)))) (B 1/2
The result follows by using the definitions of 6,. and 60.4.
The next Lemma is crucial, in that it shows how the distillation loss from SiD Zhou et al. (2024) encourages the generator Gg to produce images whose measurements match the distribution of the true measurements.
Lemma 8. For 0, define r(O) == Es [oav x2 x2 (P. Then we have that the distillation loss satisfies
R F ,lllx,,) 2e 20) -2 2T(0) < Liisui
Proof of Lemma
We first consider the decomposition
fo fo (sY $8,Y.) + 6s 8z.
Using || + y 212 3(1 | 11v12 + ||m| for x sY 50,, y = 8g and 8Φ and
taking expectations, we have that for fixed
1loy 1) - fo (Tr, 01 < 3p 18 (Tr, 1) 80,Y, (ūt, 1) + 31E pllppg.r. (ir 1) + 3 pa,... (ūr, 1)12
3 pa. (ūr -0.r. t) (2222 Sa. x2 (Po..,.lllY..)) + 3e
where the last line follows by Lemma 7 and the definition of a (0). Taking an expectation over t yields
Lassu (0) 3E, FE.
3e ( (0)
The lower bound holds by using the bound || + y- w| > | - 2(11v11 + || ll). applying the same bounds, and taking expectations. Note that this bound holds because for any 2,
|| + 2112 = || 112 - 2(z, 2) + ||=1|2.
Recall Young's 's inequality:
[xx,)| < o |00 2 + 2े2 ||:1 for > 0. Hence we have the lower bound
|| + 2 1/2 > || a² || a - 2 + ||
Choosing a 1/2, setting inequality.
y w, and using y w 11 < 2 1111|13 || (2) yields the desired
An additional ingredient we need control over the x" distance and relating it to the Fisher divergence. For that, we need the following Lemma.
Lemma 9, Under Assumption
we have that
A(0)
(B.FlPgy..cllee) 1/2
r(0)
do
Proof of Lemma 9 yields
We first fix and set q = PY,t, P Pá.v. and w p/g. Then Assumption
V
A())
/|
A( F(pllg)
xp(1)) & 25'
/iwll3d- 25
dg
dg
w
No
w
Au
37

---

Published as a conference paper at ICLR 2026
where the first line follows by assumption, the second inequality follows by definition of A and the last equality follows by definition of the Fisher divergence. Ineed, for the last equality, note that p/g. then Vw (p//g) = ş log log w(sp - 8q) where Sp and By are the scores of and g, respectively. This ensures that
V V dq -
w 1|pp
sy1l
[1sp sq pdl F(pllg)-
w
Taking square roots and applying expectation over t along with Jensen's 's inequality for the concave map u + ve yields the desired bound.
Armed with these technical results, we now prove Theorem
Proof of Theorem 5 First, recall that by Lemma we have the lower bound
Caisui
E ( IPY:) 2=) 2= - (0)
(12)
Moreover, by definition of 0", we have that
(f") = Et. llfe - Er. - fo pr. sy & 2
where we used Pa-.y = py in the second equality and the assumption on in the second-to-last inequality. Then note that by Assumption [ we have that
Lasstall ( < (f") + Eopt & 2 + Eopt.
(13)
Combining and 13 along with Lemma 9 yields the quadratic inequality _ < A B where
we have set X (.. A 35 Eopt 2 (0) and 2= A(O)
Solving
Aa
2
this inequality, we have that X < (B + 1 2A the desired result.
Substituting the values of X, A, and B yields
Finally, we show the denoising Corollary.
Proof of Corollary 7 The Corollary is consequence of the fact that when A I, have that
Py. = T[xx] * (o, of = Px * *N o² I) NO, of = px * (O, (o² + a l)) Hence if we define the time-shift map T (t) by o²(0) = of a², we have that for any and f, FPe. ((pe...:llpY..) = (llxx)))) Denote the distribution of ~ where p has support in T [[tmin- (max) and set
p(T(t))+((t) (0)
sup
p(1)
tE te(tmin.ffoxx
Then we have that
E, E, (Pxx)) < R- 16 = R LE,
Define A, == E,F(py: (pxx)). Then using the measurement injectivity bound in Theorem there exists universal constant C' such that if
(()) An
( An
R.
+
+ + 2622++82 +0) + Eopt
< A..C
then we have
E,F(pb,lexx: < E.F(prallxx.).
38

---

Published as a conference paper at ICLR 2026
Connection to mode-seeking. The previous theory connects mode-seekine behavior in the following way. In particular, under the setting of our theory, obtaining a small Fisher divergence encourages small reverse KL-divergence. To see this, recall that the KL-divergence and -i-diverge satisfy DKL. (pllg) & x²(1ll)) 4)) when < (which follows by the inequality log L & t - for 0). Following the proof of Lemma this shows that we have the inequality
A F(poy, 70
(PL & x2 (elpy;)
where A A(0) is the density ratio Ya,Y, and Py; and is the constant in Assumption 5 Hence one can minimize the Fisher divergence and the ratio A, this will also encourage smaller reverse KL-divergence (mode-seeking behavior). further theoretical investigation of this connection an important direction for future work.
G VISUALIZATION FOR OVERESTIMATED DATA NOISE LEVEL
In Section 4.5
we discussed handling an unknown data-noise level a and showed that slight overes- timation preserves strong performance, consistent with blind unverse-rroblem solvers /Zhang al. 2017]20018 Here, we provide an intuitive 2D toy example demonstrating that modest overestimation yields clean generations, whereas underestimation produces noticeably noisier samples.
We construct a noisy training set with ground-truth a = 0.05. During both pretraining and distillation, vary the assumed noise represent underestimation (ā a), accurate estimation (ā σ), and overestimation (ō > o). As shown in Fig. [4] a slight overestimation increases effective regularization, helping generated samples better adhere the data manifold.
99
455
650
425
45
408
400
-13
-18
40 485
66 69 25 18
-26
JAn
do
15 a/ 65
-18 -b6 .098
35 ale 63
(a) 0 (underestimation)
(b) 0.05 (accurate)
(c) ā 0.10 (overestimation)
Figure 14: Effect of noise-level misspecification in 2D toy example. We train with noisy dataset
at σ = 0.05 and vary the assumed noise ē during pretraining and distillation. Underestimation (ā σ)) yields noisy generations; accurate estimation recovers the target structure; and slight overestimation (ā σ)) acts as additional regularization, improving adherence to the data manifold.
See also Section 4.5 for ablations.
H PHASE I: VARIANTS OF PRETRAINING DETAILS
In Section 3.1 introduced four variants of diffusion model pretraining methods. Here, provide the training details for each of these variants.
H.1 STANDARD DIFFUSION
Standard Diffusion aims learn the distribution of y(() A(x(0) directly. To do this, we train the diffusion model on the corrupted dataset (g(() i=1- The training objective is given by:
(1)
LSD Eo..g [tt
(14)
where X(t) is a time-dependent weighting function and o, is sampled from predefined noise schedule. We using the EDM schedule for X(t) and same in Karras et al. (2022).
39

---

Published as a conference paper at ICLR 2026
The detailed training procedure is summarized in Algorithm
Algorithm 2 Standard Diffusion Training
1: procedure STANDARD- BRAADARR-DDFFUSSOON(/OCCC
p(or), K)
2: for k = 1
K do
3:
Sample a batch y ~ { Ji=1
4:
Sample noise level d, p(
5:
Sample noise ~ (0, La)
6:
Construct noisy imput: 3/ = + 3
7: Update parameters via gradient descent on LsD (Eq. 14
8: end for
9: return Trained diffusion model fø
end procedure
H.2 DIFFUSION FOR DENOISING
Diffusion for denoising aims to learn the distribution of A(x(1) from noisy observations y A(x(1) + de, where a is known (constant) noise level. The objective to mitigate the impact of additive noise during training. Related denoising strategies have been explored in Daras et al. (2025).
The Tweedie adjustment is compatible with more advanced diffusion training techniques (Ap- pendix H.3 Appendix H.4), tailored for random inpainting and Fourier-space inpainting, respec- tively.
Training loss.
We use the following objective:
of 0² o²
a² yc
27
LD = Ror v.c
So(y,, 1) +
(15)
o
where ot is sampled from the diffusion noise schedule and clipped so that az 2 a, and yt y + /o² oec.
This formulation induces differences in both training and inference. Full details are given in Algo- rithm 3 and Algorithm 4 For sampling, we denote by Teacher-Full the standard EDM sampling from max down to 0, and by Teacher-Truncated the EDM sampling truncated at the data noise level ar. See Algorithm for details.
Algorithm Diffusion for Denoising (Training)
1: procedure DIFFUSION- TMPFUSSOOFFORDDEEOONNNATAAAAAAA DENOOISNGG-
p(o))) K)
2: for to K do
3: Sample minibatch y (N) ~ p(ot), NNO, Ia)
4: Gx <- max(o,
b. Clip noise level
5: Yt < + V oF o2
6: Update f by descending V CD in Eq.
15
7: end for
8: return trained diffusion model fs
9: end procedure
40

---

Published as a conference paper at ICLR 2026
Algorithm 4 Diffusion for Denoising (Sampling)
-OR-DEORIOIINI-SAAAAA 2: Sample 2T (1 o7
procedure
o, {at)(=0)
3: for f= = -,. do
4:
Bg + fz)
5:
if truncation enabled A ot-1 then
6:
return T0
Fruncerr
7: end if
8:
2;-1 - 21
[2 80)
b EDM-style update
9: end for
10: return 20
Teacher-Full
11:
end procedure
H.3 DIFFUSION FOR RANDOM INPAINTING TASK
Daras et al. Daras et al. 2023a) introduced a diffusion training objective specifically designed for the random inpainting task. The goal is to learn the underlying clean data distribution px from partial observations of the form ( (t) where M is a binary inpainting mask applied to the image. To introduce further stochasticity, a secondary corruption mask M is applied during training.
The training loss for random inpainting is given by:
||A (Mo w)
CRI Eo...
(16)
where V1 y ore, and is conditioned both the corrupted observation and the masking pattern. All training schedules and hyperparameters follow the same configuration as in the original paper (Daaa et al. (2023b).
The full training procedure is outlined in Algorithm
Algorithm 5 Diffusion Training for Random Inpainting
procedure DIFFUSION-LNPAINTINGG INA.IINIIGG-TTANNNNNGGG T M, p(o.), K)
2:
for k = to K do
3:
i== Sample batch y N
4:
Sample noise level a, p(o))
5:
Sample noise ~ N' Ia)
6:
Sample a further corruption mask M conditioned on
7:
Compute V/ y o.
8: Update fe using gradient descent on LRi in Eq. 16
9:
end for
10: return Trained diffusion model fo
end procedure
H.4 DIFFUSION TRAINING FOR FOURIER SPACE INPAINTING
Aali et al. 2025) introduced diffusion training objective specifically designed for the multi-coil MRI on Fourier Space. The goal is to learn the underlying clean data distribution from
HH f MFS))a
y
41

---

Published as a conference paper at ICLR 2026
where St denotes the coil sensitivity profile of the i-th coil, F is the Fourier transform, and M is the masking operator in Fourier space. The further corrupted observation would be
No.
F z.
The training loss for multi-coil MRI is given by:
||A[fo(O. M, 2
CFs = Eo..y.
(17)
where y/: = + ose, and fa conditioned on both the corrupted observation and the masking pattern.
The full training procedure is outlined in Algorithm 6 All training schedules and hyperparameters follow the same configuration as the original paper Aali et al. (0255)
Algorithm Diffusion Training for Fourier Space Inpainting
1: procedure
AAANAAAAIIAANNNNNNTTAAAAAAA (a (y M, p(o)). K,
2: for to K do
3: Sample batch y (
Ti=l
4: Sample noise level 0, ~ p(())
5:
Sample noise N/ Ia)
6:
Sample further corruption mask M conditioned M
7: Compute Yt + y + 0₂
8: Update fo using gradient descent on LRi in Eq, 17
9: end for
return Trained diffusion model fø
11: end procedure
I PHASE II: DISTILLATION
In Section introduced the SiD generator loss (Eq. 5 The SiD objective admits additional design choices, discussed in the original paper Zhou et al. (2024). For completeness, we present the exact generator-loss formulation used in our implementation and defer details such as time-step scheduling and hyperparameter settings to the cited work.
SiD generator loss.
Let Fig = Go(z) and 2; = Fig + are with 2, 2, (O, Ia). The loss is (sw((w)) = E Ex,,,e a) A(t) [ - fo(
A(t) (fo( - fa(r,, () xg)]
(18)
where /s is the teacher (pretrained diffusion model), fa is the fake diffusion model, A(L) is time-dependent weight, and balances the two terms. Unless otherwise noted, we use a 1.2. following Zhou et al. (2024), which reports strong performance across tasks.
Note that, following common practice, we initialize both the auxiliary diffusion model fg and the generator Go from the teacher diffusion model fo Zhou et al. (2024); Yin et al. (2024a b), which three networks share the same
crucial for facilitating and stabilizing training. Importantly, architecture and capacity. Nonetheless, the generator has been shown to outperform the teacher Zhou al. 2024), as it avoids the accumulation error inherent multi-step sampling.
IMPLEMENTATION DETAILS
Hardware and measurement. Unless otherwise noted, all pretraining and distillation runs SxNVIDIA A6000 GPUs. Inference wall-clock time measured on 4x NVIDIA A6000 GPUs with batch size of 1024. Images are normalized to [-1, prior adding Gaussian noise.
42

---

Published as a conference paper at ICLR 2026
Teacher pretraining: denoising task. For the denoising task, we follow the EDM setup Karras et al. (2022). On CIFAR-10, we train for 200 M image iterations to match the EDM computational budget; FFHQ. CelebA-HQ, and AFHQ-v2 we train for 100 M iterations (half of EDM's budget). We adopt EDM hyperparameters verbatim Karras et al. 20222. All images are corrupted with additive Gaussian noise at the prescribed level.
Distillation: denoising task. For the distillation phase, we train the one-step generator on CIFAR- 10 for 100 M image iterations and on FFHQ, CelebA-HQ, and AFHQ-v2 for M iterations; this budget suffices to reach competitive FID. Unless stated otherwise, hyperparameters mirror those of SiD Zhou et al. 2024). For CelebA-HQ, we use the same configuration as FFHQ/AFHQ-vZ except for dropout rate of 0.15.
Teacher-consistency.
For experiments in Tab involving Teacher-CCosssseee Daras et al. (2024) use reverse steps and 32 Monte Carlo samples approximate expectations. The consistency-loss weight is selected from {0.1, 1.0, 10.0) as a fixed coefficient to maximize performance.
General corruption tasks: pretraining. We again follow the EDM training protocol Karras et al.] (2022) and pretrain for 100 M image iterations: (i) For Gaussian deblurring and super- resolution with a 0, we use Standard Diffusion objective (Eq. 14). (ii) For the same tasks with 0.2, we adopt the Diffusion for denoising loss (Eq, 15p. (iii) For random inpainting with O, we use the publicly available Diffusion for random inpainting checkpoint at hnttps::: /github com//iaanisddazan
main (iv) For random inpainting with 0 = 0.2, we train with the Diffusion for denoising loss (Eq, 15); we avoid the dedicated inpainting loss (Eq. [6) in this noisy regime due instability, consistent with Daras et al. (2023b). (v) For Fourier-space random inpainting on MRI, we initialize from the checkpoint at ht pssttps:.u /////gtuuuu com/utcs /ab/ ambi ent di f fusionunonooooo mri git Unless noted, pretrained diffusion models
amb wmwient-tttnnnmmm wmm.ieentddfffssotttetteteteeeet
use EDM hyperparameters Karras et al. (2022).
General corruption tasks: distillation. During distillation we train the one-step generator for 50-100 M image iterations, with early stopping if the validation FID begins diverge.
Reproducibility. Upon acceptance, we will release code and checkpoints to facilitate reproduction and further research.
43

---

Published as a conference paper at ICLR 2026
Figure 15: CIFAR-10 10 32x32 noisy dataset with a 0.1 (FID: 73.74).
K
ADDITIONAL QUALITATIVE RESULTS
In this section, we present additional qualitative results. A quick view is in Appendix A
44

---

Published as a conference paper at ICLR 2026
Figure 16: Unconditional CIFAR-10 32x32 random images generated with RSD training with noisy dataset with a = 0.1 (FID: 3.98).
45

---

Published as a conference paper at ICLR 2026
Figure 17: CIFAR-10 10 32x32 noisy dataset with a 0.2 (FID: 127.22).
46

---

Published as a conference paper at ICLR 2026
Figure 18: Unconditional CIFAR-10 32x32 random images generated with RSD training with noisy dataset with a = 0.2 (FID: 4.77).
47

---

Published as a conference paper at ICLR 2026
Figure 19; CIFAR-10 32x32 noisy dataset with a = 0.4 (FID: 205.52).
48

---

Published as a conference paper at ICLR 2026
Figure 20: Unconditional CIFAR-10 32x32 random images generated with RSD training with noisy dataset with a = 0.4 (FID; 21.63).
49

---

Published as a conference paper at ICLR 2026
Figure 21: Unconditional FFHQ 64x64 random images generated with RSD training noisy dataset with σ = 0.2 (FID: 6.29).
50

---

Published as a conference paper at ICLR 2026
Figure 22: Unconditional CelebA-HQ 64x64 random images generated with RSD training noisy dataset with a = 0.2 (FID: 6.48).
51

---

Published as a conference paper at ICLR 2026
Figure 23: Unconditional AFHQ-v2 64x64 random images generated with RSD training noisy dataset with a = 0.2 (FID: 5,42).
52

---

Published as a conference paper at ICLR 2026
Figure 24: Examples from the training dataset used for the Gaussian blur task with a = 0.
53

---

Published as a conference paper at ICLR 2026
Figure 25: Qualitative results of RSD generation for the Gaussian blur task with o = 0.
54

---

Published as a conference paper at ICLR 2026
Figure 26: Dataset samples for the random inpainting task with p = 0.9 and 0.
55

---

Published as a conference paper at ICLR 2026
Figure 27: RSD generation results for the random inpainting task with 0.9 and o = 0.
56

---

Published as a conference paper at ICLR 2026
Figure 28: Dataset samples for Super Resolution x2 Task with a = 0
57

---

Published as a conference paper at ICLR 2026
Figure 29: RSD generation results for Super Resolution x2 Task with o = 0
58

---

Published as a conference paper at ICLR 2026
Figure 30: Dataset samples for Gaussian Blur with o 0.2
59

---

Published as a conference paper at ICLR 2026
Figure 31: RSD generation results for Gaussian Blur with a = 0.2
60

---

Published as a conference paper at ICLR 2026
Figure 32: Dataset samples for Random Inpainting with = 0.9 and a = 0.2
61

---

Published as a conference paper at ICLR 2026
Figure 33: RSD generation results for Random Inpainting with
p = 0.9 and o 0.2
62

---

Published as a conference paper at ICLR 2026
Figure 34: Dataset samples for Super Resolution x2 Task with o = 0.2
63

---

Published as a conference paper at ICLR 2026
Figure 35: RSD generation results for Super Resolution x2 Task with o = 0.2
64

---

Published as a conference paper at ICLR 2026
Figure 36: Dataset samples for Random Inpainting with 0.8 and o = 0
65

---

Published as a conference paper at ICLR 2026
Figure 37: RSD generation results for Random Inpainting with
p = 0.8 and = 0
66

---

Published as a conference paper at ICLR 2026
Figure 38: Dataset samples for Random Inpainting with 0.6 and o = 0
67

---

Published as a conference paper at ICLR 2026
Figure 39: RSD generation results for Random Inpainting with p = 0.6 and a = 0
68


Reviews:
[{"review_id": "dlTtURpxsX", "reviewer": "Reviewer_ZwLf", "summary": "This paper proposes a framework named Distillation from Corrupted Data (DCD) for learning high-fidelity, one-step generative models from corrupted data. The framework is implemented in two stages. A diffusion model is first pretrained on the corrupted data as a teacher model, and the knowledge of the teacher model is then distilled into an efficient one-step generator. The contribution of this manuscript lies in applying score distillation technology to learning generative models from corrupted data, providing a new solution for the relevant field.", "questions": "1. What are the innovative points of the DCD framework in terms of methodology and how do these innovative points give it a unique advantage in learning generative models from corrupted data?\n\n2. What is the additional error introduced by approximation with auxiliary fake diffusion model?\n\n3. How do the results change with resolution and class diversity?\n\n4. What is the performance of DCD under more metrics like KID, Precision/Recall, and Density/Coverage?\n\n5. Could the authors provide ablation studies on the fake diffusion surrogate, shared initialization, and training stability?\n\n6. What is the rationality of the assumption that the score fields between the teacher model and the generator could be aligned?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces DCD, a novel framework for learning generative models from corrupted data without needing clean data. This approach is original in its unified treatment of diverse corruptions.\n\n2. The research is of high quality, with a rigorous two-phase framework combining corruption-aware diffusion pretraining and score distillation. Theoretical analysis supports the method, and comprehensive experiments validate its effectiveness across various datasets and tasks.\n\n3. The proposed DCD framework is potential to improve generative modeling in scenarios with limited clean data.", "weaknesses": "1. The DCD framework extends score distillation to corrupted data but lacks novel methodological contributions. Score distillation has already been widely used for learning generative models from clean data, and is extended to the scenario of corrupted data in this paper. However, it seems that DCD relies on existing diffusion models and score distillation techniques and offers limited innovations and does not provide unique paradigm for learning from corrupted data. \n\n2. In the distillation phase, the authors used an auxiliary fake diffusion model to approximate the distribution induced by the generator. The use of an auxiliary fake diffusion model during distillation introduces potential approximation errors. This method may not accurately capture the distribution induced by the generator and affect the overall performance of the framework.\n\n3. Experimental results are not sufficient to validate the effectiveness of DCD.\n\ni) Evaluations are performed on small, single-type datasets like CIFAR-10 and CelebA-HQ. The results on these small, homogeneous datasets cannot demonstrate the generalization of DCD on different image sizes, diverse classes, or various corruption types/levels/schedules.\n\nii) In most experiments, FID is adopted as the main evaluation metric. There lacks of a comprehensive assessment of the performance of DCD under varying metrics.\n\niii) Ablation studies on the fake diffusion surrogate, shared initialization, and training stability are missing. Both the fake diffusion model and the generator are initialized from the teacher model. Since the teacher model itself may learn incorrect features, this could cause staying in local optima during training. \n\n4. In the mathematical derivation of score distillation, the authors assume that the score fields between the teacher model and the generator could be aligned, but do not explain the rationality of the assumption or provide supporting evidence.\n\n5. Minor issue. The names of the horizontal and vertical coordinates are missing in Figure 12.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mwCTDFu2Ak", "reviewer": "Reviewer_4EL1", "summary": "The paper proposes a two-stage framework named DCD for learning a one-step generative model from corrupted observations. Stage one involves training a diffusion model teacher on the corrupted data using a suitable corruption-aware objective. Stage two distills this teacher into a one-step generator, where the key step is to apply the known corruption operator to the generator's output during the distillation loss computation. Extensive experiments show that the second stage not only makes the model fast but also significantly improves sample quality (FID) over the teacher model across various restoration tasks and MRI reconstruction.", "questions": "Please refer to the section of Weaknesses. I suggest that the authors provide a precise positioning of their contribution and a more intuitive analysis on the underlying mechanism to explain why distillation improves quality.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses an important practical problem. Learning from corrupted data is particularly useful in scientific areas like medical imaging. A notable contribution of this paper is that it shows that score distillation can improve generation quality from corrupted data in addition to accelerating sampling.\n\n2. The method is empirically strong. The proposed method is tested on multiple datasets, diverse corruption types, and a real-world application. The gains over the teacher models are consistent and significant.\n\n3. The two-phase design is practical to allow leveraging the rapidly growing literature on corruption-aware diffusion models for the teacher pretraining stage. This makes the framework adaptable.", "weaknesses": "1. The primary novelty of framework lies in specific composition of established techniques and the empirical validation that distillation is particularly effective in this setting. It leverages corruption-aware diffusion training and score distillation as constituent elements but is limited in inventing new components. The manuscript would be strengthened by more precise positioning of its contribution and clarifies its novelty and contribution in synthesizing existing methods to solve a new problem, rather than the introduction of fundamentally new techniques.\n\n2. The central claim that distillation improves quality is empirical. A deeper and more intuitive analysis of the underlying mechanism is necessary to clarify the claim. In addition, the crucial question on the reason that enforcing self-consistency on a corrupted output space results in a more faithful representation of the clean data manifold remains unaddressed in the main text. Existing explanation for this phenomenon is cursory, and the core theoretical justification is deferred to the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AlzsblM2iE", "reviewer": "Reviewer_AX6j", "summary": "This paper introduces **Distillation from Corrupted Data (DCD)**, a two-stage framework for training one-step generative models directly from corrupted observations without access to clean data. The method first trains a corruption-aware diffusion teacher on corrupted data (e.g., noisy, blurred, masked, or undersampled measurements), then distills it into a one-step generator using a score-matching objective that respects the corruption operator. The authors demonstrate state-of-the-art performance across a variety of corruption types (denoising, deblurring, inpainting, super-resolution, MRI reconstruction) on multiple datasets (CIFAR-10, CelebA-HQ, FFHQ, AFHQ-v2, FastMRI), consistently improving FID over teacher models while achieving up to 30× speedup in inference.", "questions": "1. **Generalization to Non-Linear Operators:** Can DCD handle non-linear corruption operators (e.g., non-linear blur, sensor saturation)? If not, what are the limitations?\n2. **Robustness to Mismatched Corruptions:** How does DCD perform if the corruption operator during distillation differs from that during pretraining?\n3. **Scalability to Higher Resolutions:** Have you tested DCD on higher-resolution datasets (e.g., ImageNet, 256×256)? Are there any architectural or training adjustments needed?\n4. **Comparison with GANs:** Have you considered comparing with GAN-based methods that also train on corrupted data?\n5. **Real-World Deployment:** Are there any latency or memory constraints when deploying the one-step generator in real-time applications (e.g., MRI reconstruction)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- **Originality:** Combines corruption-aware diffusion training with score distillation in a unified framework, with a focus on real-world scenarios where clean data is inaccessible.\n- **Quality:** Extensive experiments across multiple tasks, datasets, and noise levels demonstrate robust and consistent improvements.\n- **Clarity:** The paper is well-written, with clear explanations of both methodology and experimental design.\n- **Significance:** Addresses a critical challenge in generative modeling for scientific and medical imaging, with practical speedups and performance gains.", "weaknesses": "- **Limited Comparison to GANs:** While diffusion-based methods are the focus, a comparison with GAN-based approaches trained on corrupted data could provide a broader perspective.\n- **Complexity of Theoretical Analysis:** The theoretical section is dense and may be challenging for readers without a strong background in divergence metrics and Gaussian analysis.\n- **Hyperparameter Sensitivity:** The performance of distillation losses (e.g., SiD vs. DMD) may depend on hyperparameter tuning, which was not fully explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JdfpX83pV7", "reviewer": "Reviewer_5ZSz", "summary": "This paper introduces a unified and powerful framework, Distillation from Corrupted Data (DCD), for a long-standing and critical challenge: training high-fidelity generative models using only corrupted or degraded data. The core idea is a two-phase approach: first, a \"corruption-aware\" diffusion model (teacher) is trained directly on the imperfect observations; second, this slow, multi-step teacher is distilled into a highly efficient one-step generator (student). The central and most significant finding is that this distillation process does not merely accelerate sampling but consistently and substantially *improves* the generative quality, often surpassing the teacher by a large margin. The authors provide an exceptionally thorough empirical validation across a wide array of tasks—including denoising, deblurring, inpainting, super-resolution, and real-world multi-coil MRI reconstruction—demonstrating state-of-the-art performance in the challenging \"zero-shot\" setting where no clean data is ever used.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a problem of immense practical importance. The discovery that score distillation can serve as a powerful tool for *improving* generation quality from corrupted data, rather than just for acceleration, is a significant conceptual contribution that could shift how we approach learning from imperfect sources.\n\n2. The experimental results are comprehensive and consistently impressive. DCD achieves remarkable FID improvements over its teachers across all tested corruption types, datasets, and severity levels. Its ability to outperform specialized \"few-shot\" methods without access to any clean data is a testament to the framework's power and robustness.\n\n3. The two-phase design is clean, intuitive, and highly flexible. By decoupling the corruption-aware pretraining from the distillation, the framework can readily incorporate new and future advances in diffusion modeling for inverse problems. This modularity makes DCD a highly practical and future-proof solution.\n\n4. The successful application to the multi-coil MRI problem, a challenging scientific domain with complex-valued data and real acquisition constraints, strongly underscores the real-world viability and potential of this work.", "weaknesses": "1. While the paper mentions theoretical support in the appendix, the main text could benefit from a more intuitive explanation of *why* distillation provides such a significant quality boost in the corrupted data regime. Is it a form of regularization? Does it find a more stable generator mode? A brief, high-level discussion would further strengthen the paper's narrative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)