Is Your Diffusion Model Actually Denoising?
Authors: Daniel Pfrommer
Zehao Dou
Christopher Scarvelis
Max Simchowitz
Ali Jadbabaie
Venue: NeurIPS 2025 poster ()
Keywords: diffusion
inductive biases
science of ml
Primary area: deep_learning
TL;DR: We examine the inductive biases of conditional diffusion and devise a rigorous metric for when diffusion models do not behave like denoisers.

Abstract: We study the inductive biases of diffusion models with a conditioning-variable, which have seen widespread application as both text-conditioned generative image models and observation-conditioned continuous control policies. We observe that when these models are queried conditionally, their generations consistently deviate from the idealized "denoising" process upon which diffusion models are formulated, inducing disagreement between popular sampling algorithms (e.g. DDPM, DDIM). We introduce *Schedule Deviation*, a rigorous measure which captures the rate of deviation from a standard denoising process, and provide a methodology to compute it. Crucially, we demonstrate that the deviation from an idealized denoising process occurs irrespective of the model capacity or amount of training data. We posit that this phenomenon occurs due to the difficulty of bridging distinct denoising flows across different parts of the conditioning space and show theoretically how such a phenomenon can arise through an inductive bias towards smoothness.

Is Your Diffusion Model Actually Denoising?
Daniel Pfrommer MIT
Zehao Dou Yale University New Haven, CT 06520 zehao dou@yale, edu
Christopher Scarvelis MIT Cambridge, MA 02139 scarv@mit edu
Cambridge, MA 02139 dpfrom@mit... temiimmii.eeei
Max Simchowitz CMU Pittsburgh, PA 15213
Ali Jadbabaie MIT Cambridge, MA 02139 jadbabai @mit edu
msimchow@andrew.. cmu edu
Abstract
We study the inductive biases of diffusion models with a conditioning- variable, -vaiiiia,iinn.aiiblllleee which have seen widespread application as both text-conditioned generative image models and observation- conditioned continuous control policies. We observe that when these models are queried conditionally, their generations consistently deviate from the idcalized "denoising" process upon which diffusion models are formulated, inducing disagreement between popular sampling algorithms (c.g. DDPM. DDIM). We introduce Schedule Deviation, a rigorous measure which captures the rate of deviation from a standard denoising process, and provide methodology to compute it. Crucially, we demonstrate that the deviation from an idealized denoising process occurs irrespective of the model capacity or amount of training data. We posit that this phenomenon occurs due to the difficulty of bridging distinct denoising flows across different parts of the conditioning space and show theoretically how such phenomenon can arise through an inductive bias towards smoothness.
Introduction
Diffusion models (DMs) have widespread adoption in domains diverse robotic control, molecule design, and image generation from text prompts. The diffusion formalism is popular both because enables stable training of neural generative models, via the denoising training objective, and because it offers broad menu of mathematical and algorithmic techniques for inference [Albergo al., 2023]. For example, inference can be conducted via both Stochastic Differential Equation (SDE) [Ho et al., 2020] and Ordinary Differential Equation (ODE) [Karras et al., 2022] formalisms, the latter of which can be distilled further for accelerated sampling [Song et al., 2023]. The design of these inference strategies hinges on the following fact: for given (forward) diffusion process, there are many distinct "reverse" stochastic processes, each of which can produce the same marginal distribution over generated samples. Hence, from sampling perspective, the various stochastic processes are effect equivalent.
trained diffusion model only an imperfect neural approximation to the idealized reverse processes. Nevertheless, we might hope that this approximation not too inaccurate, For example, even if diffusion model does not perfectly capture the target training distribution, one might conjecture that the denoising training objective ensures that the model is at least consistent, in some appropriate sense, with the forward processes mapping its own generated samples to noise. At the very least, one would hope that as diffusion model is trained on more data, or conditioned on contexts that are well relreseerded in a training dataset, a learned diffusion model will converge towards its mathematical idealization.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Contributions. In this work, we initiate the study of the inductive biases of conditional diffusional models: diffusion models whose generations depend on some context 2. For example, the context could represent text descriptions of an image, observational inputs to robotic control policy, or molecular properties of protein binding target. We investigate the extent to which the path probabil- ities, (Ps, Definition 2.1) deviate from those of an idealized diffusion path (p Definition 2.4) with the same initial and terminal distribution as our learned model (note: not necessarily the ground truth data distribution). To facility this study, we introduce novel, rigorous metric, Schedule Deviation, that is designed to precisely measure the extent to which a flow field induces non-denoising behavior in intermediate marginal densities.
In short, we find
Conditional diffusion models routinely and consistently deviate from the idealized model- consistent diffusion probability path, p This effect is direct byproduct of the inductive bias of conditional diffusion, and is strongly correlated with the discrepancy between pop- ular diffuusion samplers which, mathematically, should be equivalent in the limit of small discretization error.
In more detail, our contributions are as follows:
We introduce Schedule Deviation (SD), our new metric which quantifies the extent to which a diffusion model (conditional or otherwise) deviates from the idealized diffusion probability path (Definition 3. We show that (SD) is closely related to the average total variation distance between path measures (Theorem 1), and that SD can be efficiently evaluated as consequence of the transport equation (Proposition 3.1). Moreover, unlike most prior metrics used to study non-denoising behavior, SD does not require access to the true score function or training data, and can be evaluated with access to only the (potentially conditional) flow-based model. Using Schedule Deviation, we show that the probability path of conditional diffusion models consistently and routinely deviates from the idealized path (Figure 3, Figure 4). Our findings are consistent across toy examples, conditional image generation, and trajectory planning. Fur- thermore, we show that SD is often predictive of the Earth Mover Distance (EMD) between the samples generated by popular inference algorithms.
We demonstrate the Schedule Deviation cannot be significantly ameliorated by increased model capacity training data, and even varies significantly between different classes which equally represented in the training data (Figure 5). Rather, we posit that SD in conditional settings arises as natural inductive bias of conditional diffusion when interpolating between multimodal distributions.
We provide theoretical model (Section 4, Theorem and 3) of Schedule Deviation that shows the deviation can be attributed to an inductive bias of conditional diffusion involving smoothing with respect to the conditioning variable. We prove that, under appropriate conditions, conditional diffusion engages in "self-guidance, combining scores from nearby points in the training data set and demonstrate that this causes deviation from the idealized denoising process.
1.1
Related Work
The origins of diffusion models in machine learning trace back Sohl-Dickstein et al. [2015] and were made practical by Ho et al. [2020], Song and Ermon [2019], which show such models are capable of producing state-of-the-arr image generation results. The DDIM sampling scheme [Song et al., 2020a] generalizes the reverse "denoising" process to allow for a variable level of stochasticity in the reverse sampling process, spawning large body of work on improved sampling methodology [Bansal et al., 2024, Kong and Ping, 2021, Salimans and Ho, 2022, Permenter and Yuan, 2023].
A recent line of work has investigated closed-form diffusion models [Scarvelis et al., 2023] and their relation to phenomena observed in diffusion models, such as hallucination [Aithal et al., 2024]. Drawing on the well-appreciated inductive bias of neural networks towards low-frequency functions [Rahaman et al., 2019, Cao et al., 2019], these works highlight the importance of smoothing biases in understanding hallucination and generalization in unconditional generation, with focus an implicit bias towards "smoothed" versions of the "true" denoiser of the training data.
A concurrent line of works [Vastola, 2025, Bertrand et al., 2025] have also investigated the non- denoising properties of diffusion models, with a particular emphasis on the role of target stochasticity in potentially inducing non-denoising behavior. However, these works focus on deviation from the

---

Figure 1: We principally consider three datasets: conditional MNIST [LeCun et al., 1998] (left), conditional Fashion-MNISS [Xiao al., 2017] (middle), and endpoint-conditional maze path gen- cration (right). For MNIST and Fashion-MNIST we condition on the I-SNE embedding of the images (pictured above) opposed the classes as a proxy for wel-eembedddngcoonniiooooe image generation.
MNIST
Fashion-MNN -MNIST
Maze Solutions
DDPM/DDIM OT Distance
0.4
0.3
N=10000
2
1=10000
N=2000
N- 30000
N-30000
0.1
N-1000
N 60000
N-60000
N 8000
0.0
50
100
150
200
300
80
100
120
DDPM Schedule Deviation
DDPM Schedule Deviation
Schedule Deviation
Figure For conditioning values 2 Uni we plot the Total Schedule Deviation (for Po sampled using DDPM) and optimal transport distance between DDPM/DDIM samples (as measured by Wawsserstein/ErrbhMMooo DisttrDDttaaee,,... demonstrating that our prposed metric, Schedule Deviation, is indeed predictive of divergence between different samplers. In Appendix C we demonstrate these trends hold across different choices of samplers and show additional experiments on attribute- conditional Celeb-A, where the conditioning space is more uniform.
true denoiser specified by the training data, whereas we consider denoising cuff-onssstency of the model itself. In this regard our work is similar to Daras et al. [2024], which introduces an additional loss term to induce better denoiser celf-consistency. Unlike Vastola [2025] and Bertrand et al. [2025], which reach differing conclusions on whether deviation from an ideal denoiser benefits sample quality, we take no position on whether non- non-denoii behavior is beneficial.
Our work orthogonal yet complementary to these efforts, elucidating the effect of inductive biases for conditional diffusion models [Ho et al., 2022, Song al., 2020b]. Unlike the unconditional setting, where generalization necessitates underfitting the training data, we show that an implicit bias towards smoothness with respect the conditioning variable can bias the model to duitr--diittibutiinn conditioning variables while simultaneously overfitting the training data. Of particular interest to this work the methodology of classifier-free guidance [Ho and Salimans. 2022], which performs conditional sampling via a combination of both conditional and unconditional diffusson models. Although we consider purely conditional models, we show that under the appropriate implicit biases, form of classifier-ff ffree guidance, which we term "self geiff-guidacee"" can naturally arise. The exact effect of guidance on diffused samples remains the subject of ongoing inquiry, but recent work has shown that special forms of guidance can be (approximately) understood as either sampling from the manifold of equiprobable density [Skreta et al., 2024) or, alternatively, as a combination of Langevin dynamics on the weighted product distribution [Bradley and Nakkiran, 2024, Shen et al., 2024]. We do not further investigate the mechanism by which guidance yields high quality intermediate samples, but rather demonstrate simple datasets that the form of "self-guidance" has predictive capabilities.
3

---

Prior work has shown that classifier- free guidance causes the resulting diffusion to no longer constitute a denoising process [Bradley and Nakkiran, 2024]. We note that our notion of consistency is unrelated to that of Consistency Models [Song et al., 2023]. which may in fact be schedule-inconniiteet under our framework. Daras et al. [2024] explore similar notion of consistency with a denoiser and attempt to enforce this condition by a data-augmention-styy
loss. In contrast, our metric is tailored to specifically measure how much the evolved density diverges from a reference denoising process, as opposed to how much it deviates from being diffusion model.
2 Preliminaries
We apply the framework for flow-based generated models in Lipman et al. [2022] and Albergo et al. [2023], adapted to our conditional setting. For mathematical elegance, we adopt continuous-time formalism; see e.g. Chan et al. [2024] for the discrete-time formalism. We consider pairs (z, 2). where
E 2 C Rd= is conditioning value and x X C RR is a datapoint. For a given conditioning value z, we seek to generate samples from a continuous conditional distribution p" (xr =). For instance, r may be an image or action while 2 can be a text prompt or observation. We use Z, X denote random variables over Z and X' respectively, and E Z, X' for particular values.
Flow-based generative models parameterize a time-varying family of conditional densities P. ( E [0, 1], where p1 (z]2) can be easily sampled, and po(2|2) a conditional density aimed to approximate p These densities can be specified conditional normalizing flows [Lipman et al., 2022], which describe the per-ti ser-time-stepp marginals p, via the evolution of particles moving according to specified velocity field (x(x..)
Definition 2.1 (Probability Paths and Conditional Flows). Let E [o, 1] be time index. Fix flow field v [0, 1] x
X and a family of conditional densities p. (x|2²) indexed by time s E [0, 1]. which we refer to as probability path. We say (v, p) is conditional normalizing flow if particles evolved under e match the marginal densities ps of the probability path, i.e..
Ps (+12)) Law X, Z 2), as [0, E
(2.1)
d x, = =x(,,,
where I pı (+1=) ds
The stochastic interpolants framework [Albergo et al., 2023] provides an alternative description for flow-based models, with a particularly succint description of diffusion models (e.g. Ho et al. [2020]).
Definition 2.2. A diffusion schedule (a, a) consists a noise schedule o : [0, 1] + R2 and signal
schedule [0, 1] + Rzo such that (1)- = (0)) 1, and (11 (0)) = 0.
Definition 2.3 (Diffusion Probability Path). Given a diffusion schedule (a, a), target and initial densities po and Pt, the diffusion probability path given by,
P. (-1z) Law 2. 2)
where X, = a(s) (,) denotes the stochastic interpolant [Albergo al., 2023], where X1 2 = 2 P1 = NN0, I) and Xol = ~ po(-1z) with Xi 1 Xo.
Diffusion Flows Under appropriate regularity conditions, all diffusion probabil- ity paths can be realized by an infinite family of conditional normalization flows (u. p). We focus on one such path, the model-consistent diffusion flow, which corresponds to the solution of the DDPM objective [Ho et al., 2020]. We use i and 5 throughout the rest of this paper to emphasize conditional normalizing flows which may be associated with learned model, rather than the true data-generating distribution, i.e. where Po p=
Definition 2.4 (Ideal Model-Consisten Flow). Fix a (potentially learned) flow and associated family of densities (6. as defined in Definition 2. For a given diffusion schedule (a, a),, let / = (f/)))(<=)))) (s<l=))<<(0.1) be the diffusion probability path associated with Po- The ideal denoising diffusion flow (IMCF) of 9. written (Mer IMCF((e), is the unconstrained global minimizer of
L(v] Elll (X., Z) - (á Xo (1)1))
(2.2)
Above, the expectation E. is taken w.r.t. the diffusion path Definition 2.3 with s ~ Uni 1]). The IMCF is unique, and furthermore the optimization in (2.2) can be decoupled across conditioning

---

N-10000
100
DDPMIDDIM on Distance
50
t-NNE Secord Component ISNE Second
-50
-100
100
166
DDPM Schochi Devialoo
100
-50
-80
50
100
1-SNE First Component
1-SNE First Component
I-SNE First Component
Figure 3: For t-SNE-conditional MNIST generation, evaluate the Schedule Deviation and em- pirical |- I-Wassertstein Distance between DDPM/DDIM samples, ablated over the training dataset size N (10000, 30000, 60000). We note strong structural similarity between the two metrics that appears related to the contours of the conditioning distribution and the conditional data distributions.
variables Thus, will often write (, IMCF(Ao( =)) for fixed value of The IMCF corrrsponns to the unique velocity-minimizine flow consistent with the diffusion probability path, which can be characterized explicitly follows (see Appendix A.I.I).
Remark 2.1 (Ideal Model-Consisten Flow Ground Truth Flow). We use instead of p in Definition 2.4 from here on to emphasize that Po # p", meaning that ( is the ideal flow associated with the distribution of learned model under given sampling algorithm; not necessarily the "true" flow v* Hence, schedule deviation is potentially orthogonal as a metric to whether po matches p". However, note that, by the construction of u U/CC Po Po (MCF and f1 plMcr.
Proposition 2.1. Adopt the setup of Definition 2.4. Then IMCT IMCF(po) is given explicitly by any of the following identities
MMCC (x, 2)
EEla(s) + + = x.Z
(2.3)
=>) (x))
(2.4)
=[) X, 22 =] cs(x
(2.5)
a(s) where cı(s) (ss) aG (s)) c2(s)
o(e) c(अ). 71 (8) == (s)2 ò (()) 72(8) d(s)
o18)
Eq. (2.4) represents the MCF terms the score functions V loo (ar 2)), whereas Eq. (2.5)
expresses the flow in the classical "denoising" form [Ho et al., 2020] via the conditional expectation of the "clean" datapoint Xg given the "noised" X, aas))Xi, + o {2) Hence, the IMCF |MCE is just the (continuous-timg time denoising objective from DDPM [Ho al., 2020].
Conditional Diffusion is Not Denoising.
This section introduces our first main finding: the probability paths in diffusion consistently deviate from the idealized model-consistent probability path MMCF often in regions with high data densiry, and a manner that does not abate with increased number of training examples.
3.1 Measuring Schedule Deviation
To quantify this effect, we start by introducing Schedule Deviation, a novel, natural metric which evaluates the extent to the flow field induces instantaneous deviations from the idealized probability path MCE associated with //M = IMCF()o). Crucially, Schedule Deviation analyzes the behavior of the learned model on m that is, the forward process associated with the distribution ýo generated by the model and not the reverse process Pe itself (as in Daras et al. [2024]).
5

---

Trajectory N=4000
Rashion-MNiS MNIST N=30000
Fashion Rashion-MNiST MNIST N=60000
DDPM//DDI O/T Disanno
L-SINE Secoad Componint.
Start Point
6.8
6.6
6.6
-6.
0
DDPM Schedull Doviation
Second Companent
126
200
Start Point
250
200
150
DDPM
100
-00
Start
L-SNE First Component
1-SNE First Component
Figure 4: Analogous to Figure 3, show that Schedule Deviation is predictive of divergence between the DDPM/DDIM samplers for the trajectory (left) and Fashion-MNIST datasets (right). Note that the structure of the maze (shown in Figure 1) can clearly be observed in the Schedule Deviation. We defer full ablations over the training data for both to Appendix C.
Definition 3.1 (Schedule Deviation). Fix a diffusion schedule (a, σ) and conditional flow model (u, where pi [ar|=) (O, 1). Let 1.MCP(P (x|=) be the diffusion probability path (Def. 2.3) for (o) and consider the tangent probability path PSje = Law Law(X"j which begins at time / with X ~ placr (x) 8t and evolves as de Xxx (X.). We define the Schedule Deviation at z E 2, [0, as
a/c
| 0r as
OpEnr
(D(U=2,)) =
dr.
Ös
We additionally define the Total Schedule Deviation ut e by SD (1 == So' SD 2)ds. Where appropriate, we simply refer to SDrotal as the Schedule Deviation at a given 2 E Z.
We refer the random variable X coincides with pa at s = f, but the evolves differently according the flow field dictated by v. Schedule Deviation (SD) therefore measures the rate at which learned flow instantaneously departs from the IMCF associated with its own data generating distribution Po-
tangent process [Falconer, 2003] because it initially
In addition to its natural relationship to the instantaneous deviation from the IMCF, Schedule Deviation can also be tractably estimated (proved in Appendix A.3.2). Proposition 3.1. It holds that SD(v; z, s) Eppic V (v. - UNCCF) + - (U,-R V log
IMCF
For fixed, can directly sample from |MCP (æ|z),
Algorithm 1 Schedule Deviation
pleer and estimate the score V log and consequently, the idealized flow MMCF (2, 2). by constructing an empirical distribution In sampled from po (x]2), convolving each 4i with Gaussian noise dictated by the Diffusion Schedule, and estimating the corresponding score in closed form. See Algorithm for high-level pseu- docode for computing Schedule Deviation given a particular sampling algorithm using n independent estimates using n (N 1) total samples and Appendix for implementation details. Importantly, because we evaluate log ب ()CC((r)>) for z, s fixed, our estimator is not subject to the inductive biases of function approximation that induce schedule deviation in the learned neutral network.
input: Z, E [0,1] R +
for E [L..... n. do
Za + sample(v,2)
x. a())= (6) + asse where -~~~NN( 10,
S
for j E [l...., NJ do S SS (samplee) 2)})
end for compute V llggp!rr IMCF - IMC IMCE
a
using Eq. (2.3), po 2) N EzesC 8z
E
T1 V [v, ,.MCF [(x 2)
T2 [V. 2 (II, 2). V log PMCP (0MCF(22,))
R RU (ri+12)
end for return mean

---

MNIST
MNIST Model
MNIST Training Samples
MNIST
0.090
DDPP Schhedlee Devition
130
3M persinetess
140
1.. SM, parsmmltara
23.5M persinttess
23. 5M parumetera
120
persimetees
parameters
140
Test Laxs
120
110
120
0,060
100
100
80
0.075
80
60
100l 15 2008 200k
2001 2006 3002
10t
20k
602
Iteration
Troining Dataset
Digit
Figure 5: We visualize the test loss (left) and total schedule deviation (center left) for three different model capacities over the course of training run. For the 13.3M parameter model, we show the effect of training dataset size on schedule deviation (center right), and, for the full dataset, the distribution of total schedule deviation across different classes (right). The median, 30th, and 70th percentile values are shown across the left three plots for sampled training batches and conditioning values.
The definition of Schedule Deviation closely resembles the formulation of the classical Performance Difference Lemma in Reinforcement Learning [Kakade and Langford, 2002]. Indeed, under appropri- ate smoothness assumptions, the schedule deviation both upper and lower bounds the total variation difference the probability paths Ml and p. We defer the proof to Appendix 1.3.1.
Theorem 1. Consider the setting of Defimition 3.1, for conditional flow (v,p), p). For any probability measure μ over [0, 1], the total variation distance TV (q) := _ {p - q dr between ps and pr over μ is upper bounded by integrated schedule deviation:
f TV(ps(12)), JMCF (-1=))dp(s)= & L. SD(1; 2, tfddt SDiotal 2).
P.NC
Moreover, if 10 10aP 1ppal S M- < co, there exists constants E [0, c 2 0
depending on M such that, for all 0 < c0. 8 E [0,
sup [E [Elooc
TV(:: (. (+|=)) 2 ceSD(1;
(3.1)
Remark 3.1 (Schedule Deviation v.s. Generation Fidelity). Note that po A p", so (MCP is the IMCF associated with the distribution of learned model under a given sampler; associated with the true data distribution p". Hence, schedule deviation is potentially orthogonal as a metric to whether po matches p². Moreover, we note that po pa (MCF and p1 = pJMCE, so that schedule deviation principally captures deviations from the reference process in the "middle" of the denoising.
necessarily the IMCF
Remark 3.2. It should be noted that Schedule Deviation is distinct from consistency distillation [Song et al., 2023]. Consistency distillation enforces a related condition: that a few-step model is consistent with the integrated flow map of the ODE induced by the flow-field v, whereas Schedule Deviation measures the deviation of the flow map from the denoising probability path.
3.2 Schedule Deviation is Widely Prevalent
We evaluate the schedule deviation of trained neural networks two distinct settings and 3 datasets, as described below. We use a U-Net architecture similar to Dhariwal and Nichol [2021] for all experiments. For full experiment details, see Appendix C.
Setting 1 (Conditional Image Generation). We evaluate the schedule-deviation of conditional image diffusion models. For ease of visualization and in order keep the associated dimensionality of low (as Definition 3.1 requires computing the divergence w.r.t. I), we consider the MNIST [LeCun et al., 1998] and Fashion-MNISTT [Xiao al., 2017] datasets, each conditioned on 2-dimensional "latent" obtained via a t-SNE [ [Van der Maaten and Hinton, 2008] embedding of the data. In Appendix C. we additionally consider a much larger model trained on the Celeb-A dataset, using t-SNE of the discrete attribute space for the conditioning variable. We note that the Celeb-A experiments, which use simplified Schedule Deviation without the divergence component, are much more noisy and less conclusive than the corresponding MNIST or Fashion-MNIST experiments.
7

---

Setting 2 (Conditional Maze Paths). Second, we construct simplified path- plannnilnnng problem consisting of generating trajectories in a fixed maze. For a given randomly chosen starting point. we consider all paths (ri) to the center of the maze and sample path with probability p(r;) e-(dr)))d((")) where d(r:)) is the length of the ith path and d(r*) is the length shortest path. This artificially introduces multimodality around points where multiple solutions are approximately equally optimal. For each sampled path, we use 64 points along smooth Bezier curve fit to the path such that X' C R21x2 Z C I²
Finding 1: Prevalence of Schedule Deviation. Our experiments broadly demonstrate that Schedule Deviation is prevalent across all datasets. We visualize the total Schedule Deviation over & Z in Figure 3 and Figure 4 for each of our datasets with varying subsets of the training data.
Finding 2: Schedule Deviation Persists with Model Size and Data Amount. In Figure 5, explore the schedule deviation for the MNIST dataset in-depth for both model and data ablations. Interestingly, we find that while larger models tend exhibit slightly lower Schedule Deviation- improvements appear to diminish as the model size increases and more training data can potentially (and somewhat counter-intuitivlly) increase the Schedule Deviation. Furthermore Figure 5 (right), shows that the Schedule deviation can vary dramatically between different classes, suggesting that the Schedule Deviation is both function of the density and structure of underlying dataset.
Key Takeaways: Our experiments indicate several key properties on the Schedule Deviation of condi- tional diffusion models: (1) even high-capacity models can exhibit significant Schedule Deviation, (2) Schedule Deviation appears to be intrinsically related to the underlying structure of the dataset, rather than the amount of data, and, perhaps most importantly, and (3) as we will show in the following section, Schedule Deviation is strongly predictive of divergence between different samplers.
3.3 Schedule Deviation Predicts Disagreement Between Samplers
Many popular sampling algorithms, such as DDPM [Ho et al., 2020] and DDIM [Song et al., 2020a] leverage an SDE formalism to sample from the target distribution p* (see Appendix B for details). These sampling algorithms implicitly make use of the equivalence in Proposition 2. between the learned flow u and V log (=(22) to traverse the same denoising probability path with differing levels of noise in the reverse process. Thus, when v IMCF(po). i.c. there is schedule deviation, both are guaranteed to generate samples X, whose marginals coincide with the conditional flow in Definition 2.1 (provided the number of steps is sufficiently large that discretization error is negligible).
Empirically, significant task-specific differences in performance between DDIM and DDPM have been observed Chi et al. [2023], Song et al. [2020a]. Karras et al. [2022]. In Figure 2, we show that Schedule Deviation is strongly correlated with the difference between these samplers, as measured by The heatmaps Figure 3, Figure 4 further demonstrate the structural similarity of Schedule Deviation and DDPM/DDIM divergence across the conditioning space. Recall Theorem 1, which confirms that SD is proxy for the TV distance between the traversed path the idcal denoising path. Taken together with the strong correlation between SD and OT Distance (Figures 3 and 4), we conclude
the empirical I-Wasserstein (i.e. Sotth--Movrr---sttaae
DDPM and DDIM deviate specifically for conditioning values where the trained diffusion model deviates from the idealization of denoising its generations.
We believe that this finding both (1) sheds light the underlying cause for sampler divergence and (2) demonstrates the utility of our proposed metric as an investigatory tool. In Appendix C, we show our metric is predictive for other sampling strategies, such as the Crrdienn-Essimaate (GE) sampling algorithm [Permenter and Yuan, 2023].
Explaining Schedule Deviation via Smoothness and Self-Guidance
Generalization in unconditional diffusion is broadly understood as a phenomena that arises from capacity-related underfitting of the empirical score function [Yoon et al., 2023, Scarvelis et al., 2023]. thereby preventing memorization of the training data. Prior work this area has examined the effect of an implicit bias towards smoothness and its relation to generalization [Scarvelis et al., 2023, Pidstrigach, 2022, Aithal et al., 2024]. These works, however, do not fundamentally challenge the assumption that the learned flows denoise and instead show how "better" denoisers arises by manifold
8

---

learning [Pidstrigach, 2022] or interpolation over convex hulls of the data [Scarvelis et al., 2023]. In fact, as we elucidate in Appendix B, the "natural" nonparametric extension of the closed-form in [Scarvelis et al., 2023] constitutes an ideal flow. In this section, we provide intuition for how non-denoising paths can arise from smoothness with respect to the conditioning variable through a phenomena we term self-guidance.
Self-Guidance and Schedule Deviation under Discrete Support. We begin by considering the special case where the data generating distribution p=) is supported by a discrete set Sz. We can observe in Figure 6 that for the discrete conditioning distribution shown, the schedule deviation is almost uniformly 0 for 2 E Sz. Thus, we motivate the following assumptions: (1) the model has sufficient capacity to exactly fit the training data everywhere where there is conditioning support, (2) the model is second-order smooth with respect to 2, and (3) of all flows which fit on the support of the training data, the model generalizes via the "smoothest" flow, as measured by 1 2) lcgc Under these assumptions, we develop the following result for Z C R. (with proof in Appendix A.4.1):
Theorem 2 (Discrete-support Smooth Interpolant). Lei Z be supported on finite set Sg h== C. IR for distinct 2(1) <... ... < 2 ordered without loss of generality, For each 2(0 Sz. let vf (0)) El + d
2(0) Then, there are piecewise cubic polynomials p() with pieces defined by the intervals [2(4), 20++11] such that
f /IV2ollade = Σ
arg inf vEarg inf, L[u]
In the case where pz) are linear finctiion.
Remark 4.1. Because diffusion models exhibit both smoothness biases in both x (e.g. Aithal et al. (2024]) and (this work), the most comprehensive proxy would consider smoothness penalty on the joint Hessian "xx, z). This makes a closed form solution considerably more involved, and thus focus solely on the V effect isolate functional dependence on 2.
The optimal low-z-curvature flow characterized in Theorem 2 extrapolates to -uitof-distribution variables 2 to by linearly combining flows associated with in-distribution variables 2; E 52, with weights depending only on the conditioning variable z. We refer to the phenomenon of extrapolating via combinations of flows from other parts of the conditioning space as self-guidance, as these linear combinations of flows mirrors the practice of classifier free guidance [Ho and Salimans, 2022], which composes conditional and unconditional flows vxz22), o(x) via a linear combination.
Schedule Deviation Emerges from Smoothing. Linear combinations of flows in general cannot be written as denoising flows (e.g. in classifier guidance, [Bradley and Nakkiran, 2024]). In particular, for unconditional diffusion probability paths pa(1|z = 2(+)), linearly combining vg 2()) =
(x)) (ls))log..(a (</ = =0() + 72($)x ) (recall Proposition 2.1) does not yield a diffusion probability path, i.e. for weights c1, C2
ci V logg! + 02 V log ( = V log (p!9) (p)))o))))) # V log[(p6 (1) (pb (1) )=
where [p). denotes the distribution of X, Xx + (s) in Definition 2.3 under Law [Xo] p. Thus, Theorem 2 suggests that a simple inductive bias towards smoothness can naturally lead to schedule deviation in e when the interpolated r.Mor (s. 200). We show specifically how the schedule deviation arises for particular choices of diffusion probability paths, ps -ps
(4) (i) in Appendix B.
Self-Guidance with Uniform Conditioning. We additionally show that self guidance can occur even in the presence of continuous densities, where the minimize u* of Eq. (2.2) is uniquely specified by considering a A-weighted penalty term with respect to the Frobenius norm of the appropriate Hessian:
[x[r] [(u] + A EzEx, V (X.. ?)
(4.1)
where L[u] is the original loss in Eq. (2.2) above and the expectation X, Z is taken with Defini- tion 2.3, whose density we recall is pPMCF(2: (z.
Theorem 3. Fix some diffusion schedule (a, a) and let oz (x. 2) be the IMCF flow associated with (*(z)) i.e. the minimizer of Eq. (2.2). Assume that p² is uniform density over some set S, i.e. (z) = e. for some e > and where 1s the characteristic function of S. Then the minimizer to Lx [v) for any E S is given by.
us 1,(,,2)= JE 1 ++ U (x,,2')d2'1
(4.2)
9

---

Neural Dilfusion
Neural Network
Cloadd-Form
Closed-Fomm
1.0
1.0
Geserated Value
1.0
Octerated Value
1.0
0.8
0.8
0.5
0.6
0.6
0.4
0.4
-0.5
-0.5
-1.0
-1.0
Generates Value
Conerated Value
0.5
0.5
0.6
8.6
0.4
8.
0.2
0.2
0.0
-1.0
-100
0.6
1.0 0.0
Cooditioning Value
Conditioning Value
Conditioning Value
Cooditioning Value
Figure We train an MLP and construct a closed-form denoiser for both the dataset (top row) and continuous-suppor dataset (bottom row) described below. For both the NN and closed-form interpolator we show generated values across 2 E [0, 1] as well as the Schedule Deviation over time. In particular, we note that for each dataset the NN and its closed-form analogue exhibit similar inductive biases well as Schedule Deviation away from the training data.
support
Discrete-Suppor Dataset
In the case of the uniform densities, Theorem reveals that self-guidance occurs via a local convolution with Fourier- Fourier-weights. Ttllee e'f: (a-((==))) The frequencies & are attenuated polynomi- ally as 1/ 1 As + O, the attenuation is removed, and the integral f (===== behaves like Dirac 8 around 2 [Duistermaat al., 2010, Chapter 14]. This passes the "sanity check" that, as the smoothness penalty vanishes, averaging becomes ever more local.
Generated Valuxe
0.0
0.5
1.0
Conditioning Value
Continuous-Supee Dataset
Toy Datasets. Motivated by Theorem 2 and Theorem 3, we con- sider two synthetie datasets with scalar X R. and condition E 2 C [0. consisting of mixtures with components centered at I E {00, 1),, (1, 0), (1,1)] The first dataset (with "discrete sup- port") has Gaussian noise with scale
Generated Valure
0.1 applied only to the
component, while the second (with "continuous support") has IID noise of magnitude o applied to both the x, components.
0.0 0.5
1.0
Conditioning Value
We visualize these datasets and samples from a learned denoiser, as well as closed-form interpolants, in Figure 6. For the discrete- conditioning setting, our closed form interpolant considers a simple linear guidance-style interpolation of the flows at 0 and the pantinuouu-conditioinnn setting, inspired by the Fourier- convolution weighting in Theorem 3, we construct an interpolation with a nonlinear guidance function. See Appendix C for additional details. These experiments validate that self-guidance can be a fundamental primitive for extrapolation in conditional settings and predict the learned behavior of neural networks.
Figure 7: Training data sam- ples from each of the toy datasets used in Figure 6.
5 Discussion
We introduce Schedule Deviation, a novel, principled metric for measuring divergence of diffusion models from their idealized denoising paths. This metric is strongly predictive of deviation between different samplers and is difficult to ameliorate via increased model capacity and data quantity. Taken together, our findings reveal that the central mathematical abstraction upon which equivalent inference algorithms are derived may not be representative of actual diffusion models trained in practice, and the breakdown thereof cause seemingly equivalent methods differ. This finding has major implications for the development of future sampling and distillation methods, and serves as a broader word of caution for the use mathematical principles, isolation, as a sole basis for algorithm design. However, our study has number of limitations (see Appendix D: in short, our metric requires computing the divergence of the flow over generated samples, an inherently expensive operation).
10

---

Acknowledgments
DP and AJ acknowledge support from the Office of Naval Research under ONR grant N00014-23-1- 2299 and the DARPA AIQ program. DP additionally acknowledges support from a Math Works Research Fellowship. MS was supported by Google Robotics Research Grant.
References
Sumukh K Aithal, Pratyush Maini, Zachary C' Lipton, and Zico Kolter. Understanding hallucinations in diffusion models through mode interpolation. arXiv preprint arXiv:2406.09335. 2024.
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnded Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2333.08797, 2023.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. Advances in Neural Information Processing Systems, 36, 2024.
Richard H Bartels, John C Beatty, and Brian A Barsky, An introduction splines for use in computer graphics and geometric modeling. Morgan Kaufmann, 1995.
Quentin Bertrand, Anne Gagneux, Mathurin Massias, and Rémi Emonet. On the closed-form of flow matching: Generalization does not arise from target stochasticity. arXiv preprint arXiv:2506.03719, 2025.
Arwen Bradley and Preetum Nakkiran. Classifier-free guidance is predictor-corrector. arXiv preprint arXiv:2408. 000000000800000,, 2024.
Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint arXiv: 19XXv::912.01 1eet1191200999 098.0919110001111111111111 2019.
Stanley Chan et al. Tutorial on diffusion models for imaging and vision. Foundations and Trends@ in Computer Graphics and Vision, 16(4):322-471, 2024.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal Robotics Research, page 02783649241273668, 2023.
Giannis Daras, Yuval Dagan, Alex Dimakis, and Constantinos Daskalakis. Consistent diffusion models: Mitigating sampling drift by learning to be consistent. Advances in Neural Information Processing Systems, 36, 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Johannes Jisse Duistermaat, Johan AC Kolk, JJ Duistermaat, and JAC Kolk. Distributions. Springer, 2010.
Kenneth J Falconer. The local structure of random processes. Journal of the London Mathematical Society, 67(3):657-672, 2003.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12544 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 25:8633-8646, 2022.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference machine learning, pages 267-274, 2002.

---

Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Doris HU Kochanek and Richard H Bartels. Interpolating splines with local tension, continuity, and bias control. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques, pages 33 41, 1984.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00132, 2021.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2222, 1998.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Frank Permenter and Chenyang Yuan. Interpreting and improving diffusion models from an opti- mization perspective. arXiv preprint arXiv:2306.. 04848, 2023.
Jakiw Pidstrigach. Score-based generative models detect manifolds. Advances in Neural Information Processing Systems, 35:35852-35865, 2022.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pages 5301-5310. PMLR, 2019.
Hannes Risken. Fokker-planck equation for several variables; methods of solution. In The Fokker- Planck Equation: Methods of Solution and Applications, pages 133-162. Springer, 1996.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.0055 2022.
Christopher Scarvelis, Haitz Sáez de Ocáriz Borde, and Justin Solomon. Closed-form diffusion models. arXiv preprint arXiv:2310.1223 12395, 2023.
Yifei Shen, Xinyang Jiang, Yezhen Wang, Yifan Yang, Dongqi Han, and Dongsheng Li. Un- derstanding training-free diffusion guidance: Mechanisms and limitations. arXiv preprint arXiv:2403. 12404, 2024.
Marta Skreta, Lazar Atanackovie, Avishek Joey Bose, Alexander Tong, and Kirill Neklyudov. The su- perposition of diffusion models using the it) o density estimator. arXiv preprint arXiv:2412.17762. 2024.
Jascha -och--iicsttei, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pages
PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Sasre-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456,
2020b. 2020b.
12

---

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pages 3221 1-32252. PMLR, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9(11), 2008.
John J Vastola. Generalization through variance: how noise shapes inductive biases in diffusion models. arXiv preprint arXiv:2544.25332 2025.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.0711 077447 2017.
TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K Ryu. Diffusion probabilistic models generalize when they fail to memorize. In ICML 2023 workshop on structured probabilistic inference {{&) generative modeling, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce and extensively explore a new metric for investigating the properties conditional diffusion models and show how non-denoising processes naturally arise in conditional models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include an explicit Limitations section in the supplementary appendix and briefly reference the limitations in the discussion at the end of the main body.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification:
We provide full proofs for all results in the deferred proofs appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the experimental methodology (including architecture descrip- tions) in-tepth in our supplementary material, in addition to providing the underlying code for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We include the full code from running sweeps and measuring the schedule deviation in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips..
(eglli//gideesCodeeeee
cwteimisssnooooiiiii onPolicy) icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wttpp.ccpppiiiiieeeeeeeepppppppp wctpublic//aeccccccccccccccc /gwii/gs///////eeer /Ca///ooeubbiiicc sereesubmissioon. ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See the relevant experimental details appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification; Most of our plots are qualitative heatmaps or scatter plots. Our line plots depict median values with 30th and 70th percentile values shown across random samples from the latent space/test error for given run.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these details in our experiments appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any models or data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit the MNIST and Fashion-MNIST datasets. All other data and images were synthesized by the authors.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We provide code for reference, but paper does not inherently introduce new assets
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: human subjects were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No human subjects were involved.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used as part of this research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp///errpp/////ee cc/Conffeencees22255 /LLM) for what should or should be described.
20

---

A Deferred Proofs
In this section, we provide the omitted proofs of all propositions and theorems in our main text.
A.1 Appendix for Preliminaries (Section 2)
In addition to estabilishing the characterization in Proposition 2.1, we also show an additional fact that aids in interpretation: the IMCF flow minimizes Euclidean velocity.
Proposition A.I. v IMCF(p) is the unique flow minimizing the objective
L[v] == E2,,, 1x, 2)11";
subject to the constraint that (v.p) is a conditional normalizing flow.
The proof of this statement and Proposition 2.1 rely on the following standard transport equation, for a flow.
A.1.1 Proof of Proposition 2.1, Proposition A.1
Proposition 2.1. Adopt the setup of Definition 2.4. Then MMT IMCF(po) is given explicitly by any of the following identities
U
IMCF
(x,, 2) = - + + = xz, ==
(2.3)
V, ( +2(s))
(2.4)
= =x](s) X, =x
7 2| + (2(0)
(2.5)
where (1)) (ss)
a(*) c(5)
a(s), (s))
a(a) c(s)' (8)) = 2s) - 72(6): = d(s) a(s)
Proof. For simplicity, without loss of generality we consider unconditional flows u, (z) and distribu- tions (x).
We begin by showing that Eq. (2.3) is contained in Vp)), reproducing the proof Albergo et al. [2023), Proposition 2.6, using characteristic functions. The characteristic function g(s, k) for ps (x) is given by:
g(s, k) = Ee (al))Xoto())x1))
= -f "p
Taking the time derivative, by the Liebniz rule we have,
f et *..ps<xdd= 0.g(s,h) k)
Elik (d(s)Xo+o(s)))) (axxx))))
= -ikT Bla(s)Xo + + = xjeih (x)dx
= =ik f (0
Note that for differentiable scalar function f R. + R such that lim f(x) 0, integration by parts yields the basic property of Fourier transforms,
ikn df
(x)dx== lim (f - 1 f(x) d dx
e dx dx
de
-ik f f stoddd
21

---

Applying this in the above, we have
Σ iky / e ()),))d)))
ik (1())s)))))
= =- Σ
ui Ox
1y, (z)
- 1 | Σ Fe lus s)pd(()) dzj
/ ck V (x))
We can conclude therefore that,
(z) (v.(2)p((x)) (C O,
meaning that (v,p) constitute a normalizing flow per Eq. (A.2).
We now proceed to additionally show that Eq. (2.3) the minimum- norm norm such solution. Consider any perturbation de such that v + de remains a solution of Eq. (A.2), i.c. V (6),(())p)) = 0.
We begin by forming the Lagrangian:
inf sup L(e, A) = / [lul)pa(x)dd + A(x)VV (up,) + (=p)(r))(z)dz
We now apply the optimality condition and use integration by parts to obtain:
0 = L( + ōv, X) - L(v, A) O(1óv[1²)
f 8v (x)( / {(r:VV (dvps)ds
4 8'
/ Oo (pp)( + / V V\(x)dæ
480
pc(r)o0 (0 VX(z)dr
vdv
V = VX))).
This implies that flow is optimal provided it satisfies the constraint (xp.(z) + (up)(z)) and there exists a À such that Ass). i.e. if v is conservative. Therefore all that remains is to show that is in fact conservative. We use Tweedie's 's formula to rewrite D:
U. (x) E(i(s)Xo + + = 2]
((s)
E(d(s) + o(s)
Xx, - xo)s) = x,
((s)
E[XolX, = =z, == ((s)
(((s)
((s) (())
als)
0(s)
(z ( VV logp, (z| + o(8)
((s))
o(s) a(s)
a(s)
(ss)
a(s) a(s) (I (I o(s lolp) + ((s)
a(s) ((8)
a(s)
a(s) (s)² -(())((s) V. V. (zg + a(s)
a(s)
a(s)
This shows both the equivalence of Eq. (2.3) and Eq. (2.4) and that Eq. (2.3) is conservative and hence, the ideal flow.
22

---

A.2 Proofs Regarding Forward and Reverse Processes (Appendix B)
Proposition B.1 (Stochastic Generative Processes). Given a conditional flow (0, p) and conditioning
value E E, let 00, 1] + Rzo be lime-dependent noise scule. Use { )e and {X.") / .||Z denote the forward and reverse processes where Law(Xg Z 2) (-)-2)) (aw(X))
2) pı(+12) and _ ,x, are evolved according to.
lzx [x, 2) vo) + V 2e(s)dB.
(B.1)
x Z) = 1x 2) + c()) logps [X, 2) + 22(s))BB
(B.2)
where 8 1= s and Bug B3 are standard Brownian noise processes. In particular, x X these processes satisfy,
Law(X = = Law R = =) = pa(-12)
Proof. For simplicity we only consider the following unconditional forward SDE.
ax. f(x. s)ds + /
(A.1)
Let (x) be the continuous density associated with (A.1) with Xo po(x). The Fokker-Planck equation [Risken, 1996] yields,
dz
0
0²
0s (x))= = Σ ((,xx))).(z)) + <(t)
11 (x())
dz,
dz,Ox,
i
82 llog
Σ
If s)pa + e(t)ps (x)
dz:
dxy
V [f(z (1x,s)p.(0) (x) (1),, logr,
Letting f(x,8) = v2(2) ((s) V log P. = 0. we have
0
0s Ps - (x)
Which we can see is precisely the transport equation Eq. (A.2) for the (u,p) p) flow. The reverse follows similarly:
0s Ps V (vs()))p(=)) V. (-vg(=)pa(z)).
Let + ($) Pz Then,
0s Ps - (g(x, 5)pa(x) - (ps)))o(())
Thus we can see that,
dx 9(x," 8)ds + / (e(
A.3
Proofs for Section
A.3.1 Proof of Theorem 1
The proof of Theorem is a variant of the performance-difference lemma, adapted to the control of the solution to PDEs.
Lemma A.2 (Finite-Horizon Deterministic Performance- eeffrrmance-Differeneee Lemma). Consider states a S and actions E A and a continuous-timg time dynamical system á(t) fi(z(t), u(() defined over
23

---

t E [0, Ti for some T > O. Let (x, t) to denote feedback policy π S x [0,T] - A and μ be any finite positive measure over [ T)), the Borel o-algebra on [0. T Define,
Vi (1) = ( (r(x" (8), < (<)(())))))))) (+)))
where x" (=(ss) = f.((((),, (T((s))) x (41) = x.
(x,,t)
ift +e
12 (I,, x') = lim (Venta (4) Vt (x))
where te, (a, (t(,()
c-0
ift> 4t
for any p-integrable function r. (2(x" π (x" (s)). Then, for any 11 E [0, T) E and policies n, n²,
V (x) (er 1/ A7 A7 (1),
Proof. Consider any c E (0 t2 - -1). Then,
V ti (x) V (x))
Vi (())) V Vistota (x (11 t]] V (1)+to()) Viste (x + +c)) Vi (x" (h1 + +c) Vir titett (x= (11 + ())
+ Viitetig (xt (t1 + c)) Vi titri (x (t1 + c)) + V Vott)t(()) Via Vi (x)
Ve' Vittee (I*
(+1 + +)) Virtes (x" (t1 +e)
Vhen (x) Vi, (z)
Choose c:= 12-11 for some K 2 O. Then, recursively applying the above identity yields, K
K-1
V (x) 1/7 ViT (x)
Σ
tkee (a (11 + kc)) VTs (x (41 + kt))
11.12
litket2
k=0
Taking the limit K + 00.
K-1
Va' (x) - VH,,a(()) (x) = lim
Σ
[A7 (41 + ke), x')) o(c)]
K0000
k=
S. (z) x'),
Notation. We use p to denote the solution to Eq. (A.2) with e and initial condition P3
Lemma A.3 (Diffusion Performance-Diffen Differrece Lemma). Let vs be some time-varying functional defined over E [0, 1] which maps continuous densities to scalar values. For any finite positive measure μ over B(, 1)), define
('*) = [ re(pij))d(())
1 V.. (p*)
lim c-0 c
if ift S Ut otherwise.
where w/
Then the difference between V. (ps) and (px) can be written:
V. (p.) 1 (p.) S A (Vis: v')dt
Proof. This is just Lemma A.2, where S C¹ C(,,R, densities over X, actions A are maps X + d2 and policies are flows v 00, 1] X + L
24

---

Lemma A.4. Let Ps.P', P', be densities and u be a flow such that (Po, w). (P/s v) satisfy Eq. (A.2). Then for any Ps. [0,1]
- (ps, S 2TV(p,
Proof. Fix any 8, [0, 1] and let TV(pt,P)). Without loss of generality assume that (.). 2 TV)...P))
Decompose pi (1 a)p: + apr. where Pe is signed density such that f [ = 1 and (fa, v) satisfy Eq. (A.2), Since Eq. (A.2) is linear in p, we can write p/g a)ps + oń, for s. Thus,
TV Ps) S (1 TVVV to
Combining, have that
(TV('s.P.) TV ps) S (1 aV) oTV((() TV(P,,
< <l-aTV(p..p.) + aTV((p..) 5 2a.
Lemma A.5. Let (Ps; Us), (fe. 6.) be pairs of solutions to Eq. (A.2). Let (Potp Vs) be a solution to Eg. (A.2) such that Pelt Pt- Then, for any i, 8
1 TV 66) 2 (V TV
2
Proof. Applying Lemma A.4 using p', = Pale: P. = p..
2TV pqp) 2 [TV p's) TV Since we chose Pijt pc, this yields the desired statement
TV(),,
[V(p. - TV(p..fa))
IV
Db) TV(p,,
Theorem 1. Consider the setting of Definition 1. for conditional flow (v.)). For any probability measure μ over [0, 11, the total variation distance TV (q.p) == 1 1p- q dx between pa and pMcc over p is upper bounded by integrated schedule deviation:
f TV(p=(-12)) IMCF
(...)).)((()))))))))) (-12))du(s) & L. SD(E; 2, f)dt SDiota 2).
Moreover, if (.R 1.a.Ptph 10ppl S M < co, there exists constants cu E [0, 1/2], 2 depending on M such that, for all 0 < S c s [0,
sup 10ls
TV INCP (+)=)))))))))))>>))))))))))))))) 2 ocS((,;,ss).
(3.1)
Proof. We omit without loss of generality and consider the value function (p=) given by
('.) - 1(t [} TV Pt IMCF dd((t)
The Diffusion Performance Difference Lemma Lemma A.3 gives the upper bound
(V(pi.PMC=)ddpe Vo ((o))) ViOi (po)
S(
25

---

Expanding Ap ()McF)) (
1 / lll [8,8 + te} 1-TV s. "! (ddu(t)
lim A (olcer)
-0
(TV / 1112 ( (MCE) (pq.. Pt JNCF )du(())
Applying Lemma A.4, we can bound the integrand in the second integral by TV P8tte (MCP)
(p 2 (MMC)))
- lim
sup c [E|s,ete]
2TV (Pts:: MM
Taking first order expansion, we have that for small
IMCF = 2 / TV (P Pote)
Op Os
McF
dæ + o(E)
Os
rms
Thus (v; (p | & SD(U; 2, Note that the proof also holds for the time-reversed direction since pl pfaci This yields the upper bound,
/ (V(). IMCF (()))) - /. 2D,
Lower Bound: For the lower bound, assuming that |002 |0. @.[0aP"jele=sl and 10ppl all
bounded by M, there exist constants > 0, E 1/2 depending only on M such that for any,
E [0, t,t [s co, s col.
TV pl 2 30E /
Öptta Øt
0p IMC
dx = 3ceSD(e;
Ös
t=s
Fix any (0, 8 E 10,1-
Case 1: TV PMC S ceSD(W, 2, Pick any e' E [--c,e). By Lemma A.5,
TV(pste.Pot IMCF
TV P (MCF) - (p..,.1. JMCF)
[3ee (S('SD( ceSD(u; 8)
(eSS((;;,,s)
Thus,
sup TV TV(pste.Po 2 conSD(v; s). ('(00coo]
Case 2: (V.
2 ccSD(v, 2, s). In this case we trivially have
sup TV(pete Potc) 2 ceSD(v; s). <'E(
Note that since the argument is symmetric with respect to time, for s 2 co. we can consider [-c, OJ. Thus for any [0, e) [0,
sup 1E(s-6,0te
TV(p,, (NCF) 2 ceSD(u;
A.3.2 Proof of Proposition 3.1
Proposition 3.1. It holds that SD(;; 2, Eppes | (v, D.NCP) + (v. D.CCT) V log [MCP]
26

---

Proof. An equivalent condition for the pair (p. u) to constitute a conditional flow is that it satisfies the differential transport equation [Albergo et al., 2023],
O.p. + V [ox =)p+() O.
(A.2)
This permits us to compute
SD(e; 2,
/
Op
s/t
0p
dæ
V [upp
V
[U! Ps
pe dæ
8s
0s
t=
V (2's IMCP)
+ (2', - US
V logp pM
p.acr (x))dr
x
Eygnr
V (we
U/2
(NCP)
+ (we Us
INCP)
logps IMCF]]
(A.3)
A.4 Appendix for Extrapolation Behavior (Section 4)
A.4.1 Proof of Theorem 2
Theorem 2 (Discrete-support Smooth Interpolant). Let Z be supported finite set Sz [20 C for distinct z(1) < < (N,,, ordered without loss of generality. For each 200 E Sz, ler (x) == El + o())x11Z = =(0) Then, there are piecewise cubic polynomials (z), with pieces defined by the intervals [z(5), 2(1++1]]
[2(0),((++1] such that
/ 11V2ol2 = (z) 22,,))))2
(x, = arg inf vEarg inl, L(u)
In the case where |S2| = 2, (2) are linear functions.
Proof. This is the cubic spline interpolation, which traces back to the classic work of Kochanck and Bartels [1984], but we apply here in function space. The Euler-Lagrange equation for the functional
J()) = [ (z) dx is:
da
f(x = 0 when E (21,
dx
Thus, applied to our setting, we have,
az4 us (2, =0 Vz E ( (0, X,
((++1)),
0, o..,..., N, where for convenience we let 200) -0, 2(1+11 +00. This means for each 0, that on each interval E (2(), 2((+1),, E (l....,N - 1}, the interpolator us (x, 2) can be written piecewise cubic polynomial in = of the form
(z 2) = a (x) + b1 (z) - 20)) + (z)(z od (z))(x)(2= (2)))
(x)
where for E (-xc, 22) we let (2 = (,, + (1) (x) (=- (2) and similarly for the interval 2 oo), we have (2, (2, + (x)
(x) (z (()))
Let Az = (i+1) and using a00 6(0), c( ( as shorthand for ( (z), 6.00 (z), cs df)
have the following boundary conditions for the endpoints:
( 1 2x, Vi N -1} (0) + + AZ) + (0) A A (x ( ((it1))
Vi E (I....,N -
and additionally, with boundary conditions ensure the first and second derivatives match between the different pieces
b (0 Az + Ad( (i+1) Vi (1,.., - 2]
(0 6d A2; (++1) Vi E f1,...,, 2}
and we additionally constrain the second derivatives at the endpoint to be zero so that c() 20(0) + (d Az = 0.
27

---

This yields linear system with 4(N - 1) unknowns and equations. In fact, Bartels et al. [1995] shows that this system is guaranteed to be linearly independent.
Thus, can write a 6(1), clt), d' "s as linear combination of the o² ((1)ss Therefore there exist piecewise polynomials y( (ar) such that,
v(2,2) 2) (2),
A.4.2 Proof of Theorem Lemma A.6.
L Ex,x, 10. (X,, Z) ox 2)11 L'' / ||u((x, -1( 2)) (2, z)drdzds
(A.4)
Theorem 3. Fix some diffusion schedule (a, σ) and let vi (z, 2) be the IMCF flow associated with p" (xlz), ie. the minimizer of Eq. (2.2). Assume that p" (2) a uniform density over some set S, i.e. (2) = e. is for some >> 0 and where 1 the characteristic function of S. Then the minimizer to [a for any E given by,
2
Us 2x
(x,,2')dz'
(4.2)
JE 1 +1E/1"
Proof. In light of Lemma A.6.
(1la((e,)) [
Tx =
(1, =)1 -)
Au,V =) 2) (err 2)ds.
(A.5)
Notice that the loss decouples across & [0. and x E X. Hence, let us consider the optimization problem for fixed. Set w(2) PIMCF(z (x, z). f(z) v,(x,2) 2) and (z) (z 2). Then the corresponding optimization for 8, x fixed becomes
/ [(IS))) - (=))]
(A.6)
Since w(2) = c- is is uniform, the Euler- LagranLa equation yields that for & E S,
20(s(=) (2)) 2cA Σ
(A.7)
0220022 in
Thus, equivalently, for all & E Z,
n
g
f(z) 1s
f(z)) 1s] (2) 1s
02,023
1.
We may therefore apply a Fourier transform to obtain
F(=)+ + = (=)
(A.8)
where F(2) and (z) denote the Fourier transforms of f Ls and f ig respectively, and where invoke the conversion between differentiation and multiplication under the Fourier transform. Solving for F(2), we have
F(2) I+
(*)
(*())E 2mig= dz'
(A.9)
1+ ALETT
Inverting gives, for any = E S.
f(z)
2Tiς=
1 (=)e 2migs' de'
(A.10)
1 ALE
2mi5(2=22) (+
(A.11)
G.Z'S
1 A
28

---

B Sampling Algorithms and Schedule Deviation
The sampling process Definition 2.1 can equivalently be described using the following stochastic differential equation (see Appendix A,2 for proof):
Proposition B.1 (Stochastic Generative Processes). Given a conditional flow (v, p) and conditioning
value 2 E 2, let e [0, - Rzo be a time-dependent noise scale. Use {x= )e([0.1] Z and
{x 1 [ to denote the forward and reverse processes where Law(Xo Z = 2) = Po (-(+)) Law(X) 1 - z) p1(+)z) and , are evolved according to.
d(x;1 () = [u,(X, Z) c(s) 1o, + 2c(s)dB,
(B.1)
d(x: [) = ( Z) + logp: Z) + 2e(
(B.2)
where 1- and B,, Be are standard Brownian noise processes. In particular, X , these processes satisfy,
Law 1 Z = 2) = Law (x = = (-
Sampling Algorithms with IMCF Flows. Given Corollary B.1, the continuous-time analogous of the sampling algorithms we chiefly consider (DDPM [Ho et al., 2020], DDIM [Song et al., 2020a], GE [Permenter and Yuan, 2023]). In particular, we note that DDPM/DDIM thus should sample from the equivalent distributions (in the continuous-time limit) under the assumption that the learned flow is IMCF:
Corollary B.1 (LMCF-based) generation). Given an (a, -IMCF flow (p.v), for any c: [0, 1] + R+, using Eq. (2.4) the forward and reverse processes Eq. (B.1), and Eq. (B.2) can be written as,
a(x.1z)= = 1 <(s)
(2(s)
Us (x Z)
X
ds + 2e(()BB...
(B.3)
(1))
(8))
(( c(5)
((((5)
72(5) x
d(x!'lz) =
U/ (X Z)
d8 + V (c
(B.4)
(S)
(8)
where 71 (8) = - and 72(8) = a(a)
Example B.1 (DDPM [Ho et al., 2020D) The SDD-analogue of the DDPM sampling algorithm corresponds to the choice c(s) -7 (s) (note that >1 (s), (s))) < 0) making the forward process -eaeeinnUhlenbeck process and the reverse process simplifies to,
independent of v and thus a purely
72(())
R d(x R 2) = -20 (X 8
8 2) +
X R d3 + 2e
(B.5)
Example B.2 (DDIM [Song et al., 2020al). The DDIM algorithm strictly generalizes DDPM and technically allows for any choice of c(s) 2 0. In practice (e.g. Karras et al. [2022], Chi al. [2023]) DDIM is used in a "noiseless" fashion with e = 0, in which case the reverse process simply becomes the regular flow ODE,
d(x = - (X 2)ds
Example B.3 (Gradient Estimation (GE) [Permenter and Yuan, 2023]). The Gradient Estimation algorithm is variant of DDIM which introduces correction term based on the previous estimate of v, i.e. it uses the filtered flow ,(x, (2) = (u(r,,)) + (1 ())((++6512)) where 8s is the discretization interval of the SDE. Note that in the continuous time limit where ōs O. we recover the standard DDIM. We use f = 2 for the experiments presented here.
B.1 Schedule Deviation Emerges from Linear Interpolation
To examine how schedule deviation can emerge from linear interpolation, we consider combining normal distributions with differing means and variances, respectively:
Lemma B.2. Consider two scalar normal distributions, (x) N(m, and p(o) (x)
N NH d²). W For a given a Diffuion Schedule (a, o.), the associated score functions of the distributions
at time s are,
-- - a r² o2
x- ==a(s)pp
V log (x)
V logg!?(())
-2 02
29

---

and, for any e E B, the combined score function cV Vogy's ( (x) + (1 V log px (x) is also consistent with the Diffusion Schedule (a, a)::
x. (c) + c))u))
cV (ogg + (1)
σ² + 0
We can recognize this as the score function for p(z) N(eu + (1 c)/22 o), interpolated in accordance with the Diffusion Schedule (a,a). a).
Proof. The proof follows by straightforward substitution.
Lemma B.3. Consider two scalar normal distributions, (x) oO) and (z)
(0 o²) for k 2 0, & 1. Then, given Diffusion Schedule ao, the associated score functions of the distributions time are.
v log Foogp!" (x)
(2 +02(s)
a²(8)
σ² + 02
V (x) 12 08())
a²(s)
62
where (ss) o(a) given by.
Then for any E R \ {0, 1}. the linear interpolation of the score function is
a(s)'
cllogg, (1) (x) + (1 V Vogg 12) a?(s) +2+ c)(k2 - (s)
8² 8²-0" (e)
where (s) [T+c(k] 1))o
Proof.
(1 - ()2 +c)))) +
+
cV log + (1 - v)V log (x)
(2
(K202 o0))
1
(1 (1=
(8)
(k272 (+)
(1 - c+ c1-
a (2(s)
+ (1+)+++
x
k2044( (2)))
(a4(1 (c+(lo))2200)
x
(2())
(1- +3*(()
1
(s)))
Letting (s))) = k =k1"""tt(0=0kk"""0bb0
(I=ctok'13'4o"00 we can see that the linear combination is consistent with the Diffusion Schedule (a. only if (s) is independent of s.
(s k2 + (3+
(1 +2)2
(1- [
1302+
5²
(1 c ck ck2) 02 + - (1- (c-1+(( c))2))2(s)
x²
cI-c+ck2)p (s))
(c 1+ (1-1+))k)k))) (c- - (0
= 8² +0²
(1 (1-c
0² 02(s)
= = -22+0 + + ck2 -1)
(1 c(k - 1))6² 02
30

---

Thus,
cV logp 1 (x)) (1 c)V log ,, (x)
(2(6) 0² @2 c)( 1 -)
[T+ck x ا2
+ (s)
1110 +))00
We can see that for any fixed s, the combined cV log logp!" (1 c) log pa resembles the score function of normal distribution, but the dependence of (s. on in the denominator indicates that it is noised according different schedule from (a, a.)
2)
Taken together Lemma B.2 and Lemma B.3 suggest that schedule inconsistency can arise through differences in variance, but not simple transformations such translation.
C Experiment Details
All experiments were performed using a cluster of 4 NVIDIA A100 GPUs and took approximately 100 GPU/hrs of compute to train and evaluate all visualized experiments. Combined, all experiments took approximately one week worth of GPU/hours to train and evaluate.
C.1 Conditional CelebA Experiments
We additionally consider an ablation on the CelebA [Liu et al., 2015] dataset, using t-SNE of the 40-dimensional conditional attribute space conditionally generate 64x64 images, described in Appendix C.5. We use training datasets of size N E f50000, 100000, 160000} for our experiments.
Notably, for these experiments omit the divergence term from the Schedule Deviation computation (i.e. setting 1i = 0 in Algorithm 1). This is principally for computational reasons-we do not have enough memory evaluate the divergence.
In Figure we visualize the corresponding conditioning space, highlighting 3 different conditioning at- tributes (Male/Female, Young/Not Young, Smiling/Not Smiling), well as the OT distance/Schedule Deviation over the space.
Similar to the MNIST, MNIIST and trajectory datasets, we observe little change in Schedule Deviation based on the amount of training data used. However, unlike the MNIST, Fashion-MNIST and Maze datasets, we observe no correlation between Schedule Deviation and the computed Optimal Transport distances, despite similar overall structure in the OT distances and Schedule Deviation as visualized in Figure 9.
We hypothesize the lack of correlation is related the relatively uniform coverage of the dataset over the conditioning space and the much higher dimension for the generated space X. Notably, we also observe very little variance in the OT cost. By contrast, MNIST, Mashion-MNIST, and the Maze experiments all have very non-uniform coverage over the conditioning space and exhibit much lower noise (i.e. smoother heatmaps) for the Schedule Deviation estimates.
Overall, we believe these experiments are somewhat inconclusive, partially due to the modified methodology (using r1 0) and the lack of significant structure either for the OT distances the Schedule Deviation.
C.2 Measuring Schedule Deviation
For simplicity we consider Diffusion Schedules (Definition 2.3) which are purely noising, i.e where ass) 1. In practice assume that (0, 1), any such schedule can easily be normalized into the standard form via the transformation x I+o[s]
Thus, in this simplified setting, the loss simply becomes.
L, := E Ex,elllu,(x (s)c, 0()cc)1
(C.1)
31

---

where p (z) and ~~ (N I). Under this framework, per Eq. (2.4), the minimizer to Eq. (C.1) can be written o (x, 2) == ((s) log DM (2. For convenience, we use the "--parameterization" of the flow introduced in [Ho et al., 2020], wherein (z( 2) - (
Schedule Deviation Estimator Thus, we can estimate the schedule deviation using,
SD(u Eppcr [V (V/- ((MCT) (V/- U/ (OMCT) V
log Ps IMCP
((( Eppies V (E. - 6x e(MC)) )- + (c3 (E
Ex) (
JMCF
Eypur [|a(s) (E. - - (MCF) ((s)
(Eg
IMCF Ex)
IMCF
o ((8)
Empirical Estimation of IMCF The quantity e, above is directly parameterized by the neural network and we can sample from Ps (xx/(=)))) by generating samples from po (z/|2) (using the
designated sampling algorithm) and then noising to time using the forward process. Estimating & IMCr less straightforward. Here we use that for N samples (x((9) Ji=1 from po(2)2), we can approximate V log p.MCF (xl 22) and therefore JMCF (using (V p. oF) to denote the Gaussian PDF with mean j and variance o²):
N N Marz!00,0 (s)))
IMCF (I, 2) 2 V log
im
1
|| ||0- x01
(==(6))
-((8)
N
| أ(ا
exp
20
(o(ss)
2i=j i=
exp
2
exp
(x)
x(1)
Eラ=1 exp
N
(1--000 ( 18
a(s)
2x)
In the MNIST, Fashion-MNIST. CelebA, and Trajectory experiments N 128 whereas for the toy experiments in Section 4 we use N 2000.
Empirical Estimation of V (e, c .MCC). Computing V (e. - (MCF) requires computing the divergence of a neural network, i.e. the trace of the Jacobian. For a function ʃ R - IR",, computing the divergence requires Onn) Jacobian-vesd vector products, i.c. it is computationally expensive as materializing the full n x n Jacobian. Thus, we consider instead a randomized approximation to the divergence wherein randomly sample a standard basis vector (i.e. an element of the diagonal of the Jacobian) to use as an estimate of the true divergence for each sample x'a ~ p (#.|z).
Schedule Deviation with Log-Linear Schedules. In practice (see Appendix C.5) we use a log-linear noise schedule (t), where (s) = c1e and C1, C2 are chosen based on the desired (0)) and (1).
i.c. (0), = log( (og(a(())/4) (1)) Thus, as (s)) = C20(s) we can write
SD(,; 2, C2 (lo(s) (l))) (c.
(cccr)
(MCC) [c.M (x) .
IMCF
(C.2)
For simplicity we compute and report SD(E; 2, s) using C2 = 1. This is such that the "schedule deviation" " at given noise level (s)) can be computed independent of the upper and lower bounds on
Empirical Estimation of Optimal Transport Distances: We use N 128 samples from each sampler, for each conditioning value, to estimate the 1-Wasserstein (i.e. earth-mover distance). Computations were performed using the Python Optimal Transport toolbox. We used the exact LP-based solution, opposed to e.g. entropic Optimal Transport using Sinkhorn.
32

---

Calel Conditionng
Fanalo:
Fanalo: Sniäag
Fenalo, Saiiing
Malr, Smils, Not
Mule.
-00
Mula.
Mule.
-100
Compament
OelebA SD OT Distance
CalebA SD vs OT Distance
CalebA SD va OT Distacce
DDPMADIN OT Detanne
DDPXICE O2 Discanee
32.5
28
27.5
24
N-50200
22.5
N-50000
N-100000
N- 1G0000
N-
N-000000
20.0
N- G0000
16
30
35
40
45
35
40
45
50
30
35
40
50
65
60
DDPM Schedula Deviation
DDIM Schedile Deviation
GE Schedule Devintion
CalebA 500000
CelehA N=100000
CelebA N=160000
100
25
DDPMIDDIM OT Distance
50
24
25
23
22
21
-75
-10
100
Secord Component
75
DDPM Schedule Deviation
50
46
-25
-50
-100
1-SNE First Component
t-SNE First Component
5-NN First Component
Figure 8: Similar the MNIST/Fashion-MNNES datasets, visualize the Celeb-A condition- ing space (top) and show the clustering of different attributes (Male/Female, Smiling/Not Smil- ing, Young/Not Young). Although there some similarity visually between the OT distance and DDIM/DDPM OT distance, the correlation between Schedule Deviation and OT is weak. We discuss these experiments further in Appendix C.I.
33

---

C.3 Closed-Form Interpolants
Note that we can write the flows 1² (x, 0) and (=(x, 2= = 1) under Eq. (C.1) as
(z) +1)
0" (0 0.12 + (2)
( - 1)
=x) =)= =-
(x 1)a 200.1 2t0.12 (z)
0.12 + 0² (2
20 11
210.1 (4))
σ(s) 0.13 + 0²(s)
(e-1)2
(1))
(0))
Discrete-Support Interpolant. For the discrete support dataset (Figure upper), we use the interpolant based on Theorem 2:
v(x, 2) 2)=
(z, -1-2)-0""z,
0) + = v²(x,==1)
Continuous-Support Interpolant. For the distribution with continuous support (Figure 7, lower), take inspiration from Theorem 3, which suggests that for uniform densities, the flow v(2, 2) should be convolved with the kernel
[
2
+ X||E|| de.
(C.3)
We use the approximation F-1 as the Fourier transform of associated experiments. Thus, we use,
2rite T+
(1+225+)877- We note this has the same tail behavior, ga In particular, we use e 1.5, C2 16 for the
[ell attenuates with 1
7(1 - x) 1² (2, v((x2)) 2) = 7(x) + 17 (2, ====0 + 7(x) 71 x)
7(x)
where y(x) 1.5(1 + 162z))3/22
C.4 Datasets
We visualize the conditioning spaces for the datasets in Section 3 in Figure and show the conditioning space for the CelebA dataset in Figure
MNIST/FashionMMNISS For the MNIST and Fashion-MNIST datasets, we t-SNE [Van der Maaten and Hinton, 2008] the images to construct the two-dimensional latent spaces seen in Figure 1. We used SciKit-Learn implementation (which uses Barnes-Hil Hut style approximation for large datasets) with perplexity of 30 and an early exaggeration of for both MNIST and Fashion-MNIST
for the CelebA dataset, except that we (a) downsample the images 64x64 and (b) t-SNE the 40 discrete attributes provided by CelebA (using a hypercube), as opposed to the images themselves. We visualize the resulting embedding in Figure 8.
CelebA:
We use a setup similar to the MNIST/Fashhon-MNNNI
The maze dataset (Figure left) consists of maze solutions from different starting points to the center of the maze. We use the fixed maze depicted in Figure 1. The row of the starting cell is picked uniformly at random and the column c (indexed at 0) is picked with probability p(c) x ee/2 (7--e)/ i.e. we are more likely to pick starting points near the end in order to start further from the goal point. For each starting point,
Maze Solutions Dataset:
randomly sample solution based on its length compared to the optimal solution such that p(s) x e (E(()))((("") where ((s) is the length of s and {(s*) ) is the length the shortest path to origin. For given solution, we then construct a Bezier curve which fits control points along the trajectory which have been slightly perturbed by noise w (0; 0.04). We then take 64 evenly spaced points along the Bezier curve use this as the final "trajectory" in the maze which we attempt to generate.
34

---

C,5 Model Architectures and Hyperparameters
For all experiments used the e-parameterization introduced in Ho et al. [2020] and a "variance- exploding" setup for the Diffusion Schedule as detailed Appendix B. particular, we use log-linear noise schedule where d(s), cie with 512 training timesteps (and 64 sampling timesteps) ranging from o = 5 x 10 to a = 5 for the experiments in Section and o = 0.01 to 35 for the CelebA experiments.
For the toy dataset in Section 4, we used 1024 training timesteps (and 128 sampling timesteps) with noise values ranging from = 8 x 10-3 to 10.
MNIST/FashionMMNST:: We MNIST and Bushion-MNISTY experiments. We use the same UNet architecture described in Dhariwal and Nichol [2021] with a base channel dimension of 64 and using ResNet blocks per down- sampling/upsampling step. We used GroupNorm, with 32 channels per group, for the ResNet normalization layers.
U-Net with 5.9 million parameters as the "default" for the
After the first downsampling step we use 128 dimensions (2x the "base channels"). We additionally include an attention block before the 3rd downsampling layer.
Conditioning on the time 8 and conditioning value is performed via first embedding each into a 256 dimension (4x "the base channels") into an MLP and a 2 layer MLP. The time is fed in using sin/cos embedding.
In Figure 5, we show an ablation where we increase the base channels to [96, 128, 160), corresponding 13.3M, 23.5M, and 36.8M parameters respectively.
For both MNIST/Fashion-Mi MNIST we train the model using Adam (with weight decay 1 x 10-4) and cosine decay schedule with an initial learning rate of 3 x 10 over 300, 000 total training iterations and batch size of 256 samples.
Maze Solutions: For the Maze solutions, we consider similarly constructed UNet to the MNNSS/Fashion-MNIIST, but using 1-D convolutions instead of 2-D convolutions. Our training parameters are also similar to the MNIST experiments, but we use instead 100, 000 iterations, an initial learning rate of 5 x 10 4, and a batch size of 128.
Toy Data:
For the toy datasets in Figure 6, we consider 5 layer MLP with a hidden dimension of 64, imput dimension of 2 (value + conditioning) and output dimension of ("denoised value"). The time value is first embedded using sin/cosine and then mapped to 64 dimensional vector. For each layer in the MLP, we modulate the activations using FiLM [Perez et al., 2018] conditioning scheme.
For training we use AdamW with 10, 000 iterations, cosine decay with an initial learning rate of 4 x 10-3 and weight decay of 0.01. We batch size of 128 and generated synthetic datasets of size N = 100, 000 samples as described in Section 4 and shown in Figure 7.
C.6 Full Main-body Experiment Set with Additional Samplers
For completeness, we include additional visualization for the DDPM Ho et al. [2020], DDIM Song et al. [2020a], and Gradient Estimator (GE) Permenter and Yuan [2023] sampling algorithms (described in Appendix B) for each of the MNIST (Fig. 12), Fashion-MNIST (Fig. 13), and Maze solution (Fig. 14) datasets.
Additionally in Figure 10 we show scatter plots analogous to those in Figure for all sampler/dataset combinations we evaluate. In Figure 11 we show an ablation over training samples and per-class Schedule Deviation distributions for the Fashion-MNIST dataset.
D Broader Impacts and Limitations
We believe that Schedule Deviation is an important step towards understanding the generalization behavior conditional diffusion models. This may yield insights into downstream phenomena such as hallucination and the synthesis of completely "novel" samples. The insight that conditional diffusion models generally do not denoise also has implications for the design of future sampling algorithms, and more broadly cautions against making strong assumptions on the properties
Broader Impact:
35

---

of learned models in generative settings, irrespective of the original objective inherent in the training loss (in this case, for the model "denoise").
We hope that these results will motivate greater theoretical and empirical study of non-denwoising flows and, specifically, phenomena such self-guidance. The exact effect of classifier-free free guidance remains poorly understood, despite widespread adoption and deployment. This work highlights that better theoretical and practical understanding different flow composition rules can potentially yield insights into the behavior of trained models.
Our proposed metric, Schedule Deviation, has number of limitations. Namely, it requires sampling from (2]2) (i.e. running the reverse process of chosen sampling algorithm) and necessitates estimating both (1) the divergence of the neural network and (2) the gradient of the pn(2)(2) noise distribution. Estimating the gradient for small noise levels can require a large number of samples do accurately, the variance of the V Vlog Ps (arlz) estimator increases as O. Furthermore, computing the divergence through the neural network using back propagation as requires Jacobian-
Limitations:
expensive in practice as computing the full n n input-output Jacobian, vector-produt queries to compute. Both of these computational bottlenecks suggest that computing our metric may be difficult for X' of very high dimensions. However, with greater computational resources and alternative methods for computing the divergence, these concerns may ultimately prove negligible.
We demonstrate feasbility on a toy Canditional-NNISS (d = 784) problem, but note that even in this setting, computing the Schedule Deviation for the several thousand points needed to create the heatmaps seen in Figure 12 took multiple hours per checkpoint, longer than the time to train the model itself.
Lastly we note that in this work we consider conditional diffusion three particular contexts: a low dimensional "toy" environment, maze solving dataset, and conditional image generation dataset. We hope that the trends we have identified hold other domains (e.g. audio or video synthesis) but have not thoroughly investigated precisely how universal the Schedule Deviation is in other settingngs. The evidence we present does suggest that the behavior of diffusion models should not be taken for granted.
36

---

CelabA 5-50000
OslebA N- 150000
1-NN Second Companees
12
H-NNS Seorchl Comprnsnt
I-SNN Swoood Componmnt
6-SNN Secord Compotent
+-SNE Second Compenent
M-NN Swornd Compomen
44
-75
-25
25
75
-75
-25
25
75
-75
-25
6SNE Pirst Comporent
+SNE Pirst Compoment
L-SNB Piint Component
Figure 9: Analogous Figure 12, visualize the Schedule Deviation and OT distances for different choices of sampling algorithms.
37

---

MNIST
Fashion MNIST
Maze Solutions
DDIM/DDIM OT Distanos
0.4
0.3
N=10000
1=00000
N=0000
N-30000
30000
0.1
N-4000
N N=60000
N- 60000
N=8000
50
100
150
200
100
150
200
250
300
350
80
100
120
DDPM Schedule Deviation
DDPM Schedule Deviation
DDPM Schedule Deviation
MNIST
Fashion MNIST
Maze Solutions
DDPM/GE OT Distance
30
0.4
4
0.2
N=10000
N=10000
N=2000
69
N N-30000
N-
N-4000
N-60000
60000
N-8000
0.0
100
300
400
80
120
GE Schedule Deviation
GE Schedule Deviation
GE Schedule Deviation
MNIST
Fashion-MNIST
Maze Solutions
DDIM/CE OT Distance
0.4
0.3
*
N=10000
N=10000
N-2000
N-30000
N-30000
0.1
N-4000
N N-60000
N- 60000
N-8000
50
200
200
400
100
DDIM Schedule Deviation
DDIM Schedule Deviation
DDIM Schedule Deviation
Figure 10: Optimal transport distances vs Schedule Deviation using Po corresponding to the DDPM, DDIM, and GE samplers, for each of the MNIST, Fashion-MNIST, and Maze datasets.
Dashion-MNIST Training Samples vs SD
Fashion- MNIST SD by Class
300
DDPM Scbedalo Deviatioon
220
DDPM Schadule Deviation
250
180
160
140
40000
60000
Fishirt
Thauser
Pallover
Dress
Sandal
Shirt
Bag
Training Samples
Figure 11: Ablation over training samples and per-class Schedule Deviation for Fashion-MNISTT For the left, 30th, median, and 70th percentiles are visualized for z sampled uniformly over Z.
38

---

TONE second Componest
M POPMBDMM... 4
-75
t. socood Companees
6
E-NN Seccie Compeseen
ESNE Secoeaa Compoomm
160
160
120
100
160
ESSNL SocONi Componcnn
140
120
100
ESNU Secces comppeoeee
140
120
100
-75
-25 LSKE First Component
-25 L-SNE First Componntt
-25
25
75
USNE Fucal Composent
Figure 12: Optimal transport distances (as measured by the empirical 1- I-Wasserstein distance) and Schedule Deviation for each of the DDPM/DDIM/GE, sampling algorithms on the Conditional MNIST dataset.
39

---

TONE Second Componest
DDPMEDIM A
-28
-75
t.N. socoed Componeet
E-NN Secrne Compescat
3000
ESSE Secoeaa Composmm
25.0
201
150
100
-50
ESNN Stocnn: Componen
280
300
200
200
150
-75
SSNU Sicceca componeess
400
200
100
-75
-75
-25
-25 L-NE First Componntt
-25
25
75
LSKE First Component
USNE Ficul Component
Figure 13: Optimal transport distances (as measured by the empirical I-Wasserstein distance) and Schedule Deviation for each of the DDPM/DDIM/GE sampling algorithms on the Conditional Fashion-M dataset, Note the per-row scaling on the right differs between sampling algorithms.
40

---

0.79
1.0
D00 DOPSSDDDI 01 Distron
Steat Poin
-0.75
1.0
0.0
Start Poine Y
0.29
Stus Phint
-0.3
-0.75
130
120
110
Start Poit
0.25
100
-0.75
-30
120
110
Start Point X
0.25
-025
0.75
116
Stert Dint
0.2
100
90
-0.75
-0.75
0.25 Start
-0.25
0.75
-0.75
-0.25
0.25
0.75
-0.75
-0.22
0.35 Starr Point X
25
0.75
PotntX
Start
Point
Figure 14: Optimal transport distances (as measured by the empirical I- I-Wasserstein distance) and Schedule Deviation for each the DDPM/DDIM/GE sampling algorithms on the Maze Solutions dataset.
41


Reviews:
[{"review_id": "J6LQyaEll1", "reviewer": "Reviewer_daH1", "summary": "The authors question the denoising capability of continuous conditional diffusion denoising models, and propose a principled metric for measuring the gap between diffusion models and their idealised denoising path. Both the gap, and the metric that measures it have a strong mathematical motivation and are supported by empirical evidence. The authors show that this deviation is not caused by model capacity or amount of training data, and pose it as a difficulty of the model to bridge between various conditional areas of the space.", "questions": "1. How can SD be decreased for a specific task? \n2. The paper explores SD only in the context of U-Nets, and MLP, how do you see that the architecture of the network impacts your metric? Do different architectures smooth differently?\n3. Is the author's vision that because this gap (deviation) can’t be filled by increasing model capacity or the amount of training data, the problem cannot be solved?", "limitations": "I will rate as borderline acceptance, but willing to increase in the case in which:\n- The presentation of the paper would be improved (less-heavy first Figure 1, introduce important concepts earlier on).\n- Related work is more rooted, and it explains why the classifier-free guidance paper is different [Bradley and Nakkiran, 2024] is different from the author's work.\n- Perhaps a real-world example would be present, for example, in continuous control policies, and why it is important that the model doesn't “interpolate” between unseen conditions.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Please provide a thorough assessment of the strengths and weaknesses of the paper. A good mental framing for strengths and weaknesses is to think of reasons you might accept or reject the paper. Please touch on the following dimensions: Quality, Clarity, \n\n-  Significance, and Originality. For more information, please see the NeurIPS 2025 Reviewer Guidelines (https://neurips.cc/Conferences/2025/ReviewerGuidelines). You can incorporate Markdown and LaTeX into your review. See https://openreview.net/faq.\nMajor Strengths:\n-  The theoretical framework through which the importance of the introduced measure is very nice. The authors upper and lower bound total variation between the idealized and actual paths, with the introduced measure (Theorem 1). \n- Self-guidance and linear interpolation of newly seen conditioned point is nicely proven in (Theorem 2-3)\nThe paper is theoretically sound and it makes a contribution in that regard. \n\n\nMajor Weaknesses:\n- While the denoising mechanism fails only when the conditioning space is continuous and relevant applications for this are presented in the abstract (text-conditioned, or control policies), no experiments in this regard are present (only toy-examples). \n- While the abstract is well-written, Figure 1 is too heavy to offer initial intuition on the problem. An unpopular task: t-SNE-condition MNIST generation is introduced without explanation, and then the similarity between Schedule Deviation (proposed method), and OT between samples produced by two methods (DDPM and DDIM). While the similarity is somewhat obvious it is not clear why the comparison is relevant (why compute OT between DDPM and DDIM samples?). The paper would benefit perhaps from a more intuitive first figure. Also, I am not sure why it is relevant to increase the size of the training data? \n - Related work discussed only in 6 lines  (from 93-99), is very sparse. It is mentioned that “[Bradley and Nakkiran, 2024] show that classifier-free guidance results in the diffusion not to be denoising anymore”, but it is not mentioned why their work is different.\n\n\nMinor Weaknesses:\n- It would be interesting to see how the samples from DDIM and DDPM actually differ (rather than the OT distance)\n- I would introduce quicker why the disagreement between DDPM and DDIM is important and offer intuition of it. \n- Same for Figure 2, it is not obvious why the SD is predictive of this divergence, it is not clear what is meant by trajectory N=4000 means (why is it relevant?). While I see value in the fact that the demonstration of the importance of SD is made through empirical effort, I believe that at least for Figure 1 the introduction is too heavy.", "was_revised": true, "final_justification": "I think this work has a lot of compelling insights that merit discussion at neurips 2025.", "rebuttal": "We thank the reviewer for their feedback on the presentation of the paper and recommendations. Unfortunately we cannot update the paper until the camera ready deadline, but will detail the changes we intend to make given the reviewer’s constructive feedback.\n\n1. **Experiments:**\n\n    Although we present only proxies for large-scale models/tasks, we are able to evaluate models up to the ~40 million parameter regime without issue. Thus we believe that with compute resources beyond those in academia this could be applied to smaller-sized text-conditioned models or robot foundation models (given perhaps several weeks compute time on a couple dozen GPUs).\n\n    In addition to the existing image experiments, we will replicate our methodology on either CelebA-Dialog or an annotated FFHQ which contains fine-grained labels for attributes such as age, degree of smile, amount of facial hair, etc. This will serve as a better proxy for text-based conditioning than t-SNE conditional MNIST without incurring the expense of evaluating models in the billion-plus parameter regime. \n\n    We originally ran experiments on the Push-T continuous control environment but found essentially no Schedule Deviation for the state-conditioned models, which we attribute to a lack of multi-modality in the action distributions. We believe this is due to overfitting of these large (50m+ parameter models) to very few (~200) human demonstrations, meaning these models behave more like an adaptive nearest-neighbor approximator than a multi-modal action model. Guidance between with identical, translated distributions simply shifts the mean and does not result in Schedule Deviation (see Appendix B for a discussion of Schedule Deviation for Gaussians), so it is unsurprising that we found no Schedule Deviation for essentially deterministic Diffusion Policies. We conjecture that for robot foundation models this “effective unimodality” is not present with large multi-task training datasets and image-conditioned models, although we acknowledge that additional studies of these models are needed.\n\n    When designing the maze navigation task, we explicitly introduced multi-modality in the action distributions by considering a weighted mixture of all possible paths to the target point when generating the data. Indeed, higher Schedule Deviation can be observed at “bifurcation” points in Figure 3 where the trajectory distribution is more multi-modal.\n\n    We would certainly like to run additional continuous-control experiments in other environments. However, we are unsure what an environment would look like where the diffusion policy is not effectively deterministic. We welcome suggestions from the reviewer on what environment they feel would be most appropriate.\n\n2. **Presentation:**\n\n    We acknowledge that placing Figure 1 so early is too heavy and confusing for readers. We will move this to coincide with the main experiments to make the introduction more logical and make an easier-to-digest figure (perhaps based on the low-dimensional experiments in Section 4) to familiarize the reader to the problem setting we consider.\n\n   Through different values of “N,” the total amount of training data, we wanted to demonstrate that Schedule Deviation is present even when the model is given access to different quantities of training data at the “low-density” regions–i.e. the effect depends on the relative density rather than the total amount of training data and so we conjecture that Schedule Deviation cannot be ameliorated simply by scaling the amount of training data available. Note that we also provide ablations over the training data for Fashion-MNIST and the Maze Trajectories datasets in our experimental appendix.\n\n 3. **Prior Work:**\n\n    We will definitely expand on the prior work section. In particular, we will include significantly more details as to how our metric relates to work in unconditional models and how previously proposed metrics are either inappropriate for conditional models or otherwise differ from our proposed approach.\n\n    [Bradley 2024] observes that classifier free guidance does not yield a denoiser, although they have no methodology for quantifying the degree of this divergence. In our evaluation setup there is no classifier-free guidance–only a single conditional diffusion model. We use the observation of Bradley et al. that interpolating scores yields non-denoising probability paths to leverage our theoretical results in Section 4 to explain Schedule Deviation. We agree that this part of the prior work was not well-phrased and will better explain the relation to Bradley et. al. in our subsequent revision. \n\n 4. **Predicting OT Distance:**\n\n     Both DDPM/DDIM produce “reasonable” looking samples across the conditioning space. We conjecture that Schedule Deviation is predictive of the OT distance as DDIM/DDPM leverage the assumption that the flow is a denoiser differently in their sampling procedures. This is immaterial when the neural network well-approximates a denoiser (low Schedule Deviation) but can lead to divergence with high Schedule Deviation, which is precisely what we observe. \n\n 5. **Reducing SD:** We take no stance as to whether SD is “good” or “bad.” However, the widespread use of classifier-free guidance to improve generation quality suggests that any Schedule Deviation arising from “self-guidance” may in fact be an excellent inductive bias for models, in which case reducing Schedule Deviation may be detrimental. As we note in Remark 3.1, Schedule Deviation is not directly related to generation fidelity but rather measures the self-consistency of the reverse process.\n\n    Methods such as [Daras 2023] try to induce more denoiser-like behavior by augmenting with samples drawn from the reverse process. Augmenting with an entirely new, much larger, dataset using a pretrained model eliminates the interpolation ambiguity and specifies the correct interpolation. Whether this is beneficial likely depends on the inductive biases of both the model and the sampler used in this data re-generation process. This effectively would “fix” the interpolation to either the distribution generated by DDIM or DDPM.\n\n    Note that our metric of schedule deviation is different from the analogous quantities used in [Daras] or [Scarvelis] in that (1) we run a full reverse process and then re-noise our samples, whereas [Daras] uses only reverse process samples and (2) we more directly measure how the evolved density deviates from the ideal denoiser. It is possible to deviate from the ideal denoiser by adding “curl” components that do not change the probability path. By incorporating the divergence in our metric, we are agnostic to such changes.\n\n    We will expand on the above in the prior work section.\n\n\n 6. **Regarding Architectures:** \n\n    It would be quite surprising if different architectures interpolated the same way, as we observe empirical differences in the sample quality for different architectures. However both U-Nets and MLPs exhibited schedule deviation. Our paper conjectures this is because “reasonable” smooth interpolations between different denoisers are fundamentally non-denoising. Therefore any architecture which does not exhibit Schedule Deviation would likely need fundamentally very different and seemingly \"unnatural\" inductive biases compared to the architectures we consider here. We leave an in-depth exploration of the Schedule Deviation of different architectures to future work but believe it is of great interest, both theoretically and empirically.\n\n[Daras 2023] Daras, G., Dagan, Y., Dimakis, A., & Daskalakis, C. (2023). Consistent diffusion models: Mitigating sampling drift by learning to be consistent. Advances in Neural Information Processing Systems, 36, 42038-42063.\n\n[Scarvelis 2023] Scarvelis, C., Borde, H. S. D. O., & Solomon, J. (2023). Closed-form diffusion models. arXiv preprint arXiv:2310.12395.\n\n[Bradley 2024] Bradley, A., & Nakkiran, P. (2024). Classifier-free guidance is a predictor-corrector. arXiv preprint arXiv:2408.09000."}, {"review_id": "IGiU1ZczT9", "reviewer": "Reviewer_zm5H", "summary": "This paper derived a metric Schedule Deviation from theoretical motivations, measuring the difference of the probability path from the ideal or theoretical ones. Then they derived a tractable way to estimate it from expectation. Using this metric they measured the schedule deviation in some practical conditional diffusion models, and found it’s prevalent and related to smoothing w.r.t. conditions. Finally they derived a closed form score smoothened w.r.t. condition, and checked empirically that neural networks exhibit similar generalization behavior and schedule deviation from this closed form.", "questions": "### Questions\n\n- One major concern is, The Schedule Deviation is nice and novel, but seems there are some simpler and easier to compute metrics that could do similar things, that authors didn’t mention or compare.\n    - For example, one simple ways is to compare the **deviation of neural score function from “ideal” score**. Recently a few papers have computed this through the diffusion trajectory, they found non-negligible deviation esp. at low noise scale and they found this is exactly the reason why diffusion generalize beyond training set. ( for uncond diffusion [Wang, Vastola (2023/2024), Li, Dai, Qu, (2024)]).\n    - It seems the identity in Prop 3.1 is different but quite related to averaged score deviations Eq.28 in [Wang, Vastola (2024)], authors may want to discuss this link. The way authors compute it is also quite similar, basically assuming a delta mixture model of the underlying distribution.\n    - Per Girsanov theorem, I think score deviation or velocity deviation can also leads to bound in distribution difference.\n    - Maybe the authors could plot this score deviation on X axis for Figure 4, I imagine the effect will be similar.\n- Not really sure how Schedule Deviation could be computed in Figure 5 for condition values outside the support of training data. When the condition z is out of training set, how do you compute the ground truth score and velocity and compare to neural network? (as described in L183-188)\n- Could the Theorem 3 be generalized to more complex topology of Z set? like MNIST case, it’s 2d embedding or some high dim embeddings of classes?\n- Did the authors used CFG in their main experiments? Should we describe the effect of it on schedule deviation, since it’s so commonly used in conditional diffusion model? \n\n- Regarding the score smoothing idea, authors may also cite the recent theoretical treatment in [Chen 2025] and empirics in [Wang, Vastola (2024)].\n    - More generally I feel smoothing the velocity field will leads to schedule or score deviation, either to x or z.\n\n[Chen 2025] Chen, Z. (2025). On the interpolation effect of score smoothing. *arXiv preprint arXiv:2502.19499*.\n\n[Wang, Vastola (2024)] The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications. TMLR *arXiv:2412.09726*.", "limitations": "yes in appendix D.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strength\n\n- Conditional diffusion models and esp. classifier free guidance have been less theoretically examined than unconditional one. So the subject is quite interesting. The conceptual point of conditional diffusion not denoising is interesting. (though in a sense\n- The observation of discrepancy between scheduler / solvers and schedule deviation (or score difference in my reading) is interesting and relevant to practical usage.\n- Evaluated of a few different settings.\n- Smoothness with respect to the conditioning variable is interesting and less studied before. So the theorem 2 is a nice contribution to the field.\n- Theorem 3 is nice extension of closed form diffusion (score-smoothing) model to conditional diffusion. It also illustrates these smooth score can be understood as consequence from certain regularization / constraint Eq. 4.1. This is quite aligned with some recent work of how some inductive biases / constraint of diffusion induces the actual neural score and samples. [Kamb, Ganguli, 2024; Niedoba 2024]\n- The conceptual link of deviation of path from ideal one and score smoothing w.r.t. conditioning label is interesting (previously more people think about smoothing w.r.t. x).\n\n[Kamb, Ganguli 2024]: Kamb, M., & Ganguli, S. (2024). An analytic theory of creativity in convolutional diffusion models. *arXiv preprint arXiv:2412.20292*.\n\n[Niedoba 2024] Niedoba, M., Zwartsenberg, B., Murphy, K., & Wood, F. (2024). Towards a Mechanistic Explanation of Diffusion Model Generalization. *arXiv preprint arXiv:2411.19339*.\n\n### Weakness\n\n- The conditional flow notation / framework is quite elegant but a bit confusing… the equivalent form showed in Proposition 2.1 is more common and familiar to the diffusion field. Eq. 2.4 and 2.5 are commonly found in Karras 2022 (EDM paper) etc. Maybe authors should cite them there too.\n    - For example, the IDDF seems not different from the velocity field determined by the “ideal / true score”. Though what is true score is still tricky, if we have only finite samples from dist.\n- Empirically the connection of Schedule deviation and DDPM DDIM distance is clean in the image case, but quite fuzzy in the maze case (Fig2 left and Fig4 right). This made us question the generality of this connection. Maybe further experiments could clarify when this work.\n- Theoretically, not sure if the proposed Schedule deviation metric cannot be substitute by some simpler and easier to compute metrics, e.g. difference of neural score from the ideal / true score. (details see questions)\n\n- One conceptual issue for the paper is that IDDF as defined in the paper is based on the empirical score of the training set, and being equal to that will render the model only able to sample from the delta mixture of the training set (as noticed by many authors [Scarvelis 2023], [Wang Vastola 2023], [Li, Dai, Qu, (2024)]). So opposing to L225, deviation from IDDF (as defined in the paper) may be a problem, but being equal to IDDF is not a solution either.\n    - The paper above found the score deviation (esp. at low noise scale) and they found this is exactly the reason why diffusion generalize beyond training set. So to say that SD is a bad thing to be alleviated is not totally valid.\n\n- The final experiment toy datasets is a bit too toy or simple. I feel at least a MNIST or human face (e.g. FFHQ) experiment could be done to make it more impactful. As you know face and digit datasets have discrete attributes that could be interpolated.\n\n- [Minor] L274 Not really sure why self guidance is called self guidance… it’s more like score smoothing w.r.t. condition. But personal opinion.\n\nSome typos \n\nL145 *consistency → consistently*\n\nL230 *correlated → correlated with* \n\nL287 *presence → presence of* \n\n[EDM] Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. *Advances in neural information processing systems*, *35*, 26565-26577.\n\n[Scarvelis 2023] Scarvelis, C., Borde, H. S. D. O., & Solomon, J. (2023). Closed-form diffusion models. *arXiv preprint arXiv:2310.12395*.\n\n[Wang Vastola 2023] Wang, B., & Vastola, J. J. (2023). Diffusion models generate images like painters: an analytical theory of outline first, details later. arXiv preprint arXiv:2303.02490.\n\n[Li, X., Dai, Y., & Qu, Q. (2024)] Understanding generalizability of diffusion models requires rethinking the hidden gaussian structure. NeurIPS", "was_revised": true, "final_justification": "I think in the most recent round of rebuttal the authors have successfully addressed most of my technical concerns about the paper! As long as the authors improve the clarity of the paper by moving some text & figures from the appendix to the main text in the final version, I feel it's ready for acceptance.", "rebuttal": "Thank you for the thorough reading and the in-depth review. We will clarify several potential sources of confusion that we realize may not have been sufficiently emphasized in the text.\n\n1. **Clarifying the Construction of the IDDF:**\n\n    The $p_t(x|z)$ (including $p_0(x|z)$) used in the definition of Schedule Deviation are the marginals consistent with the learned flow $v$, not the true flow $v^\\star$. That is to say, $p_0$ arises from the reverse process under the learned model and is *not* the distribution of the training set (which we define as $p^\\star(x|z)$ at the start of Section 2). This makes $p_0(x|z)$ and, by extension, $v^{IDDF}$ well-defined even when far from the support of the training data. We address this point explicitly in Remark 3.1.\n\n    Given the understandable confusion, we will (1) move Remark 3.1 into the preliminaries and (2) make the preliminaries less dense to read, with the appropriate references to Karras and other similar frameworks, (3) we will use a different symbol besides $v$ in the definition of Schedule Deviation to denote the learned flow and (4) we will also rename IDDF from “Ideal Denoising Diffusion Flow” to “Model Consistent Denoising Flow” (MCDF) in order to make the distinction between the ground-truth flow $v^\\star$ and $v^{IDDF}$ immediately clear.\n\n    Hopefully this mitigates your concern that the IDDF is based on the empirical score of the training set. In fact, because we do not rely on our training set at all, the schedule deviation can be computed solely given the pretrained model, an advantage we will highlight in our revised introduction.\n\n2. **Positioning with Respect to Prior Work:**\n\n    We use a flow-based formulation as it is simpler to reason about what it means to diverge from the ideal flow and mathematically simpler to manipulate than the SDE formulation. We based our formulation off of [Albergo 2023] but naturally the same framework exists in other works as well. We include an SDE formulation as well in Appendix B for completeness and will of course cite [Karras] as well.  \n\n    Previous work (e.g., Scarvelis, Chen) uses smoothing of the ideal denoiser to study generalization in unconditional models. In contrast, we argue that for conditional diffusion, \"approximately denoising\" is less informative due to the much higher  Schedule Deviation, which we explain by our model of self-guidance. We will include [Wang], [Li], and [Chen] in our prior work and explain how our proposed metric and interpretation of conditional diffusion are fundamentally different from the variants they consider.\n\n3. **Schedule Deviation and “Fuzziness” on the Maze Dataset:**\n\n    The correlation between Schedule Deviation and DDPM/DDIM for the Maze dataset is still quite strong. The “fuzziness” in Figure 4 is due to when the conditioning is exactly on the walls between the cells in the Maze (see Figure 7), which exhibit extremely high OT distance between DDPM/DDIM exactly on the boundary (Figure 2, upper left) but where the Schedule Deviation (Figure 2, lower left) looks much more clean. Removal of these boundary points makes the association much more clear.\n\n    We believe the noise in the optimal transport distance is due to the OT distance we use (1-Wasserstein distance) being very sensitive to changes in the probabilities for modes that are very far apart, such as when evaluating on the “walls” of the maze. Using “clipped” distances in the cost matrix for the OT distances can remove the associated “fuzziness” you observe, but we did not want to create the appearance of hacking the metric to be more favorable and justify why we used a non-standard metric. In short, we believe this is due to the OT calculation rather than our metric, which does not exhibit such outliers.\n\n4. **Relation to Simpler Metrics:**\n\n    As you point out, error between the learned model and the ideal flow is sufficient to give an upper bound on the deviation of the probability densities. The key strength of our metric is that it also has a lower bound for the deviation of the probabilities.\n\n    Any simpler metric based on error with the ideal flow cannot yield a lower bound. By the transport equation (Eq A.1), any divergence-free component of  $p_t(x|z)v_t(x|z)$ has no effect on the evolution of the marginals of the reverse process. Thus it is possible to add an arbitrary “curl” component to $v^IDDF$ while preserving the same marginals. Thus the “error” between $v$ and $v^IDDF$ can be arbitrarily high despite the fact that both the intermediate marginals and the final distributions are identical. By contrast, our metric directly measures the instantaneous deviation in how the densities are evolved and is agnostic to any such “curl” components in the flow. We will better explain this fact and improve the motivation for our metric in the preliminaries.\n\n    Comparing our metric empirically to the error wrt $v^{IDDF}$ is an excellent suggestion, and we will develop some simple experiments or examples which contrast the two.\n\n5. **Regarding the low-dimensional experiments:**\n\n   We welcome the reviewer’s suggestion of using a human-faces dataset, and intend to incorporate an experiment on either FFHQ or CelebA using continuous conditioning attributes for a subsequent revision in addition to our existing experiments in Section 3.\n\n   The datasets for the experiments in  Section 4 are designed to mirror the settings of Theorem 2/Theorem 3, where we can analytically compute the smoothest possible interpolant. Unfortunately deriving a closed-form analogous to Theorem 2 or 3 for the conditioning space of more complex data such as MNIST or FFHQ seems generally infeasible. The setting of Theorem 2 is solvable due to specific properties of cubic spline interpolations and Theorem 3 is due to the uniform weighting of the conditioning distribution analyzed.\n\n    Although Theorem 3 could potentially be generalized to non-uniform densities, the final expression would involve nested Fourier and inverse Fourier transforms in the final expression which cannot be further simplified in the general case. We believe that deriving the closed form under other “toy” conditioning distributions or smoothness penalties (yielding correspondingly different spline interpolations in discrete conditioning support analogous to Theorem 2) may be more pedagogically instructive for understanding biases in conditional diffusion. We believe such investigations would perhaps be a good fit for a future, explicitly theoretically-focused work.\n\n   In the case of MNIST or FFHQ, any predictive model would almost certainly need to incorporate inductive biases of the NN architecture as well. Modeling these biases is out of scope for this work and we direct the reader to [Kamb 2024] which investigates locality biases in purely convolutional architectures. We will be sure to expand on this in our prior work.\n\n6. **Schedule Deviation vs Fidelity:**\n\n    Schedule deviation measures whether a model is “consistent” in representing the denoiser for its own final distribution. It is possible to construct models with low schedule deviation that poorly model the desired conditional distribution or, vice-versa, models with high schedule deviation that model the distribution extremely well. For this reason we note in Remark 3.1 that schedule deviation is orthogonal to fidelity. \n\n    We do not claim that schedule deviation needs to be mitigated. In fact, given how effective classifier-free guidance is, schedule deviation arising from “self-guidance”-style smoothing may be an excellent inductive bias.\n\nAddressing other miscellaneous questions/comments:\n\n1. As discussed above, our metric is much stronger than the prior variants used in the literature. We will be sure to expand the prior work to better distinguish the advantages of our approach to justify the additional computational complexity of our approach. We encourage the reviewer to read our response to similar questions raised by reviewer daH1.\n\n2. We did not use classifier-free guidance in these experiments as we were interested in the inductive biases of learning the conditional score by itself.  It is known that classifier free guidance results in non-denoising paths [Bradley 2024] (also potentially relevant to the effect of guidance is Skreta 2024), so it would certainly be interesting to investigate empirically the Schedule Deviation of either classifier-based or classifier-free guidance. We believe there are many settings where evaluating the Schedule Deviation may be of interest and unfortunately are not able to address all of them in this paper, but will do so in a follow-up work.\n\n3. We emphasize the term “self guidance” to highlight that better understanding of guidance (which remains poorly understood) may simultaneously yield better understanding of how conditional diffusion models generalize. While it is perhaps intuitive for Theorem 2/3 that smoothness regularization yields a linear combination of the flows at the original datapoints, that this combination is independent of the choice of z is surprising.\n\n[Albergo 2024] Albergo, M. S., Boffi, N. M., & Vanden-Eijnden, E. (2023). Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797.\n\n\n[Kamb 2024] Kamb, M., & Ganguli, S. (2024). An analytic theory of creativity in convolutional diffusion models. arXiv preprint arXiv:2412.20292.\n\n\n[Bradley 2024] Bradley, A., & Nakkiran, P. (2024). Classifier-free guidance is a predictor-corrector. arXiv preprint arXiv:2408.09000.\n\n[Skreta 2024] Skreta, M., Atanackovic, L., Bose, A. J., Tong, A., & Neklyudov, K. (2024). The Superposition of Diffusion Models Using the It\\^ o Density Estimator. arXiv preprint arXiv:2412.17762."}, {"review_id": "5e8hj0e314", "reviewer": "Reviewer_jj8Z", "summary": "This paper investigates a fundamental mismatch between the theoretical formulation of conditional diffusion models as denoisers and their practical behavior during inference. It introduces a novel metric called **Schedule Deviation (SD)** to measure how much the learned generative path deviates from the idealized denoising diffusion path (IDDF). The authors find that **conditional diffusion models routinely deviate** from their idealized dynamics—this deviation persists even with more data or larger models and is theorized to result from **inductive biases favoring smoothness** in the conditioning space. The paper provides both empirical and theoretical analyses, showing SD correlates with sample quality degradation and divergence between popular sampling algorithms like DDPM and DDIM.", "questions": "- Why introduce a new metric? Why not just try to find a way to efficiently compute TV distance? Would TV distance also have similar explanatory power?\n- How difficult would it be to compute TV vs SD ?\n- What precedent in the SDE literature exists for a metric like SD? Can you show you have no reinvented the wheel?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- High relevance: addresses a core assumption in conditional diffusion\n- Originality: novel explanation of the workings of diffusion models\n- Rigorously defines a principled and tractable metric (Schedule Deviation)\n\nWeaknesses:\n- Lack of motivation and intuition for new metric, Schedule Deviation (SD)\n- No survey or connection to literature on SDEs", "was_revised": true, "final_justification": "The authors have provided adequate explanation of the significance, motivation, and intuition behind their results. Response to other reviewer's concerns about experiments were also satisfactory. Thus this reviewer recommends an accept.", "rebuttal": "We thank the reviewer for their review and questions. We address your concerns in order:\n\n1. **Motivation:**\n\n    We consolidate the different motivations and interpretations of our metric below and will be sure to revise our introduction and preliminaries to make the motivation more explicit:\n\n    **We address shortcoming with metrics developed for unconditional settings:** It is believed the score-smoothing/non-denoising behavior of models is fundamentally related to generalization in unconditional diffusion [see Scarvelis, Aithal 2024]. However, existing metrics such as the “error from the ideal closed-form denoiser” ($||v^\\star_t(x) - v_t(x)||$) [see Scarvelis] require samples from the true data distribution $p^\\star(x)$ to compute $v^\\star_t(x)$. For conditional diffusion (particularly with continuous conditioning values) we may potentially have either one or zero samples for a given “z” value and therefore cannot estimate the ideal denoiser for $p^\\star(x|z)$. This is what principally motivates the construction of a new metric which can be evaluated even for conditional models.\n\n    Existing metrics are ad-hoc designed in other ways. Namely, while the true denoiser/flow $v^\\star$ can be estimated from the training data, there are many choices for the distribution over $x$ on which to evaluate $||v^\\star_t(x) - v_t(x)||$. It is unclear whether to use (1) reverse process under the learned v_t starting from noise or (2) the forward process using samples from $p^\\star(x)$. \n\n    We argue that the **principled** choice of distribution over x to consider is the one induced by (1) running the full reverse process under the learned $v$ and then (2) subsequently re-noising the generated samples using the forward process. We use $v^{IDDF}$, “the idealized flow” to refer to the **flow for the forward process applied to the p(x | z) distribution of the reverse process under the learned v**. Crucially, we can compute $v^{IDDF}$ without access to $p^\\star(x|z)$ using only the learned $v$, meaning our metric can be computed for conditional models.\n\n2. **Intuition:**\n\n    Note that while we evaluate the Schedule Deviation using Proposition 3.1 (which resembles earlier metrics), we cleanly derive the form in Proposition 3.1 from the instantaneous rate of deviation of $p^v$ from $p^{IDDF}$ as defined in Definition 3.1. In other words, for some intermediate timestep $t$, if we start at $p^{IDDF}_t(x)$, the Schedule Deviation measures the total rate at which the density under $v^{IDDF}$ diverges from the same density evolved under $v$. We believe this is much easier and more intuitive than the “error from a reference denoiser” as in prior work.\n\n    Proposition 3.1 gives an alternate interpretation of our metric as the error between $v$ and $v^{IDDF}$ along the direction of $\\nabla \\log p^{IDDF}_t(x)$, plus a divergence-based correction term. This reveals that not all forms of error between $v$ and $v^{IDDF}$ cause deviation from a denoising probability path. At first order, it is only the error along $\\nabla \\log p^{IDDF}_t(x)$ which causes deviation, a property which our metric makes immediately clear. This makes intuitive sense, as the transport equation (Eq A.1) implies that adding a “curl” component to $p_t(x)v_t(x)$ has no effect on the evolution of the density $p_t(x)$, i.e. $v$ can be arbitrarily far from $v^{IDDF}$ without affecting the density. \n\n   Lastly, as we note just before Theorem 1, our metric resembles the advantage function used in the storied Performance Difference Lemma from reinforcement learning, applied to the evolution of densities. Indeed, an extension of the Performance Difference Lemma is central to the proof of Theorem 1 found in Appendix A.\n\n 3. **Novelty and Connection to SDE Literature:** To our knowledge these contributions above are all novel and we are unaware of a similar metric in either the SDE, optimal transport, or diffusion literature beyond the aforementioned prior works.\n\n    We base our formulation of diffusion on the Stochastic Interpolants and flow-matching frameworks [Albergo, Lipman], introducing diffusion models using its flow-based ODE formulation rather than the (equivalent) SDE formulation as it simplifies the notion of having a non-denoising flow and are generally easier to reason about the SDEs, which require significantly more technical mathematical treatment.\n\n    Note that we additionally provide SDE-based sampling procedures and equivalent formulations in Appendix B for interested readers who wish to understand SDE-based sampling works in the context of our paper. We will signpost to this appendix for readers who are interested in this.\n\n    We are of course happy to reference the connections to the SDE-based literature beyond the original diffusion works by Sohl-Dickenstein, Ho, and Song, which were of course principally SDE-based. We also will create an extended related works in the appendix with the relevant SDE related work. \n\n4. **Why not estimate TV directly?**\n    Estimating the TV distance directly requires calculating the difference in densities between $p_t$ and $p_t^{IDDF}$ over the entire space x. As we cannot integrate over the space of all images, it is not easy to compute the TV distance directly. Estimating the associated densities $p_t(x)$, $p_t^{IDDF}$ (or their ratios) is also numerically unstable in practice.\n\n    In contrast, our method for computing Schedule Deviation only requires sampling x from $p_t^{IDDF}$, meaning we can restrict our computation to areas with high likelihood. Furthermore, the quantities involved are all log-densities or their gradients, \n\n    Note that $TV(p_t^{IDDF} | p_t)$ depends only on $v_s(x)$ for $s > t$, not $v_t(x)$ itself. Thus, given only the flow $v_t(x)$ at time $t$, it is more intuitive to relate the value of $v_t(x)$ to the different in the *change* of the density $p_t$ compared to $p_t^{IDDF}$, which is precisely what Schedule Deviation captures.\n\n[Albergo 2023] Albergo, M. S., Boffi, N. M., & Vanden-Eijnden, E. (2023). Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797.\n\n[Aithal 2024] Aithal, S. K., Maini, P., Lipton, Z., & Kolter, J. Z. (2024). Understanding hallucinations in diffusion models through mode interpolation. Advances in Neural Information Processing Systems, 37, 134614-134644.\n\n[Lipman 2022] Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., & Le, M. (2022). Flow matching for generative modeling. arXiv preprint arXiv:2210.02747.\n\n[Scarvelis 2023] Scarvelis, C., Borde, H. S. D. O., & Solomon, J. (2023). Closed-form diffusion models. arXiv preprint arXiv:2310.12395."}, {"review_id": "AfoBhVhAOM", "reviewer": "Reviewer_i2tR", "summary": "The paper proposes a new metric to analyse the sampling behavior of diffusion models, Schedule Deviation. The authors show that the intermediate denoising predictions of trained diffusion models deviate from the predictions of an ideal denoiser, with this effect being consistent across model and dataset scales. To explain it, they posit that neural network smoothness with respect to the conditioning variables leads to unwanted 'self-guidance' in the model, which incorrectly interpolates predicted flows from different conditions.", "questions": "- **Unconditional diffusion**: A question that quickly comes up when reading the paper is whether Schedule Deviation exists in unconditional diffusion models, and if so, could it be because of another reason other than the self-guidance due to conditioning presented in the paper?\n\n- **Computing the proposed Schedule Deviation metric**: The authors acknowledge that computing the proposed metric is very expensive. Even for the toy datasets they demonstrate their results on, computing the divergence term $\\nabla (v_s - v_s^{\\text{IDDF}})$ is too computationally heavy, so they approximate it with a single value from the Jacobian diagonal. This raises the question of whether there's even a chance of demonstrating the paper's findings for practical diffusion models (e.g. text-to-image models).\n\n- **More complex conditions**: Do the same findings regarding self-guidance hold when the ideal denoiser is non-linearly interpolating the conditions? As the conditioning increases in dimensions ($2$-dim in the paper experiments vs $77\\times1024$-dim for text) and becomes more complex, it is unlikely that the model is simply linearly interpolating conditions to generalize to unseen prompts. Would you get similar results when interpolating across some learned geodesics on the condition manifold?", "limitations": "Yes, limitations have been addressed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- **Novelty**: The authors devise a novel metric to measure the discrepancy between the predictions of a trained and an ideal denoiser. The proposed metric, Schedule Deviation, is well-motivated and a similar approach has not been discussed in the literature before.\n\n- **Findings**: Using the proposed metric, the authors demonstrate a variety of findings that showcase how trained diffusion models do not perform as expected when drawing samples. Going a step further, the authors provide a possible explanation for this discrepancy, which lies in the smoothness of the conditioning used in these models. These findings persist across datasets/model sizes and sampling algorithms and can be of great importance to the development of future diffusion sampling and training algorithms. \n\n**Weaknesses**:\n- **Self-guidance experiments**: The authors posit that the issue of *self-guidance*, or the denoiser linearly interpolating flows from different conditions, leads to a larger Schedule Deviation. Theorems 2 and 3 motivate this assumption by showing that a smoothed denoiser interpolates between discrete and continuous conditions. To translate these findings to trained diffusion models, the authors perform the toy experiments in Figures 5 and 6. There, they show that the neural network and closed-form denoisers draw similar samples, but the similarity in Schedule Deviation is not as evident. Additionally, it is unclear why showing that having similar samples and Schedule Deviation means that self-guidance is true for the trained diffusion models.\n\n- **Scale**: The scale of experiments is limited to toy datasets, which is an inherent weakness of the proposed metric. Computing the Schedule Deviation requires (i) approximating the ideal denoiser with a large enough set of samples drawn from the model and (ii) computing the divergence of the difference between the trained and ideal denoiser scores. Although the arguments made in the paper are mostly sound for the toy experiments shown, there may be interesting findings that deviate from the ones shown in the paper in bigger models, such as text-to-image models. It is unclear whether the proposed metric can be used in larger models.", "was_revised": true, "final_justification": "The paper presents an interesting new metric to measure the deviation of the trained and an ideal denoiser, which is shown to correlate with observed deviations (DDPM vs DDIM samples) in the model. They also come up with a theoretical motivation for this discrepancy, based on the smoothness of the networks used to implement the denoiser. While the scale of the experiments is limited, this work could be a meaningful stepping stone towards better understanding the inner workings of diffusion models.", "rebuttal": "We thank reviewer i2tR for their thoughtful feedback and address their concerns, approximately in order:\n\n1. **Self-guidance experiments:**\n\n    We acknowledge that we do not directly demonstrate self-guidance in learned models, but instead show the capability of self-guidance to reasonably predict the samples and Schedule Deviation of a learned model.\n\n\n    It is unclear how to demonstrate self-guidance in itself besides comparing generated samples. Using the error in the flow between the closed-form and the learned model is difficult to interpret as there is no “baseline” flow that we can contrast self-guidance to in order to show that the computed error is reasonably low. We agree that showing that self-guidance is inherently “true” of models would greatly strengthen this paper and are interested in any ideas the reviewer has for how to effectively design such an experiment.\n\n    We naturally do not expect self-guidance to be exactly true for larger models, which necessarily must have more complicated inductive biases than purely linear interpolation of different scores–there is likely significant coupling between x and z which, as we note in Remark 4.1, our simple model cannot capture. However, we believe the “self-guidance” model gives useful intuition for why Schedule Deviation may be fundamental to conditional diffusion.\n\n2. **Scaling/computing the proposed schedule deviation metric:**\n\n   While we acknowledge that our metric is expensive, we emphasize that it is not prohibitively so: we demonstrate scalability up to ~40 million parameter models without issue.\n\n\n   We believe that with sufficient compute resources (we would estimate at most several dozen GPUs and a couple weeks’ time) the metric could provide meaningful evaluation of smaller-scale text-conditioned models (~1 billion parameters for e.g. Stable Diffusion 3).\n\n    Unfortunately, this is beyond the limitations of the resources we can acquire, although we believe it is feasible to scale: the metric is highly parallelizable and has approximately the same memory complexity as performing a single forward-backward pass.\n\n    We use t-SNE-conditioned image generation in this work as a small-scale proxy for larger text-conditioned diffusion models and, following the suggestions of Reviewer zm5H, will incorporate a human-faces dataset conditioned on continuous attributes such as age for our next revision as a better proxy for the continuously-conditioned diffusion task.\n\n    We welcome any suggestions from the reviewer for additional conditional generation tasks they believe would be interesting in the ~100 m parameter regime which we could include in the final revision.\n\n3. **Unconditional diffusion:**\n\n   Unconditional generation should behave analogously to conditional generation in a region with a high density of conditioning data–that is to say, low schedule deviation. While there is still some Schedule Deviation in these regions (presumably due to smoothing/inductive biases with respect to the x and t inputs that our analysis does not capture), this effect is fairly minimal compared to regions with low-density of conditioning support.\n\n   Because the forward process has support everywhere, with a sufficiently large model and enough training time it will always be possible to exactly model the denoiser of the training dataset and overfit to the training data, e.g. achieve low Schedule Deviation for the unconditional setting. However, for conditional models, we show that Schedule Deviation cannot be easily ameliorated by increasing the training data or the model capacity.\n\n   That is not to say there is no deviation from the IDDF in unconditional settings, just that the flow is believed to much more closely approximate a denoising process. The effect of this kind of smoothing has been studied in prior work such as [Aithal 2024], [Scarvelis 2023]. We believe follow-up work explicitly contrasting Schedule Deviation in conditional and unconditional settings to be of interest.\n\n4. **More complex conditions:**\n\n   Naturally, we do not expect that larger trained models will interpolate via linear combination of the scores–their true behavior is likely a much more complex, architecture-dependent non-parametric. We use Section 4 to highlight how natural and simple inductive biases can be fundamentally at odds with denoising in a manner that is distinct from the smoothing effects studied in unconditional models.\n\n    Although we present no evidence in this paper that self-guidance is directly predictive for larger models, the widespread use and strong empirical performance of classifier-free guidance suggests that self-guidance is an excellent inductive bias. We suggest that better theoretical understanding of classifier-free guidance may be applicable to conditional models as well.\n\n    For higher dimensions, we believe the correct primitive to consider is some complicated nonlinear spline rather than a learned geodesic, as we cannot expect the flow along a path to be purely a combination of its two endpoints. We are considering follow-up directions for larger models motivated by the concept of self-guidance along the lines of what you suggest: namely, can we express a conditional score in terms of a finite basis of unconditional functions, with learned interpolating coefficients? Whether conditional diffusion models interpolate their flows in some low-dimensional manifold is an interesting hypothesis. We hope to study this question in future work.\n\n[Aithal 2024] Aithal, S. K., Maini, P., Lipton, Z., & Kolter, J. Z. (2024). Understanding hallucinations in diffusion models through mode interpolation. Advances in Neural Information Processing Systems, 37, 134614-134644.\n\n[Scarvelis 2023] Scarvelis, C., Borde, H. S. D. O., & Solomon, J. (2023). Closed-form diffusion models. arXiv preprint arXiv:2310.12395."}]

Author rebuttal:

Meta-review: This paper presents a metric for quantifying how much the flow estimated by a diffusion model deviates from the flow defined by the minimizer of the DDPM objective [Ho et al. 2020] (aside - there's typo on line 128 where this is referred to as the DPPM objective). They show that trained diffusion models deviate from this idealized probability path, and offer an explanation for this phenomenon as a consequence of smoothness inductive biases. They also show that the score correlates with the differences between DDPM and DDIM samples. I agree with the reviewers that the paper is an insightful contribution that should be discussed at the conference. Beyond the clarifications brought up by the reviewers (e.g. emphasising the dependence on the estimated $p_0$, updates to Figure 1, etc.), I also agree with Reviewer zm5H that including a description of the algorithm in the main text would help many readers.
Decision: Accept (poster)