CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation
Authors: Bowen Song
Zecheng Zhang
Zhaoxu Luo
Jason Hu
Wei Yuan
Jing Jia
Zhengxu Tang
Guanyang Wang
Liyue Shen
Venue: NeurIPS 2025 poster ()
Keywords: diffusion models
controllable sampling
noise perturbation
Primary area: deep_learning
TL;DR: We unveil a novel linearity relationship between the input noise and outputs in diffusion models.

Abstract: Diffusion models have emerged as powerful tools for generative tasks, producing high-quality outputs across diverse domains. However, how the generated data responds to the initial noise perturbation in diffusion models remains under-explored, hindering a deeper understanding of the controllability of the sampling process.
In this work, we first observe an interesting phenomenon: the relationship between the change of generation outputs and the scale of initial noise perturbation is highly linear through the diffusion ODE sampling process. We then provide both theoretical and empirical analyses to justify this linearity property of the input–output (noise → generation data) relationship.
Inspired by these insights, we propose a novel **C**ontrollable and **C**onstrained **S**ampling (CCS) method, along with a new controller algorithm for diffusion models, that enables precise control over both (1) the proximity of individual samples to a target image and (2) the alignment of the sample mean with the target, while preserving high sample quality.
We conduct extensive experiments comparing our proposed sampling approach with other methods in terms of both sampling controllability and generated data quality. Results show that CCS achieves significantly more precise controllability while maintaining superior sample quality and diversity, enabling practical applications such as fine-grained and robust image editing. Code: [https://github.com/efzero/diffusioncontroller](https://github.com/efzero/diffusioncontroller)

Controllable and Constrained Sampling in Diffusion Models via Initial Noise Perturbation
Bowen Song'
Zecheng Zhang?
Zhaoxu Luo
Jason Hu
Wei Yuan
Jing Jia
Zhengxu Tang
Guanyang Wang**
Liyue Shen
1 'Department of Electrical Engineering and Computer Science, University of Michigan "TraceRoot.Al 3 Department of Statistics, Rutgers University *Department of Computer Science, Rutgers University
Abstract
Diffusion models have emerged powerful tools for generative tasks, producing high-quality outputs across diverse domains. However, how the generated data re- sponds to the initial noise perturbation in diffusion models remains under-explored, which hinders understanding the controllability of the sampling process. In this work, first observe interesting phenomenon: the relationship between the change of generation outputs and the scale of initial noise perturbation highly linear through the diffusion ODE sampling. Then we provide both theoretical and empirical study to justifyy this linearity property of this input-output (noise- generation data) relationship. Inspired by these new insights, we propose novel Controllable and Constrained Sampling (CCS) method, along with a new controller algorithm for diffusion models, that enables control over both the proximity of individual samples to target image and the alignment of the sample mean with the target, while preserving good sample quality. We perform extensive experi- ments to compare our proposed sampling approach with other methods both sampling controllability and sampled data quality. Results show that our CCS method achieves more precisely controlled sampling while maintaining superior sample quality and diversity, enhancing the applications of precise image editing. The code is available at
Introduction
Recently, diffusion models achieve remarkable success in generative tasks such text-to-image generation, audio synthesis [14 as well as conditional generation tasks including inverse problem
solving, image video restoration, image editing, and translation 3 28 44
34
9 26
37
20 45)
Despite these successes, real-world scientific and engineering problems pose more challenges on requesting reliable and controllable generation well data privacy.
To tackle this, one important question is: How to control the distribution of samples from a diffussio model to match a specific turget? Previous works on controllable generation with diffusion models mostly focus on constraining the generation process sample-br sample-by-samen by-sample using either plug-and and-play 16 6). that each sample approaches [29 26 37] modifying the unconditional score [3 45
can satisfy a measurement constraint. However, most prior works focus on per-sample control, with limited exploration of how to regulate the overall distribution of generated samples meet specific
Joint senior authors.
Correspondence: Bowen Song (bowenbw @ @umich.edu), Guanyang Wang (guanyang wang @ @rutgers.cdu), and Liyue Shen (liyues @umich.edu)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

statistical constraints, which is a crucial requirement in differential privacy [1] This inspires the novel task for controllable and constrained sampling we are targeting this paper. Considering the unique mechanism in diffusion sampling, we are motivated to exploit the initial noise control by studying this key question: How do the initial noise perturbations affect the generated samples in diffusion models? Previous works 2 41 suggest that the learned posterior mean predictor function is locally linear with perturbation among a certain range of timesteps for diffusion models. However, this linearity cannot be applied to every timestep nor to the samples of diffusion models, From new perspective, this work sheds lights on the relationship between input noise perturbations and generation data in diffusion models, by proposing a training- free approach.
First of all, we observe an interesting phenomenon that when using denoising diffusion implicit models (DDIM) sampling, the initial noise has a highly linear effect on the generation data at small or moderate scales. Motivated by this observation, our study tries to justify this linearity property via initial noise perturbation theoretically and empirically.
Based on the spherical interpolation to perturb the initial noise vector, we propose novel Controllable and Constrained Sampling method (CCS) for diffusion models to sample with target rMSE level, enabling the sample mean to be close to the target image, while preserving high quality and adjustable diversity. The motivation for this task stems from a fundamental need in image editing and controllable generation: preserving key source features while allowing controlled variation. However, few studies benchmark sample quality and key feature preservation at target controlled variation level. Our first key idea is to fix the average distance (rMSE) between samples and the target. enabling a fair comparison of sample diversity and feature preservation. Our second insight is to evaluate the distance between the sample mean and the target image, which reveals how well common features are preserved. In addition, our CCS algorithm enables coer-controllable "diversity slider": tool that adjusts how far generated samples deviate from the input image. This fine-grained control over similarity can be vital for practical applications such as photo editing apps.
Furthermore, we conduct extensive experiments validate the linearity phenomenon and then inves- tigate the controllability performance of our proposed CCS method by generating images centered around a specified target mean image with a certain distance. Results demonstrate the superiority of our CCS method in both controllability and sampled image quality compared with baseline methods. Moreover, we show the potential of proposed CCS sampling for broader applications including precise image editing.
Our contributions can be summarized as below:
We unveil a novel linear relationship between the initial noise and generated samples for DDIM sampling. We justify it theoretically, validate it thoroughly through extensive experiments, and discuss practical implications.
We propose novel task of controllable generation with the goal of making sample mean close to target mean while controlling the MSE of samples to a target level. To the best of our knowledge, we are the first to study this task. This task can be useful for benchmarking the performance of personalized image generation.
We propose a novel controllable sampling method based on our discovered linearity relationship. Extensive experiments with both pixel and latent diffusion models demonstrate the superior performance of our algorithm in achieving precise controllability within a our proposed constrained sampling framework.
2 Background
Diffusson Models. Diffusion models consists of a forward process that gradually adds noise to a clean image, and a reverse process that denoises the noisy images [35] 38]. The forward model is given by X, = X,-1 0,5,8, + Atw where w E N(0. I) and ((t) is the noise schedule of the process. The distribution of x, is the clean data distribution, while the distribution of x is approximately a standard Gaussian distribution. When we set At - the forward model becomes dx, -0 + VB,dw,, which is stochastic differential equation (SDE). The reverse of this SDE given by:
dæ, = (- (-B()) B(t)
B(t)) logp((x)) dt + +.+/V /B(t)dt.
2
2

---

3
Cu
Co
Figure 1: Qualitative demonstration of linearity when increasing scale of perturbation. For each target mean, we sample a perturbation noise and gradually increase Co (0.1 at a time) to increase the magnitude of the perturbation.
One can train a neural network learn the score function Xx, log (Xe). However, this formulation involves running many timesteps with high randomness. We can also compute the equivalent Ordinary Differential Equation (ODE) form to the SDE, which has the same marginal distribution of p(xx). A sampling process, called denoising diffusion implicit models (DDIM), modifies the forward process be non-markovian, so as form deterministic probability-flow ODE for the reverse process [3). In this way, we are able to achieve significant speed-up sampling. More discussion on this can be found in Section 3
Constrained Generation with Diffusion Models. Constrained generation requires to sample Xo subject certain conditions or measurements y. The conditional score at T can be computed by the Bayes rule, such that
Xx, (x Vx, (x+) + Vx, log (y(peey))))
(1)
The second term can be computed through classifier guidance [3] where an external classifier is trained for ()(y)x)) or (y xx), and then can be plugged into the diffusion model through Eq. [
Diffusion posterior sampling further refines this formulation by proposing perform posterior
sampling with the approximation of p(y xx) so p (Ro), where Xo is the Minimum Mean Square Error (MMSE) estimator of xo based on Xt.
Another line of works exploit hard consistency, which projects the intermediate noise to a measureeeent-consistenn space during sampling via optimization and plug-and-play 7 9 29 34
However, the projection term can damage the sample quality (6]. However, these works all target on controlling each individual sample. To the best of our knowledge, few works explore how control the distribution of generated samples to match certain statistical constraints, such as centered around specified target mean with certain distance, which is the target for this work.
Noise Perturbation in Diffusion Models. Noise adjustment for diffusion models has been explored [8,
42 46) for changing the style or other properties of the generated data. However, a principled study on how the noise adjustment affects the samples is limited in diffusion models. Recently, 2 41 observe the local linearity and low-rankness of the posterior mean predictor Xo based on x, in large timesteps, but this study cannot extend to the analysis of generated samples. this work, we investigate how initial noise perturbations affect generated samples from the diffusion model in the ODE sampling setting.
in image editing, video generation, and other applications [28] 44 10,
3 Linear Relationship between Initial Noise and Outputs in Diffusion Models
This section analyzes how small perturbations the input noise affect the generation data under the DDIM sampling framework. We show that a slight change in the initial noise leads to an approximately linear variation the sampled images. This result quantified from two perspectives: the discretized DDIM sampling process [36] and the associated continuous time ODE, Our mathematical analysis
3

---

relies on minimal assumptions, which also serves as the foundation for our proposed CCS algorithm in Section
3.1 Preliminary: DDIM Sampling
Fix the total sampling timesteps T and an initialization noise sample xT. [36] generates samples from the backward process XT - XT-1 7 ..... Xo using the following recursive formula:
X v1 Cee Co (xx)
Xt-1 vac-1
+ V a--1 of Eg (xx) dyct,
(2)
Var
where corresponds to the noise schedule in DDPM, (1) (xx) is the predicted noise given by the pre- trained neural network with parameter 0, e, the standard Gaussian noise, and o, a hyperparameter. The DDIM sampler [36] sets = 0 to make the backward process deterministic once Xy is fixed.
known (e.g,, eq (11) of 14 that predicting the noise equivalent to predicting the score
function up to normalizing factor, i.e., <g (xx) ~ - vI - Vx (x()). By setting 0
and substituting Eg with its corresponding estimand, we obtain the idealized DDIM process:
(e)
X₁ + (1 04 Vx. log Pt (x)))
- -
X a1-1
( aI- at) Vx, (xt))
(3)
If we treat the index f as a continuous variable (and rewrite 0t as (t) avoid confusion), we can write the idealized ODE as:
X
dx, - log
dor(t..
(4)
(o2 +1
We now examine how a small perturbation XT + XT + AAx would affect the output sample at time t 0 through both the discrete 3 and continuous time (4) perspectives.
Related work: Theorem in 3 presents a related result on the impact of initial noise perturbation. Our study differs from theirs in a variety of aspects. Firstly. they study E[x, X + AAx] - EE [xxo Xx]
under the (stochastic) diffusion process. In contrast, we directly examine the output Xo given the
initializations x, and x, + XAx under the deterministic DDIM (3) or the ODE process 4 Secondly,
assumes that Po a low-rank mixture of Gaussian distributions, which allows for an analytical
solution for pt. In contrast, our weaker assumptions render analytically intractable. Consequently, we use very different techniques, such as ODE stability theory and Grönwall's 's inequality, to study the system's 's behavior.
3.2 Linearity in DDIM Discretized Sampling
Previous works reveal the local linearity of the denoiser (which learns the score function) is quite strong 2 27 in certain range of timesteps for diffusson models. Indeed, we can demonstrate that at very large noise levels, the score function approximately linear. a distribution is Gaussian, its score function is a linear function. Let: x ~ Nu, Σ) the score function is given by: Vx log p(x) -x (1) which linear in x.
This explains why the denoiser exhibits high linearity at large timesteps as observed in 27 For DDPM, since x, Vo,x, + /1 - ar e where (O, I), / increases, the noise term dominates. and pxx.) approaches a Gaussian. Based on this observation, we can derive an approximately linear relationship between change the input initial noise and output of DDIM sampling as demonstrated in Proposition [
Proposition 1. With all the notations defined as above, assuming log pt is second-order differentiable for every > there exists matrix-valued function "Yo such that
functiion
(x + AAx, T') = xo(xT,T) + Xx +
In turn.
|xo AAx, T)) xo [xT, T [1 Ax o(2).

---

Proposition I shows that a linear perturbation of the imput with magnitude À and direction Ax results in an approximately linear change in the output, with magnitude (olll Ax||2 and direction (n(x))xx Recalling Eq. 2, each idealized DDIM sampling can be viewed as línear combination of the current intermediate noisy input x, and the score function Vx, log pt (x)). Based on this observation, Proposition [ can be derived recursively using the linear approximation of the score function since each DDIM sampling step takes a linear combination of the predicted score and the intermediate noise. The derivation can be found in Appendix A. Our assumption is based solely on the second-ordes smoothness of the score, which weaker than most existing assumptions depending on the data distribution po. For example, assumptions hold under common conditions in the literature, such as the manifold hypothesis [I 38 or the mixture of (low-rank) Gaussian assumption 17 [.
Furthermore, at large I, Pt. is approximately Gaussian and Vx, 1o (x+) ) is smooth, which implies small linear approximation error. The reason for the small error is that: 1) the score function of Gaussian is linear in x by Vx log p(x) = -E-1 (x (4), 2) when a function is smooth and its higher- order derivatives are small in magnitude, it has fewer abrupt changes, and the linear approximation error is bounded by the norm of the Hessian of the score function (through the Taylor Remainder Theorem), leading low linear approximation error. However, one might be concerned that the linear approximation error could grow significantly when / decreases and po contains multiple clusters with low-density regions in between. Nevertheless, we now explain why this concern does not arise in practice. The coefficient f(t) == === 1 ax) + v1 04-1 of & (xx) in (2) is close for small I, as O₂ 1. Moreover, the structure of the neural network eg ensures that the output is
normalized and bounded in norm, so the change in output is also bounded. Consequently, for a small
perturbation in x,, we have (t()
(t) (x; Ax) (2) (x1))ll2 R O when & is small.
Linear Approximation Error. We provide further analysis of this linear approximation error in the Appendix. We derive that this error is affected by the magnitude and the smoothness of local probability density p(zo- = sample)
3.3 ODE Stability
Let Xo [x T) be the solution of (4) with initialization xy (i.c., X7 and zo (x, T') = a(0))x(xx T"). With some technical assumptions that is detailed in Appendix, we have the following:
= xT= =>\va(T) at timestep T,
Proposition 2. There exists a matrix valued function tho such that:
8x(x7 + AAx T) = Xx T) + X (xbo(x)) Ax o(A).
In turn,
Xo(XT AAx, 7) Xo T) + ( )x o(A).
Proposition 2 mirrors Proposition I but is formulated in the continuous- time ODE setting. Its proof relies on ODE stability theory, showing that the output change is "approximately linear" for sufficiently small A. Furthermore, under the same assumption, we establish that the change remains "at most linear" for all X. The proof, which applies Grönwall's 's inequality, is provided in Appendix. Proposition 3. With the same assumptions as above, there exists a constant C(T) depending on T such that for any X:
[X AAx, -x (xT,
T)lk S C(T))]|AX|]
4 Sampling with Control
Our objective is to perturb xT into random x/. such that the generated image x such that has 1. a sample mean close to Xo while maintaining 2. sufficient diversity and difference from the original image and high image quality. We preserve the notation Xy to denote a "target image" or "target mean". We also preserve the notation Xy := DDIM 0, T), the "noise" by finding a reliable initial noise Xy, such that DDIM(x,) = Xu The closeness is quantified by L-2 norm distance ||[[[[] - Xoll2, and the diversity is measured by E[|lx| - xxl xol [2] A notable feature of our algorithm is that users can specify a desired level of diversity (such as using Co in Fig D. and the generated images will match this level while ensuring [R[6]] & Xo. Our mechanism is defined as x/ = axt + bd, where A is a random perturbation, and a and b are parameters to be specified shortly.
5

---

4.1 Sampling around a Center
For an input of the form x4- oxy + Pλ with random A, when b small and a close to 1, it can be regarded as slight perturbation of Xy. Based on Section the output will remain close to with an additional linear adjustment applied lm. Thus, we define X : Xo + bAÅ as an approximation for Xo. where A 7a(ax ba) specified in Proposition]] Since A is the only source of randomness in X/y we can easily calculate E[xy] Xo + bAETA] and Var[&]] = 6² A Cov(A)
We will now
discuss the principles for our sampling design.
High-quality image generation: we first note that the input to both DDPM and DDIM samplers is standard Gaussian noise. The following feature is known as the "concentration phenomenon" of high-dimensional Gaussian:
Proposition 4. Let ~ N(O, Ia), then for any 5 E (0,1)
P [ E (1 + 5)d] 2 2exp
(-2
0
- 83)
This result suggests that standard Gaussian noise vector remains close to hypersphere of radius vd.
Our approximation X/ has expectation E[&] = X + bAF[AA. Thus, is sufficient select A such that B|A] 0 in order to achieve: |[xx]] [[x] Xo, where the first approximation is justified by Proposition [ aandd 2 with further empirical validation in Appendix.
Close to target mean:
4.2 Centering Feasibility
The simplest strategy is to add a random noise vector Ax directly xy, expressed as xyy XT + Ax (with = 1, 6Λ = Ax). However, the following proposition demonstrates that this approach cannot produce high-quality images.
Proposition 5. For any fixed vector x, and any random vector Ax such that E[Ax] 0, the following holds:
[|[[ Ax ||x12 + tr(Cov(Ax)) 2 |x|
with equality if and only if Ax 0 almost surely.
Proposition indicates that directly adding noise, Xy Xy Xy Ax, pushes X7 farther from the spherical surface. This partly explains why the average image becomes blurrier or noisier as the scale of Ax increases, since the drift term tr tr(Cov[Ax)) grows larger, causing x/. deviate further from the sphere with radius ||x|||2- On the other hand, a simple linear interpolation such as for also cannot produce high-quality images", because this will shrink the magnitude of the interpolated vector, which we demonstrate in the experiments. This inspires us to consider the spherical linear interpolation method [33] for sampling, described below. Similar approaches have been proposed by [46]] 35), but only for interpolating between two images.
4.3 Spherical Interpolation
Let vectors a and b satisfy ||a||2 |blla and form an angle 6, Then for any E (0. 1), the vector obtained through spherical interpo-
Algorithm 2 Controller Tuning (CT)
1: Input: target mean target diversity level MSE tolerance; tol, Co. and Ckigh
lation с : sin 0 a + sin((1-a)0) satisfies
sin(a()
2: Initialize: Co + C.n+-Cugh
sin(0)
||c||2 |all2 |/ll2. In case, for stan-
2: while not converged do
dard d-dimensional normal noise vector e, it is known || x Vd ||x Therefore, we can do spherical interpolation between Xy and to obtain x Our CCS algorithm is described in I
3: Sample a batch of x/, by Alg.
4: if [[[[[x]] - xo||2] MSFn < tol then
5: Break
6: else if B[[[x] X [2] > MSEnrgee then
7: (Crigh: CT) <- (Co. Co+Cli
8: else
The perturbation mechanism corresponds to
9: (Clow, Cb) <- (Co, Co
x = axT + bA with a sf
10: end if
Ci)))in(00, sin(Co)/ sin and A is
10: end while=0

---

Algorithm (Full Inversion) CCS Sampling
Requires:
target mean Xo. perturbation scale Co. number of diffusion model timesteps T
Step 0: Compute the DDIM inversion of Xp. i.e. XT = DDIM (xo,
(x,,0, 0,
Step 1: Sample noise N(0, I). Then compute
i cos-
X.XT
Step Compute x using spherical interpolation formula:
sin(Co)
sin(0 Cu)
XT
sin(
(0)
Step 3: Output sample x/1 DDIM
standard Gaussian noise. Co == a& is defined as the parameter of perturbation scale. This mechanism satisfies the design principles described in Section 4.1 E|c] = ensures that the new sample remains close to the target mean, while the Gaussian concentration and spherical interpolation ensure that |xx llz Re ||x||22, resulting in high-quality generated images. Parameter Co controls sampling diversity. In the extreme case Co O, we have x xy, x matches X exactly but has no diversity. A larger Co makes the perturbed input deviate more from the original image and gets closer to noise. This leads greater diversity in the generated image.
Algorithm alloows users to control the desired level of diversity. It works by calling Alg. | for different values of Co. which are determined through binary search. The process is repeated until the desired diversity level (up to a small tolerance threshold) reached: if the MSE of generated images to target mean is below target threshold, Co is increased; otherwise, it is decreased.
The following theorem demonstrates that the CCS algorithm is able to precisely control the input distance.
Proposition 6. Denote the dimensionality of XT by d. Given an initial noise XT with ||xT l| (1+o(1)))Vl and fix small 8 > 0. For any M < (2 8) 8) then can find Co in Algorithm 1 such that with probability Pa + as + oo, we have |2- Xrl12 -M.
Since the dimensionality of our problem is sufficiently large, Proposition 6 allows users to control M as the input distance. Consequently, Algorithm can generate random interpolants with an exact distance of M from the input. Furthermore, since the direction uniformly distributed, and when Co
is small, E [
R E[xr], and E[x]] ~ E[[x]), which satisfies our design goal.
In other cases when the inverted noise does not lie the standard Gaussian hypersphere, argue that our proposed spherical interpolation leads the second moment closer to standard Gaussian. Formally, Fix any vector x E Id,, and let N(O, la). Let 0 E [0, be the angle between and e. We define the interpolated vector:
sin(cf)
sin((1 c)0) sin(O)
y
sin 0
Our goal is to show is closer than to a Gaussian in the second-moment (energy-shell) sense. Since a standard Gaussian has second moment E|| [2] d, we define the gap second moment as:
8(Y) | d|.
Proposition 7. For any (0, 1), we have:
8(y) S 8(x).
In summary, we argued that we can center our samples around the target mean better through spherical interpolation with random noise as in Prop. 5 and control the distance to the sample mean through adjusting the perturbation scale Co in Prop. 6 and we can also improves the sample quality even the initial noise is not Gaussian hypersphere demonstrated in Prop.
7

---

4.4 Extension to Conditional Latent Diffusion Models
Conditional diffusion models usually compute the conditional score with classifier-free guidance (CFG). Let (x(x,,1) 1) be the predicted noise, it can be written in 8o(x,,) () so(xe, L, Cooi) + ((5g(X,, t, c) se (x,,, t, Craa)) where yy is the CFG term, c is the condition and Coall is the null condition. The computation more expensive, and drastically by a small perturbation. Motivated by this, we propose a algorithm (P-CCS). Instead of starting from the T, we pick an intermediate timestep to- Then, we compute the noise term from DDIM inversion by subtracting the clean component, sample a new noise from N(0, (1 (t2) )), and then perform spherical interpolation. Details of this partial inversion algorithm (P-CCS) can be found in the Appendix.
may not want change the semantics
CCS Sampling
Applications and Experiments 5
In this section, we will discuss the applications of the observed linearity property with the proposed controllable sampling techniques.
5.1 Lincarity Property
Table 1: Rº between input perturbation and normalized
residual norms
Experimental Validation. We perform extensive experiments on both pixel diffu- sion models on the FFHQ (22) and CIFAR- 10 (25 dataset and latent diffusion models on the Celeba-HQ and fMoW dataset [5]
Pixel Diffusion Models
Latent Diffusion Models
FFHQ
CIFAR-10
CelebA-HQ
fMoW
0.995
0.988
0.959
0.947
For each experiment, we first sample 50 images target images from each validation dataset from FFHQ, CIFAR- 10, and Ce. We also pick one images from each class from the validation set of the fMoW dataset for further verification. Then for the FFHQ and CIFAR-10 selected data, we use pixel diffusion models as backbone; for Celeba-HQ and fMoW we use stable diffusion 1.5 as the backbone. For each target image, we sample eight Co from uni- form [0, 0.9] distribution. For each Co. we sample 24 images. Then we compute the average L distance between the sampled images and the target mean for each scale. We compute the R-squared coefficient (R²) between the input perturbation scales and the normalized average resid- ual norms (scale between 0-1). As shown in Table 4 we observe very strong linearity in the above experiments. Fig. also demonstrates linear semantic change visually. We provide ad- ditional analysis in the Appendix. In summary, the linearity widely exists for DDIM sampling regardless of dataset or model backbone, and it may heavily depend on the dataset distribution.
To further investigate how the linearity changes with the complexity of dataset, and different diffusion model backbones. We perform experiments to test the linearity of (I) pretrained diffusion models on a simple dataset such as FFHQ (2) pretrained dif- fusion models multimodal dataset (with many classes) such as ImageNet. (3) large pretrained foundation models such as Sta- ble Diffusion Model (trained on complex multimodal dataset). We hypothesize that the linearity on out-of-distributiin dataset will decreases, so we test the pretrained pixel-diffusion models on OOD datasets. While the training data for SD1.5 is very large (LAION-5B), we just test it on other multimodal datasets such as UCF-101 [39 and ImageNet [12]. Results show that the linearity decreases significantly when testing on OOD dataset for pixel-diffusion models. For foundation models, complexity of dataset for sampling does not affect the linearity significantly. The linearity decreases slightly comparing diffusion models trained on multimodal dataset to those trained on simple dataset. These results validate the analysis in the linear approximation error, and imply that low probability density and sudden change in the
zomoated restaal nome
sormakoed www.aaooddddduaeeeeeeee resdbal
12
Quantitative demonstration of linearity when increasing scale of perturbation. With increased sin(Co), the magnitude of perturbation increases, and the average L distance between samples and the target image increases linearly. Left is the linearity on FFHQ dataset using pixel diffusion; Right is the linearity on Celeba-HQ dataset using Stable Diffusion 1.5.
Figure 2:
8

---

YHE
1.0
0.83
0.67
0.5 Edit sirength (C)
0.33
0.16
Figure 3: Precise image editing with the proposed P-CCS algorithm. Source prompt: "A high-quality portrait of man". Target prompt: "A high-qualing portrait of woman". The right-most images with edit strength are the source images.
data distribution may contribute to low linearity. We also find significant drop of sample quality with low linearity. Details and numbers can be found in the Appendix.
Application 1: Precise Image Editing. With the aid of linearity property, we can perform the application of precise image editing, by having the user enters value of edit strength to precisely control the target image edited that extent. The key idea is through the DDIM inversion project both the source and target images back to the initial noise manifold. Specifically, we first compute
IT DDIM (zo, Gsouce)) ZT DDIM (20, Cugaa). and then perform spherical interpolation
(1)
between ut (1) and 2/ according to the user-specified editing strength. This is implemented using the proposed P-CCS algorithm with more details described in the Appendix (Alg. 4 As shown of
two example images in Fig. 3 our algorithm can easily achieve smooth and precise image editing
guaranteed by the aforementioned lincarity property.
5.2 Controllable Sampling
We propose CCS (Alg. D and P-CCS (Alg. algorithms for controllable sampling close to a specified target image, constrained by target MSE the target mean. We validate that algorithm achieve this better than baselines while preserving good image quality. In addition, we demonstrate algorithm's capability in generating personalized albums and improving sample quality.
Application 2: Generating Personalized Album. We perform experiments generating per- sonalized albums using both pixel diffusion models and latent diffusion models with our (P-) (P-)CCS algorithms. For benchmarking performance of different baselines, we propose novel task of fixing MSE to a target image, and compare other metrics, which we call controllable sampling. The goal is sample images as close source possible while keeping target diversity (MSE).
Experimental setup:
We FFHQ-256 22 and CelebA-HO 43 test set images as target images.
We use ADM (a pixel diffusion model) for FFHQ, and Stable Diffusion for CelebA-HO Baselines: We self-implement baselines as comparison since no existing work is designed for the target task so some adaptation is necessary. Naive Linear interpolation with Controller (LP-C), Gaussian
CCDF-Controller CCS (ontsl-Cr Target Mean Figure 4: We sample 120 images with fixed target mean using different methods and analyze their sample mean (average pixel intensity). Our observations show that the sample mean of our method closely matches that of the original image.
GP-Controller
DPS-Controller

---

Perturbation with Controller
Table 2: Results of Pixel Diffusion models the FFHQ Dataset
(GP-C), Diffusion Posterior
with target rMSE set as 0.12.
Sampling 66 with Controller
Method
PSNR
SD
CLIP-IQAT
MUSIQU
(DPS-C), ILVR with Controller
GP-C
18.88
0.028
0.701
45.88
(ILVR-C), Come-Closer-Diffue
ILVR-C
20.04
0.070
0.746
62.45
Faster with Controller (CCDF-C
DPS-C
21.02
0.069
0.738
64.60
or SDEdit-C). Metries: We adopt Peak Signal to Noise ratio (PSNR) for measuring whether sample mean close to target mean; LPIPS for measuring samples similarity to the source image; CLIP-IQA and MUSIQ for semantic/low-level image quality respectively; Standard Deviation (SD) for measuring sampling diversity. More experimental details and baseline implementation can be found in the Appendix.
CCDF-C
23.52
0.088
0.746
66.15
CCS (Ours)-C
25.13
0.104
0.750
66.79
Results. We observe that our CCS sampling method signif- icantly outperforms all other methods in centering at tar- get mean constrained by a fixed rMSE distance, while surpris- ingly maintaining superior image perceptual quality and diversity. Other posterior sampling meth- ods such DPS suffer from image quality degradation and diversity decrease, shown by the quantitative results reported in Table. iaee.2ee and 3 Qualitatively, we observe that the sample means of other methods look blurry or noisy, as demonstrated in Fig. 4 More qualitative results can be found in the Appendix.
Table 3: Results of the Stable Diffusion 1.5 on the CelebA-HQ
dataset with target rMSE set as 0.07.
Method
PSNR
SD
CLIP-IQAT
MUSIQ
GP-C
23.02
0.045
0.721
48.91
LDPS-C
24.56
0.034
0.721
29.07
CCDF-C
27.66
0.051
0.735
49.29
CCS (Ours)-C
30.29
0.053
0.732
49.66
Application 3: Improving Im- age Quality through P-CCS Sampling. Note that our (P- CCCS sampling algorithms, we perform spherical interpolation with random Gaussian noise. Intuitively, the initial noise is not Gaussian like falling in low- density probability region, con- ducting such interpolation will make "more Gaussian" to in- crease the likelihood of that ple so as to enhance image qual- ity. We provide formal argument for this in the Appendix. Motivated by this, we propose perform
Figure 5: Top: Corrupted images with artifacts or unreason- able structures. Bottom: Improved images by P-CCS algorithm through spherical interpolation of initial noises.
P-CCS at some timesteps of reverse DDIM sampling. As shown in Fig. 5 we observe that the sample quality can be improved significantly by this simple method, which supports the potential of our findings to introduce new post- sest-training mechanism for enhancing image generation. More experiment details and quantitative results are described the Appendix.
Conclusion
In this work, we unveil an interesting linear response to perturbation phenomenon both theoretically and empirically in diffusion models. we also study a new problem: how to sample images with target mean and target MSE. We present novel sampling algorithm along with new controller method for achieving this goal. Extensive experiments show that our proposed method samples the closest to the target mean when controlling the MSE compared to other methods, while maintaining superior image quality and diversity. The limitations of our work include: (1) Controlling other interesting statistical properties beyond sample mean with MSE is left as future work. (2) There might be some artifact samples that exhibit overlapping patterns. (3) DDIM inversion may not be perfectly standard Gaussian, which may hurt sample quality. We believe the linearity property will be important for designing better latent space for large-scale diffusion models.
10

---

Acknowledgments
LS acknowledges funding supports from National Science Foundation (NSF) (IIS-4435746), Defense Advanced Research Projects Agency (DARPA), Hyundai America Technical Center, Inc. (HATCI), University of Michigan MICDE Catalyst Grant Award and MIDAS PODS Grant Award. GW acknowledges support from the National Science Foundation (NSF) (DMS (2M--2210899) and the Adobe Data Science Award.
References
[1] Sitan Chen, Vasilis Kontonis, and Kulin Shah. Learning general gaussian mixtures with efficient score matching. arXiv preprint 1xXiv:240411892 2024.
[2] Siyi Chen, Huijie Zhang, Minzhe Guo, Yifu Lu, Peng Wang, and Qing Qu Exploring low- dimensional subspace diffusion models for controllable image editing. The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[3] Siyi Chen, Huijic Zhang, Minzhe Guo, Yifu Lu, Peng Wang. and Qing Qu. Exploring low- dimensional subspace diffusion models for controllable image editing. The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[4] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14367-14376, 2021.
[5] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In CVPR, 2018.
[6] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. The Eleventh Interna- lional Conference on Learning Representations.
[7] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems,
2022.
[8] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-diffuse-ff faster. Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12413-12422, 2022.
[9] Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. arXiv preprint arXiv:2310.01110, 2023.
[10] Jiwoo Chung, Sangeek Hyun, and Jae-Pil Heo. Style injection diffusion: A training-free approach for adapting large-scale diffusion models for style transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8795-8805, 2024.
[11] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions Machine Learning Research, 2022.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255. IEEE, 2009.
[13] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances neural information processing systems, 34:8780-8794 2021.
[15] Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming, pages 1-12. Springer, 2006.

---

[16] Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems, 36: 16222-16299, 2023.
[17] ahashayar Gatmiry, Jonathan Kelner, and Holden Lee. Learning mixtures of gaussians using diffusion models. arXiv preprint arXiv:2404. 18869, 2024.
[18] Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting text-to-image diffusion models via initial noise optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9380-9389, 2024.
[19] Philip Hartman. Ordinary differential equations. SIAM, 2002.
[20] Yutong He, Naoki Murata, Chich-Hsite Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjum Kim, Wei-Hsiang Liao, Yuki Mitsufuji, Zico Kolter, Ruslan Salakhutdinov, al. Manifold preserv- ing guided diffusion. In The Twelfth International Conference on Learning Representations.
[21] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-comphench: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723-78747, 2023.
[22] Tero Karras. A stylc-based generator architecture for generative adversarial networks. arXiv preprint arXiv: 04948, 2019.
[23] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5148-5157 2021.
[24] Zhifeng Kong, Wei Ping, Jiaji Huang. Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[26] Taesung Kwon and Jong Chul Ye. Solving video inverse problems using image diffusson models. arXiv preprint 240.02400.02574 2024.
[27] Xiang Li, Yixiang Dai, and Qing Qu. Understanding generalizability of diffusion models requires rethinking the hidden gaussian structure. Advances in neural information processing systems, 37:57499-57538 2024.
[28] Haofeng Liu, Chenshu Xu, Yifei Yang, Lihua Zeng, and Shengfeng He. Drag your noise: Interactive point-based editing via diffusion semantic propagation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6743 6332, 2024.
[29] Sai Shankar Narasimhan, Shubhankar Agarwal, Litu Rout, Sanjay Shakkottai, and Sandeep P Chinchali. Constrained posterior sampling: Time series generation with hard constraints. arXiv preprint arXiv:2410. 12652, 2024.
[30] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162-8888888 8171. PMLR, 2021.
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[32] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[33] Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pages 245-254, 1985.
12

---

[34] Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfth International Conference on Learning Representations.
[35] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.00500, 2020.
[36] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2021.
[37] Jiaming Song. Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and controllable generation. In International Conference on Machine Learning, pages 32483-32498. PMLR, 2023.
[38] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019.
[39] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human 2012.
actions classes from videos in the wild. CoRR,
[40] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2555-2563, 2023.
[41] Peng Wang. Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409.02426 2024.
[42] Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero-shot image editing and guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7378-7387, 2023.
[43] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[44] Jiaxin Zhang, Kamalika Das, and Sricharan Kumar. On the robustness of diffusion inversion in image manipulation. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models, 2023.
[45] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[46] PengFei Zheng, Yonggang Zhang, Zhen Fang, Tongliang Liu, Defu Lian, and Bo Han. Noised- iffusion: Correcting noise for image interpolation with diffusion models beyond spherical linear interpolation. In The Twelfth International Conference on Learning Representations, 2024.
13

---

A Proofs
A.1 Proof in Section 3.2
Proof of Proposition 7 Let L (x) == 1/X + Vx. log (x) be the one-step recursion. Our Xo t) formally defined as L1 L2 .... (a).
The second-order differentiability of pt implies the score function V logp Pt is first-order differentiable. Let H, be the Hessian matrix of logpt (H, = 8 ftgp:/00p 000;). We have
V lx = (w) + H₂(w) (x w) + of|xx - w112).
Therefore, for any fixed direction of unit length and 6 R,
LT(x + dw)
= pr(x + dw) +
= -rxx \T loggpr(x + (TÓHH S1 + o(8) -x((x) + 8(1r + Xy (x)) + o(8)
= LT()x + 6x)( o(6)
where (r( is defined
(x) =
\y Hy(x),
is matrix-valued function which is bounded if the norm of the Hessian of log is bounded.
Applying Lr-1 both sides of the above formula:
LT-1 Lx(x + dw) = LT-1 (x
o())
- -ww - Vog-Poogpr- LT(x) (x)
= ( + - - (1- -
recursion unperturbed data
linear
lower order term
LT (T(x 1 + o()).
where
-x = (1r-11 - (Lr (x)
So we have
xx 8w, 8w, := Lo o. LT (x( + 5% (x) o(8)
Now let À be the scale of the perturbation, such that A > 0 and À E R, and let Ax be the unit-length perturbation the initial noise XT, have:
XxT AAx,T) (XT, T) + Ao0xx) o(AA)
We could continue applying LT-2, Lr Ia on the above formula, and conclude:
xo(xr + AAx, T) = xx()x) + Axo(x)) Ax + o(A).
(5)
We might be particularly interested the distance [|x + AAx, T) - xo(xr, Tb. our calculation directly implies:
[|xo(xT + AAx, 7) (x(, T)ll2 = (xxox))Ax((/ + o(A) = (x) + o(A).
(6)
by applying triangle inequality twice:
(Ava(x)) Axx|2 [|o(A)]12 I |xx(Xx AAx, T)
(xxxx1l22 - [xx Axx(22 ||o(A)]l2
14

---

A.2 Proof in Section 3.3
We first state the detailed assumptions posed in Section 3.3
Define the function
h(t.y):= :- =2aa))))Vat) a (t)
V logpe
y
of
(o +1
We assume this function has continuous derivative (i.e., (a the whole space [0, 2] R Moreover, we assume there exists C(t) such that:
[| y² - htt, x)ll2 & C(t)lyy - x||22
for every x, x, t, and [a Ct)) < C oc.
Proof of Proposition 2 as:
exists a unique solution. We can rewrite the We first show the ODE 4
dx, = -- aa(VVVoog
da((1)
(t² +1
-d'(t)/1-- logat
Xe
dt
Vo2(t) +
va(t) 0'1 at
X
logp:
dt
as a(t) = ((1-a(t))/a(
+1 (1)
d,)tt
Given h(t, C and unifomrly Lipschitz in y, it follows from the Picard-Lindelöf Theorem (e.g..
Theorem 1.1 of 19]) that our ODE (4) has a unique solution for any initialization Xy X.
Next, it follows from Theorem 3.1 of [19] that the solution (x 7) C², i.e., the solution depends continuously and differentiably on its initialization x. Thus,
xx(X7 XAx, 7) (x T) AJ& (Jx(xx) Ax o(A),
where Jx is the Jabobian matrix of the function Xo 1) with respect to x. This concludes the proof of Proposition 2
Proof of Proposition Let Xy and Xy. + XAx be two fixed initializations. Deñin
Yr = X4 (Xr) & (Xr + AAx)
as the difference between the solutions of 4 time E [0,
Taking derivative on y with respect to yields:
y' h(t, (XT)) lutt, & (XT AAx)).
By the Lipschitz continuity:
llx'll2 S (x)))(8)) xx(xr + AAx)]12 = C(t)llxellz
Denote yt by (y1,1, V2,
Vmmt) T, we have:
d d Eis v².
dlly:l12
dt
dt Em 2y 2y
Em y2,
Em Fiaa'a
livellz
||yellallvdll
Cauchy-Schwer inequality
-1l//l2-
15

---

Thererfore, we have
dlly=lle
S (y)ll.
dt
Applying Grönwall's incquality on the function lly.ll: ll. we have:
[['cejat) ||
lyell S exp
for every Stt < T. Taking we have
xx( + AAx, 7) - xo(xr, 7)l2 s exp (f. ((oos))) |lllax|22
as claimed in Proposition
A.3 Proof in Section
Proof of Proposition
It is known
E x
Ax
[|
||[ + Ax tr(Covíx Ax))
| + tr(Cov[Ax))
2 |
The equality is taken if and only if tr [Ax] = E; Var[Ax, = 0. This is equivalent to saying that all components of Ax are deterministic. Therefore, almost surely, Ax E[Ax] 0.
Proof of Proposition o Given a standard normal vector e, we claim the following holds:
xT
||e||2
|xTIlè
-2e XT
in T,2 d - To this, notice the first term is
Σ
which converges to 1 by the law of large numbers, since B[e]] by our assumption. The last term converges to 1,2 as
1. The second term converges to 1
2c Xx
2
4E
X2 1E+
4(d + o(d) + 0.
d
d2
d²
Therefore the distance ||c - Xrll2 converges to 2 d as d - xc. Similarly we can show {(c, xy), the angle between and Xy converges to x/2 as d co. In other words, e is approximately orthogonal to Xy when the dimension d is large.
Therefore, with probability 1 - o(), ), the angle 0 in Algorithm 1is 7/ 011. and [ndd||e-- ||- 1 + o1) as d + co. Fix any M < (2 6) d. since the spherical interpolation smoothly interpolate between Xa and e, there exists
satisfying Algorithm with input C' output x/ with distance M
to Xy with probability 1 o(1).
We indeed find explicit Co with slightly weaker guarantees, set
M
Co = cos
2 2||x,11
1
Then with probability 1 - o(1), Co E (0, क 7/2), and
sin sin)Co)
sin( + sin(0 - Co)
XT xr| S sin(@)
son(Co) sin(
sin(O - Co)
.e -c
Xy sinnCnnnC - sin(f - Co)xr
sin(f)
||in + sin(O Co)xT - XT
16

---

by triangle's inequality. Meanwhile. the first term is of /d) sin(O) - sin()))+ = ++) and cos(8) (1). The square of the second term
[ssn + sin(@ Co)xr - xrl2 = sin (1o) + - + Co) - 1)c x1
sin(Co)(d++ + - (1-sin(0 Co))
+
The last term is o(d) as e X - as we analyzed above. Using again 0 = x/2 - 7 we know sin(O Co) sin( Co) + o11 cos(Co) + o11) Hence we clean the above equation:
||sin(Co)e + sin(0 Co)xT xT|| d(sin (C))2 2 + (1 - cos)C)))) + o(d)
d(2 2 cos (Co) + o(d)
M
d 2 2 + M² o(d),
+ o(d)
||x|||
where the last equality follows from ||x 112 d + o(). Finally, taking the square root and plugging back into the triangle inequality, we have:
|x
xT| M + o(Vd).
Proof of Proposition We can write Tn where ro ||||| and X, x/ro belongs to Sri-1 the unit sphere in RR
Meanwhile, it well known that we can generate N(0, La) via sample rî (d) from the chi- chi-squared distribution with parameter d, 2) sample Unif(S (td uniformly the d-dim unit sphere, 3) set e = uy Therefore, let
0 = f(1) arccos(Xo: u)
sin(cf)
s0) sin 0
((0) = sin (1 - c)0) c)00
sin 0
We rewrite y as
y s(0) u t(0)oox
Now we calculate |
E|lly| | = E[s(0) ((((0) (() xo)]
Our first claim is the cross term E E(2rorss(0) x.)] zero. To see this, we first observe since and u are independent, we have
2r E (())))
We examine the expectation ([s(O))(0)( (u, xo)] where the only random variable is the direction vector u. Replacing with sends the angle 0 to 0. which will not change the value of s00), (0)). However, the inner product flips sign: uu, x,) = - (o) Consequently, the function of interest:
F(u) (((()))(((u)))))
is an odd function of u. Therefore,
Eu (s())(0)) xo)) =
Now the expected distance simplifies to
17

---

1 || E[s(0) | (0)
==E[s0)
The next observation is the following trigonometrie identity: +00 + (()) + ())()(())))((()) = 1
for any (0, 0 (0,
To show this identity, set a = c0, = (1 (1-c)0. Then sin(a)/ sitp:/sin(a++).... sin(3)/ sin( 3). The left hand side of the claimed identity equals
sin (a) + (3) 2sin(a) (in()) cos(a (3)
sin + 3)
Now we expand sin² + 3) as
sin? (a + B) (sin(a) cos cos(a) s3))
sin cos" (3) cos (a) sin" + 2 sii((a) cos(a) sin cos(B)
sin'(a) sin 2 si) sin
+ sin?(3) - sin (a) sin + sann(a) cos(a) sin cos(3)
=sin(()) + sin + aa) sin (3c(3) - sin(a) sin(3))
san)(a) + sin ( (3n cos(a + 8),
as claimed.
Leveraging this equality s(0) + +(0) 22()))(0) cos(0) = 1 and taking expectation with respect to implies:
((+0))+E(t((()))) = 1,
the cross term is proved before.
Now let s, = E [1s (0, 1), we have
[|]||||]] ds. r1(1 8c)
Therefore
8(y) | d sc)ld -1il +=)|||x| dll |||||| d) = -((x)
A.4 Why linear interpolation fails
To further support this point, we provide additional theoretical analysis to justify why simple linear interpolation does work well for sampling. Formally, let ||Ar||2 [æ]12, and its direction is uniformly distributed, and Az] = 0. For 0 < a < 1, we have:
[|x + (1 (1--al Az < |
Proof.
We have:
||ax (1 (1-a)Azl13
a² a²
2a ax Ax) (1-1) (1-a)?|AA[1
Since B[a(1 - (x Ax)] = a(1 - a) A = 0, we have:
||a - (1 a Ar = || + (1 - a) |
+ (1)
a) < 1, and we can conclude that using simple linear interpolation cannot preserve the norm, leading to falling apart from the Gaussian sphere.
Since 201 - 1) 0, we have (o² + (1-a)
18

---

B More Analysis on Linearity
B.1 Validation of Linearity Phenomenon changing Co
Experimental setting. We perform extensive experiments on both pixel diffusion models on the FFHQ and CIFAR-10 dataset and latent diffusion models on the Celeba-HQ and fMoW dataset. For each experiment, we first sample 50 images as target images from each validation dataset from FFHQ [2)] CIFAR-10 25 and Celeba-HQ 43 We also pick images each class from the validation set of the IMoW dataset [5] for further verification. Then for the FFHQ and CIFAR-10 selected data, we use pixel diffusion models backbone; for Celeba-HQ and IMoW we use stable diffusson 1.5 as the backbone. The prompt for Celeba-HQ is given by "A high quality photo of face" and the prompt for fMoW is given by "satellite images". Then, we use each image as a target mean and perform CCS sampling as in Alg
For each target image, sample eight Co from uniform [0, 0.9] distribution. For each Co, sample 24 images. Then we compute the average T,2 distance between the sampled images and the target mean for each scale.
Evaluations. To quantitatively evaluate the linearity phenomenon, we compute the R-square between the input perturbation scales and the normalized average residual norms (scale between 0-1) for datasets with both pixel diffusion models and latent diffusion models. Note that since different target means can lead to different slopes by different Hessian matrices, we normalize the residual norms. Specifically, we compute empirical slope a and bias b between x = sin(Co) and y = E[][x] - xolla] each target mean, and then normalize the average 2 distance to be: y セーb
Results. We observe very strong linearity in the above experiments. Especially for pixel diffusion models, the R-square exceeds 0.98 for both datasets, which indicates almost a perfect linear relation- ship. For latent diffusion models, the linearity is slightly weaker, but still above 0.94 in R-square for both datasets. This is expected since Stable Diffusion use a nonlinear autoencoder and trained on a different dataset. We also present more quantitative results in Fig. 2 and qualitative results in Fig. Surprisingly, we also observe a very linear semantic change in additional to pixel-value change.
Pixel Diffusion Models
Latent Diffusion Models CelebA-HQ
FFHQ
CIFAR-I0
IMoW
0.995
0,988
0.959
0.947
Table 4: R-square between scales of input perturbation and normalized residual norms
B.2 Validation of Compositional Linearity
Experimental setting, Here we just have motivation similar to the previous one. We first sample random noise E N0, I). and then another random noise E (O, I). We perform spherical interpolation on 6g and 61, and inference using a joint noise. We can adjust Co decrease the strength of sspherical interpolation to preserve orginal image structure. This. We sample Co uniformly and continue sample 6g and 6 for evaluation. Then we compare the cosine distance between DDIM(slerp(co. 61, Co)) and aDDIM((<)) + (DDIM((1), where and b are given by slerp(<o: (1, Co). We still find very large linearity, and the linearity decreases for Latent Diffusion Models. The quantitative results using four datasets, each with 100 evaluations are attached below in Table 5
Pixel Diffusion Models
Latent Diffusion Models
FFHQ
CIFAR-I0
CelebA-HQ
IMoW
0.958
0.942
0.901
0.920
Table 5: Cosine similarity between samples generated by spherical interpolated initial noise, and linear combination of samples.
B.3 Mathematical Explanation of the Linearity
Bound the Linear Approximation Error of Score Function
19

---

Let
voxo + v1 -1-66 dc, f(z) data distribution
1
va(a)= = 1 v8 V8¹
u(x) ~O - 6)
Then the density of z is:
fz(z) * (2)
Score Function and Its Derivatives
The score function is:
v log fz()) = Bu(z)) Vu(= 2 + z)] Bu(w) [M< x)]
Let:
A(z) F>)) + x)),
B(2) = Bu()) Vu(2+ C(z) = (V x)]
Then the second derivative is:
C(2)
(())
V log
A(z)
A(z)²
Assuming VA(=) = B()), VB(=) = C(z), and VC(2) exists, the third derivative becomes:
log = VC(2) C(=) @ B(z)
A(z) A(=)²
C(2)B(2)T + B(2) @ C(2)
2B(x)B(2)T & B(2)
A(z)²
A(2)3
Norm Bound on the Third Derivative
Suppose:
|C(())la s.
|IB(2)]l bb A(z)2a3) 0, ||VC(=)]l s
Then:
30c a a3
2b
v [1())l2 <
Taylor Approximation and Lincarization Error Bound
Let 2g be a reference point. Then the second-order Taylor expansion of the score function is:
log fz()) Vlog + lo f2(20) 20)
The remainder (linearization error) satisfies:
||V (oog log (=(0)(((==)))))))))) (2022) S sup rE|0.1]
R log f2(20 11= =00))]1-11==-00)
Using the third derivative bound, we get the formula for linear approximation error bound:
d
3bc
2h
(8(2,20)1l: -
+
||2 2011
a
a²
By this bound, we may argue that the probability density of the sampling center, the curvature and gradient of the distribution (smoothness) impact the linearity error the most. If we are the high probability region (a local maximum), assume that the curvature of probability distribution at that place is small, we will have low linear approximation error. So more complicated dataset may have less linearity since it is more likely to have discontinuous regions or low-density regions. This mathematical derivation explains the decrease of linearity from Pixel diffusion models LDMSs as shown in Table: 5
20

---

B.4 Linearity Analysis for OOD and Multimodal data
To further investigate this relationship, we perform additional experiments on three more datasets to test how the linearity changes for multimodal distributions. We are interested in:
1. How is the linearity when the model is trained on a multimodal dataset and samples an out-of-distribttion target image from another highly multimodal dataset, or an OOD simple dataset?
2. How is the linearity when the model is trained on a simple dataset (for example, FFHQ) and samples on an -ittt--diittitttionn multimodal dataset, or an OOD simple dataset?
3. How is the linearity when training on a simple dataset and sampling on the same dataset?
4. How is the linearity comparing model trained on multimodal dataset in-distribution and model trained on simple dataset testing in-distribution?
In our experiment section of the paper, cover partially 1, 3, and 4. We test pixel diffusion models on FFHQ and CIFAR datasets. We also test the Stable Diffusion model (1.5) trained on a complex dataset (LAION-5B) on the human face dataset (CelebA-HQ), We observe that diffusion models trained on simple datasets (FFHQ) and tested on distributions exhibiting significant differences show stronger lincarity than Stable Diffusion trained on CelebA-HQ.
We observed in our paper that linearity decreases slightly when comparing diffusion models trained on multimodal datasets to diffusion models with simple training data. CIFAR-10 10, being a dataset with different classes, shows a slightly lower linearity score compared to FFHQ even though CIFAR- 10 is lower resolution. Empirically, we observe sudden changes of image semantics occasionally.
1. We pick 4 images from classes 4,..., 99 from the validation set of ImageNet.
2. We pick five videos from classes: "Applying Eye Makeup". "Baby Crawling". "Billiar"', and "Blow Dry Hair" from the UCF-101 dataset, and sample five frames per video.
3. We pick 10 images from each organ site the AAPM dataset, which consists of CT scans of different body parts. These datasets are different from the training data of SD1.5.
We use the same linearity testing methods as in Section 5.1 of our main paper (and also in Appendix B). We first summarize the results for Stable Diffusion 1.5:
Dataset
ImageNet
UCF-101
CelebA-HQ
fMoW
R²
0.960
0.962
0.959
0.947
Cosine similarity of linear combinations
0.922
0.924
0.901
0.920
Table 6: Linearity statistics across multimodal datasets for Stable Diffusion I.5.
The results show that for multimodal datasets (containing many classes like ImageNet or UCF-101), there no evidence of a decrease in linearity compared to simple datasets (CelebA-HQ). For the same backbone model trained on a large multimodal dataset (LAION-5B), the slightly lower linearity CelebA-HQ may be due to data processing techniques such as upscaling, which introduces blurriness and removes latent noise patterns from the Gaussian sphere.
When testing the linearity on OOD datasets for models trained on simple datasets, we observe a significant linearity drop. We compare models trained multimodal data and tested on simple datasets. Results are computed using the same testing method as before and as described in Appendix B. Here, "trained - tested" means that the model is trained on the training set of the "trained" dataset and tested on the validation set of "tested"".
For fair evaluation, we use the same model architecture (DDPM++) with the same training loss for these two pixel-space diffusion models. We find that for non-foundation models, OOD linearity drops significantly. The multimodal backbone has a slightly lower linearity score than the simple data backbone due to complexity in its training data (it trains on LAION-5B). The probability distribution of multimodal images plays important role in linearity.
21

---

Dataset (trained - tested)
ImageNet
FFHQ
FFHQ ImageNet
FFHQ + FFHQ
R
0.934
0.938
0.995
Cosine Similarity
0.902
0.905
0.958
Table Cross-dataset linearity comparison between multimodal and simple dataset training/testing. Results are computed using the same testing method as before and as described in Appendix B. Here, "trained - tested" means that the model is trained on the training set of the "trained" dataset and tested on the validation set of "tested".
C Limitations and Clarifications
C.1 DDIM inversion not conforming to standard Gaussian distribution.
1. Investigation of Sample Quality Drop. To investigate whether there is sample quality drop, we use our CelebA-HQ validation set for additional experiments. This dataset is originally of size 256 256, and we upscale it 512 x 512, so it becomes slightly blurry and sometimes gives inversions not perfectly on the sphere. We partition the CelebA-HQ data into two sets:
encoded noise with mean deviation > 0.03 std deviation > 0.03 from standard Gaussian distribution.
Not STG noise:
STG noise: all remaining samples.
We compute performance metrics these images on Stable Diffusion with rMSE target 0.07. The
results summarized below:
Set
PSNR
MUSIQ
CLIP-IQA 1
SD 1
Not STG noise
30.86
49.43 49.74
0.734
0.053
STG noise
30.10
0.731
0.054
Table 8: Comparison of image quality for STG vs. non-STG noise at rMSE target 0.07 CelebA- HQ.
We do not find significant image quality differences between these two sets. Indeed, after our CCS interpolation the 0.07 rMSE target, the interpolated noise of the non-STG noise group all falls within 0.01 difference between zero mean and unit standard deviation. Empirically, at low rMSE levels, the noise may non-stan non-standard Gaussian, but the samples remain close the input image. At higher rMSE levels, the interpolated noise becomes more standard Gaussian, resulting in good image quality.
2. Verification of Gaussianity with Interpolation Strength. To verify that the Gaussian distri- bution becomes more standard with stronger interpolation, we compute the average deviation of mean (from 0) and deviation of variance (from 1) for the CelebA-HQ experiment. We find that the interpolation strength Cy increases, the deviation quickly narrows:
Co
Deviation in mean
Deviation in std
0.0
0.025
0.023
0.2
0,013
0.016
0.3
0.011
0.009
0.4
0.010
0.008
0.5
0.009
0.006
0.6
0.007
0.005
Table Deviation of mean and standard deviation across interpolation strength Co CelebA-HQ experiments.
22

---

3. Discussion. In our paper, Co is mostly between 0.3 and 0.6, so we do not worry much about the interpolated noise being non- non-standard Gaussian. We also conduct experiments at different rMSE target levels (with varying Co) and observe that increasing interpolation strength may lead to slightly better image quality when the input image is not very good.
C.2
Some other common confusions
There might be some misunderstanding of our method and experiments. To clarify:
the baseline CCDF just special type of SDEdit.
Our method does not specifically focus on improving quality in our main experiments in the main paper (the improvement in sample quality is very nice add-on). Instead, it focuses on controllability (how to sample around a mean with a target MSE). So even though our performance gain not that large, we achieve much better controllability over the edit strength which is measured as the distance between sample mean and the real input image (as demonstrated in Fig.
D Additional Results and Experiments
In this section, we clarify some implementation details, providing more details on algorithmss and visualization. We also provide more quantitative results and computational efficiency analysis.
D.1 More Details on Controllable Sampling
Experimental Set up For pixel diffusion models, we use the first 50 images from the validation data from the FFHQ-256 (6) dataset. Then we set each image as the target mean and then sample 120 images (6000 images in total) with each target mean with target rMSE (square root of average L-2
norm of the residuals between the sample and target mean) of 0.12. Then we test on the CIFAR-10
dataset. We randomly sample 20 images serving as target means, and then sample 120 images for each target mean with target rMSE level of 0.11.
For Stable Diffusion, we use the SD1.5 checkpoint 31 We study a more challenging scenario (degraded low-resolution input images with conditional text-guided latent diffusion model). We sample 50 images from the validation set from Celeba-HO dataset with resolution 256 x 256, and then use bicubic upsampling to upscale it to 512 x 512. Note that SD1.5 is not trained on the Celeba-HD HQ dataset so this demonstrates the generalization capability of algorithms. We use the same prompt and CFG level in the linearity control experiments.
Implementation Detail
We follow Alg. [ in implementing our methods for pixel diffusion models, and Alg. 3 for latent diffusion models. We take the pretrained models for FFHQ and CIFAR-IO from the saproved/guides diffusion repos (30 [3] for the pixel diffusion experiments, and the Stable Diffusion 1.5 33 for latent diffusion experiments. For LDMs, we set to = 45, where = 50 due DDIM inversion performing worse with classifier- free guidance than unconditional models. We set the rMSE target to be 0.1 0.11 1 for FFHQ and CIFAR-10 respectively, and 0.07 for Stable Diffusion experiments to test diverse control targets. The tolerance is set to be 0.01 in all cases. More details in the Appendix. Baselines Since we are doing a novel task, we self-design the baselines with our proposed controller algorithm as an add-on.
Gaussian Perturbation with Controller (GP-C): We add a Gaussian perturbation the initial noisy image ZTo, where the perturbation scale is determined by our controller. This method resembles works that perform local editing [
(Latent) Diffusion Posterior Sampling 6 34 with controller (DPS-C) C): We perform posterior sampling with 2g as the measurement. The scale of the gradient term in (L)DPS can control the randomness, so we design a controller based on this. Details in the Appendix.
ILVR with controller (ILVR-C): the ILVR algorithm 4 is for sampling high quality images based on a reference image. The larger the downsampling parameter gives a better diversity, we dynamically adjust that parameter as by our controller algorithm. Since it is designed only for DDPM, we do not experiment it with LDMs. Details in the Appendix.
23

---

semee--losr-ddffusee faster with controller (CCDF-C): CCDF use DDPM forward to find starting noise at En. and then perform reverse sampling based on that noise [ We adjust to based on our controller algorithm.
Linear Interpolation with controller (LPP-C): Replacing CCS spherical interpolation with
linear interpolation
Evaluation Metrics
first compute pixel-wise metrics to validate our hypothesis that sample mean is close to the target mean.
PSNR (Peak Signal-to-Noise Ratio): quantifies the pixel-wise difference between the target mean and the sample mean.
SD: the average of standard deviations of pixel intensities for each sampled image, which used to measure the diversity of images.
Then we compute perceptual and reference-free metrics measure the sample quality:
measures the perceptual image quality, which focuses on low-level perceptual quality and is sensitive to blurs/noise/other distortions
MUSIQ |
-CLIP-QQA [40]:
measures the semantic image quality, which is more higher-level than
MUSIQ
Inception Score (IS) 32
is used the CIFAR- 10 dataset to further measure image quality and diversity. Since CIFAR-10 has low resolution and images blurry, we report IS score instead of MUSIQ and CLIP-IQA for CIFAR-10.
Additionally, compute LPIPS between sampled image and target mean, this reflects how the samples are preserving source information even though the MSE those samples are controlled for a fair comparison.
Results
We observe that method achieves preserving source information while generate superior quality images with sufficient diversity. The Table below shows our superior performance this direction:
Table 10: Performance with Stable Diffusion with MSE level 0.07
Methods
PSNR f
LPIPS
GP-C
23.02
0.306
LDPS-C
24.56
0.351
CCDF-C
27.66
0.318
LP-C
29.59
0.322
CCS (Ours)-C
30.29
0.252
Table 11: Performance with Pixel Diffusion on FFHQ with MSE level 0.12
Methods
PSNR t
LPIPS
GP-C
18.88
0.596
ILVR-C
20.04
0.443
DPS-C
21.02
0.459
CCDF-C
23.52
0.461
LP-C
23.41
0,489
CCS (Ours)-C
25.13
0.332
24

---

Table 12: Performance with Pixel Diffusion on CIFAR-10 with MSE level 0.1
Methods
PSNR t
LPIPS
GP-C
24.66
0.409
DPS-C
23.13
0.567
CCDE-C
24.63
0.529
CCS (Ours)-C
26.05
0.328
D.2 Sampling Efficiency
We provide sampling speed and sampling NFE results in the table below. The sampling procedure consists of steps:
Controller tuning for statistical constraint. This only needs to be performed once for each album. For method and GP, we have an additional one-time single-image inversion step (which turns out to be very fast).
2. Sampling with the tuned parameters. Thanks to the linearity property, the binary search algorithm can efficiently find the feasible scale perturbation, achieving the best controller tuning efficiency. Otherwise, the feasible scales may lie in narrower region due to abrupt output changes or variability among samples, requiring more search rounds. We will include detailed discussion of sampling efficiency in revision.
We provide inference time for sampling 120 images around a target mean (with a tuning batch of size 20) for different methods below, using Stable Diffusion tested on one A40 GPU. The table reports the controller uning NFE per batch, and sampling NFE per batch for each baseline.
Method
Sample time image
Controller Tuning NFE
Sampling NFE
GP-C
1.66s
CCDF-C
1.73s
163
42
LDPS-C
5,84s
234
50
CCS-C (Ours)
1.65s
94
45
Table 13: Sampling efficiency comparison on Stable Diffusion tested on one A40 GPU.
The advantage of CCDF-C (or in other words, SDEdit-C) that it does not require DDIM inversion, and requires fewer timesteps for denoising. However, it needs more controller tuning rounds since the outputs can be highly sensitive to some timesteps.
D.3 More Results on Adjusting rMSE Control Levels
We perform additional experiments with sampling quality benchmarks using rMSE targets from [0.05, 0.06, 0.07, 0.08, 0.09, 0.10] for Stable Diffusion on CelebA-HQ. We observe that our method consistently performs quite well (PSNR is for sample mean vs. target image, SD is for diversity). Numbers are reported the order CCS-C / CCDF-C (the stronger/best baseline).
Target rMSE
PSNR 1
MUSIQ 1
CLIP-IQA
SD
0.05
32.22/30.86
49.60/49.53
0.729/0.730
0.036/0.034
0.06
31.44/29.03
49.58/49.02
0.731/0.731
0,043/0.040
0.07
30.29/27.66
49.66/48.91
0.732/0.735
0.053/0.051
0.08
30.10/26.80
49.85/48.23
0.742/0.729
0.056/0.052
0.09
29.74/25.98
49.82/48.01
0.740/0.732
0.061/0.054
0.10
29.31/25.20
49.80 49.80/46.74
0.731/0.727
0.063/0.057
Table 14: Sampling quality comparison for different rMSE targets on CelebA-HQ of CCS-C v.s.
CCDF-C: CCS on the left, CCDF on the right
25

---

We also perform experiments on the FFHQ dataset for rMSE levels of [0.09, 0.12, 0.15), summarized below:
Target rMSE
PSNR 1
MUSIQ 1
CLIP-IQA
SD 1
0.09
28.25/27.08 .08
66.53/66.10
0.749/0.750
0.078/0.069
0.12
25.13/23.52
66.79/66.15
0.750/0.746
0.104/0.088
0.15
23.45/20.64
66.71/65.24
0.743/0.740
0,131/0. 104
Table 15: Sampling quality results FFHQ for different rMSE levels FOR CCS-C v.s. CCDF-C:
CCS on the left, CCDF on the right
Observations
We observe several interesting phenomena in these additional experiments:
1.
With more perturbation of noise (i.e., sample images farther from the target image), there is no decrease in sampling quality, Instead, there is slight increase in MUSIQ for stable diffusion, meaning the image quality increases and looks sharper. One explanation is that the target image may not have good image quality or that the DDIM inversion slightly apart from the Gaussian sphere. As the perturbation level increases, more Gaussian noise is interpolated, making the interpolated noise more "Gaussian" and improving image quality. Since Stable Diffusion is conditional and the input image is not from its training distribution, the DDIM inversion lies slightly off the Gaussian sphere. For FFHQ, however, the inversion significant change in sample quality.
is quite Gaussian, so there
2.
The sampled mean remains close to the target image as the rMSE level increases. We do not observe sudden drop in controllability or diversity (PSNR and SD). However, with increasing rMSE targets, it becomes harder to control the sample mean close to the target mean, demonstrated by declining PSNR. Nevertheless, this trade-off brings diversity improvement.
D.4
More Details improving sample quality.
Based on the observation that interpolates with Gaussian make non-Gaussian random variable more Gaussian, previous work points out that this gives a higher likelihood [30]. Hence, we propose to apply P-CCS on every step of reverse sampling with very small interpolation factor. The algorithm
is stated at Alg. 5 We observe significant gain in sample quality when testing on T21Bench [2] Table. T6 shows the quantitative performance.
Table 16: Image Quality Scores for with and without P-CCS purified on T21Bench
Metric
With
Without
MUSIQ Score
55.499 0.541
52.951
CLIP-IQA Score
0.530
D.5
More Algorithms
We describe more details for the proposed P-CCS (Partial inversion CCS sampling) algorithms for different applications including constrained sampling, precise image editing and improving sampling quality.
Alg.
demonstrates using P-CCS for constrained sampling based on Stable Diffusion.
Alg.
demonstrates using P-CCS for precise image editing based on Stable Diffusion.
Alg.
demonstrates using P-CCS for improving the sample quality, instead of controllability
26

---

Algorithm 3 P-CCS for Constrained Sampling
Requires: target mean zo, perturbation scale Co. inversion time steps to, Encoder & and Decoder D,
a prompt c.
Step 0: Compute Zo
(xv), then compute the DDIM inversion of Zo. i... Zy
DDIM-'(zo 0, to
Step 1: Compute the noise from Zio by ét. Zio Votto 2o 0²).. Then compute
Step Sample noise N(0, 1
( Ctarget [1el22||
cos
Step Compute c/ using spherical interpolation formula:
sin(Co)
sin(O Co) s0)
1 sin(O)
eto
Step 4: Compute z/, 00oo 20 4
Step 5: Output sample x ((2)) D(DDIM (2/2 l0 o,))
Algorithm 4 P-CCS for Precise Image Editing
Requires: target mean zo, perturbation scale Co. inversion time steps to. Encoder & and Decoder D, source prompt target prompt Ct.
Step 0: Compute 2g (xr), then compute the DDIM inversion of za with the source prompt,
i.c. Zio.a DDIM (zo, 0, to c.), and the DDIM inversion with the target prompt, i.e. Zoo.t
DDIM (Zo, to. ct)
Step 1:
Compute the noise from Zt..s by Eto. Ztg,s Vatta
Step Compute the noise from Zio,ts by Eto.! Zta,t /ato
Zo
Step 3:
Compute
cos
Cen.N Geast (letosal2|leto.t
Step
Compute </ using spherical interpolation formula:
(Cn)
sin(O Cb)
sin(0)
Ctg.
sin(f)
Eta.c
Step Compute Z ato Zn é, Step 5: Output sample x/= ((2)) D(DDIM(Z/, lo,00 o+)), which is the precisely edited image with strength given by Co-
D.6 More Figures
Fig. demonstrates the generated personalized album for CCS. Fig. 7 demonstrates example of applying P-CCS with SDI.5 on the Celeba-HQ dataset, we demonstrate that our algorithm can work well on in-the-wild images which are very different from the training.
Fig.
10
demonstrates the linear trend for each target mean on the FFHQ dataset.
Fig.
X
demonstrates example of image editing controlled sampling with Alg.
Fig. demonstrates the linearity drop with OOD data.
27

---

Algorithm 5 P-CCS for Improving Sampling Quality
Requires: target mean zo. perturbation scale Co, inversion time steps to. Encoder & and Decoder D,
condition
Step Start by computing zo (xo), then compute the DDIM inversion of zo with the condition, Le. Zt, DDIM (Zo, O, c).
While lo > 0: Step I: Compute the noise from Ztg. by Zto virto 20(2T) using Tweedie's
formula to compute (2r).
Step 2: Sample noise NO0, a4)).
Step 3: Compute
Coo c |1ecollel|e11
i cos
Step 3:
Compute <, using spherical interpolation formula:
sin(Co) sin(O)
Ec. + sin - Co)
sin
Step 4:
Van," Zo cy Step 5: Reverse Sampling using DDIM formula, and the modified Z1, to = to -1
Compute z
End While
Step 6:
Output sample x D(z)) = D(DDIM(ZI, la, O, ct)), which is the purified (improved)
image with purification strength given by Co-
Target Image
Sample Mean
Figure 6: demo of sampled album with CCS algorithm on FFHQ dataset. Note that the sample mean is almost the same as the input image.
28

---

Sample
(c) Target Moan
Generated Album
Gererated Album
Sample
Figure 7: CCS-CT Sampled Images with Stable Diffusion 1.5. (a) Samples with an in-the-wild target mean (e) and target rMSE 0.09; (b) Samples with target mean (d) from Celeba-HQ HQ dataset and target rMSE 0.07; (d): sample mean from (a); (f): sample mean from (b).
Source Sample Mean
Figure 8: Image Editing Samples with Stable Diffusion 1.5, the source prompt is given by 'a high- quality portrait of man', and the target prompt is given by "a high-quality portrait of a woman', the target MSE level is given by 0.10
Source Image
Sample Mean
Figure 9: Image Editing Samples with Stable Diffusion 1.5, the source prompt is given by 'a high- quality portrait of an old man', and the target prompt given by 'a high-quality portrait of a young man', the target MSE level is given by 0.09
29

---

target mean
target mean
120
target mean
average L-2 distance to the target mean
target mean
100
target mean
target mean
target mean
target mean
target mean 9
target mean
20
0.1
0.5
sin(Ca)
Figure 10: We first sample 50 images around each target mean the FFHQ dataset. We then obtain the DDIM inverse of each target mean, and then add spherical perturbation to it. When the scale of perturbations sin(Co) increases, the average norms of the residuals between each sample and target mean approximately increases linearly.
Figure We demonstrate that there sudden change in output when testing on OOD input perturbed with increasing Gaussian noise. The backbone model is a pixel diffusion model (DDPM++), which is only trained on FFHQ. The top row uses an OOD input from ImageNet, and the bottom row uses an image from FFHQ validation set input. Co from left to right: 0.0, 0.2, 0.4, 0.6, 0.8
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contribution and scope are stated in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification:
We have provided a discussion about limitations the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:[Yes)
31

---

Justification:
We have clearly stated our assumptions, propositions and results in Sec. and 4 with proof in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We have fully explained the setup of our experiments in Sec.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

reproducible, they Answer: [Yes]
Justification: This paper utilizes publicly available datasets, with detailed access instructions and appropriate citations provided. While the code is not publicly released at the time of submission, we intend to make available upon acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co cc/
public/guides /tt////SSuumisssiooooooo.oo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/publ htappss.ccpublic/uuudeeee sot.:d///o/oooeeeedisiiisissssee
ssiooiooisiiioiociicc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: Yes]
Justification: is clearly stated in the experiment section (Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have reported multiple metrics in our experiments with average values in main paper and the standard deviations in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
33

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information about computer resoures Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research complies fully with the NeurIPS Code Ethics. does not involve sensitive data, human subjects, or potentially harmful applications.
The research complies fully with the NeurIPS Code of Ethics. It does not involve sensitive data, human subjects, or potentially harmful applications.
Guidelines:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: This paper presents work whose goal is to advance the field of generative AL. There many potential societal consequences of work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
34

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not introduce release any new models or datasets that could pose risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: We have properly cited and credited the pretrained diffusion models (Stable Diffusion 1.5), and datasets used in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA].
Justification: At the time of submission, we have not released any new assets, but after acceptance, we will release new assets including code and trained models along with detailed documentation for them.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA].
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
36

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA].
Justification: The core methods and contributions of the paper do not involve the use of LLM in any important, original, non-standard way.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) hatt:p//eearrpppcc///ooeeeeeeee/ cc///oofeeeeeneee/222////.. for what should or should not be described.
37


Reviews:
[{"review_id": "JShkAxnhBH", "reviewer": "Reviewer_ZiPM", "summary": "This paper proposes CCS, a training-free method to guide diffusion model outputs toward target statistical properties by operating on the initial noise. The authors identify a linear relationship between initial noise perturbations and final outputs under DDIM sampling. The paper offers theoretical analysis, empirical validation, and applications in controllable sampling and precise image editing.", "questions": "1. How does the linearity relationship hold in complex or highly multimodal distributions?\n2. Can CCS be combined with per-sample controls like classifier guidance or universal guidance?\n3. How sensitive is CCS to the choice of timestep in full vs. partial inversion?", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. The linearity identified in DDIM sampling between the initial noise and the output is interesting.\n2. The authors have provided a rigorous analysis using ODE theory and validated it with extensive experiments.\n3. I appreciate the writing of this work with strong algorithmic and experimental clarity.\n\nWeakness:\n\n1. The generalization of linearity across diverse distributions (e.g., highly multimodal datasets) is not thoroughly examined. I wonder this part.", "was_revised": true, "final_justification": "I thank the authors for the detailed and thoughtful rebuttal and for revising the manuscript. I appreciate the additional experiments on multimodal and out-of-distribution data, as well as the clarifications on combining CCS with classifier guidance and the effects of inversion timestep. I also noticed other concerns raised by fellow reviewers have also been nicely addressed. Thus I lean towards acceptance for this work.", "rebuttal": "We thank the reviewer for the careful and insightful review!\n\n***1. Q: How does the linear relationship hold in complex or highly multimodal distributions?***\n\n> We thank the reviewer for this inquiry, which is also a question that we are very interested in. We include more results in this rebuttal and will include these new results in our revision. By our analysis of the linear approximation error (in Appendix B.3), we bound the error by the reciprocal of the lowest probability density in a local region, and we also bound the linear approximation error by the (higher-order) gradients of the probability distribution of $p_t(x_t)$. We conjecture that for multimodal distributions the linearity will be weaker at the low-density regions (regions between each modalities). \n\n> To further investigate this relationship, we perform additional experiments on three more datasets to test how the linearity change for multimodal distributions. We are interested in \n> 1. How is the linearity when the model is trained on multimodal data and sample with out-of-distribution target image from another highly multimodal dataset? or an OOD simple dataset?\n> 2. How is the linearity when the model is trained on a simple dataset (for example face dataset like FFHQ), and sample on out-of-distribution multimodal dataset? on OOD simple dataset?\n>3. How is the linearity when training on simple dataset and sample on the same dataset? \n>4. How is the linearity comparing model trained on multimodal dataset testing in-distribution and model trained on simple dataset testing in-distribution?\n\n>In our experiment section of our paper, we cover partially 1,3, and 4: we test pixel diffusion models on FFHQ dataset, and the CIFAR dataset. We also test the Stable Diffusion model (1.5) [1]  which is trained on a complex dataset LAION-5B, on human face dataset (CelebA-HQ). We observe that diffusion models trained on simple dataset (FFHQ) and testing in-distribution exhibits significant stronger linearity than the Stable Diffusion model on the CelebA-HQ dataset.\n\n>We observed in our paper that linearity decreases slightly when comparing diffusion model with multimodal training data to diffusion model with simple training data. CIFAR-10 is a dataset with different classes, we have slightly lower linearity score for CIFAR-10 compared to FFHQ even though CIFAR-10 is of lower resolution. Empirically, we observe sudden change of image semantics occasionally. \n\n> In this rebuttal, we provide additional results for 1,2, and 4. We include three more datasets:\n> 1. We pick 4 images from the class 0, 4, ...,99 from the validation set of ImageNet [2] \n> 2. We pick five videos from the classes: \"Applying Eye Makeup\", \"Baby Crawling\", \"Billiard\", and \"Blow Dry Hair\", and sample five frames from each video in the UCF-101 dataset [3]\n> 3. We pick 10 images from each organ site in the AAPM dataset as in [4], which is CT scans of different body parts.\nThese datasets are different from the training data of SD1.5. We use the same linearity testing methods as in Section 5.1 in our main paper (also in Appendix B. We first summarize the results below for Stable Diffusion 1.5:\n|Dataset | ImageNet | UCF-101| ```CelebA-HQ``` | ```fMoW``` |\n|-----------|------| --------| --------| ------------|\n| R-square of output change v.s. perturbation scale | 0.960 | 0.962 | 0.959 | 0.947|\n| cosine similarity of linear combinations | 0.922 | 0.924 | 0.901 | 0.920 | \n\n> The results show that for multimodal dataset (containing many classes like ImageNet or UCF-101), there is no evidence of decrease in linearity compared to simple datasets (CelebA-HQ) for the same backbone model trained on a large multimodal dataset (LAION-5B). The less linearity on CelebA-HQ may be due to our data processing techniques (upscale 256 to 512 introduces bluriness and let initial noise apart from the Gaussian sphere). \n\n> When testing the linearity on OOD dataset for model trained on simple dataset, we observe significant linearity drop.  We  compare it model trained on multimodal data and tested on simple dataset. Results below use the same testing method and dataset as mentioned before and in the paper Appendix B. Note that the trained->tested here mean that the model is trained on the training set of \"trained\" and tested on the validation set of \"tested\". \n|Dataset (trained->tested) | ImageNet->FFHQ | FFHQ->ImageNet | FFHQ->FFHQ | ImageNet->ImageNet | FFHQ->AAPM|\n|-----------|------| --------| --------| ------------|-----------|\n| R-square of output change v.s. perturbation scale | 0.934 | 0.938 | 0.995 | 0.983 | 0.951 |\n| cosine similarity of linear combinations | 0.902 | 0.905 | 0.958 | 0.941 | 0.912 |\n\n> For fair evaluation we use the same model architecture DDPM++ with the same training loss for these two pixel-space diffusion models. We find that for not foundation models, the OOD linearity drops significantly. The multimodal backbone has a slightly lower linearity score than the simple data backbone due to the complexity of its training data (it learns abrupt changes). The drop in OOD linearity is for both multimodal data and simple data, suggesting that the probability density of input image plays an important role in linearity, in agreement with our theoretical analysis in Appendix B.3. \n\n> Overall, we observe that this linearity exists for different settings, and changes based on the probability density of input data and the curvature/smoothness of the training data distribution. OOD data tends to have lower linearity score. backbones trained on multimodal data may have lower linearity score. In addition, well-trained diffusion models trained on the same dataset should have similar linearity score by the reproducibility of diffusion model in [5] (they learn the same probability density which determines linearity). \n\n\n\n***2. Q: Can CCS be combined with per-sample controls like classifier guidance or universal guidance?***\n\n> Yes, it can be combined with per-sample control method such as CFG (other guidance method should also be fine). We just need to replace the score with the CFG score to perform CCS sampling. Indeed combining with CFG yields very interesting results. We will add new results to our revision.\nWe perform additional experiments testing the constrained sampling with the target rMSE of 0.07 on the CelebA-HQ dataset with the SD1.5 checkpoint with different CFG levels [1.5, 2.5, 3.5, 4.5] (we perform inversion using this specific CFG value and then perform sampling with the same CFG value.) The starting t is set to be 45 which is consistent with our main experiments. The results are summarized in the table below (MUSIQ is for image visual quality, CLIP-IQA for semantic quality, SD for diversity and PSNR for controllability):\n | CFG | PSNR↑  | MUSIQ↑ | CLIP-IQA↑ | SD↑ |\n| ---------| ---- | ----- | -------| -----|\n|1.5 | ***30.19***  | 49.70| 0.733 | 0.056 |\n| 2.5 | 29.98 | 51.28 | ***0.737*** | ***0.057***|\n| 3.5 | 29.15 | 51.29| 0.730| 0.052|\n| 4.5 | 27.28 | ***52.85*** | 0.729| 0.053|\nWe observe that \n>1. When CFG gets higher, the controllability decreases, but images get sharper with better visual quality\n>2. There seems to be a sweet spot for the diversity. While high CFG hurts sampling diversity, medium-level may be not bad.\n\nWe also want to note that other baselines can fail at high CFG. CCDF-C (Our strongest baseline) uses SDEdit-type noise adding, so it cannot preserve source image information at a large rMSE target (it always generate CFG-guided image). At CFG=4.5, we have PSNR at 25.73 and SD of 0.079 at rMSE target 0.1 for our method, but CCDF-C drops to 20.19 and SD of 0.032, which signals significant loss of controllability and diversity.\n\n\n***3. Q: How sensitive is CCS to the choice of timestep in full v.s. partial inversion?***\n\nWe perform additional experiments to test the different inversion timesteps on the Stable Diffusion 1.5 for timestep [37, 40, 42, 45, 48] on the CelebA-HQ dataset. Results show that small timestep leads to better controllability, but worse diversity and slightly worse image quality.\n\n | Inversion Timestep | PSNR↑  | MUSIQ↑ | CLIP-IQA↑ | SD↑ |\n| ---------| ---- | ----- | -------| -----|\n|37 | 31.23 | 49.19 | 0.729 | 0.041 |\n| 40 | ***31.28*** | 49.31 | 0.732 | 0.047|\n| 42 | 30.78 | 49.53 | 0.730| 0.049 |\n| 45 | 30.29 | 49.66 | ***0.732*** | ***0.053***|\n| 48 | 30.15 | ***49.87*** | 0.731| 0.052|\n\nWe conjecture that editing image is harder with smaller inversion timesteps, so there is a diversity drop but with a better controllability. Also, more denoising steps can improve the sample quality. We will add these new results to our revision.\n\n\n[1] High-resolution image synthesis with latent diffusion models. CVPR 2023\n\n[2] Imagenet: A large-scale hierarchical image database. CVPR 2009\n\n[3] Ucf101: A dataset of 101 human actions classes from videos in the wild.\n\n[4] Piner: Prior-informed implicit neural representation learning for test-time adaptation in sparse-view ct reconstruction. WACV 2023\n\n[5] The emergence of reproducibility and consistency in diffusion models. ICML 2025"}, {"review_id": "orF8QXCH7x", "reviewer": "Reviewer_6ZCA", "summary": "Despite the current success of the Diffusion model, the connection between the initial noise perturbation $x_T$ and the resulting generated data $\\bar{x}_0$ remains underexplored, limiting our understanding of how to control the sampling process. In this paper, the authors reveal that the change in generated outputs scales linearly with the magnitude of the initial noise perturbation. Building on this insight, the Controllable and Constrained Sampling (CCS) method is proposed to enable sampling with desired statistical properties while maintaining high sample quality.", "questions": "1. What is the relationship between the linear approximation of the score function and linearity in the output of Proposition 1? Why is the linear approximation error in output low when the score function is smooth (Lines 147-148)? Furthermore, should it be $o(||\\lambda w||_2^2)$ instead  $o(\\lambda)$? The proof A.1 needs to be carefully revised to enhance understanding, e.g., why not use $\\lambda$ in Eq. 5 instead $\\delta$ from scratch, the Taylor expansion should be revised too. Is the matrix-value function $\\gamma_0$ bounded or not?\n\n2. There are missing references in the appendix, e.g., [18] in the proofs of Proposition 2.  There might be an error in derivative of $\\sigma(t)$ in the proof of Proposition 2 in Lines 820-821.\n\n3. While the propositions are proved with $a=1$ and $b=\\lambda$, do they still hold with $a \\neq 1$?\n\n4. In section 4.2, why can this approach not produce high-quality images when $x’_T$ is pushed farther from the spherical surface?\n\n5. The paper only reports 1 selected MSE target value for each dataset. What happens when changing it? In addition to this, as the authors self-implement other baselines (Lines 315-317), are the corresponding hyperparameters of these baselines tuned?\n\n6. Other minor points:\n   - Typos: Line 79: $T$ -> $t$; Line 122: $x_t$ -> $x_T$; Line 160: $\\alpha_0$ -> $\\sqrt{\\alpha_0}$;\n   - repeated citation [2], [3]\n   - Eq 4 needs further detail about the formula of $\\bar{x}_t$ and $\\sigma(t)$ \n   - Inconsistent format of some equations, e.g. $x$ in Line 134 versus others \n   - Error in Proposition 1  e.g. $x_0(x_T)$ -> $x_0(x_T, T)$, I suggest the author should denote the $x_0(x_T, T)$ by another notation like denoising operator $D :x_T$ -> $x_0$\n    - Algorithms 1 and 2 should be put side-by-side for improving representation.", "limitations": "Please see the above questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper uncovers a novel linear relationship between the initial noise and the generated samples under DDIM sampling, offering new insights into the diffusion sampling process.\n\n- It introduces a new task of controllable generation, which aims to match the sample mean to a target mean while constraining the sample MSE to a desired level. To address this task, the authors propose an original controllable sampling method.\n\nWeaknesses\n\n- The writing quality requires improvement to enhance clarity and readability throughout the paper.\n\n- Several key aspects of the methodology and presentation are confusing, as elaborated in the Questions section below.", "was_revised": true, "final_justification": "Before the rebuttal, my main concerns centered around ambiguous arguments (as noted in my questions 1 and 4), issues with the empirical results (question 5), and an incorrect proof (question 2), along with several minor typographical errors. During the rebuttal, the authors made a commendable effort to address these concerns and proposed concrete revisions to improve the quality of the paper. Therefore, I have decided to increase the score from 2 to 4, on the condition that the authors implement the suggested revisions in the final version.", "rebuttal": "We thank the reviewer for the careful and detailed review!\n\nHere we provide the answers to the questions:\n1. ***Q: What is the relationship between the linear approximation of the score function and linearity in the output of Proposition 1***:\n> **A**: The way we prove Proposition $1$ is by recursively using the linear approximation of the score function. In more details,\nsince each DDIM sampling step takes a linear combination of the predicted score and the intermediate noise, we prove that with the assumption of linear approximation of score function, we can derive the linearity behavior of DDIM sampling output and input. Formally as in the proof of Proposition 1 (in Appendix Line 805), let $L_T$ be a one step DDIM sampling step on x, and $\\eta_T$, $\\lambda_T$ be DDIM sampling constants, $w$ be a unit direction vector, and $\\delta$ be the perturbation scale, we have $L_T(x + \\delta w) = \\eta_T (x + \\delta w) + \\lambda_T \\nabla_{x}\\log p_{T}(x + \\delta w) = \\eta_T x + \\lambda_T \\nabla_{x}\\log p_{T}(x) + \\lambda_T \\delta H_T(x)w + \\delta\\eta_T w+ o(\\delta)$. This demonstrates that for one-step DDIM sampling, the linearity between the output and the input is proportional to the linear approximation error of the score function. Then we can further extend this to the entire DDIM sampling process as demonstrated in line 808-813 in the Appendix. We will revise our paper to clarify this relationship and describe this reasoning process clearly. \n\n2. ***Q: Why is the linear approximation error in output low when the score function is smooth?***\n> **A:** When a function $f(x)$ is smoother, it has lower magnitude for the higher-order derivatives (second-order or higher). Intuitively, it has fewer abrupt changes. So when zooming in on the plot of $f(x)$ versus $x$, a smoother function appears more like a straight line locally. \n\n>  In the Appendix B.3. (Line 896-919), we have provided a theoretical analysis of this linear approximation error, and derived the bound for this error. Based on reviewer’s comment to improve the clarity, we will revise our paper to include the Taylor remainder theorem in the main text, and provide a more thorough reasoning and derivation as below to support the previous claim: less smoothness comes with larger magnitude of higher-order (>2) gradients, which introduces more local nonlinearity.\n\n> Formally, we can use the Taylor remainder theorem here:\n> \n> $$R_n(x) = \\frac{f^{(n+1)}(c)}{(n+1)!} (x - a)^{n+1}$$\n> This theorem demonstrates that the linear approximation error is bounded by the magnitude of its second-order derivative at the point of approximation, which explains why a smoother function has less linear approximation error.\n> \n> Take three functions as examples:\n> \n> 1. $f(x) = |x|$: This function is not differentiable and not smooth at $x = 0$ because it has a sharp corner. Using sub-gradient for linear approximation leads to large error.\n> \n> 2. $f(x) = e^x$: This function is smooth everywhere, with all derivatives equal to $e^x$. The linear approximation error at $x = 0$ is very small and the linear approximated value is very close to the true function value because the higher-order derivatives are well-behaved and continuous.\n> \n> 3. $f(x) = \\sin(\\frac{1}{x^2 + 0.0001})$: Even though this function is differentiable, it has rapid oscillation at x = 0 (much less smoother than $e^x$), Its linear approximation at $x = 0$ is very inaccurate and very different from the true value of the function. \n\n3. ***Q: Is $\\gamma_0$ bounded?***:\n>If we assume the Hessian of $\\log p_t(x_t)$ is bounded (it changes less abruptly), then it is bounded. \n\n4. ***Q: Should it be $o(||\\lambda w||^2_2)$ instead of $o(\\lambda)$? why not use $\\lambda$ in Eq. 5 instead $\\delta$ from scratch?***\n> There seems to be a confusion on the notation. First, from line 805 to line 808, we refer to an arbitrary fixed-direction perturbation with a unit length $w$, so we use the notation $\\delta$ as the scale of that direction. In Eq.5, we refer to a specific unit-length perturbation to the initial noise in DDIM sampling, so we use a different syntax. We realize that this notation system can be confusing, and we will add more context in our revision to explain these notations clearly in the Appendix. \n\n> To our best knowledge, the error term in Proposition 1 should be $o(\\lambda)$. The reason we exclude $w$ in the error term is that we have $||o(\\lambda) w || \\leq o(\\lambda) ||w || \\leq o(\\lambda)$ since we denote $w$ as a unit direction vector as in line 805. In line 810, we get that $x_0(x_T + \\lambda \\Delta x, T) = x_0(x_T) + \\lambda \\gamma_0(x_T) \\Delta x + o(\\lambda)$, by rearranging the terms, we now have $x_0(x_T + \\lambda \\Delta x, T) - x_0(x_T) = \\lambda \\gamma_0(x_T) \\Delta x + o(\\lambda)$. Now let's apply the triangle inequality of the norm twice. First:\n> 1. we notice that $||x_0(x_T + \\lambda \\Delta x, T) - x_0(x_T)  ||_2= ||\\lambda \\gamma_0(x_T) \\Delta x + o(\\lambda)||_2 \\leq ||\\lambda \\gamma_0(x_T) \\Delta x|| + ||o(\\lambda)||_2$\n> 2. we also notice that  $||\\lambda \\gamma_0(x_T) \\Delta x||_2 \\leq ||x_0(x_T + \\lambda \\Delta x, T) - x_0(x_T) ||_2 + ||o(\\lambda)||_2$\n>Combine these two inequalities, we have $||\\lambda \\gamma_0(x_T) \\Delta x||_2 - ||o(\\lambda)||_2 \\leq ||x_0(x_T + \\lambda \\Delta x, T) - x_0(x_T) ||_2 \\leq ||\\lambda \\gamma_0(x_T) \\Delta x||_2 + ||o(\\lambda)||_2$, so we can conclude that $||x_0(x_T + \\lambda \\Delta x, T) - x_0(x_T) ||_2 = \\lambda|| \\gamma_0(x_T) \\Delta x||_2 + o(\\lambda)$. \n\n> Indeed, $o(||\\lambda w||^2_2)$ is tighter than our current error bound. We do not achieve this tight bound in our paper, and will investigate this in future work. We will clarify the notation in our revision.\n\n5. ***Q: There might be an error in derivative of $\\sigma_t$***\n> Thanks!  There is a typo in line 820. There should be an additional $\\frac{1}{\\alpha(t)^2}$ term in the derivative of $\\sigma(t)$. We will revise the $h(t,y)$ in line 815, and line 820. Nevertheless, adding this term does not affect the assumption of continuous derivative since $\\alpha(t)$ is smooth function that is larger than zero between [0,T], so that other places in the proof and the conclusion is not affected. We will revise the proof in our revision.\n\n6. ***Q: Does the Propositions still hold for $a \\neq 1$?***\n> Only Proposition 5 has the setting of $a = 1$, and $b = \\lambda$, while Proposition 6 has a different spherical setting. Proposition 5 just explains why using $a = 1$, and $b = \\lambda$ not working. We can also demonstrate using simple linear interpolation ($a \\neq 1$) not working for sampling as well:\nFormally, let $||\\Delta x||_2 = ||x||_2$, and its direction is uniformly distributed, and $\\mathbb{E}[\\Delta x] = 0$, for $0 < \\alpha < 1$, we have \n> $$\\mathbb{E} [ || \\alpha x + (1-\\alpha) \\Delta x ||_2^2] < || x ||_2^2$$\n\n> Proof: we have  $||\\alpha x + (1-\\alpha) \\Delta x ||_2^2 = \\alpha^2 || x ||_2^2 + 2\\alpha(1-\\alpha)(x \\cdot \\Delta x) + (1-\\alpha)^2 || \\Delta x ||_2^2$, since we have $\\mathbb{E}[\\alpha(1-\\alpha)(x \\cdot \\Delta x)] = \\alpha(1-\\alpha) x \\cdot \\mathbb{E}[\\Delta x] = 0$, we have $\\mathbb{E} [ || \\alpha x + (1-\\alpha) \\Delta x ||_2^2] = \\alpha^2 || x ||_2^2 +  (1-\\alpha)^2 || \\Delta x ||_2^2 = (\\alpha^2 + (1-\\alpha)^2) || x ||_2^2$, now since $2\\alpha (\\alpha - 1) < 0$, we have $(\\alpha^2 + (1-\\alpha)^2) < 1$, and we can reach the conclusion that using simple linear interpolation cannot preserve the norm leading to falling apart from the Guassian sphere.\n\n> We will include this analysis as well as experimental results on simple linear interpolation (also mentioned by R3) in our revision.\n\n7. ***Q: Why can this approach not produce high-quality images when is pushed farther from the spherical surface?***\n> Intuitively, if initial noises not on spherical surface, this means that the noise level is either too low or too high by Proposition 4 in our paper, which both leads the trained denoiser of the diffusion model not effectively removing the noise effectively (noisy image) or removing too much noise (blurry image). Only if we ensure the initial noise level matches the pre-defined noise level at T, the diffusion model can sample as it is designed to. This phenomenon is also observed in [1], and a qualitative example is demonstrated in Fig.3 in that paper. \n\n> We will revise our paper to include an explanation and qualitative examples on why sampling quality degrades with initial noise not on the spherical surface.\n\n8. ***Q: What happens with different MSE targets? How are baselines tuned?***\n> Thanks for the suggestion! The target in this paper is set for best visualization quality (both preserving source image feature and having variations). We perform additional experiments with sampling quality benchmark with rMSE targets from [0.05, 0.06, 0.07,0.08,0.09, 0.10] for Stable Diffusion on CelebA-HQ. We observe that our method consistently performs quite well (PSNR is for sample mean v.s. target image, SD is for diversity). (more numbers in R4's response). \n\n> We tuned hyperparameters based on validation set. More details can be found in Appendix D.1 and R1 response. Indeed most hyperparemeters of our method and baselines are searched by our controller. We will revise our paper to include all baseline details. \n\n(numbers are reported in the order of CCS-C/CCDF-C (the strongest baseline))\n| Target rMSE | PSNR↑  | MUSIQ↑ | CLIP-IQA↑| SD↑ |\n| ---------| ---- | ----- | -------| -----|\n|0.05 | 32.22/30.86   | 49.60/49.53 | 0.729/0.730 | 0.036/0.034|\n| 0.06 | 31.44/29.03| 49.58/49.02 | 0.731/0.731| 0.043/0.040|\n| 0.07 | 30.29/27.66 | 49.66/48.91 | 0.732/0.735| 0.053/0.051|\n| 0.08 | 30.10/26.80| 49.85/48.23 | 0.742/0.729| 0.056/0.052|\n| 0.09 | 29.74/25.98 | 49.82/48.01 | 0.740/0.732| 0.061/0.054|\n| 0.10 | 29.31/25.20 | 49.80/46.74 | 0.731/0.727| 0.063/0.057|\n\n9. ***Q: Minor issues***\n> Thanks! We will revise all points.\n\n[1] NoiseDiffusion: Correcting Noise for Image Interpolation with Diffusion Models. ICLR 2024"}, {"review_id": "rhoUotsqd7", "reviewer": "Reviewer_ivEh", "summary": "This paper investigates the linear relationship between the initial noise and the generated outputs of diffusion models under DDIM sampling. The authors provide both theoretical and empirical evidence to support this linearity property. Based on this insight, they propose a method called CCS, which enables the generation of desired results under specific conditions such as image editing, personalized image generation, and quality improvement.", "questions": "An ablation study comparing the proposed spherical interpolation with simpler alternatives such as linear or additive interpolation could further validate the design choice.\nIt is unclear whether the initial noise obtained through DDIM inversion always conforms to a standard Gaussian distribution. If not, how sensitive is the proposed method to such deviations? An analysis of its robustness in such cases would be helpful.\nI am also curious whether the method could be extended to enable fine-grained image editing—for example, preserving the identity of a character while modifying the surrounding environment. This seems related to selectively controlling noise components: keeping certain directions fixed while perturbing others more strongly.", "limitations": "The authors claim they discuss limitations in the conclusion section, but I could not find it.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n    The observed linearity phenomenon is interesting and novel.\n    The authors provide both theoretical analysis and empirical validation.\n    Extensive experiments are conducted across various applications to demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n\n    The proposed CCS method relies on spherical interpolation. An ablation study comparing it with alternative interpolation strategies (e.g., linear or additive perturbations) would strengthen the justification for this design choice.\n    The assumption that DDIM-inverted noise lies on a Gaussian hypersphere is not rigorously supported. In practice, the inversion result may deviate from a standard Gaussian distribution, potentially affecting the method’s reliability.\n    The paper does not discuss limitations or provide visualizations of failure cases, which would help clarify the method’s robustness and boundary conditions.", "was_revised": true, "final_justification": "The reviewer answered to my questions and I am keeping this positive score.", "rebuttal": "We thanks the reviewer for the careful and insightful review!\n\n***1. Q: An ablation study comparing it with alternative interpolation strategies***\n> A: Thanks, this is also an important question we have. We perform additional experiments that compares the performance linear interpolation for the pixel diffusion experiments on FFHQ and Stable Diffusion experiments on CelebA-HQ following the same setting in the main paper (rMSE target = 0.07 for CelebA-HQ, and 0.12 for FFHQ, both using the same controller).  We will include the new results in our paper revision. \n\n>Results showing that through spherical interpolation we achieve better controllability demonstrated by higher PSNR (between sample mean and target image), better diversity, and better image quality. Visually, using linear interpolation causes very blurry images. \nWe include the results for Stable Diffusion on CelebA-HQ here (PSNR for controllability, SD for diversity, MUSIQ for image visual quality, and CLIP-IQA for image semantic quality):\n| Method | PSNR↑ | MUSIQ↑ | CLIP-IQA↑ | SD↑ |\n| ------| -----| -----| -----| ----|\n| CCS-CT-C | 30.29 | 49.66 | 0.732 | 0.053|\n|Linear interpolation-C | 29.59 | 42.98 | 0.722 | 0.032|\n\n> Also the results on FFHQ:\n| Method | PSNR | MUSIQ | CLIP-IQA | SD |\n| ------| -----| -----| -----| -----|\n| CCS-CT-C | 25.13 | 66.79 | 0.750 | 0.104|\n|Linear interpolation-C | 23.41 | 53.26 | 0.731 | 0.057|\n\n> The result show severe degradation in MUSIQ and diversity, with significant drop in PSNR, suggesting linear interpolation leads to worse quality images with decreased diversity, and less controllability. \n\n\n> To further support this point, we provide additional theoretical analysis as below to justify why simple linear interpolation does not work well for sampling:\nFormally, let $||\\Delta x||_2 = ||x||_2$, and its direction is uniformly distributed, and $\\mathbb{E}[\\Delta x] = 0$, for $0 < \\alpha < 1$, we have $$\\mathbb{E} [ || \\alpha x + (1-\\alpha) \\Delta x ||_2^2] < || x ||_2^2$$\n\n> Proof: we have $||\\alpha x + (1-\\alpha) \\Delta x ||_2^2 = \\alpha^2 || x ||_2^2 + 2\\alpha(1-\\alpha)(x \\cdot \\Delta x) + (1-\\alpha)^2 || \\Delta x ||_2^2$, since we have $\\mathbb{E}[\\alpha(1-\\alpha)(x \\cdot \\Delta x)] = \\alpha(1-\\alpha) x \\cdot \\mathbb{E}[\\Delta x] = 0$, we have $\\mathbb{E} [ || \\alpha x + (1-\\alpha) \\Delta x ||_2^2] = \\alpha^2 || x ||_2^2 + (1-\\alpha)^2 || \\Delta x ||_2^2 = (\\alpha^2 + (1-\\alpha)^2) || x ||_2^2$, now since $2\\alpha (\\alpha - 1) < 0$, we have $(\\alpha^2 + (1-\\alpha)^2) < 1$, and we can reach the conclusion that using simple linear interpolation cannot preserve the norm leading to falling apart from the Guassian sphere. On why not falling apart from Gaussian sphere may lead to degraded image quality. We argue that if initial noises not on spherical surface, this means that the noise level is either too low or too high by Proposition 4 in our paper, which both leads the trained denoiser of the diffusion model not effectively removing the noise effectively (noisy image) or removing too much noise (blurry image). \n\n***2. Q: DDIM inversion not conforming to standard Gaussian distribution. Are there any sensitivity to those deviations***\n> Thanks reviewer for pointing this out. We agree that the DDIM inversion may not follow standard Gaussian distribution. However, through our spherical interpolation method, we are able to make the interpolated noise more \"standard Gaussian\". The risk is that spherical interpolation for a non-standard Gaussian and a standard Gaussian noise may still be quite non-standard Gaussian, and cause the image sample quality degradation. However, we observe that in practice this is rarely the case.\n\n> 1. To investigate whether there is sample quality drop, we use the our CelebA-HQ validation set in our paper for additional experiments. This dataset is originally of size 256x256, we upscale it to 512x512, so it is blurry and sometimes gives inversion not on the sphere. We partition the CelebA-HQ into two sets:  the encoded noise that has a mean deviates from 0 by more than 0.03 or std deviates from 1 by more than 0.03 (not STG noise), and the rest. We compute the performance metric of these images on Stable Diffusion with rMSE target 0.07 and summarized below:\n| Set | PSNR | MUSIQ | CLIP-IQA | SD |\n| ------| -----| -----| -----| -----|\n| not STG noise | 30.86 | 49.43 | 0.734 | 0.053|\n|STG noise | 30.10 | 49.74 | 0.731 | 0.054|\n> We do not find significant image quality difference between these two sets. Indeed after our CCS interpolation at the 0.07 rMSE target, the interpolated noise of not STG noise group all fall within 0.01 difference between zero mean and unit standard deviation. Empirically, at low rMSE levels, the noise may be non-standard Gaussian, but the samples are close to input image. At higher rMSE levels, the interpolated noise are more standard Gaussian resulting in good image quality. \n\n> To verify that more standard Gaussian distribution with more interpolation strength. We compute the average deviation of mean to 0, and deviation of variance to 1, for the CelebA-HQ experiment. We find as the interpolation strength $C_0$ increase, the deviation quickly narrows:\n| $C_0$ | deviation in mean | deviation in std |\n| ------| -----| -----| \n| 0.0 | 0.025 | 0.023 |\n|0.2  | 0.013 | 0.016 | \n|0.3  | 0.011 | 0.009 | \n|0.4  | 0.010 | 0.008 | \n|0.5  | 0.009 | 0.006 | \n|0.5  | 0.007 | 0.005 | \n\n> In our paper, our $C_0$ is mostly between 0.3 to 0.6, so we do not worry about the interpolated noise not standard Gaussian a lot. Also, we conduct experiments on different rMSE target level (with different $C_0$), and observe that increasing interpolation strength may lead to slightly better image quality if input image is not that good (the detailed experimental results can be found in R4's response). We will include the additional results in our revision.\n\n***3. Q: More discussions on the inversion of DDIM not on Gaussian sphere***\n> 1. We will add a new proposition in our revision about more interpolation leads the second moment closer to a standard Gaussian. Formally: Fix any vector $\\mathbf{x} \\in \\mathbb{R}^d$, and let $\\boldsymbol{\\epsilon} \\sim \\mathcal{N}(0, I_d)$. Let $\\theta \\in [0, \\pi]$ be the angle between $\\mathbf{x}$ and $\\boldsymbol{\\epsilon}$. We define the interpolated vector:\n\n> $\\mathbf{y} = \\frac{\\sin(c\\theta)}{\\sin(\\theta)} \\mathbf{\\epsilon} + \\frac{\\sin((1-c)\\theta)}{\\sin(\\theta)} \\mathbf{x}.$\n\n> Our goal is to show $\\mathbf{y}$ is closer than $\\mathbf{x}$ to a Gaussian in the second-moment (energy-shell) sense. Since a standard Gaussian has second moment $\\mathbb{E}[\\lVert Z \\rVert^2] = d$, we define the gap of second moment as:\n\n\n> $\\delta(Y) := \\lvert \\mathbb{E}[Y^2] - d \\rvert.$\n\n> For any $c \\in (0,1)$, we have:\n\n> $\\delta(\\mathbf{y}) \\leq \\delta(\\mathbf{x}).$ \n\n> Proof sketch: expand all the terms and apply trigonometric identities. \nIn addition, some discussion about interpolating with Gaussian nosie or adding Gaussian noise to improve image sampling quality can be found in the literature [1,2]. \n\n> 2. Furthermore, in the extreme case that the inverted noise is very not-Gaussian (for example a very blurry or noisy image input). We can perform Algorithm 5 in our paper. We observe that the sample image quality constantly improving by iteratively applying CCS interpolation sampling many times. In this sense, our method has the potential of performing image restoration, which is demonstrated in Fig.5 in our paper. Nevertheless, we acknowledge that non-standard Gaussian initial noise is a significant limitation of our work and will be better handled in future work. \n\n\n***4. Q: Extending it to fine-grained image editing***\n> Thanks reviewer for pointing this direction! We are able to extend our CCS method to fine-grained image editing using null-space projection on the perturbation direction. The idea is that first we find the editing direction and then using Algorithm 4 in our paper for precise fine-grained editing. To do this, we can borrow the algorithm from LOCO-edit [3] The pseudo-code is listed below:\n\n> Step 1. Obtain mask of area of interest (for example excluding the background)\n\n> Step 2. Compute DDIM inversion of the input image with source prompt\n\n> Step 3. Follow LOCO-Edit [3] Algorithm 4 to obtain an local edit direction (here we need to tune the direction length by image quality)\n\n> Step 4. Now we have two initial noises, one by adding the local edit direction, the other is obtained through LOCO-Edit. We can then apply Algorithm 4 in our paper for precise local editing with user-specified editing strength.\n\nWe perform experiments on this, which shows promising preliminary results. However, this is under development and will be investigated more thoroughly in future work. \n\n***5. Q: Limitations not found***\n> Sorry for the confusion. We place the limitation section in the Appendix C, stating that our CCS algorithm does not purely focus on improving sampling quality but improving controllability while preserving sample quality from a novel and orthogonal noise space perspective. So there may be cases that other algorithms achieve better sample quality but with worse controllability. We will revise our paper to put it in our Conclusion section. On the other hand, we will include more limitations in our paper including: \n\n> DDIM inversion not perfectly standard Gaussian.\n\n> Some sampled images may be having issues like overlaying of two patterns or so on.. \n\n\n[1] NoiseDiffusion: Correcting Noise for Image Interpolation with Diffusion Models. ICLR 2024\n\n[2] DreamClean: Restoring Clean Image Using Deep Diffusion Prior. ICLR 2024\n\n[3] Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editing. NeurIPS 2024."}, {"review_id": "VDxbM1BBaW", "reviewer": "Reviewer_qRjk", "summary": "- Introduces new problem of ensuring that generated samples from diffusion model are within specified range of target mean\n- Analyze and characterize linearity of DDIM sampling process with respect to small perturbations in initial noise, both theoretically and empirically \n- Propose an algorithm based on spherical interpolation to ensure generated samples from DDIM fall within certain MSE range of target mean, with hyper-parameter to adjust for diversity in generated samples \n- Compare against self-implemented baselines to demonstrate effectiveness of proposed algorithm", "questions": "*Empirical* \n1. In the experiments, it seems that the target rMSE from the given mean is set to be .12 for Table 2, and .07 for Table 3. Why are these selected as target rMSE? It seems clear that the proposed method is better than baselines at attaining this target rMSE, but it is not explained why attaining this specific value is important or relevant. Does it have to do with balancing fidelity to the target and reasonable diversity? It would be good to explain why targeting this specific value is reasonable in the context of this task. \n\n2. As explained, Algorithm 2 performs a binary search to achieve a target level of diversity by repeatedly calling Algorithm 1, which is where the spherical interpolation is applied to the DDIM sampling process. But this Controller Tuning algorithm seems to be expensive, as it requires multiple calls of the DDIM sampling process. I am not familiar with the baselines, so I am uncertain as to whether this is a reasonable expense. It would be very helpful to provide something along the lines of an average NFE per generated album for the methods to compare the running cost — while CCS outperforms the baselines, this outperformance may not be as significant if it turns out that it is significantly more expensive than the other methods. \n\n*Significance*\n1. Why is it important to be able to control the mean / distance from the mean? I am not fully convinced of the significance of this problem. \n\nCurrently, I am leaning towards rejection. If the significance is clarified, along with some of the technical details, I am happy to recommend acceptance.", "limitations": "The authors acknowledge that their method only controls the target mean and spread from the target mean. However, the abstract and the introduction make it seem as if they are capable of controlling various statistical properties. I think it may be better to align the introduction and the abstract with what is actually shown, as the current method can only control the mean / distance from the mean and not higher level moments.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "*Empirical*\n\nThe empirical comparisons seem to be thorough — the baselines selected seem relevant towards the specific task. The experimental setup itself seems well motivated. The appendix contains sufficient details to reproduce the experiments, or at least understand all of the specific setup parameters. \n\n*Theoretical*\n\nThe theoretical contributions seem quite clear, and the proofs in the appendix are easy to follow. I liked the core finding that the overall change is at most linear, given that small changes are approximately linear in effect. \n\n*Algorithmic*\n\nThe algorithm boxes themselves are laid out nicely and are easy to follow. However, I think it would be good to discuss the computational cost of the proposed algorithm in comparison to the baselines, as otherwise it is difficult to make fair comparisons.", "was_revised": true, "final_justification": "I decided to increase my score by two points from a 3 to a 5. I believe that the strengths of the paper are as follows: \n\n1. Well-motivated problem within diffusion and image generation \n\n2. Theoretical explanation for linearity effect and DDIM inversion\n\n3. Practical effectiveness in both level of control and computational cost. \n\nPrior to their rebuttal, I was only confident in the second point. However, the authors have convinced me of the first and third point. The explanation of the problem's relevance in their rebuttal convinced me of the first point; and the pSNR ablation + NFE comparison convinced me of the third point. I was willing to increase my score if I was convinced of either point, but as I have been convinced of both points, I felt that this paper warranted higher than a 4. I feel that this paper would be a good contribution to NeurIPS.", "rebuttal": "We thank the reviewer for the careful and insightful review!\n\n\n***1. Q: Why is it important to be able to control the mean / distance from the mean?***\n\n> Thanks the reviewer for raising this important question. This question points to the core novelty and impact of our work.\n\n> We may be confused about why we want to control the distance of the samples to the target image? or why we want the distance between sample mean and target image to low? Intuitively a sample mean should be blurry since the averaging of different image features should be blurry. Then why computing PSNR between sample mean and target image make sense?\n\n> 1. For image editing, there are numerous work built on DDIM inversion. However, to our best knowledge, there is no systematic quantitative study and benchmark on why DDIM inversion is better than other techniques such as SDEdit [1] or Classifier-guidance (DPS) [2]. Most of the papers give vague intuitions such as: \"Other techniques struggle to accurately preserve the input image details.\" (comparing with SDEdit) [3].  A fundamental issue here is how much percent the input image details are lost by techniques such as SDEdit.\n\n> One may think that we can compute the similarity (such as LPIPS) between sampled/edited images and the input image to measure how much the input image features are preserved. However, this metric can be misleading since if the samples and the input images are identical, we can have 100 percent similarity. So here comes our ***first novelty***: we control the average distance between the samples and the input image (rMSE) to be fixed and then compute the sample diversity and feature preservation. \n\n> 2. Then, to measure the source feature preservation capability with fixed rMSE level, we may want to know what specific features can be preserved? Also are there some ***common*** features been preserved for each sample consistently? These questions are important for image editing users since they may want human faces been preserved but not the background or other textures. If we compute the average LPIPS distance between samples and the input image, there is no way we can learn about the common features preservation. Then here comes our ***second novelty***: measuring the distance between sample mean and input image.\n\n>Thank to the linearity property revealed in our paper, we found that the sample mean using CCS sampling is surprisingly sharp and not blurry. We then have an idea that the sample mean should contain information about the common features in all samples, so if we directly compute the distance between sample mean and input image, then we can reveal both how well samples contain details from input image and how well the common features being preserved in all samples. A visual example can be found in Fig.6 and Fig.7 in the Appendix (for example in Fig.6 all sample images have eyes looking similar to the input image, which is revealed by the sample mean (otherwise the sample mean will be blurry)).\n\n>3. Furthermore, users may want fine-grained control over source feature preservation. Suppose that we are building a photo editing/creation app, and users want to generate new photos based on their input photo. we may need a sliding bar functionality that when a user slides that bar, the more different generated photos are from the original photo. Different users may have different preferences about how the generated photos look similar to their original photos in texture, facial expression, hair style and so on. Providing this sliding bar functionality will be very useful to satisfy their needs.\n\n>This is the motivation about why we propose this novel problem. We believe this problem and evaluation method can be impactful for image editing or conditional image generation. We will revise our paper to explain and highlight this.\n\n\n\n\n\n***2. Q: Why are these selected as target rMSE? Does it have to do with balancing fidelity to the target and reasonable diversity?***\n\n> Thanks the reviewer for raising this important and interesting question. \nThe target rMSE in this paper is set for best visualization quality (let readers see samples close to source image and have sufficient variations). Investigating how performance changes with different rMSE is interesting and we will add in revision.\n\n> We perform additional experiments with sampling quality benchmark with rMSE targets from [0.05, 0.06, 0.07,0.08,0.09, 0.10] for Stable Diffusion on CelebA-HQ. We observe that our method consistently performs quite well (PSNR is for sample mean v.s. target image, SD is for diversity). (numbers are reported in the order of CCS-C/CCDF-C (the strongest/best baseline))\n\n| Target rMSE | PSNR↑  | MUSIQ↑ | CLIP-IQA↑| SD ↑|\n| ---------| ---- | ----- | -------| -----|\n|0.05 | ***32.22***/30.86   | ***49.60***/49.53 | 0.729/***0.730*** | ***0.036***/0.034|\n| 0.06 | ***31.44***/29.03| ***49.58***/49.02 | ***0.731***/0.731| ***0.043***/0.040|\n| 0.07 | ***30.29***/27.66 | ***49.66***/48.91 | 0.732/***0.735***| ***0.053***/0.051|\n| 0.08 | ***30.10***/26.80| ***49.85***/48.23 | ***0.742***/0.729| ***0.056***/0.052|\n| 0.09 | ***29.74***/25.98 | ***49.82***/48.01 | ***0.740***/0.732| ***0.061***/0.054|\n| 0.10 | ***29.31***/25.20 | ***49.80***/46.74 | ***0.731***/0.727| ***0.063***/0.057|\n\n\n> We also perform experiments on FFHQ dataset for rMSE levels of [0.09, 0.12, 0.15], and summarize the results below:\n\n| Target rMSE | PSNR  | MUSIQ | CLIP-IQA| SD |\n| ---------| ---- | ----- | -------| -----|\n|0.09 | ***28.25***/ 27.08   | ***66.53***/66.10 | 0.749/***0.750*** | ***0.078***/0.069|\n| 0.12 | ***25.13***/23.52| ***66.79***/66.15 | ***0.750***/0.746 | ***0.104***/0.088|\n| 0.15 | ***23.45***/20.64 | ***66.71***/65.24 | ***0.743***/0.740| ***0.131***/0.104|\n\n\n> We observe several intersting phenomenon in this additional experiment:\n> 1. With more perturbation of noise (the sample images farther from the the target image), there is no decrease in sampling quality. Instead, there is slight increase in MUSIQ for stable diffusion which means that image quality increases and looks sharper. One explanation for this is that the target image may not have good image quality or the DDIM inversion is slightly apart from the sphere, so when increasing the perturbation level, more Gaussian noise is interpolated so the interpolated noise becomes more \"Gaussian\" and the image quality may improve. Since the Stable Diffusion is a conditional model and the input image is not from its training data distribution, we observe that the DDIM inverted is slightly off the Gaussian sphere. But for FFHQ, the inverted is quite Gaussian, so there is no significant change in sample quality. \n> 2. the sampled mean is still close to the target image when increasing rMSE level. We also do not observe sudden drop in controllability and diversity (PSNR and SD). However, with increasing rMSE target, it becomes harder to control the sample mean close to target mean, as demonstrated by declining PSNR. Nevertheless, this trades off with diversity improvement.\n\n\n***3. Q: But this Controller Tuning algorithm seems to be expensive, as it requires multiple calls of the DDIM sampling process.. It would be very helpful to provide something along the lines of an average NFE per generated album for the methods to compare the running cost***\n\n> Thanks reviewer for raising this! This is also an important question. We will add this in revision. Our method actually achieves the best overall efficiency. The controller tuning can be costly fror our goal, but the zero-order controller should still be one of the best choices compared to first-order gradient searches.  We further use binary search to improve time complexity to log scale, which should be faster than other zero-order methods [4].\n\n> We provide sampling speed and sampling NFE in the table below. The sampling procedure consists of two steps: \n> 1. controller tuning for statistical constraint. This only needs to perform once for each album. For our method and GP, we have an addition one-time single-image inversion step (turns out to be very fast). \n> 2. sampling with the tuned parameters. \n> Thanks to the linearity property, the binary search algorithm can find the feasible scale of perturbation efficiently, and we achieve the best controller tuning efficiency. Otherwise, the feasible scales may lie in a much narrower region due to abrupt change in output or vary a lot for different samples, and we may need more rounds searches to find them. We will add a discussion about sampling efficiency in our revision. We provide the inference time for sampling 120 images around a target mean for different methods below for Stable Diffusion tested on one A40 GPU. We also provide the inversion NFE, controller tuning NFE per image, and sampling NFE per image for different baselines. \n\n|Method\t|sample time per image| total inversion NFE | total controller tuning NFE | sampling NFE |\n| -------| -----| ------| ------|-----|\n|GP-C|\t1.66s|  45 | 96 | 45 |\n|CCDF-C|\t1.73s|  ***0*** | 163 | ***42*** |\n|LDPS-C|\t5.84s|  ***0*** | 234 | 50 |\n|CCS-C (Ours)|***1.65s***| 45 | ***94*** | 45 |\n\nThe advantage of CCDF-C or in other words (SDEdit-C) is that it does not require DDIM inversion, and requires fewer timesteps for denoising, but it needs more controller tuning rounds since the outputs can be very sensitive to some timesteps.\n\n****4. Q: Limitations: introduction not clear****\n> Thanks for pointing out! We will revise our introduction to \"We study controllable sampling with mean close to a target mean at a target diversity level for diffusion models.\"\n\n[1] SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. ICLR 2022\n\n[2] Diffusion Models Beat GANs on Image Synthesis. NeurIPS 2021\n\n[3] Null-text Inversion for Editing Real Images using Guided Diffusion Models. CVPR 2023\n\n[4] Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps. CVPR 2025"}]

Author rebuttal:

Meta-review: * Summary
Authors propose a new method to guide diffusion models in a training-free way, denoted as CCS. The method relies on performing a conformal/near-linear transformation on the noise factor itself, rather than performing a diffusion process on the data explicitly. This actually sheds light on not just the noise but how the entire noise distribution gets changed under the dynamics through linear steps.

* Strengths and reasons for acceptance 
The paper addresses an essential aspect of diffusion models, the actual noise dynamics, which is novel in both its examination and by providing tractable analytical arguments as well as empirical evidence on several datasets. It's also, to the best of my knowledge, the first time someone does that by moment matching on the noise function. On the empirical side, the authors (after rebuttal) also performed a thorough ablation test of hyperparameters. The paper (in the appendix) also gives all of the required theoretical proofs for the results and statements made in the main paper. All of the above makes me recommend this paper for a weak accept. I have no issue in bumping this down if needed, especially given the weakness. 

* Weaknesses 
The paper is not as clear or well-written, while the authors have raised the level post rebuttal, which is still contingent on integration post rebuttal comments. While authors do give some of the experimental results, they are still on a pretty small dataset, and more importantly, the examples that are shown visually seem a bit cherry-picked, and there is no negative example shown.
Decision: Accept (poster)