Manifold Preserving Guided Diffusion
Authors: Yutong He
Naoki Murata
Chieh-Hsin Lai
Yuhta Takida
Toshimitsu Uesaka
Dongjun Kim
Wei-Hsiang Liao
Yuki Mitsufuji
J Zico Kolter
Ruslan Salakhutdinov
Stefano Ermon
Venue: ICLR 2024 poster (2024)
Keywords: generative model
diffusion model
controllable generation
Primary area: generative models
TL;DR: We propose a training-free guided diffusion framework that can generate high quality samples with minimal additional resources

Abstract: Despite the recent advancements, conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training. In this paper, we propose Manifold Preserving Guided Diffusion (MPGD), a training-free conditional generation framework that leverages pretrained diffusion models and off-the-shelf neural networks with minimal additional inference cost for a broad range of tasks. Specifically, we leverage the manifold hypothesis to refine the guided diffusion steps and introduce a shortcut algorithm in the process. We then propose two methods for on-manifold training-free guidance using pre-trained autoencoders and demonstrate that our shortcut inherently preserves the manifolds when applied to latent diffusion models. Our experiments show that MPGD is efficient and effective for solving a variety of conditional generation applications in low-compute settings, and can consistently offer up to 3.8× speed-ups with the same number of diffusion steps while maintaining high sample quality compared to the baselines.

Published as a conference paper at ICLR 2024
MANIFOLD PRESERVING GUIDED DIFFUSION
Yutong Hel.2+1, Naoki Murata 2M Chieh-Hsin Lai? Yuhta T
Toshimitsu Uesaka' Dongjun Kim?t Wei-Hsiang Liao², Yuki Mitsufuji2.
J. Zico Kolter' Ruslan Salakhutdinov' Steefano Ermon "Carnegie Mellon University. 2Sony AI, 'Sony Group Corporation, *Stanford University yutonghe@ cmu. edu, naoki mwr.uuratt..... muraaatsonny com
ABSTRACT
Despite the recent advancements, conditional image generation still faces chal- lenges cost, generalizability, and the need for task-specific training. In this pa- per, propose Manifold Preserving Guided Diffusion (MPGD), a training-free conditional generation framework that leverages pretrained diffusion models and off-the-shelf neural networks with minimal additional inference cost for broad range of tasks. Specifically, we leverage the manifold hypothesis to refine the guided diffusion steps and introduce shortcut algorithm in the process. We then propose two methods for on-manifold training- free guidance using pre-trane pre-trained au- toencoders and demonstrate that shorteut inherently preserves the manifolds when applied to latent diffusion models. Our experiments show that MPGD is efficient and effective for solving variety of conditional generation applications in low-compute settings, and can consistently offer up to 3.8x speed-ups with the same number of diffusion steps while maintaining high sample quality compared to the baselines. Our code is available via the project page here
Noisy Linear Inverse Problems
FaceID Guidance Generation
MPGD (- (s/mmg)))
Input Reference
w/o Proj.
MPGD
C1.IP Guidance Generation
Style Guidance + Stable Diffusion
FacelD Guidance + Stable Diffusion
Unconditional
MPGD
Toput Reference
MPGD (00mmmg))
Inpor Reference
MPGD
Prompt: beaasforrgg weartng lipnne"
Prompt: "a conal Vericee"
Prompt: bendvlot of mulle paner choraniee"
Our proposed MPGD training-free sampling method for both pre-trained pixel-space diffusion models and latent diffusion models in variety of conditional generation applications. MPGD can be applied broad range of tasks with minimal sampling time and high sample quality.
Figure 1:
"Equal contribution
Work done during internship Sony AT

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Generative modeling has witnessed extraordinary breakthroughs in recent years (OpenAl 2023 Ho
et al. 2020: RRobacc et al., 202T Conditional generation, in particular, stands out crucial lask, as it underlies solutions to several real-worid problems, including image restoration, super- resolution, and creation of content with specific styles. However, despite the significant attention, conditional generation still faces its own set of challenges related to cost and generalizability: typical conditional generation requires either additional task lask-specific training, data collection, model archi- tecture designs, or extra assumptions about the conditional generation tasks (Zhang et al., [223] Ruiz et al., 2023; Isola et al. 2017; Park et al., 2019, Li et al., 2023). These requirements not only escalate costs but also restrict the range of applications and potential users.
Recent developments in diffusion models offer potential solutions to overcome these challenges (Song et al., 2021b] Dhariwal & Nichol, [021] Wallace et al., 2023). In particular, Chung et al. (223a) and many of its followup works (Song et al.] 2023b] Bansal et al. 2023; Yu et al. 2023) off-the-shelf loss functions to guide the sampling process. While these methods avoid extra training of diffusion models, their reliability remains inconsistent - at times they exhibit impressive performance, while in other instances they struggle produce realistic images. Moreover, these methods tend to be extremely slow because they rely on extensive sampling time optimization and/or exceedingly large number of diffusion time steps produce satisfactory samples. Above all, current literature has very limited understanding of when, why, and how these methods succeed or fail, making it difficult design practical implementations in real-life applications.
In this paper, we propose Manifold Preserving Guided Diffusion (MPGD), a framework for con- ditional generation using unconditionally pretrained diffusion models with (1) no extra training (2) minimal additional computation and sampling time (3) generalizability to a broad range of tasks and (4) high sample quality. Central to our method, we leverage the so-called manifold hypothesis - the fact that the real data does not lie within the totality of the pixel space, but instead lies on a very small underlying manifold. Our key idea is that instead of guiding the diffusion process without constraint (until the last time step, when it hopefully arrives at the manifold), we can project the guidance to the manifold, via its tangent spaces, throughout the diffusion process. Moreover, when using the DDIM (Song et al., 2021a) sampling approach, we also show the method leads to effi cient "shortcut" for guidance gradients that saves both time and memory and substantially improves the sample quality over competing approaches in low-resource settings.
With this new framework, we derive several novel methods to perform training-free guided diffusion generation. We specifically analyze two different practical approaches to manifold projection using off-the-shelf unconditionally pretrained autoencoders for pixel-space diffusion models. We also show that applying our shortcut latent diffusion models is naturally manifold preserving and can significantly improve the sample quality and the inference speed. Finally, we can extend the current framework incorporate multi-step optimization algorithms to further improve the performance.
We empirically test our methods against competitive training-fres guided diffusion baselines on various conditional generation tasks, including solving noisy linear inverse problems, human face generation with facial recognition model (FacelD) guidance, and text-to-image generation guided by certain input style, as illustrated in Figure Experiments show that our methods find bet- ter tradeoff between fidelity and controllability compared to the baseline methods and are able to consistently achieve up to 3.8x speed-ups while maintaining high sample quality.
CONVENTIONAL TRAINING FREE GUIDED DIFFUSION
2.1 PROBLEM FORMULATION
Let æ E X C Rd be a d-dimensional sample in the support of the data distribution and y E )' be the given input condition such as text description and an input reference image. In this paper, we aim at solving the problem of conditional generation by attempting to sample from the posterior distribution p(r y). We assume we have access pretrained generative models for the prior distribution p(x), and a differentiable loss function L(x; y) giving us the posterior p(r (y) oc p(z)) exp( [xx; y)).
2

---

Published as a conference paper at ICLR 2024
We target solutions that are: (1) Training free: Pretrained models can be deployed without extra training. (2) Low cost: The method should require minimal additional computational resources and time, (3) Generalizable: We only require black-box access to the loss function and its gradients, (4) High quality: The samples should come from a distribution that close to the true posterior.
2.2 DIFFUSION MODELS
The score-based generative models (Song & Ermon 2019; Song et al. b021b), diffusion mod- els (Sohl-Dickstein et al. 2015; Ho et al. 2020), enable sampling from a clean data distribution by iteratively using the dime-dependent score function vx, log (c for noisy data 2,, where E [0, T] and T > 0. In DDPM (Ho et al.) 2020), noisy data 2t is obtained by adding Gaussian noise 4 (O, T) to the scaled clean data x ~ p(X), as St = vāyz + vI agft where āg > is a scaling parameter. The score function is frequently parameterized through denoiser co(22, t) trained with the loss function E,, |||e - (Q(x,,))l22 so that cy estimates Gasussian noise cluded in noisy sample Tt. In the inference time, we can obtain clean data samples by applying the score function iteratively to noisy samples (Song & Ermon 2019). In particular, DDIM (Song et al. 2021a) performs each step of the sampling with the update rule
Zt
/1 (12)(I2.
+ . - a o 0e6
Ft-1 v
Voz
where the first term the right-hand side corresponds to direct estimation of the clean data from the noisy data 2e by the diffusion model, which is derived from Tweedie's formula (Efron 2011). denoted as Zofe- In this formulation, 5₁ = /(1 ā a - -ā) - á corresponds to the DDPM sampling, and with o, 0 the sampling procedure becomes deterministic.
2.3 RELATED WORKS ON TRAINING-FREE GUIDED DIFFUSION
Building upon early efforts such as Song et al. (202lb) and Meng et al. 2022) (detailed discus- sion is in the Appendix A), many recent papers, including classifier-guidance diffusion (Song al., 20216, Dhariwal& & Nichol 202T), DPS (Chung et al. 2023a). IIGDM (Song et al. 2022). Free- DoM YYu et al.) 2023), UGD (Bansal et al.) 2023). attempt to leverage pretrained diffusion models for various conditional generation tasks. The common underlying concept shared by these methods to decompose a conditional score function V., log (x ly) into the unconditional score function and the loss-based term Vz, log p(gg(()))) = Vz, log p(z)) + V., (x.
The discretized sampling procedure with the decomposed conditional score can be interpreted as a two-step process based on the additivity of the terms. When an initial noisy sample is given, denoised -z-1 is obtained by the sampling process with the unconditional diffusion model. Subse- quently, Iz-1 is further updated using the gradient of L, (z y) with respect to Zt.
In particular, if we assume the noisy log likelihood can be accessed by time-dependent differen- tiable function, Læ(x;)) y). the second step be regarded
optimization step by gradient descent to minimize the guidance loss in the vicinity of the denoised sample xz:
It - It Pt V x, Le (Zx;z))
(2)
where Pe is a time-dependent step size parameter. Hence, the optimization problem solved in this step is represented as:
min Le (z)(vv)) #/EN(.))
where N(x)) fx E IR 21) <n) Rd is neighbourhood around x, in Rd bounded by some radius r2 which is related to the optimization step size pt.
However, one caveat exists: usually the pretrained guidance loss function is only defined on clean data & instead of noisy data xt. In other words, we usually only have access to an I. that is trained on clean data rather than L 's that are trained noisy data. To solve this problem, Chung et al. (202a)) uses the clean data estimation Tal: Va, (I: VI - azeg(Ie, t)) from the Tweedie's formula (Efron 201T) as point estimation of the true loss term Therefore, we can rewrite the update rule as
f - Jg Pr V., (
and many previous methods follow this formulation. For example, DPS deals with inverse problems of the form y = A(z))+, where A(-) is differentiable function of a and is additive observation
3

---

Published as a conference paper at ICLR 2024
M:
M
M,
Maniiiok
Ma-1
Mi-r
M
Bi-
M
Tangers
Spoor
DOEM
2se(F
Tan M
Guidanee
M
MM
Toys
Paj
Zays
Projection
Maniodi
(a) DDIM (Unconditional)
DPS DDIM (Baseline)
MPGD Proj. (Ours)
MPGD (Ours)
Figure 2: A schematic overview of our proposed approaches and an illustrative comparison with DDIM (Song et al. 2021a) and DPS (Chung et al. 2023a).
noise. In cases with Gaussian observation noise, it defines the loss as L(z; y)
y A(=)
LGD (Song et al., 2023b), UGD (Bansal et al.; 2023), and FreeDoMM (Yu et al. 2023) offer more flexibility in designing loss functions, allowing them handle variety of tasks. For instance, in the case of FacelDDguudde generation, FreeDoM adopts a loss that calculates the & distance between
the features obtained by facial recognition model and the features for the target face image.
ISSUES IN THE PREVIOUS FORMULATION: THE MANIFOLD HYPOTHESIS
In the previous formulation in equation 3 the neighborhood (xx)) for the optimization objective resides the ambient space R. However, in practice the data lies in a much dimensiinas space than the ambient space. Typically, the following assumptions can be made:
Assumption (Manifold Hypothesis). The support X' of the data distribution of interest lies on a k dimensional manifold M that is embedded in d dimensional ambient space IR' such that <
Assumption 1.1 (Linear Subspace Manifold Hypothesis). The data manifold M C Rd is a linear subspace of dimension < d.
Chung al. 2022 2023b) have shown that with linear subspace manifold hypothesis and Gaus- sian annulus theorem BBlum et al.] 22000) at a certain diffusion time step I, the noisy data I; also manifold M, that has dimension d - and shell-like geometric
probabilistically concentrate structure to the original data manifold M. Here we provide an extended version of the proposition stated mathematically as follows:
Proposition 1 (Concentration of Noisy Samples (Informal, extended from Chung et al. 2022 2023b))) Define d(x, v, M. := in ||x vz'll2 for v > O, and B(M;r) == x Rd
dlz M) r} for > 0. Consider the distribution of noisy data (xx) f where p(z,(x) := (N (1 I)). Then under Assumption 1.1 Pe(xe) is "probabilistically concentrated" the (d -1) 1)-dimensional manifold Me defined
M₁ = (a e Rdd d(xx, Ta. V(
The formal version of Proposition D and its proof is provided in Appendix B
As a result, because the neighborhood resides in the ambient space rather than on the manifold Mr. not points in (axe) Equation 3 are necessarily close to or included in the manifold M,. This finding, which is empirically verified in Figure 3 suggests that the results obtained through optimization within (xx) may deviate from M, and adversely affect the evaluation of the score function (or sampling by the diffusion model) in the following steps, since the score function is Therefore, optimization within N(x)) observe
trained only with samples close to M, due to Proposition]] I cannot guarantee that the final result will correspond to realistic image. In practice, that methods such as DPS, UGD and FreeDoM require detailed fine-tuning of step size scheduling, techniques such repainting (Lugmayr et al. 2022), large number of diffusion time steps to ensure that gradient updates don't 't deteriorate the final results.
4 MANIFOLD PRESERVING TRAINING-FREE GUIDED DIFFUSION
Based on our analysis in Section 3 we reformulate the objectives in Section 2.3 to address the manifold hypothesis and propose the following framework to perform on-manifold guided diffusion.
4

---

Published as a conference paper at ICLR 2024
4.1 OBJECTIVE
We first rewrite the minimization objective in Equation 3 by considering a different neighborhood than N (xx). Since Mr is manifold, the "neighborhood" can be represented an open subset of the tangent space Ta,Mp of Iy. which is homeomorphic to an open subset in R& for k < d (Shao al., 2018). Intuitively, optimizing on small neighborhood on the tangent spaces allows us only make "reasonable" changes to the samples. With tangent spaces, we can write our objective as
L( [ ( C. Vā=
min
(5)
where (xx = Me d(I,,,) < <r) C M2 is small neighborhood around zy in its the tangent space Mr and r; is the radius of the neighborhood related to the optimization step size Pt. The objective is to find the point 2 in that neighborhood such that its Tweedie's estimation Foje
Vas (at VI (xt ()) of the clean data best aligns with the given conditions y.-
4.2 METHOD
Conventionally we can estimate the tangent spaces of a data manifold using an autoencoder (Shao et al., 2018 Bordt et al.] 2023 Srinivas et al.) 2023 Anders et al. 2020). The key idea is that, the information bottleneck in autoencoders naturally incorporates manifold hypothesis and a well- trained autoencoder yields latent representations that implicitly capture the local lower dimensional coordinates for the data manifold. However, while most off-the-shelf autoencoders are trained on the clean data, notice that in Equation 5 we need access to the tangent spaces of the noisy samples Ie in M2. So how should we achieve this goal with only access to clean data manifold M
4.2.1 THE MPGD SHORTCUT
Combining the results of Proposition I and Lemma 2 in Appendix B we first obtain the following theorem that facilitates us to perform manifold preserving guidance with access to only the clean data manifold: If guidance gradient preserves the manifold for clean data, it also brings a noisy sample on a noisy manifold.
Theorem 1. (Informal) Assume the gradient V. (. y) lies on the tangent space T M, and the diffusion model co(2,, t) is optimal. Then with Assumption 7.1 scalar G2 > and update rule
It-1 Vāt-i( ca von u)) / â -
OsEt:
(6)
we can obtain an - whose murginal distribution probabilistically concentrated on M
The formal statement, proof and discussions are provided in the appendix. Therefore, we can derive a simplified update rule for manifold preserving guided diffusion that only requires access to the tangent spaces of the clean data manifold M as long as we can ensure that V zoo [(zoje y)) is also on the tangent space Taoq M:
Foje < Foje - 4 V Kajs M(Fo(iy) v)
(a step of gradient descent)
F--1 t- vār- + /1 (co1co(24, (rescaling of the clean data and the noise)
(8)
This algorithm can also be intuitively viewed as updating the DDIM clean data estimation Toje at time t with the guidance gradient with respect to that estimation. While this approach is generally faster since it doesn't t require computing the gradient with respect Ia for the score function, should note that it requires the guidance gradient V Vz L(zgjs y)) to reside in the tangent space of the manifold M. leading on-manifold samples. Therefore, we further investigate the ways to project the guidance onto the manifold, and refer to this shortcut as manifold preserving guided diffusion without projection, MPGD w/o Proj..
M.
4.2.2 MANIFOLD PROJECTION WITH (PERFECT) AUTOENCODERS
Now that we have established an algorithm that only requires access to the clean data manifold, we can use an off-the-shelf autoencoder to project the guidance onto the tangent spaces. To demonstrate the process, first showcase the derivation where we have access to a perfect autoencoder. Note that the following is inspired by (Shao et al. 20188] Anders et al. 2020). but does not perfectly match with them.
5

---

Published as a conference paper at ICLR 2024
Algorithm MPGD for pixel diffusion models
Algorithm 3 gM: On-Manifold Guidance
1: 2r ~ -/( I)
1: if MPPD-AA then
2: for t= T. T..... do
2: Zuje = Zajt C. V
Fojt (()(E(20))))))
3:
6, ~ 10,, -
3: else if MPGD-Z then
4: Zuje y (Te V1 ageg o (z,, 1))
4: Z01+ E s)
5:
if requires manifold projection then
5:
Z01+
V #o. ((((2o))):
Fal: = (Zvee (Z e)
6: ZDje = ((
7:
else
7: end if
8:
F zote V ((
8:
return dfujc
9:
end if
10:
2t-1 = v 12
MPGD who thi
11: +V1 51-1 c0 f) tve
MPGD-AE (3.5 0.33)
12:
end for
Devedonn from the marfood
MPGD-AT (2.5 -1
13: return 150
0.125
0.100
Al Algorithm 2 MPGD for latent diffusion models
0.075
1: 2r I)
2: for T. T...., ...,,1 do
0.050
6, ~ (O.
0.025
4: 20/r vaa (Zi - v1 (2),))))))
1
5: Zoit = :0 c+V
#01t L((((zo)))e)) V
0.000
1.0
0.8
0.5
0.4
0.2
0.0
21-1 = 101-
Dimuuion Time Stop
+VI ā,-1 of c0 Oeft
Figure 3: We analyze the deviation from the manifold throughout the diffusion process for different methods (details are Appendix C
8: end for
9: return D(=0)
Assumption 2. (Perfect Autoencoder) Assume for the support X C. M the data distribution, there exists a perfect autoencoder with encoder E X Z and decoder D 2-> X with Z I2 for k- < d. This autoencoder exhibits zero reconstruction error for each point on Da MM, i.e., TD = E(xa). Furthermore, the decoder D is surjective M, and the encoder function and the decoder function D form pseudoinverse pair (Sorrenson al. 2023), implying E D is identity map.
Under the assumptions of perfect autoencoder, we can obtain gradients that preserve the manifold, as supported by the following theorem, the proof of which is provided in the appendix.
Theorem 2. If an autoencoder with encoder E and decoder D is a perfect autoencoder for the 0f. 'D BE E M.
support X C M of the data distribution, then ao L(o((((((r (x)(()(())); 3) 3D aE Azgit
Therefore, to achieve the local minima of the objective function in Equation update rules in Equation| as:
5 we can modify the
Foj < Foj c2 V L(D(E(=()))):)
where with linear manifold hypothesis, the guided Zoje is on the tangent space Tag M and 21-1 is concentrated on M We refer to this method as MPGD-AE.
Although our analysis consists of perfect autoencoder assumption, in practice, we find that well- trained imperfect autoencoders such as VQGAN's (Esser et al., (2000) also have similar effects for mapping the guidance the data manifold. In Figure we empirically verifies VQGAN's manifold preserving ability by using it as the manifold projection function of MPGD-AE. Detailed discussion included Appendix C We also present further analysis and experiments with empirically well- trained imperfect autoencoders in Section
Manipulating the Latents The idea of preserving the manifold using perfect autoencoder can be also achieved as the following: Rather than updating Tujz with gradient decent, we modify the encoded latent variable Zujs with its gradients instead. After updating Zuja, we then map it back to the data space with the decoder to obtain a new estimation of zo We refer to this method as MPGD-Z. The details and comparison among three methods are provided in Algorithms [ and where we refer to the manifold projection function as 9MM- We also provide additional analysis in Appendix B.4

---

Published as a conference paper at ICLR 2024
Messurement
Ground Truth
MPGD wio Proj. (Ours) MPGD-AE (Ours)
MPGD-Z (Ours)
DPS (Baseline)
Voiey Suppe-Resolutioon (X4, = 0.05)
DDIM Steps 20
DDIM Stops 100
DDIM Steps 20
Noiey Gaussian Deblar (a 0.05)
ImageNNet 256X256
DDIM 100
Figure 4: Qualitative examples of solving noisy linear inverse problems with our proposed MPGD and baseline DPS.
4.2.3 MPGD WITH LATENT DIFFUSION MODELS
Latent diffusion models (LDM), proposed by Rombach et al. (2021). is a procedure to gradually transform sample 2T R to 20 Z where the same space as the latent space of a well- trained autoencoder such as VQGAN (Esser et al. 2020 or VQVAE (van den Oord et al. 2017).
With the same intuition, we can also manipulate the latents in LDM using the same technique de- scribed in the previous section. Since LDM operates on the latent space of the autoencoder, the de- coded latent guidance D(V z01 (=D(= v)) is on the tangent spaces of the data manifold. There- fore, with linear manifold hypothesis, the final sample æp is the manifold M. We refer to this approach with LDM as MPGD-LDM and provide the details Algorithm 2 and Appendix B.5
4.3 MULTI-STEP OPTIMIZATION
The current framework performs one-step gradient descent the clean data Zoje for the objective in equation 5 Nevertheless, for this objective, we can also employ more sophisticated optimization solvers such nonlinear conjugate gradient method (Hager & Zhang 2006), and provided the manifold remains preserved, execute multiple optimization iterations. This can potentially lead to improvements in both quality and speed.
Although motivated differently, "Time- "rinee-raveeing" or "Repainting" (Lugmayr et al., 2022 Wang al. 2022 Yu al. 2023) is another technique that implicitly performs multi-step optimization to minimize the guidance loss. Specifically, the process of adding noise after each gradient descent step can be interpreted stochastic optimization via stochastic gradient Langevin dynamics (Welling & Teh 2011). We show the results where we employ the both nonlinear conjugate gradient method and travellig for faceID guidance Stable Diffusion generation in Appendix E.S] Figure
5 EXPERIMENTS
We empirically compare the performance of our proposed methods with baselines in three experi- mental settings. For the pixel domain diffusion, we test our methods with simpler linear inverse problem and more complex nonlinear problem. For the latent diffusion models, we evaluate our method with the two conditions the same time, one included in the pre-trained model setting and the other provided by the loss function, to examine its ability to understand compositional condi- tions. We also provide further details of the experiments in Appendix D
7

---

Published as a conference paper at ICLR 2024
MPGD wio MPGD AE
IN Prg. MOG
wio Pro.
12
MPGD 2
DPS LSD-NC
10
MCG
4-
Time
DOIM
DDIM
DDM
Figure 5: Quantitative results of FFHQ super-resolution experiment that compares fidelity (log KID), guidance quality (LPIPS) and inference time acToss different numbers of DDIM steps.
Isput Reference
DDIM (Uncondisonaa) MPGD wio Proj, (Ours) MPGD-AE (Ouns)
MPGDD (Ouns)
FreeDoM (laseline)
LGD-MC (Baseline)
Figure 6: Examples of the FacelD guidance generation with pre- pre-trained CelebA-HQ models.
5.1 PIXEL SPACE DIFFUSION MODELS
In this section, evaluate the performance of our proposed pixel domain methods (i.e., MPGD w/o Proj.. MPGD-AE, and MPGD-Z) with two different sets conditional image generation tasks: solving linear inverse problems and human face generation guided by face recognition loss, which we refer to as Facell) guidance generation. For MPGD-AF and MPGD-Z, we use the pre-trained VQGAN models provided by RRombach et al. (2021). To further demonstrate the applicability of our method, we add the results of CLIP-guided generation experiments in Appendix E.]
5.1.1 NOISY LINEAR INVERSE PROBLEM
For linear tasks, we use noisy super- super-resolution and noisy Gaussian deblurring as the test bed. We choose DPS (Chung et al. 2023a), LGD-MC (Song al. 2023b), and MCG (Chung et al. (022)
as the basleines. We test each method with two pre-trained diffusion models provided by Chung al. (2023a):
trained FFHQ dataset (Karras al.) 2019) and another on ImageNet (Deng et al. 2009). both with 256 x 256 resolution. Mcasurements in both tasks have a random noise with variance of x² 0.052. We evaluate each task on a set of 1000 samples. We use the Kernel Inception distance (KID) (Binkowski et al., 2018) to assess the fidelity, Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et 2018 to evaluate the guidance quality, and the inference time to test the efficiency of the methods. All experiments are conducted on single NVIDIA GeForce RTX 2080 Ti GPU. Figure the generated examples for qualitative comparison, and Figure presents the quantitative results for the super-resolution task FFHQ All three of our methods significantly outperform the baselines with all metrics tested across a variety of different numbers of DDIM steps, and we observe manifold projection improves the sample fidelity by large margin.
5.1.2 FACEID GUIDANCE
We also evaluate our proposed methods on the more challenging nonlinear task of FacelD guided human face image generation. The goal of this task is to generate facial images that resemble reference faces. We choose FreeDoM (Yu et al. 2023) and LGD-MC (Song et al. (023b) as baseline methods. We test all methods with the pretrained diffusion model for the CelebA-HO HQ 256 256 dataset provided by Yu et al. (2023) and DDIM steps. We generate 1000 facial images using the CelebA-HQ test set reference images and evaluate the results using KID and FacelD Loss with
8

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative results for CelebA-HQ
Table 2: Quantitative results for style guidance with
256 FacelD guidance experiment.
Stable Diffusion experiment. Our method finds the sweet spot between following the prompt and follow- ing the style guidance.
Method
KIDI
Time!
DDIM
0.0442
1.3914
3.41s
Method
Style|
CLIP
Time!
VRAM.
FreeDoM
0.0452
0.5690
10.65s
DDIM
761.0
31.61
13.89s
10.80 GB
LGD-MC
0.0448
0.6783
14.64s
FreeDoM
498.8
30.14
26.50s
17.30 GB
MPGD
0.0473
0.5163
5.82s
LGD-MC
404.0
21.16
37.43s
31.65 GB
MPGD-AE
0.0467
0.5309
7.78s
MPGD-Z
0.0445
0.5791
6.93s
MPGD-LDM
441.0
26.61
19.83s
15.53 GB
Primpt
Refermnce
MPGIDLDMM (Oum)
ProdDoo (Rascling)
LGD-MC (Rancline)
min
"e Big Ben clock lowering the city of London"
girl with long curly blonde and sunglasses"
Figure 7: Qualitative results for text-to-image style guidance experiment with Stable Diffusion.
single NVIDIA GeForce RTX 3090 Ti GPU. Figure shows the generated samples for qualitative comparison, and Table [ presents the quantitative metrics. Our methods demonstrates comparable or superior sample quality with substantial speed-ups compared to the baselines. In addition, we also notice that our methods are able maintain the overall geometry generated by DDIM and only make changes to the semantics that are relevant to the guidance. This observation suggests that our method is able to operate guidance in the tangent spaces of the DDIM samples.
5.2
LATENT DIFFUSION MODELS
To evaluate MPGD-LDM, we test our methods against the same baselines as the pixel-space FaceID experiments with text-to ext-to-image style guided generation task. The goal of this task is to generate im- ages that fit both the text input prompts and the style of the reference images. We use Stable Diffu- sion (Rombach et al. 202T) as the pre-trained methods to incorporate style loss, which is calculated by the Frobenius norm between the Gram matrices of the reference images and the generated images. For reference style images and text prompts, randomly created 1000 conditioning pairs, using images from WikiArt (Saleh & El- gammal (2015) and prompts from PartiPrompts (Yu et al. 2022) dataset. Figure 7 and Table 2 show qualitative and quantitative results for this experiment respectively. All the samples are generated on a single NVIDIA 1100 GPU with 100 DDIM steps. Our method finds the sweet spot between following the text prompts, which usually instruct the generation to generate photo realistic images that do not suit the given style, and following the style guidance, which deviate from the prompts. Notably, because MPGD does not require propagation through the diffusion model, our method can provide significant speedup and can be fitted into a 16GB GPU while all the other methods cannot.
exxt-to-mmage model and deploy the guided sampling
6 CONCLUSION
In this paper, we proposed Manifold Preserving Guided Diffusion (MPGD), a novel framework an- chored in the manifold constraint within the diffusion generation process for conditional generation. By focusing on manifold preserving guidance, our approach promises high quality conditionally generated samples, while reducing the computational cost and memory, paving the way for more accessible and reliable guided generation processes. This approach leverages the pretrained autoen- coders to ensure the manifold constraints, offering an efficient solution to the challenges in guided generation. Furthermore, our method incorporates the optimization strategies that enhance the ef- fectiveness of the sampling process.

---

Published as a conference paper at ICLR 2024
7 ETHICS STATEMENT
As training-free guided generation method, our MPGD offers way to approach low-resource con- trol of the large scale pre-trained models. However, while MPGD facilitates low-cost human control over pre-trained models, our method is still subject to potential risks including biases, copyright is- sues and intentional malicious content generation that currently exist in large-scale pre-trained mod- els. We acknowledge the importance of addressing these ethical considerations. Our commitment to ethical research practices extends to ensuring that our contributions do not exacerbate existing inequalities perpetuate harmful biases. Upon the release of our code, will implement safe guards to mitigate inappropriate content creation and we are dedicated to update our safe guarding system keep up with the research regarding safer content generation in the future.
ACKNOWLEDGEMENT
This work is partially supported by ONRN000142312388 We would like to thank Zhengyang Geng, Ruitian Zhai, Martin Q. Ma, Jing Yu Koh and Ellie Haber for their helpful feedbacks.
REFERENCES
Christopher J. Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robert Müller, and Pan Kessel. Fairwashing explanations with off-manifold detergent. In Proc. International Conference on Machine Learning (ICML), pp. 314-323, 2020.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In 2023 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 843-852, 2023.
Georgios Batzolis, Jan Stanezuk, and Carola-Bibiane Schönlieb. Your diffusion model secretly knows the dimension of the data manifold. arXiv preprint arXiv:2212.12611, 2022.
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In Proc. International Conference on Learning Representation (ICLR), 2018.
Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of data science. Cambridge University Press, 2020.
Sebastian Bordt, Uddeshya Upadhyay, Zeynep Akata, and Ulrike von Luxburg. The manifold hy- pothesis for gradient- based explanations. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3696-3701, 2023.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Conditioning method for denoising diffusion probabilistic models. In Proc. IEEE International Conference on Computer Vision (ICCV), pp. 14347-14356, 2021.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Improving diffusion models for inverse prob- lems using manifold constraints. Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 25683-25696, 2022.
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusson posterior sampling for general noisy inverse problems. In Proc. International Conference on Learning Representation (ICLR), 2023a. URL ptttps://openrevieee net/ FForum?/d-ODDyzAAGT0
Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye. Fast diffusion sampler for inverse problems by geometric decomposition. arXiv preprint arXiv:2303.0577 05554, 2023b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. IEFE, 2009.
10

---

Published as a conference paper at ICLR 2024
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. AreFace: Additive angular mar- gin loss for deep face recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4690 4699, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs image synthesis. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 8780-8794, 2021.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Associa- tion, 106(496):1602-16114 2011.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020.
Zalan Fabian, Berk Tinaz, and Mahdi Soltanolkotabi. Adapt and Diffuse: Sample-adaptive recon- struction via latent diffusion models. arXiv preprint wrXiv:2339.066// 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.00 01611,, 2022.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as Plug-and-Play priors. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2022.
William W Hager and Hongchao Zhang. A survey of nonlinear conjugate gradient methods. Pacific Journal of Oplimization, 2:35-58, 2006.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Proc. Ad- vances in Neural Information Processing Systems (NeurIPS), 33:6840-683 2020.
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with
conditional adversarial networks. CVPR, 2017.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Proc. European Conference Computer Vision (ECCV), pp. 694-711.
Springer, 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4401-4410, 2019.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems stochastically. In Proc. Advances Neural Information Processing Systems (NeurIPS), vol- ume 34, pp. 21757-21769, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selec- tion. Annals of statistics, pp. 1302-1335 2000.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: Open-set grounded text-to-image generation. CVPR, 2023,
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. RePaint: Inpainting using denoising diffusion probabilistic models. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11661-11471, 2022.
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. variational perspective on solving inverse problems with diffusion models. arXiv preprint ar X3v:22305.04311, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. Interna- fional Conference on Learning Representations, 2022.

---

Published as a conference paper at ICLR 2024
Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2I-Adapter: Learning adapters to dig out more controllable ability for to-iimage diffusion models. arXiv preprint arXiv:2302.08455. 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proc. International Conference on Machine Learning (ICML), pp. 16784-16804, 2022.
OpenAl. GPT-4 technical report, 2023.
Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun- Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alexandros G Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. XrXiv preprint arXiv:2307.0066 2023.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proc. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 22500-22510, 2023.
Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint trXiv:1505.0085 2015.
Hang Shao, Abhishek Kumar, and Thomas Fletcher. The Riemannian geometry deep generative models. In 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 428 4288, 2018. doi: 10. 109//////2 8.0071..
Jascha Dock--iccstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solv- ing inverse problems with latent diffusion models via hard data consistency. arXiv preprint arXiv:2307.08123. 2023a.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In
Proc. International Conference Learning Representation (ICLR), 2021a. URL https:
//openreview wwwp:rvvieww.ee/ffooooooo
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guidee diffusion models for inverse problems. In Proc. International Conference on Learning Representation (ICLR), 2022.
Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable genera- tion. In Proc. International Conference on Machine Learning (ICML), pp. 2023b.
PMLR,
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proc. Inter- national Conference Learning Representation (ICLR), 2021b.
Peter Sorrenson, Felix Draxler, Armand Rousselot, Sander Hummerich, Lea Zimmerman. and Ull- rich Köthe. Maximum likelihood training of autoencoders. arXiv preprint arXiv:2306. 230.033300.0.8343, 2023.
12

---

Published as a conference paper at ICLR 2024
Suraj Srinivas, Sebastian Bordt, and Hima Lakkaraju. Which models have perceptually--ligner gradients? an explanation via off manifold robustness. arXiv preprint 2tti::2335.99100,, 2023.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learn- ing. In Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. http:///////////poeeeee.....ooaa
www.:/nnnnnneeuuinooo co/paer files/ http:///w.A.aaa.aaa....oooo11a10
Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent opti-
mization improves classifier guidance, 2023.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.00442 2022.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proc. International Conference on Machine Learning (ICML), pp. 681-688, 2011.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. Transactions on Machine Learning Research, 2022.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. FreeDoM: Training-free
energy-guided conditional diffusion model. arXiv:2303.09883, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proc. IEEE International Conference on Computer Vision (ICCV), 2023.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proc. IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 586-595, 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
13

---

Published as a conference paper at ICLR 2024
A RELATED WORKS
Methods that try to address the manifold-related issues Several papers (Chung et al. 2022 2023b) have raised similar issues in the context of solving linear inverse problems using pre-trained diffusion models. In particular, Chung et al. (2023b) attempts to tackle the linear inverse problems by using the conjugate gradient method to maintain the samples solution defines the linear data manifold as Krylov subspace of the linear operator, which limits the applicability linear inverse problems. Moreover, the Krylov subspace usually does not precisely align with the data manifold in many application scenarios and therefore their analysis does not generalize to many practical setting.
linear data manifold. However, this
Methods that require fine-funing of pretrained models Prior to the introduction of the diffusion model, there were some methods that try to finetune Generative Adversarial Networks (GANs) for various tasks such as image-to-image translation (Isola et al. 2017; Zhu et al., 2017). Methods such as ControlNet (Zhang et al. (023) and T21-Adapter (Mou et al. 2023) are known for adding controllability of the model by finetuning pretrained diffusion models, for new conditional settings. Textual Inversion (Gal et al. 2022) DreamBooth (Ruiz et al., (023) also requires finetuning with a small sef of images customize (or personalize) generated images.
Methods that don't require fine-tuning of pretrained models Using pre- pre-trained model to ad-
dress various tasks without additional training is idea shared by many papers including Song
et al. 20021b) SDEdit (Meng et al. 2022) and Repaint (Lugmayr et al. 2022). SNIPS (Kawar
al. 2021), DDRM (Kawar al.] 2022), DDNM (Wang et al.] (022). DPS (Chung al. 2023a)
and IIGDM (Song et al., (2022 have expanded this framework to general linear inverse problems (specificallyy DPS and IIGDM also encompass nonlinear inverse problems) and have broadened their applicabiliiyy Additionally, methods such as PnP (Graikos et al. 2022) and RED-Dif (Mardani et al., 2023) achieve this objective by solving optimization problems that incorporate pre-trained diffusion models. Recently, UGD (Bansal et al., (023), FreeDoM (Yu et al. (023), and LGD (Song et al.) 2023b) have been proposed to increase the range of tasks they can handle by making the design of the loss function more flexible. Attempts have also have been to apply latent diffusion models (LDM) within this framework, UGD and FreeDoM have enabled the use of LDM by incorporat- ing the decoder into the loss function, leveraging the differentiability of the decoder. Moreover, papers (Song et al., 2023a] |Rout et al., [023), Fabian et al.) 2023) focus on the utilization of latent diffusion models and address problems that arises in such situations.
B PROOFS AND THEORETICAL ANALYSIS
B.1 PROOF OF PROPOSITION 1
Proposition [ (Formal, Extended from Chung et al. 2002 2023b ) Define d(z, v, M) ==
inf ||. 11z-vaa vz'lla for for the distribution of noisy data (xt) := plz, (r)(z)dd where p(az(z) = W(Varz, (1 I))
> O, and B(M;T) = E Rd d(z M) < r for >>> O. Consider
f
Then under Assumption 1.1 (x())) is "probabilisticall" concentrated" the (d 1)ddimensional manifold Mr defined as
Me = fo Rdd
d(z, M) (I-
That is, for any < S there an < 6 S which is monotonically decreasing with respect to 8 and (d k) such that
P(ae E BBM
(d) k))) 2 -
Proof. The proof follows Chung et al. 2022 2023b and here we provide an extended version. Without loss of generality, we define M for E R Ia++idd = 0} from the linear subspace manifold
assumption. Let X be x² random variable with degrees of freedom. concentration bound
by Laurent & Massart (2000) implies that for all > 0 we have
P(X - 2 2vT + 27)= < P(X - 15 -2V(7) Se
(10)
14

---

Published as a conference paper at ICLR 2024
Since Σ _i=k+1 /1/ āe) is x² random variable with d k degrees of freedom, by plugging
into (d - kije' we have
22. (d k) $ 2(d (ddd-)((V -))((++())))
IP
2(d - k)ves E
1 att
1=+11
d
(e (rrmmax00.12223..n 14
=P
Σ
E
k=
2 2e (d-k)e'
(11)
where T2 == v(1 - āe )
As a result, for any 0 < d s 1, by setting k).
8 2
60 k
log
+ 26 2 -1)
- V ma (0,1 - 2 2
6 min
(12)
we have an < c I such that
P(x, B(Miie,aAA (1) -
1-
6 is monotonically decreasing with respect to 6 and d since 63, is monotonically decreas- ing with respect to 5 and d and 68,- monotonically increasing with respect to 6
B.2 PROOF OF THEOREM
First, we confirm the following lemmas.
Lemma 1. (Total noise (Chung et al. (023bb) Consider the optimality of the diffusion model, i.e., col solvar2 v1 āect: Ci for E M. For some غ ~ -~NO0,1 I). the of noise components
/1 āt-1 - ( + OrE in DDIM sampling (Equation 1 can expressed
ā 07E0(I2, 020 / ā
(13)
(zt, t) and die; are independent, their sum is the sum of indepen- Proof. Since V1- ū;-1 Eo c0 dent Gaussian random variables. Consequently, the resulting Gaussian distribution has a mean and variance of (1 ã - of) + of az-1).
Lemma 2. Let the data distribution p(r) be probability distribution with support on the lin- ear manifold M that satisfies Assumption 1.1 For any x p(x), consider Iı-1 vāi-1æ + V1 1--aa ō4-1 - 07co t) + O2c4- Then its the marginal distribution pe-1(xx-1). which is defined as
P N(ELi / + -1-- t),
(14) probabilistically concentrated on M,-1 for & ~ N(, 1). pre-trained optimal diffusion model noise estimator co, and ity corresponding variance schedulers ot, a+-
Proof. By Lemma] 1 the multivariate normal distribution in Equation 14 has mean var- jæ and
covariance matrix [ (- Consequently, the marginal distribution of the target be repre- sented as
f N( (1 -
(15)
which is the same the marginal distribution defined in Proposition 1 Therefore, in accordance with Proposition the probability distribution (x;-() probabilistically concentrates on M,
Finally, using both Lemma | and Lemma 2 the following informal statement.
we can prove Theorem the main text, we include
15

---

Published as a conference paper at ICLR 2024
Theorem 1 (Informal) Assume the gradient V soo (zoje y) lies the tangent space TaojeM, M, and the diffusion model co(21, t) is optimal. Then with Assumption 1.1 scalar c2 0 and update rule
Ie-1 ( ce vo (xokc: /1 011-0-1-00 ofc 1) + orct:
6
we can obtain an 2 whose marginal distribution is probabilistically concentrated on M
Here we also include and prove the formal statement below.
Theorem (formal) Let the data distribution p(x) be a probability distribution with support on
the linear manifold M that satisfies Assumption 7.1 and ct >> 0 is scalar function depending on Assume that the gradient V zo [zooee y) lies on the tangent space Tzon for Zojt Vas (It
v1- āica t)). and consider the diffusion model co co(xi,t) t) is optimal. Let
(=)1 / ct V sa [(oops)) V1 0t-1 - cc
(16)
Then for rt-1 ~ (( 1). that is,
It-1 = (244) oeet, 4 ~ ~((;,,
(17)
its marginal probability distribution
pm (x M( ( (ohoolo))ooo
(18)
probabilisticcaly concentrated on Mi-1-
Proof. Firstly, we prove that for all t, there exists E M such that the Ia generated from Equa- tion 17 can also be generated by the forward process of diffusion from x. In other words, It v Gr -I- d for
MM(c,~000, We prove this by induction.
For the base case, let = T. Since we use the same initial noisy sample ør from the Gaussian prior, trivially Fr also be expressed as Varz + vT cTTT for some x p(æ) by construction of the diffusion process. Since the support of p(x) lies on M, this æ is on the data manifold M.
Now suppose for all t 2 Ti, there exists (O 1). Then since the diffusion model is optimal, 6a("T,, = =el c0(vān, + aT, Therefore, FulT; x M.. Then, under the linear manifold hypothesis and considering the gradient V Vaoi L v) lies the tangent space Taoin, M, for any c, 0, can have z' (Fofz, -. VV 201y, L (Zojr; ; v)) is on M, since the tangent space itself coincides with the manifold.
E M such that x; Vāyz VI die for
By Lemma we know that / 11-ai ãπ -1 o7 co( T1) + on en = VI ân- é for some
~~ (0, 1). Therefore, IT:-1 can also be expressed as Vār,-12' + VI -r é for x' E M. Hence complete the proof by induction.
Now that we have proved that for all t, there exists an x E M such that the Fa generated from Equa- tion 17 can also be generated by the forward process of diffusion from x, we can directly apply
Lemma and hve the marginal distribution Pmr-. (IT-1), as obtained by the update rule in Equa-
tion 17
IS probabilisticaly concentrated on Mr-1-
Besides being on the manifold, the generated noisy samples should also reflect the guidance cor- rectly. Here we theoretically verify the quality of the guidance by showing that samples obtained from our new update rule is in the vicinity of the samples obtained from the DPS update rule:
Proposition With the same assumptions and notations as Theorem 1 denote
for certain given Ie. Ep
(MPGD) -oo--1 C2 V zoos [ v)) + V āx-1 o2co + oec. --
to be the updated sample obtained from Equation 6 and
(DPS)
va--12ojt + V āo-1 ofco(Ee.t + OFE P. LZ v)
1-1
16

---

Published as a conference paper at ICLR 2024
be the updated sample obtained from Equation If OL Bea(zee)) || is upper bounded by small
dzzuee (MMPD)) and x upper
positive constant K, then with some Ct > O, the distance between Lt-1 TT-Bt In other words,
bounded by constant Kpr yas
VI- āe V
2/11 S kpt
Proof. By chain rule, we know that
ØL
dep(
(2),
( y)
(1 VI at
Eooie vor
0x.
1 ØL
1
OL Oco(xe.t)
0t Fro Ox.
Vou drof
Vot
VI- āi OL deg(ar, t)
V soje
((
Var
Ezope
dx.
Since
(ea(xs.)) 8i,
is upper bounded by some constant x, we can have
Boott
V at
||
L(zoje= y) V., L(roj, 80l
(19)
Vā
soos
vāa
As result, for ot Vā--da; 0
(PS)
(MPGD)
2 I--1
1l Var-1 + v 8.-1 ofco(xe,t) + +oo V L ((0
Va--i [zoj - a V L + v 1- āt-1 ofeo (z,, t) + ofes
(21,
Vāi-1 Pt V., L (2 1) Vār-1 c V (
14v01-1 V xg L Pc V ((
Pt
|| Vat-iatt 10t
a V slll L (Fojr: V Le
1
Prll
V
Vanie
((zopio)
Ix v)
Vas
VI
ai
S kp:
a
Batzolis al. (022) shows that as decreases, the score, i.e., eg (x (), becomes perpendicu- lar to the clean data manifold practice. As result, if Siooe on the tangent space T (cu(a..))
0 as f decreases. And therefore, empirically the upper bound constant k is very close to 0 when small. Hence, in practice, our method can provide updated samples that reflect similar guidance to the ones from DPS while having marginal distributions that are probabilistically concentrated on the correct manifolds.
Jasoo dez
B.3 PROOF OF THEOREM 2
To prove Theorem 2 examine the following Lemmas 3 and 4
17

---

Published as a conference paper at ICLR 2024
Lemma 3. Let E, D be the encoder and decoder. respectively, of a perfect autoencoder for a data support X' C .M. For any 24 E X, it holds that 2u = (o),, where 20 = E(zo). Then, the
Jacobian 8E of the encoder evaluated at 20 and the Jacobian Uzg of the decoder evaluated at 2o
Ozo satisfy DE aD I, where I is the identity matrix.
8zg B5e
Proof. Given an encoder E and a decoder D. for certain in Z, it holds that 20 = (zo)). For convenience, we denote zo (zz). Differentiating both sides 20 E(D(zo) with respect to we obtain:
a 0D
1
(20)
Ozo Ozo
Lemma 4. With perfect autoencoder and Lemma words, the subspaces spanned by the column vectors of both matrices are identical.
and aD share the same range. other
Ozg
Dzn
Proof. We aim show that the image spaces of aE and are identical. By Lemma Org I. Let the vectors of DE be denoted as e ".. E.E and the column vectors of 0zo as azo UD.1 VD holds that vf UD, and for + j. "D, =
3
Uzo
Ban 3zo
aD
Now, considering any column vector uD of 8D it can be expressed in terms of the row vectors of 050
|io,lll This implies that any element of the subspace spanned by the column Conversely, the same holds true. Therefore, the subspace spanned by the column vectors of coincides with the 02g
8E Bag
as UD Vg X
Tug,illa"
vectors of 8D can be expressed as a linear combination of the row vectors of aE 72g
Ueg
subspace spanned by the row vectors of aE In conclusion, the image spaces of 8mg identical.
aE and 0D are 85o
8mg"
Theorem If an autoencoder with encoder E and decoder D is perfect autoencoder for the support X C M of the data distribution, then V [.((((((oo)))) y) aD aE degi
OL aD aE E &Tz
Proof. As stated in Shao et al. (2018), given the assumption of perfect autoencoder, for any 2g Rk, the Jacobian 879 82p maps the tangent space T2 Z at 2g to the tangent space T of the data
manifold zo D(a)). In other words, the range of the Jacobian 8D lies within the tangent U20
space at zo. Since Z = IR*, its tangent spaces are isomorphic to R Therefore, for any vector
Us IR* the vector aD 7za v. lies in the tangent space at zp. This means that when taking the gradient aD 82p to the gradient, the resulting aE and Dag
of the loss function with respect to and applying the Jacobian
8D 8D T 8L also lies in the tangent space at To. Finally, by Lemma 4 since
vector
Dzp 05e O
aD share the same range, Vz, (D((((((())))) = 8D BE Braie
aL aE (0g T aD OL T)T lies in the Ldzo 820 Aro
dzg
tangent space at z0-
B.4 THEORETICAL ANALYSIS ON MPGD-Z
In this section, we provide the theoretical analysis and proof for algorithm MPGD-Z as proposition Theorem 2
Proposition 3. If an autoencoder with encoder E and decoder D is a perfect autoencoder for X' C. M, then D(V s. L(D(za)): = ( 0L aD E M.
BD Dzg
Proof. Similar to Theorem by Lemma aD (OE) Therefore, V L(D(2)))) =
azo
8L aD 2L aE T E Ta Z C Z. Because we have linear manifold assumption, the updated
3D 05
3D Jmm
Zoje Z0jt + V., L(D(z))) y) is also on Z. Since D is surjective to M, D(2o + V. L(D22))) on the data manifold.
18

---

Published as a conference paper at ICLR 2024
Hence the update rules for MPGD-Z is on-manifold.
Notice that in practice the autoencoder can exhibit reconstruction error. To mitigate this problem, add the inference time reconstruction enror zaje ((E()))) back to the guided clean data estimation after the update. In other words, we use the empirical update rule
20jt = E(za)t)
AToje = Joje (=
20/+ 20lt c v z01e ((((0))))
zojt D(2of() + Arop
This empirical update rule can be viewed as adding a weighted regularization term Asce dool| (G(((E((z)))) to the guidance loss where Ave scalar weight and (G(((E(z))))) is the reconstructed clean data estimation that is fixed before any guidance update (SG here denotes "stop gradient").
B.5 THEORETICAL ANALYSIS ON MPGD-LDM
In this section, we provide the theoretical analysis for algorithm MPGD-LDM with perfect autoen- coder assumption.
Proposition 4. If an autoencoder with encoder E and decoder D is a perfect autoencoder for X C M, then a guided latent diffusion sampling described in Algorithm 2 can generate a sample To E M.
Proof. Since we have perfect autoencoder, the latent space is exactly R As result, the latent diffusion process will not move the latent sample out of the latent space. Because is surjective to the manifold, D22o) is on the data manifold.
C EMPIRICAL VERIFICATION OF THE MANIFOLD PRESERVING ABILITIES
0.755
DPS NPGD we Prot
0.50
NPGD-AE -0.5-
Devatton from the manriio
-0)
0.125
0.100
3.075
0.050
0.000
1.0
0.6
0.4
0.2
0.0
Ditusion Time Stop
Figure 8: We analyze the deviation from the manifold throughout the diffusion process for different methods using the inner products between normalized score and the Jacobians from the guidance loss function.
In Figure which we also include in this section as Figure g we empirically verifies VQGAN's manifold preserving ability by using it as the manifold projection function of MPGD-AE.
We use the diffusion model predicted score as a first-order Taylor series approximation of the log likelihood and calculate the inner product between the normalized score and the normalized Jacobian of the guidance loss as an indicator of how much the guidance deviate the intermediate samples from the original distribution, i.e., off the manifolds.
As a comparison, we also show the inner product curve for the baseline DPS and MPGD without manifold projection. We witness significant deviations in DPS at the beginning of the sampling
19

---

Published as a conference paper at ICLR 2024
process and moderate ones in MPGD the end. When applying VQGAN in diffusion time steps t= 0.5 to t = 0 (denoted as "MPGD-AE (0.5 + 0)" in the plot), we can observe that the manifold projection effectively climinates the deviation the inner products become close to 0.
Empirically, we only apply autoencoder projection for = 0.5 to = 0.3 (denoted as "MPGD-AE (0.5 -> 0.3)") for efficiency purpose, but our method is still able to produce high quality samples that follow the guidance.
D DETAILS ON EXPERIMENTS
D.1 LINEAR CASE
Baselines We employ DPS (Chung et al.) 2023a), LGD-MC (Song et al.) 2023b), and MCG (Chung et al., 2022) as baseline methods. Both DPS and LGD-MC approximate the log likelihood for noisy data that of clean data, which is similar approach. MCG introduces technique to correct intermediate samples from the generative process that deviate from the manifold in linear inverse problem settings.
Experiment Setting and Datasets We evaluate our approach to the super-resolution task and the Gaussian deblurring task. In both experiments, the measurement process is given by y Ax + 2, where A is a known linear operator and 2 is the measurement noise. The objective is to estimate the original data x from the measurement y. We assume that the measurement noise is Gaussian in both cases. The log-likelihood for the clean data can be represented as (x;;y) = ? Ax where is a constant value, which we use as the loss function. More specifically, for the super-resolution task, the linear operator consists of a bicubic downsampling operator, which downsamples 256 x 256 images 64 64. The variance of measurement noise is 0.05². For Gaussian deblurring, the linear operator is convolution operator with 61 x 61 Gaussian blur kernel with an intensity value of 3.0. The variance of the measurement noise is also set to 0.05².
We evaluate approach using the FFHQ 256 x 256 (Karras et al. 2019) and ImageNet 256 x 256 (Deng et al. 2009) datasets. For FFHQ, we utilize a pretrained model from Choi et al. 2021) [ and for ImageNet, we employ a pretrained model from Dhariwal fhrriwa including the proposed method, the same pre-trained models are used for each dataset.
& Nichol 2021 For all methods,
For all methods, the number of DDIM steps is tested in three cases: [20, 50, 100]. The parameter set to 0.5. The weight parameter scheduling is based on the implementation of DPS. The guidance weight hyperparameters for all of MPGD w/o proj.. MPGD-AE, and MPGD-Z are 20,10,5 for DDIM steps 20, 50, 100 respectively. The weights for DPS 0.3 as their default, for MCG 100.0 and for LGD is 0.05 for the best empirical results we obtain. We follow the super-resolution setting in the LGD paper for its additional weight scheduling. The number of Monte Carlo samples for LGD is set to 10.
Evaluation Metries For each dataset, we perform inference using 1000 images from the test set. As evaluation metrics, we use the Kernel Inception distance (KID) (Bnnkowski al., 20188) to asses the fidelity, Learned Perceptual Image Patch distance (LPIPS) (Zhang et al. 201 8) to evaluate guid- ance quality, and the inference time to test efficiency of the method. For linear cases, all the ex- periments are conducted on single NVIDIA GeForce RTX 2080 Ti GPU. The inference time is measured by averaging the time taken to generate 20 images. During this evaluation, the batch size is set to 1.
D.2 NONLINEAR CASE: DATA DOMAIN DIFFUSION MODELS
Baselines As baselines that can solve general tasks using pretrained models, FreeDoMM and LGD- MC are compared. FreeDoM and LGD have similar ideas to DPS, using a loss for clean data to approximate the log-likeli for noisy data.
20

---

Published as a conference paper at ICLR 2024
Experiment setting and Datasets The objective of FacelD-guider face image generation is to generate facial images that resemble reference faces. As with FreeDoM, we use a pretrained human face recognition network (Deng al., 2019) to extract facial features. Specifically, we calculate the &z distance between the facial features extracted from the zoj and those from the reference face image.
We test all methods with the pretrained diffusion model for the CelebA-HQ 256 x 256 dataset provided by Yu et al. (2023 and 50 DDIM steps. All the samples are generated
single NVIDIA RTX3090 GPU. v is set to 0.5. The weight parameter scheduling is based on the implementation of FreeDoM. We set the guidane weights to the value 0.015, 0.015, 0.015, 100, and 50, for MPGD w/o proj., MPGD-AE, MPGD-Z, FreeDoM, and LGD, respectively. The number of Monte Carlo for L.GD set to 3, and the Monte Carlo parameter Tr is set to 0.1 v at--
Evaluation Metrics We generate 1000 facial images using the CelebA-HQ test set as reference images and evaluate the results using KID and FacelD Loss. The inference time measured by averaging the time taken to generate 10 images. During the inference, the batch size is set to 1.
D.3 NONLINEAR CASE: LATENT DIFFUSION MODELS
Baselines As baselines, we compare the proposed method with FreeDoM and LGD, similar to the case of data domain diffusion models.
Experiment setting and Datasets We have the evaluation the text-to- style guided gener- ation task, where the goal is to generate images that fit both the text input prompts and the style of the reference images. As the pretrained diffusion model, we use the stable Di fufuuiin pusionnvi con-v-1 checkpoint Rombach et al. 22021) The loss function involves calculating the Gram matrices (John- son et al., 2016) of the intermediate layers of the CLIP image encoder for both the generated images and the reference style images, then using their Frobenius norms as the objective. More specifically, for a reference style image Jret and a decoded image (2 from the estimated clean latent variable Zoje, we compute the Gram matrices (ziet)) and G ( 1)B; corresponding to the features of the j-th layer of the image encoder. The loss function is then calculated as follows:
[22 v5 Teer) [G - G(D( 11:
(21)
where || } denotes the Frobenius norm of matrix. We adopt the third layer's features, consistent with the configuration used in FreeDoM. All the samples are generated on a single NVIDIA A100 GPU with 100 DDIM steps. n set to 1.0. The weight parameter scheduling is based the implementation of FreeDoM. We set the parameter to the values of 17.5, 0.2, and 15.0 for MPGD- LDM, FreeDoM, and LGD, respectively. Additionally, configure the classifier-free guidance scale parameter to the value of 7.5, 5.0, and 5.0 for MPGD-LDM, FreeDoM, and LGD, respectively.
We use Style Score and CLIP score for evaluation. For reference style images and text prompts, we randomly created 1000 conditioning pairs, using images from WikiArt Saleh & Elgammal 2015 5 and prompts from PartiPrompts Yu et al. (2022) dataset. The inference time measured by averaging the time taken to generate 5 images. During the inference, the batch size is set to 1.
Evaluation Metrics
E ADDITIONAL RESULTS
E. CLIP GUIDED GENERATION WITH PIXEL-SPACE CELEBA-HEE HQ MODEL
To further demonstrate the applicability of our method, we conduct another experiment where we use pre-trained CLIP model and text prompt to guide the generation of human faces using the pixel-space CelebA-HQ model, which the same model used the FacelD) guided generation experiment. We use the f Euclidean distance between the provided text prompts and the images
wtw.lle-dfffffooooooooooo ffusion-v- 1------i-----iiaa
https://uww.wkkiarttt
21

---

Published as a conference paper at ICLR 2024
"o Aeadsnot
heodsmot mon"
to headstio of perion
Figure 9: CLIP guided generation with Pixel-space CelebA-HQ Model
NPGD We Proj
NPGD wia Proj.
NEGD-ME
MPGD Proj
NPGD-H
NFGGD
12
NPGD
DPS
MPGD-Z
DPS
LGD MCC
LGD MC
-3-
MCG
MCG
MCG DINNH
DONN
Time
20
60
60 Steps
200
Stops
Figure 10: Quantitative comparison between our methods, DDNM and other baselines.
as the guidance loss. We also sample all images with 50 DDIM steps with 1j 1.0. Images with prompt "a headshot of person with blond hair" and "a headshot of man" are generated with MPGD-Z, and images with prompt "a headshot of a person wearing red lipstick" is generated with MPGD-AF. For other hyper-parameters such as p and time traveling steps, different prompts require different choices, which we have detailed discussions in later sections. In general, we find E [1, 3] and less than 10 steps of time traveling for only subset of diffusion step (similar FreeDoM) to work well.
In Figure we exhibit examples of samples guided by the text prompt, compared with unconditional DDIM samples generated from identical random seeds. Our method is able to create images that follow the text description provided while maintaining high fidelity.
E.2 COMPARISON WITH DDNM
In this section, we compare our method with one of the state-of-the-aat diffusion based inverse problem solver, Denoising Diffusion Null-space Model, DDNM (Wang al., 2022) the super- resolution task on the FFHQ dataset. Before we dive into the discussion about the experiment, we would like emphasize that DDNM is designed for only solving linear inverse problems, and it requires direct access to the operation matrix, its pseudo-inverse/SVD and the noise scale for the measurement, which not parts of the assumptions that we have in our problem setting. As result, DDNM is not applicable to the general setting of paper. Nevertheless, we also think it is valuable better position our paper the literature of linear inverse problem solving, and therefore we conducted the experiments described below.
Table 3: PSNR comparison between DDNM and MPGD-Z.
PSNR
Method
DDIM Step = 20
DDIM Step =
DDIM Step = 100
DDNM
27.53
29.38
29.47
MPGD-Z
25.40
25.40
24.97
22

---

Published as a conference paper at ICLR 2024
Measurement
Ground Truth
MPGD wio Proj. (Ours)
MPGD-AE (Ours)
MPGD-Z (Ours)
DDNM (Beseline)
DDIM Steps 20
Noisy Super. Super-Resal Resolution (X4, a = 0.05)
DDIM Steps 50
DDIM Steps 100
Figure Qualitative comparison between our methods, DDNM and other baselines.
Ground Truth
MPGD w/o Proj. (Ours)
MPGD-AF (Ours)
DDNM (Baseline)
DDIM Steps = 20
Noisy Super-Resolution (X4, a = 0.05)
DDIM Steps = 50
DDIM Steps = 100
Figure 12: Detailed enlargement of the generated images produced by our methods in comparison with the ones produced by DDNM.
23

---

Published as a conference paper at ICLR 2024
MPGD (Ours) w/ Mulli-Step Time Traveling
Prompt
DDIM (Unconditional)
Step
2 Steps
Stops
"a headshat of a man"
"a headshot of person weoring red lipstick"
Figure 13: Qualitative showcase of the effect of different number of time traveling steps used in -ulP-guidence pixel-space CelebA -HQ model generation.
To make a fair comparison, we use the simplified version of DDNM with no time traveling and the same unconditional diffusion model pre-trained by the authors of DPS, which is a smaller model compared to the one DDNM used in their paper. Due to code availability, we use average pooling as the interpolation method, which is different from our original experiment setting.
Figures II I2 and 10 illustrate the qualitative results, detailed enlargements, and quantitative outcomes, respectively. As we observe in the figures, the simplified version of DDNM achieves an equally fast sampling speed compared to our method and obtains similar guidance quality. However, the images generated from DDNM exhibit various artifacts, such as high-frequency circular patterns and overly smooth generation. These artifacts prevent DDNM from maintaining high fidelity while our method can generate more realistic details.
Despite these findings, it is worth noting that the images generated from DDNM tend to maintain better shapes, whereas our method hallucinates small details more than DDNM. Consequently, re- garding Peak Signal-to-note (Table 3. We also acknowledge that the original paper DDNM, in order to solve the noisy inverse problems, the authors suggest using multi-step time traveling to improve the performance, which we did not deploy in order to make a fair comparison in terms of run time. Therefore, DDNM still has certain advantages over our method in terms of solving specific inverse problems. We be- lieve that integrating the consistency constraint from DDNM with our approach could potentially strengthen both methods, presenting promising direction for future research.
ratio (PSNR) values, DDNM significantly outperforms our approach
E.3 IMPACT OF THE NUMBER OF OPTIMIZATION STEPS
We explore the impact of varying optimization steps on generated samples. To illustrate, first consider the time-traveling algorithm as performing more Langevin steps in one step, thereby in- creasing the likelihood of reaching optimal solution in terms of loss, especially when these optima are significantly from the starting point.
For instance, in the task of sampling "a headshot of man" using CLIP guidance and the pixel space CelebA-HQ model, if our initial unconditional DDIM sample produces headshot of a woman, im- plementing multi-step optimization process proves be more advantageous. This is because the samples aligning more closely with the prompt are likely to be farther from the original uncondi- tional sample. Conversely, for tasks that require only minor modifications, such as generating "a headshot of a person wearing red lipstick," achieving high quality is possible with fewer optimiza- tion steps. The generated images with various number of steps are provided in Figure [3] Overall, results suggest that the more challenging the task (i.c.. the greater the deviation required to reach the desired result in expectation), the more beneficial multi-step optimization becomes.
That being said, we do observe in practice that large number of steps does not always benefit the generation. For example, we can start to observe unnatural artifacts appearing in the background of the image when sampling with 7 and 15 steps in the "red lipstick" experiment. In fact, hypoth- esize that with a step size that is not infinitesimally small, asymptotically infinite-step optimization may lead to significant deviation from the data distribution.
24

---

Published as a conference paper at ICLR 2024
Additionally, explore the use of various optimization algorithms, such nonlinear conjugate gradient, and the application multi-step optimization to select subsets of steps in line with the FreeDoM framework. We also observe that step sizes need to be adjusted according the number of steps used. The asymptotic behavior of the multi-step optimization and selecting appropriate hyper-parameters for these variations are promising areas for future research.
E.4 INFLUENCE OF THE CLASSIFIER- FIREIE GUIDANCE SCALE STYLE GUIDANCE GENERATION EXPERIMENT
Table 4: Influence of the classifier-free guidance (CFG) scale on the style score and the CLIP score in MPGD-LDM style guided generation experiment.
CFG Scale
Style (L)
CLIP (1)
2.5
493.5
26,98
5.0
459.8
27.08
7.5
441.0
26.61
We expand our analysis include quantitative comparison of style-guided Stable Diffusion gen- eration with various classifier- free guidance (CFG) scales. The parameter p is set to 17.5, while the CFG scale is selected from the set [2.5, 5.0, 7.5]. Table 4 shows the influence the CFG scale on the style score and the CLIP score.
It's worth noting that the CFG scale has a positive impact on the style score, and strong CFG scales appear to help decrease the loss function. However, although in vanilla text-to-image generation tasks larger CFG scale tends to lead to a higher CLIP score (Nichol et al. 2022p, since the distri- bution aim sample from is also guided by another external loss function, the same trend in our setting. In practice, we would suggest our users to adjust this hyperparameter suit their preference of tradeoff between the style guidance and the text prompt condition.
do not observe
E.5 ADDITIONAL QUALITATIVE RESULTS IN THE EXPERIMENTS
We provide diverse additional qualitative results in Figure
66
E.6 USER STUDY
Table 5: User study results on style guidance Stable Diffusion generation task. "Style", "Text" and "Overall" represent style consistency, text prompt consistency and overal user preference. "(W/L/D)" represents the win/lose/draw ratios.
Method
Style (W/L/D)
Text (W/L/D) Overall (W/L/D)
MPGD-LDM v.s. FreeDoM 47%
32%/66%/2% 27%/64%/9% 69%/29%/2%%
49%.44%%/6% 53%/44%%/4%
MPGD-LDM v.s. LGD-MC
We conduct user study for the style-guided Stable Diffusion generation task on Amazon Mechan- ical Turk to compare our method (MPGD-LDM) and two baselines (FreeDoM and LGD-MC). The user study consists of three parts: assessing the style consistency between the generated image and the reference style image, evaluating how well the generated image follows the text prompt, and the overall user preference.
We perform each part of this study with separate questionnaire posted on Amazon Mechanical Turk. Each HIT task contains one multiple choice question. Example surveys are provided in 22 We generate 100 images from each method using randomly sampled WikiArt-ParriPromppt image- caption pair we create from the style guidance experiment. Each annotator is compensated with 12. USD for each HIT task, and we estimate the annotators to complete each HIT in 30 seconds to minute, which yields hourly earning rate of $7.2 to $14.4 USD.
The results are shown in the Table 5 Users' response regarding Style and Text generally align with the style score and CLIP score. Our method outperformed both methods in terms of overall user preference, which suggests that our method finds a better sweet spot balancing the text prompt condition and the style guidance.
25

---

Published as a conference paper at ICLR 2024
(Ouza)
(Baidino)
DDM Stpps 50 DOMM 20
-006)
Nonoy Supo (344,
DRIM Suppe (00)
DOIM 30 DO0M
Naioy Guzasia Debku (F 0.03)
DOOM Shops 100
Figure 14: dataset.
Additional qualitative examples of solving noisy linear inverse problems on FFHQ
(Our) MPGD-AF
MIGID- (Baseline)
(Boseline)
DDM Stego 20
Nevay Sesppottttttttte (34, 6.85)
IDMM Shepe 30
DIMM Stepe 100
DOMM Stego 20
Nevyy Grovttn 0.05)
DIIN Strpe 20
DUIMSSpe 100
Figure 15: Additional qualitative examples of solving noisy linear inverse problems on ImageNet dataset.
26

---

Published as a conference paper at ICLR 2024
Iepui
DDSI (Uhrunitional
Proj (Oue)
MEGD-AE (Qun)
MPGD-Z (Oun)
(Beding)
(Beeclnn))
Figure 16: Additional qualitative examples of facelD guidance experiment.
Input Reference
MPGD (Ours)
Figure 17:
Additional qualitative examples faceID guidance experiment to showcase the diversity of our generated images. With the same input reference image, method is able to generate human face images that consist of the same identity as the reference image and that are diverse in many aspects including color, style and facial expressions.
27

---

Published as a conference paper at ICLR 2024
MPGD-LDM w/ Time Traveling
MPGD-LDM Conjugate
Prompt
Input Reference
MPGD-LDM
Gradicat Method
"a headshot of girl with blond hair and space background"
t
Figure 18: Results of FacelD guidance generation with Stable Diffusion and different optimization algorithms. In particular, non-linear conjugate gradient method improves the guidance quality while maintaining the fidelity, suggesting a promising direction for future investigation.
Text prompt: "a wearing glasses"
Input Reference
MPGD-LDM
w/o time-traveling (Ours)
FreeDoM
w/o time-traveling
(Baselinc)
FreeDoM
w/ time-traveling
(Baseline)
Figure 19: Additional qualitative examples of style guidance Stable Diffusion generation.
F LIMITATIONS
F.1 FAILURE CASES OF PIXEL-SPACE DIFFUSION GUIDANCE
In this section, we investigate the limitations and the failure modes of our proposed methods.
We observe common failures in solving noisy linear inverse problems with small numbers of DDIM steps. When the background of the image is predominantly white, prominent Gaussian noise- like patterns remain in the final results. Figure 23 shows an example of this kind of failure. While manifold projection can help mitigate the problem, we find that with small number DDIM steps. it is usually not enough completely reduce the Gaussian noise.
We also discover that the quality of the guided generation heavily depends on the performance of the guidance loss. If the loss function is not properly chosen, then it is very difficult to obtain satifactory results. For example, the ArcFace model is trained on centered and cropped humean headshot images that only recognizes the identity of person by their facial landmark features. As a result, is very hard to guide the sample to have other features such as skin tones that the model doesn't 't detect. However, these features are erucial for identifying individual well. Therefore, we advise our users to cautiously select the guidance loss functions.
F.2 FAILURE CASES OF STYLE GUIDANCE GENERATION EXPERIMENT
In Figure 24 we present some notable failure cases encountered during the style guidance generation experiments with Stable Diffusion.
28

---

Published as a conference paper at ICLR 2024
Reference image
"A cute dog with cakes"
"A cute hedgehog curled up in cozy den"
"A playful panda splashing in a river"
"An owl reading a book at night"
"A wolf howling at the moon"
Figure 20: Additional qualitative examples of style guidance Stable Diffusion generation.
1. Photo-realistic Outcomes from Painting References. (Fig. 24(a)) In this instance, despite the reference image being realistic painting, the generated image resembles photograph. This may be attributed to the inability of the loss function, used in this case, to effectively differentiate between realistic painting and actual photograph.
2. Inadequate Reflection of Simplistic Reference Styles (Fig. b)) In this example, the reference image is a monochrome line drawing. However, the generated image, while partially capturing the color scheme, fail to replicate the style of the reference.
3. Complex Prompts Leading to Incomplete Representation. (4) The prompt in thie example "a corgi's head depicted simple text-to-image generation), the "explosion of nebula" aspect is evident in the generated image. However, the result with the style guide, the aspect not represented, likely due lack of correlation between the specified aspect from the prompt and the provided style.
explosion of nebula." Without style guide (i.e., in
29

---

Published as a conference paper at ICLR 2024
Input Reference
MPGD-LDM (Ours)
FreeDoM (Baseline)
LGD-MC (Baseline)
DDIM (Unconditional)
Prompt
horse in forest"
aa chimpanzee"
group of penguins snowstorm
family four pasing front house"
horse"
key"
aerial of the Great Pyramid
Figure 21: Additional qualitative examples of the style guidance experiment.
30

---

Published as a conference paper at ICLR 2024
care magin
Coptien: "hhe offed tover
cocsad pertoy the ne pike and saree style as vefrence
sirvlle style the refermnce bebova?
captio
oandono inugo abows
Carideiide
Image
desoriee the caction and hows sorvo inde the infeeece pailt
Whech carciibee paitt parning portraye
painting?
Remember the painting nende crtdyy BOTH repacts
Can' doode
Paintig
Paining B
Figure 22: Example questionnaires in the user study.
Figure 23: An example of the failure cases of method. The Gaussian noises from the measure- ment remains in our reconstruction.
31

---

Published as a conference paper at ICLR 2024
DDIM
Input reference
MPGD-LDM
(w/o guidance)
(Ours)
(a)
"a kachina doll standing in sand"
(b) "A trash bin"
(c)
"a corgi's head depicted as an explosion nebula"
Figure 24: Noteworthy failure cases from images generated in the style guidance generation task with Stable Diffusion.
32


Reviews:
[{"review_id": "YthDf2lHpw", "reviewer": "Reviewer_Yt4g", "summary": "The paper applies the manifold hypothesis to enhance the control in unconditional diffusion models. One key finding from the paper is that the guidance term in the diffusion process can potentially have a detrimental effect on the evaluation of the score function. In other words, it can cause deviations from the data manifold and lead to the generation of unrealistic images. To address this issue, the paper suggests a solution where the guidance gradient of clean samples is projected onto the tangent space of the manifold using an autoencoder, allowing the model to stay closer to the manifold.", "questions": "- What is the method's guidance consistency in comparison to DDIM?\n- Is there any behavior change in the classifier-free guidance scale for text in the style guided experiments?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper presents a versatile idea that can be integrated with existing methods. This approach offers a balance between efficiency and quality, showcasing lower overhead compared to baselines, while also demonstrating improved quality and faster inference speed.\n- The paper provides valuable insights, and these insights are effectively supported by experiments, as in Section 3.\n- The development of the method in Section 4 is presented in a clear and accessible manner. The logical flow and explanations make it easy for readers to follow the methodology.\n- The paper demonstrates thoroughness in its presentation of baselines for experiments in both the pixel space and latent space. The dataset choices are reasonable, coupled with compelling figures and metrics, enhances the persuasiveness of the results.", "weaknesses": "I don't see major weaknesses in the paper, it is cohesive and well-rounded.\n\n- While the paper appropriately utilizes KID as a less biased metric, it could benefit from including FID (Fréchet Inception Distance) as a more widely recognized and standardized metric for related work comparison. The absence of FID makes it challenging for readers to assess the method's performance in relation to existing or future approaches.\n- The paper misses the opportunity to provide insight into failure cases when it comes to text-guided generation. This is particularly important given its widespread use in the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ke4wlF2SF4", "reviewer": "Reviewer_obSn", "summary": "The paper introduces a novel way of generating conditional samples from unconditional diffusion models without additional fine-tuning. The method is based on modifying the DDIM sampling algorithm by performing gradient steps on the $x_0$ estimate with an off-the-shelf \"clean-image\" classifier.", "questions": "- Could the authors clarify what the $L$ function that is used in the different experiments is?\n\n- Can the authors showcase their method on an experiment that is not based on sampling with a reference image? Is this a limitation of their method or can it be generalized to other formulations of $L$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose a novel and flexible way to utilize the priors of large diffusion models in conditional generation tasks. The main advantage is the ability to use any off-the-shelf classifier to provide guidance, which previously relied on expensive, slow procedures that required multiple optimization steps during sampling.", "weaknesses": "- The experiments do not clearly demonstrate the idea of guiding the sampling process with an off-the-shelf network. In all three experiments, a reference image is used to guide the sampling. The projection to the manifold in these cases is shown to successfully guide the diffusion towards an image in the reference image neighborhood.  However, it is not clear how that would work in the case where the guidance is given by a more general differentiable loss $L$. This raises the question of whether the proposed method is applicable in cases other than the limited set of inverse problems presented, where the guidance may be less informative. An example would be an $L$ that measures the log-likelihood of an attribute in the image (e.g. blonde hair). \n\n- The qualitative results in Figure 6 raise questions about the diversity of the samples drawn. The generated images shown, are almost all identical and deviate from the distribution of images in the original CelebA dataset (i.e. the colors are completely unnatural). The baseline also performs badly but the distribution seems closer to the original $p(x)$ that the model learned.\n\n- The related works section should be in the main paper. This is necessary to draw clear distinctions between the existing works and the proposed approach and highlight what the authors' main contributions are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PTHWcRS7k7", "reviewer": "Reviewer_mY9K", "summary": "This paper proposed a guided diffusion model based on manifold preserving. The basic idea is to insert the manifold constraint into the pre-trained diffusion model, by gradient-descent projection of the estimated clean image onto the manifold tangent space. The manifold constraint is implemented by the gradient descent of auto-encoder w.r.t. the estimated clean image. The paper presented theorems for the manifold preserving. The experiments mainly show that the proposed approach achieves high-quality generated image while having 3.8 times speed-up.", "questions": "Please refer to the weakness for my questions, mainly on the insufficient experimental justifications.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The idea of imposing data manifold on the diffusion model using auto-encoder is an interesting idea.\n\n(2) The paper presented sufficient proof for the manifold constraints.\n\n(3) The generated images are overall good, especially with higher generation speed.", "weaknesses": "1. The paper claimed that the proposed MPGD can generate images with 3.8 times speed up compared with the baseline. The main paper should give more details on why it can speed up with the same number of iterations?\n\n2.  The theoretical analysis on the auto-encoder in theorem 2 says that the gradient of auto-encoder lies on the tangent space of manifold. The analysis relies on the linear manifold hypothesis. Is this assumption realistic for real image set, and how to understand that the image manifold is linear?\n\n3.  In the experiments for super-resolution, there are several methods using diffusion model for image super-resolution or other restoration methods, e.g., Denoising diffusion null-space model, etc. How about the comparisons with these Sota diffusion-based image inverse models?\n\n4. The paper mentioned multi-step optimizations in the manifold projection. How does the number of iterations affect the conditional generation results in quality measures?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DqHOkNqjdT", "reviewer": "Reviewer_smw4", "summary": "The paper introduces Manifold Preserving Guided Diffusion(MPGD), a framework that leverages the manifold hypothesis to enhance conditional generation tasks. By guiding the diffusion process using the underlying manifold structure, it achieves better sample quality and faster inference. The authors propose novel training-free methods applicable to both pixel and latent space models, demonstrating improved sample quality and inference speed.", "questions": "Please address my concerns or correct me if there is anything wrong in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem that traditional training-free guidance fails to guarantee the interfered results located in the manifold is quite interesting. This seems to exist in a wide range of existing approaches. Beisides, the proposed method is training-free, simple, efficient and easy to be implemented.", "weaknesses": "- Although the problem to be addressed looks quite fundamental and the presented approach is quite general, the experiments look quite limited. It only contains a restoration task, a face-id transferring task and a style guidance task, which is quite narrow considering that the proposed method is applicable to a lot of editing tasks. Besides, the paper repetitively uses several cases in both the manuscript and the appendix, which looks a bit boring. \n\n- The results of style guidance look not that appealing. \n\n- If possible, user study would make the experimental part stronger. \n\n- There are some typos, e.g., in Eq. 2 (\\alpha_t --> \\alpha_{t-1}) and Fig. 14 (bitterfly --> butterfly).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)