Universal Guidance for Diffusion Models
Authors: Arpit Bansal
Hong-Min Chu
Avi Schwarzschild
Roni Sengupta
Micah Goldblum
Jonas Geiping
Tom Goldstein
Venue: ICLR 2024 poster (2024)
Keywords: Generative Models
Computer Vision
Diffusion Models
Primary area: generative models
TL;DR: We propose universal guidance, an algorithm that achieves conditional generation with any base diffusion model and guidance functions without any retraining.

Abstract: Typical diffusion models are trained to accept a particular form of conditioning, most commonly text, and cannot be conditioned on other modalities without retraining. In this work, we propose a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain any use-specific components. We show that our algorithm successfully generates quality images with guidance functions including segmentation, face recognition, object detection, style guidance and classifier signals.

Published as a conference paper at ICLR 2024
UNIVERSAL GUIDANCE FOR DIFFUSION MODELS
Arpit 8
Hong-Min Chu University of Maryland
Avi Schwarzschild University of Maryland
University of Maryland bansal01 0 umd. edu
Soumyadip Sengupta University of North Carolina
Micah Goldblum New York University
Jonas Geiping
University of Maryland
Tom Goldstein University of Maryland
ABSTRACT
Typical diffusion models are trained to accept particular form of conditioning, most commonly text, and cannot conditioned on other modalities without re- training. In this work, propose universal guidance algorithm that enables diffusion models to controlled by arbitrary guidance modalities without the need to retrain any use-specific components. We show that algorithm success- fully generates quality images with guidance functions including segmentation, face recognition, object detection, and classifier signals. Code is available hittllocc/mpiibaass2//////ieaead
Target Segmembrtion Map
Tampet Identity
Walker hound, W alker foxhound space
headshot of blonde woman a sketch
Target Object Locatia
Tarpet Style Image
A headshot a woman with a dog in winter.
Portrait of woman
Figure I: Diffusion guided by off-the-shelf networks. Top left: segmentation guidance, top-right: face recognition guidance, bottom-left: object detection guidance, bottom-right: style-transfer.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Diffusion models are powerful tools for creating digital art and graphics. Much of their success stems from our ability to carefully control their outputs, customizing results for each user's individual needs. Most models today are controlled through conditioning. With conditioning, the diffusion model is built from the ground up to accept a particular modality of input from the user, be it descriptive text, segmentation maps, class labels, etc. While conditioning is a powerful tool, it results in models that are handcuffed to a single conditioning modality. If another modality is needed, a new model has to be trained, often from scratch. Unfortunately, the high cost of training makes this prohibitive for most users.
A more flexible approach controlling model outputs is use guidance. In this approach, the diffu- sion model acts as generic image generator, and is not required to understand a user's instructions. The user pairs this model with guidance function that measures whether some criterion has been met. For example, one could guide the model to minimize the CLIP score between the generated image and a text description of the user's choice. During each iteration of image creation, the iterates are nudged down the gradient of the guidance function, causing the final generated image to satisfy the user's 's criterion.
In this paper, we study guidance methods that enable any off-the-shelf model or loss function to be used as guidance for diffusion. Because guidance functions can be used without re-training or modification, this form of guidance is universal in that it enables a diffusion model to be adapted for nearly any purpose.
From a user perspective, guidance is superior to conditioning, as single diffusion network is treated like foundational model that provides universal coverage across many use cases, both commonplace and bespoke. Unfortunately, it is widely believed that this approach is infeasible. While early diffusion models relied on classifier guidance (Dhariwal & Nichol 2021). the community quickly turned to classifier-free free schemes (Ho & Salimans 2022) that require model be trained from scratch on class labels with a particular frozen ontology that cannot be changed (Nichol et al. 2021
Rombach al. 2022 Bansal al. 2022).
The difficulty of using guidance stems from the domain shift between the noisy images used by the diffusion sampling process and the clean images which the guidance models trained. When this gap is closed, guidance can be performed successfully. For example, Nichol er al.] (202T) successfully use a CLIP model as guidance, but only after re-training CLIP from scratch using noisy inputs. Noisy retraining closes the domain gap, but at very high financial and engineering cost. To avoid the additional cost, we study methods for closing this gap by changing the sampling scheme, rather than the model.
To this end, our contributions are summarized follows:
We propose an algorithm that enables universal guidance for diffusion models. Our proposed sampler evaluates the guidance models only on denoised images, rather than noisy latent states. By doing so, we close the domain gap that has plagued standard guidance methods. This strategy provides the end-user with the flexibility work with wide range of guidance modalities and even multiple modalities simultaneously. The underlying diffusion model remains fixed and fine-tuning of any kind is necessary.
We demonstrate the effectiveness of our approach for variety of different constraints such as classifier labels, human identities, segmentation maps, annotations from object detectors, and constraints arising from inverse linear problems.
BACKGROUND
We first briefly review the recent literature on the core framework behind diffusion models. Then, we define the problem setting of controlled image generation and discuss previous related works.
2

---

Published as a conference paper at ICLR 2024
2.1 DIFFUSION MODELS
Diffusion models are strong generative models that proved powerful even when first introduced for image generation (Song & Brmon 2019 Ho al. 2020). The approach has been successfully
extended to a number of domains, such as audio and text generation (Kong et al. 2020 Huang et al.
2022 Austin et al. 2021 Li et al. 2022).
We introduce (unconditional) diffusion formally, as it is helpful in describing the nuances of different types of models. A diffusion model is defined combination of '''step forward process and Ttstep reverse process. Conceptually, the forward process gradually adds Gaussian noise of different magnitudes to a clean data point Sy. while the reverse process attempts gradually denoise a noisy input in hopes of recovering a clean data point. More concretely, given an array of scalars representing noise scales {ar ) and an initial, clean data point zo, applying steps of the forward process to 20 yields noisy data point
24 20 + (v1 - az)E, 10,
diffusion model is a learned denoising network eg. It is trained so that for any pair (zo, t) and any sample of e,
c0(2,.))= === 2t va
vI at
The reverse process takes the form Q(2-1/26 20) with various detail definitions, where g(-l.) is gener- ally parameterized Gaussian distribution. Different works also studied different approximations of the unknown q(=t11 lt; zn) used to perform sampling. For example, denoising diffusion implicit model (DDIM) (Song al., 2021a) first computed a predicted clean data point
2 (VI - co)co(2e.>> (2),
20
Vas
and sample R from q 2n) by replacing unknown 2o with Eg. On the other hand, while the details of individual sampling methods vary, all sampling methods produce 2 based current sample 2t, current time step and a predicted noise ē. To case the notation burden, we define a function S(,,,-) an abstraction of the sampling method, where - = &t
2.2 CONTROLLED IMAGE GENERATION
In this paper, we focus on controlled image generation with various constraints. Consider dif- ferentiable guidance function f. for example a CLIP feature extractor or segmentation network. When applied to an image, we obtain a vector measures the closeness of two vectors and c'. Given a particular choice of c, which we call a prompt, the corresponding constraint (based on l, and f) is formalized as f(c, f(=)) x O, and we aim to generate a sample z from the image distribution satisfying the constraint. In plain words, we want to generate in-distribution image that matches the prompt.
fa)) We also consider function ((::-) that
Prior work on controlled generative diffusion falls into two main categories. We refer to the first category conditional image generation, and the second category guided image generation. Next, we discuss the characteristics of each category and better situate our work among existing methods.
Conditional Image Generation. Methods from this category require training new diffusion models that accept the prompt as an additional input (Ho & Salimans/ 2022) Bansal et al. 2022) Nichol
al. 202T Whang et al., 2022 Wang et al. 2022al Li et al.) 2023 Zhang Agrawala (202)) 3). For
example, Ho & Salimans (2022) proposed classifier-free guidance using class labels as prompts, and trained diffusion model by linear interpolation between unconditional and conditional outputs of the denoising networks. |ansall et al. (2022) studied the case where the guidance function is known linear degradation operator, and trained a conditional model to solve linear inverse problems. Nichol et al. 2021) further extended classifier-free guidance to text-conditional image generation with descriptive phrases as prompts, and trained a diffusion model to enforce the similarity between the CLIP (Radford et al. (021) representations of the generated images and the text prompts. These methods are successful across different types of constraints, however the requirement to retrain the diffusion model makes them computationally intensive.
3

---

Published as a conference paper at ICLR 2024
Guided Image Generation. Works in this category employed frozen pre-trained diffusion model as foundation model, but modify the sampling method to guide the image generation with feedback from the guidance function. Our method falls into this category. Prior work that studied guided image
generation did so with a variety of restrictions and external guidance functions (Dhariwal & Nichol
2021 Kawar al. [022]] Wann et al. 2022b Chung et al. [022a Lugmayr al. [022]] Chung
et al. 2022b;/Graiko Graikos et al. (022). For example, Dhariwal & Nichol (222)) proposed classifier
guidance, where they trained a classifier on images of different noise scales the guidance function
f. and included gradients of the classifier during the sampling process. However, a classifier for noisy images is domain-speciiii
and generally not readily available an issue our method circumvents. Wang et al.: (2022b) assumed the external guidance functions to be linear operators, and generated the component of images residing in the null space of linear operators with the foundation model. Unfortunately, extending that method to handle non-linear guidance functions is non-trivial. [Chung] ct al. (022a) studied general guidance functions, and modified the sampling process with the gradient of guidance function calculated the expected denoised images. Nevertheless, the authors only presented results with simpler non-linear guidance functions such as non-linear blurring.
In this work, we study universal guidance algorithms for guided image generation with diffusion models using any off-the-shelf guidance functions f. such as object detection or segmentation networks.
UNIVERSAL GUIDANCE
We propose guidance algorithm that augments the image sampling method of a diffusion model to include guidance from an off-the-shelf auxiliary network. Our algorithm motivated by an empirical observation that the reconstructed clean image &o obtained by Eq. (3). while naturally imperfect, is still appropriate for a generic guidance function to provide informative feedback to guide the image generation. In Sec. 3.1. we motivate our forward universal guidance by extending classifier guidance Dhariwal & [ to leverage this observation and handle generic guidance functions. In Sec. 3.2 we propose supplementary backward universal guidance to help enforce the generated image to satisfy the constraint based on the guidance function f. In Sec. Sec.3.3] we discuss a simple yet helpful stepwise refinement trick empirically improve the fidelity of generated images.
3.1 FORWARD UNIVERSAL GUIDANCE
To guide the generation with information from the external guidance function f and the loss function f, an immediate thought is to extend classifier guidance (Dhariwal & Nichol (021) to accept any general guidance function. Concretely, given a class prompt c, classifier guidance performs caasifiiationnguiide sampling by replacing cn t) in each sampling step S(z,,)) with
(0(2,,t) ez( vI aj V log
Defining (-))-.) to be the cross-entropy loss and fai to be the guidance function that outputs classification probability, Eq. 4 can be re-written as
Ea(ze, t) (=(,,t) + v1 - a, V fz(z))).
However, directly replacing fei and lee with any off-the-shelf guidance and loss functions does not work in practice, as f is most likely trained on clean images and fails to provide meaningful guidance when the input is noisy.
To address the issue, we observe that
p(c]z)) / p(c]20, 2 2))p(2)22)da =E (p.
(6)
where c is conditionally-indeeeeee predicted clean image 2, by Eq. (3) with (2, t), we approximate the expectation in Eq. (6) as
with given 20- Leveraging the fact that obtain
Pesolp [p[c|2o)] p.) This leads to our proposed guided sampling procedure
4

---

Published as a conference paper at ICLR 2024
Eo(2),t) = co(2e, t) + s(t) V f(c, f(20) where s(t) controls the guidance strength for each sampling step and
v (1t-a1 22, f(a))) V., (
2t
Vas
as in Eq. 33. We term Eq. 7 forward universal guidance, forward guidance in short. In practice, applying forward guidance effectively brings the generated image closer to the prompt while keeping the generation trajectory in the data manifold. We note that related approach is also studied in Chung et al. (2022a), where the guidance step is computed based on E 20 2r]. The approach drew inspiration from the score-basee generative framework (Song et al. 2021b). but resulted in a different update method.
3.2 BACKWARD UNIVERSAL GUIDANCE
As will be shown in Sec. 4.2 observe that forward guidance sometimes over-prioritizes maintaining the "realness" of the image, resulting in an unsatisfactory match with the given prompt. Simply increasing the guidance strength s(t) suboptimal, as this often results in instability the image moves off the manifold faster than the denoiser can correct it.
Algorithm 1 Universal Guidance
To address the issue, we propose backward universal guidance, or backward guidance in short, to supplement forward guidance and help enforce the generated image to satisfy the constraint. The key idea of back- ward guidance is to optimize for clean im- age that best matches the prompt based on 2g, and linearly translate the guided change back to the noisy image space at step t. Concretely, instead of directly calculat- ing V E(c, f'(,)), we compute a guided change Azg clean data space
Parameter:
Refinement steps k, backward guidance steps m, and guidance strength st), Required: 2, the noisy vector a given time-step t, diffusion model eg, noise scale Os,, guidance function f, loss function É, and prompt
for 1,2,..... do
Calculate 2n as in Eq. (3
Calculate ég using forward universal guidance
in Eq. 7
if m then
Calculate Azo by minimizing Eq. 8 with steps of gradient descent.
zg = arg min f(c, f(2 + A)).
(8)
Perform backward universal guidance by
Empirically, we solve Eq. (8) with m-step gradient descent, where we use starting point. Since ÅP + A minimizes f(c, f())) directly, A is the change in clean data space that best enforces the con- straint. Then, we translate Azo back to the noisy data space of 2; by calculating the guided denoising prediction ? that satisfies 2, ( + Az)) v1 a,ē
En & - /at/(1 a,) Azo (see Eq. 100.
0 as
end if
2--1 t S(2,,6o,t) é' N(0,
24 + Vaa/aa-122 + V1 a1/au-1é.
end for Return 2
(9)
Reusing Eq.
3 we can rewrite ē as an augmentation to the original denoising prediction e0(2,, () by
== c (z,, Var((1 Az)
(10)
Comparing to forward guidance, backward guidance (as Eq. 10 produces an optimized direction for the generated image to match the given prompt, and hence prioritizes enforcing the constraint. Furthermore, calculation of a gradient step for Eq. (8) is computationally cheaper than forward guidance (Eq. (7)), and we can therefore afford to solve Eq. with multiple gradient steps, further improving the match with the given prompt.
We note that the names "forward" and "backward" are used analogously to the forward and backward Euler methods.
5

---

Published as a conference paper at ICLR 2024
3.3 UNIVERSAL STEPWISE REFINEMENT
Upon applying our universal guidance to stan- dard generation pipelines, we observe that in some instances, the images generated exhibit artifacts or unusual behaviors that differentiate them from natural images. Similar findings have been reported in (Lugmayr et al. 2022 Wang et al. (022b), where linear guidance functions were explored. Although we attempted to en- hance the realism by adjusting the guidance strength s(t), finding an optimal balance that both guarantees image realism and adherence to guidance constraints proved challenging, particularly with complex guidance functions. We hypothesize that the direction of guidance offered by our universal method might not always correlate with image realism, diverting the image from a natural image sampling trajectory.
Figure 2: An example of how the Universal Step- wise Refinement (USR) helps segmentation-guicl -uidded generation. The left-most figure is the given seg- mentation map, and the images generated with USR steps 1, 4 and 10 follow in order. Notice the increase of quality with USR steps.
Drawing motivation from Lugmayr al. (2022) Wang al. (2022b). we propose more nuanced approach, termed Universal Stepwise Refinement (USR), to navigate these challenges. Specifically, after sampling - = S(z,, Ex, l), we reintroduce Gaussian noise e' (N, I) žt- deriving z/ as per the equation:
Vav//aai- 2t-1 VI-a//ot-- 'e'.
(1I)
Equation I1 ensures 2 retains the appropriate noise scale for input at time step t. Given refinement step k. we repeat the Universal Stepwise Refinement l: times before advancing with the sampling for step f- This refined process enables the exploration of various regions of the data manifold at consistent noise scale to reach a solution that aligns with both the guidance and image quality requisites. Our empirical assessments indicate that using USR mechanism on top of proper guidance strength s(t) significantly enhanced the image realism while still maintaining the adherence to the prompt, as shown in Fig. 2
We summarize our universal guidance algorithm composed of forward universal guidance. backward universal guidance and universal stepwise refinement for a single sampling step in Algorithm [ For simplicity, the algorithm assumes only one guidance function, but can be easily adapted handle multiple pairs of (f, Additionally, the objectives of the forward and backward guidance do not have be identical, allowing different ways simultaneously utilize multiple guidance functions.
4 EXPERIMENTS
In this section, we present results testing our proposed universal guidance algorithm against wide variety of guidance functions. Specifically, we experiment with Sttable Diffusion (Rombach et al. 2022), diffusion model that is able to perform text-conditional generation by accepting text prompt as additional input, and experiment with a purely unconditional diffusion model trained on ImageNet (Deng al. 2009), where we use pre-trained model provided by OpenAI (Dhariwal & Nichol, 2021). We note that Stable Diffusion, while being a text-conditional generative model, can also perform unconditional image generation by simply using an empty string the text prompt. We first present the experiment on Stable Diffusion for different guidance functions in Sec. 4.J] and present the results on ImageNet diffusion model in Sec. 4.2 Hyper-parameters used for different guidance functions, further ablation studies and selection procedures for suitable guidance strength s(t) and refinement step le can be found in the appendix.
4.1 RESULTS FOR STABLE DIFFUSION
In this section, we present the results of guided image generation using Stable Diffusion as the foundation model. The guidance functions we experiment with include a segmentation network, face recognition network, object detection network and style guidance with CLIP feature extractor (Radford et al.] 2021). For experiments on Stable Diffusion, we discover that applying forward guidance already produce high-quality images that match the given prompt, and hence set m = O. To perform forward guidance on Stable Diffusion, we forward the predicted clean latent variable computed by Eq. through the image decoder of Stable Diffusion obtain predicted clean

---

Published as a conference paper at ICLR 2024
Headshot of with
Promps Walker lannd. jcuhvand
Witker hound. Walker hound.
Prompt blonde hair
Headshot of a A headshot of woman made of marble.
Guide
(ashoand on shom.
painting
Guide
with
background.
a woman looking like Lara Croft.
(N(A)
(NAA
Figure 3: In addition to matching the text prompts (above each column), these images are guided by an image segmentation pipeline. Each col- umn contains examples of images generated to match the prompt and the segmentation map in the left-most column. The top-most row contains examples generated without guidance.
Figure 4: Each column contains examples of im- ages generated to match the prompt and the iden- tity of the images in the left-most column. The top-most row contains examples generated with- out guidance.
images. We discuss the qualitative results and implementation details for each guidance function in its corresponding subsection. We summarize our quantitative evaluation Tab. [ where we evaluate how well the generated images match the external guidance with task-specific metrics, and include the similarity of CL.IP embedding between text prompts and images generated with and without guidance. For all tasks, the minimum difference between CLIP similarities indicates that our algorithm performs guidance without sacrificing the effectiveness the underlying diffusion model. We also include additional quantitative and qualitative results for text-guided generation in the appendix.
Task
Metric
Value
Segmentation Map Guidance. To guide im- age generation using a segmentation map as prompt, we use MobileNerV3-Largee (Howard et al. 2019) with a segmentation head, and a publicly available pre-trained model in Py- Torch (Paszke et al. 2019). We use standard per- pixel cross-entropy loss between a given prompt and the predicted segmentation of generated im- ages as our loss function l.
mloU
0.898
Segmentation
CLIP similariiy
(0.247) 0.249
Face similarity
0.642
Face
CLIP similarity
(0.287) 0.234
mAP@50 50
0.634
Object detection
CLIP similarity
(0.263) 0.246
Table 1: Quantitative analysis of different guidance applied on Stable Diffusion. The reference value in parenthesis is obtained with external guidance.
In our experiment, we combine segmentation maps that depict objects of different shapes with new text prompts. We use the text prompt text-conditional sampling, and guide the text-conditional generated images to match the given segmentation maps. Qualitative results are presented in Fig- From Fig. [ see that the generated images show a clear separation between object and background that matches the given segmentation map nearly perfectly. The generated object and background also each match their descriptive text (i.e. dog breed and environment description). Furthermore, the generated images are overall highly realistic. In Tab D we evaluate mloU between the ground truth segmentation map and the predicted segmentation of generated images to assess the match with constraint.
fixed additional input to Stable Diffusion to perform
Face Recognition Guidance. For guiding generation to include specific person's likeness, we propose combining face detection and facial recognition modules into one guidance function. This setup produces facial attribute embedding from image of a face. We use multi-task cascaded convolutional networks (MTCNN) (Zhang et al. (016)) for face detection, and we use facenet (Schroff) et al. 2015) for facial recognition. The guidance function f then crops out the detected face and
7

---

Published as a conference paper at ICLR 2024
Prompe Heudshot of woman with dog.
Headshos of An oil phottii of woman with a headshot of dog on beach. women with
Prompt
A colorful photo an Eiffel Tower
A fantasy photo of volcanoes
of woman
Style
Guide
(N/A)
(N/A)
In addition to text prompts (above each column), these images are guided by an object de- tector. Each column contains examples of images generated to match the prompt and the bounding boxes used for guidance.
Figure 5:
Figure 6: In addition to text prompts, these im- ages are guided by a style image. Each column contains examples of images generated match the text prompt and the style used for guidance.
outputs facial attribute embedding prompt and we use an lloss between embeddings as the loss function É.
We explore different combinations of face guidance and text prompts. Similarly to the segmentation case, we use the text prompt as a fixed additional conditioning to Stable Diffusion and guide this text-conditional trajectory with our algorithm so that the face in the generated image looks similar to the face prompt. In Fig. we clearly see that the facial characteristics of a given face prompt reproduced almost perfectly material, or style is also realized correctly and blends nicely with the generated faces. We again summarize quantitative evaluation in Tab. We evaluate the similarity between facial attributes of ground-trum truth identity and the generated faces. In general, two faces are considered to be from the same person if the similarity is over 0.5, and our algorithm can effectively guide the generated face to meet the criteria.
the generated images. The descriptive text of either background,
Object Location Guidance For Stable Diffusion, we also present the results of guided image generation with an object detection network. For this experiment, 2015) with Resnet-50-FPP. backbone (Li et al. 2021). a publicly available pre-trained model in Pytorch, as object detector. We use bounding boxes with class labels as object location prompt. We construct a loss function l by the sum of three individual losses, namely (1) anchor classification loss, (2) bounding box regression loss and (3) region label classification loss.
Faster-RCNy RCNN (Ren et al.)
We again experiment with different combinations of text prompt and object location prompt, and similarly the text prompt a fixed conditioning Stable Diffusion. Using our proposed guidance algorithm, we perform guided image generation that generates and matches the objects presented in the text prompt the given object locations. The results are presented in Fig. We observe from Fig. 5 that objects in the descriptive text all appear in the designated location with the appropriate size indicated by the given bounding boxes. Each location filled with appropriate, high-quality generations that align with varied image content prompts, ranging from "beach" to "oil painting". In Tab. we use mAP@ 50 to measure how well the generated images satisfy the constraint.
Style Guidance Finally, we conclude our experiments on Stable Diffusion by guiding the image generation based on a reference style given by a style image. To achieve so, we capture the reference style from the style image by the image feature extractor from CLIP, and use the resulting image embedding as prompts. The loss function calculates the negative cosine similarity between the em- bedding of generated images and the embedding of the style image. Similar to previous experiments,
8

---

Published as a conference paper at ICLR 2024
Object Location
Furward Only
Forward + Backward
Masked Image
CHf. Guided
Clf. Seg. Guided
bird
Figure 7: Generation guided by object detection with the unconditional ImageNet model. While
Our guidance algorithm can incorporate feedhack from multiple guidance functions. Left both forward and backward guidance produces re- to right: The inpainting prompt; The classifier- alistic images with the desired objects in the desig-
Figure 8:
guided inpainting output; Images generated with nated locations, forward guidance alone produces both classifier and segmentation guidance, where the wrong objects or the wrong locations/sizes. realistic dogs are generated exactly on the mask.
control the content using text input as additional conditioning to the Stable Diffusion model. We experiment with combinations of different style images and different text prompts, and present the results in Fig. 6 From Fig. ( can that the generated images contain contents that match the given text prompts, while exhibiting style that matches the given style images.
4.2 RESULTS FOR IMAGENET DIFFUSION
In this section, we present results for guided image generation using an unconditional diffusion model trained ImageNet. We experiment with object location guidance and a hybrid guided image generation task which we term segmentation-gunn guidedd inpainting. We also include additional experiments for CLIP guidance in the appendix. We will discuss results of each guidance separately.
Object Location Guidance. Similar to object location guidance for Stable Diffusion, we also the same network architecture and the same pre-trained model as our object detection net- work, and construct identical loss function for our guidance algorithm. However, unlike Stable Diffusion, object locations the only prompts available for guided image generation. We experiment with different object location prompts using either (1) only forward universal guidance and (2) both forward and backward universal guidance. We observe from Fig. that applying both forward and backward guidance generates images that are realistic and the objects match the prompt nicely. On the other hand, while images generated using only forward guidance remain realistic, they feature objects with mismatching categories and locations. The observation is further backed by quantitative evaluation presented Tab. 2 The evaluation is based on mAP@ between the ground truth object locations and the predicted bounding boxes of generated images, and clearly shows that the combination of forward and backward guidance leads to a much better match with the constraint. The results demonstrate the effectiveness of our universal guidance algorithm, and also validate the necessity of our backward guidance.
Object Location Bounding box 0.39
Fwd. Only
Fwd. + Bkd.
0.90
Bounding box
0.18
0.36
Table 2: Quantitative analysis of forward guidance
only versus combination of forward and backward guidance for object detection guidance on Ima- geNet with bounding boxes Fig. 7 The metric mAP@ 50.
Segmentation-Guidte Inpainting. this experiment, we aim to explore the ability of our algo- rithm to handle multiple guidance functions. We perform guided image generation with combined guidance from an inpainting mask, a classifier and a segmentation network. We first generate images with masked regions as the prompt for inpainting. We then pick an object class the prompt for classification and generate a segmentation mask where the masked regions are considered fore- ground objects of the same class We use & loss the non-masked region the loss function for inpainting, and set the corresponding st)) = 0, or equivalently only use backward guidance for inpainting. We use the same segmentation network described in Sec. 4.1 For classification

---

Published as a conference paper at ICLR 2024
guidance, the classifier that accepts noisy input (Dhariwal & Nichol 2021). and perform the original classifier guidance as Eq. (4) instead of our forward guidance. The results in Fig. show that when using both impainting and classifier guidance, our algorithm generates realistic images that both match the inpainting prompt and are classified correctly to the given object class. Adding in segmentation guidance,
algorithm further improves by making near-perfee match both the segmentation map and the inpainting prompt while maintaining realism, demonstrating that our algorithm effectively combines feedback from multiple guidance functions.
CONCLUSION
We propose a universal guidance algorithm that is able to perform guided image generation with any off-the-shelf guidance function based on fixed foundation diffusion model. Our algorithm only needs guidance and Ioss functions to be differentiable, and avoids any retraining to adapt the guidance function or the foundation model to specific type of prompt. We demonstrate promising results with our algorithm on complex guidance including segmentation. face recognition and object detection systems- and multiple guidance functions can even be used together.
6 ACKNOWLEDGEMENT
This work was made possible by the ONR MURI program and the AFOSR MURI program. Com- mercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation (IIS-2112182) and by the NSF TRAILS Institute (2229885).
REPRODUCIBILIIY STATEMENT
we describe guidance functions and foundation diffusion models for experiments presented Sec. in the corresponding subsections. Hyperparameters for experiments described in Sec. of the main paper can be found Sec. B in the appendix. We also include the source code used to conduct the experiments described in the paper in our supplementary material.
REFERENCES
Jacob Austin, Daniel Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models discrete statesspaces, Advances in Neural Information Processing Systems, 34:17981- 17993, 2021.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jic S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208. 09392, 2022.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.1 14687, 2022a.
Hyungjin Chung. Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models inverse problems using manifold constraints. arXiv preprint arXiv:2206.0000 00041, 2022b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icce, 2009.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. volume 34, 2021.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and and and-play priors. arXiv preprint arXiv:2206.00000 (9012, 2022.
Jonathan Ho and Tim Salimans, Classifier-free diffusion guidance. arXiv preprint arXiv:2207.1559. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 32, 2020.
10

---

Published as a conference paper at ICLR 2024
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1314-1324, 2019.
Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Ren, and Zhou Zhao. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. arXiv preprint arXiv:2204,099.. 2022.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint Arviv:2001.11 11799, 2022.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint wrXiv:200900976 2020.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. arXiv preprint 2tt.:22205.4211.. 2022.
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers, arXiv preprint arXiv:2111.111 2021.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jac Lee. Gligen: Open-set grounded lext-to-image generation. arXiv preprint arXiv:2301.00090 2023.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte. and Lue Van Gool. Repaint: Inpainting using denoising diffusson probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 10741 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, gegh-performance deep learning library, Advances in neural information processing systems, 32, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Robin Rombach, Andreas Blatmmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. righuuesolution image synthesis with latent diffusion models. In Proceedings of CVPR, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479- 36494, 2022.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 815-823, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. International Conference on Learning Representations, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Soch--iccstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021b.
Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Semantic image synthesis via diffuusion models. arXiv preprint arXiv:2207.00058 2022a.
Yinhuai Wang. Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:221,,00 00490 2022b.
11

---

Published as a conference paper at ICLR 2024
Jay Whang. Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12293-16303, 2022.
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li. Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23(10):1499-1503 2016.
Lvmin Zhang Maneesh Agrawala. Adding conditional control sext-to-image diffusion models, arXiv preprint arXiv:2302.05520 05543, 2023.
12

---

Published as a conference paper at ICLR 2024
A LIMITATIONS
Generation using universal guidance is typically slower than standard conditional generation for several reasons. Empirically, more than one iteration of denoising is required for the given noise level t to generate high-quality images with complex guidance functions. However, the time complexity of our algorithm scales linearly with the number of refinement steps k; which slows down image generation when A is large. That being said, for the applications presented in this paper, we were successful in generating images with smaller values of Also, as demonstrated in the main paper, backward guidance is required in certain scenarios to help generate images that match the given constraint. Computing backward guidance requires performing minimization with a multi-step gradient descent inner loop. While proper choices of gradient-based based optimization algorithms and learning rate schedules significantly speed up the convergence compute backward guidance inevitably becomes longer when the guidance function is itself very-large neural network. Finally, we note that, to get optimal results, sampling hyper-parameters must be chosen individually for each guidance network.
minimization, the time it takes
B HYPER-PARAMETERS
In this section, we present the hapereparameters for the different guidance functions i.e. face, segmentation, object location, and style guidance. We present the hyperparameters for experiments on Stable Diffusion in Sec. [.] in the Tab. where we include coefficient so to compute 8t = sovll a and the number Universal Stepwise Refinement (k). We also provide hyperparameters for experiments on ImageNet in Sec. 4.2 in Tab.
Guidance
80
k
Face
20000
2
Object Location
100
Style Transfer
6
Segmentation
400
10
Table 3: Hyper-parameters used in this paper for different guidance functions to reproduce the results for Stable Diffusion.
Guidance
80
k
Object Location
100
3
Segmentation
200
10
Table 4: Hyper-paranne used in this paper for different guidance functions to reproduce the results for ImageNet.
C CLIP GUIDANCE FOR STABLE DIFFUSION
CLIP (Radford et al. 2021 is a state-of-the-art text-to-image similarity model developed by OpenAL. We use the image feature extractor CLIP to do text-guided image generation with algorithm. We construct loss function that calculates the negative cosine similarity between an image embedding and the CLIP embedding produced by given text prompt. We use s(t) 10vI- a4 and and use Stable Diffusion as an unconditional image generator.
We generate images guided by a number of text prompts. To further assess our universal guidance algorithm and compare guidance and conditioning, we also generate images using classical, text- conditional generation by Stable Diffusion with identical prompts as inputs, and summarize the results in Fig. 9 The results in Fig. show that our algorithm can guide the generation to produce high-quality images that match the given text description, and are comparable with images generated by the specialized text-condie text-conditioning model. We also include qualitative results from experiments BrawBench (Saharia et al.) (022). DrawBench is a widely-used and diverse list of text prompts. We randomly select 20 prompts and generate 10 images for each individual prompt. We compute CL.IP
13

---

Published as a conference paper at ICLR 2024
Conditional
Guided
A photograph of an astronaut riding a horse.
An oil painting of a corgi wearing party hat.
Figure 9: We compare the ability to match given text prompts between our universal guidance algorithm and text-conditional model trained from scratch. The results demonstrate that our universal algorithm is comparable to specialized conditional model on the ability to generate quality images that satisfy the text constraints.
score, the cosine similarity between CLIP feature of the prompt and the associated generated im- age, on both Stable Diffusion and our algorithm, and report the average in Tab. [5] As demonstrated in the table, the performance of our algorithm is quantitatively similar to dedicated text-conditional gen- erator, while requiring no additional training at all for the underlying unconditional diffusion model.
Stable Diffusion
Universal Guidance
CLIP Score
0.2818
0.2632
Table 5: Quantitative results on DrawBench for Stable Diffusion and our algorithm.
14

---

Published as a conference paper at ICLR 2024
D CLIP GUIDANCE FOR IMAGENET DIFFUSION MODEL
English foxhound by Edward Hopper
Van Gogh Style
Cake
Figure 10: We show that unconditional diffusion models trained on ImageNet be guided with CLIP to generate high-quality images that match the text prompts, even if these generated images should be oul of distribution.
CLIP Guidance. We use the same construction of f and é for Stable Diffusion to perform CLIP- guided generation. We use only forward guidance for this experiment. To assess the limit of our universal guidance algorithm, we hand-crafted text prompts such that the matching images expected to be out of distribution. In particular, our text prompts either designate art styles that are far from realistic designate objects that do not belong to any possible class label of ImageNet. We present the results in Fig. 10 and from the results, we clearly see that our algorithm still successfully guides the generation produce quality images that also match the text prompts. For all three images, we have s()) = w VI - dg, where w is 2, 5 and respectively and & 10, 5 and 10 respectively.
15

---

Published as a conference paper at ICLR 2024
Sogmentation Map
C-
c=50
C=100
G-200
c=400
c-
k=4
Figure 11: The qualitative results for effect of different guidance strength s(t) and universal refinement step k segmentation guidance for Stable Diffusion. Here we use stt) v and compare different e instead.
E ABLATION STUDY AND PROCEDURE TO PICK HYPERPARAMETERS
In this section we present our ablation study on the effect different parameters, namely guidance strength st) and refinement step k, for algorithm. We summarize the qualitative results in Fig. [I] on segmentation guidance for Stable Diffusion. From the figure, we observe that increasing c alone leads to better matching between the segmentation map and the position of the generated dogs, but the quality of images also degrades, especially when c is large (c >= 200). On the other hand, increasing k noticeably improves the quality of images. For example, comparing the best image from (k, c) (4, 800) and c) (10, 800), the latter clearly exhibits significantly better realness. We also quantitatively evaluate the effectiveness of the two parameters, and present the results in Tab, 6 We calculate the mIOU between the ground truth segmentation map and the segmentation map of generated images predicted by the given segmentation network, which assesses the match between generation and the given constraint. We also show cosine similarity between the clip attribute vectors of the generated images and the given text phrase. High cosine similarity indicates that generation guided with segmentation does not stray away from the text-conditional generation of the underlying diffusion model. Tab. 6 we observe that when the step m is fixed, increasing the guidance strength leads to a better match with external guidance. On the other hand, the clip similarity score is stable for e in range of (10, 200) before showing a sudden drop when e >= 400, indicating that it is more difficult for the underlying diffusion model to control the guided generation match the text when large. As for universal refinement step k, increasing k generally improves both mIOU and clip similarity.
Metric
Refinement step
c=10
c 20
c= 50
c=
100
=200
4000
=800
k=2
0.52
0.52
0.67
0.79
0.87
0.89
0.89
mloU
k=4
0.50
0.59
0.71
0.80
0.87
0.89
0.88
k=10
0.57
0.62
0.76
0.80
0.89
0.90
0.92
k=2
0.26
0.26
0.25
0.24
0.24
0.20
0.16
CLIP sim.
k=
0.26
0.27
0.26
0.25
0.25
0.21
0.14
k=10
0.24
0.25
0.26
0.26
0.25
0.22
0.18
Table 6: Quantitative evaluation for different combination parameters of segmentation guidance on Stable Diffusion.
We also want to remark that the ablation study formalizes principled way to pick suitable k and s(t) for different guidance functions. In particular, increasing & is always beneficial to the generation quality, and the value of & is limited only by the computational budget. Given a fixed k there is generally a sweet spot for s(t) that ensures both match to the target and sufficient quality, and this sweet spot can be found with standard parameter search, described above.
16

---

Published as a conference paper at ICLR 2024
F MORE RESULTS ON STABLE DIFFUSION
(a) Walker hound, Walker foxhound on snow.
(b) Walker hound, Walker foxhound under water.
Walker hound, Walker foxhound oil painting.
Figure 12: More images to show Segmentation guidance. In each subfigure, the first image is the segmentation map used to guide the image generation with caption as its text prompt.
17

---

Published as a conference paper at ICLR 2024
(a) Headshot of person with blonde hair with space background.
(b) A headshot of a woman looking like lara croft.
(c) headshot a blonde woman as a sketch.
Figure 13: More images to show Face guidance. In each subfigure. the first image is the human identity used to guide the image generation with its caption as its text prompt.
18

---

Published as a conference paper at ICLR 2024
person
dog
(a) A headshot a woman with a dog winter.
person
dog
(b) headshot of woman with dog beach.
person
dog
(c) An oil painting of headshot of woman with a dog.
Figure 14: More images to show Object Location guidance. In each subfigure, the first image is the object location used to guide the image generation with its caption as its text prompt.
19

---

Published as a conference paper at ICLR 2024
(a) colorful photo of an Eiffel Tower.
(b) A fantasy photo of lonely road.
(c) A fantasy photo volcanoes.
Figure 15: More images to show Style Transfer. In each subfigure, the first image is the styling image used to guide the image generation with its caption as its text prompt.
20


Reviews:
[{"review_id": "3yjTwEPTlQ", "reviewer": "Reviewer_9puQ", "summary": "The paper proposes a universal guidance algorithm to make the diffusion model condition on multiple modalities without retraining the model. The core idea is to use the estimated clean image in the intermediate step to provide the classifier guidance. The experiments demonstrate the effectiveness of the proposed method.", "questions": "The method uses the predicted clean image as the input of the conditioning model. Would the inaccurate predicted clean image affect the performance? The authors need to study this issue.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The proposed method enables the pretrained diffusion model to condition multiple modality controls without any retraining.\n\n2) The paper is well-written and easy to follow.\n\n3) The experiment demonstrates the effectiveness of the proposed method.", "weaknesses": "1) The method uses the predicted clean image as the input of the conditioning model. Would the inaccurate predicted clean image affect the performance?\n\n2) Some previous work [1, 2, 3] on the conditional diffusion model should be discussed. \n\n[1] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang#, Zhongang Qi, Ying Shan, Xiaohu Qie. T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models\n\n[2] Lvmin Zhang, Anyi Rao, Maneesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models\n\n[3] Ziqi Huang, Kelvin C.K. Chan, Yuming Jiang, Ziwei Liu. Collaborative Diffusion for Multi-Modal Face Generation and Editing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HJNdxE4z44", "reviewer": "Reviewer_Dj3j", "summary": "Sampling images from pretrained diffusion models using conditions is now a very hot topic and plays very important roles in many application scenarios. There are two categories of methods for conditional generation: 1) retraining with new conditions, such as Control-Net; 2) using a guidance function to optimize a latent in the pretrained image space. Considering the cost of re-training, this paper targets a universal guidance strategy for conditional sampling. The experiments demonstrate the results of many different conditions: classifier label, human identity, segmentation maps and object locations etc.", "questions": "See weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The motivation is very clear and the writing is very easy-to-follow. \n- Evenly the idea seems simple, it is novel and reasonable. \n- The experiments contain many different condition scenarios and the results are visually good.", "weaknesses": "My major concern is on the experiments:\n\n* It lacks ablation: as claimed in Sec 3.1, \"directly replacing fcl and lce with any off-the-shelf guidance and loss functions does not work in practice\". I think it is needed to include an experiment to support this claim. In addition, only \"object location guidance\" in Sec 4.2 provided the ablation of forward and backward guidance, the results for at least more than one tasks are helpful. \n\n* I appreciate the experiments on conditional stable diffusion in Sec4.1 and unconditional imagenet diffusion in Sec 4.2. However, I am curious why not also conducting segmentation map guidance in 4.2. I think including same task in both 4.1 and 4.2 can be helpful for readers to understand the differences between the two diffusion models. \n\n* My primary concern is: all presented visual results are not diverse - dogs are everywhere. I also checked all results in the supplemental, it is similar. How about other cases?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ygv8g21R22", "reviewer": "Reviewer_xJFT", "summary": "This submission deals with designing a universal guidance for diffusion models that can adapt to various guidance modalities such as segmentation masks, bounding boxes, in a plug-and-play fashion without any retraining from the scratch or any finetuning. To solve the problem, it proposes to add a loss gradient term to the denoising score function. Let c be the guidance that is the output of some function c=f(z0) for the unknown sample z0. Then the loss is defined to be defined between c and f(\\hat{z}), where \\hat{z} is an estimate for the z0 based on noisy observation zt based on MMSE estimation. This approximation has been commonly used in the diffusion literature e.g., in DDIM, DPS [Chung et al’22 ], PGDM [Song et al’22]. Experiments show that this loss-guidance works to properly guide stable diffusion sampling. \n\n[Song et al’22] Song, Jiaming, Arash Vahdat, Morteza Mardani, and Jan Kautz. \"Pseudoinverse-guided diffusion models for inverse problems.\" In International Conference on Learning Representations. 2022.\n\n[Chung et al’22 ] Chung, Hyungjin, Jeongsol Kim, Michael T. Mccann, Marc L. Klasky, and Jong Chul Ye. \"Diffusion posterior sampling for general noisy inverse problems.\" arXiv preprint arXiv:2209.14687 (2022).\n\n[Song et al’23] Song, Jiaming, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming-Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. \"Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation.\" (2023).", "questions": "The authors need to clarify the contributions of this work compared with previous works, especially the ones in PGDM [Song et al’22], DPS , and [Song et al’23]. I am willing to change my score if the author could clarify the contributions and differences from the work in [Song et al’23].", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Controllable generation from diffusion models without re-training or finetuning is an important problem", "weaknesses": "The idea in this work doesn't seem to be novel. Loss guidance for diffusion models for the same purpose has been studied in previous works that have not been cited; see [Song et al’23]. Also, the idea of using \\hat{z} to approximate z0 based on the score has been used several times in the samping diffusion models for example for inverse problems as in PGDM [Song et al’22], and DPS [Chung et al’22 ]. \n\n\n[Song et al’23] Song, Jiaming, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming-Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. \"Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation.\" (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sO7tXs18mC", "reviewer": "Reviewer_SEzD", "summary": "The authors propose a novel method to control generation process by diffusion models using arbitrary guidance functions without any retraining. The proposed method comprises two types of guidance: forward guidance and backward guidance. In the forward guidance, an estimated clean data is used to compute the guidance function, and its gradient is directly employed to adjust the estimated noise at each timestep. On the other hand, in the backward guidance, the adjustment is obtained by multiple step gradient descent, which leads to more faithful results to the constraint incurred by the guidance. The exprimental results show that the proposed method works well across various types of guidance.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method can be applied to a wide variety of conditional generation tasks without retraining the base diffusion model.\n\n- In the experimetns, the proposed method performs well in terms of the quality of the generated images. It is also impressive that the proposed method allows for a variety of conditional generation.\n\n- Overall, the manuscript is well-written and easy to follow.", "weaknesses": "<Major ones>\n\n- The novelty in methodology is marginal.\n  - In the forward guidance, the loss function is computed based on the clean data predicted from the noisy data at each timestep, and its gradient with respect to the original noisy data is used for the guidance. This approach is quite similar to that employed in DPS and FreeDoM [R1]. \n  - In addition, the stepwise refinement is also similar to time-travel strategy in FreeDoM.\n    - [R1] \"FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model,\" ICCV 2023.\n\n- The advantage of the proposed method over existing methods is not clear. The experiments lack comparison with baseline methods, though several related studies are refered in Section 2. Specifically, Diffusion Posterior Sampling (and FreeDoM as well) is closely related to this work and should be compared qualitatively and quantitatively.\n\n\n<Minor ones>\n\n- As any kind of guidance function can be used in the proposed method, it would be interesting to see how its performance varies according to the design of the guidance function. For example, in the case of segmentation map guidance, we may have a lot of publicly-available segmentation models that can be utilized for the guidance. In this case, can we simply choose the best performing model? This point is not clear in the manuscript, because only single model is examined for each conditional generation task.\n\n\n---\n\n<Update after the rebuttal period>\n\nThanks for the response. I especially appreciate the additional results on comparison with LGD. As LGD can be seen as a modified version of DPS, it becomes clear to see the advantge of the proposed method over DPS as well as LGD. Recognizing the significance of the empirical performance of the proposed method, I have revised my score from 5 to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)