AdjointDPM: Adjoint Sensitivity Method for Gradient Backpropagation of Diffusion Probabilistic Models
Authors: Jiachun Pan
Jun Hao Liew
Vincent Tan
Jiashi Feng
Hanshu Yan
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion Probabilistic Model
Neural ODE
Adjoint Sensitivity Method
Primary area: generative models

Abstract: This paper considers a ubiquitous problem underlying several applications of DPMs, i.e., 
optimizing the parameters of DPMs when the objective is a differentiable metric defined on the generated contents. 
Since the sampling procedure of DPMs involves recursive calls to the denoising UNet, naive gradient backpropagation requires storing the intermediate states of all iterations, resulting in extremely high memory consumption. 
To overcome this issue, we propose a novel method AdjointDPM, which first generates new samples from diffusion models by solving the corresponding probability-flow ODEs. It then uses the adjoint sensitivity method to backpropagate the gradients of the loss to the models' parameters (including conditioning signals, network weights, and initial noises) by solving another augmented ODE. 
To reduce numerical errors in both the forward generation and gradient backpropagation processes, we further reparameterize the probability-flow ODE and augmented ODE as simple non-stiff ODEs using exponential integration. 
AdjointDPM can effectively compute the gradients of all types of parameters in DPMs, including the network weights, conditioning text prompts, and noisy states.
Finally, we demonstrate the effectiveness of AdjointDPM on several interesting tasks: guided generation via modifying sampling trajectories, finetuning DPM weights for stylization, and converting visual effects into text embeddings.

Published as a conference paper at ICLR 2024
ADJOINTDPM: ADJOINT SENSITIVITY METHOD FOR GRADIENT BACKPROPAGATION OF DIFFUSION PROBA- BILISTIC MODELS
Jiachun Pan" National University of Singapore pan. joenus edu. sg
Jun Hao Liew
ByteDance
junhao liew@bytedance, com
Vincent Y. F. Tan
Jiashi Feng
National University of Singapore vtanênus. sg
ByteDance
sg
jstrfengebbyteaaccecoom
Hanshu Yan*
ByteDance
hanshu. waaahh..yynbbttdnnne..cmm
ABSTRACT
This paper considers a ubiquitous problem underlying several applications of DPMs, i.e., optimizing the parameters of DPMs when the objective is differen- tiable metric defined on the generated contents. Since the sampling procedure of DPMs involves recursive calls the denoising UNet, naïve gradient backpropaga- tion requires storing the intermediate states of all iterations, resulting in extremely high memory consumption. To overcome this issue, we propose a novel method Adjoin(DPM, which first generates new samples from diffusion models by solv- ing the corresponding probability- method to backpropagate the gradients of the loss to the models' parameters (includ- ing conditioning signals, network weights, and initial noises) by solving another augmented ODE To reduce numerical errors in both the forward generation and gradient backpropagation processes, we further reparameterize the probability-o ODE and augmented ODE as simple non-stiff ODEs using exponential integration. AdjoinDDPM can effectively compute the gradients of all types of parameters in DPMs, including the network weights, conditioning text prompts, and noisy states. Finally, we demonstrate the effectiveness of AdjoinDDPM on several interesting tasks: guided generation via modifying sampling trajectories, finetuning DPM weights for stylization, and converting visual effects into text embeddings."
ODEs. It then uses the adjoint sensitivity
INTRODUCTION
Diffusion Probabilistic Models (DPMs) constitute family of generative models that diffuse data distributions into white Gaussian noise and then revert the stochastic diffusion process to synthesize new contents (Ho al., 2020; Song al., 2021). DPM-based methods have recently achieved state-of-the-ane performances in generating various types of contents, such as images (Saharia et al., 2022; Rombach et al., 2022; Ramesh et al., 2022), videos (Blattmann et al., 2023; Zhou al., 2022; Ho et al., 2022), and audio data (Liu et al., 2023a; Schneider, 2023). To promote the development of downstream tasks, several pre-trained geggh-eeffrmance models, such as Stable Diffusion (SD) (Rombach et al., 2022), have been made publicly available. Based on these public large-scale models, researchers have developed many algorithms for creative applications (Gal et al., 2022;
Daras & Dimakis, 2022; Kawar et al., 2023; Ruiz et al., 2022; Fei et al., 2023; Wen et al., 2023;
'Equal contribution. This work completed during Jiachun Pan's internship ByteDance.
Projectt Lead.
Gittub link for codes: httpt::///gg//t http://ppw.gg..ooo....o////////a

---

Published as a conference paper at ICLR 2024
Molad et al., 2023). For example, a line of customization algorithms of DPMs, such Textural- Inversion (Gal et al., 2022) and DreamBooth (Ruiz et al., 2022), have been proposed to adapt DPMs for generating images/videos that share certain styles identities. Researchers also have proposed some guidance algorithms (Bansal et al., 2023; Ho & Salimans, 2021) to make the generation process more controllable.
ubiquitous problem in customization and guidance applications is the optimization of the diffusion models' parameters so that the final generated contents satisfy certain properties. For example, to customize models for certain styles, we need to optimize the model weights to minimize the style distance between the generated images and the reference. Alternatively, concerning the guidance of sampling, we need to adjust the intermediate noisy states via the gradients of the guidance loss computed on the final generated data. Generally, the parameters to optimize include the conditional text embeddings, the network weights, and the noisy states, as they all affect the sampling trajectories. We formulate this optimization problem as follows. Denote the DPM as : which generates samples by iteratively calling function Eg. The desired properties can be defined by the loss function L(-) computed based on the generated contents Xa = o(xr, C, Eg). We aim to minimize the loss by optimizing the variables v, including the weights 8, conditioning signals c, or initial noise Xr, i.e.
min ((((XT,C,E)) WE (xT:,00))))
To solve the optimization problem (1), an effective backpropagation (BP) technique is required to compute the gradient of the loss function L(xo) with respect to the optimization variables. Song et al. (2021) showed that the DPM sampling process is equivalent to solving probability-flow ODE. Thus, many efficient sampling methods have been developed using adaptive ODE solvers. The sampling process involves recursive calls to the denoising UNet eg (x,,,, e) for multiple iterations. Using naive gradient BP requires intermediate state storage for all iterations, resulting in significant GPU memory consumption. To overcome this problem, we propose AdjointDPM, a novel gradient BP technique based on the adjoint sensitivity method (Chen et al., 2018). AdjointDPM computes the gradient by solving a backward ODE that only needs to store the intermediate state at the time point of function evaluation, resulting in constant memory usage. Moreover, we reparameterize the diffusion generation process to simple non-stiff ODE using exponential integration, which helps reduce discretization errors in both the forward and reverse processes of gradient computation.
We evaluate the effectiveness of AdjoinDDM by applying it to several interesting tasks, involving optimizing the initial/ineerediatee noisy states, network weights, and conditioning text prompts, respectively. 1) Guided sampling. Under the supervision of ine-grained vision classifiers, Adjoint- DPM can guide the Stable Diffusion to synthesize images of certain breeds of animals. 2) Security auditing of image generation systems. AdjointDPM successfully finds a set of initial noise whose corresponding output images contain NSFW (not safe for work) content, but can sneakily bypass the moderation filters. This triggers alert about the potential security issues of existing Al generation systems. 3) Stylization via a single reference image. AdjointDPM can finetune a Stable Diffusion model for stylization defined by the Gram Matrix (Gatys et al., 2015) of reference image. The stylizing capability of the fine-tuned model can generalize to different objects. All the empirical results demonstrate the flexibility and general applicability of AdjointDPPM We summarize the main contributions of this paper as follows:
1. We propose novel gradient backpropagation method of DPMs by applying the adjoint sensitivity method to the sampling process of diffusion models.
2. To the best of our knowledge, AdjoinDDPM the first general gradient backpropagation method that can be used for all types of parameters of DPMs, including network weights, conditioning text prompts, and intermediate noisy states.
3. AdjointDPM can be used for several creative applications and outperforms other baselines.
BACKGROUND
2.1 PROBABILITY-FLOW ODES CORRESPONDING TO DPMS
The framework of diffusion probabilistic models involves gradually diffusing the complex target data distribution simple noise distribution, such white Gaussian, and solving the corresponding
2

---

Published as a conference paper at ICLR 2024
reverse process to generate samples (Ho et al., 2020; Song al., 2021). Both the diffusion and denoising processes can be characterized by temporally continuous stochastic differential equations (SDEs) (Song al., 2021). Song al. (2021) derive deterministic processes (probability-fon -foww ODEs) that are equivalent to the stochastic diffusion and denoising processes the sense of marginal probability densities for all the time steps.
Let do denote the unknown d-dimensional data distribution. Song et al. (2021) formulated the forward diffusion process {x(()) te[o,]] follows
dx = f(t)x, dt + g() dw., Xo g0)) t E [0, T).
(2)
where x denotes the state at time and W, is the standard Wiener process. f(t)x is a vector-valued function called drift coefficient and g(t) is a scalar function known diffusion coefficient. For the forward diffusion process, it is common to adopt the conditional probability as qos (xe(xo) =
(xx Xo If and the marginal distribution of Xy be approximately standard Gaussian. For sampling, we have a corresponding reverse process as
dx; = [f(1)x+ - Vx,)2xx, (x())) dt + g(t) dw:, X- of0, LE [0,
(3)
In Eqn. (3), the term Vx log qe (x), is known as the score function. We can train a neural network Co (x(X,, t) to estimate -a, v log qi (x,) via denoising score matching. As discussed in Song al. (2021), there exists a corresponding deterministic process whose trajectory shares the same set of marginal probability densities (4: (x) as the SDE (3). The form of this deterministic probability- flow ODE is shown in (4). We can generate new samples by solving Eqn. (4) from T to 0 with initial sample Xy drawn from (O0 II.
dx [(()xx + g(t)² 20, co (x,, cx 1) dt
For conditional sampling, classifier-free guidance (CFG) (Ho & Salimans, 2021) has been widely used in various tasks for improving the sample quality, including text-to-image, image-to-image. class-to-image generation (Saharia et al., 2022: Dhariwal & Nichol, 2021; Nichol et al., 2022). We can use CFG to generate new samples by solving Eqn. (4) and replacing eØ Ex(x t) with ex(x,, e). Ex 1, s (x + (1 5) e0 (x Ø),
2.2 ADIOINT SENSITIVITY METHOOSS FOR NEURAL ODES
Considering neural ODE model
dx s(x,,, 1,0),
the output X = Xx + JP (xx, 0) dt. We aim to optimize the input Xy or the weights 0 by dL minimizing loss L defined the output Xg. Regarding öx. Chen et al. (2018) introduced adjoint aL state a()) = dx, which represents how the loss w.r. the state x, at any time f. The dynamics of a()) are given by another ODE,
da(t) aa())" Ōs((i,t,0) t,
(5)
dt
Ox,
0L is known, we can which can be thought of as the instantaneous analog of the chain rule. Since Oxo
OL by solving the initial value problem (IVP) backwards time to 0 of ODE in (5). compute dxT Similarly, for 0, we can regard them as a part of the augmented state:
x
(t) = [s( t, 8), 0, 1].
dt
The corresponding adjoint state to this augmented state are a (t) = (tt) where ØL and aL. The augmented adjoint state anug is governed by: ag 00
da
(6)
dt
3x
OF
By solving the IVP from time T to of Eqn. (6), we obtain the gradients of w.r.t {x,, 0, The explicit algorithm (Chen et al., 2018) is shown in Algorithm 1.
3

---

Published as a conference paper at ICLR 2024
Algorithm I Reverse-mode derivative of ODE initial value problem
Input: Dynamics parameter 0, start time to, end time t1, final state Xty loss gradient /L/ŌXt.. aL
a(t1) 8x:; ag (t1) 0, 20 [ a(t1),(()))))))))) (ti))
> Define initial augmented state.
def AugDynamics(xx. (x,, a,, 1 1.
D Define dynamics on augmented state.
return (s(x,, 1, 0), - ox
dx
a
Us 00
D. Concatenate time-derivatives
[xto
at. at = ODESolve(zo. AugDynamics, Lı, 4,,40, 0)
Solve reverse-time ODE
8x46 00
Return: AL al
Return gradients
0xig 36
ADJOINT SENSITIVITY METHODS FOR DIFFUSION PROBABILISTIC MODELS
In this section, we develop the AdjointDPM for gradient backpropagation in diffusion models based on the adjoint sensitivity methods from the neural ODE domain. When optimizing the model's s parameters Xy (including the conditioning c), AdjointDPM first generates new samples via the forward probability-fin ODE (4). Through applying the adjoint sensitivity method, we then write out and solve the backward adjoint ODE (6) compute the gradients of loss with respect to the parameters. One can apply any general-purpose numerical ODE solver, such as Euler-Maruyamy and Runge-Kutta methods (Atkinson et al., 2011), I), for solving the ODE. To further improve the efficiency of the vanilla adjoint sensitivity methods, we exploit the semi- linear structure of the diffusion ODE functions (4), which has been used in several existing works for accelerating DPM samplers (Lu et al., 2022a;b; Karras et al., 2022; Zhang & Chen, 2022), and reparameterize the forward and backward ODEs as simple non-stiff ones.
3.1 APPLYING ADIOINT METHODS TO PROBABILITY-FLOW ODES
Sampling from DPMs, obtain the generated data x. =x + fo s(x, t,0,e) dt, where
g(t)2
s(x,, t, c) f(t)x)++
Eo tx),
20.
Concerning the customization or guidance tasks, we aim to minimize a loss L defined on Xy, such as the stylization loss or semantic scores. We plug the equation (7) into the augmented adjoint ODE (5), and obtain the reverse ODE function in Algorithm 1 as:
X
-f(t)x g(t)²
20r čo (x(,xtt,c)
al
OL al g(t)² (zo(xe.t.e) f(t
d
8x:
0x. oxg 20:
0x
dt.
(8)
aL
OL g(t) aé (x..t))
a0
0xi 2a:
09
0L
df(t) OL
DL Big(t) (20,ea(xx.,tt
dt
0x:
x2 +
8x:
We observe that the ODEs governing X; and 8x; both contain linear and nonlinear parts. we directly use off- aff-the-sheile numerical solvers on Eqn. (8), causes discretization errors of both the linear and nonlinear terms. To avoid this, in Section 3.2, we exploit the semi- linear structure of the probability- flow ODE better control the discretization error for each step. Thus, we are allowed to use smaller number of steps for generating samples of comparable quality.
3.2 EXPONENTIAL INTEGRATION AND REPARAMETERIZATION
We use the exponential integration to transform the ODE (4) into a simple non-stiff ODE, We multiply an integrating factor exp(- Jo f(T)dT) on both sides of Eqn. (4) and obtain
de f(r)dr xx
11 f(r)dr g(t)² Eo (xe, (x,,t, t).
20,
Let y, denote e f(r)dr x1, then we have
dy:
di = e 1e f(r)dr
g()) 2
20, ég (J f(r)dr yr, y,,t,)
(9)
4

---

Published as a conference paper at ICLR 2024
We introduce a variable p = 7(1) and da ar e- Jo f(r)dr g(t)²
20, In diffusion models, 7(1) usually
monotonically increases when increases from 0 to T. For example, when choose f(t) log and (1) dou 2 2 in VP-SDE (Song al., 2021), have 7(1) ao a bijective mapping exists between p and and we can reparameterize (9) as: az oo. Thus,
dy
Eo (eli"
(1 f(rjdr y, (D).c)
(10)
dp
We also reparameterize the reverse ODE function in Algorithm as follows
-EG (()dd (p),
01.
Bēa
(050
1.60(0) Krmry (p)).0
By
By
8y 1(00
d
dp.
(11)
al
HEg
(0l
serdr y. (p).c)
0e By
8L
80
0ēa (el0 ar.
(6) Merrr (p)).e)
Op
Now instead of solving Eqn. (4) and Eqn. (8), we use off-the-shelf numerical ODE solvers to solve Eqn. (10) and Eqn. (11). This method is termed AdjoinIDPM. Implementation details are provided in Appendix E.
3.3 ERROR CONTROL
Here, we first show that the exact solutions of the reparameterzied ODEs are equivalent to the original ones. For the equation in the first row of Eqn. (11), its exact solution is:
| 60 E0(el0
p(())
ton y. y. (p), dp.
p((1) = Vo(e) +
(12)
p(s)
We can rewrite it as e f(r)dr x; = c- s())dxx + Set de c) dr. Then, we have
sorar + S' oft s(rjdr g(T)² Eo (x. T, dr,
X, = eli
el ((r)dr
20,
which is equivalent to the exact solution of the equation in the first row of Eqn. (8). Similarly, for other equations in (11), their exact solutions are also equivalent to the solutions in (8). Thus, when numerically solve non-stiff ODEs in Eqns (10) and (11), there only discretization errors for nonlinear functions and the closed form of integration of linear parts have been solved exactly without any numerical approximation.
In summary, we reformulate the forward and reverse ODE functions and show that by using off- the-shelf numerical ODE solvers the reparameterized ODEs, AdjointDPM does not introduce discretization eerror to the linear part. In Section 3.4, we experimentally compare the FID of generated images by solving Eqn. (4) and Eqn. (10) with the same number of network function evaluations (NFE). The results verify the superiority of solving Eqn. (10) regarding error control.
3.4 SAMPLING QUALITY OF ADJOINTDPM
To evaluate the effectiveness of the reparameterization in AdjointDPM, we generate images by solving the original ODE (4) and the reparameterized one (10) respectively. We also other state-of-the-art samplers to synthesize images and compare the sampling qualities (measured by FID). We follow the implementation of DPM in Song et al. (2021) and use the publicly released checkpoints" (trained on the CIFARIO dataset) to generate images in an unconditional manner. We use the torchdiffeq numerical solver with order 4.
package" and solve the ODE (4) and (10) via the We choose a suitable NFE number for ODE solvers so that the DPM can generate content with good quality while not taking too much time. We compare the performance of AdjointDPM (solving the reparameterzied ODEs) to the case of solving the original ones under small NFE regions (< 50).
'https: http:::///ttthbb htt.://www..oooogggoooo////e//ee 'https: // thub. com rtg.cheennnn htaggichhh//tttffffeeee
5

---

Published as a conference paper at ICLR 2024
Table FID (L) for VPSDE models evaluated on CIFARIO under small NFE regions.
We generate the number of images the training set and compute the FID between the generated images and the real ones. From Ta- ble 1, we observe that, after reparameterizing the forward generation process to non-stiff' ODE function, we can generate higher-quality samples with lower FID values under the same NFEs. The sampling qualities of our method are also comparable to those of the saate-of-the-ane
NFE
Solving (4)
DPM-solver
Solving (10)
10
9.50
4.70
4.36
20
8.27
2.87
2.90
50
5.64
2.62
2.58
sampler (DPM-solver (Lu et al., 2022a)).
APPLICATIONS
In this section, we apply AdjoinIDPM to perform several interesting tasks, involving optimizing initial noisy states or model weights for performing guided sampling or customized generation. Due to the space limitation, we provide another application using AdjointDPM for converting visual effects into identification prompt embeddings in Appendix A. The experimental results of all applications demonstrate that our method can effectively back-propagate the loss information on the generated images to the related variables of DPMs.
4.1 GUIDED SAMPLING
In this section, output samples, such as the classification score. We aim to optimize the sampling trajectory, (xe)__r to make the generated images satisfy certain requirements.
AdjointDPM for guided sampling. The guidance is defined by the loss on the
4.1.1 VOCABULARY EXPANSION
The publicly released Stable Diffusion model is pre-trained very large-scale dataset (e.g., LAION (Schuhmann et al., 2022)), where the images are captioned at a high level. It can generate diverse general objects. However, when using it for synthesizing specific kind of object, such as certain breeds of animals or species of plants, we may obtain suboptimal results in the sense that the generated images may not contain subtle characteristics. For example, when generating picture of the "Cairn" dog, the Stable Diffusion model can synthesize dog picture but the shape and the outer coat may mismatch.
Roal examples
vanill CFG
DOODI
AlljinDDM
Orchard Onide
Bhbe Winged Mearleer
Figure 1: Examples for Vocabulary Expansion. The original Stable Diffusion cannot generate samples whose features exactly match the ground-truth reference images. Using the FGVC model, AdjointDPM can guide the Stable Diffusion to synthesize certain breed of animals. Here we can generate images where the dog's face closely resembles target breeds. Besides, we generate birds with features that more similar to real images, such as black heads for Orchard Oriole and blue feathers for the Blue- Winged Warbler.

---

Published as a conference paper at ICLR 2024
Here, use fine tne-grained visual classification (FGVC) model as guidance to help the Stable Dif- fusion model generate specific breeds of dogs. The FGVC model can distinguish objects with subtle vari- ations between classes. Under the guidance of a dog-FGVC model, diffusion models can accurately generate dog images of a specific breed. In other words, the vocabulary base of the diffusion model gets expanded. We formulate this task as follows: Let f()) denote the FGVC model. The guidance Lyy f (xx)) is defined as the prediction score of the generated image Xy for class y. During sampling. in each time step I, we obtain the gradient of guidance L with respect to the noisy state x,, namely Øt, to drive the sampling trajectory. Ix.
Table 2: Per-Class % FID change (+).
Dataset
DOODL
AdjointDPM
Dogs
-5.3%
-7.1%
We present the visual and numerical results in Fig. 1 and Table 2, respectively. By visual comparison to the vanilla Stable Diffusion (SD), we observe that under the guidance of AdjoinDPMM, the color and outer coat of the generated dog images align better with the ground-truth reference pictures. Besides, we compute the reduced FID values on Stanford Dogs (Dogs) (Khosla et al., 2011) and find that the FID values are also improved. We also compare AdjointDPM to a state-of-the-arr baseline, DOODL (Wallace et al., 2023a). AdjointDPM outperforms in terms of visual quality and reduced FID values compared to SD. Refer to more results, optimization details, and comparison with the existing models in the Appendix B.
4.1.2 SECURITY AUDITING
DPMs like Stable Diffusion have been widely used in content creation platforms. The large-scale datasets used for training may contain unexpected and harmful data (e.g., violence and pornography). To avoid generating NSFW content, AI generation systems are usually equipped with safety filter that blocks the outputs of potentially harmful content. However, deep neural networks have been shown to be vulnerable against adversarial examples (Goodfellow et al., 2015; Yan et al., 2020). This naturally raises the concern- -my existing DPM generation systems output harmful content that can sneakily bypass the NSFW filter? Formally, denote f(-) as the content moderation filter and as the conditioning prompts containing harmful concepts. We randomly sample initial noisy state XT, the generated image (XT,, C,E will likely be filtered out by f(-). We want to audit the security of generation systems by searching for another initial noisy state xy, which lies in small 6-neighborhood of xy, such that the corresponding output 4 (xxy, C, 6g) may still contain harmful content but bypass f()). If we find it, the generation systems may face serious security issue.
This problem can also be formulated as a guided sampling process. The guidance T. is defined as the distance between harmful prompt and the prediction score f( C, co))) The distance is measured by the similarity between CLIP embeddings. We optimize the perturbation 5 on Xy to maximize the distance. The norm of the perturbation is limited to be 7 as we want ensure the newly generated image is visually similar to the original one.
max ST
L(c, f(0(XT - 5, 3,c,
We use AdjointDPM to solve this optimization problem and find that there indeed exist initial noisy states of the Stable Diffusion model, whose corresponding outputs can mislead the NSFW filter. Results are shown in Fig. 2. This observation raises an alert about the security issues of AI generation systems. Our research community has to develop more advanced mechanisms ensure the isolation between NSFW content and users, especially teenagers.
Table 3: Success ratio (%) of adversarial ini- tial states bypass the classifier. We show the results for five classes from the ImageNet dataset.
For numerical evaluation, we also audit the security of smaller diffusion model which is trained on the ImageNet dataset. We chose ten classes and sampled hundreds of images. For each sample, we search for the adversarial perturbation that maximizes the classification error. We record the resultant noisy states that can mislead the classification module. The success ratio achieves around 51.2%. Qualitative results about ImageNet are shown in Fig. 7 and more
Index
242
430
779
859
895
Ratio
63.9
75.8
45.3
58.22
52.6
experimental details are provided in Appendix C.
7

---

Published as a conference paper at ICLR 2024
Figure 2: Adversarial samples against the NSFW filter. We show the image generated by conditioning on harmful prompts (e.g., "A photograph of naked man ") the left. These images will be blocked by the NSFW filter. However, the images generated from adversarial initial noises circumvent the NSFW filter (Black squares are added by authors for publication).
4.2 STYLIZATION VIA SINGLE REFERENCE
Seen objects
Unseen objects
Stylization
Stylization
Figure 3: Stylization examples. Images generated by the original Stable Diffusion are shown at the top. The bottom are samples of the stylized Stable Diffusion.
We consider using AdjoinDDPM to fine-tune the weights of the UNet in Stable Diffusion for stylization based on a single reference image. We define the style of image x by its Gram matrix (Gatys et al., 2015), which is computed based on the features extracted from pre-trained VGG model.4 Here denote the features extracted from the VGG as Fxx) and the Gram matrix G = FF"..
Given reference image, we denote the target style Gstyle We aim to fine-tune the weights of the UNet so that the style generated images matches the target one. This task is formulated as the optimization problem (13). The objective contains two terms. Besides the style loss Lg (mean squared error), we also add term content loss L,- The content loss encourages the model to preserve the ability to generate diverse objects when adapting itself to certain style. In specific, sample multiple noise-prompt-mmage triplets, ((x}, c², > where x- denotes the clean image generated by the pre-trained Stable Diffusion with the input of (xf,ct). The content loss is defined as the mean squared error between the features of originally generated images F(x)) and those generated by optimized weights F F(b(xf, c², ea)). Two coefficients, w, and We. balance the strength of the two loss terms.
Σ LwgLo (Gstyles (( co))) + weLe (F(x'), Ff min 9
(13)
N
htm
8

---

Published as a conference paper at ICLR 2024
Table 4: CLIP similarity scores (f) between sam- ples and the conditioning prompts.
We construct 10 prompts corresponding to ten of the CIFARI00 classes and sample starting noises to generate 100 images (10 images for each prompt) to form our training dataset. Vi- sual and numerical results are shown in Fig. and Table 4 respectively. We observe the SD model fine-tuned by AdjointDPM can generate stylized images of different objects. The styliz- ing capability also generalizes to the concepts unseen during fine-tuning (shown in the right part of Fig. 3). addition to the high visual quality, the samples also align well with the conditioning prompts according to the high CLIP similar- ity scores. We compare AdjointDPM with other methods for stylization, including DreamBooth (Ruiz et al., 2022) and Invterio-n (Gal et al., 2022) (see the qualitative comparisons in Fig. 12). We observe that AdJ achieves better alignment between image samples and the prompts. In addition, these existing methods barely can be generalized to unseen objects in this case only one reference image is available. More details and examples of stylization are shown in Appendix D.
Prompt
DreamBooth
Inv
AdjointDPM
Airplane
21.98
24.44
27.34
Clock
28.23
26.74
30.00
House
25.40
25.91
29.03
Cat
25.90
22.30
26.83
Apples
28.10
24.06
28.59
5 RELATED WORKS AND DISCUSSION
Customization of Text-to- Text-to-Image Generation Text-to-image customization aims personalize generative model for synthesizing new images of specific target property. Existing customization methods typically tackle this task by either representing the property via text embedding (Gal et al., 2022; Mokady et al., 2022; Wen et al., 2023) or finetuning the weights of the generative model (Ruiz et al., 2022; Kawar et al., 2023; Han et al., 2023). For example, Textual-Inversion (Gal et al., 2022) inverts the common identity shared by several images into unique textual embedding. To make the learned embedding more expressive, Daras & Dimakis (2022) and Voynov et al. (2023) generalize the unique embedding to depend on the difffusion time or the layer index of the denoising UNet, respectively. In the other line, DreamBooth (Ruiz et al., 2022) learns a unique identifier placeholder and finetunes the whole diffusion model for identity customization. To speed up and alleviate the overfitting, Custom Diffusion (Kumari et al., 2022) and SVDiff (Han et al., 2023) only update a small subset of weights. Most of these existing methods assume that handful of image examples (at least 3-5 examples) sharing the same concept or property are provided by the user in the first place. Otherwise, the generalization of resultant customized models usually will be degraded, i.e., they barely can synthesize unseen objects (unseen in the training examples) of the target concept. However, in some cases, it is difficult or even not possible collect enough data that can represent abstract requirements imposed on the generated content. For example, we want to distill the editing effects/operations shown in single image by a media professional or a novel painting style of unique picture. In contrast, this paper relaxes the requirement of data samples and proposes the AdjointDPM for model customization only under the supervision of diffcrentiable loss.
Guidance of Text-to-Image Generation Concerning the guidance of diffusion models, some algorithms (Bansal et al., 2023; Yu et al., 2023) mainly use the estimated clean state for computing the guidance loss and the gradient with respect to intermediate noisy states. The gap the actual gradient is not negligible. As result, the guided synthesized results may suffer from degraded image quality. Instead, some other methods (Wallace et al., 2023a; Liu et al., 2023b), such as DOODL, compute the gradients by exploiting the invertibility of diffusion solvers (e.g., DDIM). This paper also formulates the sampling process ODE and proposes AdjointDPM to compute the gradients in more accurate and adaptive way.
The main contribution of this paper is that we propose a ubiquitous framework to optimize the related variables (including UNet weights, text prompts, and latent noises) of diffusion models based on the supervision information (any arbitrary differentiable metric) on the final generated data. To our best knowledge, AdjontDPM is the first method that can compute the gradients for all types of parameters of diffusion models. In contrast, other methods, such as DOODL (Wallace et al., 2023a), FlowGrad (Liu et al., 2023b), and DEQ-DDIM (Pokle et al., 2022)), either only work for the noisy states or require the diffusion sampling process having equilibrium points. In the future, we will explore more real-world applications of AdjointDPPM.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research/project is supported by the Singapore Ministry of Education Academic Research Fund (AcRF) Tier under grant number 1-8000423-000 and the Singapore Ministry of Education AcRF Tier under grant number A-8000189 09-0019--0-00. We would like acknowledge that the computational work involved in this research work is partially supported by NUS IT's Research Computing group.
REFERENCES
Kendall Atkinson, Weimin Han, and David E Stewart. Numerical solution of ordinary disferential equations. John Wiley & Sons, 2011.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal Guidance for Diffusion Models, February 2023. arXiv:2302.0712. [es].
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models, April 2023. arXiv:2304.08818 [es].
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, volume 31, 2018.
Giannis Daras and Alex Dimakis. Multiresolution textual inversion. In NeurlPS 2022 Workshop
Score-Based Methods, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, volume 34, 2021.
Zhengcong Fei, Mingyuan Fan, and Junshi Huang. Gradient-Free Textual Inversion, April 2023. arXiv:2304.05818 [cs].
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion, August 2022. arXiv:2208.01618 [cs].
LA Garys, AS Ecker, and M Bethge. neural algorithm of artistic style. Nature Communications, 2015.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR, San Diego, CA, USA, 2015.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. SVDiff: Compact Parameter Space for Diffusion Fine-Tuning, April 2023. arXiv:2303.I 11300 [cs].
Jonathan Ho and Tim Salimans. Classifier-fre free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22 2022.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-Based Real Image Editing with Diffusion Models, March 2023. arXiv:2210.0927 [cs].
10

---

Published as a conference paper at ICLR 2024
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Graines Grained Visual Categorization, IEEE Conference Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-conceptt customization of text-to-image diffusion. arXiv preprint arXiv:2212.0444 0444888, 2022.
Haohe Liu, Zehua Chen, Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang. and Mark Plumbley. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models, February 2023a. arXiv:2301. 12503 [cs, eess].
Xingchao Liu, Lemeng Wu, Shujian Zhang, Chengyue Gong, Wei Ping, and Qiang Liu. Flowgrad: Controlling the output of generative odes with gradients. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 24335-24344 June 2023b.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solvert+: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX1v:221..01095 2022b.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text Inversion for Editing Real Images using Guided Diffusion Models, November 2022. HtXiv:2211.09799 [cs].
Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yacl Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video Diffusion Models are General Video Editors, February 2023, arXiv:2302.01329 [cs].
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text- text-guided diffusion models. In International Conference Machine Learning, pp. 16784-16804.. PMLR, 2022.
Ashwini Pokle, Zhengyang Geng, and J Zico Kolter. Deep equilibrium approaches to diffusion models. Advances in Neural Information Processing Systems, 01137975-37990 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings the 38th International Conference on Machine Learning, pp. 8748-8763 8763. PMLR, July 2021. ISSN: 2640-3498.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents, April 2022. arXiv:2204.06125 [cs].
Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramer, Red-teaming the stable diffusion safety filter. In NeurlPS ML Safety Workshop, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine Tuning Imaae Diffusion Models for Subject- Driven Generation. Tech- nical report, August 2022. arXiv:2208. 12242 [cs].
sabject-Driven
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text- to-image diffusion models with deep language understanding. In Advances Neural Information Processing Systems, volume 35, 2022.
11

---

Published as a conference paper at ICLR 2024
Flavio Schneider. ArchiSound: Audio Generation with Diffusion, January 2023. arXiv:2301.13267 [cs, eess].
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev, LAION-5B: An open large-scale dataset for training next generation image-text models, 2022.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. P+: Extended Textual Condi- tioning in Text-to-Image Image Generation, March 2023. arXiv:2303.09522 [es].
Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimiza- tion improves classifier guidance, 2023a.
Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transfor- mations. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 22532-22544 2023b.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard Prompts Made Easy: Gradient-Basse Discrete Optimization for Prompt Tuning and Discovery, February 2023. arXiv:2302.03666 [cs].
Hanshu Yan, Jiawei Du, Vincent Tan, and Jiashi Feng. On robustness of neural ordinary differential equations. In International Conference on Learning Representations, 2020.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang, Freedom: Training-free energy-guided conditional diffusion model. arXiv:2303.09833, 2023.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In NeurlPS Workshop on Score-Based Methods, 2022.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv. Yizhe Zhu, and Jiashi Feng. Magic Efficient Video Generation With Latent Diffusion Models, November 2022. 1xXiv:2211.1100
[cs].
12

---

Published as a conference paper at ICLR 2024
CONTENTS
Introduction
Background
3 Adjoint Sensitivity Methods for Diffusion Probabilistic Models
4 Applications
Related Works and Discussion
A Text Embedding Inversion
13
B Experimental Details on Vocabulary Expansion and Comparison with Existing Models
15
Experimental Details and More Results on Security Auditing
15
More Examples Finetuning Weights for Stylization
17
E Implementation of AdjointDPM
22
A TEXT EMBEDDING INVERSION
In addition to the applications shown in Section 4, we consider another application concerning using AdjointDPM to convert visual effects (e.g., bokeh and relighting) into identification text embedding given image pair, namely original image and its enhanced. the enhanced
# Suppose version is edited by some professional and appears with certain fascinating visual effects. After optimization, we combine the obtained embedding with various text prompts to generate images with the same visual effect. Here, we simulate this real setting by using a text-to-image model to generate images with and without certain effect.
Suppose we are provided a text-to-image DPM ((.), we can generate an image x by denoising randomly sampled noise Xy in the condition of the base prompt Obase We further improve the aesthetic quality by inserting some keywords Curget. like "bokeh", into the conditioning prompt. The newly generated images are denoted by x". We x" or its feature L(-) that measures the distance to the target effect, such as the f2 or perceptual loss. We aim to optimize special embedding that can recover the visual effects in x":
reference to define loss
min L (x", (XX,, {Chuse #), co))
We utilize the publicly released Stable Diffusion models for image generation and the loss function as the mean squared error (MSE) between the target images and the generated images. We aim optimize a prompt embedding the CLIP (Radford al., 2021) embedding space and use the obtained embedding for image generation by concatenating it with the embeddings of other text prompts. As shown in Fig. 4, we observe AdjointDPM successfully yields an embedding that can ensure the appearance of target visual effects, including the bokeh and relighting. Furthermore, the obtained embeddings # also generalize well other starting noise and other text prompts. For example, the bokeh-# is optimized on pair of totoro images; it also be used for generating different images of totoro and other objects like dog. Similarly, the obtained corresponding to manual editing ("converting black and white") also be used for novel scene generation.
13

---

Published as a conference paper at ICLR 2024
Target concept: "bokeh"
Target concept: "purple neon lighting"
Chase
Cbeve + Crarget
Chasp + #
Chase
Chase +
Clarger
Chuse
Seen noises
Cao "Fallout concept art school interior render grim, unreal engine 5"
Chase "A cute totoro in yard"
Unseen noises
hare "Fallout concept art school interior render grim, unreal engine 5"
Chase
cute totoro in yard"
Different Chare
Chase
"A cute puppy in yard"
Chase "A cat running in the dark night"
Figure 4: Examples prompt inversion part
Target concept: "in style of Van Gogh"
Target concept:
"Black and white"
Cheae
Chose
Easrge
Chose
Chere
C Carger
Chose
Seen noises
Chase
"Photo from city street in the 1970s"
Chsse "A photo of Emma Watson, close up"
Unseen noises
Chose
"Photo from city street in the 1970s"
Chos "A photo of Emma Watson, close up"
Different Choss
Char "Photo city square the 1970s"
Clase
"A photo of Morgan Freeman, close up"
Figure 5: Examples on prompt inversion part 2
14

---

Published as a conference paper at ICLR 2024
B EXPERIMENTAL DETAILS ON VOCABULARY EXPANSION AND COMPARISON WITH EXISTING MODELS
Comparison with Existing Models. DOODL (Wallace et al., 2023a) optimizes the initial diffusion noise vectors w.r.& a model-based loss on images generated from the full-chain diffusion process. In their work, they obtain the gradients of loss w.r.t noise vectors by using invertible neural networks (INNs). There are three main differences between DOODL and our work. First, while DOODL optimizes the initial diffusion noise vectors, our work optimizes related variables, including network parameters, initial noises and textual embeddings w.r.t a model-based loss on images generated from the full-chain diffusion process. Thus, we consider the broader cases of DOODL. Second, in the calculation of gradients w.r.t initial noises, DOODL uses the invertibility of EDICT (Wallace et al., 2023b), i..., Zo and 2T are invertible. This method does not apply to the calculation of gradients w.r.t network parameters and textual embeddings as they share across the full-chain diffusion process. Finally, with regard to the memory consumption when calculating gradients with respect to the initial noise, our experimental results are as follows: we utilized the stable diffusion v1.5 checkpoint to run both the AdjointDPM and DOODL models V100 GPU (32GB memory). For the AdjoinDDPM method, backpropagating the gradients with respect to a single initial noise required 19.63GB of memory. In comparison, the DOODL method consumed 23.24GB for the same operation. The additional memory consumption in DOODL is mainly from the dual diffusion process in EDICT. Thus, our method is more efficient in memory consumption. In terms of time consumption, DOODL relies on the invertibility of EDICT, resulting in identical computation steps for both the backward gradient calculation and the forward sampling process. Besides, they usually use DDIM sampling methods, which is equivalent to the first-order neural ODE solver. However, our Adjoin(DPM methods have the flexibility to apply high-order ODE solvers, allowing for faster backward gradient calculation. See the following for an experimental comparison between method and DOODL.
We also make a comparison with Flow (Liu et al., 2023b). FlowGrad efficiently backpropagates the output to any intermediate time steps the ODE trajectory, by decomposing the backpropagation and computing vector Jacobian products. FlowGrad focuses on refining the ODE generation paths to the desired direction. This is different from our work, which focuses the finetuning of related variables, including network parameters, textual embedding and initial noises of diffusion models for customization. Besides, FlowGrad methods also can not obtain the gradients of loss w.r.t. textual embeddings and neural variables as these variables share across the whole generation path. Then for the gradients w.r.t the latent variables, we could show the memory consumption of our methods is constant while they need to store the intermediate results.
Experimental Details on Vocabulary Expansion During the optimization of the noise states under the guidance of FGVC model, we adopt the Euler ODE solver in our AdjoinDDPM method with 31 steps. We optimize the noise states using the AdamW optimizer for 30 epochs with different learning rates for different breeds. For the implementation of DOODL, we follow the officially released code and we set the sampling steps also to be 31 and optimization steps for 30. Following the DOODL, we also measure the performance by computing the FID between set of generated images (4 seeds) and the validation set of the FGVC dataset being studied. We do experiments on Stanford Dogs (Dogs) (Khosla et al., 2011) 1) datasets and calculate FID values. More qualitative results are shown in Fig 6.
C EXPERIMENTAL DETAILS AND MORE RESULTS ON SECURITY AUDITING
In this section, we provide explicit details about generating adversarial examples against ImageNet classifier and the NSFW filter in Stable Diffusion, respectively.
Security Auditing under ImageNet Classifier. To generate adversarial samples against an ImageNet classifier, we follow the implementation of classifier guidance generation of DPM and use
hhtpps: http://///ggttt.cc////a//ee///// httpps //gi com http:///ww/.cccceggggg/ooooooooo wer/t//eeeeeeeeeeeeeeeeeeeeeee n//eampp es /ddppaa__d___ guided-di ffusion
15

---

Published as a conference paper at ICLR 2024
Real examples
vanilla CFG
DOODL
AdjointDPM
Brabancon Griffon
Caim
Orchard Oslole
Bine- WiiiWiing Winged Warbler
Real examples (Stanford Dog)
vanilla CFG
DOODL
AdjointDPM
black blackk-and aa----n--mm tan coonbound
Whippet
Weimaraner
Figure 6: More Examples for Vocabulary Expansion.
16

---

Published as a conference paper at ICLR 2024
the publicly released checkpoints trained on the ImageNet 128x128 dataset to generate images in conditional manner. We adopt the pre-trained ResNet50 as our ImageNet classifier.
To generate adversarial examples, first randomly choose ImageNet class and set it the class label for classifier guidance generation, and then we pass the generated images to the ResNet50 classifier. If the outputs of ResNet50 classifier aligned with the chosen class label, begin to do an adversarial attack by using AdjointDPM. For the adversarial attack, we adopt the targeted attack, where choose a target class and make the outputs of ResNet50 close the pre-chosen target class by minimizing the cross entropy loss. We also clamp the updated initial noise in the range of (ET 0.8, 2T + 0.8] (i.e., set T 0.8 Sec. 4.3 ensure that generated images do not visually change too much compared with the start images). We show more adversarial examples against the ImageNet classifier in Fig. 7. Besides, define the attack rate as the ratio between the number of samples with incorrect classification results after the attack and the total number of samples. We also get the attack rate 51.2% by generating 830 samples from 10 randomly chosen classes. The class labels here we choose are [879, 954, 430, 130, 144, 242, 760, 779, 859, 997].
wall clock
birdhouse
candie
Touster
sharpouer
Pomsian
Angora
starfish
toddy bear
Boxer
Sorrel
Pension
Sloeping lag
Television
Stove
Cock
Hen
Umbrella
Bathing cap
Warplane
Aifliner
Figure 7: Adversarial examples against the ImageNet classifier. We show the originally generated images with their class names on the left; these images are correctly classified by ResNet50. On the right, we show the corresponding adversarial images which successfully mislead the classifier.
Experimental Details on the NSFW Filter. In this case, we set r = 0.9. We follow the im- plementation of Stable Diffusion and set the loss function the cosine distance between CLIP embeddings Radford et al. (2021) of generated images and unsafe embeddings from Rando et al. (2022).
D MORE EXAMPLES ON FINETUNING WEIGHTS FOR STYLIZATION
In this section, we introduce the experimental details of stylization and present more stylized examples on seen noises and seen classes, unseen noises and seen classes, and unseen noises and unseen classes.
For training, we choose ten classes from CIFAR- 100 classes, which are ['An airplane". " "A cat", "A truck". "A forest", "A house", "sunflowers". "A bottle". "A bed", "Apples", "A clock"]. Then randomly generate 10 samples from each class to compose our training dataset. Besides, we directly use these class names as the input prompt to Stable Diffusion 9. We optimize the parameters of cross
17

---

Published as a conference paper at ICLR 2024
attention layers of UNet for epochs by using AdamW optimizer with learning rate 10 4 We show more stylization results on 100 training samples (seen noises and seen classes) in Fig. Meanwhile, also show more examples of scen-classe-unge unsse---esee and examples of umseen-class-unee sataass-useeennoiss -oise in Fig. 10. In Fig. 11, we also show the stylization results on other target style images, in which one downloaded from the showcase set of Midjourney and the other is the Starry Night by Van Gogh. We also present that the finetuned networks under ODE forms can still apply SDE solvers (such as DDPM) in Fig. 9.
A clock
Apples
Styfuatton
An sirplane
A
Stylonnon
Styluation
house
Sunflowers
Stylizution
A bed
D
Styliation
Figure Stylization examples on seen classes and seen noises
D.1 QUALITATIVE COMPARISONS TO TEXTUAL-INVERSION AND DREAMBOOTH
We also provide visual comparisons to Textual Inversion Gal et al. (2022) and DreamBooth Ruiz et al. (2022) in Fig. 12. We follow the implementation of Textual Inversion !1 and DreamBooth For textual inversion, we use the same target style image in Section 4.2 as the training dataset. As in our AdjoinDDPM model, we use one style image for training, for fair comparison, the training
/
webp "I
18

---

Published as a conference paper at ICLR 2024
Seen class unseen noise
Style Image
Figure 9: Stylization examples on seen classes and unseen noises using DDPM.
Seen class unseen noise
Unseen class
Style Image
Figure 10: Stylization examples on seen-class-wnseennnooeee and unseen-class
19

---

Published as a conference paper at ICLR 2024
Seen classes and noises
Seen classes and unseen noises
Stylization
Seen and noises
Unseen classes and noises
Stylization
Seen classes and seen noises
Seen classes unseen noises
Stylization
Seen classes and seen noises
Unseen classes and unseen noises
Stylization
Figure 11: Stylization examples on other style images
20

---

Published as a conference paper at ICLR 2024
Stylization
XI II
Stylization
(a) Stylization examples generated by Textual Inversion.
Stylization
Stylization
(b) Sty Stylization examples generated by DreamBooth.
Figure 12: Comparison to
and DreamBooth for stylization.
21

---

Published as a conference paper at ICLR 2024
dataset to Textual Inversion and DreamBooth also include only one style image. We set the learnable property as "style", placeholder token as "<bengiles>", initializer token as "flowers" in Textual Inversion. Then we run 5000 epochs with learning rate 5 10 to train the Textual Inversion. For DreamBooth, we set instance prompt as "bengiles flowers". Then we run 1000 epochs with a learning rate 1 x 10-6 to train the DreamBooth. In Fig. 12, we show the stylization examples generated by using Textual Inversion and DrcamBooth. We can observe distinct differences in the stylization outcomes when comparing the Textual Inversion and DreamBooth approaches to our Adjoin(DPM methods. In some cases, for Textual Inversion and DreamBooth, we have noticed that the main objects within an image can vanish, resulting in the entire image being predominantly occupied by the applied "style."
E IMPLEMENTATION OF ADJOINTDPM
In this section, we present the explicit AdjointDPM algorithm. For VP-SDE, we have f(() and (1)
d'log
de
do² 2 qdlog a
2 dt o.. Based on the definition of yt and we obtain
Yr
X
P 7t) 000
00.
Oz
We denote the timesteps for solving forward generation ODE as {4)) where N is the number of timesteps. Then based on this re- parameteriizaion, we can show our explicit forward generation algorithm and reverse algorithm obtaining gradients for VP-SDE Algorithm and Algorithm 3.
For the choice of as, and sampling steps {t, | we adopt the implementaiion of DPM-solver",, Specifically,, consider three options for the schedule of Og and o, discrete, linear, and cosine. The detailed formulas for obtaining 2022a, Appendix D.4). The choice of schedule depends on the specific applications. We usually solve the forward generation ODE function from time T to time (c > 0 is a hyperparameter near 0). Regarding the selection of discrete timesteps (44), in numerically solving ODEs, we generally divide the time range [T' using one of three approaches: uniform, logSNR, or quadratic. The specific time splitting methods can be found the DPM-solver. Subsequently, we obtain the generated images and gradients by following Algorithm 2 and Algorithm 3. To solve ODE funetions in these algorithms, we directly employ the odeint adjoint function the torchdiffeq packages
and o, for each schedule choice are provided in (Lu et al.,
Algorithm 2 Forward generation by solving an ODE initial value problem
Input: model Eg, timesteps (t, (t) initial value Xt.
Yia Xtoi N {pr J- ((tt))
Re-parameterize Xto Re-parameterize timesteps
YEN = ODESolve Nfor (ot) a y-y,,
D. Solve forward generation ODE
Return: X1,
Oln ytx
0ig
Algorithm 3 sevverse-ode. derivative of ODE initial value problem
Input: model Eg, timesteps {p. J.N final state yox- loss gradient aL/ōypN-
ar. a(px)= = oy (p(p)) = 0, 20 [y a(px),())))))))))))()) ag (px))
> Define initial augmented state.
def AugDynamics( [y, ap, }
b Define dynamics on augmented state.
return [s(y, P. 0, c). a by:
0s
a
50
> Concatenate time-derivatives
[y po> dyeg ODESolve AugDynamin {pr (
0r.
Solve reverse-time ODE
Return: 8L 8L 0xi," 38
Return gradients
sol ver ffeq.
22


Reviews:
[{"review_id": "1DScnbxrW8", "reviewer": "Reviewer_6hcA", "summary": "The paper proposes an interesting idea, named AdjointDPM, merging Diffusion Models and techniques from Neural ODE literature. The core offering of the paper is a way of backpropagating gradients of any loss computed using the output of a (trained) Diffusion model. Specifically the authors used the well-known Adjoint Backpropagation method from Neural ODE, which is a backprop algorithm with $\\mathcal{O}(1)$ memory w.r.t the *discretization* of the ODE solver.\n\nThe authors applied their AdjointDPM method on three tasks that either require gradients w.r.t initial state $X_T$ of the reverse process, all intermediate states $\\\\{ X_t \\\\}\\_{t=1}^T$ of the reverse process or the parameters $\\theta$ of denoising model $\\epsilon_{\\theta}(\\cdot)$. They showed good performance in terms of quantitative metrics and also showed qualitative results.", "questions": "I have the following questions for the authors.\n\n- I am confused about section 3.4. That is not AdjointDPM — that is just unconditional generation with an already known sampler (DEIS, which used Adam-Bashforth) that exploits the semi-linear nature. What part of this is your contribution ? Am I missing something here ?\n- In the “security auditing” application, what exactly is the guidance $L$ ? What is the meaning of “distance between a harmful prompt and a prediction score” ? Also, what exactly is the NSFW filter $f(\\cdot)$ ? Can you provide more details please ?\n\nMinor questions or suggestions:\n\n- The unnumbered eq b/w Eq. 5 & 6 — what is the meaning of that line (”Similarly, for $\\theta$, we can regard ..”) ?\n- “adjoint state $\\mathbf{a}(t) = ..$, which represents how the loss depends on the state ..” → “adjoint state $\\mathbf{a}(t)$ = .., which represents how the loss **changes w.r.t the** state ..”.\n- The AdjointDPM eq. 8 shows gradient w.r.t $t$ — is it acutally used anywhere ?\n- Why is it called “vocabulary expansion” ? I still don’t get it.\n- “Security Auditing” is just fancy name for Adversarial Samples ?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposal of the paper is overall good, theoretically sound and shown to have worked well.\n\n- Theoretically, it makes sense to use the Adjoint method on the reverse ODE.\n- The authors exploited the semi-linear nature of the ODE even in the Adjoint backprop, following DPM-Solver/DEIS.", "weaknesses": "- While the proposal is quite novel, one might still argue that it is not really necessary to use Adjoint Backprop. One can very well accomplish the same task by backprop-ing through the solver machinery (maybe by deceasing sampling steps and using better sampler), which of course, won’t be very efficient. So, at the end, it all boils down to compute/memory efficiency. While I understand the memory advantage, sadly, the paper barely talks anything about computational requirements of the method. BTW, Neural ODEs (and Adjoint Backprop) are known to be not very scalable.\n- Experiments are okay-ish, but not really extensive. Qualitative samples are lacking in some experiments (e.g. vocabulary expansion). Also Vocabulary Expansion is shown for only two classes.\n- No comparison or mention of methods that DO backprop through the ODE solver.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ep4mPDIu6T", "reviewer": "Reviewer_5NNi", "summary": "This work leverages the adjoint methods for optimizing the parameters and/or samples of diffusion ODEs under a given differentiable scalar-valued function. To efficiently solve the adjoint ODE, this work also leverages the expoenential integrators and introduce a change-of-variable formula to obtain a simpler ODE. Experiments show that the proposed method can be used for classifier-based sampling, adversarial sampling and stylization with a single reference.", "questions": "1. What is the training time for each experiment?\n\n2. After training, can the model be used for classifier / classifier-free guidance and diffusion SDEs?\n\n==========\n\nThanks for the detailed reply! After reading the authors' rebuttal, I raised my score to 6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is easy to understand and the writing is clean and easy to follow.\n- The proposed adjoint method is novel to the diffusion model community and the combination with exponential integrator is useful.\n- The studied topic is important to the field.", "weaknesses": "- Major:\n\n  - The proposed method seems to be quite **inefficient** because it needs to optimize the model / sample at each specific task, while other guided sampling methods (e.g., classifier guidance or classifier-free guidance) do not. Note that the optimization procedure needs to solve the whole ODE at each training step, the training cost (i.e., total training time) seems to be quite expensive.\n\n  - The proposed method cannot guarantee the property of diffusion models, i.e., the noise-pred network corresponds to the score functions, because it directly train the neural ODE. Thus, it may be hard to leverage the other properties of diffusion models, such as classifier / classifier-free guidance, and it may be hard to further use diffusion SDEs for better sample quality. Instead, other guided sampling methods introduce another guidance model at each time step, which is based on the score functions and thus can maintain the diffusion property.\n\n- Minor:\n\n  - Equation(9) is exactly the EDM sampler so it is not a new method. It should be compared and discussed in detail.\n\n[1] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2GVGf7QNJU", "reviewer": "Reviewer_CqXY", "summary": "This paper...\n- proposes AdjointDPM for differentiating through the diffusion sampling process,\n- reparametrizes PF ODE and augmented ODE to reduce numerical errors,\n- applies AdjointDPM to a wide variety of tasks, such as vocabulary expansion, security auditing, stylization, etc.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- This paper is a nice application of adjoint sensitivity methods to diffusion models. To the best of my knowledge, such application of adjoint sensitivity methods has not been explored before.\n- The proposed method can potentially be applied to a wide variety of downstream tasks for diffusion models.", "weaknesses": "I must clarify that I am not very familiar with application of diffusion models to tasks such as vocabulary expansion, security auditing, etc. Hence, I am not sure whether the authors have chosen an appropriate and comprehensive set of baselines, or have followed proper evaluation protocol. So, my current score for this paper is \"marginally above the acceptance threshold\", and I will adjust my score based on other reviews and authors' reply to my concerns.\n\n- NFE and wall-clock time for AdjointDPM and the baselines is missing, so it is difficult to gauge the efficiency of AdjointDPM.\n- A background section explaining and comparing other related backpropagation methods, such as DOODLE, FlowGrad, DEQ_DDIM in detail would help readers understand the position of AdjointDPM w.r.t. previous work. Specifically, this paper lacks a discussion of theoretical and practical advantages/disadvantages of AdjointDPM w.r.t previous work on backpropagation through diffusion sampling. For instance, the authors state DEQ-DDIM require the diffusion sampling process to have equilibrium points -- is this a significant drawback? Are there certain tasks where this equilibrium assumption do not hold, so AdjointDPM is applicable while DEQ-DDIM is not?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)