AdjointDEIS: Efficient Gradients for Diffusion Models
Authors: Zander W. Blasingame
Chen Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: continuous adjoint equations
neural differential equations
neural ODEs
adjoint sensitivity method
diffusion models
guided generation
Primary area: diffusion_based_models
TL;DR: We propose a novel family of bespoke ODE solves for solving the continuous adjoint equations for diffusion models.

Abstract: The optimization of the latents and parameters of diffusion models with respect to some differentiable metric defined on the output of the model is a challenging and complex problem. The sampling for diffusion models is done by solving either the *probability flow* ODE or diffusion SDE wherein a neural network approximates the score function allowing a numerical ODE/SDE solver to be used. However, naive backpropagation techniques are memory intensive, requiring the storage of all intermediate states, and face additional complexity in handling the injected noise from the diffusion term of the diffusion SDE. We propose a novel family of bespoke ODE solvers to the continuous adjoint equations for diffusion models, which we call *AdjointDEIS*. We exploit the unique construction of diffusion SDEs to further simplify the formulation of the continuous adjoint equations using *exponential integrators*. Moreover, we provide convergence order guarantees for our bespoke solvers. Significantly, we show that continuous adjoint equations for diffusion SDEs actually simplify to a simple ODE. Lastly, we demonstrate the effectiveness of AdjointDEIS for guided generation with an adversarial attack in the form of the face morphing problem. Our code will be released on our project page [https://zblasingame.github.io/AdjointDEIS/](https://zblasingame.github.io/AdjointDEIS/)

AdjointDEIS: Efficient Gradients for Diffusion Models
Zander W. Blasingame Clarkson University bl blasinzw@clarkson edu
Chen Liu Clarkson University cliu@clarkson. edu
Abstract
The optimization of the latents and parameters of diffusion models with respect some differentiable metric defined on the output of the model a challenging and complex problem. The sampling for diffusion models is done by solving either the probability flow ODE or diffusion SDE wherein a neural network approximates the function allowing numerical ODE/SDE solver be used. However, naïve backpropagation techniques are memory intensive, requiring the storage all intermediate states, and face additional complexity in handling the injected noise from the diffusion term of the diffusion SDE. We propose a novel family of bespoke ODE solvers the continuous adjoint equations for diffusion models, which we call AdJoinDEEIS. We exploit the unique construction of diffusion SDEs further simplify the formulation of the continuous adjoint equations using exponential integrators. Moreover, we provide convergence order guarantees for bespoke solvers. Significantly, we show that the continuous adjoint equations for diffusion SDEs actually simplify to a simple ODE. Lastly, we demonstrate the effectiveness of AdjoinDDEIS for guided generation with an adversarial attack in the form of the face morphing problem. Our code will be released at https: //github. thub. com/ zblas wnw.aaiingae////m.m AdjointDEIS.
Introduction
Diffusion models are large family of state-of-the-art generative models which learn to map samples drawn from Gaussian white noise into the data distribution [1, 2]. These diffusion models have achieved state-of-ther the-tte-rrt performance on prominent tasks such as image generation [3-5], audio generation [6, 7], video generation [8]. Often, of-te-of-the-arr models them is prohibitively expensive [9]. As such, it is fairly common to adapt a pre-trained model to a specific task for post-training. In this way, the generative model can learn concepts, identities, tasks without having to train the entire model [0-12]. Additional work has also proposed algorithms for guiding the generative process of diffusion models [13, 14].
quite large and training
One method guiding or directing the generative process to solve an optimization problem w.r.t. some guidance function L defined in the image space Rd This guidance function works on the output of the diffusion model and assesses how "good" the output is. However, the diffusion model works by iteratively removing noise until clean sample is reached. As such, need to be able to efficiently backpropagate gradients through the entire generative process. As Song et al. [15] showed, the diffusion SDE can be simplified to associated ODE, and such, many efficient ODE/SDE solvers have been developed for diffusion models [16-18]. However, naively applying backpropagation the diffusion model is inflexible and memory intensive; moreover, such an approach is not trivial to apply to the diffusion models that used an SDE solver instead of an ODE solver.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Diffusion Sampling
GB
Go
Xty
XtN-1
X10
EiN
Et1
AdjointDEIS
6/
60
L
X
Xig
Xig
ac
Ux
2x (LM)
ax (41)
axx (To)
Figure 1: A high-level overview of the AdjoinIDEIS solver to the continuous adjoint equations for diffusion models. The sampling schedule consists of { timesteps for the diffusion model and {fn }M timesteps for AdjointDEIS. The gradients ax (T) be used optimize Xy to find some optimal x7.
1.1 Contributions
Inspired by the work of Chen al. [19] we study the application of continuous adjoint equations to diffusion models, with a focus training-free free guided generation with diffusion models. We introduce several theoretical contributions and technical insights both improve the ability perform certain guided generation tasks and to gain insight into guided generation with diffusion models.
First, we introduce AdjoinIDEIS bespoke family of ODE solvers which efficiently solve the continuous adjoint equations for both diffusion ODEs and SDEs. Moreover, we show that the continuous adjoint equations for diffusion SDEs simplify to mere ODE. Next, we show how to calculate the continuous adjoint equation for conditional information which evolves with time (rather than being constant). To the best of our knowledge, we the first consider conditional information which evolves with time for neural ODEs. Overall, multiple theoretical contributions and technical insights are provided bring new family of techniques for the guided generation of diffusion models, which we evaluate experimentally on the task of face morphing.
1.2 Diffusion Models
In this subsection, we provide a brief overview of diffusion models. Diffusion models learn generative process by first perturbing the data distribution into an isotropic Gaussian by progressively adding Gaussian noise to the data distribution. Then a neural network is trained to perform denoising steps, allowing for sampling of the data distribution via sampling of a Gaussian distribution [2, 9]. Assume that we have an d-dimensional random variable x E Rg with some distribution Pdata (x). Then diffusion models begin by diffusing Pxama() according the diffusion SDE [2], an Itō SDE given
dx. = f()xx, dt + g(t) dw,
(1.1)
where []. denotes time with fixed constant
O, f(:) and g(-) denote the drift and diffusion coefficients, and w, denotes the standard Wiener process. The trajectories of follow the distributions pc(x,) with (xx) = Pdma(x) and px(x)) x A' I). Under some regularity conditions Song et al. [15] showed that Equation (1.1) has reverse process as time runs backwards from to with initial marginal distribution pr (xr) governed by
dx. [f(t)x+ -9 (t) log)) dt + gt) dw,
(1.2)

---

where W, is the standard Wiener process time runs backwards. Solving Equation (1.2) is what allows diffusion models draw samples from Pasm( by sampling pr (xTx)) The unknown term in Equation (1.2) is the score function Vx (og)((x)) which in practice is modeled by neural network that estimates the scaled score function, 6n (x;, R - of Vx log pr (X+), or some closely related quantity like Xo xo-prediction [1, 2, 20].
Probability Flow ODE. The practical choice of step size when discretizing SDEs is limited by the randomness of the Wiener process as large step size, i.e., small number of steps, can cause non-oonvergence, particularly in nigh-dimensional spaces [16]. Sampling an equivalent Ordinary Differential Equation (ODE) over an SDE would enable faster sampling. Song et al. [15] showed there exists an Ordinary Differential Equation (ODE) whose marginal distribution at time is identical to that of Equation (1.2) given as
dx;
1
= f(t)xx (t) llg Px
(1.3)
dt
The ODE in Equation (1.3) is known as the probability flow ODE [15]. As the noise prediction network, (x,,, t), is trained to model the scaled score function, Equation (1.3) can be parameterized as
dx: [(()xx+ +
g²()) 20, (x1,()
(1.4)
dt
w.r.t. the noise prediction network. For brevity, we refer to this as a diffusion ODE.
Although there exist several popular choices for the drift and diffusion coefficients, we opt to use the de facto choice which is known as the Variance Preserving (VP) type diffusion SDE [1, 15, 21J. The coefficients for VP-type SDEs are given as
d log Ore dt
do? d log 0t
f(t)
g²(()=
dt 2 dt 0::
(1.5)
which corresponds to sampling x, from the distribution q(x, xo) = (a o2
Adjoint Diffusion ODEs
Problem statement. Given the diffusion ODE in Equation (1.4), wish to solve the following optimization problem:
[ 0 f()x, + 200 20,
arg min L XT + xx,200
Ex 2, )1.
(2.1)
Le., seek to find the optimal XT, z, and 0 that satisfies our guidance function L. N.B., the noise-prediction model is conditioned on additional information z.
Unlike GANs which update the latent representation through GAN inversion [22, 23], in Equation 1) diffusion models require more care as they model an ODE or SDE and require numerical solvers. Therefore, update the latent representation, model parameters, and conditional information, we must backpropagate the gradient of loss defined on the output, OL(x)) through the entire ODE or SDE.
key insight of this work the connection between the adjoint ODE used in neural ODEs by Chen et al. [19] and specialized ODE/SDE solvers by [16-18] for diffusion models. It has been well observed that diffusion models are a type of neural ODE [15, 24]. Since a diffusion model can be thought of neural ODE, can solve the continuous adjoint equations [25] to find useful gradients for guided generation. We can then exploit the unique structure of diffusion models to develop efficient bespoke ODE solvers for the continuous adjoint equations.
Let fa describe parameterized neural field of the probability flow ODE, i.e., the R.H.S of Equa- tion (1.4), defined as
(2(1)
cx f (x., = f(1), + En Xt. 2, l).
(2.2)
20:
Then (x(, 2, 1) describes neural ODE which admits an adjoint state, ax == aL/0x, (and likewise for (t) and as (t)), which solve the continuous adjoint equations [25, Theorem 5.2] in the form of
3

---

the following Initial Value Problem (IVP):
ac
dax (1) ax (1)
Of (xx, t) Öx,
(0)
0xo
dt
dag (t) = 1---yy, ax ax())) Of (x,, z,t)
ag (0) = 0,
dt
Oz
das
dt (1) = (1) 05 (Xx, 2, 1)
ag (0) = 0:
(2.3)
00
We refer this system of equations in Equation (2.3) as the adjoint diffusion ODE it describes the continuous adjoint equations for the empirical probability flow ODE.
N.B., in the literature of diffusion models, the sampling process is often done in reverse time i.e., the initial noise is xT and the final sample is Xo- Due to this convention, solving the adjoint diffusion ODE backwards actually means integrating forwards in time. Thus, while diffusion models learn to compute x, from X, with 8 4, the adjoint diffusion ODE seeks to compute ax (x)) from ax (1).
2.1 Simplified Formulation of the Empirical Adjoint Probability Flow ODE
We show that rather than treating fa as a black box. the specific structure of the probability flow ODE is carried over to the adjoint probability flow ODE, allowing the adjoint probability flow ODE to be simplified into a special exact formulation.
By evaluating the gradient of fo w.r.t. X2 for each term in Equation (2.2) we rewrite the adjoint diffusion ODE for (t) in Equation (2.3) as
dax (t) f()
g2(t) (t) dep (x),
(2.4)
dt
20.
0x,
Due to the gradient of the drift term in Equation (2.4), further manipulations are required to put the empirical adjoint probability flow ODE into a sufficiently "nice" form. We follow the approach used by [16, 18] to simplify the empirical probability flow ODE with the use of exponential integrators and change of variables. By applying the integrating factor exp (So ((r) dr) to Equation (2.4), we find:
d d f(t) de axx = eSo ((F) (2) (1)
(x.,
(2.5)
dt
20
8x.
Then, the exact solution at time given time t < ssi is found to be
ax (s) elt [(T) dr ax(t)
el. f(r) dr 92 (2(
ax(u) T Eo (xx, du.
(2.6)
20
Fx,
lineur
nom lineur
Like with solvers for diffusion models which leverage exponential integrators, we are able to transform the adjoint diffusion ODE into a non-stiff form by separating the linear and non-linear component. Moreover, we can compute the linear in closed form, thereby eliminating the discretization error in the linear term. However, we still need approximate the non-linear term which consists of difficult integral about the complex noise- -reedicrein model. This is where the insight of Lu et al. [16] to integrate the log-SNR domain becomes invaluable. Let X == log(a,/ (a.//a)) be one half of the Iog-SNR. Then, with using this new variable and computing the drift and diffusion coefficients in closed form, we can rewrite Equation (2.6) as
ax (s)
at - ax (x(()))
Cs 1 L." auou
dyp du
ax(")) Ox.
Cx(Xu, u) du.
(2.7)
As Ae is a strictly decreasing function w.r.t. it therefore has an inverse function tx that satisfies (A.) = t, and, with abuse of notation, we let Xλ (x)))). (A) == (tx (t)),,,,, &c. and let the reader infer from context if the function is mapping the log-SNR back into the time domain or already in the time domain. Then by rewriting Equation (2.7)
exponentially weighted integral and performing an analogous derivation for ag (t) and ag (t), we arrive at the following.
A more precise and technical name would be the empirical adjoint probability flow ODE; however, for brevity's 's sake we prefer the adjoint diffusion ODE and leave the reader to infer from context that this describes empirical model with learned score function.

---

Proposition 2.1 (Exact solution of adjoint diffusion ODEs), Given initial values (ax(t), az a at time E (O, the solution (s) as(s), ag (s(s) time s E (t, T) of adjoint diffuusion ODEs in Equation (2.3) is
ai xx(())
1 hmm ase age=Axx( (2)
1
dea (xo(xx,2,)) dd,
ax
+
(2.8)
0ss
ass
Ux.
)Eo(Xx, 1) dd,
dEG
az (s) =t,(t)
age ax
(2.9)
Oz
%% are pae 2x 2)
(s))) ao(() +
dd.
(2.10)
00
The complete derivations of Proposition 2.1 can be found in Appendix B.I.
There is nice symmetry between Equations (2.8) to (2.10), while the adjoint of the solution trajectories evolves with a weighting of at ag in the linear term and the integral term is weighted by af //d / reflecting the double partial ox, in the adjoint and Jacobian terms. Conversely, the adjoint state for the conditional information and model parameters evolves with no weighting on the linear term and the integral is only weighted by az/a,- This follows from the vector fields being independent of ag and ag. These equations, while reflecting the special nature of this formulation diffusion models, also have appealing parallel with the exact solution for diffusion ODEs Lu al. [16, Proposition 3.1].
2.2 Numerical Solvers for AdJointDEIS
The numerical solver for the adjoint empirical probability flow ODE, now in light of Equation (2.8), only needs to focus on approximating the exponentially weighted integral of Eş from A to A. well-studied problem in the literature on exponential integrators [26, 27]. To approximate this integral, we evaluate the Taylor expansion of the Jacobian vector product to further simplify the ODE. For notational convenience let V(x; t) denote the scaled vector- Jacob-Jacobian product of the adjoint state ax (t) and the gradient of the model w.r.t. x,, i.e.,
DEp (x,, z,
V(x; = aiax(1) (1)
(2.11)
Ox,
and likewise we let (n) (x; A) denote the n-th derivative w.r.t. A. For 1, the (A: 1)-th Taylor expansion at d is
Σ (A Ax)
V(x;A)
(x; + - A1)²))
(2.12)
Plugging this expansion into Equation (2.8) and letting h = 02 de yields
ax (s)
Ct Oi axx
Σ
vin (x; Ae)
-->1)" dx+
O(h
(2.13)
O4 n=0
n!
Linear Exactly computed
Derivatives Approximated
Coeflicients
Higher-urder Omitted
Analytically computed
With this expansion, the number of terms which need to be estimated is further reduced as the exponentially weighted integral fo (A- e dA can be solved analytically by applying times integration-by- bynn--b--rttss [16, 28]. Therefore, the only errors in solving this ODE occur in the approximation of the n-thh order total derivatives of the vector-Jacobian product and the higher-order error terms Oh++)). By dropping the O(h++1) eтror term and approximating the first (k: - 1) derivatives of the vector-Jacobian product, we can derive k-th order solvers for adjoint diffusion ODEs. We decide to name such solvers as Adjoint Diffusion Exponential Integrator Sampler (AdjointDEIS) reflecting our use of the exponential integrator to simplify the ODEs and pay homage to DEIS from [18] that explored the use of exponential integrators for diffusion ODEs. Consider the case of 1, by dropping the error term (hh2) we construct the AdjointDHIS- solver with the following algorithm.
5

---

AdjointDEIS-1. Given an initial augmented adjoint state (t). ag (t), ao(t)] at))) at time t E (0, T), the solution |ax (5), az(s). a at time (t, 7 is approximated by
ax(s) = at
=x(()) (1) + 0x(6* -
or
ax (1)
DEB Xr, 2,
as
ax, deg (Xe, (Xe, 2,
a2(s) = =z((t) + oz(e" -1) -1) ax (t(
a1
ass
Oz dEp (x: 11,1)
(s =oo(t) +0((c* -1) ax (t)
at
(2.14)
as
00
Higher-order expansions of Equation (2.12) require estimations of the n-th order derivatives of the vector Jacobian product which can be approximated via mulli-step methods, such as Adams-Bashfone methods [29]. This has the added benefit of reduced computational overhead, as the multi-step method just reuses previous values to approximate the higher-order derivatives. Moreover, multi-step methods are empirically more efficient than single-step methods [29]. Combining the Taylor expansions in Equation (2.12) with techniques for designing multi-step solvers, we propose novel multi-step second-order solver for the adjoint empirical probability flow ODE which we call Adjoin(DEIS- This algorithm combines the previous values of the vector Jacobian product at time t and time to predict a, without any additional intermediate values.
AdjointDEIS-2M. We assume having previous solution ax and model output e (x,,,. r) at time < < 8, let p denote p x -r Then the solution a, at time to Equation (2.4) is estimated
be
ax(+)
Qt ax(1)
+ a
1)
ar
(1)
DEa dxa( 2,
Oxs
Ox,
+ 0x ch -
a
(xo 2,
og a ax (r)
Ox,, dea (x,(xx,2)11)) 2,1)
ax (t)
(2.15)
2p
a
0x.
For brevity, we omit the details of the AdjoinDDEIS-2M solver for at and a²(t); rather, we provide the complete derivation and details in Appendix B. Likewise, the full algorithm can be found in Appendix G.I. The advantage of a higher-order solver is that is generally more efficient, requiring fewer steps due to its higher convergence order. We show that AdjoinDEIS-K & is k-th order solver, as stated in the following theorem. The proof Appendix C.
Theorem 2.1 (AdioinDEIIS-k
k-th order solver). Assume the function E ( 2, and its associated vector-Jacobian products follow the regularity conditions detailed in Appendix C, then for k = 1, 2, AdjointDEIS-k is k-th order solver for adjoint diffusion ODEs, i.e., for the sequence (āx J computed by AdjointDEIS-A k, the global truncation error at time T satisfies ā (e(ta)) ax (T) (hmaz), where hmazz maxi <jSM (dz, >.t_..) Likewise, AdjoiniDEIS-K a k-th order solver for the estimated gradients w.r.f. Z and O.
As previous work has shown that higher-order solvers may be unsuitable for large guidance scales [16- 18 88] we do explicitly construct or analyze any solvers for k > 2 and leave such explorations for future study.
2.3 Scheduled Conditional Information
Thus far, we have held the conditional information constant across time, i.e., at each time / E [0, T] the conditional information supplied to the neural network is z. What if, however, we had some scheduled conditional information z,? We show that with some mild assumptions, using scheduled conditional information z; does not actually change the continuous adjoint equation for Z, from the equations derived from sans substitution of z, with
While motivated by the case of scheduled conditional information in guided generation with diffusion models, this result applies to neural ODEs more generally, which could open future research directions. We state this result more formally in Theorem 2.2 with the proof in Appendix D. Note, as this applies more generally than to just AdjoinDEESS so we express this result for some arbitrary neural ODE with vector field f (x,, 2,, 1) and use the forward-time flow convention rather than the reverse-time convention of diffusion models.

---

Theorem 2.2. Suppose there exists function z [0. + RR' which can be defined as cådlåg²
piecewise function where z is continuous on each partition of [0. T given by II {0 to tj <
bn = 3} and whose right derivatives exist for all t [0, Let fa R R + R be
continuous in t, uniformly Lipschitz in x, and continuously differentiable in x. Let x R + R be the unique solution for the ODE
dx (t) = fx Zt,
dt
with initial condition x(0) = Xy. Let C R - R be a scalar-valued loss function defined on the output of the neural ODE. Then ÖL (z)))(t)) (t) and there exists a unique solution az ER - R to the following IVP:
daz (t) ax =x) (t))TO
05 (x,, Zz,
ag (T) =0
dt
Ozi
3 Adjoint Diffusion SDEs
As recent work [30, 31] has shown, diffusion SDEs have useful properties over probability flow ODEs for image manipulation and editing. In particular, it has been shown that probability flow ODEs are invariant in Nie et al. [31, Theorem 3.2] and that diffusion SDEs are contractive in Nie et al. [31, Theorem 3.1], i.e., any gap in the mismatched prior distributions pe (Xx) and (x(x)) for the true distribution Pt and edited distribution Pr will remain between po(x,) and (xo), whereas for diffusion SDEs the gap can be reduced between pi (xx) and pa (xx) as tends towards O. Motivated by this reasoning, we present a framework for solving the adjoint diffusion SDE using exponential integrators.
The diffusion SDE with noise prediction model is given by
dx,- [[()xx+ + (²(1) Ex 2. 1)] di g(() dw.,
(3.1)
ot
where 'dt' is an infinitesimal negative timestep. Note how the drift term of the SDE looks remarkably similar to the probability flow ODE sans missing factor of 1/2 in front of the noise prediction model. This is due to differing manipulations of the forward Kolomogorov equations-whic describe the evolution of p. (xx) used by Anderson [32] to derive the reverse-time SDE and later by Song et al. [15] to derive the probability-foon ODE. This connection is very important as it enables one to simplify the Adjoin(DEIS solvers for the adjoint diffusion SDE.
We show that for the special case of Stratonovich SDEs not depend on the process state X,, then the adjoint process has unique strong solution that evolves with what is essentially an ODE. Intuitively, this tracks as the stochastic term g(t) o dw; has nothing to do with x,. We state this observation somewhat informally in the following theorem. The proof can be found in Appendix E.
with a diffusion coefficient gt) which does
Theorem 3.1. Lei f Rdd x R- R2 be in Cp and g R ~> Rdxx be in Cb. Let L: Rd + R be vcalar-valued differentiable function. Let W. : [0,T] - IR_ be a u-diimessional Wiener process. Let
x: [0, 17 + Rd solve the Stratonovich SDE
dx, f(x, dt g(t) dw.,
with initial condition Xg. Then the adjoint process ax (t) = al(x)) is strong solution to the backwards-ig -nd--i-----e ODE
da)(t) (tx)) 0s 0x;
(x,, t) dt.
(3.2)
This is a boon for us, as diffusion models use only a mere scalar diffusion coefficient, gt).. Therefore, the continuous adjoint equations for the diffusion SDE just simplify to ODE. Not only that, but as mentioned before, the drift term of the diffusion SDE and probability flow ODE differ only by factor of 2 in the term with the noise prediction network. As only the drift term of the diffusion
2 Frenchh: continue à gauche, limite à detroite.
'The notation "o dw,' denotes Stratonovich integration which differs from integration in the Itô sense. More details on this are found in Appendix E.
7

---

SDE is used when constructing the continuous adjoint equations, it follows that the only difference between the continuous adjoint equations for the probability flow ODE and diffusion SDE is factor of 2. Therefore, the exact solutions are given by:
Proposition 3.1 (Exact solution of adjoint diffusion SDES). Given initial values [ax(0), (1))) a (() (1, T) of adjoint diffusion SDES
time & (0, T) the solution [ax (+), az (8), [s))] time
0t ax
2
[ age ax 2o(x).2,1) di,
ax(s)
(t)
+
(3.3)
as A
0x1
ax(s) az( +2 /
do
dea 2, 2, dd, Max (2)
aNE ExE-aan
(3.4)
On
/"a ONE ax
dEp(XA, 2,
a0(s) an(() +
dA.
(3.5)
00
Remark 3.1. While the adjoint diffusion SDES evolve with an ODE, the same cannot be said for the underlying state, xG. Rather this evolves with a backwards SDE (more details in Appendix E) which requires the same realization of the Wiener process used to sample the image as the one used in the backwards SDE.
3.1
Solving Backwards Diffusion SDEs
Lu et al. [17] propose the following first-order solver for diffusion SDEs
X x, 20, (x,; 8) 8) + a e2h 16s;
(3.6)
where 6, (O, I). To solve the SDE backwards in time, we follow the approach initially proposed
by Wu and Torre [33] and used by later works [31]. Given a particular realization of the Wiener
process that admits X, ~ NNa,,00 I) then for two samples and x, the noise 6g can be calculated by rearranging Equation (3.6) to find
X₁ x, + 20 1)c, 2,
(3.7)
o.Ve 2h
With this the sequence {er, J of added noises can be calculated which will exactly reconstruct the original input from the initial realization of the Wiener process. This technique is referred to as Cycle-SDE after the CycleDiffusion paper [33].
Related Work
Our proposed solutions can be viewed training-free method for guided generation. As active area of research, there have been several proposed approaches to the problem of training-free guided generation, which either dynamically optimize the solution trajectory during sampling [34-36], or optimize the whole solution trajectory [37-40]. Our solutions fall into the latter category of optimizing the whole solution trajectory along with additional conditional information.
While Nie et al. [37] explored the use of the continuous adjoint equations optimize the solution trajectories of diffusion SDEs they don't It consider the ODE case and make use of the special structure of diffusion SDEs to simplify the continuous adjoint equations as we did. Recent work by Pan et al. [39] explore the using continuous adjoint equations for guided generation but does not consider the SDE case and uses different scheme simplify the continuous adjoint equations. We provide more detailed comparison against these approaches and further discussion related methods in Appendix A.
Experiments
To illustrate the efficacy of our technique, examine application of guided generation in the form of the face morphing attack. The face morphing attack is a new emerging attack on Face
(ab) that Recognition (FR) systems. This attack works by creating singular morphed face image X.

---

(a) Identity a
(b) Face morphing with AdjoinDEES
(c) Identity
Figure 2: Example of guided morphed face generation with AdjoinDDEIS on the FRLL dataset.
Figure 3: Comparison of DiM morphs the FRLL dataset. From left to right, identity a, DiM-A, Fast-DiM, Morph-PIPE, AdjointDEIS (ODE), AdjointDEIS (SDE), and identity
(b) [41-43]. A successfully shares biometric information with the two contributing faces x6" and X
created morphed face image can trigger false accept with either of the two contributing identities in the targeted Face Recognition (FR) system, see Figure 2 for an illustration. Recent work in this space has explored the use of diffusion models to generate these powerful attacks [41, 44, 45J. All prior work on diffusion-basee face morphing used pre-trained diffusion autoencoder [46] trained on the FFHQ [47] dataset at a 256 x 256 resolution. We illustrate the use of AdjointDEIS solvers by modifying the Diffusion Morph (DiM) architecture proposed by Blasingame and Liu [41] to the AdJointDEIS solvers to find the optimal initial noise x and conditional Zab- The AdjointDEIS solvers are used to calculate the gradients with respect to the identity loss [45] defined as
LID d(Uabi Vo) + d(vas, vb), Laiss
|d(vab. Vo) d(Vab, 06))
(5.1) ((x(ab)) and F X V is FR system which embeds images into vector space V which is equipped with measure of distance, d. We used the ArcFace [48] FR system for identity loss.
Lip LID + Laiff:
where va F( v2 F(x
uob "eb
We compare against three preexisting DiM methods, the original DiM algorithm [41], Fast-DiM [44], and Morph-PiPE [45] as well
LAN-inversion--asee face morphing attack, MIPGAN-I and MIPGAN-II [49] based on the StyleGAN [47] and StyleGAN2 [50] architectures respectively. Fast- DiM improves DiM by using higher-order ODE solvers decrease the number of sampling steps required to create a morph. Morph-PIPP performs a very simple version of guided generation by generating large batch of morphed images derived from discrete set of interpolations between x(() and (b) and Z, and 2₁. For reference purposes, we compare against a reference
method [49] which uses GAN-inversion w.r.t.to the identity loss find the optimal morphed face, and we include prior state-of-ther Webmorph, a commercial off-the-shelf system [51].
We run our experiments SYN-MAD 2022 [51] morphed pairs that are constructed from the Face Research Lab London dataset [52], more details in Appendix H.4. The morphed images are evaluated against three FR systems, the ArcFace [48], ElasticFace [53], and AdaFace [54] models;

---

further details are found in Appendix H.5. To measure the efficacy of a morphing attack, the Mated Morph Presentation Match Rate (MMPMR) metric is5] is used. The MMPMR metric as proposed by Scherhag et al. [55] is defined as
{
M Σ
( M
S > 8
(5.2)
[ne.,... N..
m=1
where ò is the verification threshold, Sm is the similarity score of the n-th subject of morph m, Nm is the total number of contributing subjects to morph m, and M is the total number of morphed images.
In our experiments, we used a learning rate of 0.01, _N- = 20 sampling steps. M = 20 steps for AdjointDEIS, and 50 optimization steps for gradient descent. For the sampling process we used the DDIM solver [2]. a widely used first-order solver. Following [56] we observed that using recorded values of {x,, J.m for the backward pass improved performance. Note, this does not mean stored the vector-Jacobians or any other internal states of the neural network. Moreover, due to our use of Cycle-SDE this choice was mandated for the SDE case. We discussion this decision further in Appendix F.1.
Table 1: Vulnerability of different FR systems across different morphing attacks on the SYN-MAD 2022 dataset. FMR = 0.1%.
MMPMR(1)
Morphing Attack
NFE())
AdaFace
ArcFace
ElasticFace
Webmorph [S1]
97.96
96.93
98.36
MIPGAN-I [49]
72.19
77.51
66.46
MIPGAN-II [49]
70.55
72.19
65.24
DiM-A [41]
350
92.23
90.18
93.05
Fast-DiM [44]
300
92.02
90.18
93.05
Morph-PIPE [45]
2350
95.91
92.84
95.5
DiM AdjointDEIS-1 (ODE)
2250
99.8
98.77
99.39
DIM + AdjointDEIS-1 (SDE)
2250
98.57
97.96
97.75
In Table we present the effectiveness of the morphing attacks against the three FR systems. Guided generation with AdjointDEIS massively increases the performance of DiM, supplanting the old state-of-the-art for face morphing. Interestingly, the SDE variant did not fare as well as the ODE variant. This is likely due the difficulty discretizing SDEs with large step sizes [15-17]. We present further results in Appendix F that explore the impact of the choice of learning rate and the number of discretization steps for AdjointDEIS.
6 Conclusion
We present a unified view on guided generation by updating latent, conditional, and model information of diffusion models with guidance function using the continuous adjoint equations. We propose AdjointDEIS, a family of solvers for the continuous adjoint equations of diffusion models. We exploit the unique construction of diffusion models create efficient numerical solvers by using exponential integrators. We prove the convergence order of solvers and show that the continuous adjoint equations for diffusion SDEs evolve with ODE. Furthermore, that is scheduled in time, further expanding the generalizability of the proposed technique. Our results in face morphing show that the gradients produced by AdjointDEIS can be used for guided generation tasks.
show how to handle conditional information
There several limitations. Empirically, we only explored a small subset of the true potential AdjointDEIS by evaluating on a single scenario, i.e., face morphing. Likewise, we only explored a few different hyperparameter options. In particular, we did not explore much the impact of the number of optimization steps and the number of sampling steps for diffusion SDEs on the visual quality of the generated face morphs.
Limitations.
Broader Impact. Guided generation techniques can be misused for a variety of harmful purposes. In particular, our approach provides a powerful tool for adversarial attacks. However, better knowledge of such techniques should hopefully help direct research in hardening systems against such kinds of attacks.
10

---

Acknowledgments
The authors would like thank Fangyikang Wang for his helpful feedback the Taylor expansion of the vector Jacobians. The authors would also like to acknowledge the fruitful discussions with Pierre Marion and Quentin Berthet on using the adjoint methods from bilevel optimization perspective.
References
[1] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic mod- els. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 6840-6851, Curran Associates, Inc., 2020. URL https://proceddngs wwweiiinnnnnnuuureeeeeee neurips- cc/paperr2000/filee httpa//gnrrrrrpp...////////0000
pdf. 1,3,33
[2] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL https: /ttp///pppnree..eee//eee
net/forum? net/forum?id-Stlge id-SttdiarCHit. 1, 2, 3, 10
[3] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[4] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv e-prints, art. arXiv:2204.06122 April 2022. doi: 10448500axxi.22244.00222
[5] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Den- ton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Sali- mans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to- image diffusion models with deep language understanding. hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 36479-36494. Curran Associates, Inc., 2022. URL http::ttp//// httptpppp//prooeedingnssss neurips cttpcc//pppaeee..c seeppppe__aa files// wwles/paper//e////eeee ppapr/2022/Eilll pdf.
In S. Koyejo, S. Mo-
[6] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. AudioLDM: Text-to-Audis Generation with Latent Diffusion Models. arXiv e-prints, art. arXiv:2301.12503 Jamuary 2023. doi: 10.48550/arXiv.230111 125033
[7] Scott H. Hawley. Pictures of midi: Controlled music generation via graphical prompts for image-based diffusion inpainting, 2024. URL https: //apivv////iiv////aaeeaaeaa org/ harg//bs//e abs/240/ 01499. 1
[8] Andreas Blattmann, Robin Rombach, Huan Ling. Tim Dockhorn. Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models. arXiv e-prints, art. arXiv:2304.088 April 2023. doi: 10,485500arXiv.304. 08818.
[9] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Proc. NeurIPS, 2022. 1,2
[10] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. /2242, 2022.
[11] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. arXiv e-prints, art. arXiv:2208.01611 August 2022. doi: 10.48550/ MrXiv.228.00615
[12] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106 09685, 2021.
11

---

[13] Jonathan Ho and Tim Salimans. Classifier--ree diffusion guidance. In Neur/PS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL aattps://ppenrvieee net/forum? dddermmm=ddduuuuxxfx.y.b I
[14] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal Guidance for Diffusion Models. arXiv e-prints, art. arXiv:2302.07122 February 2023. doi: 21.S500/aXXi.2230207711
[15] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https: https://openre/// /openeevie..
net/forumiid 1d=-xTIG12RRRREE 1, 2, 3, 7, 10, 18, 33
[16] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan LI, and Jun Zhu. Dpm- solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems, volume 35, pages 5775-5787. Curran Associates, Inc., 2022. URL http:: /ttpttp:::///cooe https: pttptpeee///ooooeeeeenggg newereeeeddddnnnnnnneeooeooop.p neurips cc/ pww.repp...ppppprereeeeeeec..eee files/ wttpeeeeiiieespppp////////////// pdf. 3, 4, 5, 6, 20, 21, 22
[17] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models, 2023. 8, 10, 23
[18] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In International Conference on Learning Representations, 2023, 1, 5,
[19] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL https: htttps::/proceeedinggg mew.eeeeeeddinneeeeee.see....ep neurips pappe__filesppappppppppppapleeep 6bb
pdf. 2,
[20] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models.
In International Conference Learning Representations, 2022. URL /ttttps://ppprreeeeee
net/forum? -d-TIddXIZThhhhooooo 3
[21] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Curran Associates Inc., Red Hook, NY, USA, 2019. 3
[22] R. Abdal, Y. Qin, and P. Wonka. Image2stylegan: How to embed images into the stylegan latent space? In IEEE/CVE Int'l Conf. Comp. Vision (ICCV), pages 4431-4440, 2019. doi: 20.100//CC222219.0000
3
[23] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embed- ded images? In Proceedings of the IEEE/CVF conference on computer vision and pullern recognition, pages 8296-8305, 2020. 3
[24] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023. URL https: http:///peeeeeeeeeeeeee wwt. id PQVMRDCJTTt..
[25] Patrick Kidger. On Neural Differential Equations. PhD thesis, Oxford University, 2022. 3, 28
[26] Marlis Hochbruck and Alexander Ostermann. Exponential integrators. Acta Numerica, 19:
209-286, 2010. doi: 10. (0.10177/5062299910000000
5, 20
[27] Iyabo Ann Adamu. Numerical approximation of SDEs & the stochastic Swift-Hohenberg equation. PhD) thesis, Mleriot-Watt University, 2011.
[28] Martin Gonzalez, Nelson Fernandez Pinto, Thuy Tran, elies Gherbi, Hatem Hajri, and Nader Masmoudi. Seeds: Exponential sde solvers for fast high-quality sampling from diffusion models. In A. Oh, T. Naumann, A. Globerson, Saenko, M. Hardt, and Levine, editors, Advances in
Neural Information Processing Systems, volume 36, pages 68061 10661-61120 120. Curran Associates, Inc., 2023. URL htwp::ttt///// httpappps////occeeeeeeee nww.ips.cc. hatec/pper__ffee files/ httpa//eeeeeaapeee////////////// /ppper/2023/filee Conferenceeeeeee
d6f 0B87744.e888000f3288888777711110 Sorereeeeereeerrrreeeeeeeeeeeeee pdf.
12

---

[29] K. Atkinson, W. Han, and D.E. Stewart. Numerical Solution of Ordinary Differential Equations. Pure and Applied Mathematics: A Wiley Series of Texts, Monographs and Tracts. Wiley, 2011. ISBN 97811 118164525. URL. https: //bookss////oo.....// google e. com com/books Q 6
[30] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
[31] Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng. and Chongxuan Li. The blessing of randomness: SDE beats ODE in general diffusion- based image editing. In The Twelfth International Conference on Learning Representations, 2024. URL https:
//openreview. wet/ffoumt disimmddddDDeeOOODD 7,8
[32] Brian D,O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326 1982. ISSN 0304-4149. doi: https://doi.org/10..t 0304-4149(82)9005115. URL https:/ http///wwwcceenceieeee... com ht////cincc/arrrcc// p11/00304414982005515 7
[33] Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero-shot image editing and guidance. In ICCV, 2023.
[34] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free energy-guided conditional diffusion model. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 8, 16
[35] Zander W. Blasingame and Chen Liu. Greedy-dim: Greedy algorithms for unreasonably effective face morphs. In 2024 IEEE International Joint Conference on Biometrics (IJCB), pages 1-10, September 2024. 16, 17,31
[36] Xingchao Liu, Lemeng Wu, Shujian Zhang, Chengyue Gong, Wei Ping, and Qiang Liu. Flow- grad: Controlling the output of generative odes with gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24335 24335-24344, 2023. 8, 16
[37] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. In International Conference on Machine Learning (ICML), 2022. 8, 16, 17
[38] Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimization improves classifier guidance, 2023. 16, 17
[39] Jiachun Pan, Jun Hao Liew, Vincent Tan, Jiashi Feng, and Hanshu Yan. AdjoinDPPM: Adjoint sensitivity method for gradient backpropagation of diffusion probabilistic models. In The Twelfth International Conference on Learning Representations, 2024. URL https::/
net/forumidd -3y31IDDRBW. 8, 16, 17
[40] Pierre Marion, Anna Korba, Peter Bartlett, Mathieu Blondel, Valentin De Bortoli, Arnaud Doucet, Felipe Llinares-López, Courtney Paquette, and Quentin Berthet. Implicit diffusion: Efficient optimization through stochastic sampling. arXiv preprint arXiv:2402.05-463 2024. 8, 16, 18
[41] Zander W. Blasingame and Chen Liu. Leveraging diffusion for strong and high quality face morphing attacks. IEEE Transactions on Biometrics, Behavior, and Identity Science, 6(1): 118-131, 2024. doi: 00.119//B10M.222444499577 9, 10, 31.32
[42] R. Raghavendra, K. B. Raja, and C. Busch. Detecting morphed face images. In IEEE 8th Ini'l Conf. on Biometrics Theory, Applications and Systems (BTAS), pages 1-7, 2016. doi: 10.1 2L1109/BAA..2216.791111
[43] Eklavya Sarkar, Pavel Korshunov, Laurent Colbois, and Sébastien Marcel. Are gbn-d morphs threatening face recognition? In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2959-2963, 2022. doi: 10.1 109/ CCASSP44992..222297446664
32
13

---

[44] Zander W. Blasingame and Chen Liu. Fast-dim: Towards fast diffusion morphs. IEEE Security & Privacy, 22(4):103-114, June 2024. doi: 10.100/MSEE 9,
[45] Haoyu Zhang, Raghavendra Ramachandra, Kiran Raja, and Busch Christoph. Morph-pipe: Plugging in identity prior to enhance face morphing attack based diffusion model. In Norwegian Information Security Conference (NISK), 2023, 9, 10, 32
[46] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakom. Diffusion autoencoders: Toward meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10619-10629, 10629, June 2022.
[47] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396-4405, 2019. doi: 10.1 10.119/CVPR.221..0453..
[48] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1990-4699, 2019. 9,
[49] Haoyu Zhang, Sushma Venkatesh, Raghavendra Ramachandra, Kiran Raja, Naser Damer, and Christoph Busch. M generating strong and high quality morphing attacks using identity prior driven gan. IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(3): 365-383, 2021. doi: 10.1 109/TBB0...021.0777777
9, 10, 32
[50] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8107-8116, 2020. doi: 10.1 1..1099CCR44260022000.000
9
[51] Marco Huber, Fadi Boutros, Anh Thi Luu, Kiran Raja, Raghavendra Ramachandra, Naser Damer, Pedro C. Neto, Tiago Gonçalves, Ana F. Sequeira, Jaime S. Cardoso, João Tremoço, Miguel Lourenço, Sergio Serra, Eduardo Cermeño, Marija Ivanovska, Borut Batagelj, Andrej Kronovšek, Peter Peer, and Vitomir Struc. Syn-mad 2022: Competition detection based on privacy-awar synthetic training data. In 2022 IEEE International Joins Conference on Biometrics (IJBB), pages 1-10, 2022. doi: 10. N0.1109/0CB542222222...00000
face morphing attack
9, 10, 32
[52] Lisa DeBruine and Benedict Jones, Face Research Lab London Set. 5 2017. doi: 10. 6084/m9. 6084/m9..fgsharee.500 50g55aree.5447666655555 v5. URL https: //f wfifggagrre... wtt./ooo///t/eeee aariclessddaaset//Eeee
Research_ LabarchhLLa_a__a Baaib___odoon__e Serrrdon_Set/004766666 9, 32
[53] Fadi Boutros, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper. Elasticface: Elastic margin loss for deep face recognition. In Proceedings of the IÉEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1578-1587, June 2022. 9, 32
[54] Minchul Kim, Anil K Jain, and Xiaoming Liu. Adaface: Quality adaptive margin for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 9, 32
[55] UIrich Scherhag, Andreas Nautsch, Christian Rathgeb. Marta Gomez-Barren Raymond N. J. Veldhuis, Luuk Spreeuwers, Maikel Schils, Davide Maltoni, Patrick Grother, Sebastien Marcel, Ralph Breithaupt, Raghavendra Ramachandra, and Christoph Busch. Biometric systems under morphing attacks: Assessment of morphing techniques and vulnerability reporting. In 2017 International Conference of the Biometrics Special Interest Group (BIOSIG), pages 1-7, 2017. 10
doi: 9/BI/SII..001...0...
[56] Suyong Kim, Weiqi Ji, Sili Deng, Yingbo Ma, and Christopher Rackauckas. Stiff neural ordinary differential equations. Chaos: An Interdisciplinary Journal of Nonlinear Science, 31(9), 2021.
10, 29
[57] Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 22532-225 2023. 17
14

---

[58] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33: (696-6707, 2020. 25
[59] Xuechen Li, Ting-Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference tics, volume 108 of Proceedings of Machine Learning Research, pages 3870-3882. PMLR, 26-28 Aug 2020. URL httph: SBRLttts///rroceedingg mlr www.:presssvvve ppwss/vv08///e 11t//112..i11 html. 26, 27
Artificial Intelligence and Statis-
[60] Hiroshi Kunita. Stochastic differential equations and stochastic flows. Stochastic Flows and Jump-Diffusions, pages 77-124, 2019. 26, 27
[61] Patrick Kidger, James Foster, Xuechen (Chen) Li, and Terry Lyons. Efficient and accurate gra- dients for neural sdes. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 18747- 18761. Curran Associates, Inc., 2021. URL httpt: httptttpppppppppppppppppeeeee/ee fi
ctcccc//ppa _
pdf.. 28
[62] Ionut Cosmin Duta, Li Liu, Fan Zhu, and Ling Shao. Improved residual networks for image and video recognition. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 9415 9415-9422, 2021. doi: 10.1 20tp://CPR4880.2222..411111
32
[63] Xiang An, Xuhan Zhu, Yuan Gao, Yang Xiao, Yongle Zhao, Ziyong Feng, Lan Wu, Bin Qin, Ming Zhang. Debing Zhang, and Ying Fu. Partial fc: Training 10 million identities on single machine. In 2021 IEEE/CVF International Conference Computer Vision Workshops (ICCVW), pages 1445-1449, 2021. doi: 10.1 10.1109/ 202.00000 1.0016.. 32
15

---

Organization of the appendix. In Appendix A we provide detailed comparison between our work and related work. Appendix provides the full derivations for the construction of the AdjointDEIS & solvers. Likewise, in Appendix C we present the proof for Theorem 2.1. In similar manner Ap- pendix presents the proof for Theorem 2.2. A more detailed discussion of the case of adjoint diffusion SDEs is held in Appendix E along with the proof of Theorem 3.1. We include additional experiments on the impact of AdjointDEIS on the face morphing problem in Appendix F. The details of the implementation of our approach are included in Appendix G. Additional details specific to our experiments are likewise included in Appendix H. Finally, for completeness, we include Appendix to show the closed-fome solutions of the drift and diffusion coefficients.
A Additional Related Work
In this section, we compare several recent methods for training- free guided generation which we broadly classify into two categories:
1. Techniques which directly optimize the solution trajectory during sampling [34-36]
2. Techniques which search for the optimal latents xy and or z (this can include optimizing the solution trajectory well) [38, 39].
In Table 2 we compare several different techniques for training-free free guided diffusion whether they explicitly optimize the whole solution trajectory, iee., optimizing xT, if they optimize additional information like z or 0, and whether the formulation is for diffusion ODEs, SDEs, or both.
Table Overview of different training- free guidance methods for diffusion models.
Method
ODE
SDE
Optimize
Optimize (z, 0)
FlowGrad [36]
X
X
FreeDoM [34]
X
/
x
Greedy [35]
/
J
X
DOODL [38]
/
X
V
DiffPure [37]
X
AdjointDPM [39]
X
V
Implicit Diffusion [40]
AdJointDEIS
Using Table 2 high-level overview, spend the rest of this section providing a more detailed comparison and discussion of the related works which we separate by the two categories.
A.1 Optimizing the Solution Trajectory During Sampling
FlowGrad. The FlowGrad [36] technique controls the generative process by solving the following optimal control problem
min L(xx) + [ [[((t) dt,
(A.1)
S.L. No = xT + [ fo(x,,2,1) + ull) de
=
(A.2)
where u is the control function. This optimization objective learns to alter the flow, fg. by u. In practice, this amounts to injecting control step governed by u(t) for discretized schedule. This technique does not allow for learning an optimal XT, z, or 0.
FreeDoM. Another recent work, FreeDoM [34] looks at gradient guided generation of images by calculating the gradient w.r.t. x, by using the approximated clean image
X, G,6 (X,, 2,t)
(A.3)
a1
16

---

at each timestep. Let h4 = Xe, -
The strategy can be described as
X1, at1-1 Xx;
201:
(e (xe: Xt, 2,,1) + ar, V 2h. let,
(A.4)
Ott:
X1, 0t. (x( t,tz)
80
(A.5)
at:
@c(x))
B: 8xt
(A.6)
Xt,-1 XX,1 Bt.
(A.7)
where ne, is learning rate defined per timestep. Importantly, FreeDoM operates diffusion SDEs. They have an addition algorithm which the add noise back to the image, in essence going back one timestep and applying the guidance step again.
Greedy. Similar to FreeDoM, greedy guided generation [35] looks to alter the generative trajectory by injecting the gradient defined on the approximated clean image; however, this technique does so w.r.t. the prediction noise, i.e. é, = stopgrad(Ea (Xr, t.))
(A.8)
No X o1.
(A.9)
Ot1;
aL(x))
g1. = deL.
(A.10)
4, = é, B):
(A.11)
The technique would work for either diffusion ODEs or SDEs.
A.2 Optimizing the Entire Solution Trajectory
DOODL. The algorithm DOODL [38] looks at the gradient calculation based on the invertibility of EDICT [57]. This method can find the gradient w.r.t. Xy; however, it cannot for the other quantities. DOODL additionally has further overhead due the dual diffusion process of EDICT. Further analysis of DOODL compared to continuous adjoint equations for diffusion models can be found in [39].
Table 3: Comparison of solvers for the continuous adjoint equations for diffusion models.
DiffPure [37]
AdjoinDDPM [39]
AdjointDEIS
Discretization domain
eg over /
Ep over p
Es over À Custom solver
Solver type Exponential Integrators
Black box SDE solver
Black box ODE solver
X
Closed form SDE coefficients
X
Interoperability with existing samplers
X
Decoupled ODE schedule
X
Supports SDEs
The remaining methods are much closer to our work as they use continuous adjoint equations for guidance. We provide high-level summary that compares these methods ours in Table 3.
DiffPure, Work by Nie et al. [37] examined the use continuous adjoint equations for cleaning adversarial images, application of guided generation. There exist few key differences between their work and ours. First, while they consider the SDE case they use an Euler-Maruyama numerical scheme, meaning there is discretization error incurred in the lincar term of the drift coefficient. Moreover, the solver for the continuous adjoint equation results in an Euler scheme which suffers from low convergence order and poor stability, particularly for stiff equations. Our approach used exponential integrators to greatly simplify the continuous adjoint equations whilst improving numerical stability by transforming the continuous adjoint equations into non-stiff form.
AdjointDPM. More closely related to our work is the recent AdjointDPM [39] which also explores the use of adjoint sensitivity methods for backpropagation through the probability flow ODE. While
17

---

a₂ (0) 0, they also propose to use the continuous adjoint equations to find gradients for diffusion models, our work differs in several ways which we enumerate Table 3. For clarity, we use orange denote their notation. They reparameterize Equation (1.4)
dy dp
= Ea(e
(0) (()) dr y, 7 (p).c)
(A.12)
= e ((r) (0 20
where с denotes the conditional information, (tt, and them the following ODE for calculating the adjoint.
which gives
de
ac T dea(eld t00 s(7) y 7 (p))
ac dp oy θy
(A.13)
0y
In our approach we integrate over X whereas they integrate over p. Moreover, provide custom solvers designed specifically for diffusion ODEs instead of using a black-box ODE solver. Our approach is also interoperable with other forward ODE solvers, meaning our AdjointDEIS solver is agnostic to the ODE solver used to generate the output; however, the AdjointDPM model is tightly coupled to its forward solver. Lastly and most importantly, method is more general and supports diffusion SDEs, not just ODEs.
Although the original paper omitted closed form expression for (p). we provide to give comparison between both methods, and to ensure that AdjointDPM can be fully implemented. In the VP SDE scheme with linear noise schedule log at is found to be
log 04 = 31 - 80 ,2 30
(A.14)
4
2
on t e [0, with Bu 0.1, 20, following Song et al. [15]. Then 7 is found to be
8g
88 + 4lg
(go B1)
2e (otto)2+1
y (p)
(A.15)
Bo 3
Implicit Diffusion. Concurrent work to ours by Marion et al. [40] has also explored the use of continuous adjoint equations for the guidance of diffusion models. They, however, focus efficient scheme to parallclize the solution to the adjoint ODE from the perspective of bi-level optimization rather than the adjoint technique itself. So, while we focused the numerical solvers for the continuous adjoint equations, they focused on an efficient implementation of the optimization problem from the perspective of bi-level optimization.
B Derivation of AdjointDEIS
In this section, we provide the full derivations for the family of AdjointDEIS solvers. First recall the full definition of the continuous adjoint equations for the empirical probability flow ODE:
ac
dax
0f (x,, 2,
(0) Ixo
(t)
-ax(0 (t)
dl
8x.
dag (t) ax(t)) (t)
Öf o (x1, 2, t)
Oz
dag (t) ax Of (x,, Z, t)
ag (0) 0,
(B.1)
00 We can simplify the equations by explicitly solving gradients of the neural vector field fa for the drift term to obtain
dax (t) f(t)ax(t)
g2(1)
dep (xx, 2, 1)
20, ax ax(t)
(t)
dt
Ox:
da (t)
(1)
T dEp (xe, 1)
0
a (t)
dt
20
Oz
dag
g2)
T DEB (x,, 2, t)
(t)
0 20:
(t)
(B.2)
df
08
18

---

Remark B.1. The last two equations in Equations (B.2) the vector fields are independent of (az. ad). reducing these equations to mere integrals; however, often useful compute the whole system augmented ODE.
Anug (ax; ag ag)
Remark B.2. Likewise, the last two equations in Equations (B.2) are functionally identical with simple swap of for 0 or vice versa.
As such, for the sake of brevity, the derivations for the AdjointDEIS solvers for (az, as) will only explicitly include the derivations for ag.
B.1 Simplified Formulation the Continuous Adjoint Equations
Focusing first on the continuous adjoint equation for ax we apply the integrating factor exp (Jo (r) dr) to Equation (B.2) to find
d
elo f(r) dr (t) = - -el f(r) dr (()
(x,,
20 ax())
(B.3)
dt
Oxi
Then, the exact solution at time given time & < 8 found to be
elō f(r) dr (8) edo f(z) xx(1)) (1) S. e5 f(T) dr g2(
(X u) ix.,
ax (1)
du
20 ax(s) eli f(z) ax J. e f(T) dr (2)
Eo (x
2, u)
20 ax (w)
du
(B.4)
Ox.
To simplify Equation (B.4), recall that f(t) is defined as
dl log Oz dt
f(L)
(B.5)
for VP type SDEs. Furthermore, let A == log(a,/ol /or) ) be one half of the log-SNR. Then the diffusion coefficient can be simplified using the log- derivavvve trick such that
do cl log a (1) = dt
[olog oe dt
llogoo at 220
ddde
dt of = 20
(B.6)
dt
dt
Using this updated expression of (t) along with computing the integrating factor in closed form enables us to express Equation (B.4) as
at ax (t)
dru (x
co( u)
ax
=
auTu du
Fxn du.
(B.7)
ag
Lastly, by rewriting the integral in terms of exponentially weighted integral and a2gu/au age Au we find
1 //x Ol A
Ex z, A)
ax ax(s)= = (t)
aa
ax())
dA.
(B.8)
OxX
This change of variables is possible as de is strictly decreasing function w.r.t. and therefore it has an inverse function tx which satisfies t (At) = t. and, with abuse of notation, we let X,, := (x)))) ax (A) = ax (12 (A))),, and let the reader infer from context if the function is mapping the log-SNR back into the time domain already in the time domain.
Now we will show the derivations find a simplified form of the continuous adjoint equation for the conditional information. Using the continuous adjoint equation from Equations (B.2) for a (t) along with the log- SNR, we can express the evolution of ag (t) as
da (t) = ot dt
dA, (x()) Oco(xi z,
(B.9)
dt
Oz
As we would like to express this exponential integrator, we simply multiply by ar/ar to obtain ag 02/0, = are Ai, as such we can rewrite Equation (B.9) as
da (t) are Ao dy, ax at())
@ee(xx,
(B.10)
dt
di
Oz
Using Equations (B.8) and (B.10), arrive Proposition 2.1 from the main paper.
19

---

Proposition B.1. Given initial values (tx, ag(t),ag(t) [t) at time t E (0. T), the solution (s), a,(s), ag()] time E (t, T" of the adjoint empirical probability flow ODE Equa- lion (B.4) is
ax(8) at ax(() +
1 Lm ase- ax (A)
dEO (XA, dA,
ass as
Ux.
(B.11)
deg deo 2, A)
(s(s) az(()
are ax())
axe
dd.
(B.12)
Oz
(8) an(() / are
(x.. a A) dA.
(B.13)
00
Then to find the AdjointDEIS solvers take k-th order Taylor expansion about X and integrate in the log-SNR domain.
B.2 Taylor Expansion
For h: 2 1, the (A: 1)-th Taylor expansion at X of the inner term the exponentially weighted integral in Equation (B.11) is
(A) dco(Xx, 2,
k-1
(A A.) d
dco 2, A)
a}ax(A)
+0(1--A0)
0xx
n!
dAn
OXA
n=0
A=de
(B.14)
Then plugging this into Equation (B.11) yields
ax(s)
at ax (t)
1
//
A, d"
a ax
(xx z, A)
dd
a2
Os
Ae
2=0
n!
dA
OX1
A=dr
+0(h+11)
d
(x, 2,
A (A- x1) c yP
(t)
(A)
ass
0ss
n=0
d
Ox,
x=
n!
estimated
analytically compated
(1 omitted
(B.15)
where h = X, Xt-
The exponentially weighted integral fe (A-Ae)" - dA can be solved analytically by applying times integration by parts [16, 26] such that
> c (1 A1)"
Ts pntt pn+ pn (h), dd
(B.16)
n!
as
with the special fefunnctooss [26]. These functions are defined as
Pn+1 (h) = S/' ell-u)h xt" n! du, sPo(h) =e",
(B.17)
which satisfy the recurrence relation (k+1(()) (px(h) - (0))/h and have closed forms for
k = 1, 2:
(hhh)) h
(B.18)
h
(2)h)) c h
(B.19)
h2
20

---

Likewise, the Taylor expansion of the exponentially weighted integral in Equation (B.12) yields
Az d
dEp(XX, 2 A)
ax(s) ax(() +
(xax(())
dd + O(h
d
On
A=
k
(xA,
(A \1)
= at) +
a1ax (A)
+ O(h
d
Oz
n!
de
estimated
analytically computed
emitted
(B.20)
B.3 AdjointDEIS-1
For h = and omitting the higher-order error term, Equation (B.15) becomes:
ax
(s)
at ax
(t)
a
(x))
(x,,
(A
-x)
Yd
ax
ag
8x.
Ae
0!
at ax (t) + 0z
dcgl
(x,, 2,
ax
(t)
By Equation (B.16).
(B.21)
as
ag
0x;
Likewise, the continuous adjoint equation for z, Equation (B.20), becomes when k = 1 by omitting the higher-order error term:
a²(s) = az (t) + a.ax(() (t) (c0(x.,22
S/m ~rr (A (A-Ap xx)
> dd
Oz
0!
(x,, 2, Oz And the first-order solver for ag (t) can found in a similar fashion, thus we have derived the AdjointDEIS-L solvers.
= (t) + (e - -1)
a1
(t)
By Equation (B.16).
(B.22)
0x
B.4 AdJOintDEIS-2M
Consider the following definition of the limit in the log-SNR domain
d a (gax(A) (A) Dea (xa z, A)
V(x; Ae) - V(x; Ar)
lim A
(B.23)
dxx
ph
where p Az with h Xa- - dr and where is some previous step < < Again, V(x; Ax) overloaded mean (Ax)) Then by omitting higher-order error O(h Equation (B.15) becomes:
Ot
v(x; / (A- (1- dd V() )c)
> (A- Yd /'
(x(5)
ax
(t)
+
O'
0!
A
at
0z - (x;(x; + Le (e - h (x:))
ax
(l)
+
(B.24)
a2
By applying the same approximation used in Lu et al. [16] of
e h
hh -
(B.25)
h
2
then rewrite the second term of the Taylor expansion
To (e h 1)v
(x; \1)
Jo
(E"
h 11-1) V(x; de) - (xx;Ap)
By Equation (B.23)
Os
a2
ph
h
(V(x: V(x; Ar)
By Equation (B.25)
0 2p
0, h -
dep 2,t)
deg (x 2,
afax(t)
axax(1)) (r)
a,
2p
0x:
0x,
(B.26)
21

---

Then Equation (B.24) becomes
Deg (Xe, 2,
(x(3) (1) + 0z(E" -
ax ax())
0xg
0x
θx,
o ea - 1
07
ax (1)
dEO (Eo(Xi, 2,,,
axxx
a
axx( (r)
dea )60(xr;2,1)
(B.27)
2p
a
ixi
a
Ùx,
Likewise, consider the scaled vector-Jacobian product of the adjoint state (1) and the gradient of the model w.r.t Z, i.e.,
(x,,xx,2.11
V(z; t) = a,ax (t)
(B.28)
Oz
along with a corresponding definition of firs-derivative w.r.t. À as defined in Equation (B.23). As such Equation (B.20), when
2, becomes the following when omitting the higher-order error term:
(8 = (1) + Ac) //M X. = at) + Ts (e* 1))
(A (A=).)0 dd (2; Ax)
//m
11 Ad
0!
[e - h ();
(B.29)
022
as
The second term of the Taylor expansion can rewritten as
Os (e h 1)V (z; Ax) as
Os (e" h 1)
V(z; Ae) V(z;
ag
ph
0 ch
(V(z:de) V(z:Ar))
By Equation (B.25)
02 2p
0. h -1
deo (xo( 2.1)
ax (r) dEp (xp 2,
m1ax(1)
0g 2p
dz
Oz
(B.30)
Then Equation (B.29) becomes
(x()) (!) as( - 1)
ar a()
dep 2,t)
Oz
h -
2p ax (t)
dEA (Xr, 2,
Or
ax(r)
ax
Qeo(Xr: 2.1)
(B.31)
8z
O2.
Oz
and the corresponding second-order solver for ao(() can be found in similar manner.
C Proof of Theorem 2.1
For notational brevity we denote the scaled vector-Jacobian products of the solution trajectory of AdjoinDDEIS
DEP (Kr, z,
(x; = a²xx (1)
(C.1)
Oep (x,,,
V(z;t) = axāx(t)
(C.2)
Oz
C.1 Assumptions
For the AdjointDEIS solvers, make similar assumptions to Lu al. [16].
Assumption C.1. The total derivatives of the vector-Jacobian products (()) z, 0), as function of X exist and are continuous for 0 S S k - (and hence bounded).
Assumption C.2. The function 6x(x, 2, f) is continuous in and uniformly Lipschitz and continuously differentiable W.E.t. its first parameter
Assumption C.3. hmoz max h, OL/M))
22

---

Assumption C.4. Ps >>>>00 for all i == 1...., M and some constant c.
The first assumption is required by Taylor's theorem. The second assumption mild assumption to ensure that Theorem C.I holds, which used replace V( z, 0).t) with V((x,, z, f. ( (t) ax (t)) so the Taylor expansion w.r.t. A, is applicable. The third assumption is technical assumption exclude a significantly large step size. The last assumption is necessary for the case when k = 2. For our proofs, we follow a similar outline to that taken by Lu et al. [17, Appendix A].
C.2 The Vector-Jacobian Product Lipschitz
Lemma C.1 WVector-Jacobiag Product is Lipschitz.). Let fo R x R x
[0
R be continuous
in and uniformly Lipschitz and continuously differentiable in x. Let x solution to
(0, T
- Re be the unique
dx: fo(x,, z,
dt
with initial condition X6. Then the following map
-5S (x,, 2, ① ax z, 8]
(a, 1)
is Lipschitz in a. Moreover, the Lipschitz constant L > 0 is given by
L sup 1e/o,T]
(C.3)
0x,
Proof. Now, as x, is continuous and fa continuously differentiable in x, so to is continuous function on the compact set [0, 2], so it is bounded by some L > 0. Likewise, for a E Rdd the map (a, L) + (xs,)) is Lipsehitz in with Lipschitz constant L and this 0 x.200
8x. (x,, z, t)
af
constant is independent of t.
C.3 Proof of Theorem 2.1 when k 1
Proof. First, we consider the case of the adjoint state ax Recall that the AdjointDEIS-1 solver for ax with higher-order error terms is given by
Ct. ax (t,) + Oeet (c - 1) at a4i1
dco(xe, 2,t)
ax(t++1)=
ax (t,)
+ Oh?),
(C.4)
0x::
where we let 4; = f, f++1 s, ti A1s41 dr from Equation (B.21). By Theorem C.I and Equa- tion (B.21) it holds that
at. (x(tt) + To (6 (
ai āx (t:)
dco( 2,t)
åx (t++1)
amh
a
0%..
0t; âx(ts) Otess (c 1)
2
dEp (xp 2,,) + O(āx(()) ax(l)))
ax(1,)
att+1
at
Ox1,
deo(Xe. 2, ((x((1,) -ax(1))
(llaax(1i) (x((l)) + T4i1 (c - 1)
ax (1:)
0tti11
0
Ox.:
ax(1+=1) + O1h + 0(xx ax (t:))
(C.5)
Repeat, this argument, from âx (lo) (0) then we find
(x((m) (T) +
= (T) + O (hmow))
(C.6)
Although the argument for the adjoint state (t) follows an analogous form to the one above, we explicitly state for completeness. Recall that the AdjointDEIS-1 solver for a with hideer-order error terms is given by
ax(1++1) at()) + Oeun 1) at. ax(tt)
Deg (X,,, 2,t)
O1h)).
(C.7)
at
Oz
23

---

By Theorem C.I and Equation (B.22) it holds that
(x(t++)) = åz (4:) + oe (e - 1) ax(ti))
at,
degl
(xx, 2,
011i11
Oz
= =4,(4) + tist (c 1)
Ol
ax (x44
de 2.
+ (()((4) - ax((1))
ax4+1
Oz
=ax(t;) Ouui (c - -1)
Ot
ax(t:))
dEG 2,1) ((xx(t;) -xi(t)
at++
a2(41)1) + (limas) + ((xx ax((1))
(C.8)
Repeat, this argument, from az áz(to) 0 then we find
a (1 az (7) + (M max) = az (T) + Olhmaz)
(C.9)
An identical argument can be constructed for ag, thereby finishing the proof.
C.4 Proof of Theorem 2.1 when k =
We prove the discretization error of the AdjointDEIS-2M solver. Note that for the AdjoinDDEIS-2M
solver we have h; At._ A and p; Àe, - Furthermore, let A | a (t:)ll-
Without loss of generality, we will prove this only for axi the derivation for az and ag is analogous.
Proof. First, we consider the case of the adjoint state ax (1). Recall that the AdjointDEIS-2, see Equa- tion (B.24), solver for ax with higher-order error terms is given by
ax (1++1))
Ott, ax (1:)
Otet1 -x)
Ou+1 (x) 04+t1
0(h)).
044+1
ax+t1
014+41
(C.10)
Taylor's expansion yields
ax(()+1)- (0-00-aa) ate
atten ax (1:)+
Tliti (xx;
Tle (c -x)
SCh,
a
at
(C.11)
where C' is a constant that depends on ((2) (Xt, !). Also note that
((( 1 (V(x:t;) - V(x;t=-1)) S SCh.
(C.12)
Pshe
Since Pi is bounded away from zero, and e -hi 1- h4 + 12/22 O(h,), know
|cot - h4 + -,
(v(x;(,) V(x: (xx)
2p,
hi
-1
chi h4
S CLL(( + A.-1) Ch₂ +
V(x;t,+ - v(x+
Pi
2
h;
s CLM(A + 4,-1) Ch. + Ch² (x;;ta)
v(x;4-1)
SCLAAA + A,-1) CMA...
(C.13)
where M, = 1+ supe.getu. ( (x: t)ll and L are the Lipschitz constants of V t) by Theo-
rem C.I. Then, Di+1 can estimated as
A < at A +
01+t1 LA,
+ Ttet
(CM,, + CLM,(A, + As+1)) + Ch
0tt++1
a
af
2
A + Ch (A; + A+t1 h1)).
(C.14)
Ott4
Thus, A+t1 (l1,aoz) as long as hyar is sufficiently small and Ag + A ((h2 which can be verified via Taylor expansion, thereby finishing the proof.
D
24

---

D Proof of Theorem 2.2
For additional clarity, we let x(t) X, and likewise, z(t) Zt.
Proof. Recall that z(t) is piecewise function of time with partitions of the time domain given by {0 = to < 4 < ... < In = =T). Without loss of generality we consider some time interval ( /m] for some 1 v S Consider the augmented state defined on the interval x:
d
oo (x((()))(() Zz(1)
fug
(D.1)
dt
z
where 8z: : [0, 7] R= denotes the right derivative of time t. Let Baug denote the associated augmented adjoint state
Baug (1) =
ax
(4)-
(D.2)
az
The Jacobian of fag has the form
f aug
@5 (x.2..))
(x.(x.z.))
8x
8x
(D.3)
0 0
0
As the conditional information z() evolves with 8 zt))) on [6m [m] in the forward flow of time. The derivative of the Oz w.r.t. z is clearly as 02 a function only of time Remark, that as the
bottom row of the Jacobian f aug is all 0 and fa is continuous in t we can consider the evolution of asu the whole interval [0, T] rather than just the partition [fm-11 tm]- Using Equations (D.2) and (D.3) we can define the evolution of the adjoint augmented state on [0, T] as
da dt
0f mug (l) axx ax)
(1).
(D.4)
0xx x
Therefore, (t) evolves with the ODE
daz
dt (t) ax(()) 05
ag (T) = 0,
(D.5)
Öz(t)
We have thus shown the evolution of (t( for some continuously differentiable function z(t).
Now we prove the solution is unique and exists. As x(t) is continuous and fa is continuously differentiable in x, it follows that the map ( + afa (x(1), z()), () is continuous function on the
bounded by some L > 0. Correspondingly, for ax RRd compact set [0, T). and therefore af. follows that the map (ax, 1) د -ax म. (x(1), z(1), 1) is Lipschitz in ax with Lipschitz constant L and this constant is independent of t. Therefore, by the Lindel-Lindelff theorem the solution (t exists and is unique.
D.1 Continuous-timm Extension of Discrete Conditional Information
The proof above makes some fairly lax assumptions about z(t). While we assumed the process was càdlàg and had right derivatives this restraints are too loose for adaptive step-size solvers. While this
is not an issue for our AdjointDEIS, Kidger et al. [58] pointed out that an adaptive solver would take a long time to compute the backward pass as would have to slow down at the discontinuities. One could follow the approach taken in [58] and construet natural cubic splines from fully observed, but
irregularly sampled time series {z, 1 with 0 = to < ... < <t T. Then define Z : [0, T + R as the natural cubic spline with knots at to ... tn such that Z(t.) Zt,
D.2 Connection to Neural CDEs
Kidger al. [58, Theorem C.1] showed that any equation of the form
X; = xo + S/' ho 2s 8) ds,
(D.6)
25

---

can be rewritten as a neural controlled differential equation (CDE) of the form
X =x + / fo(x, 8) dz..
(D.7)
where f dz, is the Shemann-Stieltiie integral. N.B., the converse is not true.
While there a certainly benefits to modeling with a neural CDE over neural ODE, diffusion models in particular pre-train the model in the neural ODE sense. Moreover, we are also interested in updating Zt, at each & via gradient descent to solve our optimization problem.
E Details on Adjoints for SDEs
In this section, we provide further details on the continuous adjoint equations for diffusion SDEs that we omitted from the main paper due to their technical nature and for the purpose of brevity.
Consider the Itō integral given by
XT
dw.,
(E.1)
where x is continuous semi- comi-martingale adapted the filtration generated by the Wiener process (wrheejo.r) (Fihe The following quantity, however, is not defined
J X, dw..
(E.2)
This is because x; and w, are adapted to {Felrejo,T which is defined in forwards time. This means does not anticipate future events only depends on past events. While this is generally sufficient when we wish to integrate backwards time, we want future events to inform past events.
E.1 Stratonovich Symmetric Integrals and Two-sided Filtration
Clearly, we need different tool to model this backwards SDE. As such, taking inspiration from the work on neural SDEs [59], we follow the treatment of Kunita [60] for the forward and backward tisk-Stratonovich integrals using two-sided filtration. Let {F FF.rhstts..eeoor be two-sided filtration,
where F., is the o-algebra generated by (w, W. : I u I t} for s, E [0, T such that S l.
Forward time. For continuous semi-martingale (xt)0) adapted to the forward filtration (Fo,)ee(o.d) the Stratonovich stochastic integral is given as
Xt.
Xiu-1 (Way. Wtz
Xx o dw, lim
(E.3)
[
2
where II {0 to <N T} is partition of the interval [o, T] and [I]| max ty fx-1- The forward filtration (Fo.heelo. is analogous to the filtration defined the prior section; therefore, any continuous semi-martingale adapted to it only considers past events and does not anticipate future events.
Reverse time. Consider the backwards Wiener process w, w. WT that is adapted to the backward filtration {F,, Jeejo,,,, then for a continuous semi-martingale (X1hejoo1y adapted to the backward filtration, the backward Stratonovich integral is
Xr
X, o dw, lim
W (
W
(E.4)
2
The backward filtration {Fs.Tsselo.r is the opposite of the forward filtration in the sense that continuous semi-men martingales adapted to it only depend on future events and do not anticipate past events. As such, time is effectively reversed.
Remark E.1. While the Stratonovich symmetric integrals give powerful tool for integrating forwards and backwards in time with stochastic integrals, important that we use the same realization of the Wiener process.
26

---

E.2
Stochastic Flow of Diffeomorpphsms
Consider the Stratonovich SDE defined as
XT -x0 xo ! t(x,,t) dt dt+ f g(xx,t) t) dw
(E.5)
where f. g.g& E C, i.e. they belong to the class of functions with infinitely many bounded deriva- tives w.r.t. the state and bounded first derivatives solution. Given a realization of the Wiener process, there exists a smooth mapping & called the stochastic flow such that (x,((,) the solution time of the process described Equation (E.5)
time. Thus, the SDE has unique strong
started at x at time s S t. This then defines a collection of continuous maps S = (ts,restt.. from IR to itself.
Kunita [60, Theorem 3.7.1] shows that with probability this collection S satisfies the flow property
(x,c(x,) = uu.: (x.. s S u S ttx, R
(E.6)
and that each ..., is smooth diffeomorphism from IRd to itself. Hence, S is the stochastic flow of diffeomorphisms generated by Equation (E.5). Moreover, the backward flow V.A, $ e, satisfies the backwards SDE:
T (x,) = Xt -f" [ (X),,)Xx)), du J: (( u) o dw
(E.7)
for all 8, [0, 7] such that 8 I t. This formulation makes intuitive sense as the backwards SDE differs only from the forwards SDE by negative sign.
E.3 Continuous Adjoint Equations
Now consider the adjoint flow (x,) (x.),. (xx)) then Axc(X)) = Ao, (V,,((.) Li al. [59] show that (x,) satisfies the backward SDE:
8x; + $f." Am.l(x)) 0f
OC
Ox., ( (V..( u) du + [' (x.)(X)) Ox.
Og (Xi,((x), u) dww.
(x,)
(E.8)
As the drift and diffusion coefficient of this SDE in Cpe,1, the system has unique strong solution.
E.4 Proof of Theorem 3.1
We are now ready to put all of this together to prove the result from the main paper.
Proof.
dx, f(x,, dt g(t) dw,-
(E.9)
By Equation (E.8) the adjoint state admitted by the flow of diffcomorphisms generated by Equa- tion (F.9) evolves with the SDE
Oc
of (x),)) u) du + ' (x.)( Ox, S Au.i Ox.
0g
Ax((xx)= = 0x:
(2) dw.
=0
+ [-¹ (x..(())) Ox.
of ( (va)( u) du.
(E.
0x.
Clearly, the adjoint state evolves with ODE revolving around only the drift coefficient, i.e. f. Therefore, we can rewrite the evolution of the adjoint state as
0f
da (1) =ax((1) Zx,
(xe, 1) dt.
(E.11)
27

---

E,5 Converting the Itō SDE to Stratonovich
The diffusion SDE in Equation (3.1) defined as an Itö SDE. However, Theorem 3.1 defined as Stratonovich SDEs. However, an Itō SDE can be easily converted into the Stratonovich form, i.e., for some Itō SDE of the form
dx, f (xx, () di + g () dw,
(F.
with differentiable function a, there exists a corresponding Stratonovich SDE of the form
1 0g (x,, t) gXx, t)] + g(x,, t) dw..
dx. (x 1) 2 0x
(E. 13)
As Equation (3.1) is defined such that gx t) g(t) and is independent of the state X,, then the SDE may be written in Stratonovich form as
[(()) + g²) c,( dt + g(t) dw.
dx
(E.14)
or
F
Additional Experiments
In this section, we include some additional experiments which did not fit within the main paper.
(a) M 5
(b) .M 10
(c) M 15
(d) M 20
Figure 4: Morphed faces created by guided generation with AdjointDEIS with differing number of discretization steps.
F1 Impact of Discretization Steps
One of the advantages of AdjoinDDEIS is that the solver for the diffusion ODE and continuous adjoint equations are distinct. This means that we do not have to force N MM enabling greater ficxibility when using AdJointDEIS. As such, explore the impact of using fewer steps estimate the gradient while keeping the number of sampling steps the change in the number of discretization steps when estimating the gradients. Unsurprisingly, the fewer steps we take, the less accurate the gradients are. This matches the empirical data presented in Table which measures the impact of the performance of face morphing measured in MMPMR.
20 fixed. In Figure we illustrate the impact of
Table Impact of number of discretization steps, M, on face morphing with AdjoinDEISS. FMR 0.1 %, 0.1.
MMPMR(1)
M (4)
AdaFace
ArcFace
ElasticFace
15
94.89
90.59
94.07
94.27
91.21
92.84
05
69.94
60.74
64.21
To explore degradation in performance further. As Kidger [25] points out, due recalculating the solution trajectory of the underlying state backwards can differ, in not significant ways from the x, calculated in the forward pass due to truncation errors. This is especially true for non-algebraically reversible solvers [61], although these can suffer from poor regions of stability. Consider the toy
28

---

example of the ODE itt forward solve, most numerical ODE solvers with non-trivial region of stability will find reasonably useful solution as the enror decay exponentially; however, during the backward solve any small error magnified exponentially instead.
>r(t), where À < 0 and some numerical ODE solver. During the
One possible solution to this is to use interpolated adjoints wherein the solution states, X,, but not the internal states of fe stored. The backward solve can then interpolate between them needed. Kim et al. [56] report that interpolated adjoints performed well on a stiff differential equation. In the case when N .M it is quite convenient to simply store {x, 1 during the forward solve. Moreover, in our use case where N is quite small this adds little overhead. In Table 5 we compare recording the solution states finding them via solving DDIM forwards in time. We observe that AdjointDEIS performed better when using the recorded states.
Table 5: Recording the solution states backwards solve with DDIM. M 20, 7 0.001. FMR 0.1%.
MMPMR(1)
Record
AdaFace
ArcFace
ElasticFace
94.27
89.78
93.87 95.91
95.5
92.64
(a) 0.01
(b) 0.1
(1)
Figure 5: Morphed faces created by guided generation with AdjoinDEIS with different learning rates. used M 20 the ODE variant.
F2 Impact of Learning Rate
We measure the impact of the learning rate on guided generation with AdjointDEIS in Table Unsurprisingly, high learning rates lower performance, especially for less accurate gradients. l.e., when M small. We illustrate example of the impact Figure 5. Clearly, the learning rate of starts to distort the images even if it still fools the FR system.
F3 Number of Steps
As alluded in the main paper, one of the drawbacks of diffusion SDEs is that they require small step sizes to work properly. We observe that the missing high frequency content is added back in when the step size increased, see Figure
G Implementation Details
G.1 AdJointDEIS-2M Algorithm
For completeness we have the full AdjointDEIS-22 solver implemented in Algorithm for solving the continuous adjoint equations for diffusion ODEs. We assume that there is another solver that
29

---

Table Impact of learning rate, 7). on face morphing with AdjointDEIS. FMR 0.1%.
MMPMR(1)
SDE
M (p)
AdaFace
ArcFace
ElasticFace
X
20
98.77
98.98
98.77
X
0.1
20
99.8
98.77
99.39
0.01
20
95.5
92.64
95.91
10
50.92
49.69
50,92
0.1
10
94.27
91.21
92.84
X
05
2,66
2.04
1.84
X
0.1
05
69.94
60.74
64.21
1
20
98.57
99.59
98.98
0.1
20
98.57
97.96
97.75
(a) N 20
(b) N 50
Figure 6: Morphed faces created by guided generation with AdjoinDDEIS with different number of sampling steps. SDE solver, M N.
solves the backward ODE yield (* MM Remark that asug [ax- Ag ag) Also Algorithm can be used to solve the continuous adjoint equations for diffusion SDEs by simply adding the factor of into the update equations.
G.2 Code
Our code for AdJointDEIS will be available here http://ggguubb..ommbbbsssagggg AdjointDEIS.
com
G.3
Repositories Used
For reproducibility purposes, we provide list of links to the official repositories of other works used in this paper.
1. The SYN-MAD 2022 dataset used in this paper can be found https://github. com/ marcohuber -SabbberrSYN---eeee -AD-SSS-------------------- MAD--MA222
2. The ArcFace models, at https://github. com/
dataset, and MSIM-ArcFace dataset can be found 3. The ElasticFace model can be found at
4. The AdaFace model can be found
5. The official Diffusion Autoencoders repository can be found at /otpp::///ittuu..cmm
phizaz/diffae.
6. The official MIPGAN repository can be found at https://github. com/ ZHYYYYNNYYY/MIIPAN- wttp:///AAAAAAAAAAAAooaaaa-- morphinn aawoggg-lggiitht.
30

---

Algorithm AdjointDEIS-2M.
Require: Initial values ax (0), monotonically increasing time steps (4) and noise prediction
model Eg (x,,
Denote h; At141 Xe, for i ......,
2: (x(o)) < ax (0)
Initialize an empty buffer Q.
3: ag (tu) < (, 0.
4: Q
butfer
(V), V(z, to), t(0
5: (x())) - ax (fo) + ot, (c 1)
aig
ax (to) T dco (co x,fo)
at,
0810
6: (t1) - (t(o + t1, (e - -1) 1) åx (t))
dea (St 2,50)
the
7: as (b1) - ( + +o (e a/e xx (10)
Dea (Rt, 2,,
80
8: Q
bitler
[(x, (1), V(z, 11). V(0,1))
9: for
+
1,2,... M. - do
10:
Pi f h111
11:
D;
11 2p:
V(x11 (,)
V(x; 4-1)
2p:
E,
[1 2p:
V(z,t,) 2p:
V(z 11-1)
13:
F,+
[+ 2p:
V(Oit)) 1 (V( 11-1)
ax(li+1) f ay -x + 0i++ (e L)D;
14:
0441
15:
(x(1++1) âz(t)) + M:11
(e 1,)E,
antt
16:
(t(4++1) - (t(t +
d+it1
(e 1)F;
0444
if M then
18:
Q
bufter (ix,l++1), (411, V(0, 44))
19:
end if
20:
end for
21:
return (tM),, (t (t
H Experimental Details
In this section, we outline the details for the experiments run Section 5.
H.1 DiM Algorithm
For completeness, we provide the DiM algorithm from [41] following the notation used in [35]. The original bona fide images are denoted x6 and ()))_ The conditional encoder is & X' - 2, & is the numerical diffusion ODE solver, $ is the numerical diffusion ODE solver time runs forwards from 0 to T. The algorithm presented in Algorithm 2.
H.2 NFE
In our reporting of the NFE we record the number of times the diffusion noise prediction U-Net is evaluated both during the encoding phase, Ng, and solving of the PF-ODE or diffusion SDE, N. We chose to report N + Ng over N + 2Ng as even though two bona fide images are encoded resulting 2N NFE during encoding, this process can simply be batched together, reducing the NFE down to Ng. When reporting the NFE for the Morph-
model, we report N + BN where B is the number of blends. While a similar argument can be made that the morphed candidates could be generated in large batch of size B, reducing the NFE of the sampling process down to N, we chose to report BN as the number of blends, B 21, used the Morph-PIPE is quite large, potentially resulting in Out Of Memory (OOM) errors, especially if trying to process mini-batch of morphs. Using NE + N reporting over NE + BN, the NFE Morph-PIPH, PIPE is 350, which is comparable to DiM. The reporting of NFE for AdjointDEIS calculated Ng + nopt (N M) where nt,p
31

---

Algorithm 2 DiM Framework.
Require: Blend parameter 0.5. Time schedule (4)m= (0, 4+1-
1: Ze + [(xD)
>> Encoding bona fides into conditionals.
2: Zo [(xD (b)
3: for i f 2,,,..., - do
4:
X1,41 t 2 (xi, EØ(x,
(a)
(a)
(a)
Za. tz), t4)
Solving the probability flow ODE time runs from
to T
5:
X1441
(b)
+ x
(x Ea(x,,
(b)
(b)
Zb. tx), t.)
6: end for
7: XT + slerp(x; (a) (b)
XT ;w) 8: Zub < (erp(z 24; w)
(ab)
Morph initial noise. Morph conditionals.
9: for N -I,... do
x x__1 + (xx(,.)
(ab)
(ab)
co (x,
(ab)
Zob, t.),t))))
Solving the probability flow ODE as time runs
from to
11: end for
(ab)
return X.)
the number optimization steps and is the number of discretization steps for the continuous adjoint equations.
H.3
Hardware
All of the main experiments were done on single NVIDIA Tesla V100 32GB GPU. On average, the guided generation experiments for our approach took between 6 hours for the whole dataset of face morphs with a batch size of 8. Some additional follow-up work for the camera-ready version used an NVIDIA H100 Tensor Core 80GB GPU with a batch size of 16.
H.4 Datasets
The SYN-MAD 2022 dataset is derived from the Face Research Lab London (FRLL) dataset [52]. FRLL is dataset of high-quality captures of 102 different individuals with frontal images and neutral lighting. There are two images per subject, an image a "neutral" expression and one a "smiling" expression. The ElasticFace [53] FR system was used to select the top 250 most similar pairs, in terms of cosine similarity, of bona fide images for both genders, resulting total of 489 bona fide image pairs for face morphing [51 as some pairs did not generate good morphs the reference set; we follow this minimal subset.
H.S
FR Systems
All three FR systems use the Improved ResNet (IResNet- 100) architecture [62] as the neural net backbone for the FR system. The ArcFace model is widely used FR system [41, 43, 45, 49]. It employs an additive angular margin loss enforce intra-class compactness and inter-class distance, which can enhance the discriminative ability of the feature embeddings [48]. ElasticFace builds upon the ArcFace model by using an elastic penalty margin over the fixed penalty margin used by ArcFace. This change results in an FR system with state-of-the-art performance [53]. Lastly, the AdaFace model employs an adaptive margin loss by weighting the loss relative to an approximation of the image quality [54]. The image quality is approximated via feature norms and is used to give less weight to misclassified images, reducing the impact of "low" quality images on training. This improvement allows the AdaFace model to achieve state-of-the-an performance in FR tasks.
The AdaFace and ElasticFace models are trained on the AreFaee dataset, whereas the ArcFace model is trained on the MS1M-RA A-MeeeinaFace dataset. N.B., the ArcFace model used in the identity loss not the same ArcFace model used during evaluation. The model used in the identity loss is an IResNet- 100 trained on the Glint360k dataset [63] with the ArcFace loss. We use the cosine distance to measure the distance between embeddings from the models. All three FR systems require images of 112 112 pixels. We resize every image, post alignment from dlib which ensures
32

---

the images are square, to 112 x 112 using bilinear down- down- sampling. The image tensors are then normalized such that they take values in [-1, 1]. Lastly, the AdaFace FR system was trained BGR images so the image tensor is shuffled from the RGB format to the BGR format.
I Analytic Formulations of Drift and Diffusion Coefficients
For completeness, we show how to analytically compute the drift and diffusion coefficients for linear noise schedule Ho al. [1] in the VP scenario Song al. [15]. With linear noise schedule log a2 is found to be
81
80
30
log ax
(I.1)
4
2
on / [0, with 80 0.1, 20, following Song al. [15]. The drift coefficient becomes
81
B0
5(1)
(1.2)
2
2
and as v1 a we find
d
exp (-
31
80
30
dt
df
4
2
81
30
= ((31 - Bo)t + (00 exp
12
2
(1.3)
2
Therefore, the diffusion coefficient g²(1) is
81 Bo 1 280l
g2(1) ((3) Bo) + Bb) exp
de? Jr
8o_ 27 Bo 12
+ +((8 Bo)t Bo) 1 exp
(I.4)
d m1 of
Importantly, da, does not exist at time t = 0, as is discontinuous at that point, and so an approximation needed when starting from this initial step. In practice, adding small < 1 to t = 0 should suffice.
de
33

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discussed the limitations of this work in Section 6.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The full set of assumptions, derivations, and proof' are found Appendices to E.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We presented the implementation details in Appendix G, including the algo- rithm as well as the repositories used.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The dataset are all public dataset, have provided links to all the repos- itories used in the experiments in Appendix G. We provide detailed derivations of the Ad- jointDEIS solvers Appendix B. Interested readers can implement the algorithms themselves. We intend to release our code at https: http//gtttttt//t////////teee.. com/ zalaaiigaaaeeeee AdjointDEIS.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental details are presented in Appendix H.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due the computationally demanding nature of the guided generation we do not report emror bars.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Question:
34

---

Answer: [Yes]
Justification: The hardware used in this paper is explained in Appendix H.3.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /wwrrisurppp..//
Answer: [Yes]
Justification: We conducted the research conforming every aspect with the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We address the broader impacts in Section 6.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The dataset used in the experiments are public datasets.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The details of the datasets and models used from other researchers are decsribed in Appendices G and H.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: No new assets were created at the time submission.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We did not perform crowdsourcing. Human faces are used in the experiments, but the datasest we used are all public dataset.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Used public datasets, as such no IRB approvals were needed,
35


Reviews:
[{"review_id": "C6a8DVdLpR", "reviewer": "Reviewer_vPC5", "summary": "AdjointDEIS uses the method of adjoint sensitivity to compute gradients of diffusion models, which is more efficient and less memory intensive, and robust to the injected noise. This work proposes efficient solvers for both the adjoint probability flow ODE and the adjoint diffusion SDE. Experiments demonstrate the efficacy of the solvers on guided generation for a face morphing problem.", "questions": "- It would be interesting to visually plot example forward and backward trajectories of the diffusion processes. What is the typical magnitude of the numerical errors that accumulate during the backward ODE solve?", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- AdjointDEIS reduces the adjoint of the PF ODE to the problem of exponential integrators, which enables use of the vast amount of literature in the areas.\n - Obtaining accurate gradients wrt initial noise x_T and the latent z can open the door to possibly novel applications of diffusion models.", "weaknesses": "- The number of experimental applications of the method seem lacking (just one application to face morphing attacks is included). I’d encourage the authors to think about applications to data assimilation or inverse problems for instance.\n- Since the primary contribution of the paper is solver, having a proof of convergence rates or empirical studies of convergence rate would be valuable additions to the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer vPC5 for the helpful comments and interest in our work. We agree with the reviewer that AdjointDEIS can be applied to more applications like inverse problems. Below we respond to the questions and concerns raised by the reviewer. We hope this addresses the questions and are happy to answer any further questions.\n\n> Since the primary contribution of the paper is solver, having a proof of convergence rates or empirical studies of convergence rate would be valuable additions to the paper.\n\nWe fully agree with the reviewer. If accepted we will include a proof the AdjointDEIS-$k$ is a $k$-th order solver with some mild assumptions to ensure the vector Jacobian product is Lipschitz w.r.t. $\\mathbf{a}_t$ and that the step size is not significantly large $h\\_{max} = \\max\\_{1 \\leq j \\leq M} h_j = \\mathcal{O}(1 / M)$. We provide an excerpt from the proof below. The proof roughly follows the structure of Appendix B.3 from Lu et al. \"DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps\" (NeurIPS, 2022).\n\n*Proof.* The AdjointDEIS-1 solver with higher-order error terms is given by\n\\begin{equation}\n\\def\\bfa{{\\mathbf{a}}}\n\\def\\bfx{{\\mathbf{x}}}\n\\def\\bfz{{\\mathbf{z}}}\n\\bfa_{t_{i+1}}=\\frac{\\alpha_{t_i}}{\\alpha_{t_{i+1}}}\\bfa_{t_i} + \\alpha_{t_i}^2\\sigma_{t_{i+1}}(e^h_i - 1)\\bfa_{t_i}^\\top \\frac{\\partial \\boldsymbol\\epsilon_\\theta(\\bfx_{t_i}, \\bfz, t_i)}{\\partial \\bfx_{t_i}} + \\mathcal{O}(h_i^2)\n\\end{equation}\nwhere $k=1, t_i = t, t_{i+1} = s, h_i = \\lambda_s - \\lambda_t$. Let $\\\\{\\tilde{\\mathbf{a}}\\_{t\\_i}\\\\}\\_{i=1}^M$ denote the sequence computed by AdjointDEIS-$k$.\nSince the vector Jacobian product is Lipschitz w.r.t. $\\mathbf{a}\\_t$, we can write\n\\begin{equation}\n\\tilde\\bfa_{t_i}^\\top \\frac{\\partial \\boldsymbol\\epsilon_\\theta(\\tilde\\bfx_{t_i}, \\bfz, t_i)}{\\partial \\tilde\\bfx_{t_i}}=\\bfa_{t_i}^\\top \\frac{\\partial \\boldsymbol\\epsilon_\\theta(\\bfx_{t_i}, \\bfz, t_i)}{\\partial \\bfx_{t_i}} + \\mathcal{O}(\\tilde\\bfa_{t_i} - \\bfa_{t_i})\n\\end{equation}\nthen\n\\begin{align}\n\\def\\bfa{{\\mathbf{a}}}\n\\def\\bfx{{\\mathbf{x}}}\n\\def\\bfz{{\\mathbf{z}}}\n\\tilde\\bfa_{t_{i+1}} &= \\frac{\\alpha_{t_i}}{\\alpha_{t_{i+1}}}\\tilde\\bfa_{t_i} + \\alpha_{t_i}^2\\sigma_{t_{i+1}}(e^h_i - 1)\\tilde\\bfa_{t_i}^\\top \\frac{\\partial \\boldsymbol\\epsilon_\\theta(\\tilde\\bfx_{t_i}, \\bfz, t_i)}{\\partial \\tilde\\bfx_{t_i}}\\\\\\\\\n        &= \\frac{\\alpha_{t_i}}{\\alpha_{t_{i+1}}}\\tilde\\bfa_{t_i} + \\alpha_{t_i}^2\\sigma_{t_{i+1}}(e^h_i - 1)\\bigg(\\bfa_{t_i}^\\top \\frac{\\partial \\boldsymbol\\epsilon_\\theta(\\bfx_{t_i}, \\bfz, t_i)}{\\partial \\bfx_{t_i}} + \\mathcal{O}(\\tilde\\bfa_{t_{i}} - \\bfa_{t_{i}})\\bigg)\\\\\\\\\n        &= \\frac{\\alpha_{t_i}}{\\alpha_{t_{i+1}}}\\tilde\\bfa_{t_i} + \\alpha_{t_i}^2\\sigma_{t_{i+1}}(e^h_i - 1)\\bfa_{t_i}^\\top \\frac{\\partial \\boldsymbol\\epsilon_\\theta(\\bfx_{t_i}, \\bfz, t_i)}{\\partial \\bfx_{t_i}} + \\mathcal{O}(\\tilde\\bfa_{t_{i}} - \\bfa_{t_{i}})\\\\\\\\\n        &= \\bfa_{t_i} + \\mathcal{O}(h_{max}^2)  + \\mathcal{O}(\\tilde\\bfa_{t_{i}} - \\bfa_{t_{i}})\n\\end{align}\nRepeating this argument, from $\\tilde{\\mathbf{a}}\\_{t\\_i} = \\mathbf{a}\\_0$ then\n\\begin{equation}\n\\def\\bfa{{\\mathbf{a}}}\n\\def\\bfx{{\\mathbf{x}}}\n\\def\\bfz{{\\mathbf{z}}}\n\\tilde\\bfa_{t_M} = \\bfa_T + \\mathcal{O}(Mh_{max}^2) = \\bfa_T + \\mathcal{O}(h_{max})\n\\end{equation}\nThus the global truncation error is $\\mathcal{O}(h_{max})$ and thus completes the proof Q.E.D.\n\n> It would be interesting to visually plot example forward and backward trajectories of the diffusion processes. What is the typical magnitude of the numerical errors that accumulate during the backward ODE solve?\n\nThe forward and backwards trajectories will look identical just with a reversal in time. The sampling trajectory of the probability flow ODE starts with white noise and slowly adds information back into a clean image. Likewise, the backwards ODE solver for the probability flow ODE takes the clean image and adds noise to it. The typical magnitude is numerically quite small as the gradients themselves have a small magnitude. From what I recall the magnitude of errors were about $10^{-5}$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "84vSrJB4gy", "reviewer": "Reviewer_jxa4", "summary": "The paper proposes an adjoint sensitivity method -- AdjointDEIS -- for efficiently calculating gradients of diffusion SDE models. Current methods for naive backpropagation rely on discrete adjoints which are memory intensive. The authors introduce an approach based on the stochastic adjoint sensitivity method to solve the gradients with respect to initial noise, conditional information, and model parameters. The authors develop custom solvers for the adjoint problems along with the proposed sensitivity method. The methods are validated on a face morphing problem.", "questions": "1. Point 2 in contributions: Aren't the continuous adjoint sensitivity methods used in Neural SDEs already general-purpose enough to handle diffusion problems?", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* **Great Flow/Presentation**: The general presentation of the paper is well done! Especially starting off with a more straightforward setting of an ODE and extending them to SDEs\n* **Specialized Solver**: Diffusion models are quite widespread and developing specialized solvers leveraging the structure of the problem demonstrates a practical application.", "weaknesses": "* **Limited Experimental Scope**: While the overall analysis and formulations are sound, the overall experimental validation for the method is limited to just a single experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer jxa4 for the interest in our work and the insightful comments. We are encouraged the reviewer found the extensions from diffusion ODEs to diffusion SDEs as a strength of the paper. Below we respond to the question raised by the reviewer. We hope this helps address the questions and are happy to respond to any further questions.\n\n> Point 2 in contributions: Aren't the continuous adjoint sensitivity methods used in Neural SDEs already general-purpose enough to handle diffusion problems?\n\nWe agree with the reviewer that the continuous adjoint sensitivity methods used in Neural SDEs are general-purpose enough to be applied to diffusion SDEs; however, w'd like to emphasize that one of contributions as we listed in the paper is \"To the best of our knowledge, AdjointDEIS is the first general back-propagation technique **for** diffusion models that use an SDE solver for sampling\".\nThat is, to the best of our knowledge we are the first group to explore using the method of adjoint sensitivity *explicitly for* diffusion SDEs.\nA significant insight in our work comes from exploiting the structure of diffusion ODEs/SDEs, where, in the VP case---this can also be shown in the Variance Exploding (VE) case---the adjoint diffusion SDE simplifies to an ODE!\nThis insight would not straightforwardly be shown by simply applying the Neural SDEs methods on the diffusion SDEs without inspecting the diffusion term of the Stratonovich SDE.\n\nAnother contribution of our work is using the specific structure of the adjoint diffusion ODEs/SDEs to transform the ODE/SDE via exponential integrators to a much simpler formulation, removing the error from the linear error term.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vI6Trld0HX", "reviewer": "Reviewer_oW3N", "summary": "In this paper, the authors proposed an accelerated  method for differentials of pretrained diffusion models with respect to its latent valuables or parameters by making use of \n1. the Taylor expansion of the log-SNR parameter, and\n2. the exact integral formula of the derivatives related to the probability flow ODE, which they call Adjoint Diffusion ODE.\n\nThey applied this method to morphing of two generated images. As they summarized in Table 1, their results achieved good performance.\n\nFor more details:\n- The setting\n    - Data diffusion SDE $d\\mathbf{x}_t = f(t) \\mathbf{x}_t + g(t) d\\mathbf{w}_t$\n        - Its integral $q(\\mathbf{x}_t|\\mathbf{x}_0) = \\mathcal{N}(\\mathbf{x}_t|\\alpha_t \\mathbf{x}_0, \\sigma_t^2)$ where $f(t), g(t)$ are related to $\\alpha_t, \\sigma_t$ by Eq.(2.5).\n    - A pretrained diffusion model $\\epsilon_\\theta(\\mathbf{x},\\mathbf{z}, t)$\n        - Its generating process: the probability flow ODE $\\frac{d\\mathbf{x}\\_t}{dt} = f(t)\\mathbf{x}\\_t + \\frac{g(t)^2}{2 \\sigma_t}\\epsilon_\\theta(\\mathbf{x}, \\mathbf{z}, t)$\n        - They name the velocity (the RHS of the ODE) as $\\mathbf{h}_\\theta(\\mathbf{x}, \\mathbf{z}, t)$\n- The purpose\n    - Calculation of gradients of $\\mathcal{L}(\\mathbf{x}_0(\\mathbf{x}_T, \\mathbf{z}, \\theta))$ with respect to $\\mathbf{z}$ or $\\theta$\n    - which are defined by ODEs in Eq.(3.3) and (3.4).\n- The method\n    - One can recover the derivatives based on a vector $\\hat{\\mathbf{a}}_t = \\frac{\\partial \\mathcal{L}}{\\partial \\mathbf{y}_t}$ where $\\mathbf{y}_t = \\alpha_t^{-1}\\mathbf{x}_t$.\n    - For $t<s$, one can get $\\hat{\\mathbf{a}}_s = \\hat{\\mathbf{a}}_t + \\int\\_{\\lambda_t}^{\\lambda_s} ... d\\lambda$ (Proposition 3.1).\n    - Applying the Taylor expansion on ... with $\\lambda$ around $\\lambda_t$, and drop $O((\\lambda_s - \\lambda_t)^{k+1})$ term, we get a concrete relation between $\\hat{\\mathbf{a}}_s$ and $\\hat{\\mathbf{a}}_t$ (Eq.(3.11)),\n    - which provides explicit (approximated) formulas for the derivatives.\n- The empirical result\n    - They apply the proposed method to morphing (Section 5 Experiments).", "questions": "- for significance improvements:\n    - **Q1(significance)** In the theoretical sections, the authors apply omitting the higher order terms of the Taylor expansion, so the resultant method should be just an approximation. Why is it good performance in Table 1 compared to other methods nonetheless the method is just an approximation?\n- for clarity improvements:\n    - **Q1(clarity)** In the Adjoint Empirical Diffusion SDE cases, the integral results in Eq.(4.12) and (4.13) look very similar to ODE cases in Eq.(3.13) and (3.14). The only difference seem to be (doubled) 2nd term. Why such similarity happens?\n    - **Q2(clarity)** In the beginning of section 4, the authors wrote their motivation to analyze SDE cases as it possibly improve the result, however, it turns to be that the SDE version works worse than the ODE cases. Why does it happens?", "limitations": "They leave comments on\n1. Lack of conditional generation in numerical experiments\n2. No application of parameter differentiation in numerical experiments\n\nin the final page.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **quality(+)** The explanations are well written. The authors provide concrete proofs on their theorems and experiments with real data.", "weaknesses": "- **significance(-)** It is difficult to understand in which respect the proposed method is superior.\n- **clarity(-)** I am almost convinced by the explanations, however there is a concern as explained in Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer oW3N for the detailed questions and interest in our work. We are glad that the reviewer thought our work was well written. We address the questions raised by the reviewer below. We would be happy to provide additional clarification or to answer further questions about our work.\n\n> * for significance improvements:\n>      * **Q1(significance)** In the theoretical sections, the authors apply omitting the higher order terms of the Taylor expansion, so the resultant method should be just an approximation. Why is it good performance in Table 1 compared to other methods nonetheless the method is just an approximation?\n\nThe reviewer is correct that the AdjointDEIS-1 and SDE-AdjointDEIS-1 solvers are numerical approximations of the true adjoint diffusion ODE/SDE. However, we believe these approximations are still quite useful even with large step sizes. The local truncation error for Eq. (3.10) is $\\mathcal{O}(h^{k+1})$, for the first-order solver this error is $\\mathcal{O}(h^2)$. Intuitively, with more discretization steps $N \\to \\infty$ then $h \\to 0$ the error for the first-order solver will disappear. In practice we found that $N = 20$ steps for the adjoint ODE and $N = 50$ steps for the adjoint SDE seemed to work well. As other works have noted, numerical SDE solvers take more steps due to the difficulty in discretezing the stochastic integral.\n\nThe improvement compared to previous works of DiM, Fast-DiM, and Morph-PIPE in Table 1 can be attributed to the identity guided nature of DiM + AdjointDEIS.\nWhile there are discretization errors, we posit that these errors are small enough that the estimated gradient is still useful for identity guidance.\nOne of the strengths of our proposed work is that the adjoint diffusion ODE solver is decoupled from the sampling procedure for the diffusion ODE, that means we can choose different numerical precision levels for sampling and estimating the gradients.\nWe note that while Morph-PIPE does identity guidance, it does so through a brute force search over a space of 21 possible morphs whereas we perform identity guidance through estimated gradients.\n\n> * for clarity improvements:\n>     * **Q1(clarity)** In the Adjoint Empirical Diffusion SDE cases, the integral results in Eq.(4.12) and (4.13) look very similar to ODE cases in Eq.(3.13) and (3.14). The only difference seem to be (doubled) 2nd term. Why such similarity happens?\n\nThis similarity is actually one of our key insights! We observed that the adjoint diffusion SDE actually simplifies to an ODE.\nThe factor of 2 is present because of the difference between the PF ODE\n$$    \\frac{\\mathrm d \\mathbf{x}\\_t}{\\mathrm dt} = f(t)\\mathbf{x}\\_t + \\frac{g^2(t)}{2\\sigma\\_t}\\boldsymbol\\epsilon\\_\\theta(\\mathbf{x}\\_t, \\mathbf{z}, t) $$\nand the diffusion SDE\n$$    \\mathrm d \\mathbf{x}\\_t= \\bigg [f(t)\\mathbf{x}\\_t + \\frac{g^2(t)}{\\sigma\\_t}\\boldsymbol\\epsilon\\_\\theta(\\mathbf{x}\\_t, \\mathbf{z}, t) \\bigg ]\\mathrm dt + g(t) \\circ \\mathrm d \\tilde{\\mathbf{w}}\\_t $$\nwhere that factor 2 differs between the ODE and drift term of the SDE.\nThis means that we can use the *exact* same solvers for the adjoint ODE as the adjoint SDE with the *only* exception being the factor of 2!\nThe only caveat being the underlying state $\\mathbf{x}\\_t$ still evolves with the backwards flow, Eq. (4.6), and uses a different solver.\n\n>  **Q2(clarity)** In the beginning of section 4, the authors wrote their motivation to analyze SDE cases as it possibly improve the result, however, it turns to be that the SDE version works worse than the ODE cases. Why does it happens?\n\nWhile recent work like Nie et al., \"The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing\" (ICLR 2024) shows the promise of SDEs in image editing, the face morphing problem is unique.\nThe goal is to fool an Face Recognition (FR) system into accepting one morphed face image as belonging to *two* identities.\nRecent work into the face morphing problem has shown that visual fidelity and morphing performance are not necessarily correlated, see the DiM and Fast-DiM paper.\nUpon visual inspection, the morphs produced using the adjoint diffusion SDE solver look more realistic and less smoothed to our eyes.\nIn terms of morphing performance in terms of MMPMR, we find it to be about the same as the adjoint diffusion ODE morphs.\nAs such we would argue that the SDE morphs are actually *slightly* better than ODE morphs as they have comparable MMPMR performance and better visual fidelity.\n\nIf accepted, we plan on making these contributions more clear in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SDSDDZOvN3", "reviewer": "Reviewer_1FE9", "summary": "This paper presents a way to optimize the initial seed or control parameters to a diffusion network to optimize a differentiable loss applied to the final samples drawn from a discretization of the diffusion SDE / probability flow ODE. Put differently the idea is to apply the white box integration trick from DPM / DPM++ paper, and the method from the neural ODE paper to differentiate through an ODE solver.\n\nIn addition, the paper presents methods for backpropagating through an SDE. \n\nIn terms of experiments the method is demonstrated on a face morphing application.", "questions": "Mainly I would like to understand a little bit more about the rationale for selecting the face morph application selection, and whether the authors think the empirical comparisons in Table 1 are apples-to-apples given the different NFE for the methods.\n\n\nSome more minor comments\n- eq (3.1) has a factor of 2 missing.\n- around eq(2.5) it might be good to explicitly clarify that while VP typically uses alpha^2 + sigma^2 = 1 ibut you are not making that assumption here.", "limitations": "Yes limitations have been addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and the derivation is mostly clear. I did not go through section 4 for the derivation of the SDE Adjoint Diffusion SDE too closely since I am not very familiar with the Stratonovich stochastic integral.\n\nThe paper is also fairly comprehensive since it presents the derivations for both the probability flow formulation as well as the SDE formulation. \n\nOverall I think the problem tackled in the paper is interesting and the solutions are elegant.", "weaknesses": "The experimental results seem to be weak.\n\nThis paper presents a way to control the final outcome from a diffusion sampler using a differentiable loss, but the only application demonstrated in the paper is on face-morphing where the NFE are not controlled across the baselines. Some of the baselines have fewer NFE and it's not possible to say conclusively whether they underperform because they are worse methods or just because they had less computational budget.\n\nAdjointDEIS-2M is not even implemented and compared. \n\nAlso, a lot more compelling demonstrations could be shown of these new algorithms. For example, a number of \"controlnet\" type applications could have been tried to create an image that matches a sketch of an output.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 1FE9 for the helpful questions and interest in our work. We are happy the reviewer found the paper clear and comprehensive tackling an interesting problem. We address the questions raised by the reviewer below. We hope our responses help address the questions and are happy to answer any additional questions.\n\n> Mainly I would like to understand a little bit more about the rationale for selecting the face morph application selection, and whether the authors think the empirical comparisons in Table 1 are apples-to-apples given the different NFE for the methods.\n\nThis work actually originated from research we were conducting on the face morphing problem. A popular technique in that area is to use identity guidance for GAN-based models (Zhang et al., \"MIPGAN—Generating Strong and High Quality Morphing Attacks Using Identity Prior Driven GAN,\" in IEEE Transactions on Biometrics, Behavior, and Identity Science, 2021) wherein the optimal latent vector $\\mathbf{z}^*$ is found by minimizing the identity loss defined between the generated morphed face $g(\\mathbf{z})$ and the original bona fide faces $\\mathbf{x}^{(a)}, \\mathbf{x}^{(b)}$ such that\n$$\\mathbf{z}^* = \\mathop{\\rm arg min}_{\\mathbf{z} \\in \\mathcal{Z}} \\mathcal{L}(g(\\mathbf{z}), \\mathbf{x}^{(a)}, \\mathbf{x}^{(b)})$$\nThis is done using a gradient descent algorithm and thus requires $\\frac{\\partial \\mathcal{L}(g(\\mathbf{z}), \\mathbf{x}^{(a)}, \\mathbf{x}^{(b)}))}{\\partial \\mathbf{z}}$ which can be found through simple automatic differentiation tools.\nWe were seeking to perform a similar guided generation procedure, but with diffusion models. \nIt is during this process that we developed the AdjointDEIS algorithm.\nBeyond the AdjointDEIS algorithm developed out of the work on guided generation for the face morphing problem, we are also interested in experimental illustration of AdjointDEIS as we compare to the original **Di**ffusion **M**orphs (**DiM**) and a recent identity guided extension called Morph-PIPE.\n\nWe believe the comparisons in Table 1 are fair despite the different NFE numbers. We first present a brief summary of the different DiM algorithms with details on the NFE calculation:\n* **DiM-A** The original DiM algorithm, the model uses 250 NFEs to encode the bona fide images $\\mathbf{x}_0^{(a)}, \\mathbf{x}_0^{(b)}$ into the initial noise $\\mathbf{x}_T^{(a)}, \\mathbf{x}_T^{(b)}$ and conditional information $\\mathbf{z}\\_{a}, \\mathbf{z}\\_{b}$. The model then takes 100 NFEs to generated the morphed image from the morphed noise and conditionals $\\mathbf{x}\\_T^{(ab)}$ and $\\mathbf{z}\\_{ab}$.\n* **Fast-DiM** An improvement on DiM-A that uses higher-order ODE solvers to reduce the NFE. Fast-DiM use 250 NFEs for encoding and 50 NFEs for sampling.\n* **Morph-PIPE** An improvement on DiM-A that generates 21 possible morphs using a blend of 21 different interpolations between $\\mathbf{x}_T^{(a)}$ and $\\mathbf{x}_T^{(b)}$, likewise with the conditional information. Due to the 21 candidate morphs, the NFE for sampling is $21 \\cdot 100 = 2100$. The morph which minimizes the identity loss is then chosen.\n* **DiM + AdjointDEIS** We apply AdjointDEIS-1 to the original DiM algorithm. We used 250 NFEs for encoding. Due to the optimization procedure, we were able to reduce the sampling NFE during each iteration of the optimization procedure to 20. We used 50 optimization steps, resulting in a total sampling NFE of $20 \\cdot 50 = 1000$.\n* **DiM + SDE-AdjointDEIS** We apply SDE-AdjointDEIS-1 solver to the original DiM algorithm. We used 250 NFEs for encoding. As prior work has noticed, due to the intricacies of discretizing the diffusion SDE, numerical SDE solvers often take more steps to achieve good performance than a numerical ODE solver. We used 50 sampling steps but 10 optimization steps for 500 total sampling NFEs.\n\nAs discussed in the Fast-DiM paper, increasing the number of sampling/encoding steps, and thereby NFE, for DiM-A, Fast-DiM, or Morph-PIPE would not meaningfully increase the effectiveness of the morphing attack in terms of MMPMR. While the higher NFE of the AdjointDEIS methods does indicate an increased computation cost over that of DiM or Fast-DiM, simply increasing the NFE for DiM or Fast-DiM would not give them equivalent performance to AdjointDEIS. Our higher NFE is due to the computation of the adjoint diffusion ODE/SDE.\nWe would like to emphasize that compared to Morph-PIPE, the other identity guided method, our approaches achieve *superior performance with fewer NFE*.\n\n> Some more minor comments\n>    * eq (3.1) has a factor of 2 missing.\n>   * around eq(2.5) it might be good to explicitly clarify that while VP typically uses alpha^2 + sigma^2 = 1 ibut you are not making that assumption here.\n\nIf accepted, both of these changes will be incorporated into the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
We thank all the reviewers for all of their time and feedback on our submitted manuscript.

---

We are delighted to see that the reviewers appreciated the practical significance of our method, highlighting that it "can open the door to possibly novel applications of diffusion models." (**reviewer vPC5**) and "demonstrates a practical application" (**reviewer jxa4**).

Additionally, we are glad to see that the reviewers appreciated that we "start[ed] off with a more straightforward setting of an ODE and extending them to SDEs" (**reviewer jxa4**) and thought that "the problem tackled in the paper is interesting and the solutions are elegant" (**reviewer 1FE9**).

Lastly, we are pleased that the reviewers found the paper to be "well written" (**reviewers oW3N and 1FE9**) and "fairly comprehensive" (**reviewer 1FE9**).

---
We primarily address the concerns and questions raised by the reviewers in our individual responses; however, we address some common concerns below.

**Significance of Contribution.** In light of the feedback we received, we make the following improvements:
* Emphasizing the significance that the adjoint diffusion SDE simplifies to an ODE and is **identical** to the adjoint probability flow ODE with the exception of a factor of 2 on the vector Jacobian term.
* Prove that the AdjointDEIS-$k$ solvers are in fact $k$-th order solvers obtaining a global truncation error of $\mathcal{O}(h^k)$. For more details, please refer to the response to reviewer vPC5.
* Highlight that the calculation of the adjoint probability flow ODE is decoupled from the probability flow ODE, i.e., separate numerical solvers can be used with **different** step sizes! This means we could use **fewer** steps to obtain a working estimate of the gradient and still perform guided diffusion. The step size could be scheduled, whereas the loss decreases with each optimization iteration using the adjoint state the step size for the adjoint ODE solver can increase to reduce computation.

**Experimental Section.** We acknowledge that additional experimental results would help strength our manuscript. We believe that the example of face morphing provides a good illustration of the utility of AdjointDEIS in guided generation problems. Our hope is that the compelling motivation of AdjointDEIS coupled with the new theoretical results will provide a springboard for further research into guided generation with gradients from AdjointDEIS.

We hope that our responses to the reviewers convincingly address the reviewers' concerns and are happy to answer any further questions.

Sincerely,

The Authors

Meta-review: This work leverages adjoint sensitivity analysis to efficiently compute gradients of the probability flow ODE implementation of diffusion model, in order to optimize latent variables and hyperparameters. Reviewers and AC agreed about the merits of this work, and acceptance is recommended.
Decision: Accept (poster)