Deep Equilibrium Approaches to Diffusion Models
Authors: Ashwini Pokle
Zhengyang Geng
J Zico Kolter
Venue: NeurIPS 2022 Accept (2022)
Keywords: diffusion models
deep equilibrium models
model inversion

Abstract: Diffusion-based generative models are extremely effective in generating high-quality images, with generated samples often surpassing the quality of those produced by other models under several metrics.  One distinguishing feature of these models, however, is that they typically require long sampling chains in order to produce high-fidelity images.  This presents a challenge not only from the lenses of sampling time, but also from the inherent difficulty in backpropagating through these chains in order to accomplish tasks such as model inversion, i.e., approximately finding latent states that generate known images.  In this paper, we look at diffusion models through a different perspective, that of a (deep) equilibrium (DEQ) fixed point model. Specifically, we extend the recent denoising diffusion implicit model (DDIM), and model the entire sampling chain as a joint, multi-variate fixed point system. This setup provides an elegant unification of diffusion and equilibrium models, and shows benefits in 1) single-shot image sampling, as it replaces the fully-serial typical sampling process with a parallel one; and 2) model inversion, where we can leverage fast gradients in the DEQ setting to much more quickly find the noise that generates a given image.  The approach is also orthogonal and thus complementary to other methods used to reduce the sampling time, or improve model inversion.  We demonstrate our method's strong performance across several datasets, including CIFAR10, CelebA, and LSUN Bedroom and Churches.

Deep Equilibrium Approaches to Diffusion Models
Ashwini Pokle Carnegie Mellon University apokle@cs. cmu. edu
Zhengyang Geng
Carnegie Mellon University zgeng20cs. cmu. edu
Zico Kolter
Carnegie Mellon University Bosch Center for Al zkolterêcs. cmu edu
Abstract
Diffusion- generative models are extremely effective in generating high- quality images, with generated samples often surpassing the quality of those pro- duced by other models under several metrics. One distinguishing feature of these models, however, is that they typically require long sampling chains to produce high-fidelity images. This presents a challenge not only from the lenses of sam- pling time, but also from the inherent difficulty in backpropagating through these chains order to accomplish tasks such as model inversion, i.e., approximately finding latent states that generate known images. In this paper, look diffusion models through a different perspective, that of a (deep) equilibrium (DEQ) fixed point model. Specifically. (DDIM) [68], and model the entire sampling chain as joint, multi-variate fixed point system. This setup provides elegant unification of diffusion and equilib- rium models, and shows benefits in 1) single image sampling, as it replaces the fully-serial typical sampling process with a parallel one; and 2) model inversion, where we can leverage fast gradients in the DEQ setting to much more quickly find the noise that generates a given image. The approach is also orthogonal and thus complementary to other methods used to reduce the sampling time, or improve model inversion. We demonstrate our method's strong performance across several datasets, including CIFARIO, CelebA, and LSUN Bedroom and Churches."
extend the recent denoising diffusion implicit model
Introduction
Diffusion models have emerged as a promising class of generative models that can generate high quality images [69, 68, 57], outperforming GANs on perceptual quality metrics [19], and likelihood- based models on density estimation [42]. One of the limitations of these models, however, is the fact that they require long diffusion chain (many repeated applications of denoising process). in order to generate high-fidelity samples. Several recent papers have focused on tackling this limitation, egg. by shortening the length of diffusion process through an alternative parameterization [68, 44], or through progressive distillation of sampler with large diffusion chain into smaller one [54, 65]. However, all of these methods still rely on a fundamentally sequential sampling process, imposing challenges accelerating the sampling and for other applications like differentiating through the entire generation process.
In this paper, we propose an alternative approach that also begins to address such challenges from different perspective. Specifically, we propose to model the generative process of specific class of
'Code available at atttp::/ggitie com/ http///avinipooeeeee-ddeaaeee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

diffusion model, the denoising diffusion implicit model (DDIM) [68], as deep equilibrium (DEQ) model [6]. Deep equilibrium (DEQ) models are networks that aim to find the fixed point of the underlying system in the forward pass and differentiate implicitly through this fixed point in the backward pass. To apply DEQs to diffusion models, we first formulate this process as an equilibrium system consisting of all T joint sampling steps, and then simultaneosly solve the fixed point of all the T steps to achieve sampling.
This approach has several benefits: First, the DEQ sampling process can be solved in parallel over multiple GPUs by batching the workload. This is particularly beneficial the case of single image (i.e.. batch-size-ong inevitably made them unable to maximize GPU computation. Second, solving for the joint equilibria simultancously leads to faster overall convergence as we have better estimates of the intermediate states in fewer steps. Specifically, the formulation naturally lends itself to an augmented diffusion chain that each state is generated according to all others. Third, the DEQ formulation allows us to leverage a faster differentiation through the chain. This enables us to much more effectively solve problems that require to differentiate through the generative process, useful for tasks such model inversion that seeks to find the noise that leads to a particular instance of an image.
generation, where the serial sampling nature of diffusion models has
We demonstrate the advantages of this formulation on two applications: single image generation and model inversion. Both single image generation and model inversion are widely applied in real world image manipulation tasks like image editing and restoration [77. 63, 35, 2, 49, 55]. On CIFAR- 10 [46] and CelebA [52]. DEQ achieves up to 2x speedup over the sequential sampling processes of DDIM [68]. while maintaining comparable perceptual quality of images. In the model inversion, the loss converges much faster when trained with DEQs than with sequential sampling. Moreover, optimizing the sequential sampling process can be computationally expensive. Leveraging modern autograd packages is infeasible as they require storing the entire computational graph for all T states. Some recent works like Nie et al. [58] achieve this through use of SDE solvers. In contrast, with DEQs we can use the implicit differentiation of (1) memory complexity. Empirically, the initial hidden state recovered by DEQ more accurately regenerates the original image while capturing its finer details.
To summarize, our main contributions are as follows:
We formulate the generative process of augmented type of DDIM as a deep equilibrium model that allows the use of black-box solvers to efficiently compute the fixed point and generate images.
The DEQ formulation parallelizes the sampling process of DDIM, and as a result, can be run on multiple GPUs instead of single GPU. This alternate sampling process converges faster compared to the original process.
We demonstrate the advantages of our formulation single image generation and model inversion. We find that optimizing the sampling process via DEQs consistently outperforms naive sequential sampling.
We provide an easy way to extend this DEQ formulation to a more general family of diffusion models with stochastic generative process like DDPM [67, 33].
2 Preliminaries
Diffusion Models
Denoising diffiusion probabilistic models (DDPM) [67, 33] are generative models that can convert the data distribution to simple distribution, (e.g., standard Gaussian, (0, I)), through a diffusion process. Specifically, given samples from a target distribution X5 ~ g(xp), the diffusion process is Markov chain that adds Gaussian noises to the data to generate latent states X1. Xy in the same sample space as Xy. The inference distribution of diffusion process is given by:
(xxT xx) = II
(xx (x:-1) t=1
(I)
To learn the parameters that characterize a distribution pe (xx) = f pe (Xo:T) )dx as an approxi- mation of q(xo), surrogate variational lower bound [67] was proposed to train this model:
L Eg1 - log po (x (x1) + Σ DKL (xxL(((xx1/xx/ xo)) (p) (x
+ D (x0)xxxxxxx)xxxxxxx) (2)

---

After training, samples can be generated by reverse Markov chain, i.e., first sampling ~ p(xr), and then repeatedly sampling x,-1 till we reach Xo-
As noted in [67, 68], the length 7 of a diffusion process is usually large (e.g. 7 = 1000 [33]) as contributes better approximation of Gaussian conditional distributions in the generative process. However, because of the large value of T', sampling from diffusion models can be visibly slower compared other deep generative models like GANs [29].
One feasible acceleration is to rewrite the forward process into a non-Markovian one that leads to a "shorter" and deterministic generative process, i.e., denoising diffusion implicit model [68] (DDIM). DDIM can be trained similarly to DDPM, using the variational lower bound shown in Eq. (2). Essentially, DDIM constructs nearly non-stochastic scheme that can quickly sample from the learned data distribution without introducing additional noises. Specifically, the scheme to generate a sample x given X, is:
(e) vI a1Eg (xL1
X1
+ V 1- a:-1 - oî (0) (xx) + OfEt
X vai-1
(3)
Var
where Ofl, OrT E [0, Gt I),, and Co. the noise given a noisy state Xt. Different values of o, define different generative processes. For a variance schedule By...Br. we use the notation a, = II'm - B.). When o, = ((1 0t-1)/(1 aa)i-aa)/ a (aoiv/1aaaa11 for all i, the generative process represents a DDPM. Setting oe = for all gives rise to a DDIM, which results a deterministic generating process except the initial sampling XT p(xT).
(1) (x,) is an estimator trained to prediet
Deep equilibrium models are a recently-proposse class of deep net- works that, in their forward pass, seek to find fixed point of single layer applied repeatedly to hidden state. Specifically, consider a deep feedforward model with T. layers:
Deep Equilibrium Models
z |+1|| =(²" (x)0)x)) for i = -..... L -
(4)
where x is the input injection, n is the hidden state of 7th layer, and f/ is a layer that defines the feature transformation. Assuming the above model is weight-tied, i.e., f!a = fo, Vi, then in the limit of infinite depth, the output z of this network converges fixed point zz.
lim fo (z (x)))))
(5)
1-000
Inspired from the neural convergence phenomenon, Deep equilibrium (DEQ) models [6] are proposed to directly compute this fixed point z² as the output, i.e.,
xx(z";x)= =
(6)
The equilibrium state z" can be solved by black-box solvers like Broyden's method [13], or Anderson acceleration [5]. To train this fixed-point system, Bai al. [6] leverage implicit differentiation to directly backpropagate through the equilibrium state zt using (11) memory complexity. DEQ is known as a principled framework for characterizing convergence and energy minimization in deep learning. We leave a detailed discussion in Sec. 6.
3 A Deep Equilibrium Approach to DDIMs
In this section, we present the main modeling contribution of the paper, a formulation of diffusion processes under the DEQ framework. Although diffusion models may seem to be natural fit for DEQ modeling (after all, we typically do not care about intermediate states in the denoising chain, but only the final clean image), there are several reasons why setting up the diffusion chain "naively" as DEQ (i.e., making fo be a single sampling step) does not ultimately lead to a functional algorithm. Most fundamentally, the diffusion process is not iime-invrriant (i.e., not "weight-tied" in the DEQ sense), and the final generated image is mauuticalll-speakkii independent of the noise used to generate it (i.e., not truly based upon "input injection" either).
Thus, at high level, our approach to building a DEQ version of the DDIM involves representing all the states Xo:T simultaneously within the DEQ state. The advantage of this approach is that I)
3

---

we can exactly capture the typical diffusion inference chain; and 2) we can create more expressive reverse process where the state x, updated based upon all previous states X;41:T, improving the inference process; 3) we can execute all steps of the inference chain in parallel rather than solely in sequence as typically required in diffusion models; and 4) we can use common DEQ acceleration methods, such as the Anderson solver [5] to find the fixed point, which makes the sampling process converge faster. A downside of this formulation is that we need to store all DEQ states simultaneously (i.e., only the images, not the intermediate network states).
3.1
A DEQ formulation of DDIMs (DEQ-DDIM)
The generative process of DDIM given by:
(t) v1 Cn (x1) + VI-av-i a-- c (x)), t= [L,..,,, T Var
X = va-1 X1
(7)
This process also lets us generate sample using subset of latent states (x Xrgh J. where (Ti...., { C T. While this helps in accelerating the overall generative process, there is a tradeoff between sampling quality and computational efficiency. As noted in Song et al. [68], larger T values lead to lower FID scores of the generated images but need more compute time; smaller are faster to sample from, but the resulting images have worse FID scores.
Reformulating this sampling process as a DEQ addresses multiple concerns raised above. We can define DEQ, with sequence of latent states X1:7 as its internal state, that simultaneously solves for the equilibrium points at all the timesteps. The global convergence of this process is upper bounded by T steps, by definition. To derive the DEQ formulation of the generative process, first we rearrange the terms in Eq. (7):
X
V
01-1
Xt
+
V
a
11-1(1 (1 as)
(0) Eg
(xx)
(8)
0t
022
14-11 (1(0)
Let c/ 11- a1-1
Then we can write
X;-1
x, Eo (xx)
(1) (t)
(9)
at
By induction, we can rewrite the above equation as:
T-1
XT
OT-k
XT
Σ T=
OT (1+1) EB
(x++1)
k E 1E,,,,,, O,..,
(10)
aT
This defines a "fully-upperttriangula"" inference process, where the update of x, depends on the noise prediction network eg applied to all subsequent states X++1:7; contrast to the traditional diffusion process, which updates x, based only on X,+1- Specifically, let h(-) represent the function that performs the operations in the equations (10) for a latent x, timestep t, and let h(-)) represent the function that performs the same set of operations across all the timesteps simultaneously. We can write the above set of equations fixed point system:
XT-1
h(xT)
h(xr-1x)
XT-2
:
h(xiT)
Xo
or,
Xo-T- h(xoTT-1:x))
(1I)
The above system of equations represent DEQ with X7 (O I) as input injection. We can simultaneously solve for the roots of this system of equations through black-box solvers like Anderson acceleration [5]. Let g(x xT) x(xo-T1::x)) - X -1, then we have
X
(12)

---

This DEQ formulation has multiple benefits. Solving for all the equilibria simultaneously leads to better estimation of the intermediate latent states x, fewer number of steps (i.e. S t steps for x,). This leads to faster convergence of the sampling process as the final sample Xo, which is dependent on the latent states of all the previous time steps, has a better estimate of these intermediate latent states. Note that by the same reasoning. the intermediate latent states converge faster too. Thus, we can get images with perceptual quality comparable to DDIM in a significantly fewer number of steps. Of course, we also note that the computational requirements of each individual step has significantly increased, but this at least largely offset by the fact that the steps can be executed as mani-batched in parallel over each state. Empirically, in fact, often notice significant speedup using this approach on tasks like single image generation.
This DEQ formulation of DDIM can be extended to the stochastic generative processes of DDIM with 1) > O, including that of DDPM (referred to as DEQ-sDDIM). The key idea sample noises for all the time steps along the sampling chain and treat this noise as an input injection to DEQ, in addition to Xy-
xOr RootSolver(g(xo:r -1 C1:T))
(13)
where RootSolver()) is any black-box fixed point solver, and c1:1 ~ (NO I) represents the input injected noises. We discuss this formulation in more detail in Appendix D.
4 Efficient Inversion of DDIM
One of the primary strengths DEQs is their constant memory consumption, for both forward pass and backward pass, regardless of their 'effective depth'. This leads to interesting application of DEQs in inverting DDIMs that fully leverages this advantage along with the other benefits discussed in the previous section.
Algorithm 1 A naive algorithm to invert DDIM
Algorithm 2 Inverting DDIM with DEQ
Input: A target image Xo D, (x(x,, t) a
Input: A target image Xo D, co t)
trained denoising diffusion model, N the total number of epochs
trained denoising diffusion model, N the total number of epochs
f denotes the sampling process in Eq (7)
g is the function in Eq. (12)
Initialize Xy ~ (N,0 for epochs from 1 to N do
Initialize X ~ (O
for epochs from to N do
for to do
}> Disable gradient computation xO.T RootSolver(g(xorr= x);
Sample - f (X),
end for
Enable gradient computation
Take gradient descent step on
Compute Loss L(X,, x5)
Use the 1-step grad to compute 8c/axr
V kx 1180 x 1o.
Take gradient descent step using above
end for Output: Xr
end for
Output: x-
4.1 Problem Setup
Given arbitrary image Xy ~ D, and denoising diffusion model ex(x,, f) trained on dataset D, model inversion seeks to determine the latent Xy ~ (0, I) that can generate an image Xy identical to the original image Xy through the generative process for DDIM described in Eq. (7). For input image Xq, and a generated image &n, this task needs to minimize the squared-Frobenius Frobenius distance between these images:
C(xo, &o) = ||xo - x
(14)
4.2 Inverting DDIM: The Naive Approach
relatively straightforward way to invert DDIM is to randomly sample X ~ (O I), and update via gradient descent by first estimating Xo using the generative process in Eq. (7) and backpropagating through this process after computing the loss objective in (14). The overall process has been summarized in Algorithm 1. This process has large computational overhead. Every training epoch
5

---

requires a sequential sampling for all T timesteps. Optimizing through this generative process would require the creation of a large computational graph for storing relevant intermediate variables necessary for the backward pass. Sequential sampling further slows down the entire process.
4.3 Efficient Inversion of DDIM with DEQs
Alternatively, we can use the DEQ formulation develop much more efficient inversion method.
We provide a high-level overview of this approach in Algorithm 2. We can apply implicit function theorem (IFT) to the fixed point, i.e, (12) to compute gradients the loss [(xo: x5) in (14) w.e.t. ()::
ac 0())
dL dxpo
(J bxi.7 (xce)))))
0h(xxor-1i22 -iiX))
(15)
O(-)
where (-) could be any of the latent states X1,..., XT, and J x is the inverse Jacobian of (Xo:T- Xr) evaluated at Tü.r- Refer to [6] for a detailed proof. Computing the inverse of Jacobian matrix can become computationally intractable, especially when the latent states x, high dimensional. Further, prior works [6, 8, 28J have reported growing instability of DEQs during training due to the ill-conditioning of Jacobian. Recent works [27, 26, 28, 9] suggest that we do not need an exact gradient to train DEQs. We can instead use an approximation to Eq. (15), i...
0C = 0()) 0x6.
OC
dh Oñ(xi (x
M
(16)
a())
where M is approximation of J
gv ix For example, [27, 26, 28] show that setting M I. i.., 1-step gradient, works well. In this work, we follow Geng et al. [28] to further add a damping factor to the 1-step gradient. The forward pass is given by:
x Roo
(17)
X = T -(x + 10--11 7) X
(18)
The gradients for the backward pass can be computed through standard autograd packages. We provide the PyTorch-style pseudocode of our approach in the Appendix B. Using inexact gradients for the backward pass has several benefits: 1) It remarkably improves the training stability of DEQs; 2) Our backward pass consists of a single step and is ultra-cheap to compute. It reduces the total training time by a significant amount. is easy to extend the strategy used in Algorithm 2 and use DEQs to invert DDIMs with stochastic generative process (referred to as DEQ-sDDIM), We provide the key steps of this approach Algorithm 4.
Experiments
We consider four datasets that have images of different resolutions for our experiments: CIFARIO (32 x32) [46], CelebA (64x64) [52], LSUN Bedroom (256x256) and LSUN Outdoor Church (256x256 256) [76]. For all the experiments, we use Anderson acceleration as the default fixed point solver. We use the pretrained denoising diffusion models from Ho et al. [33] for CIFARIO, LSUN Bedroom, and LSUN Outdoor Church, and from Song et al. [68] for CelebA. While training DEQs for model inversion, we use the 1-step gradient Eq. (18) to compute the backward pass. The damping factor for 1-step gradient is set to 0. 1. All the experiments have been performed on NVIDIA RTX A6000 GPUs We provide additional experimental details in the Appendix A. While the primary focus in this section will be on the DDIM with a deterministic generative process i.e., n = O, we also include few key results on stochastic version of DDIM (DEQ-sDDIM) here. More extensive experiments can be found in Appendix D.
5.1 Convergence of DEQ-DDIM
We verify that DEQ converges to a fixed point by plotting the values of [|ā(Xo:T) - X over Anderson solver steps. As seen in Figure 1, DEQ converges to fixed point for generative processes of different lengths. easier to reach simultaneous equilibria on smaller sequence lengths than larger sequence lengths. However, this does not affect the quality of images generated. We visualize

---

the latent states of DEQ in Figure 2. Our experiments demonstrate that DEQ is able to generate high-quality images in few 15 Anderson solver steps on diffusion chains that were trained much larger number of steps T. One might note that DEQs converge to limit cycle for diffusion processes with larger sequence lengths. This not a limitation we only want the latent states at the last few timesteps to converge well, which happens in practice as demonstrated in Fig. 2. Further, these residuals can be driven down by using more powerful solvers like quasi-Newton methods, e.g. Broyden's s method.
CIFAR10
DEQ on CelebA
10
T-10
T-10
10"
Diftrence norm (10))
1-500
Difference norm (NX)
|400
10
10'
10"
10"
10"
10"
10"
10"
10"
Steps
Figure 1: DEQ-DDIM finds equilibrium point. We plot the absolute fixed-point convergence
[}x(x):)) Xo:7 I|2 during forward pass of DEQ for CIFAR-10 (left) and CelebA (right) for different number of steps T. The shaded region indicates the maximum and minimum value encountered during any of the 25 runs.
Figure Visualization of intermediate latents x, DEQ-DDIM after 15 forward steps with Anderson solver for CIFAR- 10 (first row, T 500), CelebA (second row, T = 500). LSUN Bedroom (third row, T = 50, and LSUN Outdoor Church (fourth row, = 50). For T 500, we visualize every 50th latent, and for T 50, we visualize every 5th latent. In addition, we also visualize Xp:4 in the last columns.
5.2 Sample quality images generated with DEQ
We verify that DEQs can generate images of comparable quality to DDIM by reporting Fréchet Inception Distance (FID) [32] in Table For forward pass DEQ, we run Anderson solver maximum of 15 steps for each image. We report FID scores on 50,000 images, and average time to generate an image (including GPU time) on 500 images. We note significant gains in wall-clock time single-shot image generation with DEQs on images with smaller resolutions. Specifically, DEQs can generate images almost faster than the sequential sampling of DDIM on CIFAR- (32x32) and CelebA (64x64) We note that these gains vanish sequences of shorter lengths and images with larger resolutions as seen in case of LSUN Bedrooms, and Outdoor Churches (256x 256). This because the number of fixed point solver iterations needed for convergence becomes comparable to the length of diffusion chain for small values of T. Thus, lightweight updates performed on short diffusion chains for sequential sampling are faster compared compute heavy updates in DEQs.
We also report FID scores on DEQ-sDDIM for CIFAR1O Table 2. We run Anderson solver for maximum of 50 steps for each image. We observe that while DEQ-sDDIM is slower than DDIM,

---

it always generates images with comparable or better FID scores. For higher levels of stochasticity ie., for larger valued of 17. DEQ-sDDIM need more Anderson solver iterations to converge fixed point, which increases image generation wall-clock time. We include additional results in Appendix D.2. Finally, we also find that on full-batch inference with larger batches, sequential sampling might outperform DEQs, as DEQs would have larger memory requirements in this case, i.e, processing smaller batches of size B might be faster than processing larger batches of size BT.
DDPM
DDIM
DEQ-DDIM
Dataset
FID
Time
FID
Time
FID
Time
CIFARIO
1000
3.17
24.45s
4.07
20.16s
3.79
2.91s
CelebA
500
5.32
14.95s
3.66
10.31s
2.92
5.12s
LSUN Bedroom
25
184.05
1.72s
8.76
1.19s
8.73
3.82s
LSUN Church
25
122.18
1.77s
13.44
1.68s
13.55
3.99s
Table 1: FID scores and time for single image generation for DDPM, DDIM and DEQ-DDIM.
FID Scores
Time (in seconds)
π
T
DDIM
DEQ-SDDIM
DDIM
DEQ-sDDIM
0.2
20
7.19
6.99
0.33
0.51
0.5
8.35
8.22
0.35
0.51
1
20
18.37
17.72
0.34
0.93
0.2
50
4.69
4.44
0.88
0.88
0.5
50
5.26
4.99
0.83
1.00
1
50
8.02
7.85
0.83
1.58
Table 2: FID scores for single image generation for DDIM and DEQ-sDDIM on CIFAR1O. Note that DDPM [33] with a larger variance achieves FID scores of 133.37" and 32.72* respectively for T = 20 and T = 50, where indicates numbers reported from Song et al. [68]
5.3 Model Inversion of DDIM with DEQS
We report the minimum values of squared Frobenius norm between the recovered and target images averaged from 100 different runs in Table 3. We report results for DEQ with = 0 (i.e., DEQ-DDIM) in this table, and additional results for r) (i.e., DEQ-sDDIM) outperforms the baseline method on all the datasets by a significant margin. We also plot the training loss curves of DEQ-DDIM and the baseline in Figure 3. We observe that DEQ-DDIM converges faster and has much lower loss values than the haseline method induced by DDIM. We also visualize the images generated with the recovered latent states for DEQ-DDIM in Figure 4 and with DEQ-sDDIM in Figure 5. It is worth noting that images generated with DEQ capture more vivid details of the original images, like textures of foliage, crevices, and other finer details than the baseline. We include additional results of model inversion with DEQ-sDDIM on different datasets in Appendix D.3.
reported in Figure/ 17. DEQ
Baseline
DEQ-DDIM
Dataset
T
Min loss J
Avg Time (mins) +
Min loss +
Avg Time (mins) |
CIFAR10
100
15.74 + 8.7
49.07 = 1.76
0.76 + 0.35
12.99 + 0.97
CIFAR I0
10
2.59 + 3.67
14.36 0.26
0.68 + 0.32
2.54 4 0.41
CelebA
14.13 + 5.04
30.09 = 0.57
1.03 0.37
28.09 + 1.76
Bedroom
10
1114.49
+ 795.86
26.41 = 0.17
36.37 + 22.86
33.7 + 1.05
Church
10
1674.68 + 1432.54
29.7 = 0.75
47.94 + 24.78
33.54 + 3.02
Table 3: Comparison of minimum loss and average time required to generate an image. All the results have been reported 100 images. See Appendix A for detailed training settings.

---

Avg. Training Loss (CelebA)
Avg. Training Loss (LSUN Bedroom)
DEO Rasel
Basel
Arg. training lose (kog toak))
Avg triiiing tree
1070 1300
200 2003
2000
1000
-00
Epocha
Epochs
Figure 3: Training loss for CelebA and LSUN Bedroom over epochs. DEQ converges in fewer epochs, and achieves lower values of loss compared to the baseline. The shaded region indicates the maximum and minimum value of loss encountered during any of the 100 runs.
Related Work
Implicit Deep Learning Implicit deep learning is an emerging field that introduces structured methods to construct modern neural networks. Different from prior explicit counterparts defined by hierarchy or layer stacking, implicit models take advantage of dynamical systems [43, 23, 3], e.g., optimization [4, 72, 20, 27, 21]. differential equation [16, 22, 70, 30], or fixed-point system [6, 7, For instance, Neural ODE [16] describes continuous time-dependent system, while Deep Equilibrium (DEQ) model [6], which is actually path-indeppndent, is a new type of implicit models that outputs the equilibrium states of the underlying system, e.g., z" from z" fo(z", x) given the input x. This fixed- woidtppintt system can be solved by black-box solvers [5, 13]. and further accelerated by the neural solver [10] in the inference. An active topic is the stability [8, 28, 9] of such system as it will gradually deteriorate during training, albeit strong performances [16, 6, 8]. DEQ has achieved SOTA results wide-range of tasks like language modeling [6]. semantic segmentation [7], graph modeling [31, 51, 59, 15], object detection [73], optical flow estimation [9], robustness [74, 48), and generative models like normalizing flow [53], with theoretical guarantees [75, 38, 25, 50].
Figure Model inversion on CIFARIO, CelebA, LSUN Bedrooms and Churches, respectively, Each triplet has the original image (left), DDIM's inversion (middle), and DEQ-DDIM's inversion (right).
Diffusion Models Diffusion models [67, 33, 68), score-based generative models [69, 71], newly developed generative models that utilize an iterative denoising process to progressively sample from learned data distribution, which actually is the reverse of a forward diffusion process. They have demonstrated impressive fidelity for text-conditioned image generation [62] and outperformed state-of-the-are GANs on ImageNet [19]. Despite the superior practical results, diffusion models suffer from a plodding sampling speed, e.g., over hours to generate 50k CIFAR-sizE sized images [68]. To accelerate the sampling from diffusion models, researchers propose to skip part of the sampling steps by reframing the reverse chain [68, 45, 44], or distill the trained diffusion model into faster one [54, 65]. Plus, the forward and backward processes in diffusion models can be formulated as stochastic differential equations [71]. bridging diffusion models with Neural ODEs [16] in implicit deep learning. However, the community still lacks insights into the connection between DEQ and diffusion models, where we build our work to investigate this.

---

Figure 5: Model inversion of DEQ->DDIM on CIFARIO. CelebA, LSUN Bedrooms and Churches, respectively. Each triplet displays the original image (left), and images obtained through inversion with DEQ-sDDIM for 1) 0.5 (middle), and rj (right).
Model inversion Model inversion gives insights into the latent space of generative model, as an inability of a generative model to correctly reconstruct an image from its latent code is indicative of its inability to model all the attributes of image correctly. Further, the ability manipulate the latent codes to edit bigh-level attributes of images finds applications in many tasks like semantic image manipulation [77, 2), super resolution [14, 47J, in-painting [18], compressed sensing [12], etc. For generative models like GANs [29], inversion is non-trivial and requires alternate methods like learning mapping from an image to the latent code [11, 61, 78], and optimizing the latent code through optimizers, e.g., both gradient-basee based [1] and gradient-frre [34]. For diffusion models like DDPM [33], the generative process is stochastic, which can make model inversion very challenging. Many cxisting works based on diffusion models [55, 18, 71, 36, 40J edit images or solve inverse problems without requiring full model inversion. Instead, they do so by utilizing existing understanding diffusion models as presented in some recent works [71, 37, 391. Diffusion models have been widely applied to conditional image generation [17, 18, 57, 64, 36, 66, 40, 56]. Chung et al. [18] propose a method to reduce the number of steps in reverse conditional diffusion process through better initialization, based on the idea of contraction theory of stochastic differential equations. Our proposed method is orthogonal this work; explicitly model DDIM joint, multi-variate fixed point system and leverage black-box root solvers to solve for the fixed point and also allow for efficient differentiation.
7 Conclusion
We propose an approach to elegantly unify diffusion models and deep equilibrium (DEQ) models. We model the entire sampling chain of the denoising diffusion implicit model (DDIM) as a joint, multi- variate (deep) equilibrium model. This setup replaces the traditional sequential sampling process with a parallel one, thereby enabling us to enjoy speedup obtained from multiple GPUs. Further, we can leverage inexact gradients to optimize the entire sampling chain quickly, which results in significant gains in model inversion. We demonstrate the benefits of this approach on 1) single-shot image generation, where we were able to obtain FID scores on par with or slightly better than those of DDIM; and 2) model inversion, where we achieved much faster convergence. We also propose an easy way to extend DEQ formulation for deterministic DDIM to its stochastic variants. is possible to further speedup the sampling process by training DEQ model to predict the noise at particular timestep of the diffusion chain. We can jointly optimize the noise prediction network, and the latent variables of the diffusion chain, which we leave as future work.
8 Acknowledgements
Ashwini Pokle is supported by grant from the Bosch Center for Artificial Intelligence.
References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer
10

---

Vision, pages 4432 4441, 2019. (Cited on 10)
[2] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Styleflow: Atrriuuee-oodittooeed exploration of stylegan- images using conditional continuous normalizing flows. ACM Transactions on Graphics (TOG), 40(3):1-21, 2021. (Cited on 2, 10)
[3] Brandon Amos. Tutorial on amortized optimization for learning to optimize over continuous domains. arXiv preprint arXiv:220..00665, 2022. (Cited 9)
[4] Brandon Amos and J. Zico Kolter. OptNet: Differentiable optimization as a layer in neural networks. In International Conference Machine Learning (ICML), 2017. (Cited on 9)
[5] Donald G Anderson. Iterative procedures for nonlinear integral equations. Journal of the ACM (JACM), 1965. (Cited on 3, 4, 9, 17, 21)
[6] Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Neural Information Processing Systems (NeurIPS), 2019. (Cited on 2, 3, 6, 9, 19)
[7] Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Multiscale deep equilibrium models. Neural Information Processing Systems (NeurIPS), 2020. (Cited on 9)
[8] Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Stabilizing equilibrium models by jacobian regularization. arXiv preprint xrXi:22106.13342 2021. (Cited on 6, 9)
[9] Shaojie Bai, Zhengyang Geng, Yash Savani, and Zico Kolter. Deep equilibrium optical flow estimation. arXiv preprint arXiv:2204.084422 2022. (Cited on 9)
[10] Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Neural deep equilibrium solvers. In International Conference on Learning Representations, 2022. (Cited on 9)
[11] David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun- Yan Zhu, and Antonio Torralba. Semantic photo manipulation with generative image prior. arXiv preprint arXiv:2005.07727, 2020. (Cited on 10)
[12] Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using generative models. In International Conference on Machine Learning, pages 537-546. PMIR, 2017. (Cited on 10)
[13] Charles G Broyden. A class of methods for solving nonlinear simultancous equations. Mathe- matics of computation, 1965. (Cited 3, 9, 19)
[14] Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy. Glean: Generative latent bank for large-factor image super-resolution. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 14245-142544 2021. (Cited on 10)
[15] Qi Chen, Yifei Wang. Yisen Wang. Jiansheng Yang, and Zhouchen Lin. Optimization-inducee graph implicit nonlinear diffusion. In International Conference on Machine Learning, pages PMLR, 2022. (Cited on 9)
[16] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Neural Information Processing Systems (NeurIPS), 2018 (Cited on 9)
[17] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. lyv:: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.0299 028308 2021. (Cited on 10)
[18] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-diffuse faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction. arXiv preprins arXiv:2/ 02.05 146, 2021. (Cited 10)
[19] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021. (Cited on 1, 9)

---

[20] Josip Djolonga and Andreas Krause. Differentiable learning of submodular models. Advances in Neural Information Processing Systems, 30, 2017. (Cited on 9)
[21] Priya L. Donti, David Rolnick, and J Zico Kolter. DC3: A learning method for optimization with hard constraints. In International Conference Learning Representations (ICLR), 2021. (Cited on 9)
[22] Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural ODEs. In Neural Information Processing Systems (NeurIPS), 2019. (Cited on 9)
[23] Laurent EI Ghaoui, Fangda Gu, Bertrand Travacca, and Armin Askari. Implicit deep learning. arXiv:1908.0633 06315, 2019. (Cited on 9)
[24] Thorsten Falk, Dominic Mai, Robert Bensch, Özgün Çişek, Ahmed Abdulkadir, Yassine Marrakchi, Anton Böhm, Jan Deubner, Zoe Jäckel, Katharina Seiwald, et al. U-net: deep learning for cell counting, detection, and morphometry. Nature methods, 16(1):67-70, 2019. (Cited on 17)
[25] Zhili Feng and Zico Kolter. On the neural tangent kernel of equilibrium models, 2021. (Cited on 9)
[26] Samy Wu Fung. Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. Fixed point networks: Implicit depth models with jacobian- free backprop. arXiv e-prints, pages arXiv-2103, 2021. (Cited on 6)
[27] Zhengyang Geng, Meng- Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is atten- tion better than matrix decomposition? In International Conference on Learning Representa- tions (ICLR), 2021. (Cited on 6, 9)
[28] Zhengyang Geng, Xin-Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. Neural Information Processing Systems (NeurIPS), 2021. (Cited on 6, 17)
[29] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014. (Cited on 3, 10)
[30] Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR), 2022. (Cited on
9)
[31] Fangda Gu. Heng Chang, Wenwu Zhu, Somaych Sojoudi. and Laurent El Ghaoui. Implicit Graph Neural Networks. In Neural Information Processing Systems (NeurIPS), pages 11984- 11995, 2020. (Cited on 9)
[32] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017. (Cited on 7)
[33] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Neural Information Processing Systems (NeurIPS), 2020. (Cited on 2. 3. 6, 9, 10, 17, 20)
[34] Minyoung Huh, Richard Zhang, Jun- Zhu, Sylvain Paris, and Aaron Hertzmann. Transform- ing and projecting images into class-conditional generative networks. In European Conference on Computer Vision, pages 17-34, Springer, 2020. (Cited on 10)
[35] Thibaut Issenhuth, Ugo Tanielian, Jérémie Mary, and David Picard, Edibert, a generative model for image editing. arXiv preprint axXiv:2111.i5266 2021. (Cited on 2)
[36] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 111448--195444 14954, 2021. (Cited on 10)
[37] Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in denoiser. arXiv preprint 27X1v:2007.13640, 2020. (Cited on 10)
12

---

[38] Kenji Kawaguchi. On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers. International Conference on Learning Representations (ICLR), 2020. (Cited on 9)
[39] Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34:21757-21769, 2021. (Cited on 10)
[40] Gwanghyun Kim and Jong Chul Ye. Diffusionclip: Text-guided image manipulation using diffusion models. arXiv preprint arXiv:21100027111 2021. (Cited on 10)
[41] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. (Cited on 17)
[42] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2107.00630, 2021. (Cited on
[43] J. Zico Kolter, David Duvenaud, and Matthew Johnson. Deep implicit layers tutorial neural ODEs, deep equilibirum models, and beyond. Neural Information Processing Systems Tutorial, 2020. (Cited on 9, 17)
[44] Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00132 2021. (Cited on 1, 9)
[45] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis, In International Conference on Learning Representations (ICLR), 2021. (Cited on
[46] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. (Cited on 2, 6)
[47] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 2022. (Cited on 10)
[48] Mingjie Li, Yisen Wang, and Zhouchen Lin. Cerdeq: Certifiable deep equilibrium model. In International Conference Machine Learning, 2022. (Cited on 9)
[49] Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems,
2021. (Cited on 2)
[50] Zenan Ling, Xingyu Xie, Qiuhao Wang, Zongpeng Zhang, and Zhouchen Lin. Global conver- gence of over wver-parameterized deep equilibrium models. arXiv preprint 2xXi:22055118144 2022. (Cited on 9)
[51] Juncheng Liu, Kenji Kawaguchi, Bryan Hooi, Yiwei Wang, and Xiaokui Xiao. EIGNN: Effcienf infihite-depth graph neural networks. In Neural Information Processing Systems (NeurIPS), 2021. (Cited on 9)
[52] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015, (Cited on 2, 6)
[53] Cheng Lu, Jianfei Chen, Chongxuan Li, Qiuhao Wang, and Jun Zhu. Implicit normalizing flows.
In International Conference on Learning Representations (ICLR), 2021. (Cited on 9)
[54] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. ArXiv, abs/2101 008.2200000288, 2021. (Cited on 1, 9)
[55] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint 2rX1v:2108.01073, 2021. (Cited on 2, 10)
13

---

[56] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Hya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprini arXiv:21 10741, 2021. (Cited on 10)
[57] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021. (Cited on 10)
[58] Weili Nie, Brandon Guo, Yujia Huang. Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.004660, 2022. (Cited
on 2)
[59] Junyoung Park, Jinhyun Choo, and Jinkyoo Park. Convergent graph solvers. arXiv preprint arXiv:2106.01680, 2021. (Cited on 9)
[60] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017. (Cited on 17)
[61] Guim Peramnu, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez. Invertible conditional gans for image editing. arXiv preprint arXiv:161 1611,6635, 2016. (Cited on 10)
[62] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint urXiv:2204.0611: 2022. (Cited on 9)
[63] Ardavan Saeedi, Matthew Hoffman, Stephen DiVerdi, Asma Ghandeharioun, Matthew Johnson, and Ryan Adams. Multimodal prediction and personalization of photo edits with deep generative models. In International Conference on Artificial Intelligence and Statistics, pages 1317.
PMLR, 2018. (Cited on 2)
[64] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021. (Cited on 10)
[65] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models.
In International Conference on Learning Representations (ICLR), 2022. (Cited on 1, 9)
[66] Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05336 2021. (Cited
on 10)
[67] Jascha Sohl-Dicke Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamice In International Conference on Machine Learning (ICML), 2015. (Cited on 2, 3, 9, 20)
[68] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020. (Cited on 1, 2, 3, 4, 6, 8, 17, 20, 22)
[69] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019. (Cited on 1. 9)
[70] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICIR), 2021. (Cited on 9)
[71] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021. (Cited on 9, 10)
[72] Po-We Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. Satnet Bridging deep learning and logical reasoning using a differentiable satisfiability solver. International Conference on Machine Learning (ICML), 2019. (Cited on 9)
14

---

[73] Tiancai Wang, Xiangyu Zhang, and Jian Sun. Implicit Feature Pyramid Network for Object Detection. arXiv preprint arX1v:2012.13563, 2020. (Cited on 9)
[74] Colin Wei and Zico Kolter. Certified robustness for deep equilibrium models via interval bound propagation. In International Conference on Learning Representations, 2022. (Cited on 9)
[75] Ezra Winston and J. Zico Kolter. Monotone operator equilibrium networks. In Neural Informa- tion Processing Systems (NeurIPS), 2020. (Cited on 9)
[76] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. CoRR, abs/1506.0333 033600365, 2015. (Cited on 6)
[77] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. European conference on computer vision, pages 592-608. Springer, 2020. (Cited on
2, 10)
[78] Jun Yan Zhu, Philipp Krähenbühh, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on the natural image manifold. In European conference on computer vision, pages 597-613. Springer, 2016. (Cited on 10)
15

---

Checklist
Did you include the license the code and datasets? [No] The code and the data are The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "ZLAbbvBISae", "reviewer": "Reviewer_MgS2", "summary": "This paper proposes an extension to \"denoising diffusion implicit model\" (DDIM), a variant of denoising diffusion probabilistic model (DDPM). Here I refer to it as DEQ-DDIM. Specifically, it regards the denoising iterations as fixed-point iterations and solves all the denoised images efficiently using Anderson acceleration. This is an interesting exploration; however I have concerns of the reduced expressiveness brought about by DDIM and DEQ-DDIM.\n", "questions": "1. I understand that in the DDIM and this paper, the authors use $\\alpha$ to denote $\\bar{\\alpha}$ in the DDPM paper. However, the authors are suggested to explicitly point it out to avoid possible confusions of readers.\n2. Typos and grammar:\n\na) line 41: join equilibria -> joint equilibria\n\nb) line 137: compared to the traditional -> in contrast to the traditional\n", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. It's a novel and interesting perspective to view the denoising iterations as fixed-point iterations (seeking for converged solutions through iterations). \n\nWeaknesses:\n1. DEQ-DDIM inherits the limitations of DDIM: they remove the injected random noises during denoising, therefore the results are deterministic (given the noise image x_T). This limitation is fundamental: it makes DDIM and variants less being probabilistic models. A direct consequence is the generated images of DDIM and variants has reduced diversity compared with DDPM (refer to the recall of DDIM in [a]). It's debatable whether such a sacrifice for speedup is worthy. \n~~2. Another possible inherent flaw (but I might be wrong) of DEQ-DDIM is, as the authors correctly pointed out, \"the diffusion process is not time-invariant (i.e., not 'weight-tied' in the DEQ sense)\". More specifically, the denoising function takes the (time embedding of) time t as extra input. In my view, the time t controls the variance of the noises to be estimated, therefore it's indispensable. But if taking time t as extra input, then can we still view DDIM as fixed-point iterations? If not, then speedup using Anderson acceleration is not applicable.~~\n\n[a] Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. ICLR 2022.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LDYC4ofhu3a", "reviewer": "Reviewer_7km1", "summary": "The paper shows a deep equilibrium interpretation of denoising diffusion implicit models (DDIM). This approach provides the traditionally sequential diffusion models with parallelization capabilities, resulting in faster single image generation and model inversion with less memory consumption.", "questions": "- Is algorithm 1 the same as the inversion method suggested in DDIM (section 5.4 of DDIM's arXiv version)?\n\n- What is the reasoning behind the choices of T for different datasets?\n\n---\n\nPost-rebuttal:\n\nThe authors have sufficiently answered my questions and concerns, and have shown willingness to provide more experiments in the final version. I have thus raised my score to 7.", "limitations": "The paper holds no potential negative societal impact in my opinion, and the authors have aptly ignored this as it is not applicable.\n\nAs for the limitations, the authors touch upon their method's limitations in the paper text, addressing the larger memory requirements of each step. However, DDIM's results should be made clearer. It should also be mentioned that sequential models can be differentiated through with constant memory complexity as well (yes, current autograd methods cannot automatically achieve this, but it still is possible).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Well-founded and strong mathematical viewpoints on diffusion models. This paper provides a novel and significant view on diffusion models as upper-triangular iterative systems, and shows an interesting method to differentiate through them.\n- Impressive model inversion capabilities, especially as demonstrated qualitatively in Figure 4.\n\n\nWeaknesses:\n- Line 53: \"modern autograd packages would require storing the entire computational graph for all T states by default\". While true, it is still possible to differentiate through the entire sequential diffusion process without requiring this. For example, this was done in [R1]. This fact is not made sufficiently clear to the reader. The same point is made on line 159, without stating \"by default\". \n- The approach loses its appeal when considering batched image generation.\n- In Table 1, the reporting of DDIM results is inadequate. DDIM achieves an FID of 4.07 using a particular setting, whereas it can also achieve an FID of 3.17 using a different parameterization, under the same timeframe. The fact that it edges out this paper's result is okay, as this paper provides a significant speedup. However, this result should be adequately reported in the table.\n- Similarly, DDIM can be sped up significantly and achieves remarkable results. The authors suggest that their method can be used in tandem with other speedup methods, which sounds theoretically plausible, but such results are not reported in the paper.\n- Results seem to only mildly improve upon previous results seen in DDIM and others. An additional experiment showing the use of both the novel acceleration method and previous speedup methods simultaneously would better demonstrate the capabilities of this method.\n\nMinor comments:\n- Typo on line 41: \"join\" instead of \"joint\"\n- Typo on line 209: \"effect\" instead of \"affect\"\n- On lines 266-268, some citations are not described correctly. Citations [45, 17, 57, 32] do not manipulate latent codes to edit high-level attributes of images. They utilize the prevalent understanding of diffusion models for the respective tasks they solve, similar to ideas first presented in [57, R2, R3]. The text should be reworded and show the existing and suggested citations as methods that edit images or solve inverse problems without requiring full model inversion.\n\n[R1] Nie, W., Guo, B., Huang, Y., Xiao, C., Vahdat, A., & Anandkumar, A. (2022). Diffusion Models for Adversarial Purification. arXiv preprint arXiv:2205.07460.\n\n[R2] Kadkhodaie, Z., & Simoncelli, E. P. (2020). Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007.13640.\n\n[R3] Kawar, B., Vaksman, G., & Elad, M. (2021). SNIPS: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34, 21757-21769.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KybxDiFBnB", "reviewer": "Reviewer_cuLf", "summary": "This paper develops a variant (deep equilibrium model (DEQ)) of an existing method (DDIM) with the goal of efficient sampling and model inversion while still maintaining the performance.  In particular, it treats the sampling step sequence of DDIM（denoising diffusion implicit model, where the sampling step is deterministic and no longer a Markov chain.) as a fixed-point set, so that a fixed-point solver (e.g., Anderson acceleration, etc.) is able to jointly minimize the entire sampling chain. The authors show the practical relevance of their proposal in single-shot image sampling and model inversion on several benchmark datasets.", "questions": "1), From the high-level point of view, what is the practical meaning of single-shot image sampling? From my perspective, I think image restoration and reconstruction (e.g., Inpainting, super-resolution, medical image inverse problems, etc.) might be a potential direction. However, as mentioned by authors, the proposal may work efficiently only on shorter sampling lengths, and on images with smaller resolutions (32x32, 64x64). As a result, the rationale for single sampling need to be discussed from very beginning.\n \n2), In the same line above, there is supposed to have an original DDPM model for references in all the experiments, as presented in [Song et al., 2021], DDPM models works better than DDIM when sampling steps turn to be larger. The addition of these comparisons will clearly show the regime that suits to this proposal.\n\n3), In line 146-147, the statement, “Solving for all the equilibria simultaneously leads to a better estimation of the intermediate latent states x_t in a fewer number of steps (i.e., $≤ t$ steps for $x_t$).”, is confusing and need to be justified with some ablation studies. The jointly sampling still need to process all the T steps, right?\n\n4), In this work, it is claimed that the steps can be executed as mini-batched in parallel. Is the parallel computing already used at DDIM sampling, such as the outcomes in Table 1 and other experiments? If so, please consider adding more implementation details in the revision. Some additional explanation through Eq. (7) would be helpful. In addition, will the parallel computing work in the multi-batch cases?\n\n5), What is the initialization $x_ {0: T-1}$ of the fixed-point iterations?  In the original DEQ implementations, the initialization is usually set to zero. Will different initializations result in different convergence and sampling performance?\n\n6), I’m curious about how this proposal can be applied to a more general diffusion models that the sampling step is not deterministic. Seems that the current methodological statements in Sec. 3, from Eq. 9 to Eq. 10, are not clear for stochastic sampling. It would be very helpful for authors to discuss this point. \n\n7), When comparing with DDIM, only LSUN datasets with resolution 256x256 are considered. As exploring advances in large scale diffusion models for image generation is impactful, I wonder how the proposed jointly sampling compares to DDIM on ImageNet of resolution 512x512? The pretrained models are publicly available in https://github.com/openai/guided-diffusion.\n\n8), Since the proposed DEQ method is mainly based on diffusion sampling [30] and DDIM sampling [54], how the architecture improvements apply to the SDE version of score-based models and the corresponding conditional SDEs?\n\n9), The current gradients calculation of model inversion is inexact due to the usage of Jacobian free. I wonder the current proposal can fed into the exact gradient calculation for the backward pass (e.g., Neumann backpropagation, etc.).\n", "limitations": "The limitations of the proposal are missing or at least not well stated in the current manuscript. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper builds on other works in the recent literature and proposes something useful and novel. It is overall well written and reports (somewhat) competitive results with both quantitative and qualitative validation for image sampling and model inversion.  While I think the methodological aspect is novel enough for publication, the rationale for single image sampling is unclear and need to be further justified. Please find my technical comments bellow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept