Gradient-free Decoder Inversion in Latent Diffusion Models
Authors: Seongmin Hong
Suh Yoon Jeon
Kyeonghyun Lee
Ernest K. Ryu
Se Young Chun
Venue: NeurIPS 2024 poster (2024)
Keywords: Latent diffusion model
Inversion
Gradient-free inversion
Resource-efficient inversion
Primary area: diffusion_based_models
TL;DR: We propose an efficient gradient-free decoder inversion for LDMs for ensuring invertible latent diffusion model, which significantly reduced runtime and memory usage compared to gradient-based methods in various recent LDMs.

Abstract: In latent diffusion models (LDMs), denoising diffusion process efficiently takes place on latent space whose dimension is lower than that of pixel space. Decoder is typically used to transform the representation in latent space to that in pixel space. While a decoder is assumed to have an encoder as an accurate inverse, exact encoder-decoder pair rarely exists in practice even though applications often require precise inversion of decoder. In other words, encoder is not the left-inverse but the right-inverse of the decoder; decoder inversion seeks the left-inverse. Prior works for decoder inversion in LDMs employed gradient descent inspired by inversions of generative adversarial networks. However, gradient-based methods require larger GPU memory and longer computation time for larger latent space. For example, recent video LDMs can generate more than 16 frames, but GPUs with 24 GB memory can only perform gradient-based decoder inversion for 4 frames. Here, we propose an efficient gradient-free decoder inversion for LDMs, which can be applied to diverse latent models. Theoretical convergence property of our proposed inversion has been investigated not only for the forward step method, but also for the inertial Krasnoselskii-Mann (KM) iterations under mild assumption on cocoercivity that is satisfied by recent LDMs. Our proposed gradient-free method with Adam optimizer and learning rate scheduling significantly reduced computation time and memory usage over prior gradient-based methods and enabled efficient computation in applications such as noise-space watermarking and background-preserving image editing while achieving comparable error levels.

Gradient-free Decoder Inversion in Latent Diffusion Models
Seongmin Hong'
Suh Yoon Jeon'
Kyeonghyun Lee'
Ernest K. Ryu
Se Young Chun 13.-
Dept. of Electrical and Computer Engineering, 2IMCC & IPAI, Seoul National University Dept. of Mathematics, University of California, Los Angeles
{smhongok, euniejeon, litiphysics, sychun) ac kr, eryu@math, ucla acla Project page: https: bttps://smiir //sshhngooo... github. io /de//cciiie inv html
Abstract
In latent diffusion models (LDMs), denoising diffusion process efficiently takes place latent space whose dimension is lower than that of pixel space. Decoder is typically used to transform the representation in latent space to that in pixel space. While a decoder is assumed to have an encoder an accurate inverse, exact encoder-decoder pair rarely exists in practice even though applications often require precise inversion of decoder. In other words, encoder not the left-inverse but the right-inverse of the decoder; decoder inversion seeks the left- inferse Prior works for decoder inversion LDMs employed gradient descent inspired by inversions of generative adversarial networks. However, gradient- baseed methods require larger GPU memory and longer computation time for larger latent space. For example, recent video LDMs can generate more than 16 frames, but GPUs with 24 GB memory can only perform gradient-based decoder inversion for frames. Here, propose an efficient gradient-free decoder inversion for LDMs, which can be applied to diverse latent models. Theoretical convergence property of our proposed inversion has been investigated not only for the forward step method, but also for the inertial Krasnoselskii-Men Mannn (KM) iterations under mild assumption on cocoercivity that is satisfied by recent LDMs. Our proposed gradient-fres method with Adam optimizer and learning rate scheduling significantly reduced computation time and memory usage over prior gradient-bart based methods and enabled efficient computation image
in applications such as noise-space watermarking and editing while achieving comparable error levels.
Introduction
Deep generative models have been actively investigated over the past decade across numerous 35 38), video (5]
modalities such as image [2
41
42
45 46), audio [18 21 22
and molecular structure 17 13
They can sample new data points from the distribution of the
training set, can model priors to solve regression problems, and can be used for applications like editing, retrieval, and density estimation. Representative classes of deep generative models include generative adversarial networks (GANs), variational autoencoders (VAEs), normalizing flows (NFs), and diffusion models (DMs). They used to be known not to simultaneously satisfy the three key requirements: (i) high-quality samples, (ii) diversity, and (iii) fast sampling, also referred to as the generative learning trilemma [50]. However, recent DMs such as Rectified Flow (3), Adversarial
Diffusion Distillation 40 and Consistency models [3) require just one step for sampling. Thus,
"Co-corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: Comparison of deep generative models. DMs have overcome the generative learning trilenima [50], but they still lack invertibility compared to other models. In particular, LDMs have necessitated additional decoder inversion that has traditionally been addressed through memory- intensive and time-consuming gradient-based optimization methods. Here, we propose a method for efficiently (ie., gradient-free) ensuring invertibility in LDM
Generative learning trillema
50
High-quality samples
Fast sampling
How to achieve invertibility?
Diversity
GAN
x
Optimization / learning-based GAN inversion
NF
X
V
Naturally exact inversion
DM
V
Xcan ((29)
Diffusion: Solving ODE backward
(LDM) Decoder: Optimization saasd inversion
DMs have overcome the trilemma and the old problem of slow sampling in recent DMs is no longer a concern as shown in the last row of Table As a result, DMs have become one of the most prominent deep generative models, especially for image and video. However, there are still remaining challenges for DMs such as achieving invertibility, which is well-supported by other models (see Table D
Achieving invertibility for deep generative models has been an important topic. In GANs, prior works proposed GAN inversion techniques (49] (i.e., to find the latent vector that generated the image) and their applications such as real image editing [1. NFs are naturally invertible deep generative models by construction with related interesting applications [17. There have also been attempts achieve and utilize invertibility for DMs. The most popular one is the naive DDIM inversion that reverses the sampling process of DDIM's deterministic denoising 41, i.e., adding the estimated noise. While the naive DDIM inversion enables image editing, it is known to be somewhat inaccurate for seeking the true latent 11. for watermarking [48] and for
wwwgground-preeervii image editing [32]. To better
ensure the exactness of the inversion. several prior works 51 tailored algorithms than the naive DDIM inversion.
10 have proposed more
The use of latents in DMs has made ensuring invertibility more difficult. Latent diffusion models (LDMs) were proposed move the diffusion denoising process from the pixel space to the (low- dimensional) latent space, thus efficiently generating high-quality and large-scale samples 337 This issue may be critical since many recent popular DMs operate in latent space [37 47 52
27
40 45
5
However, latents are usually lossy compression of pixels, making one-to-one mapping between the latent and pixel spaces very challenging. Thus, the accuracy of inversion in LDMs typically lower than that in pixel-space [4], requiring additional efforts to compensate for it as illustrated in Figure [b One could naïvely employ a gradient-based GAN inversion method to the decoder of LDMs, but it required very large GPU memory and computation time [14, especially for large-scale LDMs. Moreover, recent video LDMs 5 making gradient-based GAN inversion infeasible in single GPU.
52
45
46
generate dozens of frames once,
In this work, we aim to achieve better invertibility in LDM shown in Table [ Specifically, we propose decoder inversion to overcome the difficulty of ensuring invertibility in LDM due to the inexact encoder-decoder pair. The proposed decoder inversion is gradient-fred frree, which is faster and
more memory-efficient than the gradient-based methods suggested in GAN inversion. Section 3 describes the motivation and analysis of our decoder inversion, including the theoretical convergence guarantee under mild assumption not only for the vanilla forward step method, but also for the inertial Krasnoselskii-Mane (KM) iterations. The assumption for our theorems is also validated by showing experimental convergence to the ground truth as well as the effectiveness of momentum. Section Dettionndescrinee described how to refine our algorithm by integrating popular optimization techniques such as Adam [16] and learning rate scheduling as practical extensions of our gradient-free decoder inversion, demonstrating that our method works well on various latest LDMs compared to existing gradient- based methods. Lastly, Section 5 showcases interesting applications (noise space watermarking and background-pen preserriig image editing) where our proposed gradient-free decoder inversion can be effectively utilized. The contributions of this work are:
2

---

Inversion Error
VAE
Our contribution
Diffusion
Learning based GAN inversion
2T (Noise)
20
D
(Image)
LDM
LDM Pixel-space DM
Pixel-space DM Opt.-based GAN inversion
2T (Noise)
20 (Image)
NF
Inversion Runtime
1-step grad-free
Iterative grad-based
(a) LDM vs Pixel-space DM
(b) Invertibility of generative models
Figure I: (a) The difference between LDM and pixel-space DM lies in the use of the decoder (D). (b) This difference has caused the performance gap in exact inversions. Pixel-space DMs and gradient- based GAN inversion methods are located on the right side due to iterative gradient back puc--propggttioss If gradient-base based decoder inversions are used, which are computationally intensive, LDM is located on the rightmost side to address the lossiness of latents. Our proposed gradient- decoder inversion method allows to efficiently handle the transformation between latent and pixel spaces.
proposing a gradient-free decoder inversion algorithm to achieve better invertibility diverse latest LDMs. Our method has the following advantages with experimental evidences:
Fast: up to 5x faster, 1.89 s vs 9.51 s to achieve -16.4 dB (Fig. 3c and Tab. S1c) Accurate: up to 2.3 dB lower, -21.37 dB vs -19.06 dB in 25.1 (Fig. 3b and Tab. S1b) Memory-efficient: up to 89 % can be saved, 7.13 GB vs 64.7 GB (Fig. 3b) Precision-flexible: 16-bit vs 32-bit (Fig. 3)
theoretically guaranteeing the convergence to the ground truth not only for the vanilla forward step method, but also for the inertial KM iterations,
showcasing that our proposed gradient-free decoder inversion can be used in interesting applications.
Backgrounds
2.1 Latent diffusion models (LDMs)
Stable Diffusion 2.1 37 is a widely known open-source text-to-image generation model. La LaVie 47 text-conditioned video generation model, which can generate consecutive frames per inference. InstaFlow [24] is a one-step text-to-image generation model that can generate images whose quality is good Stable Diffusion. These three LDMs will be used in all the experiments of our work.
Exact inversion of accelerated DMs. Another factor that makes the exact inversion of DMs difficult, besides the use of latents, is acceleration. Due the of high-order ODE solvers, the exact inversion of DPM-Solvers 25 26 is not as straightforward as in DDIM. Exact inversion algorithms for some DPM-Solvers have been proposed [14], but obtaining the exact inverses for other 24 has not been explored.
accelerated DMs 23
2.2 Optimization-ban GAN inversion
For target image æ, existing optimization-base GAN inversion methods typically perform the following optimization with respect to latent vector z (usually assumed to lie on simple distribution such (0, 1)):
min f(x, (())),
where & is distance metric and G the generator part GAN 49
Most works solve Eq. [1
through backpropagation with an optimizer. For optimizers, Adam [16] is employed in []] and

---

L-BFGS 20 is employed in [54]. Some more recent works optimized the transformed latent feature w. which can be defined as w MLP(z), in StyleGAN [T].
To exploit the prior knowledge that z lies on the range space an encoder, prior works often perform the following two methods:
(A) The output of the encoder & is used as initial point of the optimization process.
(B) The comain-guidess encoder is used to regularize the latent code within the semantic domain of the generator. For example, min || ((2)ll2
+ Areg | {()
Since we already have the encoder, we employ ( novel way over (B)
too. On the other hand, we will utilize the encoder
2.3 Gradient-based decoder inversion in LDMs
Recent works 14 performed the same as Eq. (hence they are iterative as illustrated in Figure lb) for the decoder inversion:
min
((z)))
with gradient-basee based method. For simplicity, we omit the subscript by setting zo = 2. The vanilla gradient descent algorithm can be expressed follows:
b+ 2 pv [Vx(x,
Advancements in backpropagation algorithms and libraries have made the computation of V V==() more efficient, but it still requires significant amount of GPU memory usage and lengthy runtime compared to inference. As the size of generated outputs of recent LDMs continues to increase, the computation of V D through backpropagation getting more and more burdensome.
3 Gradient-free decoder inversion in LDMs
In this work, an alternative method to Eq. 3 we propose the following decoder inversion method, which is form of the forward step method 39]
x² -<EE(Dzz*)) E(x)).
(4)
where the initial latent vector estimate n0 E(x) that is the same (A) in Section 2.2
3.1 Motivation
The definition of the decoder inversion problem for the given image x as follows:
find
D(2).
(5)
#ER*
Since directly dealing with Eq. (5) is impractical, relax it solving the following problem:
find
E(z) (D)()))
(6)
zER
The following Remark demonstrates that solving Eq. (6) easier than solving Eq. (5).
Remark (Eq. 6 easier than Eq.
(5) {=|æ = (=)) (=|E(x) (D())))
Remark [ true because DDx)) {xx) (z)) Equation ( equivalent the following:
find ZEBF
= == ====)((((()) E(x)), vp E
where Eq. refers to finding a fixed point of an operation I (--(E(D(.)) - E(x)) whose another form is Eq. the forward step method.

---

3.2 Convergence analysis on forward step method
Here, we demonstrate that our method converges to a fixed-poing under reasonable conditions. Furthermore, although we solved an easier problem represented by Eq. (6) rather than directly solving
Eq. ( we show that our proposed method surprisingly finds the true solution of Eq. (
Theorem 1 (Convergence of the forward step method), Let > 0. 0 p 23, and E R.V.
Assume T(-) = E D()) - {()) is continuous. Consider the iteration
*+1 = z² -pT=*
for k ==0, =,0,1
(8)
Assume x² is zero of T (t.e., T2" = 0) and
(T = 2 B for k = 0,1,...
(9)
Then, Tz + O. furthermore, z² + =²0 then ~ is a zero Tf (i.e. T= = 0).
Proof outline. Equation ( makes || == 113 S || - - p) p) x1 Then, sum for k = ,,...,00 DC.
Theorem assumes that Eq. (9) holds, which is refered as 3-cocoercivity. The assumption makes
sense because we expect ED ~ I. In fact, for linear autoencoders, it is proven by Baldi and Hornik that ED the following Remark 2
Remark (ED I in linear autoencoders). Let E R. be random vector such that B[xx*] EER N.. Assume Σ has distinct positive eigenvalues. Consider the optimization problem
min
Ex |x: DE
DERNXP, EERPxN EERPXN
Then, ED = I (identity matrix).
I is 1-cocoercive, which suggests that our assumption is reasonable. In Section 3.4 further demonstrate that the assumption is reasonable experimentally and 2 actually converges well.
3.3 Convergence analysis on momentum for acceleration
Momentum is widely used in optimization keep the optimization process going in the right direction, even when gradients are noisy or the landscape is flat. Among many momentum algorithms, employ the inertial Krasnoselskii-Man (KM) iteration [19 28 29 and analyze it since the convergence is guaranteed with some assumptions 29 be defined as follows:
The inertial KM iteration our setting can
y = 2² + - 2111)
(10)
k+1 = v² - -2
(11)
where T(-) D 0 a 1, 1.8360 O, and X 0.
We now present Theorem 2 ensuring that the inertial KM iterations converge. While the formulation
and assumptions differ from 229, the theorem and its proof heavily rely it.
Theorem 2 (Convergence of the inertial KM iterations), Let 0 < a <
O. >>
0 and x E R.
Assume T(.) a D(-) - Ex)) is continuous. Let (z* y* satisfy 10 and 11 Assume 2" is zero T (L..,, Tz* 0) and the following holds:
(Tu nh 2²) 2 B for h k= 0,
(12)
If
A(1 - a 2a²) < (1 a)²
(13)
then the followings are true:
(A) Σes1 k+1 2 xk 112 Σx>1 || - z/- |rand E | converge.
(B) There is constant M > O such that ming ,| < M =
(C) limg-400 1** -==1l| exists.
5

---

(D) (yy) and (z*) have the same limit points.
Here, similar to Theorem 1 the convergence of (z*) is guaranteed for learning rate (i.e, 2A33) smaller than some upper bound. However, unlike Theorem [T] this upper bound determined not only by cocoercivity (i.e., B), but also by the inertial parameter (Ze. a).
3.4 Validation of the assumption
In Theorems and we utilized the assumption that T()) = - E E E(x) is ß-cocoercive for the (y/ x and z². In this section, we experimentally investigate whether this assumption holds for the actual (z*) and 00 obtained the experiments of Section i.e.
(ED2>0 - ED2 200
2 3 > O.
(14)
1 ED
Equations 9 and 12 are equivalent Eq. if x²
00 and x y². By (C) and (D) of
Theorem it Eq. 14 holds while z + z², then the convergence was due to Eq. 14).
We conducted decoder inversion experiments in various recent LDMs such Stable Diffusion 2.1 [37]. Va [47]. and InstaFlow (24). With or without momentum, the learning rate was fixed
0.001. For inertial KM iterations, was set 0.9. Figure illustrates the ming of cocoercivity (14), convergence of the algorithm (]|= 2 (|2), and final NMSE (| [11//||=*10 for 100 instances in each scenario. The observations and the resulting insights are follows, and they support the validity of our assumptions and theorems.
Our assumption on cocoercivity is reasonable: most of instances showed (ED="-EDD)
0 for all k.
1ED===-E=="1
2. The better the convergence, the cocoercive it is: the fitted functions (red lines) have negative slopes.
3. Converging instances closely approximate the ground truth: the points at the bottom are darker.
-10
10.0
NASE (dB)
NMSE (dB)
|| 2=1
NMSE (dB)
7.5
100
5.0
-20
2.5
-18
0.0
0.5
0.00
0.25
0.50
0.5
min cocoercivity (z" 2")
min coccercivity (2* 2*)
min cocoercivity (z" 25
(a) SD 2.1
37
vanilla FSM
(b) LaVie
47
vanilla FSM
(c)
InstaFlow
24 vanilla FSM
NMSE (dB) d
-11
NMSE (dB)
50
NMSE (dB)
20
40
100
00
15
1
20
-0.5
0.0
0.5
0.0
0.5
0.0
0.5
min cocoercivity (2' vs 2*)
min cocoercivity (2) vs 2*)
min coccercivity (2* vs 2*)
(d) SD 2.1 37
inertial KM iter.
LaVie
47
inertial KM iter.
(f) InstaFlow
24 inertial KM iter.
Figure 2: The six subfigures represent the relationship between cocoercivity and convergence for models algorithms (vanilla forward step method and inertial KM iteration). The x-axis
represents the values of minlkejo,100 (ED= (ED="~ED=* = x= =z*), which informs whether the optimization [/ -
path satisfies the assumptions of Theorems and 2 while the y-axis represents the convergence (i.e.,
|00 200 2)) The red line shows the linear function fitted by least squares. We set x = 2300

---

Experiments with practical optimization techniques
Adam optimizer. Section we proved that our proposed forward step method (Theorem and inertial KMM iterations (Theorem 2 converge. Moreover, experimentally showed that the assumptions of the theorems hold for most instances and demonstrated their convergence as predicted (Fig. 2 However, it generally believed that using the Adam optimizer [16] is preferable in many optimization problems. In this section, although we cannot prove convergence as in Theorems [ 2 we empirically demonstrate that using the Adam optimizer achieve good runtime and memory usage compared to conventional gradient-based methods.
(Optional) Learning rate scheduling. Following common beliefs and a prior work which solves the same problem [14), one can also use cosine learning scheduler with warm-up steps. Similar to [14), the first Vio of the steps are warm-up steps, followed by the application of cosine annealing. After V/o of the total steps have passed (as the learning rate has sufficiently decreased), it is kept constant for the rest of steps.
Noise recon. NMSE (dB)
Grad-based
Grad based 32-bit
Grad free 32-bit
Grad-free 16-bit
-15
Pareto inefficient
Grad-based
32-bit
8.83
-0
32-bit
3.81
Grad-free (ours)
We seek
16-bit
3.15
25
bottom & left
10
20
30
6
8
10
Runtime for decoder inversion (sec)
Peak memory usage (GB)
(a) Stable Diffusion 2.1
37
Noise recon. NMSE (dB)
Grad-based 32-bit
Grad-free 32-bit
Grad-free 16-bit
-17
4
Pareto inefficient
+
Grad-based
32-bit
64.7
T
T
T
20
32
11.7
Grad-free (ours)
seak
16
7.13
22
bottom & left
40
80
120
0
10
20
30
40
50
60
70
Runtime for decoder inversion (sec)
Peak memory usage (GB)
(b) La video diffusion model
Noise recon. NMSE (dB)
Grad-based 32-bit
Grad-free
Grad-free
32-bit
Pareto
-16
inefficient
Brad-based
32-bit
8.38
-18
Grad-free (ours)
32-bit
3.29
16-bit
2.62
We seek
-2
bottom & loft
10
20
30
40
0
6
8
10
Runtime for decoder inversion (sec)
Peak memory usage (GB)
(c) InstaFlow, one-step image generating model
24
Figure 3: Our gradient-free decoder inversion has way shorter runtime than the gradient-based decoder inversion, and drastically reduces the GPU memory usage, on (a) SD2.1 (37], (b) La [7], and (c) InstaFlow (24]. Note that 16-bit gradient-based approach unimplementable, due to the underflow problem. Each point represents a different hyperparameter setting (e.g.. the total number of iterations, learning rate, learning rate scheduling): by collecting experimental results from such diverse settings, the Pareto frontier can be obtained fairly without manipulation.
7

---

Results. Figure 3 shows the average NMSE and peak memory usage when performing decoder inversion using practical techniques (i.e., Adam and learning rate scheduling) in three different LDMs. While the gradient-based method required much runtime and GPU memory to achieve certain accuracy, our approach achieves good accuracy much less runtime and memory usage. One advantage of our proposed method is that it enables all operations to be performed in 16-bit through gradient-free methods, which would typically be infeasible with gradient- -based approaches [30]. As result, for video LDMs where large vectors need to be estimated (Fig. 3b), memory usages can be significantly reduced by almost times.
5
Applications
5.1
Tree-rings watermarking for image generation
In this section, we show interesting application of our gradient-free decoder inversion. Wen et al. 48 proposed tree-rings watermarking, which is an invisible and robust method for protecting the copyright of diffusion-genee generated images. Watermark is embedded into the Fourier transform of zy, and detected by inversion (i.e., estimating 2T from x). Hong et al. [[4] went beyond watermark detection to attempt watermark classification, which makes the problem more difficult; the inversion should be even more accurate. the following scenarios with two image generation LDMs, we experimentally show that our decoder inversion efficiently perform watermark classification.
Watermark classification in SD 2.1. Since watermark classification requires higher accuracy than detection, iterative diffusion inversion algorithms was used such as the backward Euler instead of the naïve DDIM inversion. However, it is reported that iterative algorithms become unstable when the classifier-free guidance is large [4]. So even for watermark classification, the naïve DDIM inversion should be used for now. In this case, one promising option for improving the performance of watermark classification is decoder inversion.
Watermark classification in InstaFlow. One-step models such as InstaFlow 24 generate 2g directly from æy. Therefore, rather than the naïve DDIM inversion, the backward Euler method[[4] should be used to obtain 2r from ø0. In this case, zo is very sensitive to 2T, so æT should be estimated accurately by decoder inversion.
Results. Table shows the accuracy, peak memory usage and runtime of classifying three tree-rings watermarks using different decoder inversion algorithms. Our decoder inversion method achieves watermark classification performance comparable gradient-based method while significantly reducing runtime and peak memory usage. See the appendix for the details.
Table 2: Our gradient-fres decoder inversion method achieves the tree-rings watermark [48] classi- fication performance comparable to gradient-basee method while significantly reducing memory usage and runtime, in different scenarios.
LDM
Encoder
Gradient-
[4]
Gradient-free (ours)
Accuracy
186/300
207/300
202/300
SD2.1
37
Peak memory (GB)
5.71
11.4
6.35
Runtime (s)
5.66
38.0
22.9
Accuracy
149/300
227/300
227/300
InstaFlow
Peak memory (GB)
2.93
8.84
3.15
Runtime (s)
3.55
35.9
13.6
5.2 Background-preservvi image editing
editing, to manipulate an image based on a new condition while preserving background from the original image, requires the entire generating latent trajectory. When it is unknown, it must be recovered via inversion (32. We empirically demonstrate that our algorithm enhances the Bekkground-preserviii image editing, without the need for the original latents. Figure

---

shows the qualitative results of applying our algorithm to the experiment. To compare accuracy similar execution times, we adjusted the number iterations to match the execution time. At comparable execution times, our grad-free method better preserves the background and achieves lower NMSE.
Method
iter
Edited
Error map (x5)
Edited
Error map (x5)
Runtime
HUD ARTANUU REEAPAUCE
Oracle [32]
HUST 03333 CES1 BAGE
HUSEE 6613 NUIE E,002 SUSE 6,00
Grad- based
50
15.7s
Grad-free
HUSE CO0T AACE EOCO sust EX03
100
32bis
17.7s
(Ours)
Grad-free
200
19.3s
(Ours)
Grad-
30
based
9.31s
Grad-free
50
32his
8.538
(Ouns)
Grad-free
100
PIGS DERPORGE RECJARIS RAKES
16bit
9.56s
(Ours)
Figure 4: Our grad-free methods enables the background preserving image editing, where the original trajectory (z,) is unknown. The first row (Oracle) shows the result with using the original trajectory. The latter cases estimate the trajectory with inversion methods. Our grad-free method demonstrates better performance compared the grad-based method at similar runtime.
Discussion
6.1 Does the 3-cocoercivity hold also when using Adam? Yes.
-11
40
12
NMSE (dB)
NMSE (WB)
[1:300 2-1
IMSS (dB)
10
5-
5-
-29
-21
27
0.0
0.5
0.0
0.5
min coccercivity (zf vs z*)
min caccercivity (2 vs 2*)
min cacoercivity (2f vs z*)
(a) Stable Diffusion 2.1
371
(b) LaVie 4v
(c) InstaFlow 2Z!
Figure 5: The assumptions also hold for Adam in many cases, as similar in Fig. 2 The x-axis
represents the values of min (&D=~ -ED=" = x0=_-= -s") which informs whether the optimization path kE|0,50] ||ED= (1022-ED=21 ED2
satisfies the assumptions of Theorems [ and 2 while the y-axis represents the convergence (i.e., || - x 12). The red line shows the linear function fitted by least squares. Note that x0 was approximated by 300
Although it is challenging to provide proof regarding convergence for algorithms using Adam and learning rate scheduling (those used Section 4 and Fig, 3c). we can rather test (z*) satisfy cocoercivity (9. as in Fig. deeiics the plot of Fig under the conditions of the experiments

---

in Section 4 specifically with 16-bit precision and 50 iterations (as Adam converges faster than others). the cases Figs.(S3b.ar and SS3c cocoercivity is well satisfied, and even in Fig. [3a many instances meet the cocoercivity requirement. Additionally, as in Fig. 2 the fitted functions have a negative slope. These findings show that our analysis can help explain some aspects of Adam's convergence.
6.2 Why this method has not been proposed in GAN inversion studies
Some may wonder why this gradient-free approach (replacing gradient descent of Eq. (3) with forward step method of Eq. (4) using the encoder) has not been proposed in numerous GAN inversion studies. Here, we provide Table 3 to explain the reasons. The first reason is that GAN inversion cannot use the encoder from off-the-shelf [9]. It is because networks commonly used in GAN inversion, such as StyleGAN [ were not born with an encoder even though there are some GAN methods that are inherently an autoencoder such as VQGAN [9] that was born with an encoder. Thus many GAN inversion works needed to train the encoder [33 8 34 53 6 36 7 2] while we can use the encoder without any finetuning. The second reason is that the memory requirement was small in GAN inversion, as the size of the generated images was small. Even a standard GPU on a PC was capable of running gradient-basee algorithms, there gradient- baseed methods. On the contrary, due to the increasing size of images/videe videos generated by recent LDMs, performing decoder inversion with gradient- gradient-based algorithms requires huge memory. Another reason is that many networks for GAN inversion operate in full precision (32 bits), making it easier employ gradient-based methods here. Meanwhile, many recent LDMs often infer in half-precision (16 bits) to enhance efficiency, making gradient-base based methods challenging 33)
not much need for lighter algorithms than
Table 3: Why has gradient-free method not been proposed for GAN inversion?
GAN inversion
Decoder inversion in LDMs
Encoder usage
Limited Small Full (32-bit)
Very easy Large (e.g., video generation) Half (16-bit)
Memory requirement
Inference precision
7 Conclusion
In this work, proposed gradient-free decoder inversion methods for LDMs. The vanilla forward step method and its extension with momentum were shown to guarantee convergence, and the assumptions and theorems were validated for various LDMs. We experimentally showed that practical algorithm significantly reduced runtime and memory usage compared to existing gradient- based methods.
Limitations Existing gradient-based method is more accurate than our method sufficient runtime and GPU memory available. In applications such image editing, where the accuracy of latent reconstruction may not be critically important such as background preservation, decoder inversion is often unnecessary in most settings (i.e., using the encoder just OK).
Broader impacts This study could positively impact the protection of copyright for LDM's outputs. One day, even if new neural network architecture is invented and diffusion is longer used, our method can still be efficiently utilized to improve the accuracy of decoder inversion as long as transformations between latent and pixel-space are required.
10

---

Acknowledgements
This work was supported by Institute of Information communications Technology Planning & Evaluation (ITTP) grant funded by the Korea gOV)) [NO.202I-0-011 Artificial Intelligence Graduate School Program (Seoul National University) and National Research Foundation of Korea (NRF) grants funded by the Korea government (MSIT) (NRF-202218 A4A A4A1030579, NRF. 2022M3C1 A A309202211) I). The authors acknowledged the financial support from the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University.
References
[1] Rameen Ahdal, Yipeng Qin, and Peter Wonka. Image2StyleGAN: How to embed images into the stylegan latent space? In ICCV, 2019.
[2] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. HyperStyle: Stylegan inversion with hypemetworks for real image editing. In CVPR, 2022.
[3] Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53-58, 1989.
[4] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch. Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint axXiv:2111.15177 2023.
[5] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhor, Scung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[6] Lucy Chai, Jonas Wulff, and Phillip Isola. Using latent space regression analyze and leverage composi- tionality in gans. ICLR, 2021.
[7] Lucy Chai, Jun-Yan Zhu, Eli Shechtman, Phillip Isola, and Richard Zhang. Ensembling with deep generative views. In CVPR, 2021.
[8] Antonia Creswell and Anil Anthony Bharath. Inverting the generator of generative adversarial network.
IEEE transactions on neural networks and learning systems, 1967-1949 2018
[9] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[10] Daniel Garibi, Or Patashnik, Andrey Voynov, Hadar Averbuch-Ene Ellor and Daniel Cohen-Or. ReNoise: Real image inversion through iterative noising. arXiv preprint arXiv:2403.1400 2024.
[11] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[12] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. NeurlPS, 2020.
[13] Seongmin Hong and Se Young Chun. Neural diffeomorphic non-uniform b-spline flows. In AAAI, 2023.
[14] Scongmin Hong, Kyconghyun Lee, Suh Yoon Jeon, Hyewon Bac, and Se Young Chun. On exact inversion of dpm-solvers. arXiv preprint (ato:2331..88889 2023.
[15] Tero Karras, Samuli Laine, and Timo Aila, A style-hased generator architecture for generative adversarial networks. In CVPR, 2019.
[16] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015.
[17] Jonas Köhler, Andreas Krämer. and Frank Noé. Smooth normalizing flows. NeurlPS, 2021. 2021.
[18] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. In ICLR, 2021.
[19] Mark Aleksandrovich Krasnosel' skii. Two remarks on the method of successive approximations. Uspekhi matematicheskikh nauk, 10()):133-127, 1955.
[20] Dong C Liu and Jorge Nocedal. On the limited memory bigs method for large scale optimization. Mathematical programming, 45(1):503-52) 1989.

---

[21] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic. Wenwu Wang. and Mark D Plumbley. AudioLDM: Texi-lo-audio generation with latent diffuusion models. In ICML, 2023.
[22] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang. Yuxuan Wang. and Mark Plumbley. AMdioL.MM 2: Learning holistic audio generation with self- supervised pretraining. arXiv preprint WxXiv:2308005735 2023.
[23] Xingchao Liu, Chengyue Gong, and Qiang Liu, Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209. 030000, 2022.
[24] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et InstaFlow: One step is enough for high-quality diffusion-based text-to-image generation. In ICLR, 2023.
[25] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: fast ODE solver for diffusion probabilistic model sampling in around 10 steps. NewrIPS, 2022.
[26] Cheng Lu, Yuhao Zhou, Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 9PM-Solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv:2211.0109. 2022.
[27] Simian Luo, Yiqin Tan, Longho Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023.
[28] W Robert Mann. Mcan value methods in iteration. Proceedings of the American Mathematical Sociery, 11500-510, 1953.
[29] Juan José Maulén, Ignacio Fierro, and Juan Peypouquet. Inertial Krasnoselskii- Mann Iterations. Set- Valued and Variational Analysis. 32(2):10, 2024.
[30] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXivv1710.037400 2017.
[31] Zhihong Pan, Riccardo Gherardi, Xiufeng Xie, and Stephen Huang. Effective real image editing with accelerated iterative diffusion inversion. In JCCV, 2023.
[32] Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elow and Daniel Cohen-Or. Localizing object- level shape variations with text-to-image diffusion models. In ICCV, 2023.
[33] Guim Perarnau, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez. Invertible conditional GANs for image editing, arXiv preprint 1.X.vv6611.06355 2016.
[34] Stanislav Pidhorskyi, Donald Adjeroh, and Gianfranco Doretto, Adversarial latent autoencoders. In CVPR, 2020.
[35] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv:2204.06/25, 2022.
[36] Elad Richardson. Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Or. Encoding in style: a stylegan encoder for lange-to-image translation. In CVPR, 2021.
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusson models. In CVPR, 2022.
[38] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- Booth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
[39] Ernest K Ryu and Wotao Yin. Large-scale convex optimization: algorithms & analyses via monotone operators. Cambridge University Press, 2022.
[40] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint 231. 2023.
[41] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[42] Yang Song and Stefano Ermon. Improved techniques for training bcore-based generative models. NewrIPS, 2020.
[43] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.00 006.9.09..0....... 2023,
12

---

[44] Bram Wallace, Akash Gokul, and Nikhil Naik. EDICT: Exact Diffusion Inversion via Coupled Transfor- mations. In CVPR, pages 22532-22541, 2023.
[45] Jiuniu Wang. Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprini arXiv:2308.06:: 06571, 2023.
[46] Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Vide- oLCM: Video latent consistency model. arXiv preprint arXiv:2312.09109, 2023.
[47] Yaohui Wang, Xinyuan Chen, Xin Ma, Shangehen Zhou, Ziqi Huang. Yi Wang. Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang. al. LaVie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 2023.
[48] Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein, Tree-Ring Watermarks: Fingerprints diffusion images that are invisible and robust. arXiv:2305. 2023.
[49] Weihao Xia. Yulun Zhang. Yujiu Yang. Jin Xue, Bolei Zhou, and Ming-HSuan Yang. GAN Inversion: survey, IEEE TPAMI, 45(3)3121-3133, 2022.
[50] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffuusion GANs. ICLR, 2021.
[51] Guoqiang Zhang, Jonathan P Lewis, and W Bastiaan Kleijn. Exact diffusion inversion via bi-directional integration approximation. arXiv:2307. 10829, 2023.
[52] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint 1rX1v:2211.11018, 2022.
[53] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain GAN inversion for real image editing. In ECCV, 2020.
[54] Jun- Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei Efros, Generative visual manipulation on natural image manifold. In ECCV, 2016.
13

---

A
Appendix
A.1 Proofs
Throughout the paper and appendix, both the inner product and the norm are l2.
A.1.1 Proof of Theorem I
Proof of Theorem
=0112 1 1z x² -0=TT=Z11
= th =011 2p(T=* 2" 2/ - 22 +
(S1)
< 1zt - ==112 p2( - T)][
Sum both sides from k = O,...,
p(23 -P p)
T 112 S 2| - <00 1
T2 +0
(S2)
k=0
Finally, if k + 200 then Tz 00
by continuity.
A.1.2 Lemma 1 for Theorem
For simplicity, we set
v = (A-+ -)),
(S3)
8h u(1 - -a)ll 2 x² 1|12
(S4)
A*(²") =12* =1 || -2133 A²(=") =0
(S5)
(*()") = 1|2 ==11 - 8 C²(²*) =1|2² - =211
-
(S6)
Before we prove, we need the following lemma.
Lemma S1. Let 8 >
p 23, and RN, Assume T(.) Eo D(.) E(x) is continuous.
Assume T(.) = E o D(.) E(x) is continuous. Let (z* y²) satisfy 10) and 11
Assume z" is
zero T (i.e, Tz = 0) and Eq. (12) holds. Then, for k = 1,2,...
as <a) (all)) +ol 1-1 A k+1 + 5 + k+1 2*
(S7)
Proof of Lemma
57
Let = 2A3. From Eq.
h- x= 21 1r x= 2=11 p² (T p/
2p - ,²
(S8)
Applying Eq.
12
Eq. S8 we have
|| ++1 2012 S $11 - -1112 p(2B 211
(S9)
From Eq. 10
1 - 1= = (1 + -xxx==²"))))
=2" - a(22~1 - =21
(S10)
=(1+ a: ||| 2 21 a(1 a)ll=* 2 1112 al|zk~ -2112
Combining Eq. 59 and Eq. (100.
|| ++1 - =11 - (1+a) 201 + a (1+++a)ll= -
~ =
- -2112-p(2------- p) Ty
a
-
(S11)
Using Eq. SS4. we can simplify Eq. S11 as follows:
(x)) az" a(1 + a)llz* - - - (S12)
14

---

On the other hand, by Eq.
10 and Eq.
we have
p²11Ty*11 | 1²
||z z² - a(z* )|
= |1(1 - (=+ = + al a(z k-1
2x + x 111
(1 aa]l2 kt1 - =2 a(1 a)ll2* 112 + aal2ztt1 k+1 2x + + 111
(S13)
Let's multiply (2 p\/p to Eq. S13)) Then
p(2 - p) = 8h+1 va(1 a)]l=* 111 pall=tt1 22 + x
(S14)
Finally, subtracting Eq. (S14) from Eq. (S12). we obtain Eq. (S7
A.1.3 Proof of Theorem
Proof of Theorem 2 We firstly show that (c*(z*)), is nonincreasing and nonnegative, thus
lim C* (z") exists. By Eq.
13 there exists > 0 such that
R=+00
a(1+ ra(1 - a) S sv(1- -a)
(S15)
Combining Eq. S7 and Eq.
[115). have
(z") + gh+1 + va|l= k+1 2z* + z² 111
s (=)) + [a a) + pa(1 a)]]l** - z 11
(S16)
s a (z") + [v(1- - - =]]l=* 112
a (2*) + 8t -ell=* 112
From Eq. (S6) and Eq. (S5).
cktt((")) C² (z*) = A ++1 (2)) all|=* 11 - || -2-1)) + 8 8*
(S17)
++| (2*) + gl6t1 aA (2*) - 8t.
By Eq. S16 and Eq. (S17), we have
ck+1 (z*) + vall= 22 + 1 + +ell
2112112 < (z)),
(S18)
which shows (C& (c²)2) (z*)) is nonincreasing. To prove it is nonnegative, suppose Ch (2*) 0 for some kn 2 1. Since it is nonincreasing, from Eq. (S6).
| 1= a||= ==12 s C* (²"2))))))))))))"()) S C (x*) < 0
(S19)
for all & 2 k1 That is, || 11 2| < | - 211 1|2 + Chi (z*), hence
< 1 x= 12 < 1 1 + cki (x*) S ... - || x 1|2 + (k - kx) (S20)
for all k 2 kj, and this is a contradiction as the right hand of Eq. (S0)) diverges to 00 as k + co. Therefore, (C*()*)), k
is nonincreasing and nonnegative, thus lim C* (=*) exists.
100
Now we begin showing (A)
By the findings so far and Eq. S18 and since e > 0, E1>1 ||z
2 +2*+ 112 and
k-1 2 converge. Since Σ k>1
1 converges, Σ Σイ>1 8hh
Σ<>1 r(1 - a)]l=*
1|2 also converges. Using Eq. S13
and an identity I|a - b112 < 2|
2 we have
p= | y $11 = (1 + ++1 2012 + a(1 + a)]l=* 1
(S21)
Summing Eq.
S18 over k: > 1, have
Σ
< C²(t²) =||z² - =21112
(S22)
k2
15

---

Using the summation of Eq. S21 and Eq. (S22 we have
(1 + a)
n min 15k5n
TTy 12
ITy
1| -=-1
1=?
(S23)
cp²
which shows that Σx21 1 converges, and proves (B)
Now prove (C) From Eq. (S16), have
h+ S (2" + 8t.
(S24)
Let [++ denotes the positive part. From Eq. S24
(1 (L-a)[ARt(z>))) )|+ + alA (z ||+ S aaAA +
(S25)
Applying summation for k 2 1, we have
(1 (1-a) a) ( (2*)]+ S a +
Σ
8
8 <0.
(S26)
k
121
121
Let h* || 2 11
プド= A (> I|+ Then h++1 - n/ A
kk+1(2 (2*) - [1 }} S 0 so
(h*) is nonincreasing. Since Σx>1 / k+1 (z*) I+ is finite, lim |/ x" = lim h* exists.
Finally, we now prove (D) By A))
limg-+o | || = lim -z*--*+11 0. By Eq. 10 and
(y*) and (z*) have the same limit points.
A.2 Experiment details and more results
A.2.1 Decoder inversion in LDMs
Hong et al. [14] successfully employed the gradient-based method in their experiments and completed hyperparameter tuning. Their tailored method may work well in their setting, so we used the code from Hong et al. [14's i official repositoryPjo to perform decoder inversion on 100 images generated under the same settings (prompt, classifier- guidance, random seed). This makes our comparison fair. Table [S] shows the settings, NMSE, number of iterations, runtime, and memory usage of the experiments. Although all the values in Table are actually displayed in Fig. 3 Table ST provides additional information, including which specific settings correspond to the points in Fig.
Gradient-based method For the gradient-based based method (for comparision), we used Adam [16] with 12-loss. When optionally using a cosine learning rate scheduler, 1/10 of the total steps were warm-up steps (i.e., the learning rate increased linearly). Note that this setting is the same to [14] In [4), the learning rate was 0.1 with 100 iterations, but it showed long runtimes. Thus, we set the iterations to 20, 30, 50, and 100. We could observe that reducing the number of iterations decreases the effectiveness of learning rate scheduling. Therefore, we additionally conducted experiments with fixed learning rate of 0.01, and found that it performed better with a smaller number of iterations. As shown in Tables STa and S1c when comparing the '0.1 scheduled' and *0.01 fixed' rows under "Grad-based 32-bit' the fixed learning rate outperforms at 20 and 30 iterations, whereas the scheduled learning rate outperforms at 50 and 100 iterations. By comparing existing methods under various hyperparameter settings, we can more reliably establish the Pareto frontier of these methods, ensuring the fairness of our comparisons.
Other details For Stable Diffusion 2.1 and InstaFlow, we used prompts from https: huggingface co/ too/datasets///e mataasetts/ussttvstt..o Satoosttt///aalleeee Stt//taaaaleeeeeeeeeeeeeeee wite--Difffio----ee Prompts For La Vie, prompts were generated by ChatGPT, given the examples used in [47]. The classifier free guidance was 3.0 in Stable Diffusion 2.1, 7.5 in La and 1.0 in InstaFlow. For Vie, we found that 200 iterations for the gradient-free method is too long, so we adjusted the learning rate scheduling as if there were only 100 iterations (fixing the learning rate after 100 iterations).
16

---

Table S1:
We can further improve the reconstruction quality, by using cosine learning rate scheduler with warm-up up steps (if the number of iterations or the runtime is predetermined). We conducted experiments of the decoder inversion various I.MMs, with all other conditions being the same to that in Fig. [3] but using relatively scheduled learning rate at each iteration. + represents the 95% confidence interval.
(a) Stable Diffusion 2.1
Method
Bits
LR
NMSE (dB) # iter. / Runtime
Memory
0.1
-13.92 + 0.44
-17.84 0.55
-20.69 + 0.73
-24.82 = 1.08
Grad-based
32
8.83 GB
scheduled
2./6.11
30/ 0.31 s
50 15.7
100 / 31.7
0.01
-18.61 + 0.49
-19.51 0.55
-20.93 + 0.67
-23.00 0.85
Grad-based
32
8.83 GB
fixed
20/
30/9.37
50 16.0
/ s
Grad-free (ours)
0.01
-19.39 + 0.54
-20.84 0.66
-21.71 + 0.77
-21.85 0.82
32
3,81 GB
scheduled
3.34
50/ 8.53
100 / 17.7
200 34.5
Grad-free (ours)
0.01
-18.74 +: 0.48
-20.82 0.66
-21.72 4: 0.77
-21.86 + 0.82
16
3.15 GB
scheduled
20 s
50/4.90
100 9.56
200 19.3
"This setting is same as setting
LaVic
Method
Bits
LR
NMSE (dB) # iter. / Runtime
Memory
-17.35 +: 0.45
-18.12 0.49
-19.58 + 0.59
-22.20 1: 0.81
Grad-based 32
64.7 GB
scheduled
24.24
30/ 37.07 s
50/ 61.37
100 122.31
0.01
-19.06 +:
-19.68 0.48
-20.66 0.57
-22.14 0.71
Grad-based
32
64.7 GB
fixed
2
30.17.11
61.67
100 / 122.46
Grad-free (ours)
0.01
-20.21 = 0.43
-21.42 0.54
-21.58 0.62
-21.21 0.58
32
11.7 GB
scheduled
20 12.96
50 / 32.17 s
100 / 64.25
200 / 128.34
Grad-free (ours)
0.01
-19.86 1: 0.42
-21.37 0.55
-21.52 :: 0.63
-21.14 1: 0.58
16
7.13 GB
scheduled
20/ 8.59
/ 21.19 s
100 / 42.28
200 / 83.45 s
(c) InstaFlow
Method
bits
LR
NMSE (dB) // iter. Runtime
Memory
-15.67 0.48
-16.22 1: 0.53
-17.26 1: 0.62
-19.15 0.78
Grad-based 32
8.38 GB
scheduled
2.29
30 9.55
50 16.12
100 32.54
0.01
-15.94 0.48
-16.40 +
-17.14 0.59
-18.33 0.70
Grad-based
32
8.38 GB
fixed
20 6.30
30/ 30/9.51
50 / 15.92
100 32
Grad-free (ours)
0.01
-16.55 + 0.53
-17.46 + 0.63
-17.76 + 0.68
-17.81 + 0.70
32
3.2 GB
scheduled
20 3.17 s
50 / 8.34
100 / 16.90
200 34.21
Grad-free (ours)
0.01
-16.38 0.51
-17.46 +: 0.63
-17.76 1: 0.68
-17.82 4 0.71
16
2.62 GB
scheduled
20 1.89
50
9.999 s
200 20.04
A.2.2 Watermark detection
For both scenarios, the number steps was 100 and the learning rate was scheduled for both gradient- based and gradient-free methods. The maximum learning rate was 0.1 for grad-based methods and
17

---

0.01 for Grad-fire free methods, which is consistent with the experiments in Section 4. All experiments were conducted 32-bit.
For Stable Diffusion 2.1, through decoder inversion, we obtain 2n from æ, and perform naïve DDIM inversion for 50 steps obtain 2r from Zp- Except for employing the naïve DDIM inversion due to the use of large classifier-free guidance of 7.5 and resulting stability issues, the experimental settings are the same as those of [14]. For InstaFlow, the backward Euler (Algorithm of [44 with 100 iterations is applied for seeking 2T from Zo.
Figure [TT shows the confusion matrices for the experiments of Table in the main paper. In Figure S2 we also present the qualitative results of applying our algorithm to the watermark classification experiment. Our grad-free free method enables precise reconstruction, while reducing the runtime compared to the grad-based method.
000
Actual
1
54
14
32
Actual
1
47
24
29
Actual
52
17
31
2
27
47
2
17
66
2
18
25
3
3
1
2
3
1
2
3
1
2
3
Predicted
Predicted
Predicted
Encoder
Grad-based
Grad-free (ours)
(a) Stable Diffusion 2.1. Accuracy: 186/300, 207/300, 202/300
000
Actual
36
Actual
31
Actual
30
2
36
2
1
96
3
0
16
84
3
0
0
100
0
100
2
2
1
3
Predicted
Predicted
Predicted
Encoder
Grad-based
Grad-free (ours)
(b) InstaFlow. Accuracy: 149/300, 227/300, 227/300
Figure S1: Confusion matrices for watermark classification LDMs. Ours is better than the encoder
and works as well as the grad-based method [14]
Embedded Generated watermark
Encoder (Recon. Error)
Grad (Recon. Error) 200, 64.4s
Grad(short) (Recon. Error)
Grad-free 16bit (ours)
(Recon. Error)
iter, Runtime
100, 32.0s
200, 35.0s
NIMAE
NMAE 0.17
TOMAE 0.10
NIMAE 0.18
Figure S2: Qualitative result on the watermarking classification experiment 14]
Our grad-free free
decoder inversion aids in the accurate reconstruction of Tree-ring watermarking either reduces the processing time compared to the grad-based (column 3 vs 5) or achieves better accuracy within the same runtime (column 4 vs 5).
48 Specifically,
18

---

Table S2: Ablation studies on optimizer and learning rate scheduling. Adam showed the best result compared to vanilla and KM iterations method. The learning scheduling we used showed consistently good performance across all intervals.
(a) Ablation study on optimizer
NMSE (dB)
Method \\ # iter.
20
30
50
100
Vanilla
-16.87 + 0.38
-17.42 = 0.41
-18.21 = 0.46
-19.35 + 0.54
K.M iterations
-18.99 + 0.53
-20.72 +: 0.73
-21.46
+ 0.96
-20.91 + 1.20
Adam (orig.)
-19.39 + 0.54
-20.84 + 0.66
-21.71
+ 0.77
-21.85 + 0.82
(b) Ablation study on learning rate scheduling
NMSE (dB)
Method
20
50
100
200
1r=0.01 (fixed)
-20.05 + 0.58
-21.07 + 0.70
-21.22 = 0.74
-20.61 + 0.79
1r=00002 (fixed)
-17.85 + 0.43
-19.28 + 0.53
-20.59 = 0.64
-21.57 + 0.74
Ir scheduled (orig.)
-19.39 + 0.54
-20.84 + 0.66
-21.71 = 0.77
-21.85 + 0.82
A.2.3 Ablation Studies
We conducted an ablation study on each optimizer and learning rate scheduling, in SD2. 102.1(325 (32bit). Table S2a shows the result of only changing the optimizer while keeping all other conditions the same. In Table Table[S2b we observed what happens when using fixed learning rate instead of applying learning rate scheduling with Adam. When using a fixed learning rate, we found that with large learning rate (Ir=0.01), the performance was poor when the number of iterations was high, and with small learning rate (Ir=0.002), the performance was poor when the number of iterations was low. In contrast, the scheduled learning rate we used showed consistent performance across all intervals regardless of the number of iterations.
A.2.4 Analysis on the baaed method
In Figure S3 present the instance- -esance--iseeeee cocoercivity, convergence, and accuracy for the gradient- based method, similar to Figure Eigurenaad and Figure(3] Similar to the results observed with the gradient- free method, the gradient-based method showed that most instances satisfied cocoercivity, and better convergence often led to higher accuracy. However, it was observed that cocoercivity and convergence not significantly correlated. In other words, the correlation between cocoercivity and convergence is not a general characteristic, but a unique feature we discovered in our gradient-free method.
A.3
Supplementary explanation
Normalizing flows (NFs) are born with its analytic invertibility, hence located at the bottom left. VAEs and learning- based GAN inversion methods have corresponding encoders, so they have short computation time but come with some inversion error. Some special structured DMs [4] are also located here.
Figure 1b
A.4 Computational resources
For running Stable Diffusion 2.1, one NVIDIA GeForce RTX 3090 Ti was used. The RAM size of GPU was 24 GB. Note that most of the computation was conducted on GPU. For CPU, one 11th Gen Intel(R) Core(TM) 19-1 F900KF @ 3.50GHz was used. For running LaVie, one NVIDIA A100 SXM4 80GB and AMD EPYC 7742 64-Core Processor were used. For InstaFlow, one NVIDIA GeForce RTX 3090 GPU with one 12th Gen Intel(R) Core(TM) 17-12700K @ 3.60GHz was used.
19

---

30
-15
NMSE (dB)
NMSE (dB)
NISSE (dB)
||200
40
10
2.5
-29
31
-29
0.5 min coccercivity 2*)
min coccercivity (2f
min cocoercivity (2t v=z)
Stable Diffusion 2.1
LaVic
(c) InstaFlow
Figure S3: Analysis on the grad-based method. x-axis: min ke(0,50)
[EDx -ED=11 y-axis: conver-
gence (i.e., 50 00 [[2). The red line shows the linear function fitted least squares. Note that 00 was approximated by 3000 Better convergence and accuracy were observed, but no relationship with cocoercivity was found.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper discusses the limitations in Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
This paper provides Theorems 1,2 and their corresponding proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper provides sufficient information that can reproduce the results the main experiment in Section 4.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
This paper provides code to reproduce the experimental results in Sections
and 5.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
httpe/uuiides/oodeSuuiisssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpooliiigggddde/ooooooo http://uddd//oeeSuubntn ssicnyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper provides the details of the experiments sufficiently.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: This paper reports error bars (95% confidence interval) for the main experiment (Figure 3).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Peak memory usages and time of executions are main concerns this paper. This paper also provide information on computer resources in Section A.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, this paper does.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper discusses societal impacts Section 7.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This paper explicitly mentioned and properly respected the assets used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code the supplementary material has its document README.md.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve such crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve such crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "Wjd9FA9xpZ", "reviewer": "Reviewer_2mTK", "summary": "This work provides a method for gradient-free decoding for latent diffusion models that can reduce the amount of required GPU memory and lessen the computation time as opposed to gradient-based methods. The method focusses on providing better invertibility in ldms that is based on a theoretical assumption that guarantees the convergence of the forward step and the inertial KM iterations for the ground truth. They further showcase the proposed method for tree-ring watermark classification problem.", "questions": "Please refer the weaknesses section.", "limitations": "refer the weaknesses section.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. They propose a new method for gradient-free decoding for the ldms that has some advantage over gradient-based methods. \n2. The paper contains a detailed description of the method along with the reasonings behind the assumptions made. \n3. Detailed set of experiments are present along with a thorough convergence analysis.\n4. Well-written in terms of method description and experiments.", "weaknesses": "1. The paper is hard to follow in the methodology description and has a lot of assumptions which are difficult to verify from the paper itself. For example, in lines 152 and 165, some assumptions are being made which are not backed by proofs.\n2. Are the findings enough to support the validity of the said assumptions? A question I find hard to answer based on the described experiments. \n3. The advantages of the given approach seems limited and not quite powerful. Being able to reduce some memory for decoding is not a big enough contribution in itself. \n4. Limited novelty -- similarity with prior works like [14] mentioned in the paper.\n5. Not enough quantitative experiments to establish the utility of the said method -- Comparing for memory consumption and runtime is not enough in my opinion.\n6. Not enough applications provided to prove the merits of the proposed approach -- applying the current method on diverse set of applications like image generation, editing, interpolation, etc and comparing it with gradient-based methods and evaluating the results would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review. We are glad that you found our work new [S1], containing a detailed description of the method along with the reasonings behind the assumptions made [S2], experiments are detalied [S3], convergence analysis thorough [S3], and experiments and methods are well-written [S4]. Here, we carefully responded to your comments on weaknesses and questions.\n\n$  $\n\n### Weaknesses\n\n\n> **[W1]**: *\"The paper is hard to follow in the methodology description and has a lot of assumptions which are difficult to verify from the paper itself. For example, in lines 152 and 165, some assumptions are being made which are not backed by proofs.\"*\n\n\nWhile we have a dedicated subsection for the validation of assumptions in Sec 3.4 and related discussions in Sec 4 (as you indicated in [S2]), we will further refine it if any unverified assumption exists. Regarding your examples, one is trivial [Line 152], and the other is verified empirically [Line 165].\n- [Line 152] *$\\mathcal{T}(\\cdot) = \\mathcal{E} \\circ \\mathcal{D}(\\cdot) - \\mathcal{E}(x)$ is continuous*: \nNeural networks like $\\mathcal{E}, \\mathcal{D}$ are generally treated as continuous functions. Moreover, they are usually *differentiable*, in order to be trained. \n- [Line 165] *$\\mathcal{T}(\\cdot) = \\mathcal{E} \\circ \\mathcal{D}(\\cdot) - \\mathcal{E}(x)$ is $\\beta$-cocoercive for the $(y^k,z^k)$ and $z^\\star$*:\nSince $\\beta$-cocoercivity is hard to *prove* for all feasible $(y,z)$, we assumed  $\\beta$-cocoercivity only for $(y^k,z^k)$ and $z^\\star$. Then, we empirically verified the cocoercivity for actual $(y^k,z^k)$ and $z^\\star$, as in Figure 2, which supports [Line 165].\n\nWe will add these to the revision.\n\n$  $\n\n> **[W2]**: *\"Are the findings enough to support the validity of the said assumptions?\"*\n\nYes, they are. We have found the following things in three different models:\n| Model      | Cocoercivity (L175)   | Cocoercivity vs Convergence  (L177) | Convergence vs Accuracy (L179) |\n|--------------|------------|------------|------------|\n| SD 2.1 (Image LDM)     | ✔ Completed | ✔ Completed | ✔ Completed |\n| Lavie (Video LDM)  | ✔ Completed | ✔ Completed | ✔ Completed |\n| InstaFlow (Image Latent Rectified Flow)       | ✔ Completed | ✔ Completed | ✔ Completed |\n\nwhich can completely verify the following causality: \n\n\n**Most instances $\\overset{\\text{L175}}{\\Longrightarrow}$ Cocoercivity $\\overset{\\text{L177}}{\\Longrightarrow}$ Convergence $\\overset{\\text{L179}}{\\Longrightarrow}$ Accuracy**.\n\n$  $\n\n> **[W3]** *\"The advantages of the given approach seems limited and not quite powerful. Being able to reduce some memory for decoding is not a big enough contribution in itself.\"*\n\nOur advantages are not limited to 'reduce some memory'. We organize the advantages here. Our method is:\n\n\n- **Fast**: Our method needs shorter runtimes for achieving the same NMSEs than the grad-based method. (up to 5X faster, in Fig. 3c and Tab. S1c, *1.89 s* vs 9.51 s to achieve -16.4 dB)\n- **Accurate**: In the same runtime, our method shows smaller NMSEs than the grad-based method. (up to 2.3 dB lower, in Fig. 3b and Tab. S1b, *-21.37 dB* vs -19.06 dB in 25.1 s)\n- **Memory-efficient (significant)**: Our method consumes a smaller amount of GPU memories than the grad-based method. (up to 89% of the memory can be saved, in Fig. 3b, *7.13 GB* vs 64.7 GB)\n- **Precision-flexible**: The grad-based method requires a full-precision model that supports backpropagation. Our method, however, is flexible and can run on any precision model, even though it does not support backpropagation. Our method can be immediately applied to many LDMs distributed in half-precision.\n\n$  $\n\n> **[W4]** *\"Limited novelty -- similarity with prior works like [14] mentioned in the paper.\"*\n\nOur paper is novel, as it firstly proposed Gradient-free decoder inversion of in LDMs.\nThe following table shows that [14] is different from ours.\n\n| Work      | Problem   | Comparison | Model |\n|--------------|------------|------------|------------|\n| Ours | Inversion of *decoders*| Gradient descent (similar to GAN inversion) | SD2.1, LaVie, InstaFlow | \n| [14] | Inversion of *denoising diffusion processes* | naive DDIM inversion | Pixel space DM, SD2.1\n\nOne similar point to [14] is that we employed the forward step method, but that is just a very widely known optimization algorithm [39].\n\n$  $\n\n> **[W5]** *\"Not enough quantitative experiments to establish the utility of the said method -- Comparing for memory consumption and runtime is not enough in my opinion.\"*\n\nWe sufficiently verified four distinct advantages of our method. As mentioned in **[W3]**, we did not only compare *memory consumption* and *runtime*, but also *accuracy* and *float precision*.\n\n$  $\n\n> **[W6]** *\"Not enough applications provided to prove the merits of the proposed approach -- applying the current method on diverse set of applications like image generation, editing, interpolation, etc and comparing it with gradient-based methods and evaluating the results would help.\"*\n\nThanks for a good suggestion. We additionally conducted background-preserving image editing [14, 32]. Figure R1 (see the rebuttal pdf in \"Author Rebuttal by Authors\") shows the qualitative results of applying our algorithm to the experiment in [14], which investigates how exact inversion improves the background-preserving image editing [32]. To compare accuracy at similar execution times, we adjusted the number of iterations to match the execution time. At comparable execution times, our grad-free method better preserves the background and achieves a lower NMSE. We will add this to the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tgQJYEtARn", "reviewer": "Reviewer_DG1n", "summary": "The paper introduces a zero-order (gradient-free) inversion optimization algorithm for encoder-decoder based generative models, particularly focusing on latent diffusion models (LDM). The objective of the optimization problem is to find the latent vector $z$ for a given image $x$ such that $x=D(z)$ where D is the decoder of the LDM.  The proposed inversion algorithm updates the latent vector at the next iteration by relaxing the constraints of the objective function to find the z such that $E(x)=E(D(z))$ (E is th encoder) by iteratively taking the difference between the two push forward maps until it converges to a fix-point. The authors demonstrate their proposed method is motivated technically, they show their inversion algorithm converges to a fixed point under reasonable assumptions (section 3.4). This type of analysis is further expanded to momentum (section 3.4 demonstrates reasonable assumptions). Empirically, the authors show their method significantly decreases computational cost while maintaining similar performance to gradient-base inversion algorithms.", "questions": "1).  Could the authors please provide qualitative results similar to experiments performed in [48]?\n\n2). Could the author please comment on the scope of the inversion algorithm and what applications it is suitable for?\n\n3). Encoder-Decoder models are not bijective, therefore $E(D(Z)$ is an approximation of an invertible map, so there will be some information loss when approximating image x. In certain subfields, this is described as [\"Representation Error\"](https://proceedings.mlr.press/v70/bora17a/bora17a.pdf). Could the authors please comment on how this phenomenon affects their analysis even though their method converges to a fixed point (possibly very close to z*), there will be some information loss due encoder-decoder structure of the model.\n\n4). Could the authors explain the scaling of Figure 2? Please refer to the comment in the weakness sections.", "limitations": "Please refer to the \"weakness\" portion of the review.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "To my knowledge, the proposed method is novel, effective, and straightforward to implement for any encoder-decoder based architectures.\n\nThe proposed method demonstrates relatively similar performance with significantly less computational time in the experiments provided.\n\nThe proposed algorithm is technically motivated, and the assumptions are verified computationally.\n\nThe authors provided code for reproducibility.", "weaknesses": "Certain aspects of the writing need attention. For example, ρ is not defined in Equation 3. Additionally, contribution bullet point 3 ends with \"and,\" while the rest of the contributions end with a comma, suggesting missing information. The contribution section itself reads like a run-on sentence separated by bullet points. \n\nFigure 2: The scaling of the figure is confusing. It appears that NF is the ideal architecture because it has the lowest inversion error and the least computational runtime, which seems to argue against the proposed architecture LDM.\n\nThe limitations of the optimization algorithm are not clearly articulated in comparison to other inversion algorithms. The proposed method requires x (the image or signal) to be given, whereas other inversion algorithms require a set of measurements y, and a forward operator. \n\nThe results lack a visual verification to confirm the effectiveness of the proposed methodology. For instance, Section 5 replicates the experiments in [48], but does not reproduce the qualitative results to demonstrate the relationship between them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review, and for finding [S1] our work novel, effective, and straightforward to implement for any encoder-decoder based architectures, [S2] our method demonstrates has significantly less runtime, [S3] technically motivated, and the assumptions verified computationally, [S4] and we provided code for reproducibility. However, it is puzzling that you have given a *strong reject* despite finding so many advantages. We hope you will reconsider your review after reading our rebuttal that has thoroughly addressed your feedback regarding the weaknesses and questions.\n\n$  $\n\n# Weaknesses\n> **[W1]** *\"Certain aspects of the writing need attention. For example, $\\rho$ is not defined in Equation 3. Additionally, contribution bullet point 3 ends with \"and,\" while the rest of the contributions end with a comma, suggesting missing information. The contribution section itself reads like a run-on sentence separated by bullet points.\"*\n\n- About $\\rho$: Thanks for pointing out. We will add a sentence that $\\rho$ is the step size.\n- About contribution bullet point 3: We meant it (i.e., The contribution section itself reads like a run-on sentence). \n\n$  $\n\n> **[W2]** *\"Figure 2: The scaling of the figure is confusing. It appears that NF is the ideal architecture because it has the lowest inversion error and the least computational runtime, which seems to argue against the proposed architecture LDM.\"*\n\nIt seems you are referring to Figure 1b. Figure 1b shows the accuracy and runtime of inversion, which are merely parts of the numerous specifications of the generative model. As you said, normalizing flows (NFs) are good at inversion in accuracy and runtime. However, we are *not* finding which generative model does inversion best. LDMs efficiently generate high-quality and large-scale samples (Line 47), so we want to do inversion well **in LDMs**. We have nothing to do with NF.\n\nFor the scaling of the Figure 1b, please see **[Q4]**.\n\n$  $\n\n> **[W3]** *\"The limitations of the optimization algorithm are not clearly articulated in comparison to other inversion algorithms. The proposed method requires x (the image or signal) to be given, whereas other inversion algorithms require a set of measurements y, and a forward operator.\"*\n\nThere must be a severe misunderstanding. It seems you are looking for a competitor to our algorithm in the inverse problems in imaging, which can be represented as $y=Ax+n$. We solve a totally different problem, $x=\\mathcal{D}(z)$ (Eq. 5). Before our work, including GAN-inversion works [49], $x=\\mathcal{D}(z)$ could only be solved by gradient descent. Therefore, we compared our method with the gradient descent in Sections 4 and 5.\n\n$  $\n\n> **[W4]** *\"The results lack a visual verification to confirm the effectiveness of the proposed methodology. For instance, Section 5 replicates the experiments in [48], but does not reproduce the qualitative results to demonstrate the relationship between them.\"*\n\nAs we mentioned in Lines 204-210, We did not perform the watermark *detection* experiment from [48], but rather the watermark *classification* experiment from [14]. Nevertheless, we will add qualitative results for the watermark classification experiment [14], as in Fig. R2 (see the rebuttal pdf in \"Author Rebuttal by Authors\").\n\n$  $\n\n# Questions\n\n> **[Q1]** *\"Could the authors please provide qualitative results similar to experiments performed in [48]?\"* \n\nYes, see the response to **[W4]**.\n\n$  $\n\n> **[Q2]** *\"Could the author please comment on the scope of the inversion algorithm and what applications it is suitable for?\"*\n\nSure.\n- Scope of the inversion algorithm: As in **[W3]**, it refers to algorithms which solve $\\underset{z}{\\textrm{find}}\\;x=\\mathcal{D}(z)$.\n- Applications: As in Line 41, *\"for seeking the true latent [14], for watermarking [48], and for background-preserving editing [32]\"*. \n\n$  $\n\n> **[Q3]** *\"Encoder-Decoder models are not bijective, therefore, E(D(z)) is an approximation of an invertible map, so there will be some information loss when approximating image x. In certain subfields, this is described as \"Representation Error\". Could the authors please comment on how this phenomenon affects their analysis even though their method converges to a fixed point (possibly very close to $z^\\star$), there will be some information loss due encoder-decoder structure of the model.\"* \n\nFollowing the definition of the representation error [A], our representation error is 0, because we solve $x=\\mathcal{D}(z)$ with $x \\in \\mathrm{Range}(\\mathcal{D})$. There is no measurement error either, since we do not have a measurement operation $A$ (we are not solving $y=Ax+n$). All of the source of the error is the optimization error: *\"The optimization procedure did not find the best $z$\"* [A].\n\nAs you mentioned, $z^0 := \\mathcal{E}(\\mathcal{D}(z^\\star))$, the initial point of the optimization, is different from $z^\\star$. That is the problem that we solve in the paper: *starting from $z^0$, seeking $z^\\star$*.\n\n[A] Bora, Ashish, et al. \"Compressed sensing using generative models.\" ICML 2017.\n\n$  $\n\n> **[Q4]** *\"Could the authors explain the scaling of Figure 2? Please refer to the comment in the weakness sections.\"*\n\nIf you mean Figure 1b instead of Figure 2, roughly speaking, the first dotted line represents about 1 second, and the second dotted line represents around 5 to 10 seconds. For the inversion of LDM, referring to Table 4 in [14], it ranges from 30 to 160 seconds. Note that these can vary in different settings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "spgqYKdpmS", "reviewer": "Reviewer_9ej4", "summary": "The paper proposes a gradient-free method for decoder inversion in latent diffusion models (LDMs), which significantly reduces computational complexity and memory usage compared to traditional gradient-based methods. The approach is theoretically proven to converge and is efficient in experiments with various LDMs, achieving comparable accuracy. This method is useful for applications like noise-space watermarking, demonstrating its practical utility in scenarios requiring efficient decoder inversion.", "questions": "In Figures 2 and 4, there are some failure cases depicted. Could you provide more details on the success rate of the proposed method and how it compares to the gradient-based method? Additionally, a detailed analysis of these failure cases and potential reasons for the discrepancies would be very helpful.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written, with clear and easy-to-follow explanations.\n* The proposed method replaces gradient operations with forward inference, significantly saving memory and improving computational efficiency. Theoretical analysis ensures the convergence of the method.\n* Extensive experimental results on various latent diffusion models (LDMs) demonstrate the superior efficiency and effectiveness of the proposed approach.\n* The method's practical utility is highlighted through its successful application in noise-space watermarking.", "weaknesses": "* The theoretical convergence of the forward step method and KM iterations is provided. However, the experiments employ the Adam optimizer. It appears that the term $E(D(z^k)) - E(x)$ in Eq. (4) is treated as the gradient in Adam. More details to clarify this could help avoid potential misunderstandings. Moreover, an additional ablation study on the impact of using Adam and cosine learning rate decay may enhance the comprehensiveness of the findings.\n* The paper acknowledges that gradient-based methods can achieve higher accuracy in certain applications. A more in-depth discussion on the accuracy trade-off and scenarios where this method might fall short would provide a balanced perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging review. We are pleased that you found our work well-written, clear, and easy to follow [S1], our method significantly saves memory and improves computational efficiency [S2], our theoretical analysis ensures convergence [S2], and the extensive experiments demonstrate the superiority of our method [S3]. We are also glad that you consider our method practical and successful in application [S4]. Here, we have carefully considered the weaknesses and questions you raised.\n\n$  $\n\n# Weaknesses\n\n> **[W1]** *\"The theoretical convergence of the forward step method and KM iterations is provided. However, the experiments employ the Adam optimizer. It appears that the term in Eq. (4) is treated as the gradient in Adam. More details to clarify this could help avoid potential misunderstandings. Moreover, an additional ablation study on the impact of using Adam and cosine learning rate decay may enhance the comprehensiveness of the findings.\"*\n\n### Gradient in Adam\n\nThank you for a great comment. As you said, $\\mathcal{E}(\\mathcal{D}(z)) - \\mathcal{E}(x)$ is treated as the gradient in Adam. Actually, we are not the first to use Adam for non-gradient-based minimizations as if they were gradient-based. For example, in reinforcement learning, semi-gradient methods that use stop-grad operations are employed instead of gradients (they do not actually use gradients [A]), but Adam is still used. Additionally, zeroth-order optimization does not use gradients but still utilizes Adam [B]. Thanks for the interesting discussion topic; we will add this and clarify in the revision.\n\n[A] Nota, Chris, and Philip S. Thomas. \"Is the policy gradient a gradient?.\" arXiv preprint arXiv:1906.07073 (2019).\n\n[B] Chen, Xiangyi, et al. \"ZO-AdaMM: Zeroth-order adaptive momentum method for black-box optimization.\" NeurIPS 2019.\n\n$  $\n\n### Ablation study on optimizer\nThank you for suggesting good ablation studies. We additionally conducted an ablation study in SD2.1 (32-bit), changing only the optimizer while keeping all other conditions the same. The table below shows the results:\n\n| # iter.  | 20 | 50 | 100 | 200 |\n|---|---|---|---|---|\n|Vanilla| -16.87 $\\pm$ 0.38 | -17.42 $\\pm$ 0.41 | -18.21 $\\pm$ 0.46 | -19.35 $\\pm$ 0.54 |\n|KM iterations| -18.99 $\\pm$ 0.53 | -20.72 $\\pm$ 0.73 | -21.46 $\\pm$ 0.96| -20.91 $\\pm$ 1.20 |\n|Adam (orig.)| **-19.39** $\\pm$ 0.54 | **-20.84** $\\pm$ 0.66 | **-21.71** $\\pm$ 0.77 | **-21.85** $\\pm$ 0.82 |\n\n### Ablation study on learning rate scheduling\n\nThis time, we experimented to see what happens when using a fixed learning rate instead of applying learning rate scheduling with Adam.\n\n| # iter.  | 20 | 50 | 100 | 200 |\n|---|---|---|---|---|\n|lr=0.01 (fixed)| **-20.05** $\\pm$ 0.58 | **-21.07** $\\pm$ 0.70 | -21.22 $\\pm$ 0.74 | -20.61 $\\pm$ 0.79 |\n|lr=0.002 (fixed)| -17.85 $\\pm$ 0.43 | -19.28 $\\pm$ 0.53 | -20.59 $\\pm$ 0.64 | -21.57 $\\pm$ 0.74 |\n|lr scheduled (orig.)| -19.39 $\\pm$ 0.54 | -20.84 $\\pm$ 0.66 | **-21.71** $\\pm$ 0.77 | **-21.85** $\\pm$ 0.82 |\n\nWhen using a fixed learning rate, we found that with a large learning rate (0.01), the performance was poor when the number of iterations was high, and with a small learning rate (0.002), the performance was poor when the number of iterations was low. In contrast, the fixed learning rate showed consistent performance across all intervals regardless of the number of iterations. Again, thank you for good suggestions. We will add these ablation studies to the revision.\n\n\n$  $\n\n\n> **[W2]** *\"The paper acknowledges that gradient-based methods can achieve higher accuracy in certain applications. A more in-depth discussion on the accuracy trade-off and scenarios where this method might fall short would provide a balanced perspective.\"*\n\nAs we mentioned in Line 257, the gradient-based method is more accurate than our method if sufficient runtime is available. Nevertheless, that is mostly not a big deal, because too much accuracy is unnecessary for applications. In experiments in Tab. 2-InstaFlow, the number of iterations was 100 for the gradient-based method. As shown in Tab. S1c, more than 100 iterations are needed to be more accurate than the gradient-free method. However, in Tab. 2-InstaFlow, the accuracy is 227/300, which is the same as the gradient-free. This shows that too much accuracy is unnecessary for the watermark classification. We will add this discussion in the revision. \n\nAdditionally, we provide new qualitative results for various applications in Figs. R1 and R2 by running the gradient-based method for both longer and shorter durations. This would help a more in-depth discussion on the accuracy trade-off. Thank you.\n\n$  $\n\n# Questions\n\n> **[Q1]** *\"In Figures 2 and 4, there are some failure cases depicted. Could you provide more details on the success rate of the proposed method and how it compares to the gradient-based method? Additionally, a detailed analysis of these failure cases and potential reasons for the discrepancies would be very helpful.\"*\n\nGreat idea. In Fig. R3 (in the PDF in the common rebuttal), we display the instance-wise cocoercivity, convergence, and accuracy for the gradient-based method (similar to Figs. 2 and 4). Like the gradient-free method, the gradient-based method showed that most instances satisfied cocoercivity, and better convergence often led to higher accuracy. However, it was observed that cocoercivity and convergence are not significantly correlated. In other words, 'cocoercivity $\\Rightarrow$ convergence' is not a general characteristic, but a unique feature we discovered in our gradient-free method.\n\nAs mentioned already in Line 177, we confirmed that the more cocoercivity is satisfied, the better the convergence. However, when examining the failure cases (i.e., instances that do not satisfy cocoercivity) directly, we could not identify any significant commonalities. Figure R4 in the common rebuttal PDF shows the 8 failure cases of Fig. 4a. We will add this discussion. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AU3TMqxC6K", "reviewer": "Reviewer_YFNw", "summary": "The paper introduces a gradient-free method for LDM decoder inversion. Compared to traditional gradient-based methods, this method is computational and memory efficient which makes it suitable for large-scale tasks like video generation. They provide theoretical support for the method's convergence. Their empirical results show that their gradient-free method reduces computation time and memory usage in applications such as watermark classification using Stable Diffusion 2.1 and InstaFlow models. Using optimization strategies like the Adam optimizer and learning rate scheduling further boosts the efficiency.", "questions": "1- How robust is the assumption 𝐸∘𝐷 ≈ 𝐼 ? Can you provide empirical evidence for this assumption in the context of non-linear autoencoders?\n\n2- Have you tested the proposed inversion method on video LDMs?\n\n3- How does the accuracy of the proposed method compare to gradient-based methods in applications such as image editing?\n\n4- How were these parameters chosen? Have you done an ablation study?", "limitations": "Limitations have been briefly mentioned, yet there is not enough quantitative analysis for cases where the accuracy of gradient-based methods are significantly higher, and whether this affects applications in both image and video domain.", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1- The authors introduce a new gradient-free method for decoder inversion, which is both faster and more memory-efficient that traditional gradient-based methods.\n2- The paper provides a theoretical analysis of the proposed method, showing that it converges under reasonable conditions.\n3- The method is shown to be effective in a practical application where precise decoder inversion is necessary, and their proposed method achieves comparable accuracy to gradient-based methods.", "weaknesses": "1- The comparative experiments mainly focus on memory usage and runtime. There is limited evidence on how the accuracy of the inversion compares to gradient-based methods in various applications. Also, authors do not present images for qualitative comparison.  \n\n2- Experiments validating the convergence assumption are limited to specific hyperparameters for Stable Diffusion, LaVie, and InstaFlow. It is not clear if this behavior generalizes to other settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and we are encouraged that you found that our method is new [S1], faster and more memory-efficient than traditional grad-based methods [S1], theoretical analysis shows that our method converges under reasonable conditions [S2], effective in a practical application [S3]. Here, we provide feedback on your reviews.\n\n$ $\n# Weaknesses\n\n> **[W1]** *\"The comparative experiments mainly focus on memory usage and runtime. There is limited evidence on how the accuracy of the inversion compares to gradient-based methods in various applications. Also, authors do not present images for qualitative comparison.\"*\n\nWe have already verified that, in addition to memory usage and runtime, our method also excels in terms of accuracy and precision versatility. To summarize the advantages of our method and their experimental evidence:\n\n- **Fast**: Our method needs shorter runtimes for achieving the same NMSEs than the grad-based method. (up to 5X faster, in Fig. 3c and Tab. S1c, *1.89 s* vs 9.51 s to achieve -16.4 dB)\n- **Accurate**: In the same runtime, our method shows smaller NMSEs than the grad-based method. (up to 2.3 dB lower, in Fig. 3b and Tab. S1b, *-21.37 dB* vs -19.06 dB in 25.1 s)\n- **Memory-efficient (significant)**: Our method consumes a smaller amount of GPU memories than the grad-based method. (up to 89% of the memory can be saved, in Fig. 3b, *7.13 GB* vs 64.7 GB)\n- **Precision-flexible**: The grad-based method requires a full-precision model that supports backpropagation. Our method, however, is flexible and can run on any precision model, even though it does not support backpropagation.\n\nThe accuracy in the application was similar to the grad-based method because the grad-based method had up to 2.6 times longer runtime (Tab. 2). If the grad-based method had a runtime as short as the grad-free method, the grad-free would likely be more accurate than the grad-based. To verify this, we additionally experimented on applications:\n\n- **Background-preserving image editing**: Figure R1 (in the rebuttal PDF) shows the qualitative results of applying our algorithm to an experiment, which investigates how exact inversion improves background-preserving image editing [32]. To compare accuracy at similar execution times, we adjusted the number of iterations to match the execution time. At comparable execution times, our grad-free method better preserves the background and achieves a lower NMSE.\n- **Watermarking classification**: Figure R2 (PDF) shows the qualitative results of applying our algorithm to the watermark classification [14] in Sec. 5. Our grad-free method either reduces the runtime compared to the grad-based or achieves better accuracy within the same runtime. We will add these additional qualitative results to the revision.\n\n$ $\n> **[W2]** *\"Experiments validating the convergence assumption are limited to specific hyperparameters for Stable Diffusion, LaVie, and InstaFlow. It is not clear if this behavior generalizes to other settings.\"*\n\nOur method is generally applicable to other LDMs because the hyperparameters in our method consist solely of the learning rate and number of iterations, which work the same for any model. As proof, Table S1 shows that it works well on three different models with the same learning rate and number of iterations.\n\n$ $\n# Questions\n\n> **[Q1]** *\"How robust is the assumption 𝐸∘𝐷 ≈ 𝐼 ? Can you provide empirical evidence for this assumption in the context of non-linear autoencoders?\"*\n\nAs in Line 260, *\"using the encoder is just OK\"* means $\\mathcal{E} \\circ \\mathcal{D} \\simeq \\mathcal{I}$. Many image editing works [31, 44] have been using $\\mathcal{E}$ as an adequate left-inverse of $\\mathcal{D}$, which explains $\\mathcal{E} \\circ \\mathcal{D} \\simeq \\mathcal{I}$. As empirical evidence from our own, the 'Encoder' column in Table 2 shows reasonably good accuracies (186/300, 149/300).\n\n$ $\n> **[Q2]** *\"Have you tested the proposed inversion method on video LDMs?\"*\n\nYes, we have. See Fig. 3b.\n\n$ $\n> **[Q3]** *\"How does the accuracy of the proposed method compare to gradient-based methods in applications such as image editing?\"*\n\nOur method is more accurate than grad-based. We answered the same content in **[W1]**, so please refer to **[W1]**.\n\n$ $\n> **[Q4]** *\"How were these parameters chosen? Have you done an ablation study?\"*\n\nAs we answered in **[W2]**, our hyperparameters are model-independent and generally applicable to other LDMs. Nevertheless, we newly conducted ablation studies on the optimizer and learning rate scheduling. Due to the limited space, we provided the result in the rebuttal for reviewer **9ej4, [W1]**.\n\n$ $\n# Limitations\n\n> **[L1]**: *\"Limitations have been briefly mentioned, yet there is not enough quantitative analysis for cases where the accuracy of gradient-based methods are significantly higher, and whether this affects applications in both image and video domain.\"*\n\nAs we mentioned in Line 257, grad-based method is more accurate than our method if sufficient runtime is available. Nevertheless, that is mostly not a big deal, because too much accuracy is unnecessary for applications. In Tab. 2-InstaFlow, the number of iterations was 100 for the grad-based method. According to Table S1c, more than 100 iterations are needed to be more accurate than the grad-free method. However, in Table 2-InstaFlow, the accuracy is 227/300, which is same to the grad-free. This shows that too much accuracy is unnecessary for the watermark classification. We will add this discussion in the revision. \n\nAdditionally, we provide new qualitative results for various applications in Figs. R1 and R2 by running the grad-based method for both longer and shorter runtimes. This would help a more in-depth discussion on the accuracy trade-off. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thank you for taking the time to provide such valuable feedback. We are delighted to learn that you found many strengths in our paper. All reviewers noted that our research offers advantages over existing gradient-based methods, particularly in terms of speed and memory. All reviewers also acknowledged that our reasonable assumptions were verified through experiments and that the paper is well-written. [DG1n] found our method novel, and [2mTK, 9ej4] found our method can be successfully applied to gain benefits in practical applications. [9ej4] recognized that our extensive results demonstrate the superiority of our method. [DG1n] found our method can be easily extended to any encoder-decoder structure, and [DG1n] appreciated that we have provided the code to ensure reproducibility.

$  $

While many strengths of our manuscript were identified, here we would like to further clarify the following key advantages / contributions of our work:

## 1. Our method is fast, accurate, memory-efficient, and precision-flexible.

We have already verified that, in addition to memory usage and runtime, our method also excels in terms of accuracy and precision versatility. To summarize the advantages of our method and their experimental evidence, our method is:

- **Fast**: Our method needs shorter runtimes for achieving the same NMSEs than the grad-based method. (up to 5X faster, in Fig. 3c and Tab. S1c, *1.89 s* vs 9.51 s to achieve -16.4 dB)
- **Accurate**: In the same runtime, our method shows smaller NMSEs than the grad-based method. (up to 2.3 dB lower, in Fig. 3b and Tab. S1b, *-21.37 dB* vs -19.06 dB in 25.1 s)
- **Memory-efficient**: Our method consumes a smaller amount of GPU memories than the grad-based method, which can be a significant advantage for large-scale LDMs on a GPU with limited memory. (up to 89% of the memory can be saved, in Fig. 3b, *7.13 GB* vs 64.7 GB)
- **Precision-flexible**: The grad-based method requires a full-precision model that supports backpropagation. Our method, however, is flexible and can run on any precision model that may not even support backpropagation. Our method can be immediately applicable to many existing LDMs distributed in half-precision.

$  $
## 2. Our algorithm provably converges with rigorous proofs.
We believe and would like to emphasize that our theorems with rigorous proofs are important contributions of this work. We provided that our novel algorithm converges through Theorems 1 and 2, with rigorous proofs included in the supplementary material. Notably, proving convergence with momentum (Theorem 2) is significantly more challenging than without it. 

$  $

We carefully responded to all reviewers' comments / concerns and will incorporate all the feedback in the revision.
To see Figures R1-R4, please download the PDF below.

Meta-review: In this paper, the authors present a gradient-free approach for inverting decoders in latent diffusion models.  The approach has the strengths of reducing computational complexity and memory usage compared to prior methods, along with permitting support for half-precision LDMs.  The authors additionally establish a proof of convergence and demonstrate practical utility of the approach in noise-space watermarking.  The paper would be strengthened if it provided additional applications in which comparable empirical performance gains were replicated.  The strengths outweigh the weaknesses, and the paper is recommended for acceptance.
Decision: Accept (poster)