Lightning-Fast Image Inversion and Editing for Text-to-Image Diffusion Models
Authors: Dvir Samuel
Barak Meiri
Haggai Maron
Yoad Tewel
Nir Darshan
Shai Avidan
Gal Chechik
Rami Ben-Ari
Venue: ICLR 2025 Poster (2025)
Keywords: Deterministic Image Inversion
Image Editing
Diffusion Models
Flow Matching
Image Generation
Primary area: generative models
TL;DR: A new image inversion method for text-to-image diffusion models, that introduces real-time editing by employing Newton-Rahson numerical scheme

Abstract: Diffusion inversion is the problem of taking an image and a text prompt that describes it and finding a noise latent that would generate the exact same image. 
Most current deterministic inversion techniques operate by approximately solving an implicit equation and may converge slowly or yield poor reconstructed images.  We formulate the problem by finding the roots of an implicit equation and devlop a method to solve it efficiently. Our solution is based on Newton-Raphson (NR), a well-known technique in numerical analysis. We show that a vanilla application of NR is computationally infeasible while naively transforming it to a computationally tractable alternative tends to converge to out-of-distribution solutions, resulting in poor reconstruction and editing. We therefore derive an efficient guided formulation that fastly converges and provides high-quality reconstructions and editing. We showcase our method on real image editing with three popular open-sourced diffusion models: Stable Diffusion, SDXL-Turbo, and Flux with different deterministic schedulers. Our solution, **Guided Newton-Raphson Inversion**, inverts an image within 0.4 sec (on an A100 GPU) for few-step models (SDXL-Turbo and Flux.1),
opening the door for interactive image editing. We further show improved results in image interpolation and generation of rare objects.

Published as a conference paper at ICLR 2025
LIGHTNING-FAST IMAGE INVERSION AND EDITING FOR TEXT-TO-IMAGE DIFFUSION MODELS
Dvir Samuel
Barak Meiri
Haggai Maron
Yoad Tewel2.3
Nir Darshan'
Shai 2vidan
Gal Chechik
Rami Ben-Ari
OOriginAI, Teel-Aviv University, BBar-llan University, *Technion, "NVIDIA Research
Israel
Inversion and Editing in 0.4
Real Image
pizza + bread
kitchen knife
wine glasses
wine white wine
ELR
ICLR
coffee milk
black pen yellow
ICLR in notebook
milk flowers
Figure 1: Consecutive real image inversions and editing using our GNRI with Flux.1-sche 1-shhnell
Black-
Forest 2024 (0.4 sec on an A A100 GPU).
ABSTRACT
Diffusion inversion is the problem of taking image and prompt that de- scribes it and finding a noise latent that would generate the exact same image. Most current deterministic inversion techniques operate by approximately solving implicit equation and may converge slowly or yield poor reconstructed images. We formulate the problem by finding the roots of an implicit equation and de- vlop method to solve it efficiently. Our solution is based on Newton-Rane Raphson (NR), a well-known technique numerical analysis. We show that vanilla ap- plication of NR is computationally infeasible while naively transforming it to a computationally tractable alternative tends to converge to fiit-f--disributiion solu- tions, resulting in poor reconstruction and editing. We therefore derive efficient guided formulation that fastly converges and provides high-quality reconstructions and editing. We showcase method real image editing with three popular open-sourced diffusion models: Stable Diffusion, SDXIL-Turo,, and Flux with different deterministic schedulers. Our solution, Guided Newton- Raphhsnn In- version, inverts an image within 0.4 sec (on an A100 GPU) for few-step models (SDXL-Turbo and Flux. 1). opening the door for interactive image editing. We fur- ther show improved results in image interpolation and generation of rare objects.
INTRODUCTION
Text-to-image diffusion models (Rombach al. 2022 Saharia et al. 2022 Ramesh al.) 2022 Bal-
aji et al. 2022) can generate diverse and high- fidh-fidelity images based on user-provided text prompts. These models further used in several important tasks that require inversion, namely, discover- ing an initial noise (seed) that, when subjected to a backward (denoising) diffusion process along
"oorrespnndence Dvir Samuel dvirsamuel@ @ gmail.com

---

Published as a conference paper at ICLR 2025
with the prompt, generates the input image. Inversion is used in various tasks including image edit- ing (Hertz et al. (2022). personalization (Gal et al.) 2023b a). seed noise interpolation for semantic augmentation (Samuel et al. 2023 and generating rare concepts (Samuel et al. 2024).
As inversion became a critical building block in various tasks, several inversion methods have been suggested. Denoising Diffusion Implicit Models (DDIM) (Song al., 2021) introduced deter- ministic and fast sampling technique for image generation with diffusion models. DDIM inversion transforms an image back into a latent noise representation by approximating the inversion equa- tion. Although this approximation makes it very fast, it also introduces an approximation error (as explained in section 3). causing noticeable distortion artifacts in the reconstructed images. This is particularly noticeable in few-step diffusion models (Luo et al., [0233] Sauer et al.] [2023] Esser et al.) 2024), with large gaps between the diffusion time-steps, and where inference is achieved with only 2-4 4 steps. Several efforts have heen made to address inconsistencies in DDIM Inversion, which result in poor reconstruction quality (Mokady et al. 2023 Wallace et al.) 2023 Pan et al.) 2023). For example, (Pan al. 2023 Garibi et al.) 20224) used fixed-point iterations to solve the DDIM- inversion equation. Another approach in (Hong et al. 2024) tackled the minimization of residual error in DDIM- iive--iversoon via gradient descent. Although these methods demonstrate improvements over previous approaches, their editing quality and computational speed are still limited.
[023]]ssser
In this paper, we frame the deterministic diffusion inversion problem as finding the roots of im- plicit function. We propose a solution based on the Newton -aatonnRaphoon (NR) numerical scheme (Bur-) den et al.] 2013) very fast and well-tested optimization method for root finding. We further define a scalar NR scheme for this problem that can be computed efficiently. However, NR has a main disadvantage in our context. When applied to our highly non-convex function, it tends to find roots that drift outside the distribution of latents that the diffusion model was trained on. To mitigate this, we introduce a guidance term that leverages prior knowledge of the likely solution locations, steer- ing the NR iterations toward in in-distribution roots. This is done by adding this prior knowledge on the noise distribution at each diffusion step. Ultimately, our approach enables rapid inversion while maintaining state-of-the-art reconstruction and editing quality.
We name our approach GNRI, for Guided Newton Raphson Inversion. GNRI converges in just a few iteration steps at each diffusion time step and achieves high-quality image inversions (in terms of reconstruction accuracy). In practice, 1-2 iterations are sufficient for convergence that yields significantly more accurate results than other inversion methods. Importantly, GNRI requires no model training, fine-tuning, prompt optimization, or additional parameters, and is compatible with all pre-trained diffusion and flow-matching models with deterministic schedulers. We demonstrate its effectiveness for inverting different deterministic schedulers used with latent diffusion model (Stable Diffusion) (Rombach et al.] 2022), few-step latent diffusion model (SDXII-turbo) (Sauer et al. (023)) and few-step flow matching model (Flux.1) 1) (Black-Forest) (024).. Figure[ the quality and speed of GNRI for iterative editing using few-step diffusion models. Using such models models that require 4 denoising steps, our approach can edit real images within 0.4 seconds. This allows users to edit images on the fly, using text-to-image models.
We conduct a comprehensive evaluation of GNRI. First, we directly assess the quality of inver- sions found with GNRT by measuring reconstruction errors compared to deterministic inversion ap- proaches. Our method suppresses all methods with x2 to x40 speedup gain. We then demonstrate the benefit of GNRI in two downstream tasks (I) In Image editing, GNRI smoothly changes fine details in the image in a consistent and coherent way, whereas previous methods struggle to do so. (2) In Seed Interpolation and Rare concept generation (Samuel et al. 2023) that require diffusion inversion. In both of these tasks, GNRI yields more accurate seeds, resulting in superior generated images, both qualitatively and quantitatively.
RELATED WORK
Text-to-image diffusion models (Rombach et al., 2022; Saharia et al., 2022; Ramesh et al. 2022 Balaji et al. 2022 Esser et al. 2024) translate random samples (seeds) from a high-dimensional space, guided by a uupr-supplied text prompt, into corresponding images. DDIM (Song et al. 2021) is widely used deterministic scheduler, that demonstrates the inversion of an image to its latent noise seed. When applied to the inversion of rext-guided diffusion models, DDIM inversion suffers from low reconstruction accuracy that is reflected in further tasks, particularly when applied to few-
2

---

Published as a conference paper at ICLR 2025
step diffusion models that require only 3-4 denoising steps. This happens because DDIM inversion relies on a linear approximation, causing the propagation of errors that result in inaccurate image re- construction. Recent studies (Mokady et al. 2023; Wallace et al. [0033] Pan et al. 2023 Hong al.) 2024) address this limitation. Null-text inversion (Mokady et al. (023) optimizes the embedding vector of an empty string. This ensures that the diffusion process calculated using DDIM inversion, aligns with the reverse diffusion process. (Miyake et al. (202)) replace the null-text embedding with prompt embedding instead. This enhances convergence time and reconstruction quality but results in inferior image editing performance. In both (Mokady et al., 2023) and (Miyake et al., 2023), the optimized embedding must be stored, resulting in nearly million additional parameters for each image (using 50 denoising steps of StableDiffusion (Rombach et al. 2022)
EDICT (Wallace et al. 2023) introduced invertible neural network layers, specifically Affine Cou- pling Layers, to calculate both backward and forward diffusion paths. While effective, it comes at the cost of prolonging inversion time. BDIA (Zhang et al.] (023) introduced a novel approximation tailored for EDICT, enhancing its computational efficiency while maintaining the accuracy in diffu- sion inversion. However, it still demands considerably more time, approximately ten times longer than DDIM Inversion. TurboEdit (Wu et al. 2024) introduced encoder- -baee iterative inversion technique for few-step diffusion models, enabling precise image inversion and disentangled image editing. DirectInv (Ju et al. (023) separates the source and target diffusion branches for improved text-guided image editing, leading to better content preservation and edit fidelity. AIDI (Pan et al.] 2023) and ReNoise (Garibi et al. 2024) used fixed-point iteration technique at each inversion step to find a better solution for the implicit function posed by DDIM equations and thereby achieving improved accuracy. ExactDPM (Hong et al., 2024). similar (Pan et al.
2023 Garibi et al. 2024) proposed gradient-bart based methods for finding effective solutions for the implicit function.
2024, Deutch et al. 2024) use stochastic DDPM inversion instead of a deterministic one. Although it can recover the exact image during reconstruction, the stochastic nature of these approaches usually causes the method to struggle with reconstructing fine details during editing. These methods require many steps for high- quality editing, making the process time-consuming. Additionally, they demand a larger memory footprint, as they need to store T + 1 latents for each inverted image, restricting their use only to reconstruction and editing tasks. We also compare our approach with these methods in terms of editing quality and demonstrate that our method outperforms them all.
Alternative methods proposed in (Huberman et al. 2023 Brack al.]
3 PRELIMINARIES
We first establish the fundamentals of Denoising Diffusion Implicit Models (DDIMs). In this model, a backward pass (denoising) is the process that generates an image from a seed noise. A forward pass is the process of adding noise gradually to an image until becomes pure Gaussian noise. Inversion (Song et al. 2021) is similar to the forward pass but the goal is to end with a specific Gaussian noise that would generate the image if denoised.
Forward Pass in Diffusion Models. Diffusion models (Rombach et al. 2022) learn to generate images through systematic process of iteratively adding Gaussian noise to a latent data sample until the data distribution is mostly noise. The data distribution is subsequently gradually restored through a reverse diffusion process initiated with a random sample (noise seed) from a Gaussian distribution. In more detail, the process of mapping (latent) image to noise is Markov chain that starts with žu. and gradually adds noise to obtain latent variables 21, 22,.... 22, following distribution g(21, :2,...; 2T 20) (T==)(=((=<_).. where Vi at Rd with d denoting the dimension of the space. Each step in this process is a Gaussian transition, that is, g follows a Gaussian distribution
(<x//==1) == N(55 = 11 E, = 3,7),,
(1)
parameterized by a schedule (Bo, ro),..., (BT, E 1), As discussed below, in DDIM, 7t- = v1 - Be and in Euler scheduling 3r = 7/2 = Vt.
Deterministic schedule diffusion models. It has been shown that one can "sample" in a determin-
istic way from the diffusion model, and this accelerates significantly the denoising process. Several deterministic schedulers have been proposed (Song et al., [021] Lu et al. 2022 Karras et al. 2022).
describe here two popular ones, DDIM and Fuler schedulers.
3

---

Published as a conference paper at ICLR 2025
Denoising Diffusion Implicit Models (DDIM). Sampling from diffusion models can be viewed as solving the corresponding diffusion Ordinary Differential Equations (ODEs) (Lu et al. 2022). DDIM (Song al. 2021). popular deterministic scheduler, proposed to denoise a latent by
2t-1
at-1
2t va--i A co(zent,p),
where = 1- B. (x)) of network that trained to predict the noise to be removed.
1, and (v()r) = -ia))) and (z( p, is the output
Euler schedulers. Euler schedulers follow a similar deterministic update rule
2t-1 = + (at-1 02 (1)ua(2z: t),,
(3)
here dz dr are scheduling parameters and vi is the output of the network that trained to predict the velocity.
Diffusion inversion. We now focus inversion in the latent representation. We describe approach first for DDIM, and extension to other schedulers is discussed below. Given an image representation 20 and its corresponding text prompt p. seek noise seed 2T that, when denoised, reconstructs the latent øp- Several approaches were proposed for this task. DDIM inversion rewrites Eq. (2) as:
= =(())
f(x)
0+-1 2-1 Vor Arb(au) c0(2),
(4)
DDIM inversion approximates this implicit equation in 2 by replacing 2 with 21-1
fzx)
21-1 + voi Av(ai) c0(2)-1,t,p)
(5)
a--1
The quality of the approximation depends on the difference 2, - -1-1 (a smaller difference would yield small error) and the sensitivity of Eg to that z, See (Dhariwal & Nichol 2021 Song al.) 2021) for details. By applying Eq. (5) repeatedly for every denoising step f, one can invert an image latent 2y to latent 2y in the seed space.
DDIM inversion is fast, but the approximation of Eq. (5) inherently introduces errors at each time step. As these errors accumulate, they cause the whole diffusion process become inconsistent in the forward and the backward processes, leading to poor image reconstruction and editing (Mokady
al. 2023 Wallace al. 2023 Pan al., 2023). This is particularly noticeable in few-step and consistency models with small number of denoising steps (typically 2-4 steps), where there's ' significant gap between 7, and - Garibi al. (2024).
This inversion technique can also be applied to other deterministic schedulers. For instance, for Euler scheduler, one defines 2t f(2). f(2)) == 2t-1 + (at ot-1) 00(2,, and this is ap- proximated by 2 + (a; - ot-1) (2(_1,,,p)))
Iterative inversion and optimization methods. Several papers proposed to improve the approxi- mation using iterative methods. AIDI (Pan et al. 2023) and ReNoise (Garibi et al. 2024) proposed to directly solve Eq. 4 using fixed-point iterations (Burden 1985), widely- used method in nu- merical analysis for solving implicit functions. In related way, (Hong et al. 2024) solves a more precise inversion equation, obtained by employing bigher-order terms, using gradient descent.
Newton-Raphson Root finding. The Newton-Raphson method is widely used numerical tech- nique for finding roots of a real-valued function (Burden et al. 2015). is particularly effective for solving equations of the form F(z) = O. when F RD - RD for an arbitrary dimension D. provides fast convergence, typically quadratic, by requiring the evaluation the function and the inversion of its Jacobian matrix. It has also been shown that when initialized near a local extremum, NR may converge to that point (oscillating around it) (Kaw et al. 2003). The NR scheme in general form, is given by
A++ - 2 (( F(+)),
where -( presents the inverse of a Jacobian matrix E DxD for F (all derivatives are w.r.t 2), and stands for the iteration number. The iteration starts with an initial guess, 20.
4

---

Published as a conference paper at ICLR 2025
Zoom-in
VAE Encode
2T Noise Seed
T--1
Zt
zº -
21
20
F(
( = (i)
(z +1
Newton Raphson Iterations
Figure 2: Newton- Raptonn Inversion iterates over an implicit function Eq. Jusing Eq. 10 schemm,,, at every time-step in the inversion path. Initialized with zi = 2-11 it converges within 2 iterations, to 2t- Each box denotes one inversion step; black circles correspond to intermediate latents in the denoising process; green circles correspond to intermediate Newton Rapionn iterations.
4 OUR METHOD: GUIDED NEWTON RAPHSON INVERSION
Existing inversion methods are either sesstbut-inaccuraal like DDIM inversion, or more precise-but- slow, like (Pan et al. 2023) and (Hong et al. 2024). This paper presents a method that achieves balance of speed and improved precision compared to these existing approaches.
To achieve these features we introduce two key ideas. First, we frame inversion as a scalar root- finding problem that can be solved efficiently, and use the well-known Newton-Raphan root-finding method. Second, we make the observation that an inversion step equivalent to finding a specific noise vector, and we use strong prior obtained from the distribution of that noise. This prior during the iterative solution guides Newton-Rann Raptonn towards finding a root that adheres to the correct distribution of latents. See Figure 2 for illustration.
4.1 FRAMING INVERSION AS EFFICIENT ROOT-FINDING
Diffusion inversion can be done by finding the fixed point of the function f(z) in Eq. 4 This is defined
equivalent to finding the zero-crossings or roots, 2, of the residual function r RR. + as r2x) := Zt f(z).. One could now apply Newton-Raphson (NR) to find root. However, in our case, 7t is a high dimensional latent(D 16K in StableDiffusion (Rombach et al. (0222), making it computationally infeasible to compute the Jacobian (Eq. (6. explicitly and invert it. To address this computational limitation, we apply a norm over r(2),, f Rd + R2 yielding a vurtilvariable, scalar function, which has the same of roots:
f(zx)
1| f(x))li
and seek the roots r(=)) 0 where ||-|| denotes a I,, norm which simply sum over all absolute values of A. Applying the norm on the residual has be done in previous work (Hong et al. 20244 Garibi al. 2024), however here is used to reduce the Jacobian to vector and that can be computed We call this method Newton-Raphson Inversion, NRI.
quickly (see derivation in Appendix
This method is efficient and converges quickly in practice. However, NRI in our case has a key limitation: Eq. (4) and thus f, can have multiple solutions or local minima. The accelerated con- vergence of NR due linear "extrapolation", can cause it to "jump" solutions outside the latent variable distribution for the diffusion model. Indeed, in practice, we find that solving Eq. 7 for f(2)) 0 often converges quickly to solution that results with poor image reconstruction. We quantify and illustrate this phenomenon section 5.1 Next, we describe a remedy to this problem.
4.2 GUIDED NEWTON RAPHSON INVERSION
The NR procedure discussed above may find roots that are far from the modes of the latent distribu- tion. How
guide NR converge useful roots among all possible ones?
Note that determining a 2 given 24-1- is equivalent to finding which noise term is added during the forward pass of the diffusion process. Luckily, the distribution of that noise is known by the design of the diffusion process and we can use it as prior information. More precisely, since each step in the diffusion process follows Gaussian distribution q(=x (z-1) (Eq. []. define a positive function
5

---

Published as a conference paper at ICLR 2025
that obtains a zero value at the maximum likelihood of the distribution by taking the negative log of the exponent part of the distribution (without the normalizing factor). This can be simplified to
1| = well
DDIM,
G(2+)
mell
Euler.
where ur, 3e were defined in Section Eq. (I). To guide NR to select an "in-distribution" solution, we define new objective F + R5:
F(z.) = || - [=+)]l + AG(zt).
hyperparameter weighting factor for the guidance term. See Appendix E for ablation of À values. Note that this guidance term incorporates prior knowledge about the probable locations of the solutions in space.
Here, A
We then seek to drive F(2e) near zero. Applying the Newton Raphson iteration scheme for Eq. (9
then emerges (see derivation in Appendix Ap:
of 2t-1
1
F(+t) gi(2f) +
+ = 2 (i)
(10)
Here 9i 8F((a) is the partial derivative of F with respect to the variable (descriptor) 2; (1). The (2)
index E [0, 1..1. D] indicates z's components and i stands for the iteration number, while n is small constant added for numerical stability. The function g can be computed efficiently using automatic differentiation engines. We initialize the process with z from the previous diffusion time- step žt-1- We call approach GNRT for Guided Newton Raphson Inversion. For discussion on applying NR on our objective function and the guidance term see Appendix B
5 EXPERIMENTS
Here, we first provide motivation and analysis for the guidance term presented in Section 4.2 Then, we show evaluation results of GNRI on three main tasks: (I) Image inversion and recon- struction: Here, We assess the inversion fidelity by evaluating the quality of the reconstructed images. (2) Image Editing: We demonstrate the efficacy of our inversion scheme in image editing, highlighting its ability to facilitate real-time editing. (3) Rare Concept Generation: We illustrate how our method can be applied to improve the generation of rare concepts.
Compared Methods:
We compared our approach with the following methods. Standard DDIM Inversion (Song al., (02T), SDEdit Meng et al. (2022), Null-text (Mokady
2023) and
EDICT WWllace et al. 2023). Two fixed-point based methods: AIDI (Pan et al. 2023 and
ReNoise (Garibi et al. 2024) with further comparison to Anderson Acceleration Pan al. (2023)
shown in Appendix [| Gradient-based method ExactDPM (Hong et al., (024). Wu2024Tume Turbo (Wul eet al.) 2024) and two DDPM inversion methods: EditFriendie Huberman et al. (2023) and Gal2024Turbo (Deutch et al. 2024). In all experiments, we used code and parameters provided by the respective authors.
Implementation details: To demonstrate the versatility of our approach, conducted experiments on latent diffusion model SD2.1 Rombach et al.] (2022) and two few-step models: SDXL-turbo (Sauer et al. 2023) and the newly introduced flow matching Flux.l-schnell model (Black-Forest) 2024). Sampling steps were set to 50 for SD2. and 4 for SDXL-turbo and Flux. All methods ran until convergence. All methods were tested on single A 100 GPU for a fair comparison. PyTorch built-in gradient calculation was used for computing derivatives of Eq. (9). Editing for all iterative methods was done using Prompt-to-Prompp (Hertz al. 2022).
5.1 GUIDANCE TERM MOTIVATION AND COMPARISON TO OTHER NUMERICAL SCHEMES
To illustrate the motivation behind guidance term, compare the reconstruction quality of non-guided NRI a "vanilla" NR inversion method with two iterative numerical schemes: Fixed- point Iteration (FPT) ((Pan et al. 2023) and Gradient Descent (GD) (Hong et al. (2024). We test the

---

Published as a conference paper at ICLR 2025
(b)
1e-5
5.2
26
--tt21l
Better-
llog(q
5.0
PSNR
24
Descent
3.5
4.8
Fixed Point (AIDI)
NRI
4.6
22
GNRI (Ours)
2.5
4 6
2
4
10 12 14 16 # iterations
0.0
2.5
5.0
7.5
10.0
# iteration
# iteration
Figure 3: The effect of GNRI guiding term NR inversion, and comparison other iterative inversion methods. All results are averages computed with SDXI-Turbo applied to 5,000 COCO images. (a) Average residuals throughout optimization. NR-based methods the fastest to converge. Gradient-descent was run with the largest learning rate that was stable but still is slowest. (b) Reconstruction quality (PSNR). Adding guidance (blue) NRI (green) significantly improves the quality of the converged solution. (c) Likelihood of inferred noise. Without the guiding term, NRI (green) finds solutions that are substantially different from those found by other methods, which explains the low reconstruction quality.
ReNoise (fieed-paint)
ExactDPM (gradien-ddescent)
Input Image
DDIM
GNRI (OURS)
SDXL-turbo
26
GNRI (Ours)
SDXL turbo
kelle world
HELLLO
hulle world
&o wored
laffe world
PSNR
Wu24Turbo
ExactDPM (gadiientddesccm))
ReNoise
(fxod-palnt)
DDIM
15 Backward-Forward Time [sec]
Flux.1
26
GNRI (Ours)
PSNR
Flux1
18
ReNoise (fived-point)
ExactDPM (g (gadient-descoe descand)
14
DDIM
10
15
Brckward-Forwark Time [sec]
Figure 4: (Left) Reconstruction qualitative results: Comparing image inversiion-reconsttuciiion performance. While all baseline methods struggle to preserve the original image, GNRI successfully excels in accurately reconstructing it. (Right) Inversion Results: Mean reconstruction quality (y- axis, PSNR) and runtime (x-axis, seconds) on the COC02017 validation high PSNR while reducing inversion-recoostruciion time by a factor of x2 (compared to DDIM) and up x0 (compared to ExactDPM) on SDXL-tturo and x 10 x40 Flux.1, compared to other approaches.
Our method achieves
reconstruction of 5,000 image-caption pairs from COCO Lin et al. (2014), presenting three metrics in Fig. Fig 3 shows that FPI, GD and non-guided NRI, all converge to low residual values but the reconstruction values for NRI (green line) in Fig. 3 are notably lower. This suggests that solutions reached by NRI are areas of the latent space that yield poor results. Comparing the blue curve in Fig 3 to the green curve highlights the significant impact of adding guidance term. It accelerates convergence (Fig. 31), improves reconstruction (PSNR) (Fig 33) and reaches a better likelihood (Fig. 3e).
7

---

Published as a conference paper at ICLR 2025
ReNoise
EXactDPM
Input Image
SDEdit
EditFriienlly
(prddeent-lesone
GNRI (OURS)
fe workd
WOT
Agreement with image
SDXL-turbo
0.18
GNRI[Ours]e
Gat dog
0.23
(land--piir))
RaNolse
Wuz4furbo
S00 SOXU -turbo
LPIPS
De.to124Turbs
0.28
PnP
ExactDPM jesddeet-Suscong
0.33
EdiFriendly
SDEdit
0.38
Better
0.20
0.25
0.30
CLIP Score
Agreement with prompt
Agread Agreement with image
Flux.1
steal apple
0.13
GNRI (Ouns)
0.18
Flux.1
LPIPS
0..33
0.28
ExactDPM PaP Sandiere destenn
ReNoise (Tadd point
0.33
EditFrienddl
CBE
0.38
0.20
0.25
0.30
Thank Your!
Bank Youe!
Thoop Yavet
Tronle y
Gaod lye!!
CLIP Score
Agreement with prompt
Thank Good Bye
Figure 5: (Left) Qualitative results of image editing. GNRI edits images more naturally while preserving the structure of the original image. baselines were executed until they reached con- vergence. (Right) Evaluation of editing performance: GNRI achieves superior CLIP and LPIPS scores, indicating better compliance with text prompts and higher structure preservation.
5.2 IMAGE RECONSTRUCTION
To evaluate the fidelity our approach, we measure the PSNR of images reconstructed from seed inversions. Specifically, used the entire set of 5000 images from the MS-COCO-2017 validation dataset (Lin et al. 2014). along with their corresponding captions. For each image-caption pair, we first found the inverted latent and then used it reconstruct the image using the same caption. Given that the COCO dataset provides multiple captions for each image, we used the first listed caption as conditioning prompt. Figure (left) (left) demonstrates qualitative comparison of reconstructed images. Figure 4 (right) illustrates the PSNR of reconstructed images versus inversion time, highlighting the performance of our method compared to SoTA inversion techniques. The dashed black line represents the upper bound set by the Stable Diffusion VAE, showing that method achieves the highest PSNR among SoTA methods and approaches the VAE upper bound, while also being the fastest. Additional qualitative comparisons for SD2.1 and further analysis are in Appendix H
5.3 REAL-TIME IMAGE EDITING
Current state-of-the-art the-art methods for editing images start by inverting real image into latent space and then operating on the latent. As a result, the quality editing depends strongly on the quality of inversion (Mokady al. 2023
We now evaluate the effect inverting images with GNRI, compared to SoTA inversion baselines. We show that GNRI outperforms all baselines in editing performance while requiring the shortest time. This opens the door for real-time editing capabilities. We present below both qualitative and quantitative results. For Neural Function Evaluations (NFE) comparison Appendix H
8

---

Published as a conference paper at ICLR 2025
Table 1: Image editing comparison using descriptive text in PlE-Bench (uu al. 2023) dataset with SDXL-turbo. The efficiency is measured in single A100 GPU. Our method achieves the best background preservation and clip similarity while being significantly faster than other methods.
PlE-Bench
Background Preservation
CLJP Similarity Edited
Efficiency
PSNR
LPIPS
MSE
SSIM
Whole
Inverse
Forward
1
10a 4 10* 4 102 f
4-
(8) 4
DDIM
18.59
177.96
184.69
66.86
23.62
21.20
0.454
0.445
ReNoise (Garibi et al. 2024
27.11
49.25
31.23
72.3
23.98
21.26
5.41
0.445
ExactDPM (Hong
2024
24.54
59.88
36.49
69.18
23.77
21.23
15.11
0.445
EditFriendly [Huberman al. 2023
24.55
91.88
95.58
81.57
23.97
21.03
32.3
23.1
Direct Inv (Ju al. 2023
27.22
54.55
62.86
84.76
25.02
22.10
4.3
50
TurboEdit (Wu et al. 2024
29.52
44.74
26.08
91.59
25.05
22.34
0.981
0.569
GNRI (Ours)
30.22
40.15
20.10
98.66
26.15
23.26
0.521
0.445
Interpolation|
Centroid
Table 2: Image interpolation and centroid finding. In interpola- tion, two images x², x²,²² are in- verted to generate images between their seeds 2/. 27. In centroid- finding, a set of images is inverted find their centroid. Best perfor- mance is achieved with our GNRI.
ACC FID 4 |ACC FID 4
DDIM Inversion ExactDPM (Hong et al. 2024)
51.59
6.78
67.24
5.48
52.01
6.13
68.14
5.32
ReNoise (Garibi et al. 2024)
52.51
6.0
69.14
5.22
GNRI (ours)
54.98
5.91
70.18
4.59
Qualitative results. Figure leftt) gives qualitative comparison between SoTA approaches and GNRI in the context of real image editing. GNRI excels in accurately editing with target prompts producing images with both high fidelity to the input image and adherence the target prompt. The examples illustrate how alternative approaches may struggle to retain the original structure or tend overlook crucial components specified in the target prompt. As an example, in the first row of Figure leftt, GNRI exclusively converts the cat into dog without changing the text in the sign whereas other methods either fail to generate proper dog or change the text in the sign. Results for SD2. in Appendix H For additional qualitative results see also Appendix [ and K User Study. We further evaluated editing quality using human raters. We followed the evaluation process of Mokady et al. (2023) and asked human raters to rank edits made by the three leading methods: Ours, ReNoise (Garibi et al. 2024). and ExactDPM (Hong et al., 2024). Unfortunately. the code for TurboEdit (Wu et al. 2024)
not available at the time of our evaluation. 100 images were rated, 12 images were provided by the authors of (Mokady et al. (202)) and the rest were randomly selected from COCO (Lin et 2014) and PieBench (Ju al., 2023) benchmarks. Three raters for each image were recruited through Amazon Mechanical Turk and were tasked to choose the image that better applies the requested text edit while preserving most of the original image. GNRI was preferred with ExactDPM (Hong et al. 2024) 12.5%.
71.6% preference compared to ReNoise's (Garibi et al.
2024) 15.9% and
Quantitative results. Following (Wallace et al., 2023 Huberman et al. 2023 Garibi et al., 2024), we evaluate the results using two complementary metries: LPIPS (Zhang et al., 2018) to quantify the extent of structure preservation (lower is better) and CLIP-based score to quantify how well the generated images comply with the text prompt (higher is better). Metrics are averaged across 100 MS-COCO images. Figure riighh illustrates that editing with GNRI yields superior CLIP and LPIPS score, demonstrating the ability to perform ofate-of-the-rt real image editing with supe- rior fidelity. Following (Wu et al.
also evaluated our approach the newly introduced PieBench (Ju et al., 2023) dataset. Table [ shows that our method more effectively follows the text guidance while preserving the background, outperforming current SoTA techniques. These evalua- tions further aftirm the findings derived from the user study.
2024)
5.4 SEED INTERPOLATION AND RARE CONCEPT GENERATION
We evaluate the benefit of our inversion in a second task, of selecting latent seeds for image gen- eration. It has been shown (Samuel et al. 2024 2023) that text-to-image diffusion models poorly
9

---

Published as a conference paper at ICLR 2025
Inversion Quality Impact on Rare Concept Generation: We assess image generation using a pre-trained classifier's accuracy, comparing NAO (Samuel et al. 2023) (with DDIM inver- sion), ReNoise (Garibi et al., 2024). ExactDPM Zhang et al. (2023) and GNRI. We report average per-class accuracy for Head (over 1M samples), Medium, and Tail (rare classes < 10K samples). GNRI enhances rare and medium concept accuracy without sacrificing overall performance.
Table 3:
ImageNet1k in LAION2B
Head
Medium
Tail
Total Acc
FID T Topt (sec) (sec)
n=235
n=9
n=256
Methods
#> 1M
IM>#> 10K
10K:*
DDIM inversion
98.5
96.9
85.1
94.3
6,4
25
29
ExactDPM (Hong al.
2024
98.1
96.8
85.3
94.1
7.1
240
32
ReNoise (aaribi al. 2024
98.5
97.0
85.3
94.4
6.9
24
GNRI (ours)
98.6
97.9
89.1
95.8
6.3
17
25
generated rare concepts, and this can improved by selecting better noise latent 2T. To address this issue, SeedSelect (Samuel et al. (024) takes few images of a rare concept input and uses diffusion inversion module to iteratively refine the obtained seeds. These refined seeds are then used generate plausible images of the concept. NAO (Samuel et al. 2023) extends this by introducing new paths and centroids for seed initialization. Both methods rely on DDIM Inversions, crucial for initial seed evaluation. GNRI provides an alternative for precise inversion seeds, aiming for improved image quality and semantic accuracy. It's important to highlight that most inversion
techniques (Mokady al. 2023] Wallace al. 2023 Huberman al.) 2023 Brack al. 2024) are
not applicable in this context as they necessitate extra parameters for image reconstruction, which
impedes the straightforward implementation of interpolation.
Interpolation and centroid finding: We evaluate GNRI by following the experimental protocol of (Samuel et al., 2023) and compare images generated by four methods: DDIM, ReNoise (Garibi jet all]|22022] ExactDOM (Hong et al. (0024) and GNRI. Evaluation is conducted based on FID score and classification accuracy (by a pre-trained classifier). Results are presented in Table Notably, initializing with GNRI seeds consistently results in higher-quality images both in interpolation paths and seed centroids.
Rare concept generation: We further show the effect of our seed inversions the performance of NAO centroids with SeedSelect for rare concept generation. Specifically, we compared images generated by SeedSelect, initialized with NAO using DDIM inversion, ReNoise, ExactDPM, and GNRI, as shown Table 33. We followed the evaluation protocol of (Samuel et al. 2024 (023) on ImageNetlk classes arranged by their prevalence in the LAION2B dataset (Schuhmann al. 2022). The results, summarized in Table 3 demonstrate that our inversion method significantly boosts performance, both classification accuracy and FID, compared to other methods. Further- more, our inversions yield more precise and effective initialization point for SeedSelect (Samuel al. 2024), resulting in notably quicker convergence without compromising performance, in all categories (head, medium, and tail) along with a high gap at tail.
6 SUMMARY
Image inversion in diffusion models is vital for various applications like image editing, semantic augmentation, and generating rare concept images. Current methods often sacrifice inversion qual- ity for computational efficiency, requiring significantly more compute resources for high-quality results. This paper presents Guided Newton-Raphson Inversion (GNRI), novel iterative approach that balances rapid convergence with superior accuracy, execution time, and memory efficiency. Us- ing GNRI opens the door for high-quality real-time image editing due to its efficiency and speed. GNRI requires no model training, fine-tuning, prompt optimization, or additional parameters, and is compatible with all pre-trained diffusion and flow-matching models with deterministic schedulers.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to express our sincere gratitude Prof. Nir Sochen for the intriguing discussion during the course of this study.
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mïika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. eDiff-II: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022.
Black-Forest. Flux: Diffusion models for layered image generation. https::/github. com/ 2024.
Manuel Brack, Felix Friedrich, Katharina Kormeeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolinaros Passos. Ledits++: Limitless image editing using text-to-image models. In CVPR, 2024.
Douglas Burden, Richard L.; Faires. Fixed-point iteration. 1985.
R.I. Burden, J.D. Faires, and A.M. Burden. Numerical Analysis. Cengage Learning, 2015. ISBN 9781305465350. URL wtttps:/bboos..ooo httpb//wooo.oooooooooeoooooooooe www.ooog.......o....... le. /lwpcooks...e90---BBB.
Gilad Deutch, Rinon Gal, Daniel Garibi, Patashnik, and Daniel Cohen-Or. Turboedit: Text-based image editing using few-step diffusion models, 2024.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. ICLR, 2023a.
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. Designing encoder for fast personalization of text-to-image models. arXiv preprint arXiv:2302. 12228, 2023b.
Daniel Garibi, Or Patashnik, Andrey Voynov, Hadar Averbuch-Elow and Daniel Cohen-Or. Renoise: Real image inversion through iterative noising. arXiv preprint arXiv:2403. 14602, 2024.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arX1v:2208.01626, 2022.
Seongmin Hong, Kyeonghyun Lee, Suh Yoon Jeon, Hyewon Bae, and Se Young Chun. On exact inversion of dpm-solvers. CVPR, 2024.
Inbar Huberman, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inver- sion and manipulations. arXiv:2304.06 06040. 2023.
Xu Ju, Ailing Zeng. Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Direct inversion: Boosting diffusion-based editing with lines of code. ArXiv, 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine, Elucidating the design space of diffusion- based generative models. NeurIPS, 2022.
Autar Kaw, Nathan Collier, Michael Keteltas, Jai Paul, and Glen Besterfield. Holistic but customized resources for a course in numerical methods. Computer Applications Engineering Education, 11(4):203-100 2003.
11

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In , 2014.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Syn- thesizing high-resouution images with few-step inference, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with text- text-guided diffusion models. arXiv preprint arXiv:330..6600, 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. CVPR, 2023.
Zhihong Pan, Riccardo Gherardi, Xiufeng Xie, and Stephen Huang. Effective real image editing with accelerated iterative diffusion inversion. In ICCV, 2023.
Boris Polyak and Andrey Tremba. New versions of newton method: step-size choice, convergence domain and under-determined equations. Optimization Methods and Software, 35(6):1272-1333, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi. Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022.
Dvir Samuel, Rami Ben-Ari, Nir Darshan, Haggai Maron, and Gal Chechik. Norm-guided latent space exploration for text-to-image generation. NeurIPS, 2023.
Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. Generating images of rare concepts using pre-trained diffusion models. AAAI, abs/2304. 0..3304.1533,, 2024.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021.
Bram Wallace, Akash Gokul, and Nikhil Vijay Naik. EDICT: Exact diffusion inversion via coupled transformations. CVPR, 2023.
Zongze Wu, Nicholas Kolkin, Jonathan Brandt, Richard Zhang, and Eli Shechtman. Turboedit: 003322333..38
Instant text-based image editing. arXiv preprint arXiv:2408. 08332, 2024.
Guoqiang Zhang, Jonathan Lewis, and W Bastiaan Kleijn. Exact diffusion inversion via bi- directional integration approximation. arXiv:2307.10829, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
12

---

Published as a conference paper at ICLR 2025
Appendix
A NEWTON METHOD FOR MULTIVARIABLE SCALAR FUNCTION
In this section, we present the formulation of Newton's method for zero crossing of multi-variable scalar function. For vector R2 and function f(x) : IR + R are looking for the roots of the equation f(x) = 0. Assuming that the function is differentiable, we can use Taylor expansion:
6) = f(x) + f 8 + o(118113)
(A1)
For every 8,x RR. The idea of Newton's method in higher dimensions is very similar to the one-dimensional scalar function; Given an iterate x", we define the next iterate x*+1 by lincarizing the equation f(x*) 0 around x* above, and solving the linearized equation f(x* + 0. Dropping the higher orders we then solve the remaining linear equation for 5. In fact, in the scalar case, the solution for is not unique, with each solution potentially determining different direction of propagation of the iterate x" in the high-dimensional input space. The optimal direction cannot be known in advance, and some have proposed selecting the propagation direction by imposing constraints, such minimizing 8 (Polyak & Tremba, 2020). We adopt simple, non-sparse, yet effective solution, that allows the scheme flexibility deriving the direction of propagation:
1 01
f(x*)
(A2)
D
ar 82
ar 3PD
D22
where z; indicates the i-th component of vector x. The above can be easily proven as solution by substituting 5 into Eq. [T] dropping the higher orders setting it equal to zero. Using the relation, = x++1 x we get the final iterative scheme:
1 [(**)
(A3)
D
of
(x*)
Oo;
Note that this equation is component-wise i.e. each component is updated separately, facilitating the solution.
B DISCUSSION ON THE NEWTON-RAPHSON METHOD FOR FUNCTIONS WITHOUT ZERO CROSSINGS
Note that our objective function in Eq. 9 is a non-negative function. When the target function F 2 0 and does not have zero crossings,
the NR procedure may converge to a (non-zero) minima. common example is the function f(x) x 1 that has no real roots, but NR converges to its minimum. Note that while the Newton- Raphson method is typically used to find zero crossings of a function, our approach applies it to drive convergence toward local minima of the objective function (being locally convex around this point). The analysis of this type of convergence within the Newton-Raphson framework is left for future numerical analysis research.
C GNRI SCHEME CONTRACTION MAPPING
Fixed-point iterations are guaranteed to converge if the operator is contractive in the region of inter- est. Formally, a mapping T is contractive on domain there exists p < (also called Liphshitz constant) such that [xx) T(y)l Σ p|| yll. v(I, y) E X, where || || denotes norm. In the following we will demonstrate empirically that our GNRI scheme introduces contraction mapping, property needed for convergence of the scheme. Contractive mappings ensure that each iteration moves closer to the limit, with the distance shrinking at a predictable rate. The rate of contraction (the Lipschitz constant) can be used give bounds on how fast the error decreases, helping sess the efficiency of the scheme. Next, we show empirically that, the mapping is contractive in our region of interest.
13

---

Published as a conference paper at ICLR 2025
SDXL-Turbo
Flux.1
-1.
-1.5
-2.0
tm0
225
t=0
1ogth
-2.0
logt
-3.0
tm3
tm3
-3.5
-2.5
-4.0
-3.0
4.5
iteration
iteration
Figure Cl:
Contraction Mapping. Contraction mapping for two fast diffusion models are shown (in log-scale), each time-step. The scheme converges in 2-3 iterations with a notable contraction occurring in the first iteration, assisting the rapid convergence. t = is close to the image side and t == 3 close to seed.
According to Eq. 10 the GNRI scheme for the solution F(z) 0 (defined in Eq. 9). at each diffusion time-step, given by:
F(2
(C1)
gx(2*)
0J is the dertial-derivaivve of F with respect to the variable 2;, while i E [0, 1, D] where D: 8æ; indicates z's components. We can rewrite the NR iteration as the fixed-point iteration f(z) = by
setting f(2) as:
-(+))== 1 F(**)
C2)
D gx(2*)
Note that the fixed point of the newly introduced f is the solution of F(z) = 0.
Let us now define new residual function:
=( = 11F(2*) - =1111
(C3)
By substitution we get:
F(aht)) - 12f(2 - f(2 F(=*) 11((=)) - = || - =1111
(C4)
Therefore contraction mapping requirement for f: [1f()*+1) f(>h)]1 s p||zk+1 - - can be written as:
F(2*+1) s pp(tt)
(C5)
and the contraction mapping is satisfied 7 follows exponential decay (r(2*) S p r(20) with
some p < 1. Note that r(=*) given in Eq. C3 can be computed at each iteration step. In Fig. CI we plot F(z*), tested on 5,000 image caption pairs from the COCO dataset (the same as used Sec.
5.1 showing that f(=*) indeed decays exponentially (linearly in ocg-scale), till convergence. This
behaviour therefore implies that GNRI iteration a contraction mapping. Note that our contraction mapping does not converge to zero, namely the fixed point, since objective function F may not have an exact solution. Computing contraction rate p for GNRI shows = 0.50 (at the first iteration) while p 0.90 for FPI indicating the faster convergence of GNRI.
Note that we further show in the main paper the convergence of the equation residual f l1f(2)) zill (EQ. 7 indicator for convergence toward the inversion solution.
14

---

Published as a conference paper at ICLR 2025
"Your guess is as good as mine as to what these objects are."
"A thing is in the outline and shows up like something"
"Food on a train with a pie and some vegetables."
Figure DI: Failure cases: GNRI fails to converge where the prompt and image do not align.
26
PSNR
24
22
0.0
0.2
0.4
0.6
0.8
1.0
A
Figure D2: The influence of À on reconstruction performance. We observe that employing no reg- ularization (A 0) leads to poor reconstruction, while À = 0.1 typically yields the highest recon- struction accuracy. As increases, reconstruction accuracy declines, possibly because assigning excessive weight to the prior undermines the root-finde not-finding objective.
D FAILURE CASES
Here we provide information about an analysis to find failure cases for convergence. ran in- version and reconstruction, in scale, COCO2017 (118k caption-image pairs) and found that in 95.4% of cases, GNRI successfully converged to a solution. Specifically, residuals went down from (for DDIM) solutions. The remaining 4.6% were samples with incorrect captions, see Fig. SDI These results imply that lack of convergence may indicate text and image miss-alignment which we consider for future work.
10-4 indicating convergence, and the PSNR
25.7 indicating good
E PRIOR TRADE-OFF PARAMETER
Figure D2 shows the impact of the weight parameter on the reconstruction accuracy. We observe that using regularization (A 0) results in poor reconstruction, whereas setting = 0.1 achieves the highest reconstruction accuracy, underscoring the importance of the prior in the objective func- tion. There is gradual decrease with larger À values, where = 1 the PSNR starts to decline faster, likely because placing too much emphasis on the prior compromises the residual root-fin -indiinnig objective.
15

---

Published as a conference paper at ICLR 2025
Inversio-Reconstrrution
Editing
26.0
Ours
Agreement with image
0.15
PSNR
25.8
Better
ExactDPM (gradient-descent)
0.20
Better
EDICT
Ours
LPIPS
0.25
25.6
EDICT
0.30
NULL Text
DDIM
AIDI (fixed-point)
0.35
Better
10
20
30
40
50
60
0.26
0.28
0.30 0.32 0.34
Foowwadd Time [sec]
CLIP Score
Agreement with prompt
Figure E1: (Left) Inversion Results: Mean reconstruction quality (y-axis, PSNR) and runtime (X-axis, seconds) on the COC02017 validation set. Our method achieves high PSNR while reduc- ing inversion-reconstrutiion time by factor of x4 x14g(Right) Evaluation of editing perfor- mance: GNRI achieves superior CLIP and LPIPS scores, indicating better compliance with text prompts and higher structure preservation.
F SEED EXPLORATION & RARE CONCEPT GENERATION
Input Images
ReNoise
GNRI (Ours)
Tiger
Cat
Oxygen mask
pay-
phone
Patas Monkey
Figure F1: Generating rare concepts based on few examples with the method of (Samuel et al.) 2023) that heavily depends on difftusion-inveriion quality. In our comparison, we evaluate GNRI alongside ReNoise (refer to the discussion in the main paper). ReNoise frequently struggles to produce realistic representation of certain objects (such as Oxygen-mask or Patas Monkey) or an accurate depiction of specific concepts (like Tiger-cat Pay-phone). However, the of GNRI rectifies these issues in the results. For detailed quantitative comparison, please refer to the main paper. See the main paper for quantitative comparison.
Figure STi further displays results for the rare-concept generation task introduced in (Samuel al. 2024 2023). The objective is to enable the diffusion model to generate concepts rarely seen in its training by using few images of that concept. Both methods in (Samuel et al. 2024 2023)
utilize diffusion inversion for this purpose. In the main paper, we presented experiments showcasing
the impact of new inversion process on the outcomes of (Samuel al. 2024 2023).
16

---

Published as a conference paper at ICLR 2025
ReNoise (fixed-point)
ExactDPM
Input Image
SDEdit
EditFriendly
(gradient-descent) GNRI (OURS)
Changing object
cat tiger
Adding object
girl sitting
Changing Pose
dead + blooming tree
Changing Baokground
+ the garden
Qualitative comparison for image editing on the PlE-benchmark. Illustrates how al- ternative approaches may struggle to preserve the original structure or fail to edit the image, while GNRI succeeds in editing the image properly.
Figure F2:
We provide qualitative results based on NAO (Samuel et al., 2023 centroid computation when utilizing GNRI, contrasting them with those obtained through DDIM inversions. The illustrations demonstrate that GNRI is capable of identifying high-quality centroids with correct semantic content compared to centroids found by DDIM.
G ADDITIONAL QUALITATIVE RESULTS ON PIE-BENCHMARK
Additional quantitative results the PlE-Benchmark. Figure F2 provides quantitative results on the PIE-Benchmark using Flux.1. I. highlights how alternative methods often fail to maintain the original structure miss key elements specified in the target prompt, whereas GNRI effectively achieves accurate and proper image editing.
Quantitative results with Stable Diffusion 2.1. We present a quantitative comparison between
GNRI and baseline methods, utilizing latent diffusion SD2.1 (Rombach al. 2022). Figure ET aa illustrates the image petession-reoostruucco performance, evaluated using LPIPS and CLIP-score metrics. The results demonstrate that GNRI surpasses all other inversion methods in terms of reconstruction quality, which subsequently im- proves editing performance.
performance, while Figure El(b) highlights the editing
17

---

Published as a conference paper at ICLR 2025
Input Image
Null Text
EDICT
AIDI
Ours
"Wooden cutting board with a siice of piaza bread"
"An angry man with puffy hair"
lady with bananat oranges on her head"
"People walking on sidewalk with yaHlol red hydrant"
"A dog pants next to bowl of water food"
"An area cake on table"
"A plate with a.cake hat"
Qualitative comparison for image editing: Illustrates how alternative approaches may struggle to preserve the original structure overlook crucial components specified in the target prompt, while GNRI succeeds in editing the image properly.
Figure F3:
18

---

Published as a conference paper at ICLR 2025
Input Caption: sitting chair"
Input Image
cat carafi
cat dog
ca! boy
with sunglasses
chair sofa
Input Caption: "An angry man
Input Image
angry - happy with army shirt
with helmet
with joker mask
with silky hair
Figure F4: Various editing with same input: Note the GNRI capability in both subtle and extensive changes would expect from the particular prompt change.
Table HI: PSNR vs. NFE results using SDXL. Turbo on the COCO validation set. Our method achieves the highest PSNR with the fewest NFEs, compared to other methods.
PSNR
NFE
ReNoise Garibi et al. (2024)
18.1
36
ExactDPM Hong al. (0224)
20.0
40
TurboEdit Wu et al. (2024)
22
8
GNRI (Ours)
23.9
Table H2: Inversion comparison with Edit Friendly on SDXL Turbo (COCO validation set). While DDPM achieves higher PSNR through stochastic inversion, our deterministic approach (GNRI) pro- vides competitive results with significantly faster inversion time (0.521s 32.3s).
PSNR
Inv Time
EditFriendly Huberman et al. (0023
26.1
32.3
GNRT (Ours)
23.9
0.521
H ADDITIONAL QUALITATIVE AND QUANTITATIVE RESULTS -COMPARISON TO BASELINES
In this section, we present additional qualitative comparisons involving GNRT and baseline methods.
Figures F3 provide further comparisons for real-image editing, using SD2.1. These figures illustrate how alternative approaches may struggle to preserve the original structure or overlook crucial com- ponents specified in the target prompt. For example, n the first row of Figure F3 GNRI correctly transforms the pizza into bread, while other methods generate bad looking bread. Row three shows an example of GNRI success (bananas to oranges), where other alternatives totally fail.
Figure shows the outcomes of various edits the same image, providing additional confirmation that inversion with GNRI yields modification the pertinent parts in the image while maintaining the original structure.
Table HI provides PSNR vs NFE (Neural Function Evaluations) results using SDXL Turbo on the COCO validation set. Our method requires 6 NFEs, calculated 4 steps with an average of 1.5 NR iterations per step (some steps needing iteration, others 2). As shown the table below, our approach achieves the highest PSNR while using the fewest NFEs compared to other methods.
19

---

Published as a conference paper at ICLR 2025
Table II:
Comparison of Anderson Acceleration and the Guidance term for out-of-distribution COCO samples. A higher likelihood is better.
mean likelihood q(2x
PSNR
Inv Time
Anderson Acceleration Guidance Regularizer (Ours)
0.34
18.3
Inf 0.4
0.82
25.8
Table J1: Memory usage and convergence time for different methods A100 and RTX 3090Ti.
A100 (VRAM 40GB)
RTX 3090Ti (VRAM 24GB)
Memory
Inv Time
Memory
Inv Time
AIDI Pan et al. (2023)
15GB
sec
16 GB
13 sec
ExactDPM Hong al. (2024)
30GB
17 see
20 GB
25
GNRI (Ours)
28GB
0.4 sec
19 GB
I sec
Table H2 compares inversion performance with Edit Friendly Huberman al. (023) using SDXL, Turbo on the COCO validation set. Edit Friendly achieves the maximum PSNR possible given the VAE distortion, because it does stochastic DDPM inversion, saving additional noise matrices at every step and "overfitting" the inverted image. As a result, it requires extensive inversion time (more than 30 seconds), whereas our GNRI reaches a relatively high PSNR in under a second. Our approach solves deterministic inversion (DDIM Euler) making it suitable for applications like interpolation and rare concept generation, where DDPM is not feasible.
I COMPARISON TO ANDERSON ACCELERATION
The guiding term introduced in Sec 4.2 might appear similar to Anderson acceleration used in Pan ct al. (2023), which aims to allow more gradual changes from previous estimates. However, One key difference is that guidance term introduced in (9) does pull 2, toward 2t-1 but towards the mean 14, By that, it directs the Newton -aatonnRaphson method toward solutions having a high likelihood, using "side-information" about the noise. In contrast, Anderson acceleration relies on previous values of the implicit function f(z;) and residuals but lacks awareness of the latent distribution (see for example, line 13 in Algorithm of (Pan al. 2023)).
In GNRI, keeping close to encourages solutions that are more "in-distribution" and aligned with the diffusion model. In other words, if 2t-1 starts far -utto--distiiuution, keeping 2, close to 2t-1 is likely to yield worse solutions compared to guiding 2t toward inputs that the model expects receive.
We illustrate this effect in table [1] We analyzed the inversion of 300 COCO images that the model would consider be non-typical by selecting the samples with the lowest likelihood under ((²)) (I). The table shows that GNRI drives the latents toward significantly higher likelihood solu- tions (mean (=)))=+-)) leading to substantially better reconstructions (higher PSNR) than Ander- acceleration.
More broadly, our guidance term could potentially be applied to other methods, such as exed-point iteration approaches like those proposed by Pan et al. It would also be interesting to find methods that can include more domain knowledge into the inversion process.
J MEMORY USAGE
We compared the memory usage of our approach, with AIDI Pan al. (2023) and ExactDPM (Zhang et al. (2023) on a A100 GPU (40 GB VRAM) and a RTX 3090Ti GPU (24GB VRAM). Results given in table JT Our approach consumes about the same GPU memory as ExactDPM, however, it converges faster about x43 times than ExactDPM on an A100 GPU, and x25 faster on an RTX 3090 GPU.
20

---

Published as a conference paper at ICLR 2025
K MULTI OBJECT EDITING
We conducted several qualitative experiments to evaluate the capabilities of our method in editing multiple objects simultaneously. Our results demonstrate that the method can effectively handle complex scenes, allowing for the independent manipulation of several objects without compromising the quality of individual edits. As illustrated in Figure KT our GNRI model successfully performs multi-object transformations while maintaining scene coherence and natural composition from converting multiple dogs to cats while preserving the scene layout, to simultaneously transforming different food items (steak to apple, eggs to pancakes) on a plate. These results underscore GNRI's robustness in handling multiple concurrent edits while maintaining high visual quality and semantic coherence across the transformed scenes.
Input Image
GNRI (OURS)
dogs > cats
steak + apple eggs > pancakes
Figure KI: Multi-object editing capabilities of GNRI. Top: Simultaneous transformation of multiple dogs into cats. Bottom: Concurrent food item conversion on a plate (steak-vapple eggs Spancakes) demonstrating preservation of scene composition and context.
21


Reviews:
[{"review_id": "5HhjIisKEh", "reviewer": "Reviewer_8MQs", "summary": "This paper addresses diffusion inversion for fast and accurate inversion within text-to-image diffusion models. The authors propose Guided Newton-Raphson Inversion (GNRI), which leverages the Newton-Raphson (NR) numerical root-finding technique to achieve efficient and high-quality inversion. By constructing a modified DDIM inversion function as the target for NR and adding a guiding term, GNRI ensures convergence within the distribution of the latent space, overcoming common challenges in achieving accurate reconstructions. This approach supports tasks like real-time image editing, image interpolation, and rare concept generation, achieving significant speedups. The authors show that GNRI could invert images within 0.4 seconds on an A100 GPU while maintaining high fidelity, compared with popular diffusion models.", "questions": "The objective function $F(Z_t)$ in Eq. 9, which combines the residual term $r(z_t)$ and the regularization $G(z_t)$ raises questions about its role in achieving exact root-finding. The two terms in Eq. 9 seem to contradict each other, because a solution to $r(z_t) = 0$ will yield a $G(z_t) > 0$, which further leads to $F(z_t) > 0$ if $r(z_t) = 0$. This suggests that the method solves an approximation of the original inversion problem rather than the exact roots.\n\nIn Fig. 3(a), it is observed that GNRI achieves a lower residual than the non-guided NR inversion (NRI), which is unexpected since NRI has only one objective $r(z_t) = 0$. Why does GNRI, despite its additional regularization term, achieve a better residual than NRI? What is the residual level if you set $z_t = mu_t (G(z_t) = 0)$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: This paper introduces a novel approach to the diffusion inversion problem by leveraging the Newton-Raphson (NR) root-finding method with two modifications: (1) dimensionality reduction via a scalar formulation of the Jacobian and (2) a guiding regularization term. By reframing inversion as a fixed-point iteration (FPI) problem with an $L_1$ norm relaxation, this work combines numerical optimization techniques with diffusion models, addressing limitations in accuracy and speed seen in prior methods. The addition of the regularization term prevents out-of-distribution solutions.\n\nQuality: The methodological foundation is rigorous, with well-defined derivations and justifications for each modification to the NR method. The paper also provides extensive empirical validation across multiple tasks, including image reconstruction, editing, interpolation, and rare concept generation. The inversion time to 0.4 seconds on an A100 GPU is impressive.\n\nClarity: The paper is generally clearly written and accessible. \n\nSignificance: GNRI addresses the problem of achieving efficient and accurate image inversion in text-to-image diffusion models, which is central to tasks like real-time image editing and rare concept generation. The experimental results across models (e.g., Stable Diffusion, SDXL-Turbo, Flux) demonstrate GNRI’s versatility, with the potential for broad impact in real-time generative model applications.", "weaknesses": "The concept of utilizing FPI methods in diffusion inversion is not completely new (like AIDI, ReNoise).\n\nFigure 3 compares the performance of the method with an FPI baseline, which is not specified as the author mentioned two baseline FPI methods (AIDI, ReNoise).\n\nThere is a notational inconsistency in referencing Eq. 5 instead of Eq. 4 when describing the fixed-point function $f(z)$, which could lead to confusion regarding the intended inversion function. This typo gives the impression that the method still relies on the DDIM-inversion approximation. Please clarify these notations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cZYYiOPnaL", "reviewer": "Reviewer_Rr79", "summary": "To improve the accuracy of DDIM inversion approximations, previous studies, such as ReNoise, have utilized fixed-point iteration. This paper introduces a specialized fixed-point iteration known as Newton-Raphson (NR), which enables faster convergence. Because a straightforward application of the NR method is infeasible for high-dimensional points, this paper reduces dimensionality by applying the L1​-norm to the fixed-point function, transforming the Jacobian into a vector. Additionally, to prevent convergence to an off-manifold fixed point, a proximal term is added to the fixed-point function, steering the solution $z_t$ toward the mean of the forward distribution $q_t(z_t|z_{t-1})$ This complete framework is termed Guided Newton-Raphson Inversion (GNRI).", "questions": "- Equation (10) requires one forward pass and one back-propagation through diffusion model (UNet). In fact, with the same number of fixed-point iterations, this requires additional cost for back-propagation compared to ReNoise. However, in Figure 4 and Table 1, the reported efficiency of the GNRI is almost same as the original DDIM and much better than ReNoise. Is there some points that the reviewer is missing?\n- Efficiency analysis (Figure 4):  Could author provide PSNR vs NFE? It would be helpful to further understand the efficiency of inverse methods.\n- Is this method also capable of multiple-object editing?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The motivation and proposed method is clearly stated.\n- As the NR provides faster convergence than naive fixed-point iteration method, the proposed method achieves high-efficiency compared with SOTA inversion methods.\n- Combined with SDXL-turbo and Flux, it enables real-time image editing.\n- Provided experimental results well demonstrate the effectiveness of the proposed method.", "weaknesses": "- The reviewer does not find significant weaknesses\n- Minor: Linebreak in Figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qfEIdr3dLB", "reviewer": "Reviewer_xAUW", "summary": "This paper introduces an efficient method for accelerating image inversion in diffusion models by refining the Newton-Raphson (NR) approach. While a direct application of NR is computationally intense due to the need for Jacobian computation, the authors optimize it with dimensionality reduction using the L1 norm and incorporate a regularization term to ensure the noise latent conforms to the original distribution of the diffusion model. Experiments on models like SDXL-Turbo and Flux showcase both the speed and quality of inversion, and additional tests demonstrate the method’s capabilities in image editing, interpolation, and generating rare objects.", "questions": "I suggest the authors to include the experiments for the DDPM-Inv for both reconstruction and editing. This helps to understand the difference between using one trajectory vs sample multiple trajectory and record the noise(difference) and help us to determine which method is better. \nAlso please include more visualization results for the editing tasks. This helps us to understand the difference of different methods visually.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper introduces an optimized approach to inversion in diffusion models, significantly accelerating the process. With inversion times as low as 0.4 seconds on an A100 GPU, this method is fast enough to enable interactive applications, a notable advancement over existing techniques.\n2. By incorporating an L1 norm for dimensionality reduction and a regularization term to ensure conformity with the diffusion model's distribution, the method maintains high reconstruction quality, avoiding common issues like out-of-distribution errors in the generated latents.", "weaknesses": "1. Missing comparison with DDPM Inversion (An Edit Friendly DDPM Noise Space: Inversion and Manipulations) for the inversion performance in both quality and speed. Also, please include the DDPM-Inv result on PIE-Bench for image editing. \n2. More visualization comparison of the proposed method against other methods for editing tasks. Includes more editing results from the PIE-Benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C2WU4VZNiv", "reviewer": "Reviewer_TzFZ", "summary": "This paper introduces a new inversion method for image editing in T2I DMs. It uses the Newton-Raphson (NR) method to improve the speed and accuracy of the inversion process. The authors present two main contributions: applying a norm to the NR method (Sec 4.1) and using an anchoring technique (Sec 4.2). Experimental results show that they can edit real images lightning-fast.", "questions": "1. In L238, should it reference Eq. 4 instead of Eq. 5?\n2. Could you provide more results for the rare concept generation?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Innovative Approach**: The use of the Newton-Raphson method helps overcome common problems found in other iterative inversion techniques (AIDI, ReNoise, ExactDPM), making the process more efficient.\n   \n2. **Comprehensive Experiments**: The paper provides experiments on three practical tasks, showing that GNRI well compared to existing methods, especially in terms of speed.", "weaknesses": "1. **Questionable Contribution**: Although the idea of employing NR seems innovative, the contribution in Sec 4.1 about applying the norm seems standard and not very innovative. Optimizing norms is a common practice in optimization. For example, on page 9 of ReNoise, the convergence analysis is performed using the l2-norm, and ExactDPM also applies gradient descent to the l2-norm. Actually, anyone who tries to apply Newton method would do the same thing naturally, because calculating Jacobian is impossible while calculating the gradient of the *norm* is very easy in PyTorch. Therefore, this point does not stand out.\n\n2. **Similarity to Other Methods**: The guiding regularizer introduced in Sec 4.2 appears similar to Pan et al.'s Anderson acceleration, which also aims to prevent large changes from previous estimates. This similarity raises questions about the originality of the approach. On the contrary, the significant drop in performance when $\\lambda=0$ in Figure C2 can be seen as an indication of the instability of NRI.\n\n3. **Memory Usage**: The Newton-Raphson method needs gradients, which can lead to high memory use. (It must be using torch.grad, I guess.) Other methods, like AIDI and the forward step method in ExactDPM, do not rely on gradients and may be more efficient, in terms of memory usage. ExactDPM actually used a smaller GPU than A100.\n\n4. **Some writings are confusing**: The description of $\\mathcal{F}$ is confusing because it is defined in two different ways: as a function from $\\mathbb{R}^D \\to \\mathbb{R}^D$ in L203 and again as a function from $\\mathbb{R}^D \\to \\mathbb{R}^+$ in Eq (9). This inconsistency can mislead readers. Also, in line 577, the citation for Hong et al. is duplicated. In line 637, it should be noted that EDICT was presented at CVPR *2023*, not 2022.\n\n5. **Lack of Results for Rare Concept Generation**: The paper does not provide strong quantitative results for generating rare concepts. For example, Table 3 does not show significant improvements over existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)