Noise Map Guidance: Inversion with Spatial Context for Real Image Editing
Authors: Hansam Cho
Jonghyun Lee
Seoung Bum Kim
Tae-Hyun Oh
Yonghyun Jeong
Venue: ICLR 2024 poster (2024)
Keywords: Generative Models
Diffusion Models
Image Editing
Primary area: generative models

Abstract: Text-guided diffusion models have become a popular tool in image synthesis, known for producing high-quality and diverse images. However, their application to editing real images often encounters hurdles primarily due to the text condition deteriorating the reconstruction quality and subsequently affecting editing fidelity. Null-text Inversion (NTI) has made strides in this area, but it fails to capture spatial context and requires computationally intensive per-timestep optimization. Addressing these challenges, we present Noise Map Guidance (NMG), an inversion method rich in a spatial context, tailored for real-image editing. Significantly, NMG achieves this without necessitating optimization, yet preserves the editing quality. Our empirical investigations highlight NMG's adaptability across various editing techniques and its robustness to variants of DDIM inversions.

Published as a conference paper at ICLR 2024
NOISE MAP GUIDANCE: INVERSION WITH SPATIAL CONTEXT FOR REAL IMAGE EDITING
Hansam Cho Cho Jonghyun , Seoung Bum Kim', Tae-Hyun Oh Yonghyun Jeong"t 'School of Industrial and Management Engineering, Korea University, "NAVER Cloud DDept. of Electrical Engineering and Grad. School of Artificial Intelligence, POSTECH
"Institute for Convergence Research and Education in Advanced Technology, Yonsei University (chosam95, tomtoml 103, sbkiml J fokorea ac.kr (taehyun) @postech. ac. kr, (yonghyun. jeong @mavercorp..com com
ABSTRACT
Text-guided diffusion models have become a popular tool in image synthesis, known for producing high-quality and diverse images. However, their application to editing real images often encounters hurdles primarily due the text condition deteriorating the reconstruction quality and subsequently affecting editing fidelity. Null-text Inversion (NTI) has made strides this area, but fails to capture spatial context and requires computationally intensive per-timestep optimization. Address- ing these challenges, we present NOISE MAP GUIDANCE (NMG), an inversion method rich in spatial context, tailored for real real-image editing. Significantly, NMG achieves this without necessitating optimization, yet preserves the editing quality. Our empirical investigations highlight NMG's s adaptability across various editing techniques and its robustness to variants of DDIM inversions.
Input
DDIM
NTI
NPI
ProxNPI
NMG(Ours)
Local Fratinn
"a young man standing in front building, smiling"
Global Editing
"a large elk standing on top of lush green field, painting"
(a) Local and global editing with Prompt-to-Pront
Vieepoint Altemation
woman with her eyes closed, frontal view" (b) Viewpoint alternation with MasaCtrl
Cal Dog
(c) Zero-shot mmage-to-imager translation with pix2pix-zero
Figure 1: Compared to other inversion methods, NMG (a) demonstrates high fidelity editing when paired with Prompt-to-Promet
(b) successfully conducts viewpoint alteration via MasaCtrl, and (c) preserves the spatial context of the input image while performing zero-shot image-to-image translation with pix2pix-zera zero. Text prompt corresponding to each input image is presented beneath each sample, with words introduced for image editing distinctly highlighted in green.
First Author. Work done during an internship at NAVER Cloud. "Co-correppondig authors.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Text-guided diffusion models (Rombach et al. 2022 Saharia et al., 2022 Ramesh et al. 2022 Chang iet al. 2023 Podell et al. 2023) have recently emerged as a powerful tool in image synthesis, widely adapted for their ability to generate images of exceptional visual quality and diversity. Owing their impressive performance, numerous studies have leveraged these models for image editing (Hertz
al. 2022 Cao al. 2023 Parmar al. 2023 Epstein et 2023). While these models excel in editing synthesized images, they often produce sub-par results when editing real images. [eertz et al.
(2022) point out that this challenge mostly originates from the reliance on classifier frresffeer-freeeeeeee guidance
(CFG) (Ho& & Salimans| (021), a method ubiquitously adopted to increase fidelity
Editing real images within a diffusion framework follows a twofold approach. Firstly, an image is inverted into latent variable via DDIM inversion. This latent variable is then channeled into two paths: reconstruction and editing. While the editing path modifies the latent towards the desired outcome, it also integrates information from the reconstruction path to retain the original identity of the image, As a result, the quality of the edit is highly dependent on the reconstruction path. However, extrapolating towards the condition via CFG introduces errors in each timestep. This deviation pushes the reconstruction path away from the DDIM inversion trajectory, hindering accurate reconstruction and ultimately diminishing the editing fidelity.
Addressing this limitation, Mokady et al. (203)) propose Null-text Inversion (NTI) leading to notable achievements in real image editing. This approach optimizes the null-text embedding used in CFG on a per-timestep basis, correcting the reconstruction path to the desired DDIM inversion trajectory. By leveraging the optimized null-text embedding and integrating it with Prompt-to-Promm (Hertz et al. 2022) editing, NTI conducts real image editing. While NTI offers certain benefits, its use of per-timestep optimization can lead to increased computational demands in practical applications. As solution to this inteeintessvve optimization method, NPI (Miyake et al. (2023) and ProxNPI (Han et al. 2023) attempt to approximate the optimized null-text embedding without optimization. Although these methods succeed in correcting the reconstruction path, they often struggle to capture the spatial context in the input images.
To address these challenges, we introduce NOISE MAP GUIDANCE (NMG), an inversion methodology enriched with spatial context for real image editing. To capture spatial context, we employ the latent variables from DDIM inversion, which we refer to as noise maps. These noise maps, essentially noisy representations of images, inherently encapsulate the spatial context. To eliminate the need for optimization phase, we condition the noise maps to the reverse process. Rather than solely depending on text embeddings for image editing, our methodology harnesses both noise maps and text embeddings, drawing from their spatial and semantic guidance to perform faithful editing. To accommodate our dual-conditioning strategy, we reformulate the reverse process by leveraging the guidance technique proposed by Zhao al. (2022).
Our experimental results highlight NMG's capacity to preserve the spatial context of the input image during real image editing. Figure [ (a) and (c) reveal that NTI, NPI, and ProxNPI often struggle to capture the spatial context of the input image. Furthermore, Figure I (b) highlights a scenario where spatial context is essential for effective editing, and in such context, NMG consistently outperforms other methods. By utilizing guidance techniques of diffusion models, our optimization-free method achieves speeds substantially faster than NTI without compromising editing quality. The versatility of NMG is further emphasized by its integration with various editing techniques. e.g., Prompt-to- Prompt (Hertz et al., 2002),, MasaCtrl (Cao et al., 2023) and pix2pix -zero (Parmar et al., 2023), each grounded in an inversion methodology. Moreover, we demonstrate NMG's resilience variations of DDIM inversion. Our main contributions are summarized as follows:
We present NOISE MAP GUIDANCE (NMG), an inversion method rich in spatial context, specifically tailored for real-image editing.
Although formulated as an optimization-free method, we show that NMG maintains editing quality without compromise, even achieving superior performance over competing methods.
We demonstrate NMG's adaptability by combining it with different editing methodologies and by highlighting its consistent robustness across variations of DDIM inversion.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Inversion with Diffusion Models The initial stage of image editing typically consists of encoding the input image into a latent space, referred to as inversion. DDIM (Song et al., 2020a) first introduces the concept of inversion within diffusion models by formulating the sampling process differential equation (ODE) and inverting it to recover a latent noise as a starting point for the sampling process. However, the precision of DDIM's ' reconstruction deteriorates when being integrated with a text-guided diffusion model (Hertz et al., 2022). In response, recent studies propose the addition different inversion framework (Huberman- Spiegelglas et al., 2023). Recently, Mokady et al. (2023) proposes Null-text Inversion (NTI), which tailors inversion techniques specifically for text-guided diffusion models. NTI refines the null-text embedding used in text-guided diffusion models, achieving accurate image reconstruction. However, the intensive computation required for optimization constrains its broader applicability. To mitigate this drawback, Negative-pron -prompt Inversion (NPI) (Miyake et al.) 2023) approximates the optimized null-text embedding to achieve optimization free inversion, albeit with compromised performance relative to NTI. Subsequently, |Han et al. (2023) incorporate proximal guidance into NPI, enhancing its quality. Like NPI, we introduce an optimization inversion technique, but uniquely, our approach robustly maintains the spatial context of the original image.
ordinary
of auxiliary diffusion state (Wallace et |l.| 2023)
Editing with Inversion methods Despite the emergence of numerous studies in the field of editing via diffusion models, they often encounter challenges including the need for extra training (Kawar iet al. 2023 Valevski et al. (2023; Bar- et al., (0022). or the incorporation of additional conditional signals (Avrahami et al., 2022] 2023 Nichol et al. 2022 Rombach al. 2022). Recently, inversion- F023/Parrar
based editing methods (Hertz et al. 2022 Cao et al. 2023] Parmar et al. (023) have shown promising results that do not require additional training or conditional signals. These methods typically involve two parallel phases: a reconstruction sequence and editing sequence. In the reconstruction sequence, critical information from the input image is extracted and fed into the editing sequence for manipulation. Notably, Prompt-to-Pronee (Hertz al. (2022) leverages eross-attention maps to guide the editing sequence, while MasaCtrl (Cao et al. (2023) introduces mutual self-attention to facilitate non-rigid editing. pix2pix-zero (Parmar al. (002) utilizes cross-attention map guidance to perform zero-shot image-to-ime reconstruction path is crucial for reliable image editing, the efficiency of these methods heavily relies on the performance of the inversion methods they use. By integrating our inversion method with existing inversion-basee editing methods, we demonstrate significant improvement in preserving the spatial context of input images.
translation. Because obtaining precise information from the
3 METHOD
3.1 BACKGROUND
Text-guided Diffusion Model Text-guided diffusion models (Rombach et al.) 2022 are designed to map random Gaussian noise vector ær into an image zo while aligning with the given text condition CT. typically text embeddings derived from text encoders like CLIP (Radford et al. 202I). This is achieved through sequential denoising operation, commonly termed as the reverse process. This process is driven by a noise prediction network co, which is optimized by the loss:
Lsimple = N.o.
co(2,, t, cr)
Although eg is conditioned on the timestep t, denoted as ea (Zz, CT). we omit the timestep condition for the output of the network as ça (2,, cr) for brevity. Text-guided diffusion models typically utilize
classifier-free guidance (CFG) (Ho & Salimans 2021 incorporate text conditions during image
generation. CFG is represented as follows:
Eø(z,, (T) Eo (z,, 0) + w- (ee (Zi, cr) Eg (ze, 0)).
(2)
Here, w signifies the text guidance scale (controlling the impact of the text condition), and @ denotes the null text embedding (the embedding vector for a null-text """).
DDIM Inversion To edit a real image in the framework of diffusion models, image firstly has to be converted into a latent variable 2;, a process known as inversion. Fundamentally, the latent variable 2/. is the original starting noise that reconstructs into the image when denoised. To invert an
3

---

Published as a conference paper at ICLR 2024
2y-1
71-1
DDIM Inversion
Zy-1
Inpu: Image
Pp-1
Reverse Process (eccnstrrucioo))
Reconstruction (wlo alignment)
Reconstruction (w alignment)
(a) Image Reconstruction
Noise Map 40-1
Noise M
2
"a fire hydrant
DM
fire hyyrannt
I1
Vil-i-
Ze
DM
DM
DM
tuning
Noise Map Condition
condition
(b) Noll-text Inversion
(c) Noise Map G Guidance
(0 (Optimization)
(Condition)
Figure 2: As seen in (a), naive reconstruction often fails due to the reconstruction path diverging from the original inversion path. Achieving reliable reconstruction necessitates realigning the recon- struction path with the inversion path. As depicted in (b), NTI achieves this alignment by optimizing the null-text embedding, thereby reducing the error between the inversion and reconstruction paths. Conversely, NMG, as shown in (c), conditions the reconstruction process based on the divergence between the two paths, leveraging this variance to refine the reconstruction path.
image into its latent variable, DDIM inversion (Song et al. 2020a) is predominantly utilized. DDIM inversion is derived from the reverse process of DDIM that is formulated as:
va--1 (V a:
Zt-1
cØ c0(2,,CT)
a
with {ar]T= as predefined noise schedule. Because this DDIM reverse process can be formulated as an ordinary differential equation (ODE), the DDIM inversion process can be obtained by reversing said ODE as:
- 11 eg (Zt, CT).
Z1+1
Ce
2t
Vas+1
ar+1
Null-text Inversion Inversion- editing methods follow a twofold approach. First, an image is converted into its latent variable through inversion. This is followed by a simultaneous two-phase procedure: reconstruction and editing. While the editing phase alters the latent variable towards the desired modification, it concurrently draws on information from the reconstruction phase to maintain crucial the image's characteristics. During editing, a substantial CFG guidance value of w for generating high-fidelity images. However, the extrapolation toward the condition introduces errors at every timestep. illustrated Figure (a), large CFG guidance scale misguides reconstruction path away from the inversion path. This divergence Ieads to imprecise reconstructed image, subsequently degrading the final quality of the edited image. To address this, Null-text Inversion (NTT) (Mokady et al. 2023) optimizes null-text embeddings used in Eq. to align with the DDIM inversion trajectory. Initially, DDIM inversion is performed with w = to compute the latent variables {zī 1 termed the noise maps. As the reverse process progresses from to t 1, Z;- is derived from 2, with Eq. 2 and Eq,
Ct-1 a.
Vai-1
(V-L-1
V
ën (0(2,, er).
(5)
a-1
With both and single noise map 2|_1 available, the null-text embedding 0, is optimized every step using the loss function:
min - 2 0.
(6)
By optimizing the null-text embeddings, NTI reduces the error of the reconstruction path. Overall process of NTI is depicted in Figure (b). However, since the null-text embedding is represented as a single-dimensional vector in Rdd it struggles capture the image's 's spatial context.
4

---

Published as a conference paper at ICLR 2024
3.2 NOISE MAP GUIDANCE
Unlike null-text embeddings, noise maps {zī 1== naturally capture the spatial context of the input image. This is due the fact that noise maps originate from infusing a small amount of noise into the input image and have the same spatial dimensions as the input image. Leveraging this attractive trait, we directly employ noise maps to preserve the spatial context of an image. As part of this approach, we condition the reverse process on noise maps. Given that current text-guided diffusion models are conditioned solely on text embeddings, our initial step is to reformulate the reverse process to account for both text and noise maps.
To reformulate the reverse process to conditional form, we introduce the score function Vz, log p(z)) with the relation (zz) -v1 a V x, p(z). By replacing V., log p(z,) with V s. log pl 2 (c), we enable the network to produce outputs based on specific conditions, leading to the equation ar Zz, log pl p(z)(c). Applying Bayes's rule, Zz, log p(z,(c) Vz, ((z) =z, log ze), the network's 's conditional output is then formulated as:
<o <0(2), c)
co(2,, c) ---I
log p(z,) + 2z, log p(clz;)
(7)
In this context, V lo p( c| 21 is erucial component to condition the diffusion model. We introduce energy guidance (Zhao et al. 2022), which serves as a flexible conditioning format in our formulation. With energy guidance, Eq. 7 is revised as follows:
E0(
/1 ar <v. logp(zz) E c,
(8)
NTI Mokady et 2023) performs one-step denoising, transitioning from 2e to 2t-1. and compares the noise map - with latent variable 2 for optimization. To align with this process, we define the energy function E E(z, c, t) = || - 1l to condition the noise map for each timestep. Note that unlike NTI, we employ the LI distance to compute the distance between the noise map and the current latent variable. Based on the definition of the energy function, Eq, Bis revised as:
(o(2,,CN) V ail logp(zz) - 89 V 11 2 l11)
where 2111 Dr ze + va-- (V a--1 (V
11111) c c0(z),0)
We term the noise map eN conditioned to produce the network output as co (2 CN) in Eq. Addi- tionally, we introduce a hyperparameter 8y termed the gradient scale in Eq. 9 compensate for the scale difference between the gradient and the network's output. By adjusting the magnitude of sy, we modulate the degree of the edit from the original reverse process to the DDIM inversion trajectory. Building on the findings that substantial guidance scale can yield high-quality results (Dhariwal & Nichol 2021 I Ho & Salimans 2021). we also introduce a guidance technique for noise map condition similar Eq. as follows:
Co(2,, eN) co (z), + SN (co(ze, CN) ca (z,, 0)).
(10)
where en the guidance scale of the noise map. To derive a latent variable conditioned on the noise map, we execute one-step reverse process with Eg (2,,, cx) as follows:
Va-- (V
1 V
Es (Z),
(11)
By performing a text-conditioned reverse process with ZNM latent variable conditioned noise map, we derive our final latent variable that is conditioned on both the noise map and the text condition. Empirically, we find that can approximate N;Nad 2N² Thus, latent 2,_1 both
conditioned on noise map and text embedding is determined as follows:
Eo (2. NM Cr) co (2,N .00 ST (co(2 N. .er)
co NM (0)
(12)
2:-1
NM a--1 Cte
zt
+ Var-1 (V (VLL-11
Eg (z, NM
.TT).
(13)
To maintain consistent notation conventions with s.y in Eq. 10 we designate sy to represent the text guidance scale, instead of w in Eq. 2 In Figure (c), we display the overall process of NMG. We note that NMG is a sequential process of first conditioning the noise map and conditioning the text embedding on the outcome of the step before.
5

---

Published as a conference paper at ICLR 2024
Input
DDIM
NTI
NPI
ProxNPI
NMG(O wrs)
Siveppan
"a eat tiger sitting table next vase of flowers"
Contextual Altemation
"a woman the forest desert" (a) Local editing with Prompt-to-Promp
Van Gogh
"a woman gray dress dancing, van gogh style"
Pointillism
"a man with dreadlocks standing in front ofa building, pointillism"
(b) Global editing with Prompt-to-Prome
Figure 3: Image editing results using Prompt-to-Pronet Prompt are shown in (a) for local editing and (b) for global editing. Results show that DDIM lacks in preserving details of the input image, both NTI and NPI face challenges in maintaining spatial context, and ProxNPI exhibits limited editing capabilities. In contrast, NMG consistently produces robust results for both local and global edits.
EXPERIMENTS
Our method is tailored ensure that the reconstruction path remains closely aligned with the inversion trajectory throughout the image editing process. In the section below, we compare NMG with several methods, including DDIM (Song et al. 2020a). NTI (Mokady al. 2023). NPI (Miyake et al. 2023). and ProxNPI (Han et al., 2023). While DDIM is not inherently formulated to prevent reconstruction divergence, we include it in our comparisons to highlight how image quality deteriorates when the reconstruction path strays from the desired trajectory.
4.1 QUALITATIVE COMPARISON
Given that the primary goal inversion methods is image editing, first integrate NMG into the widely adopted Prompt-to-Prne Prompt (Hertz et al. (022) editing method to empirically validate our approach. For the integration with Hooppttoo-Propt. we incorporate NMG within the reconstruction path, rather than the editing path. This approach stems from NMG's characteristic of aligning pathway with the inversion trajectory. Such an alignment can detrimental the image's editing.
Prompt-to-Pronet performs diverse tasks by controlling attention maps from reconstruction and editing paths. Our experiments encompass four local editing tasks: object swapping, contextual alteration, facial attribute editing, and color change, well as four global style transfer tasks to styles of oil paintings, van Gogh, pointillism, and watercolor. Figures B depict our results for both local and global editing using Prompt-to- to-mropt.. Due to its limited reconstruction capability, DDIM struggles to integrate details of the input image into the edited image. Both NTI and NPI leverage null-text embeddings retain input image details, but since the null-text embedding is one-dimensional

---

Published as a conference paper at ICLR 2024
Input
DDIM
NTI
NPI
ProxNPI
NMG
Pose Modification
"a small kitten is playing laying with flower"
Viewpoint Altemation
"a close up of small toy mario, side view"
(a) MasaCtri
Cat - Dog
Dog Cat
(b) pix2pix-eero
Figure 4: Image editing outcomes are presented using (a) MasaCtrl and (b) pix2pix-zero. NMG's proficiency in retaining spatial context is highlighted in (a), while its resilience to variations of DDIM inversion is showeased in (b).
vector, it inherently struggles to preserve spatial context. ProxNPI, due to the inversion guidance, utilizes the noise maps follow an inversion trajectory and excels retaining spatial context. However, this guidance is also applied to editing paths, producing constrained results. For instance, first row of Figure 3 (b) illustrates an attempt to transition the style the image toward "van Gogh style". ProxNPI tends to adhere closely to the original context, whereas our proposed NMG effectively transforms the style.
4.2 EVALUATING PROPERTY WITH ADDITIONAL EDITING METHODS
Leveraging spatial context The distinct advantage of NMG lies in its direct utilization of the noise maps, allowing it to preserve spatial context compared to other inversion methods. To demonstrate this capability, we integrate NMG with MasaCtrl (Cao et al. 2023), an editing approach that utilizes mutual self-attention for non-rigid editing tasks. Because MasaCtri relies on the DDIM reconstruct an input image during real image editing, we substitute the DDIM with NMG and other comparison methods to demonstrate the effectiveness of approach. For spatially demanding edits, undertake pose modification and viewpoint alternation. Figure (a) showcases the results achieved with MasaCtrl. Due to NMG's capability access spatial context directly through noise maps, yields unparalleled editing results. The effectiveness of our method in preserving spatial context further highlighted in Section 4.1 Although ProxNPI employs inversion guidance capture spatial context, its reliance on single gradient descent step to align with the inversion trajectory occasionally results in inconsistent outputs. We also compare results with ProxMasaCtrl, a method proposed by Han et al. (2023) that integrates ProxNPI with MasaCtrl using different integration approach than our main comparison. For details and experiments, see Appendix A.2
Robustness to variations of DDIM inversion Most inversion techniques leverage DDIM inversion as its foundation to encode the image into the latent. However, DDIM inversion can be modified to meet certain goals. To explore the robustness of our method to variations DDIM inversion. we incorporate NMG with pix2pix-zero (Parmar et al. 2023). Pix2pix-zed zeroo designed for zero-shot image-to-image translation, modifies DDIM inversion by adding regularization term to calibrate the

---

Published as a conference paper at ICLR 2024
P2P (Local)
P2P P2P (Global)
MasaCtrl
User Study
Models
CLIP
TIFAt
CLIPt
TIFA
CLIPt
TIFA
DDIM
0.2977
0.8436
0.3066
0.8349
0.2825
0.8253
NTI
0.2983
0.9125
0.3202
0.8302
0.2903
0.8277
10.0%
NPI
0,2982
0.9076
0.3157
0.8117
0.2921
0.8188
5.0%
ProxNPI
0.2951
0.8947
0.3006
0.8463
0.2922
0.8188
12.5%
NMG(O
0.3007
0.8955
0.3221
0.8991
0.2955
0.8548
72.5 %
Table I: Quantitative evaluation of image editing using local and global editing with Prompt-to- Prompt, MasaCtrl, and user study reveals that NMG consistently surpasses other baseline methods in editing performance.
Method
Reconstruction
optimized conditional
MSEL
SSIMT
LPIPS
Time.
NTI NMG NTI+ + NMG
0.0127
0.7292
0.1588
104.85
0.0124
0.7296
0.1673
5.97
0.0124
0.7302
0.1668
107.72
Table 2: A quantitative evaluation of reconstruction assesses how closely each method aligns with the inversion trajectory. While combining both strategies yields the best results, NMG stands out as an efficient approach when considering both time and performance.
initial noise to more closely resemble Gaussian noise. In the inversion phase, we utilize the modified DDIM inversion proposed in pix2pix-zero. For the reconstruction phase, similar to our experiments with MasaCtrl, we integrate NMG and other comparison methods. Figure presents the editing outcomes achieved using pix2pix-zero on tasks like cat-to-dog, and dog-to-cat conversions. The results indicate that our method produces robust results with spatially coherent samples, even under variations of DDIM inversion.
4.3 QUANTITATIVE COMPARISON
To quantitatively measure the quality of image editing paired with our method, we utilize two metrics: CLIPScore (Hessel et al., 202T) and TIFA (Hu et al., 2023). CLIPScore gauges how closely the edited image aligns with the target text prompt in the CLIP (Radford et al. (021)) embedding space. TIFA, on the other hand, evaluates the semantic alignment given image and text prompt pairs based visual question-answering accuracy. Our evaluation, utilizing all the tasks described in Section 4.1 is conducted on four local and global editing results via Prompt-to-rromp and two tasks for non-rigid image editing via MasaCtrl. We edit 20 images for each task and report the averaging scores in Table I It can be seen that NMG consistently surpasses competing methods, an efficacy that stems from NMG's capability retain the spatial context of the input image without imposing constraints on the editing path.
While the metries previously discussed are commonly employed, their reliance on specific models often causes them to misalign with human perception. To address this and evaluate visual quality from human-centric perspective, we additionally conduct a user study. Fifty participants were recruited through Prolific (Prolific 2023). For the study, 40 sets, each containing four images edited using NTI, NPI, ProxNPI, and NMG, were presented to the participants. Each was paired with a specific editing instruction and an input image. Participants chose the image with the highest fidelity that best met the editing instructions. We note that images within each set were displayed a randomized order. In analysis, collect the method most often selected in each comparison and report the selection ratio in the final column of [ Notably, our NMG method was the most preferred by participants. These findings underscore the efficacy of our method in aligning closely with human perception of image quality. A print screen of the user study is provided in Figure T2
4.4 ABLATION STUDY
Reconstruction For reconstruction of real images, NMG adheres to the trajectory of DDIM inversion, conditioned by noise maps. To demonstrate the effectiveness of our approach, we compare it with the optimization-based method, NTI. Our evaluation utilizes the MS- MS-COCO validation set Lin et al. (2014), from which we randomly select 100 images. We assess the reconstruction quality of images using metrics such MSF, SSIM (Wang al. 2004), and I.PIS (Zhang al. 2018). The
8

---

Published as a conference paper at ICLR 2024
DDIM
= 1000
Sy -10
Chickooe
Input Image
Input Image
= 5000
30000
Sy Syy
Sy
Noise Map Guidance
"a eal tiger is sitting on a weoden chair"
"a black eat tiger walking down road"
(a) Guidance srale
(b) Gradient scale
Figure 5: Ablation results of (a) guidance scales and (b) gradient scales. In (a), we demonstrate that the noise map guidance scale governs the influence input image nuances, while the text guidance scale steers the extent of edits in the desired direction. In (b), we demonstrate that the gradient scale regulates the degree of alignment with the inversion trajectory.
results in Table 2 indicate that our method performs comparably to the optimization-based approach. Notably, the best results are achieved when both methods are used together. However, regarding reconstruction speed, our strategy outpaces its optimization counterpart by nearly factor of 20. Taking both speed and performance into account, NMG emerges as the efficient choice for aligning with the inversion trajectory during reconstruction.
Guidance scale NMG utilizes duallconditioning strategy comprising noise maps and text prompts. To discern the effects of each condition, modulate their respective scales of s and sT in Eq T0 and Eq. 12 Figure (a) illustrates the influence of each guidance scale. Prioritizing text conditioning aligns outcomes more closely the intended edits. In contrast, prominent noise map conditioning scale retains the image's ' context. However, achieving balance between the two scales is essential for a desirable outcome. Over-relying on text conditioning, evident in the grid image's ' lower left triangle, erodes the nuance of the input image. Conversely, an excessive emphasis on noise map conditioning, seen in the grid image's 's upper right triangle, may hinder successful editing. The grid image's 's diagonal showcases results from balanced scaling, indicating that appropriate scaling yields the best outcomes. Importantly, our experiments maintained consistent guidance scale, underscoring its robustness across varied samples.
Gradient scale To regulate the adherence to the trajectory of DDIM inversion, we employ a gradient scale 89 in Eq. The effects of this scale are demonstrated by varying its magnitude and presenting samples in Figure (b). smaller gradient scale offers limited alignment with inversion trajectory, leading to edited outputs closely mirroring the result using DDIM in the reconstruction path. Conversely, an overly large gradient scale results in pronounced trajectory alignments, degrading the editing quality. Therefore, the proper selection of the gradient scale is vital. Analogous to the guidance scale, we maintain a consistent gradient scale across all experiments, ensuring it remains universally effective rather than overly sensitive to individual samples.
5 CONCLUSION
In the evolving field of image editing, Noise Map Guidance (NMG) offers a notable advancement. By addressing the challenges present in current real- real-image editing methods using text-guided diffusion models, NMG introduces an inversion method rich in spatial context. NMG directly conditions noise maps to the reverse process, which captures the spatial nuance of the input image and ensures the preservation of its spatial context. Experimental results demonstrate NMG's capability to preserve said spatial context, and this property is furthermore highlighted with spatially intensive edits. NMG approach that prioritizes speed without compromising
also designed as an optimization- quality. NMG represents a significant step forward, suggesting further investigation and refinement real-in image editing techniques.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank the Image Vision team of NAVER Cloud for their thoughtful advice and discussions. Training and experiments were done the Naver Smart Machine Learning (NSML) platform (Kim et al. 2018). This study was supported by BK21 FOUR. T.-H. Oh was partially supported by Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea (MVT) 1oo.2021-0-0206 Artificial Intelligence Innovation Hub; No.RS-2023- 00225630, Development Artificial Intelligence for Text-based 3D Movie Generation).
ETHICS STATEMENT
Generative models for synthesizing images carry with them several ethical concerns, and these concerns shared by (or perhaps exacerbated in) any generative models such ours. Generative models, in the hands of bad actors, could be abused to generate disinformation. Generative models such ours may have the potential displace creative workers via automation. That said, these tools may also enable growth and improve accessibility for the creative industry.
REPRODUCIBILITY STATEMENT
The source code can be found at http://gghhh..ccmmmaaassssS/A
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Transactions on Graphics (TOG), 42(4):1-11, 2023.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 843-852, 2023.
Omer Bar-Tal, Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European conference on computer vision, pp. 707-723. Springer, 2022.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.0844 2023.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.0077
2023.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 05225683-25696 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. arXiv preprint arXiv:2306.0000 2023.
Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Yuxiao Chen, Di Liu, Qilong Zhangli, et al. Improving regative-prompt inversion via proximal guidance. arXiv preprint arXiv:2306.05414 2023.
10

---

Published as a conference paper at ICLR 2024
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt- to-prompt image editing with cross-attention control. In The Eleventh International Conference on Learning Representations, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 7514-7528, 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah Smith. Tifa: Accurate and interpretable eex-to--image faithfulness evaluation with question answering. arXiv preprint arXiv:2303. 1897, 2023.
Inbar Sutberman-Spiegelghle Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. arXiv preprint arXiv:2304.0614/) 2023.
Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1134, 2017.
Bahjat Kawar, Shiran Zada, Oran Lang. Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Trani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Hanjoo Kim, Minkyu Kim, Dongjoo Seo. Jinwoong Kim, Heungseok Park, Soeun Park. Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, al. Nsml: Meet the mlaas platform with real-world case study. arXiv preprint arXiv: 18/0///000000 18.09957 2018.
Gihyun Kwon and Jong Chul Ye. Diffusion- baaedd image translation using disentangled style and content representation. In The Eleventh International Conference on Learning Representations, 2022.
Tsung-V Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Daiki Miyake, Akihiro Tohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models. arXiv preprint 2rXiv:2305.16807, 2023.
Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-608 6044, 2023.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text text-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804. PMLR, 2022.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXin:2307.01195. 2023.

---

Published as a conference paper at ICLR 2024
Prolific, 2023. URL htt.t:::/www..rrlll.i.../
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10000 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-3666 -34494, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020a.
Yang Song, Jascha
Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2020b.
Dani Valevski, Matan Kalman, Eyal Molad, Eyal Segalis, Yossi Matias, and Yaniv Leviathan. Unitune: Text-driven image editing by fine tuning a diffusion model on single image. ACM Transactions on Graphics (TOG), -10 2023.
Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transfor- mations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. --254,,,,, 2023.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 1344)6600-612 2004.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang, Freedom: Training-free energy-guided conditional diffusion model. arXiv preprint urXiv:2303.0983. 2023.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Zhongping Zhang, Huiwen He, Bryan A. Plummer, Z. Liao, and Huayan Wang. Complex scene image editing by scene graph comprehension. 2022. URL lttpp::// appi........ semanticscholar
org/CorpusID:247 6678000
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3622 2022.
12

---

Published as a conference paper at ICLR 2024
Noise Map Guidance: Inversion with Spatial Context for Real Image Editing
SSupllementary Material
A
ADDITIONAL EXPERIMENTS
Noise Map
Text
Input
Input
L1
L2
Text
Noise Map
"a white plate topped with meatballs tofu and groens"
"a cat tiger walking on lodge body water"
"a dog sitting on the side of the road snow"
"a young man standing narrow alley way, smillng"
(a) Conditioning Order
(b) Energy Fundion
Figure 6: Ablation results of (a) conditioning order and (b) energy function
A.1 ADDITIONAL ABLATION STUDY
Conditioning Order As detailed Section 3.2, NMG employs a dual-conditioning approach that leverages both the noise map and text conditions sequentially. While our primary strategy conditions the noise map before the text embeddings, the outcomes of local editing under varying conditioning sequences: the second column depicts the canonical NMG sequence-initially the noise map, then text; the third column illustrates the reverse order. Both configurations yield promising results, underscoring NMG's robustness to conditioning sequence variations.
feasible to reverse this order. Figure (6 (a) presents
formulate the energy function using the L1 distance, diverging from the L2 distance employed by NTI (Mokady et al. 2023). Figure (b) showcases editing outcomes derived from these distinct energy functions: the second column illustrates results using L1 distance, while the third column represents those from L2 distance. Both methodologies adeptly guide the editing direction. However, the L2-based results manifest noticeable blurring in the image background. These observations align with the findings of prior research (Isola et al. 2017 Pathak al. 2016) that 1.2 distance often produces blurred results. Hence, for NMG, we intentionally select the distance for energy function formulation.
Energy Function Section 3.2,
A.2 COMPARISONS WITH PROXMASACTRL
ProxiNPI Han et al. 2023) introduces an integration with MasaCtrl (Cao et al. 2023). termed ProxMasaCtrl. This approach incorporates NPI (Myykke al. 2023) in the reconstruction path, while leveraging proximal guidance in the editing path to enhance reliability. Figure compares the results from ProxMasaCtrl with our integration, labeled as "NMG MasaCtrl". Despite ProxMasaCtrl's mitigation of unintended changes through proximal guidance, its use of NPI in reconstruction means it often misses spatial context. This lack spatial context is evident in Figure 7 (a): the first row highlights ProxMasaCtul's inability to alter poses accurately, and the second row reveals missing spatial components, like a kitten's 's tail, during editing. Similarly, when observing viewpoint alternation results in Figure 7 (b), ProxMasaCtrl struggles to deliver convincing outcomes, whereas our NMG- inGGiinteratted solution, drawing on its rich spatial context, consistently achieves superior results.
13

---

Published as a conference paper at ICLR 2024
NMG + MasaCtri
NMG M assCCri
Input
ProxMasaCtrl
Input
ProxMasaCtrt
"a baby lamb standing sitting lush green field"
"a statue of with beard, side view"
"a small kitten standing sitting on top of table"
"a white car parked in a parking lot, rear angle"
(a) Pose Modification
(b) Viewpoint Altemation
Figure 7: Comparison with ProxMasatCrrl
A.3 EXPERIMENTS DETAILS
Within experimental framework. employ Stable Diffusion (Rombach et al. 2022). standardiz- ing the diffusion steps to T 50 across all experiments. For the editing tasks, the parameters are set as follows: noise map guidance v 10, text guidance Sy 10, and guidance scale 8g 5000. For the reconstruction tasks, the configurations are set to &N = 10, $T = 7.5, and &g = 10000.
A.4 ADDITIONAL RESULTS
Local and Global Editing We present supplementary comparison results in Figure an Figure I1 showcasing local and global editing outcomes achieved with Prompt-to-Pront (Hertz et al.) 2022). These additional results further underscore NMG's proficiency in preserving spatial context without diminishing the quality the editing output. Furthermore, we present the results of diverse stylization tasks in Bj) demonstrating NMG's versatility. This evidence suggests that NMG's capabilities extend beyond painting stylizations, encompassing a broad spectrum of styles, including anime and mosaic.
Non-rigid Editing In conjunction with MasaCtrl (Cao al. 2023). NMG demonstrates its capabil- ity for non-rigid editing. Figure b) exhibits additional experimental results of pose modifications. It evident that NMG effectively conducts edits with pronounced spatial information changes.
Editing with NTI + NMG In image reconstruction. the combined use of NTI and NMG has been shown yield superior quality. To ascertain whether this synergy also extends to image editing, we have undertaken series of editing experiments utilizing both NTT and NMG. Figure C) demonstrates that while NMG alone provides dependable editing, it occasionally lacks specific spatial details. Integrating additional information from NTI effectively compensates for this shortfall, restoring the spatial context the input image. This integration potentially represents significant advancement in augmenting the capabilities of NMG for image editing tasks.
Comparisons with ProxNPT For a further comparative analysis with ProxNPI, we replicated the experiment using the same image featured in the ProxNPI study. Figure d) presents these additional comparative results. The outcomes demonstrate that NMG yields results comparable to the ProxNPI. However, it is essential to note that ProxNPI primarily concentrates not extensively explore various editing scenarios, such as contextual alterations or stylization. As discussed in Section 4.1 ProxNPI's capabilities somewhat limited due to its reliance inversion guidance. In contrast, NMG is not subject to these constraints, demonstrating its more extensive utility in various image editing tasks.
object swapping and does
14

---

Published as a conference paper at ICLR 2024
Input
Cubism
Anim Style
Mosaic
Input
Edited
"a woman in blue dress is dancing on a sidewalk, <style>"
standdngg sitting..."
"a deer standing in a forest next to a tree, <style>*
....nnning standing..."
(a) Global Editing
(b) Non-rigid Editing
Input
NMG
NMG NTTI
Input
ProxNPI
NMG(Ours)
"a eat tiger sitting next to a mirror"
"orange van with surfboards flowers on top"
"a man with beard wearing sunglasses and denim jacket, smiling"
"two origami birds sitting on branch"
(c) Combine NMG with NTI
(d) Comparison with ProxNPI
Figure 8: Additional editing results of (a) global editing, (b) non-rigid editing, (c) combine NMG with NTI, and (d) additional comparison with ProxNP1
Reconstruction We offer an additional quantitative comparison of reconstruction results. The left side of TableBppresst presents a quantitative comparison of reconstruction capabilities between NMG and other methods. While NMG demonstrates comparable quantitative results in reconstruction to these methods, it is important to emphasize that NMG's primary focus is on image editing. It should be noted that high-quality reconstruction does not invariably equate superior editing performance. ProxNPI exhibits commendable ability to reconstruct the image in Table 3 However, as illustrated in the sixth row of Figure II its editing capabilities are somewhat limited, underscoring the distinction between reconstruction proficiency and editing versatility.
Additional User Study Evaluating edited images based on their aligmment with editing instructions is crucial. yet it is equally essential to delve into more detailed aspects of evaluation. For example, in local editing tasks, it essential to maintain the integrity of unedited regions, while in global editing, preserving the overall structure of the image is essential. To explore these aspects, we conducted an additional user study similar to the approach in Section /4.3 Thirty participants, recruited via Prolific (Prolific 2023), were presented with ten sets of images. Participants were instructed to assess the preservation of unselected regions local editing and the retention the overall structure in global editing. The results of the study, displayed on the right side of Table 3 indicate that NMG performs well in both preserving unselected regions during local editing and maintaining the overall structure in global editing scenarios.
15

---

Published as a conference paper at ICLR 2024
Reconstruction
User Study
Method
MSE,
SSIM
LPIPS|
DDIM
0.159
0.368
0.521
NTI
0.013
0.729
0.159
0.0%
NPI
0.021
0.661
0.230
0.0%
ProxNPI
0.012
0.737
0.153
10.0%
NMG(Ours)
0.012
0.723
0.167
90.0%
Table 3: Quantitative comparison of reconstruction and additional user study
"a person rding beside horse
"a car bicycle driving down a sandy road"
(a) Relationship Change
(b) Object Swap with Different Structure
"two men-on one man walking down a street"
"a boat on the right side of the beach"
(c) Object A Addict and Deleting
Figure 9; Illustrations of limitations
B LIMITATIONS
Our research endeavors to enhance inversion-basee editing methods, focusing on improving spatial context during the editing process. Nevertheless, current methodology is limited to compatibility with methods that do not conform to the inversion-basse framework. For instance, SGC-Net (Zhang al. 2022), which operates text-guided diffusion model for relationship change tasks, proves challenging to integrate with NMG due to its deviation from the inversion braseddon-baeed editing paradigm. Consequently, applying NMG in conjunction with SGC-Net for tasks involving relationship changes faces significant hurdles. As an alternative, we attempted relationship change tasks using MasaC- trl (Cao al. 2023). However, Figure a) depicts the ineffectiveness of this approach, as MasaCtrl is not inherently designed for relationship change tasks.
Additionally, the editing capabilities of NMG are intrinsically linked the limitations of existing inveesio--based methods. For example, Prompt-to- Proo--rr (Hertz et al. 2022) editing involves the swapping of cross-attention maps between source and target texts, which restricts the possibility of structural changes in the edited object to the constraints of the original image's object. This limitation evident in Figure b)), which illustrates an object swap task between a car and a bicycle. The disparate structures of these objects underscore the challenges faced by NMG in ensuring reliable editing under these conditions.
Moreover, NMG's dependence text for image editing hinders its ability to perform precise spatial changes. As shown on the left side of Figure (cc), NMG can effectively remove an object, such as a man, from image. However, lacks the precision to identify and remove specific individual,
16

---

Published as a conference paper at ICLR 2024
making targeted removals unfeasible. Similarly, struggles to follow exact location directives when adding new objects. For instance, despite the instruction to place boat the right side of the beach, the edited image fails place it the right. Addressing these challenges to enable more accurate spatial control real image editing with NMG vital area for future research and development.
the right side of Figure _e). NMG
C EXTENDED RELATED WORK
Guidance in Diffusion Models To generate samples that abide by certain condition, Score- SDE (Song et al. 2020b) initially introduces a formulation for conditional diffusion models. Fol- lowing this formulation Dhariwal Nichol (222I) proposes classifier guidance, which leverages additional classifiers to enhance the quality of class conditional image synthesis. Building upon classifier guidance, |Ho Salimans (2021) proposes classifier frrefe guidance, which obviates the need for additional classifier. However, these guidance methods are often restricted to class or text conditions. EGSDE (Zhao et al. 2022) advances the field by introducing energy guidance techniques. Although EGSDE primarily focuses on image-to-image translation tasks, they notably expand the flexibility in conditioning formats diffusion models. Similarly, DiffuselT (Kwon & Ye 2022). inspired by MCG (Chung et al. 2022), focuses image translation. Successive methodologies. such as FreeDoM (Yu et al. 2023) and universal guidance (Bansal et al. 2023), broaden the scope of conditions to encompass diverse range of generative tasks. These methodologies
adaptable to various conditional signals, including segmentation maps, style images, and facial identities. Aligning with these advancements, approach also leverages the versatile framework of energy guidance, targeting the accurate reconstruction of input images.
D USER STUDY
screenshot our user study conducted through Prolific (Prolific 2023 is depicted in Figure 12 Note that Prolific receives pre-built Google form, and distributes participants of the study.
17

---

Published as a conference paper at ICLR 2024
Input
DDIM
NTI
NPI
ProxNPI
NMG(Ours)
Object Swap
"a small brown and white dog robot laying on top of lush green filed"
"a cupcake flower on plate with a fork"
Attribute Editing
woman with long blonde hair standing in front of pink wall, sunglasses"
"a young boy standing in front of pile of food, sunglasses"
Contexuual Altemation
"a woman with long hair standing in front of a wall building"
"a woman standing on bridge in the woods snow"
Color Change
"a living room with yellow red chair and ottoman"
"a pair of marvon black vans on yellow background"
Figure 10: Local editing with Prompt-to-Prompt
18

---

Published as a conference paper at ICLR 2024
Input
DDIM
NTI
NPI
ProxNPI
NMG(O
Oil painting
a man with curly hair and a hoodie looking up, oil painting"
"a woman dives to eatch a volleyball on the beach, oil painting"
Van Gogh style
woman with dreadlocks smiling the camera, van gogh style"
"a man in hat is walking through canyon, van gogh style"
Pointillism
"a couple of men walking down a street next to each other, pointillism" "
beautiful young woman leaning against a wall, pointillim"
ein
Watercolor
"a woman standing on top of sandy beach, watercolor"
"a woman in green dress and black hat, watercolor
Figure 11: Global editing with Prompt-to-Prompt
19

---

Published as a conference paper at ICLR 2024
[Sample 1] Select the image of the highest quality, considering both the original input and the editing instructions
Edit instruction: dog - robot
Image 1
Image 2
Input Image
Image 3
Image 4
Image 1
Image 2
Image 3
Image 4
Figure 12: User study screenshot
20


Reviews:
[{"review_id": "sZSuH8Hq6r", "reviewer": "Reviewer_yyX5", "summary": "The paper introduces NOISE MAP GUIDANCE (NMG), a new method for real-image editing. NMG uses noise maps derived from latent variables of DDIM inversion to capture spatial context effectively. By conditioning these noise maps to the reverse process and using both noise maps and text embeddings for image editing, NMG eliminates the need for time-consuming optimization. The results show that NMG preserves spatial context better, works faster, and integrates well with various other editing techniques while maintaining high edit quality. Furthermore, it demonstrates robust performance across different versions of DDIM inversion.", "questions": "- How about the editing performance of adding or deleting elements in the images?\n\n- While both global and local editing in the ProxNPI paper seem promising, the editing capability in this paper doesn't appear as effective. For instance, in Figure 3, the second row shows an edited result with a clear boundary between two types of backgrounds. In the third row, under \"Van Gogh,\" the overall style seems to have undergone minimal change. Can you provide an explanation for these observations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The presented concept is intriguing and efficient. It details an uncomplicated yet effective method of using noise map conditioning during real image inversion, which streamlines the reverse process and eradicates path divergence between the reconstruction path and inversion trajectory. This leads to a more precise reconstruction.\n\n- Moreover, the experiments carried out are robust. They demonstrate superior performance in real-image editing both qualitatively and quantitatively. Additionally, the tests focusing on spatial context utilization are crucial, effectively proving enhanced spatial context preservation capabilities.", "weaknesses": "- The visualization results depict many recurring stylization outcomes, such as the \"oil painting style\" and \"Van Gogh style\". It would be beneficial for the paper to exhibit a broader variety of more challenging editing instances.\n\n- The user study could benefit from providing more accurate directives in its questions; if it pertains to local editing, for instance, the question should contemplate including \"evaluate original preservation in unedited areas”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vys7R2lg1F", "reviewer": "Reviewer_9KmB", "summary": "This paper proposes a noise map guidance method to capture the spatial context information in the input image, addressing the challenge in Null-text Inversion (NTI).  The proposed method is designed for real image editing. Experiments are performed to demonstrate various image editing capabilities of NMG, such as face expression modification, style transfer, viewpoint alternation, among others.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method looks simple but effective;  \nBoth quantitative and qualitative experiments are performed to validate that NMG outperforms baselines;  \nExtensive experiments are performed to validate that NMG is able to achieve different image editing tasks.", "weaknesses": "I have the following comments about the weaknesses:\n\n1) Figure 2 seems confusing to me. Since this paper repeatedly mentioned that NMG address the challenges in Null-text Inversion, I think it would be nice to compare NMG to Null-text Inversion in this Figure, and demonstrate how NMG outperforms Null-text Inversion. In addition, it would be nice to add the corresponding text descriptions in the Figure. *E.g.*, it's unclear whether the caption indicate to change the blue fire hydrant to a red one or it's just some deviations during the reconstruction.\n\n2) Limitations and future work should be discussed. For example, whether NMG can address the relationship change task like SGC-Net [1]? Whether NMG can achieve various non-rigid image editing tasks like Imagic [2]? It seems that NMG can achieve some non-rigid editing tasks such as viewpoint alternation or face expression modification. However, spatial information between the output and input seems consistent in the majority parts, from my view. Thus, it would be great to see experiments exploring whether NMG can perform other operations (with more obvious spatial information change) such as \"from a tiger\" to \"a jumping/running tiger\".\n\n[1] Zhang, Zhongping, et al. Complex Scene Image Editing by Scene Graph Comprehension. BMVC 2023.  \n[2] Kawar, Bahjat, et al. Imagic: Text-based real image editing with diffusion models. CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ftky9Y2uN", "reviewer": "Reviewer_pVhT", "summary": "The paper \"Noise Map Guidance: Inversion with Spatial Context for Real Image Editing\" presents a novel inversion method called Noise Map Guidance (NMG) for real-image editing using text-guided diffusion models. NMG addresses the challenges faced by existing methods, such as Null-text Inversion (NTI), which fail to capture spatial context and require computationally intensive per-timestep optimization. NMG achieves high-quality editing without necessitating optimization by directly conditioning noise maps to the reverse process, capturing the spatial context of the input image. The authors demonstrate NMG's adaptability across various editing techniques and its robustness to variants of DDIM inversions through empirical investigations", "questions": "Why there are no quantitive and qualitative comparison with previous works about reconstruction? I think there should be a comparison with other methods in this aspect, or the author should explain why it is omitted.\n\nHow NTI + NMG  performs when dealing with actual editing task? it would be helpful to show the proposed method can combine with previous method to achieve a better result.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and provides sufficient background and analysis into the motivations and effectiveness of NMG. the overall framework is very straightforward, there should be no difficulty for other to reproduce. it demonstrates a strong adaptability across various editing techniques, including Prompt-to-Prompt, MasaCtrl, and pix2pix-zero.\n\nThe method is optimization-free, making it computationally efficient while preserving editing quality, it achieve a 20 times acceleration compares to null-text inversion.\n\ncomprehensive quantitative and qualitative comparison of NMG with other inversion methods, showcasing its superior performance in preserving spatial context and editing fidelity.", "weaknesses": "NMG impose a very strong spatial constraint during editing, as in the figure most showcase have almost the same geometry structure as the original picture, for case the modify geometry e.g. the cat in figure 4, the result shows an apparent artifacts in the modified region. There needs a further investigation on how will NMG perform when facing editing that requires modification on the spatial structure, for example removing a target (like \"two man ...\" ->\"one man...\") or change to a totally different object (\"...car\" to \"... bike \" ).    \n\nMoreover, there lack of discussion about possible failure cases of  NMG, the authors should add such discussion about in what circumstance NMG would fail and the reason why it fails to help the community better understand the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)