Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models
Authors: Senmao Li
Joost van de Weijer
taihang Hu
Fahad Khan
Qibin Hou
Yaxing Wang
jian Yang
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion model; Content Suppression; Image Editing; Text Embeddings
Primary area: generative models

Abstract: The success of recent text-to-image diffusion models is largely due to their capacity to be guided by a complex text prompt, which enables users to precisely describe the desired content. However, these models struggle to effectively suppress the generation of undesired content, which is explicitly requested to be omitted from the generated image in the prompt. In this paper, we analyze how to manipulate the text embeddings and remove unwanted content from them. We introduce two contributions, which we refer to as soft-weighted regularization and inference-time text embedding optimization. The first regularizes the text embedding matrix and effectively suppresses the undesired content. The second method aims to further suppress the unwanted content generation of the prompt, and encourages the generation of desired content. We evaluate our method quantitatively and qualitatively on extensive experiments, validating its effectiveness. Furthermore, our method is generalizability to both the pixel-space diffusion models (i.e. DeepFloyd-IF) and the latent-space diffusion models (i.e. Stable Diffusion).

Published as a conference paper at ICLR 2024
GET WHAT YOU WANT, NoT WHAT YOU DON'T: IMAGE CONTENT SUPPRESSION FOR TEXT-TO-IMAGE DIFFUSION MODELS
Senmao Li', Joost van Weijer", Taihang Hu', Fahad Shahbaz 34, Qibin Hou Yaxing Wang': Jian Yang!
VVCIP, CS, Nankai University, Universitat Autònoma de Barcelona Moohaeed bin Zayed University of AI, *Linkoping University
{senmaonk, wwwhhhthhhan he 1 com, joost wab. es
fahad. khan@liu. {hougb, yaxing, ca jyang] @nankai edu. en
ABSTRACT
The success of recent text-to-image diffusion models largely due to their capac- ity to be guided by complex text prompt, which enables users to precisely de- scribe the desired content. However, these models struggle to effectively suppress the generation of undesired content, which is explicitly requested to be omitted from the generated image in the prompt. In this paper, we analyze how to ma- nipulate the text embeddings and remove unwanted content from them. We in- troduce two contributions, which we refer to as soft-weighted regularization and inference-time lext embedding optimization. The first regularizes the text embed- ding matrix and effectively suppresses the undesired content. The second method aims further suppress the unwanted content generation of the prompt, and en- courages the generation of desired content. We evaluate our method quantitatively and qualitatively on extensive experiments, validating its effectiveness. Further- more, our method is generalizability to both the pixel-space diffusion models (i.e. DeepFloyd-IF) and the latent-space diffusion models (i.e. Stable Diffusion).
INTRODUCTION
Text-based image generation aims to generate high-quality images based on user prompt (Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2021). This prompt is used by the user to communi- cate the desired content, which we call the positive target, and can potentially also include undesired content, which define with the term negative target. Negative lexemes ubiquitously preva- lent and serve as pivotal components in human discourse. They are crucial for humans to precisely communicate the desired image content.
However, cxisting text-to-image models can encounter challenges effectively suppressing the generation of the negative target. For example, when requesting image using the prompt "a face without glasses", the diffusion models (i.e., SD) synthesize the subject without "glasses", as shown in Fig. 1 (the first column). However, when using the prompt "a man without glasses", both SD and DeepFloyd-IF models still generate the subject with "glasses" as shown in Fig. (the second and fifth columns). Fig. (the last column) quantitatively show that SD has 0.819 DetScore for "glasses" using 1000 randomly generated images, indicating very common failure cases in diffusion models. Also, when giving the prompt "a man", often glasses are included, see Fig. (the third and sixth columns). This is partially due to the fact that many of the collected man training images contain glasses, but often do not contain the glasses label (see in Appendix Fig. 9).
Few works have addressed the aforementioned problem. The negative prompt technique? guides diffusion model to exclude specific elements or features from the generated image. It, however, often
'The corresponding author.
also happens in both Ideogram and Mijourney models, see Fig. 'The negative prompt technique is used in text-to-image diffusion model to eliminate undesired content associated with prompt inputted into the unconditional branch instead of using the null-text

---

Published as a conference paper at ICLR 2024
AMAN
00
60
6:1
0101
Deepl
DetScore "glasses"
Figure I: Failure cases of Stable Diffusion (SD) and DeepFloyd-IF Given the prompt "A man with- out glasses", both SD and DeepFloyd-IF fail to suppress the generation of negative target glasses. Our method successfully removes the "glasses". (Right) we use DetScore (see Sec. 4) to detect the "glasses" from 1000 generated images. The DetScore of SD with prompt "A face without glasses" is 0.122. See Appendix E for additional examples.
leads to an unexpected impact on other aspects of the image, such as changes to its structure and style (see Fig. 6). Both P2P (Hertz al., 2022) and SEGA (Brack et al., 2023) allow steering the diffusion process along several directions, such as weakening target object from the generated image. We empirically observe these methods to lead to inferior performance (see Fig. 6 and Table I below). This is expected since they are not the tailored for this problem. Recent works (Gandikota et al., 2023; Kumari et al., 2023; Zhang et al., 2023) fine-tune the SD model to eliminate completely some targeted object information, resulting in catastrophic neglect (Kumari al., 2022). A drawback of these methods is that the model is unable to generate this context in future text-prompts. Finally, Inst-Inpaint (Yildirim et al., 2023) requires paired images to train model to erase unwanted pixels.
In this work, we propose an alternative approach for negative target content suppression. Our method does not require fine-tuning the image generator, collecting paired images. It consists of two main steps. In the first step, we aim to remove this information from the text embeddings which decide what particular visual content generated. To suppress the negative target generation. we climi- nate its information from the whole text embeddings. We construct a text embedding matrix, which consists of both the negative target and [EOT] embeddings. We then propose a seft-weighted regu- larization for this matrix, which explicitly suppresses the corresponding negative target information from the [EOT] embeddings. In the second step, to further improve results, we apply inference- text embedding optimization which consists of optimizing the whole embeddings (processed in the first step) with respect to two losses. The first loss, called negative target prompt suppression, weak- ens the attention maps of the negative target further suppressing negative target generation. This may lead to the unexpected suppression of the positive target (see Appendix D. Fig. 13 (the third row)). To overcome this, we propose positive larget prompt preservation loss that strengthens the attention maps of the positive target. Finally, the combination of our proposed regularization of the text embedding matrix and the inference-time emhedding optimization leads improved negative target content removal during image generation.
In summary, our work makes the following contributions: (I) Our analysis shows that the [EOT] embeddings contain significant, redundant and duplicated semantic information of the whole input prompt (the whole embeddings). This needs to be taken into account when removing negative target information. Therefore, we propose soft- ooft-weighted regularization to eliminate the negative target information from the [FOT] embeddings. (II) To further suppress the negative target generation, and encourage the positive target content, we propose inference-tine text embedding optimization. Ablation results confirm that this step significantly improves final results. (III) Through extensive experiments, we show the effectiveness of our method to correctly remove the negative target in- formation without detrimental effects on the generation of the positive target content. Our code is available in https: / thub. com/ sew-ssen-moosee /tppaaappppppp.......
RELATED WORK
Text-to-Image Image Generation. Text-to-image synthesis aims to synthesize highly realistic images which are semantically consistent with the text descriptions. More recently, text-to-image mod- els (Saharia et al., 2022; Ramesh et al., 2022; Rombach et al., 2021) have obtained amazing per-
We use text embeddings refer to the output from the CL.IP text encoder.
2

---

Published as a conference paper at ICLR 2024
formance in image generation. With powerful image generation capability, diffusion models allow users to provide text prompt, and generate images of unprecedented quality. Furthermore, a series of recent works investigated knowledge transfer on diffusion models (Kawar et al., 2022; Ruiz et al., 2022; Valevski et al., 2022; Kumari et al., 2022) with one or few images. In this paper, we focus on the Stable Diffusion (SI) model without fientuning, and address the failure case that the generated subjects are not corresponding to the input text prompts.
iiffusion-Based Image Generation. Most recent works explore the ability to control edit generated image with extra conditional information, as well as text information. It contains label-to- image generation, layout-to-image generation and (reference) image-to-image generation. Specifi- cally, label-to-image translation (Avrahami et al., 2022a;b; Nichol et al., 2021) aims to synthesize high-realistic images conditioning on semantic segmentation information, as well as text informa- tion. P2P (Hertz et al., 2022) proposes mask-fre editing method. Similar to label-to-image trans- lation, both layout-to-image (Li et al., 2023b; Zhang & Agrawala, 2023) and (reference) image-to- image (Brooks et al., 2022; Parmar et al., 2023) generations aim to learn a mapping from the input image map to the output image. GLIGEN((i et al., 2023b) boosts the controllability of the gener- ated image by inserting bounding boxes with object categories. Some works investigate Diffusion- based inversion. (Dhariwal & Nichol, 2021) shows that a given real image can be reconstructed by DDIM (Song et al., 2020) sampling. Recent works investigate either the text embeddings of the conditional input (Gal et al., 2022; Li et al., 2023a; Wang et al., 2023), or the null-text optimization of the unconditional input (i.e., Null-Text Inversion (Mokady et al., 2022)).
Diffusion- Blafusion-Based Semantic Erasion. Current approaches (Gandikota et al., 2023; Kumari et al., 2023; Zhang et al., 2023) have noted the importance of erasure, including the erasure of copy- right, artistic style, nudity, etc. ESD (Gandikota et al., 2023) utilizes negative guidance to lead the fine-tuning of a pre-trained model, aiming achieve a model that erases specific styles or objects. (Kumari et al., 2023) fine-tunes the model using two prompts with and without erasure terms, such that the model distribution matches the erasure prompt. Inst-Inpaint (Yildirim al., 2023) is a novel inpainting framework that trains a diffusion model to map source images to target images with the inclusion of conditional text prompts. However, these works fine-tune the SD model, resulting in catastrophic neglect for the unexpected suppression from input prompt. In this paper, we aim to remove unwanted subjects in output images without further training or fine-tuning the SD model.
3
METHOD
We aim to suppress the negative larget generation in diffusion models. To achieve this goal, we focus on manipulating the text embeddings, which essentially control the subject generation. Naively clim- inating a target text embedding fails to exclude the corresponding object from the output (Fig. 2a (the second and third columns)). We conduct comprehensive analysis that shows this failure is caused by the appended [EOT] embeddings (see Sec. 3.2). Our method consists of two main steps. In the first step, we propose soft-weighted regularization to largely reduce the negative target text infor- mation from the [EOT] embeddings (Sec. 3.3). In the second step, we apply inference-time text embedding optimization which consists of optimizing the whole text embeddings (processed in the first step) with respect to two losses. The first loss, called the negative target prompt suppression loss, aims to weaken the attention map of the negative target to guide the update the whole text embeddings, thus further suppressing the subject generation of the negative target. To prevent un- desired side effects, namely the unexpected suppression from the positive target in the output (see Appendix D. Fig. 13 (the third row)), we propose the positive target prompt preservation loss. This strengthens the attention map of the positive target. The inference- time text embedding optimization presented in See. 3.4. In Sec. 1. we provide simple introduction to the SD model, although our method is not limited to a specific diffusion model.
3.1 PRELIMINARY: DIFFUSION MODEL
The SD firstly trains an encoder E and a decoder D. The encoder maps the image x into the latent representation 2o = E(x), and the decoder reverses the latent representation zo into the image ã = D(zo). SD trains a UNet-based denoiser network to predict noise e, following the objective:
min E E2,, ||c co t, c)ll2
(1)
3

---

Published as a conference paper at ICLR 2024
eie ) c
Zevingg plosee
Deninng (CTT)
Zevinng Jasc
"oite vale-ty
AoT phht ginil cloo, ehme
cread gimw
( 69) 11
Figure 2: Analysis of [EOT] embeddings. (a) [EOT] embeddings contain significant information as can be seen when zeroed out. (b) when performing WNNM, we find that [EOT] embeddings have redundant semantic information. (c) distance matrix between all text embeddings. Note that each [EOT] embedding contains similar semantic information and they have near zero distance.
where the encoded text embeddings c is extracted by pre-trained CLIP text encoder I' with given a conditioning prompt p: c [(p), 2, is noise sample timestamp & ~ [1, T) and T is the number of the timestep. The SD model introduces the cross-attention layer by incorporating the prompt. We could extract the internal cross-attention maps A, which are high-dimensional tensors that bind pixels and tokens extracted from the prompt text.
3.2 ANALYSIS OF [EOT] EMBEDDINGS
The text encoder T maps input prompt p into text embeddings c 768, N = 77 in the SI) model). This works by prepending a Start oext ([SOT) symbol to the input prompt p and appending N - ppl End of Text ([EOT)) padding symbols at the end, to obtain N symbols in total. We define text embeddings
T(p) E RMX (i.e..M
SO cf' "..."..,ppp". p/pp111 C
EOT 2pl---
BOT
Below, we explore several aspects of the [HOT] embeddings.
What semantic information [EOT] embeddings contain? We observe that [EOT] embeddings carry significant semantic information. For example, when requesting an image with the prompt "a man without glasses", SD synthesizes the subject including the negative target "glasses" (Fig. 2a (the first column)). When zeroing out the token embedding of "glasses" from the text embeddings c, SD fails to discard "glasses" (Fig. 2a (the second and third columns)). Similarly, zeroing out all [EOT] embeddings still generates the "glasses" subject (Fig. 2a (the fourth and fifth columns)). Finally, when zeroing out both "glasses" and the [EOT] token embeddings, we successfully remove "glasses" from the generated image (Fig. 2a (the sixth and seventh columns)). The results suggest that the [EOT] embeddings contain significant information about the input prompt. Note that naively zeroing them out often leads to unexpected changes (Fig. 2a (the seventh column)).
How much information whole [EOT] embeddings contain? We experimentally observe that [EOT] embeddings have the low-rank property indicating they contain redundant semantic in- formation. The weighted nuclear norm minimization (WNNM) (Gu et al., 2014) is an effective low-rank analysis method. We leverage WNNM to analyze the [EOT] embeddings. Specifically, construct [EOT] embeddings matrix V fcO .EOT .OTT 22 and perform WNNM _OT
as follows D (V) VT, where V = UEV" is the Single Value Decomposition (SVD) of v, and D Σ) the generalized wofttthresholdig operator with the weighted vector w, i.c.,
*This observation is based on the results our statistical experiments on generated images, with a sample size of 100. The average PSNR 49.300, SSTM is 0.992, and the average Rank -78.3.
4

---

Published as a conference paper at ICLR 2024
cf (1 b.
1p|-1)
e00r
BB
SOT cNB of
--1P1-2
c5or
B
EN 60
EDOT
N-1p1-
BUT NB 50T
Σ
2 a
aOT N-
A.E
A
P1-P1-
SD moder
DEV?
2
DEVT
Eq.3
Eq 4
A
A NE
(a) Soft-weighted regularization
(b) Inference-imee optimization
Figure 3: Overview of the proposed method. (a) We devise a negative target embedding matrix X: X = Ve> o , - We perform SVD for the embedding matrix X UEVV. We introduce a weight regularization for each largest eigenvalue. Then we recover the embedding matrix X = UÊV". (b) We propose inference-time text embedding optimization (ITO), We align the attention maps of both _ and E. and widen the ones of both V and C.V.E.
goT
De();; soft (Es; wi) = max Ea wi, 0). The singular values do 2 ... 2 -1ppl-2 and the weights satisfy 0 wi S WN--pl
To verify the low-rank property of [EOT] embeddings, WNNM mainly keeps the top- largest singular values of E, zero out the small singular values, and finally reconstruct aofor ,afor ,
aEOT ,...,.,.....p..... N We use Rank(V)) to represent the rank of V., We explore the impact
of different Rank(() values on the generated image. For example, as shown in Fig. 2b, with the prompt "White and black long coated puppy" (here |pl = 6), we use PSNR and SSIM metries to evaluate the modified image against the SD model's output. Setting Rank(()) zeroing all [EOT] embeddings, the generated image preserves similar semantic information when using all [EOT] embeddings. As Rank(() increases, the generated image gets closer to the SD model's output. Visually, the generated image looks similar to the one of the SD model with Ra )-4. Achiev-
ing acceptable metric values (PSNR=40.288, SSIM=0,994) with Rank ( in Fig. 2b (middle). The results indicate that the [FOT] embeddings have the low-rank property, and contain redundant semantic information.
Semantic alignment for each [EOT] embedding There exist total of 76 |pl [EOT] embed- dings. However, we find that the various [EOT] embeddings are highly correlated, and they typically contain the semantic information of the input prompt. This phenomenon is demonstrated both qual- itatively and quantitatively in Fig. 2c. For example, we input the prompt "A man with a beard wearing glasses and beanie in blue shirt". We randomly select one [EOT] embedding to replace input text embeddings like Fig. 2c (left) S. The generated images have similar semantic information (Fig. 2c (right)). This conclusion also is demonstrated by the distance of each [EOT] embedding (Fig. 2c (middle)). Most [EOT] embeddings have small distance among themselves. In conclusion, we need to remove the negative target information from the 76 - |p] [FOT) embeddings.
3.3 TEXT EMBRDDING-BASED SEMANTIC SUPPRESSION
Our goal is to suppress negative target information during image generation. Based on the afore- mentioned analysis, we must eliminate the negative target information from the [EOT] embeddings. To achieve this goal, we introduce two strategies, which we refer to as soft-weighted regulariza- tion and inference-time text embedding optimization. For the former, we devise negative target embedding matrix, and propose a new method to regularize the negative target information. The inference- sitrencce-imeeeeee text embedding optimization aims to further suppress the negative target generation of the target prompt, and encourages the generation of the positive target. We give overview of the two strategies in Fig. 3.
'The selected [FOT] embedding repeated [pl times
5

---

Published as a conference paper at ICLR 2024
Soft- weig-weighted Regularization. We propose to use Single Value Decomposition (SVD) to extract negative target information (c.g., glasses) from the text embeddings. Let c = fcsOr, .f. - C EOT ,... ROT ......--p1--2 (pl-p) be the text embeddings from CLIP text encoder.
As shown in Fig. 3 (left), we split the embeddings ci = 0, Ipl 1) into the neg- ative target embedding set c eNE and the positive target embedding set C Thus we have EET (c (c of - % For - (cSOT ..E ..N a ,'',, ,, -N-Ipl- 2
EOT
=
We construct a negative target embedding matrix X: = [oNE ,0,
EOT GEOT , pl-
We
perform SVD: X UEVT, where E diag(ov. ".. ,nng), the singular values 01 2 [,
Intuitively, the negative target embedding matrix 2 0 ngy 7y = min N p - 1).
X = c E EOT
EOT N mainly contains the expected suppressed information. After
performing SVD, we assume that the main singular values corresponding to the suppressed information (the negative target). Then, to suppress negative target information, we introduce soft- weighted regularization for each singular value 6
δ -= 00
(2)
We then recover the embedding matrix x = U.VT. here E = diag(oo oi, , Note that the recovered structure is X [NE CD e ... N and é (cSOT, c NNE a
aEOT -N
We consider special case where reset top-K or bottom-K singular values to O. As shown on Fig. 4, we are able to remove the negative target prompt (e.g, glasses or beard) when setting the top-K (here, K= 2) singular values to O. And the negative target prompt information is preserved when the bottom-K K singular values are set to 0 (here. K=70). This supports our assumption that main singular values of X are corresponding the negative target information.
3.4 INFERENCE-TIME TEXT EMBEDDING OPTIMIZATION
As illustrated in Fig. 3 (right), for specific timestep t, during the diffusion process T + 1, we get the diffusion network output: eg ( E,, L, e). and the corresponding attention maps: (APE, A BOT
where e = (cSOT ..E. NN c oforr ,... CN -2)) The attention maps APE are corresponding
to C. while ANE are corresponding to NE which we aim to suppress. After soft-weighted regularization, have the new text embeddings è (cSOT, P. E ,2OOT, ... GEOT ,NNlp1122
Similarly, we are able get the attention maps: (ÁP",AN))
Here, we aim to further suppress the negative target generation, and encourage the positive target information. We propose two attention losses regularize the attention maps, and modify the text embeddings è to guide the attention maps to focus on the particular region, which is corresponding to the positive target prompt. We introduce an positive target prompt preservation loss:
Cpt
APE
A
PE
(3)
That is, the loss attempts to strengthen the attention maps of the positive target prompt at the timestep To further suppress generation for the negative target prompt. we propose the negative target prompt suppression loss:
C,t ANE ANE
(4)
Full objective. The full objective function of our model is:
L L= Apt + Anth
(5)
where Ap==1 and 00,5 are used to balance the effect of preservation and suppression. We use this loss to update the text embeddings č.
For real image editing, we first utilize the text embeddings e to apply Null-Text (Mokady et al., 2022) to invert a given real image into the latent representation. Then we use the proposed soft- weighted regularization to suppress negative target information from resulting in é. Next, we
'The inspiration for Eq. is explained in detail in the Appendix B.

---

Published as a conference paper at ICLR 2024
Table 1: Comparison with baselines. The best results are in bold, and the second best results are underlined.
image editing
Generated image editing
Method
Random negutive turget
Random negative target
Negative target; Car
Negalive larget:
lyler Ediat
Negative larget
Van Gogh
Clipseone]
IFTD) DetScore
[Clipsoree IFTD DetSeone] (Tlipsonee IFTD) DatSoore
KTlipcoree
[FIDs]CIissone IFT
image (Gienented image)
0.7986
0.3381
0.8225
0.4509
0.7414
Negative prompt
0,7983
175.8
0.2402
0,7619
169.0
0.1408
0.8458
151.7
0.5130
0.7437
233.5
0,8039
242.1
P2P (lletizet el 2022)
0,7666
92.53
0.1758
0.8118
1033
0.3391
0,5638
21.7
0.6343
0.7470
86.3
0.8849
139.7
ESD (Gandikota al., 2023)
165.7
0.2223
0.6954
256.5
0.7222
257.5
Conceptabblatinn (Kumari ot al,, 2023)
0.76412
172.3
0.0935
0.7411
211.4
0.8200
219.9
Forget-Me-Nee MetNot (Zhang
0.301
0.5867
0.7495
227.9
0.8391
203.5
Inpt-nt (Viddirm ol., 2023)
0.7527
135.5
0.1129
0.7602
1504
0.1744
1369
0.2361
SEGA (Brack et al., 2023)
0.7960
172.2
0.3005
0.5001
165.8
0.4767
0.7678
209.9
0.8730
175.0
Oars
0.6857
166.3
0.9384
(166477
176.4
20688
0.6448
apply inference-time text embedding optimization to update è during inference, resulting in the
final edited image. Our full algorithm is presented in Algorithm 1. See Appendix C for more detail about negative target generation of SD model without the reference real image.
Algorithm 1: Our algorithm
model
Inpul: A text embedddings l'(p) and real image I. Output: Edited image I.
Er Inversion| BI2).c) // e.g., Null-text 11 SNR
é SWR(e) (Eq. 2);
WoT cheet chillons
for :-::=T,,, =TT:= T-TTT11-.... 1.....do
é, =
// TTO
for ite O.... 9 do
A A c- co ta
A ANE t- co rel : An + Ant 3-6); & Ve,
end end
Figure 4: Effect of resetting top-K or bottom-K singu- lar values to O. Main singular values correspond to the target information that we expect to be suppressed.
21-1.--- + c (,,,,, Cc)
Return Edited image i = D(xg)
4 EXPERIMENTS
Baseline Implementations. We compare with the following baselines: Negative prompt, ESD (Gandikota et al., 2023), Concept-ablation (Kumari et al., 2023), Forget-Me-Not (Zhang et al., 2023), Inst-Inpaint (Yildirim et al., 2023) and SEGA (Brack et al., 2023). We use P2P (Hertz al., 2022) with Attention Re-weighting.
Evaluation datasets. We evaluate the proposed method from two perspectives: generated-image editing and real-image editing. In the former, we suppress the negative target generation from a generated image of the SD model with text prompt, and the latter refers to editing a real-image input and a text input. Similar to recent editing-related works (Mokady et al., 2022; Gandikota et al., 2023; Patashnik et al., 2023), we nearly 100 images for evaluation. For generated siterratdddddieeeeeee negative target surrpression, we randomly select 100 captions provided in the COCO's validation set (Chen al., 2015) as prompts. The Tyler Edlin and Van Gogh related data (prompts and seeds) are obtained from the official code of ESD (Gandikota et al., 2023). For real- aal-image negative target suppression, we randomly select 100 images and their corresponding prompts from the Unsplash and COCO datasets. We also evaluate our approach on the GQA-Inpaint dataset, which contains 18,883 unique source- suunee-targe--promete pairs for testing. See Appendix. A for more details on experiments involving this dataset. We show the optimization details and more results in Appendix A, D and E, respectively.
Metrics. Clipscore (Hessel et al., 2021) is a metric that evaluates the quality of a pair of a neg- ative prompt and an edited image. We also employ the widely used Fréchet Inception Distance (FID) (Heusel et al., 2017) for evaluation. To evaluate the suppression of the target prompt infor- mation after editing, we use inverted FID (IFID), which measures the similarity between two sets. In this metric, the larger the better. We also propose to use the DetScore metric, which is based on MMDetection (Chen et al., 2019) with GLIP (Li et al., 2022). We detect the negative target object in the edited image, successful editing should lead to a low DetScore (see Fig. 5 and Appendix A
httpps: /https://unslaae ash. com/

---

Published as a conference paper at ICLR 2024
Layioo
king
20%
(Da40))
(0251)
(0242)
20%
doing Dices
12N
500
(0:14)
(0.745)
(0224)
Figure 5: (Left) We detect the negative target from the edited images and and show the DetScore below. (Middle) Real image negative target suppression results. Inst-Inpaint fills the erased area with unrealistic pixels (the red dotted line frame). Our method exploits surrounding content information. (Right) User study.
Schmishubre
Yochua
woven besket with sloaa of gla6e in
traffic
with the recropilane
board
sunglasses and scart
telestsion and a table
siting frant a laptop--inne complla
light shove street Binding Sheaves with a-ean
by ven-Geglh
Real image
Negetive prompt
PZP
Enppann
Inst Impaint
SESAA
CICCA
ours
Qun
Figure 6: Real image (Left) and generated image (Middle and Right) negative target suppression results. (Middle) We are able to suppress the negative target, without further finetuning the SD model. (Right) Examples of negative target suppression.
for more detail). Following Inst- Inpainn (Yildirim et al., 2023), we use FID and CLIP Accuracy to evaluate the accuracy of the removal operation on the Inpaint dataset.
For real- real-image negative target suppression, as reported in Table I we achieve the best score in both Clipscore and DetScore (Table (the second and fourth columns)), and a comparable result in IFID. Negative prompt has the best performance in IFID score. However, it often changes the structure and style of the image (Fig. (left, the second row)). In contrast, our method achieves a better balance between preservation and suppression (Fig. 6 (left, the last row)). For generated image negative target suppression, we have the best performance for both a random and specific negative target, except for removing Tyler Edlin's 's style, for which ESD obtains the best scores. However, ESD requires to finetune the SD model, resulting in catastrophic neglect. Our advantage further substantiated by visualized results (Fig. 6).
8

---

Published as a conference paper at ICLR 2024
dlses
meak
nout muge
Shadis rertous
Cracke remcual
Zain PATOOAA
Ailing mlkjett real image Cenerating rulgeete for generiee imag
Figure 7: Additional applications. Our method can be applied image restoration tasks, such as shadow, cracks, and rain removal. Also we can strengthen the object generation (6-9 column).
As shown in Fig. (middle) and Table we achieve superior suppression results and higher CLIP Accuracy scores on the GQA-Inpaint dataset. Inst-Inpaint achieves the best FID score (Table 2 (the third column)) primarily because its results (Fig. 5 (the second row, the sixth column)) closely resemble the ground truth (GT). However, the GT images contain unrealistic pixels. Our method yields more photo-realistic results. These results demonstrate that the proposed method effective in suppressing the negative target. See Appendix. A for more experimental details.
User study. As shown in Fig. (Right), we conduct a user study. We require users to select the figure in which the negative target is more accurately suppressed. We performed septuplets comparisons (forced choice) with 20 users (20 quadruplets/user), The results demonstrate that our method outperforms other methods. See Appendix. E for more details.
Ablation analysis. We conduct an ablation study for the proposed approach. We report the quanti- tative result in Table 3. Using seithweighted regularization (SWR) alone cannot completely remove objects from the image. The results indicate that using both SWR and inference-timg text embedding optimization leads to the best scores. The visualized results presented in Appendix. D.
Additional applications. As shown in Fig. 7 (the first to the fifth columns), we perform experi- ments variety of image restoration tasks, including shadow removal, cracks removal and rain removal. Interestingly, our method can also be used to remove these undesired image artifacts. In- stead of extracting the negative target emhedding, we can also strengthen the added prompt and (EOT) embeddings. As shown in Fig. 7 (the sixth to the ninth columns), our method can suc- cessfully adapted strengthen image content, and obtain results that are similar methods like GLIGEN (Li et al., 2023b) and Attend-and-Excite (Chefer et al., 2023) (See Appendix. F for complete explanation and more results).
Table 2: Quantitative comparison on the GQA- Table 3: Ablation study. The effective-
Inpaint dataset for real image negative target sup- ness of both soft-weighted regularization and inference-time text embedding optimization.
pression task.
Paired data
CLIP Acc t
CLIP Acc (top5)
Clipscore,
IFID
DetScore
Methods
FID
SD SWR
0.8225
0.4509
X-Decoder Inst- Inpaint <
6.86
69.9
46.5
0.7996
85.9
0.3668
5.50
80.5
60.4
SWR+ Epp.
0.8015
100.2
0.3331
Ours
x
13.87
92.8
83.3
SWR Eps Lui
0.6647
176.4
0.1321
5 CONCLUSIONS AND LIMITATIONS
We observe that diffusion models often fail suppress the generation of negative target information in the input prompt. We explore the corresponding text embeddings and find that [EOT] embed- dings contain significant, redundant and duplicated semantic information. To suppress the genera- tion of negative target information, we provide two contributions: soft-weighted regularization and inference-time text embedding optimization. mation from the text embedding matrix. The inference-time text embedding optimization encour- ages the postive target to be preserved, as well as further removing the negative target information. Limitations: Currently, the test-time optimization costs around half a minute making the proposed method unfit for applications that require fast results. But, we believe that a dedicated engineering effort cut down this time significantly.
the former, we suppress the negative target infor-
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by funding by projects TED2021-132513B-000 and PID2022-143257NB. 100 funded by MCIN/AEI/ tionEU/PRTR and FEDER. The project supported by Youth Foundation (62202243). Computation supported by the Supercomputing Center Nankai University.
10001 and by the European Union NextGenera-
REFERENCES
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. arXiv preprint NXXv:22006.02779 2022a.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022b.
Manuel Brack, Felix Friedrich. Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting. Sega: Instructing diffusion using semantic dimensions. arXiv preprint arXiv:2301. 12247, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint 2tti::22/1.09000 2022.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf. and Daniel Cohen-Or. Attend-ande
excite: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13826, 2023.
Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint 19X.v:9006.07000 2019.
Xinlei Chen, Hao Fang, Tsung-Y. Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00322 2015.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth word: Personalizing text-to-image generation using textual inversion, arXiv preprint 2rXiv:2208.01615 2022.
Rohit Gandikota, Joanna Materzyúska, Jaden Fiotto-Kand Kaufman, and David Bau. Erasing concepts from diffusion models. arXiv preprint arXiv:2303.07345, 2023.
Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimization with application to image denoising. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2862-282220002 200222899220 2014.
Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Yuxiao Chen, Di Liu, Qilong Zhangli, et al. Improving negative-prompl inversion via proximal guidance. arXiv preprint arXiv: 230.:2300.00414 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompy image editing with cross attention control. arXiv preprint arXii=2208.01626, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference-free evaluation metric for image captioning. In EMNLP, 2021.
10

---

Published as a conference paper at ICLR 2024
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. pp. 6626- 1637,2017,
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Hui-Tang Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. ArXiv, abs/2210.09276, 2022.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. arXiv preprint urXiv:2212.04455 2022.
Nupur Kumari, Bingliang Zhang, Sheng-Y. Yu Wang, Eli Shechtman, Richard Zhang, and Jun- Yan Zhu. Ablating concepts in text-to-image diffusion models. arXiv preprint arXiv:2303. 13516, 2023.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq Jenq-Neng Hwang, et al. Grounded manguage-image pre- tre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
10965-10975, 2022.
Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Stylediffusion: Prompt-embedding inversion for text-based editing. arXiv preprint 2rXiv:2303.1564 2023a.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. arXiv preprint :xXiv:3311.00093, 2023b.
Chenlin Meng, Yang Song. Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Im- age synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models. arXiv preprint arXiv:2305.1680 16807, 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 10741, 2021.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Yan Zhu. Zero-shot image-to-image translation. arXiv preprint arXiv:2302.03027 2023.
Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Eie Elor, and Daniel Cohen-Or. Localiz- ing object-level shape variations with text-to-image diffusion models. In Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208 /2242, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Bureu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, Tim Sali- mans. Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text- to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 11487, 2022.
11

---

Published as a conference paper at ICLR 2024
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 1921-1930, 2023.
Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. Unitune: Text-driven image edit- ing by fine tuning an image generation model on a single image. arXiv preprint arX1v:2210.09477, 2022.
Kai Wang, Fei Yang, Shiqi Yang, Muhammad Atif Butt, and Joost van de Weijer. Dynamic prompt learning: Addressing cross-attention leakage for text-based image editing. arXiv preprint arXiv:2309. 15664, 2023.
Ahmet Burak Yildirim, Vedat Baday, Erkut Erdem, Aykut Erdem, and Aysegul Dundar. Inst-inpaint: Instructing to remove objects with diffusion models, 2023.
Yu Zeng. Zhe Lin, Huchuan Lu, and Vishal M Patel. Cr-fill: Generative image inpainting with auxiliary contextual reconstruction. Proceedings of the IEEE/CVF international conference on computer vision, pp. 14164-14173, 2021.
Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. mortet-me-not: Learn- ing to forget in text-to-image diffusion models. arXiv preprint arXiv:221..08332. 2023.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX: IMPLEMENTATION DETAILS
Configure. We suppress semantic information by optimizing whole text embeddings inference time, and it takes as little as 35 seconds. No extra network parameters are required in our optimiza- tion process. We mainly use the Stable Diffusion v1.4 pre-trained model 8, All of our experiments conducted using Quadro RTX 3090 GPU (24GB VRAM).
Early stop. Recent works Hertz et al. (2022); Chefer et al. (2023) demonstrate that the spatial location of each subject is decided in the early step. Thus we validate our method on different steps in inference time. Fig (left) shows that our method suffers from artifacts after 20 timesteps. In this paper, at inference time apply the proposed method among + 20 timesteps, and for the remaining timesteps, we perform the original image generation as done in the SD model.
Inner iterations. Fig 8 (right) shows the generation at different iterations within each timestep. We observe that the output images undergo unexpected change after 10 iterations. We set the iteration number to 10.
rta wither glaone.
(0UR)
FARTERY
FArTber
Restiin::
Figure 8: (Left) We stop optimizing at step 20 and keep the original model operating for the rest of the steps. (Right) The synthesized images with different iterations. We observe that have better performance when setting iteration to 10.
Inaccuracy label. Fig. 9 shows that the collected man training images contain glasses, but often do not contain the glasses label.
IFID We use the official FID code to compute the similarity/distance between two distributions of image dataset, namely the edited images and the ground truth (GT) images. This measurement assesses the overall distribution rather than a single image. In the ideal case, our goal is to suppress only the content associated with the negative target in the image while leaving the content related to the positive target unaffected. We evaluate the effectiveness of suppression by comparing the FID values of the image dataset before and after suppression. A higher FID indicates a more successful suppression effect (referred to as IFID). However, we experimentally observed that many suppres- sion methods (e.g., Negative prompt) can inadvertently impact the positive target while suppressing the negative target. Therefore, we will use IFID as the secondary metric, and Clipscore and DetScore as the primary metrics.
DetScore We introduce a DetScore metric. It use MMDetection (Chen et al., 2019) with GLIP (Li al., 2022) to detect the negative prompt object from the generated image and real image (e.g., the negative prompt object "laptop" in prompt "A laptop sofa" in Fig. 5 (left)). We refer to the prediction score as DetScore. We set the prediction score threshold to 0.7, and our method achieves the best value in quantitative evaluation in both generated and real images (see Table 1 (the fourth, seventh and tenth columns)).
The proposed method aims to focus attention on content suppression based on text embeddings, so we compare with various baselines on different
Generated-image editing experiment details.
httpps: http://hggggggggg wtt.faac..cc//oooo co/ a/stabllslllee.. -- ffuai on-v1-4
13

---

Published as a conference paper at ICLR 2024
W
MY MUNDEISTION MAN
BI
MAN
MAN NAHNU
This Man
MAN MUSTACHE BEARD
Who? The MAN Steve Dovison Blue Mana Development M..
Figure 9: We find the collected man training images contain glasses, but often do not contain the glasses label.
Presunt Woman Binding Shoaves Vieeant-voe Gogg
5D
Negative prompt
ESD
Corcapt Corcap--abiaion abiation
Forges Pig-toos
SEGA
Durs
traffc diy
baflic light SAVINS the
SD
Tanget prompt
F27
ESD
Cancept-ablation
Fangae -Me Stot
Trest Tert-Tnpuetn
SEGA
Ours
Figure 10: (Top) Comparisons with various baselines for generated images the style Van Gogh and Tyler Edlin. (Bottom) Comparisons with various baselines for generated car-related images.
types of generated images. (1) We compare method with various baselines for generating images in the style of Van Gogh and Tyler Edlin (see Fig. 10 (Top) and Table (the eleventh to the fourteenth columns)). The data related to Van Gogh and Tyler Edlin styles are sourced from the official code of ESD (Gandikota et al., 2023). This dataset comprises 50 prompts for Van Gogh style and 40 prompts for Tyler Edlin style. (2) To generate car-related images, we randomly select 50 car-related captions from COCO's validation set as prompts for input into SD. Additionally, we use multiple seeds for the same prompts. We chose to conduct experiments using car-related images for the specific reason that all baselines can effectively erase cars from the images, whereas the removal of other content is not universally suitable across all baselines. As shown in Table (the eighth to the tenth columns), our method achieves the best values on the three evaluation metrics compared with all the baselines. Quantitative comparisons to various baselines are presented in Fig. 10 (Bottom). (3) For the other generated images used in the experiments (see Fig. 6 (the fourth to the sixth columns) and Table I (the fifth to the seventh columns)). we randomly select 100 captions provided in the COCO's validation set (Chen et al., 2015) prompts, and input to the SD model.
14

---

Published as a conference paper at ICLR 2024
Inpainting method CRFill
Remove the airplane at the center
The airplane at the center
Figure 11: As an example, the instruction used in Inst-Inpaint "Remove the airplane at the center", while our prompt is "The airplane at the center". GT is obtained using the image inpainting method CRFill (Zeng al., 2021).
GQA-Inpaint dataset experiment details. Inst-Inpaint reports FID and CLIP Accuracy metrics for verification on the GQA-Inpaing dataset. FID compares the distributions of ground truth images (GT) and generated image distributions to assess the quality of images produced by generative model. In the evaluation of Inst-Inpaint, the target image from the GQA-Inpaint dataset serves as the ground truth image when calculating FID. In Table 2 (the third column), Inst-Inpaint achieves the best FID score on the GQA-Inpaint dataset, primarily because the erasure results produced by Inst- Inpaint (Fig. 5 (the second row, the sixth column)) closely resemble the ground truth (GT) images (Fig. (the second row, the fifth column)). Inst-Inpaint introduces CLIP Accuracy as a metric to assess the accuracy of the removal operation. For CLIP Accuracy, of Inst---npaint. Inst-Inpaint use CLIP as a zero-shot classifier to predict the semantic labels of image regions based on bounding boxes. It compare the Topl and Top5 predictions between the source image and inpainted image, considering success when the source image class is not in the Top1 and Top5 predictions of the inpainted image. CLIP Accuracy is defined as the percentage of success. In Table 2 (the fourth and fifth columns), ours achieves the highest CLIP Accuracy scores for both Top1 and Top5 predictions on the GQA-Inpaint dataset. This result indicates the superior accuracy of our removal process.
the official implementation
Inst-Inpaint requires obtaining the target image corresponding to the source image as paired data for training. It extracts segmentation masks for each object from the source image and uses them to remove objects from the source image using the inpainting method CRFill (Zeng et al., 2021). The resulting target image is used GT (e.g., Fig. 11 (the second column)).
There are 18883 pairs test data in the GQA- GQA-Inpaint dataset, including source image, target image, and prompt. Inst- Inst-Inpaint attempts to remove objects from the source image based the provided prompt as an instruction (e.g., "Remove the airplane at the center" in Fig. 11 (the third column)). We suppress the noun immediately following "remove" in the instruction (e.g., "airplane") and use the remaining part, deleting the word "remove" at the beginning of the instruction to form our input prompt (e.g., "The airplane the center" in Fig. 11 (the fourth column)).
Baseline Implementations. For the comparisons in section 4, we use the official implementation of ESD (Gandikota et al., 2023) 9, Concept-ablation (Kumari et al., 2023) Forgget-M--Not (Zhang et al., 2023) 11 Inst-Inpaint (Yildirim et al., 2023) 12 and SEGA (Brack et al., 2023) 13 We use P2P (Hertz al., 2022) with Attention Re-weighting weaken the extent of content in the resulting images.
Failure cases. Fig. 12 shows some failure cases.
"https://githube htt.aaas//ghhhccccoooooooooopppp
15

---

Published as a conference paper at ICLR 2024
A hamburger and fries
A girafie standing next to bamboo building
SD
Ours
SD
Ours
Figure 12: Failure cases.
B APPENDIX: EQ. 2 IN SOFT-WEIGHTED REGULARIZATION.
We take inspiration from WNNM, method used for image denoising tasks, which demonstrates that singular values have clear physical meaning and should be treated differently. WNNM considers that the noise the image mainly resides in the bottom-K singular values. Each singular value of the image patch can be updated using the formula o [Λ++] and set to 0 when the updated singular
(a++) introduced to ensure that components corresponding to smaller singular values undergo more shrinkage, where À is positive constant used to scale the singular values, and on our observation, the lop-K singular values in the constructed negative target embedding matrix X = NE c EOT, ..... gOT ipl mainly resides the content the expected suppressed embedding NEE Therefore, utilize the formula e to ensure that the components corresponding to larger singular values undergo more shrinkage.
value becomes less than 0. The weight À
small positive constant used avoid division by zero. In this paper, based
C APPENDIX: ALGORITHM DETAIL OF GENERATED IMAGE.
Algorithm 2: Our algorithm
Require: A text embeddings c ((p) and noise vector ž7. Output: Edited image Z.
è SWR(c) (Eq. 2); for 1 = T,,T - 2 = è;
// Soft weighted Regularization
do
// Inference-tine text embedding optimization
for ite =0, Ite- do
APE ANE t c0(2,,,,c) c);; APE ANE t ea(ž,, t, êx) (Eqs. 3-6);
+ min min(>giC Cpl Ani)
E. + à Ve,
end
-1-1---5 c,(2,,t,0)) (,)
end
î D(%)
Return Edited image Ī
D APPENDIX: ABLATION ANALYSIS
Verification alignment loss. As shown in Fig. 13, Cnt can mainly hold regions that we do not want to suppress. In addition, employ the SSIM metrie assess the influence of the Lpd- Increasing
16

---

Published as a conference paper at ICLR 2024
Yoshua Benglo with beard
containing covered
tree has dying leaves and apples in forest
snow on pathway
woman wearingg glasses and red hat
in some cooked broccon
SWAR
SWIR Cal
Eit (Ours)
SWR Erd
Figure 13: The regions that are not expected to be suppress are structurally altered without Lyt (third row). Our method removes the subject while mainly preserving the rest of the regions (fourth row).
Lpt raised SSIM from 0.407 (SWR+L..) 0.552 (SWR+L.++Cp)) indicating that 4n help preserve the rest of the regions. SWR+Lmi while capable of removing objects (DetScore=0.0692), tends change the original image structure and style (IFID=242.3).
Variant of weighweighted regularization. We also explore another way to regulate the target text embedding. We directly out the Top-K singular values of Σ here, Σ diag(an (ong(on,ee 01, Ang ) -pl ), and reconstruct X, which is fed into SD model generate image. Although directly zeroing out Top-K contributes to suppress the generation from the input prompt, it suffers from unexpected results (Fig. 14 (the third to the fifth columns)).
X UΣVT X VE _ c EOT 2
Attention map zero. Recent work Hertz et al. (2022); Chefer al. (2023); Parmar et al. (2023) explore the attention map to conduct varying tasks. In this paper, we also zero out the attention map which is corresponding to the target prompt, which is defined as atm2zero. As shown in Fig. 15, attn2zero method fails to suppress the target prompt in output images.
Analysis of our method in long sentences We use the object detection method to investigate the behavior of glasses when zeroing out both "glasses" and [EOT] embeddings in long sentences. We first randomly generate 1000 images using SD with the prompt pare "A man without glasses" while generating a version that zeros out both "glasses" and [EOT] embeddings. We use MMDe- tection with GLIP and the prompt "glasses" to detect the probability of glasses being present in the generated images and obtain the prediction score for "glasses". The average prediction scores of MMDetection of the two versions above-mentionel on 1000 images are 0.819 and 0.084 (see Ta- ble 4 (third row, first and second column)), respectively, which proves that when using prompt p pa "A man without glasses", zeroing out the text embeddings both "glasses" and [EOT] results in the disappearance of "glasses" in almost all generated images. It should be noted that the prediction
17

---

Published as a conference paper at ICLR 2024
man without glasses
man without glassas
A umbrella and chais beach
SD
Top-5
Top-1 10
Top-25
Top-4D
SWR (Gurs)
Figure 14: Variant soft-weighted regularization. zero out Top-K singular values Σ ( diag(ou, 01.-. Tno) We experimentally observe that naively zeroing out the singular values suppresses the target prompt, but in some cases leads to unwanted changes and expected results (the third to fifth columns).
Aree whtou Wenne
sitacofuellz
mon
wthrol
slandifeellz
man hold phone
holding
<lendottedi
Figure 15: We set the attention map of the suppressed subject (e.g. glasses) to 0. We find it fail to remove this subject (third column). Ours successfully remove the subject (second column).
score MMDetection does indicate that 81.9% of the 1000 images contain glasses. Instead, represents the probability that the image is detected as containing glasses.
To investigate the behavior of glasses with long sentences. we use ChatGPT generate descrip- tion words of lengths 8, 16, and 32 after prompt pate to form new prompts denoted as g Sais p pro++1uss and p #ro+32ws respectively. As shown in Table 4, when zeroing out both "glasses" and [EOT] embeddings, long sentences are harder to drop glasses than short sentences. This is due to the fact that other embeddings, except "glasses" and [EOT], contain more glasses information compared to short sentences. However, we observe that zeroing out both "glasses" and [EOT] em- beddings works when most of the words in the prompt correspond to objects in the image, even when the sentence is long. (e.g. "A man with beard wearing glasses and hat in blue shirt") Therefore, our method requires a concise prompt that mainly describes the object, avoiding lengthy abstract descriptions.
18

---

Published as a conference paper at ICLR 2024
Table 4: The average prediction score of MMDetection with GLIP using prompt "glasses".
Zeroing both "glasses" and [EOT] embeddings
SD
Method
pare
nere
p p Sis
p pare+ 16ws
DetScorel
0.819
0.084
0.393
0.455
0.427
Different suppression levels for soft-weighted regularization. We observe that the disappearance of the negative target (e.g., glasses in Fig. 2a (the sixth and seventh columns) occurs when the negative target information diminishes to a certain level. We perform an analysis experiment to validate this conclusion. For example,
oy control the suppression levels in soft-weighted regularization using = " * a (in Eq. 2). When y = 0, then = o, there is no change in singular values. When 7 1, then e a, which equals to Eq. 2 that used. When y oc, then ā = lim e a 0, which equals to zero out both "glasses" and [EOT] embeddings in
Fig. 2a (the sixth and seventh columns). As shown in Fig. 16, as y increases, the degree to which singular values are penalized gradually increases. When y increases certain level, which the content of glasses in both the "glasses" and [EOT] embeddings decreases to certain level, glasses will be erased.
without glasses
= Do
Figure 16: Different suppression levels for soft-weighted regularization.
Robustness diverse input prompts. As shown in Fig. 17, we showcase our robustness to diverse input prompts by effectively suppressing the content in image using multiple prompts. important to emphasize that the suppressed content must be explicitly specified in the input prompt to enable our prompt-base content suppression.
man with a-board weering giasoes and ashot blue shirt
man with a-beard
men glocpes
man axbeanic
N
Real
Figure 17: We can suppress the content using diverse input prompts.
Evaluation the attenuation factor We experimentally observed that employing an attenuation fac- tor (e.g., 0. for the negative target embedding matrix would impact the positive target (see Fig. 20). Hence, using attemuation factor leads unexpected subject changes well as changes to the tar- get subject. This is due to the fact that the [EOT] embeddings contain significant information about the input prompt, including both the negative target and the positive target (see Sec. 3.2). Further- more, the selection of factors needs to be carefully performed for each image to achieve satisfactory suppression results.
19

---

Published as a conference paper at ICLR 2024
A man with beard wearing glasses and a hat
Girl holding toothbrush
blue bird perched top of tree branch
Dog holding Aower
Waterfall
surrounded with
Real image
Negative prompt
P2P
Inst-Inpaint
Ours
Figure 18: Additional Sferrnce--guide negative target generation results. Comparisons with various baselines for real image and the target prompt.
[ЕOT] embedding in text prompts with various lengths. We observe that the [EOT] embedding contains small yet useful semantic information, demonstrated in Fig. 2c in our main paper. As shown in Fig. 2c, we randomly select one [EOT] embedding to replace input text embeddings. The generated images following this replacement have similar semantic information (see Fig. 2c). To further evaluate whether the [EOT] embedding contains useful semantic information in text prompts of various lengths, replace the input text embeddings with just one [EOT] embedding, but multiple. We use part of the [ЕOT] embedding when its length exceeds that of the input text embed- dings (short sentence), and we copy multiple copies of the whole [EOT] embedding when its length is shorter than input text embeddings (long sentence).
In more detail, we first randomly chose 50 prompts from the prompt sets as mentioned in Sec. 4. These text prompts include various syntactical structures, such as "A living area with television and a table". "A black and white cat relaxing inside laptop" and "There is a homemade pizza on cutting board". We add description words with lengths 8, 16, 32 and 56 following the initial text prompt T to obtain a long sentence, dubbed as p pro++8ws p aro+ 1ius sro++32wx and p pro4-50as 56ars respectively. For instance, when pd sre is "A living area with television and table", p ares would be extended to "A living area with a television and table, highly detailed and precision with extreme detail description".
20

---

Published as a conference paper at ICLR 2024
Girl without eerrings-
Woman without mask
Girl wearing jewelry
A car without flowers
An elderly woman seated at table holding pizzas-
SD
prompt
Negative
P2P
Inst-Inpaint
SEGA
Ours
Figure 19: Additional latent-guided negative target generation results. Examples of method and the baselines for generated image. We are able to suppress the target prompt, without further finetuning the SD model.
We use Clipscore to evaluate that the generated images match the given prompt, In this case, we test model under various length prompts (p*rs,p s p p (see Table 5 (the second and third rows)). As shown in Table 5, the generated images corresponding [EOT] embedding replacement prompts also contain similar semantic information compared the initial prompt. The degeneration of the Clipscore is small (less than 0.11), indicating that the [EOT] embedding also contains semantic information. Fig. 21 shows
pre+332s and p 55c4))
more qualitative results.
Related work also consider the [EOT] embedding for other tasks. For example, P2P manipulates the [EOT] attention injection when conducing image-to-image translation. P2P (Hertz et al., 2022)
21

---

Published as a conference paper at ICLR 2024
wcoring glasses
wearing a-hat
N
N
N
Input
Atteruation
(Ours)
(Ours)
Figure 20: SWR with attenuation factor and SVD. Note how the usage of attenuation factor leads to undesired changes in the hat of the man (the second column).
Table 5: Comparison results with original tokens and their replacement version. We evaluate it with Clipscore.
Mehod
pr
p parc+Sws
p parc+16ws
arc++32w
87c+56aus
SD
0.8208
0.8173
0.8162
0.8102
0.8058
SD w/ replacement
0.7674
0.7505
0.7479
0.7264
0.7035
black white cat relaxing inside a laptop
SD
SD replacement
p STC
P S
p Src+ gre+16ws 16ws
P SYC+ 32ws
p STc+-56uu
Figure 21: Both SD and its w/ replacement results.
swaps whole embeddings attention, including both the input text embeddings and [EOT] embedding attentions.
Taking simple mean of the [EOT] embedding We extract the semantic component by taking simple Mean of the Padding Embedding ([EOT) embedding). referred MPE. We evaluate the propsed method (i.e., SVD) and MPE. We suppress "glasses" subject from 1000 randomly generated images with the prompt "A man without glasses". Then we use MMDetection detect the probability of glasses in the generated images. Final, we report the prediction score (DetScore).
As reported in Table 6 (the third and fourth columns), we have 0.1065 MMDetection score, while MPE is 0.6266. This finding suggests that simply averaging the [EOT] embedding often fails to extract the main semantic component. Furthermore, we further zero the 'glasses' token embedding well as MPE, it still struggles extract "glasses' information (0.4892 MMDetection). Fig. 22 qualitatively shows more results.
Inference-tiee timee optimization with value regulation. We propose inference-time embedding opti- mization to further suppress the negative target generation and encourage the positive target content, following soft- weighted regularization. This optimization method involves updating the whole text embedding, which then transferred to both the key and value components in the cross-attentin layer. Therefore, our method implicitly changes the value component in the cross-attention layer.
22

---

Published as a conference paper at ICLR 2024
Table Comparison between ours and MPE. We report Clipscore.
Mehod
SD
Ours
MPR
MPE zeroing embedding
DetScore
0.8052
0.1065
0.6266
0.4892
SD (0.758)
Ours (0.000)
MPE (0.763)
MPE and zering "glasses" embedding (0.591)
Figure 22: The visualization and DetScore when using mean of the [EOT] embedding.
Furthermore, similar to the proposed two attention losses, we attempt to use two value losses to regulate the value component in the cross-attention layer:
Lu Apt + Anthnnt
Cpt V.P. -VP
(6)
L VINE
V1 NE
where hyper-parameters Apt and Aut are used to balance the effects of preservation and suppression of the value. When utilizing this value loss, find that it is hard generate high-quality images images (Fig. 23 (the third and sixth columns)). This result indicates that directly optimizing the value embedding does not work. The potential reason is that it also influences positive target, since each token embedding contains other token embedding information after CliptextEncoder,
man glacser-
wearing qlasses
(attention loss)
Inpux
(attention loss)
Figure 23: The results for generated image (left) and real image (right) of attention loss and value loss the inference-time embedding optimization.
E APPENDIX: ADDITIONAL RESULTS
User study. The study participants were volunteers from our college. The questionnaire consisted of 20 questions, each presenting the original image generated by SD, as well as the results of various baselines and our method. Users are tasked with selecting an image in which the target subject (i.e., car) is more accurately suppressed compared to the original image. Each question in the questionnaire presents eight options, including baselines (Negative prompt, P2P, ESD, Concept- ablation, Morget-Me-Not, Inst-Inpaint and SEGA) and our method, from which users were instructed to choose one. A total of 20 users participated. resulting in a combined total 400 samples (20 questions x option x 20 users), with 159 samples (39.75%) favoring method (see Fig.
23

---

Published as a conference paper at ICLR 2024
(Right)). In the results of the user study, the values for Ours. Negative Prompt, P2P, ESD, Concept- Ablation, Forget-Me-Not, Inst-Inpan Inst-Inpaint, and SEGA are 0.3975, 0.0475, 0.03, 0.1625, 0.0525, 0.01, 0.285, and 0.015, respectively.
Additional results in our approach method. Fig. 18 shows additional real-image editing results, and Fig. 19 shows additional generated-image editing results. It should be noted that the gener- ated images, as shown in Fig. (the first to fourth columns. i.c., "Girl without carring". "Woman without mask", "Girl not wearing jewelry" and "A without flowers".) -), not used for quanti- tative evaluation metrics Table (the fifth to the seventh columns), as the occasional failure of Clipscore (Hessel al., 2021) recognize negative words.
Real image results mask-based methods. Mask-based removal methods work well for iso- lated objects. However, they tend to fail for objects that closely related their surroundings. Compared to mask-based methods, our prompt-based method can automatically complete regions of removed content based on surrounding content and works equally well when removed content is closely related to surrounding content. For example, in Fig. 24, the prompt "A man with beard wearing glasses and hat in blue shirt" and the corresponding imput image show that "beard", "glasses", and "hat" are closely related to the man (Left). Our method can successfully remove "heard", "glasses", and "hat", and fill the removed area based on the context of the "man" (Med- dle), while the mask-based removal method appears very aggressive (Right).
man with a-boord weoring glassos and a-bat blue shirt
Image
Figure 24: Ours can successfully remove "beard", "glasses", and "hat" and fill in the removed area based on the context of the "man" (Meddle), while the mask- method (e.g., PlaygroundAl) fails (Right). The method reliant masking necessitates the provision of user-specified masks that define the erased areas during the inference process.
Real image results in various inversion methods. Our method combine various real image inversion techniques, including Null-text, Textual inversion mentioned in the Appendix (Textual inversion with pivot.) of Null-text, StyleDiffusion (Li et al., 2023a), NPI (Miyake et al., 2023) and ProxNPI (Han et al., 2023) (see Fig. 25).
woman wearing sucglacees and scart
Figure 25: Our method can combine various real image inversion techniques.
Implementation on DeepFloyd-IF diffusion model. We use Deepfloyd-IF based on the trans- former extract text embeddings using the prompt "a man without glasses" for generation. The generated output still includes the subject with "glasses" (see Fig. 26 (Up)), although the T5 text encoder used in Deepfloyd-IF has a larger number of parameters compared to the CLIP text encoder used in SD (T5: 4762.31M vs. CLIP: 123.06M). Our method also works very well on DeepFloyd-IF diffusion model (see Fig. 26 (Bottom)).
24

---

Published as a conference paper at ICLR 2024
A man without glasses
Deeproyd-lf
man without glasses
durs
Figure 26: (Up) Results from DeepFloyd-IF still generate the man wearing "glasses", (Bottom)
Implementation of our method on DeepFloyd-IF.
"A man without glasses" results on other diffusion models. When we use other diffusion models for image generation with the prompt "A man without glasses" input, the generated images still show the presence of "glasses" (see Fig. 27 (Top)). Our method can also be implemented other versions the StableDiffusson model, including StableDiffusion 1.5 and StableDiffusion 2.1 (see Fig, 27 (Bottom)).
man thout giasses
Snelehruum 1.3
520 MESDI 2.1
1000g CI
man WIDTOLE GANGSO
Stakooisoon L.
mudion
Figure 27: (Top) Results from StableDiffusion 1.5, StableDiffusion 2.1, Ideogram and Midjourney still generate the man wearing "glasses". (Bottom) Our method's implementation on StableDiffusion 1.5 and StableDiffusion 2.1.
F APPENDIX: ADDITIONAL APPLICATIONS
Additional cracks removal and rain removal results. As shown in Fig. 28, we present additional results for both cracks removal and rain removal. (Up) Additional results for cracks removal. (Mid- dle) We demonstrate additional results for the synthetic rainy image. (Down) Additionally, we also demonstrate additional results for the real-world rainy image.
Attend-and-Ewee similar results (Generating subjects for generated image). Attend-and. Excitet (Chefer et al., 2023) find that the SD model sometimes encounters failure in generating one or more subjects from the input prompt (see Fig. 29 (the first, third, and fifth columns)). They re- fine the cross-attention map to attend to subject tokens and excite activations. The Eq. 2 used in soft-weighted regularization utilizes the weight - to ensure that components corresponding to larger singular values undergo more shrinkage, as assume that the main singular values corresponding to the suppressed information. We make a simple modification to the weight in
co/ http://aaaaaecooaaaaa/////a///// wtwpunaamm//ttaleeeeee ffusion-vi-5 "https::/hugginge /tto/taaiiiii htt.c//ttaaallyyasstttaleeeeeeee ffusion-22
25

---

Published as a conference paper at ICLR 2024
photo with erocie
Real image
green
Curs
saiey image of a tiger in the forest
r-raiay image village streat with flawers and house
vaiey image boats in a river with buildings
sainy Image a fishing boat docked in the harbor
4ainy Image of a yollow and black bird
rainy Image coral and fish
the ocean
Real image [syntneticc image)
Ours
rainy- image of trees the forest
A rainy image of tree
A sainy Image of road and trees
A sainy image of grassy area with buildings
A sainy Image of street light
sainy image of person walking down an umbrella
Real Image image)
Durs
Figure 28: (Top) Cracks removal results. (Middle) Rain removal for synthetic rainy image. (Bottom) Rain removal for real-world rainy image.
Eq. by using g00 to ensure that the components corresponding to larger singular values undergo more strengthen (i.e., ā = 3 eo0 * a), where B = 1.2 and a = 0.001. This straightforward modi- fication, merely involving the update of text embeddings, addresses situations where the SD model encounters failures in generating subjects (see Fig. 29 (the second, fourth, and sixth columns)).
GLIGEN similar results (Adding subjects for real image). GLIGEN (Li al., 2023b) can enable real image grounded inpainting, allowing users integrate reference images into the real image. We can achieve results similar to real-image grounded inpainting using only the prompt (see Fig. 30 (the second, third, fifth, and sixth columns)). In detail, add the prompt (blue underline) of the desired subject to the prompt describing the real image and then adopt the same strategy as in the previous subsection lrrend-and-Excite similar results).
Replacing subject the real image with another. Subject replacement a common task in various image editing methods Meng et al. (2021); Parmar et al. (2023); Mokady et al. (2022): Li et al. (2023a); Tumanyan et al. (2023). We can edit an image by replacing subject with another using only the prompt (see Fig. 31 (the second, fourth, and sixth columns)). We replace the text of the
26

---

Published as a conference paper at ICLR 2024
lion with srown
lion with glesses
cat and frog
paining of elephant with
playful kitten butterlly wildlower
SD
Ours
Ours
Figure 29: Attend-and-Exxite similar results (Generating subjects for generated image).
man
with
woman wearing carrings
woman wearing carrings with glacces
woman
with www. nese
carrings with mask
tree in field A tree field
tree field with flowers
car near trees
car near trees
A car near trees with
near river
image
Image
Figure 30: GLIGEN similar results (Adding subjects for real image).
edited subject in the source prompt with the desired one create the target prompt. Subsequently, we translate the real image using the target prompt into latent code. We then apply the same strategy similar results) to obtain the edited image. For
in the previous subsection (Attend-and-Exoll example, we can replace the "toothbrush" in the "Girl holding toothbrush" image with the "pen". The DetScore with "toothbrush" of the source image is 0.790, and the Clipscore with "pen" of the target image is 0.728. We find that the brace is also replaced with the pen, as the cross attention map for "toothbrush" couples the toothbrush and the brace.
Girl halding boothisruch pen
river road between mountains
three white dimnoum sushi an brown bowl
M
Jumping jeep
faspberay cherry
red meat-balic apples
plate
Real Image
Durs
Real Image
Ours
Real Image
Ours
Figure 31: Replacing subject in the real image with another.
27


Reviews:
[{"review_id": "VQJ9HlclBE", "reviewer": "Reviewer_9uRw", "summary": "This paper addresses the challenge of controlling the generation of unwanted content in text-to-image diffusion models by introducing two methods: soft-weighted regularization and inference-time text embedding optimization. These techniques effectively suppress undesired content and encourage the generation of desired content, with positive results demonstrated through quantitative and qualitative experiments on both pixel-space and latent-space diffusion models.", "questions": "Please address questions in \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of \"soft-weighted regularization\" that effectively removes negative target information from text embeddings, improving the control over undesired content generation.\n\n2. The method is more efficient than previous methods: no need for fine-tuning the generator and no collection of paired images.\n\n3. An interesting and inspiring analysis is conducted in Section 3.2.", "weaknesses": "1. This work introduces some new matrix computations, such as the SVD in soft-weighted regularization and the attention map alignment in ITO. However, the authors do not discuss the additional computational overhead of these computations.\n\n2. In Tab. 1, the proposed method is outperformed by baselines on some metrics under certain settings. It would be better to analyze why this occurs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7BLwxhjWr5", "reviewer": "Reviewer_69SC", "summary": "This paper focuses on a sub-question of image editing, suppress the generation of undesired content, and proposes improvements from two aspects. Firstly, after conducting an analysis on text embeddings post text encoder encoding, it is concluded that EOT is low-rank and contains a large volume of prior information. Therefore, it also includes the information desired to be suppressed during the image editing process. Consequently, the first improvement proposed is to suppress the negative information in the text embedding to restrain its expression during the generation process. The second enhancement optimizes the attention map during the inference process to be as close as possible to the information to be preserved, while distancing from the unwanted information. Judging by the results, optimal outcomes were achieved on most datasets.", "questions": "1. In the comparative experiments, it would be beneficial to specifically list the time and memory consumption ratios of this method compared to other methods, as this is necessary for a more application-oriented task.\n2. In the first phase, this method uses coefficients to adjust the size of the negative information matrix to suppress the expression of negative information. If the singular value decomposition method is not employed, but instead, the entire matrix is multiplied by an attenuation factor, how would that affect the image editing results?\n3. From Table 3, it seems that 'the negative target prompt suppression loss' plays the most significant role. What would be the effect if only this loss is considered without incorporating any other improvements?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The analysis of the information components within text embedding provides certain guidance for subsequent T2Anything related research.\n2. Judging by the results presented in the paper, it has achieved a rather precise suppression of information from the text, also outperforming previous works on numerical indicators.\n3. There is no need for additional data training; any existing T2I model can be utilized.\n4. The logic of the work is clear, and the exploratory part of the experiment is plentiful.", "weaknesses": "1. From the algorithmic perspective, both improvement points are existing methods, and thus lack a certain level of novelty.\n2. This method requires gradient back-propagation during the inference process. Considering memory and time consumption, it doesn't seem as efficient as truly training-free methods like P2P.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9s9tSv1fLj", "reviewer": "Reviewer_XjcJ", "summary": "Aiming at supressing specific subjects from generated or real images, this work explores the hidden information of [EOT] embeddings in depth. Base on the discoveries, the soft-weighted regularization and inference-time text embedding optimization are proposed, enabling image editing without training or fine-tuning any large diffusion model. This light-weighted methology is effective and can be adapted to many applications, thus I believe it gives a solid and valuable contribution. The organization and phrasing of the article is also clear and easy to understand.", "questions": "With this methodology, we can remove subjects from an image or add subjects to it. Is it possible to change one subject to another in one go? For example, can we change the “toothbrush” in “Girl holding toothbrush” image to a “pen”?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This light-weighted methology is effective and can be adapted to many applications, thus I believe it gives a solid and valuable contribution. The organization and phrasing of the article is also clear and easy to understand.", "weaknesses": "The diffusion model is a hot topic in machine learning and computer vision community. The differences should be further highlighted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eWML9SEZGd", "reviewer": "Reviewer_B9jb", "summary": "This paper proposes a method to eliminate side effects from padding promt tokens. 1. First it reveals that some side effects are hidden in the padding tokens, then it proposes an SVD based method to remove the side effects from the padding tokens. 2. It proposes a negative target prompt suppression loss to weaken the attention of the negative target in the prompt, 2. It proposes a positive target prompt preservation loss to avoid mistakenly suppressing positive targets in the prompt.", "questions": "1. We know cross attention consists of the attention map and the value recombination (output = v*attn). Even if the attention map values are largely suppressed, the undesired semantics may still slip into the image features through the value recombination. Have the authors tried to address this issue?\n2. CLIPscores in Table 1 are a bit confusing. Are they the similarity between the images and negative prompts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The analysis of the hidden semantics in padding tokens is interesting. \n2. The design of the positive preservation and negative suppression losses are intuitive to understand.", "weaknesses": "*** Update after author responses ***\nThe authors addressed most of my concerns. Although I still feel the conclusions are a bit counter-intuitive, I'd like to raise my rating to 6, before seeing more cross validation from the community.\n\n====== original comments ======\n\n1. I'm not totally convinced that semantics in padding tokens have so much impact. My own empirical experience is that the padding tokens usually have very small attention scores (=> close to 0 attention probabilities) compared to meaningful tokens, and thus their semantics, if any, add little to the image features. Though, due to the large number of padding tokens, it might accumulate to somewhat significant impact, esp. when the prompt is short. This needs more systematic experiments to confirm, e.g. a diagram of the padding token impact w.r.t. the prompt length, where the prompts are randomly drawn from a pool.\n2. All the padding tokens are derived from the same input word embedding, and only differ in the positional encoding added to the word embedding. If you want to extract the main semantic component, why not take a simple mean of the padding embeddings? Why using SVD is advantageous?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)