DreamEdit: Subject-driven Image Editing
Authors: Tianle Li
Max Ku
Cong Wei
Wenhu Chen
Venue: Accepted by TMLR ()

Abstract: Subject-driven image generation aims at generating images containing customized subjects, which has recently drawn enormous attention from the research community. Nevertheless, the previous works cannot precisely control the background and position of the target subject. In this work, we aspire to fill the void of the existing subject-driven generation tasks. To this end, we propose two novel subject-driven editing sub-tasks, i.e., Subject Replacement and Subject Addition. The new tasks are challenging in multiple aspects: replacing a subject with a customized one can totally change its shape, texture, and color, while adding a target subject to a designated position in a provided scene necessitates a rational context-aware posture of the subject. To conquer these two novel tasks, we first manually curate a new dataset called DreamEditBench containing 22 different types of subjects, and 440 source images, which cover diverse scenarios with different difficulty levels. We plan to host DreamEditBench as a platform and hire trained evaluators for standardized human evaluation. We also devise an innovative method DreamEditor to resolve these tasks by performing iterative generation, which enables a smooth adaptation to the customized subject. In this project, we conduct automatic and human evaluations to understand the performance of our DreamEditor and baselines on DreamEditBench. We found that the new tasks are challenging for the existing models. For Subject Replacement, we found that the existing models are particularly sensitive to the shape and color of the original subject. When the original subject and the customized subject are highly different, the model failure rate will dramatically increase. For Subject Addition, we found that the existing models cannot easily blend the customized subjects into the background smoothly, which causes noticeable artifacts in the generated image. We hope that DreamEditBench can become a standardized platform to enable future investigations towards building more controllable subject-driven image editing. Our project and benchmark homepage is https://dreameditbenchteam.github.io/

Published in Transactions on Machine Learning Research (11/2023)
DreamEdit: Subject-driven Image Editing
Tianle Li, Max Ku", 4. Cong Wei", Wenhu Chen
University of Waterloo
Vector Institute, Toronto
(t291i,max. cong. wei, sennhuchen/luwaatrrloooooo
Reviewed on OpenReview:
Subject Replacement
Subject Addition
Subject Images
Source
Output
Source
Output
The leftmost column is the customized subject, the middle column is the Subject Replacement task, the rightmost column is the Subject Addition task. The output is the generated results by DreamEditor
Figure 1:
Abstract
Subject-driven image generation aims generating images containing customized subjects, which has recently drawn enormous attention from the research community. Nevertheless, the previous works cannot, precisely control the background and position of the target sub- ject. In this work, we aspire to fill the void of the existing subject-driven generation tasks. To this end, we propose two novel subject-driven editing sub-tasks, i.e., Subject Replace- ment and Subject Addition. The new tasks are challenging in multiple aspects: replacing a subject with customized one can totally change its shape, texture, and color, while adding target subject to a designated position in provided scene necessitates rational context-aware posture of the subject. To conquer these two novel tasks, first manually curate dataset called DreamEditBence containing 22 different types of subjects, and 440 souree images, which cover diverse secnarios with different difficulty levels. We plan to host. DreamEditBence and hire trained evaluators for enable standardized human evaluation. We also devise an innovative method DreamEditor to resolve these tasks by performing it- crative generation, which enables smooth adaptation to the customized subject. In this project, we conduct automatic and human evaluations to understand the performance of our DreamEditor and baselines on DreamEditBenche We found that the new tasks are challeng-

---

Published in Transactions on Machine Learning Research (11/2023)
ing for the existing models. For Subject Replacement, we found that the existing models are particularly sensitive to the shape and color of the original subject. When the original sub- ject and the customized subject are highly different, the model failure rate will dramatically increase. For Subject Addition, we found that the existing models cannot easily blend the customized subjects into the background smoothly, which causes noticeable artifacts the generated image. We believe DreamEdi tBench can become a standardized platform en- able future investigations towards building more controllable subject-driven image editing. Our project and benchmark homepage is attps:://dreamediitbnnnnct wwwpeaamm.itt..u..... github. io/
1 Introduction
Can you imagine your favorite pet cat going to any place that any other cat has been to, just by picking a photo for them? Can you picture your most familiar objects appearing in authentic locations around the world, simply by providing a photograph of the surroundings? The replacement or incorporation of personalized subjects into a source image can present an exhilarating challenge, and these possibilities remain largely unexplored in previous endcavors.
Current subject-driven image generation (as proposed by Ruiz et al. 2023 Gal et al. (022); Chen et al. (2023b)) primarily relies on textual prompts to synthesize images with the given subject situated in diverse visual scenes. However, text-gnided subject-driven image generation struggles to control the location pose of the subject, background scene, image layout, etc. This lack of controllability hinders its applicability in world applications. In contrast, text-guided image editing (Dong et al. 2017 Li et al.) 2020b a) can procisely control the subject location, image layout, and background scene but falls short in controlling the synthesized subjects. Therefore, we are motivated to close the gap between subject-driven generation and image editing, proposing the new task of subject-driven image editing.
In this work, we propose two novel subject-driven image editing subtasks, i.e., Subject Replacement and Subject Addition as shown in Figure [ The goal of subject replacement is to replace subject from source image with a customized subject. In contrast, the aim of the subject addition task is to add a customized subject to a desired position in the source image. These two tasks pose challenges in the following aspects: 1) the generated subject needs to maintain a similar location and pose to the source image, and 2) the generated subject should blend in with the environment realistically.
To evaluate the newly proposed task of subject-driven image editing, we first mamually curate new dataset, FreamEditBencee The dataset consists of examples with (source image, larget subject) as inputs, and the model needs to generate a target image with the target subject appearing in the source image. The target subject is represented by a few reference images. The dataset spans more than 20 subject classes and 440 carefully picked source images with diverse backgrounds. For the addition lask, we manually labeled different proper bounding boxes the source image to specify the desired location for the target subject.
Further, develop a novel iterative generation method, i.c., DreamEditor. Our model takes three steps: (1) we fine-tune the text-to-image generation model with DreamBooth (Ruiz et al. 2023) on the target subject images to associate the subject with a special token [V]. (2) we adopt off-the-she segmentation tool to locate the region for inpainting and then use DDIM inversion to re-paint the region with the target subject, which is guided by the special token [V]. (3) However, we observe that the generated subject can still retain unwanted properties from the source subject. Therefore, we adopt an iterative in-painting, where we food the generated result as input to perform another round of in-painting. After 2-3 itcrations, we can observe the re-painted subject will gradually match the target subject while fitting naturally with the background. We compare DreamEditor with several baselines with human evaluation and show that DreamEditon tor can achieve better overall scores consistently on both tasks.
After obtaining human evaluation results for DreamEditor and baselines, we observe that it is easier for the model to replace or add the target to the souree image in some cases than others. For Subject Replacement, the source subject can be seamlessly replaced by the target one when they share similar features like colors and shape, thus merely one iteration suffices to yield satisfactory output. However, when it comes to the
2

---

Published in Transactions on Machine Learning Research (11/2023)
scenario that the target subject differs dramatically from the source, it will entail longer iterations to fit to the target gradually as shown in Figure 2 Similarly, for the subject addition task, the position and posture of the added target are more strictly stipulated for some of the backgrounds. As shown in the right column in Figure 1 the generated sloth plush should be preciscly positioned on the chair rather than hovering in the air. For these cases, we note that DreamEditor can generate an irrational posture or position for the larget subject the initial iteration, but rectify it to a rational one in later iterations. Accordingly, we further divide the DreamEditBench into easy and hard levels. The sharp drop experiment results from easy to hard subsets reflect that the applied methods do struggle with the hard source images more than the easy ones. Moreover, we notice that the human evaluation results diverge largely from the automatic evaluation results. None of the automatic metrics like CLIP (Radford et al. 2021) and DINO (Caron et al. 2021 is able to reflect the success rate. Therefore, we advocale the necessity of conducting a rigorous human evaluation and plan to build a platform for more rigorous evaluation.
Iteration
Itenation 2
lteration
Figure 2: The visualization of DreamEditor to iteratively refine the generated target subject.
In a nutshell, the major contributions of this work are as follows:
1. Task Definition. We propose two novel new subject-driven image editing tasks, i.c. subject replacement and subject addition, and identify their challenges.
2. Benchmark. To slandardize the evaluation of the two lasks, we collect the DreamEdit containing same-typed subjects and background images. 3. Method. We develop DreamEditor, a novel itorative gencration method, to enable gradual adaptation from the source subject to the larget one.
2 Related Work
Conditional Image Editing with Deep Generative Models. The concept of srage-tt-Imagg trans- lation introduced where model is trained to learn a mapping between images distributions from two different domains that image from domain
be translated another domain (Isola et al., 2017 Zhu et al. 2017a| by Huang et al., 2018 Park et al.] 2020). Image-to-Image translation models are widely used in style transfer and photo enhancement tasks. However, Image-to- toae translation models lack the ability to edit locally on an image. Rescarchers have found that the generative prior can be used to manipulate parts of the generated image. To edit real images, they first need to be inverted into the latent space. This concept. known as 'Inversion', allows for detailed image editing. The idea of GAN inversion was first introduced by the method of Zhu al. (2016). and a significant effort has been made on efficient GAN inversion (Perarnau et al. 2016 Creswell & Bharath 2019 Abdal al. 2019) and applying GAN inversion to image editing applications [Bau et al.) 2019 Palashnik et al. 2021 Tov et al. 2021
Recently, diffusion models (Dhariwal & Nichol 2021] dominate the field of image generation due to their superiority in generating realistic images. denoising diffusion model learns to gradually denoise data starting from pure noise and a UNet (Ronneberger et. al., (0155) is leveraged as the network architecture. The objective of a diffusion model is to Iearn a reverse diffusion process that restores structure in data synthesize realistic data (Ho al. 2020). Texl-to-Image Diffusion Models later proposed and achieve state-of-the-art image generation (Nichol et al.) 2022; Ramesh et al. 2022 Saharia et al. 2022). In these Maoguage-Imagee Pre-training) RRadfrdd 2021 is used to bridge the gap between textual descriptions and visual content. Diffusion-basse image editing methods were also introduced in Lugmayr et. al. (2022): Hertz el. al. 20022) Mokady el al. 2022
methods, pre-trained text encoder such as the CLIP (Contrastive
et al.
Parmar et al. 2023 | Couairon et al. (2022) Kawar et al. (2022).
Subject-Drive Driien Image Editing with Personalization of Pretrained Diffusion Model. As previously mentioned, image editing methods heavily rely latent prior. Tn subject-driven editing tasks, the subject is
3

---

Published in Transactions on Machine Learning Research (11/2023)
often out of the latent distribution and thus it would be difficult to maintain subject fidelity when performing image editing. One way is to assume that a large-scale text-to-image diffusion model already understands a partial concept of the subject and that the subject can be derived from the right input text vector. Textual Inversion is done by performing a gradient update to optimize a text token vector to such that the text token vector represents the subject with a tiny portion of subject data. (Gal el. al. 2022). On the other hand, the straightforward approach is to fine-tune the large-scale text-to-image diffusion model to learn the concept of the subject. DreamBooth performs efficient fine-tuning on a large-scale diffusion model with a tiny portion of subject data (Ruiz et al., 2023 Following subject personalization works (Chen et al., 2023a Shi
et al. 2023 Tewel et al. 2023) investigate more efficient ways to perform subject-driven image generation. More recently, there are concurrent works like BLIP-Diffusion (Li et al. 2023a), CustomEdit (Choi et al. 2023) and PhotoSwap (Gu el. al. 2023) working on the task of subject swapping. Our work differs from these works in two aspects: (1) the previous work only focuses on subject replacement, (2) our method can work on cases where the souree and target subjects differ significantly. Our itcrative method can modify the appearance ileratively to increase subject fidelity. In the context of subject addition, the objectives and approaches significantly deviate from those employed in image composition (Niu et al.) 2021 Zhang et al. 2021). Contrary to the mere placement and subsequent harmonization typical in image composition, subject addition is inherently a generative process. This process not only introduces the subject into the background but also dynamically adjusts various attributes of the subject, such as posture, expression, and viewing angle, to achieve seamless integration with the existing background. This adaptation executed automatically with diffusion process, underscoring the advanced capabilities of this approach in generating contextually coherent images.
3 Preliminary
Text-to-Image Diffusion Models. A diffusion probabilistic model is a latent variable model that is trained to learn the image distribution by reversing the diffusion Markov chain. Specifically, are interested in large text-to-image diffusion models pre-trained on large-scale text-image pairs (Rombach et al. 2021). The model consists of a CLIP (Radford et al. 2021) text encoder r, and a U-Net (Ronneberger et al. 2015) based conditional diffusion model cg. Given a text prompt Q, the text encoder I generates a conditioning vector I'( With a randomly sampled noise e ~~ (0, I) and the time step t, we can get a noised image or latent code 24 = a'x + drC where x is the input image, ag and o are the cocfficients that control the noise schedule. Then the conditional diffusion model is trained with the denoising objective:
E. - - t, I(Q)
(1)
Where is trained to predict the noise condition on the noisy latent 2,, text prompt Q, and the time step t. At inference, given noise latent 27, the noise is gradually removed by sequentially prodicting it using for T steps. A more detailed description is given in the supplementary material.
Subject-Driven Text-to-Image Generation. Subject-driven
generation models (Ruiz et al. 2023) often
fino-tune a pre-trained text-to-image generation model co on a small set of demonstrations C, of subject s, which contains a set of image and text description pairs C, {(x,, (+) where v; means the 7th image of subject 8, while text description Q, of image i; contains special text token V. that is bound to the subject (Ruiz et al. 2023 To customize diffusion model E0,, we optimize the objective:
E, co, (2,,,,
(2)
Where eo, is trained to always denoise for the images of the subject when condition on a text description that contains the special text. Loken V,
DDIM Inversion for Real Image Edition. Text-guided editing of a real image with generative models often requires inverting the given image and textural prompt. Essentially, this means identifying an initial noise vector that can generate the given real image, while maintaining the model's ability to edit. We
4

---

Published in Transactions on Machine Learning Research (11/2023)
leverage the DDIM inversion scheme Dhariwal & Nichol 2021 2p = onto latent variable 27.
Song et al. 2021 b to encode given image
1 a1+1
(v
60 (z(
(3)
Based on the assumption that the ODE process can be revered in the limit of small steps, øL is computed by reversing the deterministic DDIM sampling. c0(2,, t) is an unconditional model or equivalently a conditioning model with text ie. 6,
4
Benchmark
4.1 Problem Definition
Subject Replacement The Subject Replacement task aims to generate a new image by replacing the subject in a source image with the subject from a set of target subject images, guided by a general description. Formally, given the target subject images set C., text prompt Q' describing the replaced image, and an image containing the same-typed subject to be replaced Srep: the lask of Subject Replacement is targeted at obtaining transformed image R SQ(C.. ,", Srep), so that the feature of the replaced subject in R aligns with the target subject in C.2 and simultancously, the background in Srep is still preserved after transformation. As shown in the first row of Figure | we replace the heart cartoon in the source image Srep with the red evil cartoon in the target subject images C, and preserve the background information in Srep-
Subject Addition The Subject Addition task seeks to place the target subject at a designated position in background image. Similar to the Subject Replacement task, given a set of images of the target subjects C., a text prompt Q' describing the after-placement image, a suitable background image Sadd for the customized subject, and a manually labeled bounding box Bbox specifying the addition position, the task can be formulated formally as A T(C,, ensure the rationality of the interaction between the generated subject and the context. For instance, the grey sloth plush in the last column of Figure the air. In addition, the background details are necessitated to be preserved.
Bbox). In the Subject Addition task, it is crucial to
[ shoull always be placed on the chair other than hovering in
4.2 Data Collection
To standardize the evaluation of the two proposed tasks, we curate a new benchmark, i.c. DreamEdi tBench, consisting of 22 subjects in alignment with DreamBooth Ruiz et al. 2023) with 20 images for each subject correspondingly. For the subject replacement lask, we collect 10 images for each lype, which include same- typed source subjects in diverse environments as shown in Figure [aa| The images are retrieved from the internet with the search query "a photo of [Class name]", and the source subject should be the main subject in the image which dominates major part of the photo. For the subject addition task, we collect 10 reasonable backgrounds for each type of subject as shown in Figure Gb In the meantime, specific location the target subject should be placed with a bounding box in the background. To collect the specific backgrounds for each subject, we first brainstorm and list the possible common environments of the subjects, then we search the listed keywords from the internet to retrieve and pick the backgrounds.
manually designate the
4.3 Dataset Analysis
We notice that for Subject Replacement, some of the collected source images share similar features with the target, while others differ dramatically from the target in color, texture, or shape. According to our observation, the significant distinction in feat ures can result in the degradation of model performance. For instance, it easier to adapt to the target teapot for the source the left column than the right in Figure Ga] as the left teapots share a more similar color to the target. Therefore, we divide the collected source images into 150 easy ones and 70 hard ones. For Subject Addition, the model likely generate the subject more reasonably in some of the backgrounds than the others. For instance, if the view and style of
5

---

Published in Transactions on Machine Learning Research (11/2023)
the background images diverge far away from the provided set of subjects, it is hard for the stable diffusion model to synthesize target subject adapting to the unseen situation (e.g. the right column of candle in Figure 6b is classified to hard as it requires the generation of the top view). Moreover, another example is the bear plush in Figure 6b, the bottom of the generated bear plush is strictly mandated to be at the bottom edge the bounding box, otherwise, makes no sense place a plush the air or below the bay window. Accordingly, we divide the 220 backgrounds into 122 easy-typed images and hard ones.
5 Methodology
M
(3)
Segmentation
Mask Diation
DDIM Encoding
Encoding
(1 - M,
DDIM Denotsing
DOIM Decoding
$
Conditioned on
T photo of [V] dog.
Segmentation
Mask Dilation
Segmentation
Dilation
Segmentation
Mask Dilation
DOIM Encading
Encoding
DDIM Do Denaising
(2)
DOIM
DDIM Denoising
Iteration
Figure 3: DreamEditor generates the results iteratively. The output of the last iteration will serve as the input for the next. In each iteration, it leverages the dilatod mask subject segmentation and the specialized prompt to guide the DDIM inversion and gradually in-paint a subject more similar to the target one.
To effectively resolve the two tasks, build novel system DreamEditor customized subject. DreamEditor can realize customized subject replacement or addition with the following steps: 1) Given à set of the target subject images, we fine-tune the text-to-image model with Dream Booth Ruiz et al.] (2023 to associate the target subject to a special token. 2) We initialize the source image with two different strategies. 3) We leverage DDIM inversion guided by the segmentation mask and target text prompt generate the initial result. 4) We repeat the process itcratively, where the output will become the input for the next. round. An overview of the proposed DreamEditor is shown in Figure
iterative adapter to the
Initialization Before inpainting, we deploy different initialization strategics. We use the source image Srep for replacement directly for the first type of initialization as it has already contained a same-typed subject as the target. However, for Subject Addition, is challenging to directly generate a context-aware target subject the designated Bhor. Instoad, we devise an infill-hhen-eneerate schema. With description of the target subject in the text, GLIGEN-infil can in-paint a subject similar the target into the desired location in the source image, which is named as GLIGEN initialization merely for Subject Addition. The

---

Published in Transactions on Machine Learning Research (11/2023)
weill-then-genettee schema enables easy adaptation from the "semi-customized" subject to the target one, as they share similar features. Moreover, the infilled subject tends to establish natural interaction with the context as it is trained with large amount of image and bounding box pairs (Li et al. 2023b). Meanwhile, we propose COPY as another initialization strategy to preserve the characteristics maximum degree. For COPY initialization ion, we segment the target subject from one of the target subject images Cs, and re-scale it to match the size of the source subject in Srep or the Bhor in Sadd- Then we replace the corresponding pixels in Srep or Sadd with the re-scaled target subject segmentation. We briefly define the initialization step as T6 = I(Stask: Cs), where Seask E [rreps Sadd]
the target to the
In-painting Given an input image To we segment the source subject from the background with Segment- anything (Kirillov et al. 2023) to obtain the mask M Then we dilate M to Mc(*) with morphological transformation and dilation kernel m as M2 D(z (6) m), so that the painted subject can have more space to harmonize with the background. To in-paint the target subject at the dilated mask, with the fine- tuned model c0,, we first encode the input z into an intermediate noised latents I&k using DDIM, where (1) conditioned on prompt Q* with special token for the target subject to
0 << k(t)) < T. Then we denoise oi obtain = where 0 <t < k. To preserve the background and solely modify the subject, we leverage the dilated segmentation mask (i) to decide the partial pixels guided by Q-, For the pixels outside MM we replace them with the recorded latents in DDIM encoding to reconstruct the background, which can be (i)
formulated as =; (()
(1 (1-M;-0) +xz + M zt Then we can ultimately obtain the denoised image =0
shown in Figure 3 We formulate the whole in-painting process as =n
(f) P(ea, (8) M, (5) Q*).
Iterative Generation According to our observation, the generated subject from the first round may still retain the features from the source subject or cannot interact with the context rationally. Therefore, it motivates us to propose the iterative generation approach as shown the orange block in Figure 3 In detail, DreamEditor (N) will treat the output from the last iteration as the input to the next iteration, i.c. (i+1)
(6) and repeal the in-painting process for all the N iterations. It is worth noting that there is an 40 =
iterative change of (, from round to round. The shape of the segmentation mask will gradually adapt to the target subject so as the generated subject. Briefly, the process is expressed as Algorithm !
6 Experiments
6.1
Experiment Setting
We use stable diffusion version 1. 1.4 to fine-tune it with DreamBooth (Ruiz et al. 2023). For all of the 30 subjects involved in DreamBench (Ruiz al. 2023), we set iteration number N 5 and the mask dilation kernel = 20. The encoding ratio / is set to be 0.8 for the first iteration and decreases lincarly as //T k1 - i * 0.1. We leverage the Gligen inpainting pipeline implemented in the official code repository for the initialization of DreamEditor for Subject Addition task. Meanwhile, we leverage the to obtain the segmentation mask of
based Segment-anything implemented at lang-segmentt-nyy the source subject. All the experiments are run on a single A6000 GPU.
6.2 Baseline Methods
We compare the proposed DreamEditor with the following baselines to show its effectiveness.
DreamBooth (Ruiz al.j 2023) We finetune the stable diffusion model with target subject images C. and the source image Stask to encode the target subject and the source image by special tokens V and Y*. Then we prompt the model with photo of V* fclass name] in Y backyround. to obtain the results.
Ihttp:://uugiingficee c http:/agggggaaaetooooooooo//aaaa CCompiiis/saaallddffttsoor. Phttpe://githhub.com /gligen httpa///....mmlleeeee///////e//e /fff///ssuueeeet////gggggg gligen
3 http:///gghhuuucmmmmm/mmmmmeeeee

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm 1: DreamEditor Algorithm
Input: A source image Stask, a target prompt Q" fine-tuned stable diffusion model 6,,, a set of target subjoct images Ca mask dilation kernel m and iteration number N. A list edited images Rsub-
Output:
20
I(Saosk: C.):
Function P(Ea, 2o (6)
M2 Q"")
(+)
DDI M Encode(Eg, (9)
T0 ,....
(1)
(1)
2k = Ik
for 1 ---1, =-1,k 2.... ...... do
2 (i) DDIMSamples
9111 Q"):
4 = (1-M) (1-, (5)) ) + (
21 =2 -(1)
10
11
end
for i == 1,2, 2, N do
13
(1 (mm (i)
m):
(1) P(co,, To M Q"):
14
20
(1+1)
(1)
15
To
20
16
end
I Red
[ 1 :....:
(1) (2)
(N)
Return Rgub
Method
Initialization
Dino-backt Dino-subt
Clipl-subt
Clipl-backt
Overall
Subject Replacement
DreamBooth
0.718
0.481
0.867
0.744
0.608
Customized-DiffFae
0.619
0.878
0.834
0.915
0.790
CopyPaste
COPY
0.775
0.822
0.874
0.896
0.819
PhotoSwap
0.584
0.842
0.824
0.893
0.757
CopyHarmonize
COPY
0.779
0.757
0.879
0.855
0.786
DreamEditor (1)
COPY
0.753
0.779
0.875
0.877
0.791
DreamEditor (5)
COPY
0.765
0.799
0.882
0.882
0.807
DreamEditor (1)
0.546
0.664
0.763
0.853
0.646
DreamEditor (5)
0.564
0.667
0.77
0.855
0.655
Subject Addition
DreamBooth Customizedi-iifFFaa
0.699
0.151
0.860
0.604
0.312
GLIGEN
0.577
0.641
0.818
0.789
0.653
CopyPaste
COPY
0.444
0.666
0.693
0.826
0.579
PhotoSwap
GLIGEN
0.550
0.684
0.808
0.809
0.660
CopyHarmonize
COPY
0.441
0.698
0.690
0.844
0.592
DreamEditor (1)
COPY
0.684
0.807
0.849
0.879
0.775
DreamEditor (5)
COPY
0.664
0.773
0.841
0.850
0.751
DreamEditor (1)
GLIGEN
0.579
0.773
0.818
0.870
0.717
DreamEditor (5)
GLIGEN
0.632
0.798
0.838
0.874
0.753
Table 1: Automatic Evaluation Results of DreamEditor and Baselines on Subject Replacement and Addition.
Customized-Diff.di (Couairon et al., 2022) DiffEdit can automatically generate the mask be edited by contrasting predictions conditioned on source and target prompts, so that it can realize edit- ing without changing the background. We replace the diffusion model in DiffFdit with fine-tuned one by
8

---

Published in Transactions on Machine Learning Research (11/2023)
Method
Initialization
Subjectf
Backgroundf
Realistict
Overall
Subject Replacement
DreamBooth
0.543
0.0
0.707
0.072
Customized-Diine
0.21
0.828
0.668
0.488
CopyPaste
COPY
1.0
0.148
0.123
0.263
PhotoSwap
0.15
0.773
0.663
0.425
CoryHnrmonize
COPY
1.0
0.552
0.147
0.433
DreamEditor (1)
COPY
0.778
0.407
0.52
0.548
DreamEditor (5)
COPY
0.817
0.505
0.54
0.606
DreamEditor (1)
0.532
0.760
0.557
0.608
DreamEditor (5)
0.630
0.800
0.582
0.664
Subject Addition
DreamBooth
0.477
0.0
0.635
0.067
Customized- DiffEddi
GLIGEN
0.288
0.302
0.252
0.280
CopyPaste
COPY
0.983
1.0
0.033
0.319
PhotoSwap
GLIGEN
0.21
0.562
0.305
0.33
Harmonine
COPY
0.983
1.0
0.295
0.662
DreamEditor (1)
COPY
0.635
0.978
0.265
0.548
DreamEditor (5)
COPY
0.633
0.973
0.393
0.623
DreamEditor (1)
GLIGEN
0.287
0.99
0.427
0.495
DreamEditor (5)
GLIGEN
0.478
0.972
0.528
0.626
Table 2: Human Evaluation Results of DreamEditon tor and Baselines on Subject Replacement and Addition.
Dooth Then we modify the source and target prompts to photo of a [class name/ and photo of a V* [class name) correspondingly to enable the generation of the customized subject. For the Subject Addition, we initialize the input the same way as DreamEditor.
Copy-Paste To have a naive but fundamental baseline, we directly use the output result from COPY initialization. Copy-Paste basically leverages Segment-anything to segment the target subject from one of the provided larget subject images set C, as "copy" and re-scale it replace the corresponding pixels the segmentation box in source image Srep the labeled bounding box in Sadd.
PhotoSwap (Gu et al. 2023) Similar with DreamEditor, PhotoSwap first learns the target concep with DreamBooth, and then swaps into the target image with attention swap during diffusion process for replacement task. We employ GLIGEN initialization for PhotoSwap in addition task enable head-to- head comparison.
CopyHarmonize Harmonize To have a intuitive and straightforward baseline, for subject addition task, we copy-paste for the target subject first, and utilize a Harmonizer (Ke et al., (022) to contextualize the copied subject in the source image. For subject replacement task, the source subject is segmentod first, and we conduct in- in-painting on the segmentation mask to fill up the background and repeat the same process with addition task.
6.3 Main Results
We run the experiments for DreamEditor and baselines on DreamEditBench. We present results in Figure 5 and Figure | To make more comprehensive comparison, we evaluate the results with both an automatic matrix and rigorous human evaluation.
Automatic Evaluation Results For both replacement and addition tasks, it crucial maintain delity both subject and background. Therefore, we evaluate the fidelity of the generated results with DINO Caron et al. 2021 and CLIP-T (Radford et al. 2021 scores for both the subject and the back-

---

Published in Transactions on Machine Learning Research (11/2023)
Subject
Source
DreamBooth
Silomized-DDifEaan
Copy-Paste
PhotoSwap
CopyHarmonize
DreamEditor
Figure 4: Results Subject Replacement Task Compared with Baselines
ground. The two matrices measure the average cosine similarity between the generated and real images with VITS/16 DINO embeddings and CLIP embeddings. We define Dino-sub, Dino-back, ClipI-sub, ClipI-back the measurements for the subject and background separately. For Dino-sub and Clipl-sub, it deploys all
10

---

Published in Transactions on Machine Learning Research (11/2023)
Subject
Source
DreamBooth
Customized-Diine Eit
Copy-Paste
PhotoSwap
CopyHarmonize
DreamEditor
Figure 5: Results Subject Addition Task Compared with Baselines
the images from the provided target subject images set C, as the real images. And for Dino-back and CilpI- back, it leverages the source image Stask the real reference. To disentangle the subject from background vice versa, the subject is segmented from the background for both the reference and generated images
11

---

Published in Transactions on Machine Learning Research (11/2023)
with off-the-shelf segmentation tool. The segmentation mask or its complementary part will be filled with white for background-orientee or subject-oriented evaluation respectively. The overall score is defined as the average of the geometric mean of the previous four measurements over all the examples. The automatic evaluation results are demonstrated in Table 1 The number in the brackets after DreamEditor refers to the number of iterations.
Human Evaluation Results We conduct a human evaluation of the results for DreamEdit to report a more realistic performance. We define three aspects to evaluate the quality of the generated results: 1) Subject Consistency: How well do the generated results preserve the feature of the customized subject in the provided set of images for the target? 2) Background Consistency: How well do the generated results preserve the background information in the source inputs? 3) Realism: Assess the overall realism and naturalness of the generated image. For each of the perspectives, we ask three experts to label the results with scores 0, 0.5, or 1. The detailed evaluation standard is described in Appendix A.2 For methods with several iterations, i.e DreamEditor (5), the expert is asked to decide the best iteration first and evaluate the result for the best iteration. After obtaining the labeling, we average each aspect over all the examples to get the final results as shown in Table 2 The overall score obtained by calculating the geometric mean of the scores from the three perspectives. Generally, the proposed DreamEditor (5) obtains the best overall score among all the methods, surpassing the best baseline Customized- subject addition task, though CopyHarmonize slightly performs better than DreamEditor (5) due to almost full score for Subject and Background consistency, DreamEditor (5) outperforms it by 0.233 in Realism score, indicating the higher fidelity of the results generated by DreamEditor. The detailed analysis for the comparison of realism is demonstrated in Appendix A.4 Moreover, it is imperative to note that the human evaluation results diverge dramatically from the automatic ones, which reffects the necessity of conducting a rigorous human evaluation for a more fair evaluation among methods.
by 0.176 in Subject Replacement. For
6.4 Result Analysis
As shown in Figure and Figure though with relatively high subject consistency and realism, DreamBooth can hardly preserve the background from the source. Bustomized-Difffal can merely either preserve the background or adapt to the target subject but can barely handle both. It is also difficult for PhotoSwap to adapt to the features of the target subject. CopyPaste although preserves both subject and background consistency, it fails to obtain realistic results in most cases, as it cannot generate context-aware subjects. The realism issue is alleviated in Caryooonniize baseline, but still yields relatively low realism score. DreamEditor with COPY as initialization further mitigates the limitation of CopyHarmonize, leading to more realistic results. It works well when the posture of the segmented subject matches the target context (e.g. monster toy in Figure 5). but fails otherwise (e.g. backpack in Figure 5 DreamEditor without initialization for replacement or with GLIGEN initialization for addition balances subject, background, and realism to some extent, thus obtaining the highest overall human evaluation scores.
It can be observed from Table ! the models initialized with COPY mechanism can achieve better scores in subject-oriented evaluation measurements. While method with weakest background preservation mechanism like DreamBooth performs worst in the background -rrennted evaluation matrix, which matches our intuition.
Besides the main results, we conduct a more fine-grained analysis on the sub-division of DreamEdi t As stated in Section 4.3 we divide the collected dataset into easy and hard subsets. We measure the overall results for the two divisions separately for each task shown in Table 3 For the majority of the cases, the models achieve better performance on the casy sub-division than the hard one by about 0.1.
In addition, we analyze the effect of the number of ilerations in an iterative generation. As shown in Table [ for Subject Replacement, the Subject Consistency will consistently increase with a larger N. Nevertheless, the Background Consistency and Realism will fluctuate and even decrease with more itcrations. Generally, it reflects a trade-off between subject consistency and the other two aspects with the Parameter.. Thus, if we can pick the best iteration for each example individually, it can achieve the best overall performance.
12

---

Published in Transactions on Machine Learning Research (11/2023)
Subject Replacement
Subject Addition
Method
Initialization
Easyt
Hard
Easyt
Hard
Ciitomize--Diffddde
/GLIGEN
0.511
0.418
0.303
0.250
PhotoSwap
/GLIGEE
0.511
0,418
0.352
0.303
CoryHarmonize
/GLLIGE
0.511
0,418
0.685
0.632
DreamEditor (1)
COPY
0.551
0.529
0.600
0.480
DreamEditor (5)
COPY
0.600
0.612
0.663
0.374
DreamEditor (1)
-/GLIGEN
0.648
0.515
0.562
0.470
DreamEditor (5)
-/GLIGEN
0.702
0.567
0.676
0.563
Table 3: Comparison of Human Evaluation Results on the Easy and Hard split of DreachditBBnchh
Iteration
Subjects
Background
Realistic
Overall
N=1
0.531
0.763
0.556
0.608
N=2
0.585
0.713
0.490
0.589
N=3
0.601
0.693
0.456
0.574
N=4
0.613
0.681
0.448
0.571
N=5
0.625
0.681
0.453
0.577
N-Best
0.630
0.800
0.582
0.664
Table 4: Human Evaluation Scores with Different Iteration Number for Subject Replacement Task.
Limitations
DreamEditor can either fail or require a large iteration number to adapt when the target subjects differ too much from the source subject. Besides, duc to the iterative gencration, the crror in roconstruction from DDIM inversion will be propagated and leads lo blurry background after N iterations, especially for large N. addition, the performance of DreamEditor is highly affected by the segmentation model and GLIGEN in-painting, if the two models fail in the first place, it is very likely that DreamEditor will also fail ultimately.
7 Conclusion
In this work, define two novel subject-driven image editing tasks, iee. Subject Replacement and Sub- ject Addition. To standardize the evaluation of the two proposed lasks, we collect the CreamEditBenchh containing 440 same-typed subjects and background images for 22 subjects in total. Meanwhile, devise DreamEditon tor to realize gradual refinement towards target subjects with itcrative generation. The systematic human evaluation shows the advanlage of our method over the other baselines on the overall performance.
Broader Impact
The precise control over subjects and backgrounds in image generation, as demonstrated in this research, can contribute to the ongoing efforts to detect and mitigate deepfake content. By improving our understanding of how subjects can interact with their environment, this research can aid the development more effective deepfake detection algorithms. In addition, leveraging the proposed method, well-controlled and relatively high-quality fake images can be synthesised automatically, which can be leveraged to train better deepfake detector. By advancing the field of subject-driven image generation, this rescarch indirectly results in the development of counternneaures against malicious uses of deepfake technology. Moreover, it raises awareness among researchers and policymakers about the evolving landscape of Al-generated content, stimulating discussions on potential regulations and safeguards.
13

---

Published in Transactions on Machine Learning Research (11/2023)
References
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun- Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH), 38(4), 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervieed vision transformers. 2021 TEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 9630 9640, 2021.
Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Identity- preserving discntangled tuning for subjoct-driven text-to-image generation, 2023a.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Was Wei Chang, and William W. Cohen. Subject-driven text-to-image generation via apprenticcship learning. ArXiu, abs/2304.00186, 2023b.
Jooyoung Choi, Yunjey Choi, Yunji Kim, Junho Kim, and Sungroh Yoon. Custom-edit: Text-guided image editing with customized diffusion models. arXiv preprint orXiv:2305.117 15779, 2023.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-baser semantic image editing with mask guidance. ArXiv, abs/2210..1122 2022.
Antonia Creswell and Anil Anthony Bharath. Inverting the generator of a generative adversarial network. IEEE Transactions on Neural Networks and Learning Systems, 30(7):1967-1974, 2019. doi: 10.1109/
TNNLS.201S 201...2111.228199.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 8780-8794. Curran As- sociates, Inc., 2021. URL https::/ hewpisseerrippccom neurips.ce/paper files//pper/222//iile/ htt.a//csseaaaae7770eeeeaaaaaaa. Paper. pdf
Hao Dong, Simiao Yu, Chao Wu, and Yike Guo. Semantic image synthesis via adversarial learning. In Proceedings of the IEEE international conference on computer vision, pp. 5706-5714, 2017.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.
URL htttps://arxiv.ot wwwp:/ivvwrgg//////// 2tt.b//22000.010
Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifci Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, el al. Photoswap: Personalized subject swapping in images. arXio preprinl arXiv:2305. 182266605.18286, 2023.
Amir Hertz, Ron Mokady, Jay M. Tenenbaum, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt- to-prompt image editing with cross attention control. ArXiv, abs/2208.0100 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc., 2020. Utt.httpss//rroccdda...ooo cttppe....aaappp.....o//oaaaaaaa
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised imago-to-image trans- lation. In European Conference on Computer Vision (ECCV), 2018.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, 2017.
14

---

Published in Transactions on Machine Learning Research (11/2023)
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Hui-Tang Chang, Tali Dekel, Inbar Mosseri, and Michal Trani. Imagic: Text-based real image editing with diffusion models. ArXiu, als//210.099276, 2022.
Zhanghan Ke, Chunyi Sun, Lei Zhu, Ke Xu, and Rynson W.H. Lau. Harmonizer: Learning to perform white-box image and video harmonization. In European Conference Computer Vision (ECCV), 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02644 2023.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr. Manigan: Text-guided image manipu- lation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1880-7889, 2020a.
Bowen Li, Xiaojuan Qi, Philip Torr, and Thomas Lukasiewiez. Lightweight generative adversarial networks for text-guided image manipulation. Advances in Neural Informalion Processing Systems, 33:22020-2222 2020b.
Dongxu Li, Jumnan Li, and Steven CH Hoi. Blip-diffusion Pre-trained subject representation for controllable text-to-image generation and editing. arXiv preprint arXiv:230..1 14720, 2023a.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pullern Recogrition (CVPR), pp. 22511 22521, June 2023b.
Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint:
Inpainting using denoising diffusion probabilistic models. 2022 IEEE/CVF Conference on Computer Vision and Pullern Recognition (CVPR), pp. 11451-11461, 2022.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models, 2022.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Tya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text- guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th Internalional Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16784-16804. PMLR, 17-23 Jul 2022. URL https://prroceedingg mlr wwwps//veee2.. www.////nno22aaaa. html
Li Niu, Wenyan Cong, Liu Liu, Yan Hong, Bo Zhang, Jing Liang, and Liqing Zhang. Making images real again: A comprehensive survey on deep image composition. ArXiv, abs/2106. 14490, 2021. URL https://pp..smmmttccoiii www.nticscholaatttt....ea Spsaattticccola..oooooooooppooos amgg/oorpusD::2565577
Tacsung Park, Alexci A. Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-
to-image translation. In European Conference Computer Vision, 2020.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. ArXie, abs/2 22222222020222 2023.
Or Patashuik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.
October 2021.
Guim Perarnau, Joost van de Weijer, Bogdan Raducanu, and Jose M. Álvarez. Invertible Conditional GANs for image editing. In NIPS Workshop on Adversorial Training, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Kruegor, and Nya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. ArXiv, abs/2204.06122 2022.
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 10674-10685, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. ArXiv, abs/ 1505. 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning lext-to-image diffusion models for subject-driven generation. In Compuler Vision and Pattern Recognition Conference, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Bureu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David J. Fleet, and Mohamnad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205.11482) 2022.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized texi-to-inage generation without test-time finetuning, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2021.
Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to- to---to-mmae personalization. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
Omer Tov, Yuval Alalff, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40:1 14, 2021.
Bo Zhang, Li Niu, and Liqing Zhang. Image composition asscssment with saliency-augmenned multi-pattern pooling. ArX abs /bs/2.0..33133, 2021. URL httpss///pp..eenatttcccooooa htt.:///a.....a.ooooooooooooooaa
Jun-Yar Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. Generative visual manipulation on the natural image manifold. In Proceedings of European Conference on Computer Vision (ECCV), 2016.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEFE International Conference on, 2017a.
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shecht- man. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems, 2017b.
16

---

Published in Transactions on Machine Learning Research (11/2023)
A Appendix
A.1 Dataset Examples
We demonstrate part of our collected data Figure The left column is examples of easy subset and the right column is that for hard subsets.
A.2 Human Evaluation
To standardize the conduction of a rigorous human evaluation, we stipulate the criteria for each measurement follows:
Subject Consistency How well do the generated results preserve the feature of the customized subject in the provided set of images for the target subject?
Score The subject is consistent and accurately represents the intended subject, closely matching the visual characteristics and appearance.
Score 0.5: The subject partially resembles the intended subject but lacks consistency in some visual attributes, such as facial features, body proportions, object shapes.
The subject in the generated image bears little resemblance to the intended subject or exhibits significant distortions in visual characteristics and appearance.
Score 0:
Background Consistency How well do the generated results preserve the background information in the provided source inputs?
Score 1:
The background consistent and scamless, demonstrating a high level coherence with the given context or scene.
Score 0.5:
The background has some minor inconsistencies or artifacts, but they are not prominent enough to significantly affect the overall coherence of the image.
Score 0: The background shows significant inconsistencies artifacts that are visually distracting and do not align with the given context or scene.
Realism Assess the overall realism and naturalness of the generated image.
Score 1: The generated image is visually convincing and closcly resembles real photograph, hibiting realistic lighting, shadows, texture delails, and overall visual coherence.
The realism of the image is somewhat compromised, showing minor visual flaws or inconsistencies that may raise suspicion but do not strongly detract from its overall appearance.
Score 0.5: Score 0.5:
Score 0: The generated image appears highly artificial and unrcalistic, with noticcable visual flaws, unnatural lighting, or inconsistencies that make it easily identifiable as a generated image.
If the score of an aspect is 0, use 0.001 approximation. We also provide example for each score with respective to different criteria as shown in Figure 7
A.3 Failure Cases
The results are generated by DreamEdi tor with unified parameters for all the examples without subtly tuming for any specific subject. Therefore, there can be various failure cases different stage of the DreamEditor pipeline shown in Figure 7
17

---

Published in Transactions on Machine Learning Research (11/2023)
Теapot
Shiny Sneaker
Easy
Hard
Easy
Hard
Robot Toy
Duck Toy
(a) Benchmark Examples for Replacement Task
Candle
Bear Plushie
Easy
Hard
Easy
Hard
Berry Bowl
Can
(b) Benchmark Examples for Addition Task
Figure 6: DreamEdi tBench Examples
18

---

Published in Transactions on Machine Learning Research (11/2023)
Subject Consistency
Background Consistency
Realism
A
0.5
0
Figure 7: Detailed human evaluation criteria for different measurements.
For instance, in subject replacement task, it is hard for DreamEditor to adapt to the target pink backpack from the green source they share few similar features, and large iteration number N may be required to transform it completely. Secondly, it will also lead to failure if the segmentation step fails at the very beginning as shown in the teapot example. The inaccurate segmentation map can result in the shift of the subject position and unnocessary artifact added to the background.
For subject addition lask, in addition to the failures from replacement, the initialization step can play vital role. It is more likely fail if the subject generated by GLIGEN differs far from the target subject like the black backpack in the 4th column of Figure 7 Moreover, the subject generated by GLIGEN may not correctly interpret its relationship with the context. The cat should be real cat sitting at the carpet other than the painting, while the bear plushie should be put on the bay window instead of suspending in the air. These failure cases may be rectified with more specific parameter tuning for each subject or better sogmentation and in-painting model.
A.4 Comparison with naive baselines
According to the human evaluation results for DreamEditor and CopyHarmonize, we conduct deeper analysis on why our method can not outperform the naive Copy Harmonize baseline in subject addition task for the overall score. As we can observe from the left column of Figure when adding the target subject to the source image, CopyHarmonize can achieve exact match with the target subject and preserve the background consistently,
pastes the segmented subject directly to the background. However, though high scores Harmonnn obtain in subject and background consistency, the copy-and- harmonize pipeline will fail the realism score when the posture or angle of the view play a vital role of making the images look realistic. For instance, as in the first row of the left column in Figure g the shoulder straps of backpack should not stand up when placed in the chair, and it also should not be floating above the chair. In the second row of CopyHarmonize, the angle of the view the toy car is not consistent with the that the table from background. Instoad, a view of top is expected as generated by DreamEditor other than a front view. Moreover, direct copy can lead to obvious artifact when the subject is the edge of the original image like the flat cut of the body of the dog in the third of Figure g Our DreamEditor generate
19

---

Published in Transactions on Machine Learning Research (11/2023)
Subject Replacement
Subject Addition
Subject
Subject
Source
Source
Segment
GLIGEN
Result
Result
Figure 8: Failure cases of DreamEditor tor in the two proposed tasks.
reasonable and realistic images when it comes to these cases as shown in Figure g In subject replacement task, Harmonizee will induce patent artifact after inpainting as shown the right. column of Figure g which will impair the background consistency and realism of the generated images dramatically.
Subject Addition
Subject Replacement
CO papier
On ppppprr
papier
Source
CopyHarmonize
DreamEditor
Source
CopyHarmonize
DreamEditor
Figure Comparison of DreamEditor with CopyHoarmniiee.
20


Reviews:
[{"review_id": "pOnZbsijym", "reviewer": "Reviewer_4Jvv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The introduced novel task is very attractive, with the compiled dataset holding promising potential for future research endeavors within this domain. \n2. The rationale and details of the method are presented with remarkable clarity, ensuring readers' ease of comprehension. The experiments results are comprehensive, including automatic evaluations, human evaluations and the effect of initialization strategy and different iteration number.\n\nWeakness:\n1. While the DreamEditor outperforms existing methods in human evaluation, its results still fall short of achieving the desired level of quality. A noticeable disparity persists between the original subject and the generated counterpart. In many instances, the generated subject fails to seamlessly integrate with the surrounding environment in a plausible manner.\n2. The Iteration number is set mannuly for each input image. Regrettably, the model lacks the inherent capability to autonomously ascertain the optimal iteration number for individual images, despite the evident impact of iteration number on the quality of generated outcomes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an innovative task called subject-driven image editing, encompassing two sub-tasks: subject replacement and subject addition. This task aims to precisely control subject placement, similar to subject-driven image generation, while also managing subject location and pose, akin to conventional image editing. The authors curate a new dataset, named DreamEditBench, dedicated to evaluating this innovative task. Moreover, the authors propose an iterative approach named DreamEditor for this task. DreamEditor employs DDIM for iterative result generation, guided by segmentation masks and text prompts. The method incorporates a customized initialization strategy. Experimental validation on DreamEditBench demonstrates the superiority of the proposed DreamEditor over existing methods in terms of human evaluation results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The equations (1), (2), and (3) lack appropriate punctuation.\n- A broader and impartial human evaluation process is recommended.\n- Further experimentation involving an ablation study is encouraged.", "broader_impact_concerns": "I do not have any concerns about the broader impact of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NNVhvowc1M", "reviewer": "Reviewer_k1Yq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The method is evaluated by human evaluation. The observation of the discrepancy between human evaluation and quantitive metrics is beneficial to the community.\n\n2. A new dataset, with moderate size, is curated.\n\n3. The limitations are discussed.\n\nWeaknesses:\n\n1. Technical contribution is limited. The proposed method is a combined application of prior works, i.e., DreamBooth, Segment Anything and GLIGEN. Though it is a recent trend to do research by pipelining, more technical insights are needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper works on subject-driven image editing, including subject replacement and subject addition, by combining DreamBooth, Segment Anything and GLIGEN. The proposed method is evaluated on a newly curated dataset via both quantitive metrics and human evaluation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Regarding the main weakness, I would not request a change since it would be too significant.\n\n\nWe can see that DreamBooth has low background scores. I am curious whether it is possible fine-tune a second model for another special token V2 to learn the background to handle this problem of DreamBooth?", "broader_impact_concerns": "It has a potential mis-use risk to produce fake contents.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jSnhrdbl20", "reviewer": "Reviewer_HtM6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Subject-driven image editing is a challenging and useful task. This work defines two sub-task settings and proposes to use DreamBooth and an iterative generation scheme to achieve reasonable results.\n\n- The challenges of subject replacement and subject addition are analyzed, and the corresponding evaluation datasets and metrics are introduced. A user study is also constructed to evaluate subject consistency, background consistency, and realism.\n\n- To further improve the subject addition performance, two initialization schemas are introduced: COPY and GLIGEN-infill, which provide a good starting point for subject addition.\n\n- Figure 1 clearly illustrates the two sub-task settings, and the overall paper organization and writing are easy to follow.\n\nWeaknesses:\n\n- The technical contributions are somewhat limited. The iterative generation scheme is not new and has been applied in several previous generation tasks, such as image inpainting “High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling, ECCV20” \n\n- A better baseline could be “Inpainting + Copy-Paste + Harmonization,” which is a straightforward workflow for object replacement or addition. It would be good to perform a comparison with such a baseline.\n\n- Regarding the newly introduced evaluation dataset, it only contains 22 subject classes, which may not be enough for a comprehensive evaluation. Collecting at least hundreds of subject classes would be ideal for a more convincing experiment result.\n\n- Regarding the introduced evaluation metrics,  Dino-sub, Dino-back, ClipI-sub, and ClipI-back may be questionable as subject and background pixels are all in the generated image, which may affect the evaluation to each other. For example, when evaluating Dino-sub and ClipI-sub, different backgrounds may cause unpredictable results.\n\n- It is mentioned in the limitation that “DreamEditor can either fail or require a large iteration number” and “leads to a blurry background after N iterations, especially for large N.” The failure cases and analysis could be presented in the appendix. \n\n- It is not clear how to select the best N, and it would be interesting to explore an automatic strategy to choose the best N. For example, we can calculate a metric to do early-stopping.\n\nMinor question: “DreamEditor (5)” in Sec. 6.3, I’m not sure what it means either as it is not explained in the text.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims at subject-driven image editing and targets two sub-tasks: subject replacement and subject addition. To evaluate the new settings, an evaluation dataset is collected, and several metrics are introduced. An iterative strategy and two initialization schemas are proposed to achieve reasonable performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical adjustments:\n- Better baseline comparison.\n- Explain the effectiveness of the proposed metrics.", "broader_impact_concerns": "A broader impact statement should be added to discuss the potential impact on deepfake.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three expert reviewers provided high-quality reviews for this paper. After the revision and the discussions between authors and the reviewers, the ratings are split, with two reviewers leaning to accept and one reviewer leaning to reject. 

The primary concerns of the limitation lie in the result quality. In particular, the comparison with the simple baseline "inpaint+paste+harmonization" because it does not require computationally expensive fine-tuning. However, while this simple baseline can preserve the high-resolution details of the input image, it is fundamentally not feasible for more advanced editing beyond simple translation and scaling. For example, in the paper, the authors did show examples of changing the subject's pose in the inserted scene. The revision also provided these baseline results and quantitative comparisons. 

Considering all the strengths and weaknesses raised by the reviewers, the AE believes that this paper has sufficient merits to be published at TMLR.
Decision: Accept (poster)