DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation
Authors: Hong Chen
Yipeng Zhang
Simin Wu
Xin Wang
Xuguang Duan
Yuwei Zhou
Wenwu Zhu
Venue: ICLR 2024 poster (2024)
Keywords: diffusion model
subject-driven text-to-image generation
disentangled finetuning
customized and personalized generation
Primary area: generative models

Abstract: Subject-driven text-to-image generation aims to generate customized images of the given subject based on the text descriptions, which has drawn increasing attention. Existing methods mainly resort to finetuning a pretrained generative model, where the identity-relevant information (e.g., the boy) and the identity-irrelevant information (e.g., the background or the pose of the boy) are entangled in the latent embedding space. However, the highly entangled latent embedding may lead to the failure of subject-driven text-to-image generation as follows: (i) the identity-irrelevant information hidden in the entangled embedding may dominate the generation process, resulting in the generated images heavily dependent on the irrelevant information while ignoring the given text descriptions; (ii) the identity-relevant information carried in the entangled embedding can not be appropriately preserved, resulting in identity change of the subject in the generated images. To tackle the problems, we propose DisenBooth, an identity-preserving disentangled tuning framework for subject-driven text-to-image generation. Specifically, DisenBooth finetunes the pretrained diffusion model in the denoising process. Different from previous works that utilize an entangled embedding to denoise each image, DisenBooth instead utilizes disentangled embeddings to respectively preserve the subject identity and capture the identity-irrelevant information. We further design the novel weak denoising and contrastive embedding auxiliary tuning objectives to achieve the disentanglement. Extensive experiments show that our proposed DisenBooth framework outperforms baseline models for subject-driven text-to-image generation with the identity-preserved embedding. Additionally, by combining the identity-preserved embedding and identity-irrelevant embedding, DisenBooth demonstrates more generation flexibility and controllability.

Published as a conference paper at ICLR 2024
DISENBOOTH: IDENTITY-PESSERING DISENTAN- GLED TUNING FOR SUBJECT-DRIVEN TEXT-TO-IMAGE GENERATION
Hong Chen', Yipeng Zhang', Simin Wu', Xin Wangh
Xuguang Duan', Yuwei Zhou', Wenwu Zhu
'Department of Computer Science and Technology. Tsinghua University
BBejjing National Research Center for Information Science and Technology
3 LLnnhou University
(h-chen20, zhang-yp28 dxgl8, 9, zhou-yw21Jemai httpemaiissssssssuuaaaaam tsinghua. edu. xin wang, wwzhu) singhua edu. en, wusm2261zu.. iess2211zzu.du.ccm
ABSTRACT
Subject-driven text-to-image generation aims to generate customized images of the given subject based on the text descriptions, which has drawn increasing atten- tion. Existing methods mainly resort finetuning pretrained generative model, where the identity-relevant information (e.g., the boy) and the identity-ireleeant information (e.g., the background the pose of the boy) entangled in the latent embedding space. However, the highly entangled latent embedding may lead the failure of subject-driven lext-to-image generation as follows: (i) the identity-irreleant information hidden in the entangled embedding may dominate the generation process, resulting the generated images heavily dependent on the irrelevant information while ignoring the given text descriptions; (ii) the identity- relevant information carried in the entangled embedding can not be appropriately preserved, resulting in identity change of the subject in the generated images. To tackle the problems, we propose DisenBooth, an ceentity-preseringg disentangled tuning framework for subject-driven text-to-image generation. Specifically, Dis- enBooth finetunes the pretrained diffusion model the denoising process. Dif- ferent from previous works that utilize entangled embedding to denoise each image, DisenBooth instead utilizes disentangled embeddings to respectively pre- serve the subject identity and capture the identity-ireleeant information. We fur- ther design the novel weak denoising and contrastive embedding auxiliary tuning objectives to achieve the disentanglement. Extensive experiments show that our proposed DisenBooth framework outperforms baseline models for subject-driver text-to-image generation with the combining the identity-preserved preserved embedding and embedding. Additionally, by embedding, DisenBooth demonstrates more generation flexibility and controllability!
INTRODUCTION
Training on billions of text-image pairs, large-scale text-to-image models (Rombach et al. 2022 Saharia et al. 2022, Ramesh et al. 2002) have recently achieved unprecedented success in gener- ating photo-realistic images that conform to the given text descriptions. Thanks to their remarkable generation ability, a more customized generation topic, subject-driven text-to-image generation, has
attracted an increasing number of attention in the community (Gal et al.] 2022; Ruiz et al. 2022
Wei et al. 2023] Shi et al. 20033 Chen et al. 2023 Kawar et al., [202]] Kumari et al. 2022). Given a small set of images of subject, e.g., 3 to images of your favorite toy, subject-driven text-to-image
generation aims to generate new images the same subject according to the text prompts, e.g., an
image of your favorite toy with green hair on the The challenge of subject-driven text-to-
"Corresponding Authors.
'Our code is available http:////ggttbb.../oo/oeeeee/e//

---

Published as a conference paper at ICLR 2024
image generation lies in the requirement that in addition to aligning well with the text prompts, the generated images are expected to preserve the subject identity (Shi et al. 2023) as well.
Existing subject-driven text-to-image generation methods (Gal et al. 2022 Ruiz et al. 2022, Kumari ctt al. 2022 Dong et al. 2022) mainly rely on finetuning strategy, which finetune the pretrained text-to-image diffusion models (Rombach al.] 2022 Saharia et al.. 2022) through mapping the images containing the subject to a special text embedding. However, since the text embedding designed to align with the given images, information regarding the subject will be inevitably entangled with information irrelevant to the subject identity, such as the background or the pose of the subject. This entanglement tends to impair the image generation in two ways: (i) the identity- irrelevant information hidden in the entangled embedding may dominate the generation process, resulting in the generated images heavily dependent on the irrelevant information while ignoring the given text descriptions, e.g., DreamBooth (Ruiz et al., 2022) ignores the "in the snow" text prompts, and overfits to the input image background as shown in row column of Figure I (ii) the identity-relevan preserved, resulting in identity change of the subject in the generated images, e.g., DreamBooth generates backpack with different color from the input image in row column 4 of Figure T Other works (Wei et al.] [023] Shh et al.] 2023: Chen et al. 2023 Gal et al.] [023a] Ma et al, 2023) focus reducing the computational burden, and investigate the problem of subject-driven text-to-image generation without finetuning. They rely on additional datasets that contain many subjects for training additional modules to customize the new subject. Once the additional modules are trained, they can be used for subject-driven text-to-image generation without further finetuning. However, these methods still suffer poor generation ability without considering the disentanglement, e.g., ELITE (Wei et al.) 2023) fails to maintain the subject identity in row 2 column 2 and ignores the action "running" from the text prompt in row 1 column 2 of Figure [
information carried in the entangled embedding can not be appropriately
input image
text description
ELITE
InstructPix2Pix
Drsambooth
DisenBooth(ours)
"a dog running
the with sunflowers
S backeack in the snews
backpekk
stands in the white flowers"
Comparisons between different existing methods and our proposed DisenBooth. "a S dog" is a special token that represents the subject identity. The non-finetuning method ELITE and the image editing method InstructPix2Pix struggle in preserving the subject identity. The existing finetuning method DreamBooth suffers from overfitting on the input image background (row 2 col- umn 4) and subject identity changes (row column 4).
Figure 1:
To tackle the entanglement problem in subject-driven text-to-image generation, we propose Disen- Booth, an dieenttty-prseerviv disentangled tuning framework based on pretrained diffusion models. Specifically, DisenBooth conducts the disentangled tuning during the diffusion denoising process. As shown in Figure 2 different from previous works that only rely on an entangled text embedding as the condition to denoise, DisenBooth simultaneously utilizes a textual identity- preeervd embed- ding and a visual dlentity-irreleann embedding as the condition to denoise for each image containing the subject. To guarantee that the textual embedding and the visual embedding can respectively cap- ture the identity-relevan and objectives, i.e., the weak denoising objective and the contrastive embedding objective. To further enhance the tuning efficiency, parameter-effin efficient tuning strategies are adopted. During the inference preserved embedding is utilized for subject-driven rext-to-image generation. Additionally, through combining the two disentangled embeddings together, we can achieve more flexible and controllable image generation. Extensive experiments show that DisenBooth can si-
information, we propose two auxiliary disentangled
stage, only the identity-preserved
2

---

Published as a conference paper at ICLR 2024
Add Noise somples
11.4 22.00 25,6
Diffusion Forward
sampled rosh: 6,.
small Image set
latent space
naiay latent
Identity-Irrelevant Branch
denoise
CLIP
Image
Adapter
Encoder Ej
denoise
denoise
Identity Preserving Branch
weok
LoRA
backpack"
"aS"
CLIP Text Encoder Ey
f
denoise
U-Net(Ee)
L2
DisenBooth conducts finetuning the denoising process, where each input image is denoised with the textual embedding fa shared by all the images preserve the subject identity, and visual embedding fi to capture the identity-irrelevant information. To make the two embeddings disentangled, the weak denoising objective L2 and the contrastive embedding objective Lg are pro- posed. Fine-tuned parameters include the adapter and the LoRA parameters.
Figure 2:
multancously capture the identity-relevan and the identity-irreleve information, and demonstrates superior generation ability over state-of-the-art methods subject-driven driven text-to-imen so-tto--mmae generation.
To summarize, our contributions are listed as follows. (i) To the best of our knowledge, we are
the first investigate the disentangled finetuning for subject-driven text-to-image generation. (ii) We propose DisenBooth, an identity-preservig disentangled tuning framework for subject-driven text-to-image generation, which able to learn a textual ddentity-preservee embedding and a visual identity-irrelent irrelevantt embedding for each image containing the subject, through two novel disentan- gled auxiliary objectives. (iii) Extensive experiments show that DisenBooth has superior generation ability in subject-driven text-to-image generation over existing baseline models, and brings more flexibility and controllability for image generation.
RELATED WORK
Text-to-Imase Generation Training on large-scale datasets, the text-to-image generation mod-
els (Zhang Agrawala 2023 Xu al. 2018 Ramesh et al. 2022 Saharia al.) 2022] Ramesh
al. 2021; Nichol et al. 2022 Chang al. 2023 Kim et al. 2022) have achieved great success
recently. Among these models, diffusion-
models like Stable Diffusion (Rombach et al. 2022).
DALLE-2 (Ramesh et al. 2022 and Imagen (Saharia al. 2022) have attracted lot of attention due to their outstanding controllability in generating photo-realistic images according to the text de- scriptions. Despite their superior ability, they still struggle with the more personalized generation, where we want to generate images about some specific or user-defined concepts, whose identities hard to be precisely described with text descriptions. This leads the emergence of the recently popular topic, subject-driven text-to-iman image generation (Ruiz et al.
2022 Gal et al. 2022).
2020 Meng et al.) Bar- Tal al., 2022 Brooks et al. 2022; Kim et al. 2022; Hertz et al.] [202] aims to edit an input image ac- cording to the given textual descriptions. SDEdit (Meng al.) and Blended-Diffussio (Avra-) hami et al. 2022) blend the noisy input to the generated image in the diffusion denoising pro- cess. Prompt2Prompt (Hertz et al.] (022) combines the attention map the input image and that of the text prompt to generate the edited image. Imagic (Kawar et al. 2022) utilizes 3-step optimization-basee strategy to achieve more detailed visual edits. A more recent SOTA method In- structPix2Pixx (Brooks et al. 2022) utilized GPT-3 3 (Brown al. 2020), Stable Diffusion (Rombach iet al. 2022) and Prompt2Prompt (Hertz et al. 2022) to generate dataset with (original image, text prompt, edited image) triplets train diffusion model for text-guided image editing. Despite
Text-Guided Image Editing Text- image editing (Li et al.
3

---

Published as a conference paper at ICLR 2024
their effectiveness, they generally not suitable for subject-driven text-to-image generation (Chen et al. 2023), which needs to perform more complex transformations to the images, e.g., rotating the view, changing the pose, etc. Also, since these methods are not customized for the subject, their ability to preserve the subject identity is not guaranteed. Some examples of InstructPix2Pix for subject-driven text-to-image generation are presented in Figure the pose of the dog is not changed to "rumming" in row column 3, and the subject identity is changed in row 2 and 3.
Subject-Driven Text-to-Image Generation Given few images of the subject, subject-driven text- to-image generation (Kumari et al. 2022) Gal et al., 2022: Ruiz el al. 2022] Han el al. 2023) aims to generate new images according to the text descriptions while keeping the subject identity unchanged. DreamBooth (Ruiz et al. (202)) and TI (Gal et al. (022)) are two popular subject-driven
text-to-image generation methods based on finetuning. They will both map the images of the subject into a special prompt token 5 during the finetuning process. The difference between them is that TI finetunes the prompt embedding and DreamBooth finetunes the U-Net model. Several concurrent works (Chen et al. 2023 Wei et al.
2023 Shi et al. 2023 Jia et al. 2023 Gal et al. 2023a) propose to conduct subject-driven text-to-image generation without finetuning, which largely reduces the computational cost. They generally rely on additional modules trained additional new datasets, like the visual encoder in WWi et al. (2023), [Shi et al. (2023) to directly map the image of the new subject to the textual space. However, all the existing methods learn the subject embedding in highly entangled manner, which will easily cause the generated image to have a changed subject identity or to be inconsistent with the text prompt. Although some works (Wei et al.) 2023 Dong iet al. 2022) use the segmentation mask to exclude the influence of the background, there are some other identty-irrelevant factors that these methods fail tackle, such as the pose of the subject as shown in row 1 column 2 of Figure I Additionally, these methods rely on an additional segmentation model or user-labeled mask, our method is free of additional annotations.
3 PRELIMINARIES
In this section, we will introduce the preliminaries about Stable Diffusion and subject-driven text- to-image generation, and also some notations will use in this paper.
Stable Diffusion Models, The Stable Diffusion model is a large text-to-image model pretrained on large-scale text-image pairs {(P,))), where P is the text prompt of the image æ. Stable Diffusion contains an autoencoder (). D(-),, a CLIP (Radford et al. (021)) text encoder (-), and a U- Net (Ronneberger et al., 2015) based conditional diffusion model E-))). Specifically, the encoder E(-) is used to transform the input image x into the latent space = E(r), and the decoder D(.) is used reconstruct the input image from the latent z, x x D(z). The diffusion denoising process of Stable Diffusion is conducted in the latent space. With a randomly sampled noise ~ NO, 7) and the time step I, we can get a noisy latent code 2 = 0z22 + ore, where a and o are the coefficients that control the noise schedule. Then the conditional diffusion model eg will be trained with the following objective for denoising (Ho et al. 2020 Song et al. 2020):
min P.P, [||e- eo (Zx, L, Er (P))1
The goal of the conditional model co()) in Eq. (1 is to predict the noise by taking the noisy latent 21, the text conditional embedding obtained by ET ET(P), and the time step as input.
Finetuning for Subject-Driven Text-to-Image Generation. Denote the small set of images of the specific subject as C. (x: /K where v, means the jth image and K is the image number, usually to 5. Previous works (Gal et al. 2022 Ruiz et al. 2022 Kumari et al., 2022) will bind special text token P.. e.g., "a S backpack" in Figure to the subject s, with the following finetuning objective:
min E. E. Il|e- co(,,, Er
(2)
Different methods use the objective in Eq. (2) to finetune different parameters, e.g., Dream- Booth (Ruiz et al. 2022) will finetune the U-Net model ea(.), while TI (Gal et al. 2022) finetunes the embedding of P. in the CLIP text encoder (-).) DreamBooth finetunes more parameters than TI and achieves better subject-driven text-to- -imaee generation ability. However, these methods bind P. to several images { making the textual embedding (P.) inevitably entangled with infor- mation irrelevant the subject identity, which will impair the generation results.
4

---

Published as a conference paper at ICLR 2024
To tackle the problem, in this paper, we propose DisenBooth, an tuning framework for subject-driven text-to-image generation, whose framework is presented in Fig- ure 2 DisenBooth utilizes the textual embedding (y,P.) and a visual embedding to denoise each image. Then, with our proposed two disentangled objectives, the text embedding Er(P:) can pre- serve the identity-relevant information and the visual embedding can capture the ideetity-imelevant information. During generation, by combining P. with other text descriptions, DisenBooth can gen- erate images that conform to the text while preserving the identity. DisenBooth can also generate images that preserve some characteristics of the input images by combining the textual identity- preserved embedding and the visual identity-ireeleant embedding, which provides more flexible and controllable generation. Next, we will describe how DisenBooth obtains the disentangled em- beddings, how DisenBooth finetunes the model with the designed disentangled auxiliary objectives, and then how it conducts subject-driven text-to-image generation with the finetuned model.
disentangled
4 THE PROPOSED METHOD: DISENBOOTH
4.1 THE
AND SEENITT--RRELEVAR EMBEDDINGS
DisenBooth will use textual embedding to preserve the identity-relevant information and a visual embedding to capture the identity- inreleattt information. Then, better subject-driven text-to-image generation can be conducted with the textual adentity-preserve embedding.
The Mlentity-Preservee
Embedding. We obtain the -dentity-preserved embedding fa shared by Branch as shown in Figure 2 tity of subject to special text token P.. Then the textual obtained through the CLIP text encoder,
{æ;) } through the
where we want to map the iden- embedding can be
f. Fr(s).
(3)
The Ideeeity Embedding. To extract the identity-ireleeant embedding of each image Ti we design an Identity-Irreeevnt Branch in Figure 2 where the pretrained CLIP image encoder E, is adopted, and we can first obtain feature f/() Er (z+). However, this feature obtained from the pretrained image encoder may contain the identity information, while we only need the identity- irrelevant information in this embedding. To filter out the identity information from n!²), we design a learnable mask M with the same dimension as the feature, whose element values belong to (0,1), to filter out the identity-relevan information. Therefore, we obtain masked feature M f(") by the element-wise product between the mask and the pretrained feature. Additionally, considering that during the Stable Diffusion pretraining stage, the text encoder is jointly pretrained with the U-Net, while the image encoder is not jointly trained, we use the MLP with skip connection to transform M f(P) into the same space the text feature f. as follows,
fi = M , MLP(M f( = 1,2, K, and in Figure we denote the skip connection with the mask as the Adapter.
4.2 TUNING WITH DISENTANGLED OBJECTIVES
With the above extracted identity-preservee preserved and cieentty--rrelevver embeddings, conduct the finetuning with a similar denoising objective in Eq. (2) on the K images in C.,
L1 || 16c-colz... co(zr,,, ft fa)
(5)
As shown in Figure 2 6, is the randomly sampled noise for the tth image, f, is the randomly sampled time step for the 2th image, and 2i,t, is the noisy latent of image obtained by 21,4, E +
or, e; as mentioned Sec. This objective means that we will use the sum of the identity embedding and the identity-irreleann embedding the condition to denoise each image. Since each image has an amage-specifff identity-ires veenttt-irreleeantt embedding. fs does not have to restore the identity- irrelevant information. Additionally, considering that fo is shared when demoising all the images, will tend to capture the common information of the images, i... the subject identity. However, only utilizing Eq. (5) to denoise may cause a trivial solution, where the visual embedding captures all
5

---

Published as a conference paper at ICLR 2024
the information of image 24, including the identity-relevant information and the identity-irrelewe information, while the shared embedding f. becomes a meaningless conditional vector. To avoid this trivial solution, we introduce the following two auxiliary disentangled objectives.
Weak Denoising Objective. Since we expect that f. can capture the identity- relevant information instead of becoming meaningless vector, fs should have the ability of denoising the common part of the images. Therefore, we add the following objective:
C2 A2 Σ ||a coleuentt fo)llE
(6)
In this objective, we expect only with the identity- preserved embedding, the model can also denoise each image. Note that we add a
22 < 1 for this denoising objective, because we do not need fs to precisely denoise each image, or f. will again contain the identity-rrrelevan
infor- mation. Combining this objective and the objective in Eq. 5 together, we can regard the process in Eq. (5) as a precise denoising process, and regard the process in Eq. (6) as a weak denoising process. The precise denoising process with fa + fa as the condition should denoise both the subject identity and some irrelevant information such as the background, while the weak denoising process with fs as the condition only needs to denoise the subject identity, it requires smaller regularization weight A < 1. We use 22- = 0.01 for all our experiments.
Contrastive Embedding Objective. Since expect fa and fi to capture disentangled information of the image 2;, the embeddings f. and fi should be contrastive and their similarities are expected to be low. Therefore, we add the contrastive embedding objective follows,
K
C== Ao Σ cos(far f.).
Minimizing the cosine similarity between fo and ft will make them less similar to each other, thus casier to capture the disentangled identity-relevant and identity-irrelevann information of zz. Ag is a hyper-parameter which is set to 0.001 for all our experiments. Therefore, the disentangled tuning objective of DisenBooth is the sum of the above three parts:
Ci C2 Ca.
(8)
Parameters to Finetune. In previous works, DreamBooth (Ruiz al.. 2022) finetunes the whole U-Ner model and achieves better subject-driven text-to-image generation performance. However, DreamBooth requires higher computational and memory cost during finetuning. To reduce the cost while still maintaining the generation ability, we borrow the idea of LoRA (Hu et al., (02T] to conduct parameter-efficient finetuning. Specifically, for each pretrained weight matrix Wo E Faxc in the U-Net co(.). LoRA inserts low-rank decomposition to it, Wo + Wo + BA, where B E Rdxr, E R r & min(d, k). A is initialized as Gaussian and B is initialized as O, and during finetuning, only B and A are learnable, while Wo is fixed. Therefore, the parameters to finetune are largely reduced from d x k to (d + k) X r. Parameters for DisenBooth contain the parameters in the previous adapter and the LoRA parameters in the U-Net, shown in the yellow block in Figure 2
4.3 MORE FLEXIBLE AND CONTROLLABLE GENERATION
After the above finetuning process, DisenBooth binds the identity of the subject s to the text prompt P., e.g., "a S* backpack". When generating new images of subject 8, we can combine other text descriptions with P. obtain the new text prompt P', e.g.. "a S backpack on the beach". Then, the CLIP text encoder will transform it to its text embedding f = Er (P.). With f. as the condition, the U-Net model can denoise randomly sampled Gaussian noise to an image that conforms to Pd while preserving the identity of s. Moreover, if we want the generated image to inherit some characteristics of one of the input images V4. e.g., the pose, we can obtain its visual identity-irrelevant embedding fi through the image encoder and the finetuned adapter, and then, use f + r/f: as the condition of the U-Net model. Finally, the generated image will inherit the characteristic of the reference image I,, and 17 is typer-parameed that can be defined by the user to decide how many characteristics can be inherited. DisenBooth not only enables the user to control the generated image by the text, but also by the preferred reference images in the small set, which is more flexible and controllable.
6

---

Published as a conference paper at ICLR 2024
input mages
Sa th fires
beach!
"ast cater
granswtt Barfooanr preued
"aneds' our
BreamBooth
Figure 3: Generated images the can given different text prompts with different methods.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Dataset. We adopt the subject-driven text-to-image generation dataset DreamBench proposed by Ruiz et al. (2022), which are downloaded from Unsplast This dataset contains 30 subjects, including unique objects like backpacks, stuffed animals, cats, etc. For each subject, there are 25 text prompts, which contain recontextualization, property modification, accessorization, etc. To- tally, there are 750 unique prompts, and we follow Ruiz et al. (2022) generate images for each prompt, and total 3,000 images for robust evaluation.
Evaluation Metrics. (i) The first aspect is the subject fidelity, i.e., whether the generated image has the same subject as the input images. To evaluate the subject fidelity, we adopt the DINO score proposed by Ruiz et al. (2022), i.e., the average pairwise cosine similarity between the ViT-S/I6 DINO embeddings of the generated images and the input real images. A higher DINO score means that the generated images have higher similarity to the input images, but may risk overfitting the identity-ireelevnt information. (ii) The second is the text prompt fidelity, i.e., whether the generated images conform to the text prompts, which is evaluated by the average cosine similarity between the text prompt and image CLIP embeddings. This metric is denoted CLIP-T (Gal al. 2022; Ruiz et al. 2022). Besides these two metrics, we also use human evaluation to compare our proposed method and baselines. Specifically,, we asked 40 users to rank different methods in their identity- preserving ability. with randomly sampled 15 unique prompts for each user, and we denote the average rank of the 600 ranks as Identity Avg. Rank. Additionally, we asked another 30 users to rank the generated images in the overall performance of the subject-driven text-to-image generation ability, i.e., jointly considering whether the generated images have the same subject as the input images and whether they are consistent with the text prompts. With randomly sampled 30 unique prompts for each user, we finally obtain 900 ranks, and denote this rank as Overall Avg. Rank.
Baselines. TI (Gal et al. 2022) and DreamBooth (Ruiz et al. 2022) subject-driven text-to- text-to-image generation. InstructPix2PPxx (Brooks et al. 2022) is a SOTA pretrained method for text-guided text-to-image editing. We also include the pretrained Stable Diffusion (SD) (Rombach et al.) (022) without finetuning as a reference model. We provide the detailed implementation of DisenBooth in A.1
finetuning methods for
5.2 COMPARISON WITH BASELINES
The scores of different methods are shown in Table I and some visualized generated results are shown in Figure and Figure More generated results can be found in the Appendix. From these results, we have the following observations: (i) as the reference model, the pretrained SD has the lowest DINO score and the highest CLIP-T score as expected. The pretrained SD is not customized
httpss:/unsppaash..

---

Published as a conference paper at ICLR 2024
Input Image
IntructPiu2Pii Dreambooth
DisenBooth
Ti
InstrucPPidPt Dreambooth
BisenBooth
shoped 5
purple dog running under the starry dky with bright moon"
5 playing water"
buttarfly flowers
dog
Figure 4: Generated examples of different subjects.
for the subject, thus having the lowest image similarity, but it has the highest CLIP-T score because it does not overfit the input images. (ii) The image editing method InstructPix2Pix is not very suit- able for the subject-driven text-to-image generation, which has the lowest CLIP-T, i.e., the lowest text prompt fidelity, because it cannot support the complex subject transformations. (iii) TI has weaker ability to maintain the subject identity and a lower prompt fidelity than DreamBooth and DisenBooth. (iv) DreamBooth has the highest DINO score, which means that the generated images are very similar to the input images. However, as observed in the generated images, this too high DINO score results from overfitting the -dentity-ireleevnnt information, e-g., in Figure 3 the gener- ated images by DreamBooth have a similar background to the last input image, making the prompts like "on the beach" and "on top of pink fabric" ignored. (v) In contrast, our DisenBooth achieves the highest CLIP-T score except the reference model, and generates images that conform to the text prompts while preserving the subject identity. Considering the images generated by our Disen- Booth have very different backgrounds from the input images, it has a little lower DINO score than DreamBooth, but it receives a higher Identity Avg. Rank from users, showing its superior ability in preserving the subject identity instead of overfitting the -deneity-ireeleant information. Addi- tionally, the Overall Avg. Rank collected from the users demonstrates that DisenBooth outperforms existing methods in subject-driven text-to-image generation. We also provide more visualizations in the Appendix, which further shows the superiority of DisenBooth.
Table 1: DINO, CLIP-T, and the user preferences of different methods on DreamBench. Except the referenced model pretrained SD, we bold the method with the best performance w.r.t. each metric.
pretrained SD InstruetPix2Pix
TI
DreamBooth DisenBooth
DINO Score?
0.362
0.605
0.546
0.685
0.675
CLIP-T Scoret
0.352
0.303
0.318
0.319
0.330
Identity Avg. Rank]
2.69
3.73
2.20
1.37
Overall Avg. Rank]
2.89
3.07
2.45
1.59
5.3 ABLATION STUDY
Disentanglement. our design, the textual embedding f. obtained through the special text prompt P. aims to preserve the subject identity, and the visual embedding fi aims to capture the identity- irrelevant information. We verify the relationship in Figure [ In each row, we generate identity- relevant images only using fa as the U-Net condition with random seeds. Then, we generate ieentity-irelevann images using f. the U-Net condition. The results show that DisenBooth can faithfully disentangle the identity-relevant and the identity-irrelevnt information. Additionally, in the second row, we can see that the identity-irrelevan information not only contains the background, but also the pose of the dog, which is described with the pose of a human by Stable Diffusion. The disentanglement explains why our DisenBooth outperforms current methods. The shared textual embedding only contains the information about the subject identity, making generating back- ground, pose or property easier and resulting in better generation results.
More Flexible and Controllable Generation with f., As aforementioned, if want to inherit some characteristics of the input image Z,, we can add the visual embedding ft to f with a user- defined weight 1), i.e., the condition + nft. In Figure with the same text prompt "a S dog on
8

---

Published as a conference paper at ICLR 2024
Figure Visualization for disentanglement. The identity-relevan images are generated using the text prompt P.. The identity-irreleann irrelevant images generated with the image-specific identity- irrelevant embedding f.. All generations are with random seeds.
"as" dog
abrd +
increases from
Image X)
I1oU iroge
12 whitete ot
-
coloriul umoretaa A
Figure 6: Generating images with different reference images with Figure Generating other ob- jects with characteristics of ft.
different
the Great Wall" to obtain f', we select two different images of the dog to obtain f., and the weight n is changed linearly from 0.0 0.8. The results show that with larger 7), the generated image will be more similar to the reference image. With a relatively small the generated image can simultaneously conform to the text and inherit some reference image characteristics, which gives more flexible and controllable generation. However, as becomes large, the text prompt will be ignored and the generated image will be the same as the reference image. This phenomenon means the identity-ireleeant part will impair the subject-driven text-to-image generation, which inspires us to disentangle the tuning process. Additionally, we can also use fi to generate other images that have the characteristics of the identity-irrelevar information of I;, In Figure 7 we use fi to generate other objects, such as "a cat". We can see that the generated image inherits the pose and the background of the input image, which further shows the flexibility of our DisenBooth.
We provide more ablation the Appendix, the effectiveness validation of the disentangled loss in A.2 the mask in A.3 etc., in both qualitative and quantitative ways.
CONCLUSION
In this paper, we propose DisenBooth for subject-driven text-to-image generation. Different from existing methods which learn an entangled embedding for the subject, DisenBooth will use an adentity-preserved embedding and several -eeeeittirreleevntt embeddings for all the images in the finetuning process. During generation, with the identity-preservee embedding, DisenBooth can generate images that simultaneously preserve the subject identity and conform to the text descrip- tions. Additionally, DisenBooth shows superior subject-driven text-to-ine to--toi-iggeee generation ability and serve more flexible and controllable framework.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by the National Key Research and Development Program of China No. 2023YFF1205001, National Natural Science Foundation of China (No. 62250008, 62222209, 62102222), Beijing National Research Center for Information Science and Technology under Grant No. BNR2023RC00003, BNR2023TD03006, and Beijing Key Lab of Networked Multimedia.
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene: Extracting multiple concepts from single image. arXiv preprint arXiv:2305. 2023.
Omer Bar-Tal Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text- driven layered image and video editing. In Computer Vision-ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pp. 707-723. Springer, 2022.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint 1rXiv:2211009800 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. arXiv preprint arXiv:2301 230.::3000,00044, 2023.
Hong Chen, Yudong Chen, Xin Wang, Ruobing Xie, Rui Wang, Feng Xia, and Wenwu Zhu. Curricu- lum disentangled recommendation with noisy multi-feedback. Advances in Neural Information Processing Systems, 34:26924-26936, 2021.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming-Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. arXiv preprint 2XXv:23004.00166, 2023.
Ziyi Dong, Pengxu Wei, and Liang Lin. Dreamartist: Towards controllable one-shot text-to-image generation via contrastive prompt-tuning. arXiv preprint arXiv:2211.11337. 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, arXiv preprint arXiv:2208.016118 2022.
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. Designing an encoder for fast personalization of text-to-image models. arXiv preprint 1222::2300.122288. 2023a.
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain tuning for fast personalization of text-to-image models. ACM Transac- fions on Graphics (TOG), -1() 2023b.
Abel Gonzalez-Garche Joost de Weijer, and Yoshua Bengio. Image-to-ines image translation for cross-domain disentanglement. In Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning, arXiv preprint arXiv:2303.1 /1305, 2023.
10

---

Published as a conference paper at ICLR 2024
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen- -r. Sromp--t--prmpee image editing with cross attention control. International Conference on Learn ing Representations, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arXiv preprint arXiv:2304.0644 2023.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276, 2022.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2426-2435 2022.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. arXiv preprint urXiv:2212.044883 2022.
Hsin-Yine Ying Lee, Hung-Yo Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr. Manigan: Text-guided image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7880-7889, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018
Yiyang Ma, Huan Yang, Wenjing Wang. Jianlong Fu, and Jiaying Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation. arXiv preprint arXiv:2303.09319. 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804. PMLR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748 $748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text- conditional image generation with clip latents. arXiv preprint 22X::222044061255 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.

---

Published as a conference paper at ICLR 2024
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assiser Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. /2e142222..222,,,.. 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-364944 2022.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint AXXi:2304.03411,, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020.
Xin Wang. Hong Chen, and Wenwu Zhu. Multimodal disentangled representation for recommenda- tion. 2021 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2021.
Xin Wang, Hong Chen, Si'ao Tang, Zihao Wu, and Wenwu Zhu. Disentangled representation learn- ing. arXiv preprint arXiv:2211.1169) 2022.
Xin Wang, Zirui Pan, Yuwei Zhou, Hong Chen, Chendi Ge, and Wenwu Zhu. Curriculum co- disentangled representation learning across multiple environments for social recommendation. In International conference on machine learning. PMLR, 2023.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized eot-to-image generation. arXiv preprint arXiv:2302. / 13848, 2023.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1316-1324, 2018.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543. 2023.
Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under spatio-temporal distribution shift. In Advances in Neural Information Processing Systems. 2022.
Zeyang Zhang, Xin Wang, Ziwei Zhang, Zhou Qin, Weigao Wen, Hui Xue, Haoyang Li, and Wenwu Zhu. Spectral invariant learning for dynamic graphs under distribution shifts. In Advances in Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.I IMPLRMENTATION DRTAIL.
We implement DisenBooth based on the Stable Diffusion 2-1 (Rombach et al. 2022). The learning rate is le-4 with the AdamW (Loshchilov & Hutter 2018) optimizer. The finetuning process is conducted one Tesla V100 with batch size of 1, while the finetuning iterations are ~3,000. As for the LoRA rank, we use r = 4 for all the experiments. The MLP used in the adapter is 2-layer with ReLU as the activation function. The LoRA and adapter make the parameters to finetune about 2.9M, which is small compared to the whole 865.9 U-Net parameters. Additionally, the special embedding the same as that of (Ruiz al., 2022). i... "a+ S + class", where S is rare token and class is the class of the subject. When comparing with the baselines, we only use the textual embedding f mentioned in Sec. 4.3 as the condition.
token P. we use to obtain the
A.2 THE EFFECTIVENESS THE DISENTANGLED OBJECTIVES.
We validate the effectiveness of the proposed two disentangled objectives in Table 2 on half of the DreamBench dataset. We can see that without the weak denoising objective (1.2), the DINO score will decrease, which means the subject identity cannot be well preserved. The contrastive embedding objective (L3) can further improve the DINO score and the CLIP-T. We also provide the generated images of the variants in Figure g which are consistent with the quantitative results. It can be seen that without the weak denoising objective, the subject identity cannot be well-preserved, e.g., circled out, the input images have 3 logos but the generated images with this variant have em-
fewer. Without the contrastive embedding objective which makes the textual bedding contain different information from the visual adentity-ineelevan embedding. the identity of the subject will be changed when the prompt is "in the while flowers" in row 2 column 3. Addition- ally, in row the generated images seem to have the same angle, which may be entangled in the shared identity-preservee embedding.
w/o weak(1.2) 0.684
w/o contrast(I.3)
w/o both DisenBooth
DINO
0.689
0.683
0.693
CLIP-T
0.329
0.328
0.331
0.331
Table 2: Ablations about the disentangled objectives.
A.3 ABLATIONS ABOUT THE MASK
We use a learnable vector M to act as a mask to filter out the identity-relevant information. We compare the performance of our method with and without the mask half of DreamBench, and the results are shown in Table 3 We can see that without the mask, the DINO score will degrade
w/o mask
DisenBooth
DINO
0.673
0.693
CLIP-T4
0.330
0.331
Table 3: Ablations about the learnable mask.
significantly. Considering that the mask is used to filter out the identity information, without the mask, the visual branch will also include some identity information, which may cause the text branch to capture less subject identity information, thus resulting in a low DINO score. We also conduct disentanglement visualization about the mask in Figure g where we use the learned fs to generate 4 adentity-relevan images about the subject, and use fi generate 4 identity-irrelevant images. It can be seen that without the mask, the subject identity cannot be well preserved (c.g.. in the identity- relevant images of w/o mask, the color and the logos different from the input image) and the feature will contain identity information (e.g., the identity-irrele iirelvtt images of w/o mask will contain the backpack color).
Comparison between the feature-level mask or pixel-level mask. In our proposed method, we apply learnable mask in the adapter to conduct feature selection. A natural question is whether
13

---

Published as a conference paper at ICLR 2024
"a S" backpack under the starry sky"
*a S" backpack the forest"
"a.5" backpack the white flowers'
S backpack beside the blue river"
w/o denaise
w/o contrastive embedding
w/o both
Input Images
DisenBooth
Figure 8: The effectiveness of the disentangled auxiliary objectives.
input image
identttyreeevant
idendii--imeeeven
DiscrBooth
mack
DisenBooth
w/o mesk
Figure 9: Disentanglement comparison between with and without the mask. Identity-relevant im- ages are generated with the text prompt with 4 random seeds. Identity erated with the image-specific identity-inelevant embedding. The results show that mask M can prevent the identity information into the identity-ireeleant feature, e.g., the red color.
images are gen-
14

---

Published as a conference paper at ICLR 2024
DreamBooth
Pixel Mask
DisenBooth
DINO
0.671
0.666
0.666
CLIP-T
0.321
0.327
0.336
Table 4: Comparison between using pixel-level masks or learnable feature-level masks.
Subject images
Pixel-level
Feature-level
Identity- ireelvant
Identity-
irrelevant
Figure 10: The generated elentity-irreleeve image comparison between using the pixel-level mask and feature-level mask.
we can use an explicit pixel-level mask before the image encoding. We compare the results of using pixel-level masks and feature-level masks on 10 subjects in Table 4 We can that both applying pixel-level masks (Pixel Mask) and feature-level masks (DisenBooth) can improve the CLIP-T score compared to DreamBooth, alleviating overfitting the edentity-irrelevann information problem. However, it shows that applying the feature-level mask brings better performance. The reasons are as follows:
Masks the feature level filter information at the semantie level instead of the pixel level, enabling it to capture more identity-irrelevan irrelevant factors such as the pose or position of the subject. Therefore, the ddentity-preservine thus achieving a better CLIP-T score and better text fidelity. However, using the masks on the pixels can only disentangle the foreground and the background, ignoring other identity- irrelevant factors such as pose.
branch does not have overfit these factors,
The mask at the feature level is learnable and jointly optimized in the finetuning process, while adding an explicit mask the image is two-stage process, whose performance will be little worse.
In Figure 10 we use the identity-irrelevntt features extracted with the feature-level mask and the pixel-level mask to generate images. We can see that when the subject is the dog, the pixel-level mask can only learn the background information of the input subject image. In contrast, our feature- level mask can not only learn the background, but also the pose and the position of the dog, repre- senting it with human. The qualitative results are consistent with our previous analysis.
A.4 TUNING WITH 1 IMAGE
We compare different methods when there is only 1 image for finetuning for each subject, on 1/3 of the DreamBench dataset. The average quantitative results are presented in Table We can see that our proposed method achieves both the best image and text fidelity, which further shows the superiority of our proposed method in this scenario.
Additionally, we also try to explore whether our method can disentangle the identity-relevan and -deetity-ireeeeant information under this --image setting. The corresponding results are presented in Figure II] We can see that, under the 1-image finetuning setting, the identity-relevant and identity- irrelevant information customization, we can disentangle the beach and the duck toy. In the fourth example, we can disen- tangle the cat, and its background and pose.
still be disentangled by our method. For example, in the first subject
15

---

Published as a conference paper at ICLR 2024
pretrain_St
Pix2Pix
TT
DreamBooth
DisenBooth
DINO
0.352
0.613
0.538
0.610
0.617
CLIP-T
0.351
0.302
0.288
0.297
0.321
Table 5: Comparison among different methods when tuning with only 1 image for each subject. Except for the reference pretrain_Sh baseline, bold the method with the best performance.
Input Image
Identity-relevant
Identity inreleann
Input Image
relevay
Idendit-irreleeant
Figure 11: The disentanglement achieved by DisenBooth using only image for finetuning.
A.5 INFLUENCE OF FINETUNING TEXT ENCODER
We also try to explore the influence of finetuning the CLIP text encoder, where we both finetune the text encoder of DreamBooth and our DisenBooth on 20 subjects in DreamBench. We report the average quantitative results in Table 6 We see that finetuning the text encoder for both DreamBooth and DisenBooth will increase their DINO score, which means the generated images will be more similar the input images. However, they suffer significant CLIP-T drop, which means they overfit the input images while ignoring the given textual prompts. Therefore, finetuning alleviate the text encoder will result in more overfitting, but it can be also seen that DisenBooth the overfitting problem with clear margin, whether training the text encoder or not. We also provide the corresponding qualitative comparison in Figure 12 which is consistent with the quantitative results.
A.6 COMPARE WITH MORE BASELINES
In our main manuscript, we compare DisenBooth mainly with the finetuning method for subject customization. Here, we compare method with the following more baselines DreamBench.
(short as Custom) Kumari et al. (2022) proposes to only finetune the parameters in the attention layers of the U-Net. SVDift Han al. (2023) decomposes the parameters with SVD and only finetunes the corresponding singular values. Break-A- A-Scene//a Avrahami et al. (2023) focuses on the scenario where one image may contain several subjects and can customize each subject the image. ELITE Wei et al. (2023) trains an image-to-ter to-text encoder to directly customize each image without further finetuning. E4T Gal et al. (2023b) proposes to train an encoder for each domain and then only very few steps of finetuning are needed for customization.
The corresponding quantitative comparison presented in Table 7 and the results show that our proposed BisenBooth has better subject-driven text-to-image generation ability than all the base- lines. Note that E4T is a pretrained method and its open-source version only for human subjects,
DreamBooth
DreamBooth-text DisenBooth
DisenBooth+text
DINO
0.703
0. 7,728
0.694
0.719
CLIP-T+
0.322
0.286
0.333
0.301
Table The influence of finetuning the CLIP text encoder. "+text" means the version of finetuning the text encoder.
16

---

Published as a conference paper at ICLR 2024
Input Images
DisenBooth
DisenBooth+text
DreamBooth
DreamBooth-text
SA backpack the beach"
"AS" backpack with mountains the background
backpack"
Figure 12:
The influence finetuning CLIP text encoder.
Custom
SVDiff
Break-A-Scene
E4T
ELITE
DisenBooth
DINO
0.675
0.668
0.639
0.241
0.606
0.675
CLIP-T
0.314
0.318
0.313
0.262
0.291
0.330
Table 7: Comparison with more baselines on DreamBench.
which faces a severe out-of-domain problem and cannot achieve satisfying performance on Dream- Bench. This phenomenon also indicates that although there are some non-finetuning methods for fast customization, effective finetuning to adapt to new out-of- concepts is still important.
We also provide qualitative comparisons with the baselines Figure the superiority of our method.
which further demonstrates
Input image
Custom
SVDiff
Break-A-Scene
E4T
ELITE
DisenBooth
S backoack top of pink fabric
Appiae
"AS" on
A' dog with cities in the background
Figure 13: Qualitative comparison with more baselines.
17

---

Published as a conference paper at ICLR 2024
A.7 HYPER-PARAMERR PARAMETERS
There are hyper- paper-pprameters in our method, one is the weight of the weak denoising loss, A2, and the other is the weight of the contrastive embedding loss, Å3. We use different values of X2 and A to tune the model on the "berry. bbwl"" subject. The results are reported in Table 8 and Table We can see from the results that increasing X will make the identity-relevan embedding contain more input image information, which will result in high DINO score, but faces the problem of overfitting and a low CLIP-T score. Also, increasing X3 will make the embedding disentangled and improve performance, but too large value like 0.1 will make this objective dominate the optimiza- tion process, which will harm the denoising process, and result in a low DINO score and CLIP-T score. Empirically, setting these two parameters to 0.001-0.01 will be OK.
A2
0.0
0.001
0.01
0.1
DINO
0.770
0.790
0.792
0.891
CLIP-T
0.259
0.267
0.274
0.207
Table 8: parpmmtr experiments on X
A
0.0
0.001
0.01
0.1
DINO
0.777
0.792
0.797
0.749
CLIP-T
0.264
0.274
0.265
0.255
Table 9: Syrer-parameter experiments on X3
A.8 CUSTOMIZING MULTIPLE SUBJECTS IN A SCENE
We also try to explore whether our DisenBooth can be used when there are multiple subjects in scene, and the example is shown in Figure [4] As shown on the left of the figure, we first prepare two images for the two subjects, i.e., the duck toy and the cup the figure, and we want to generate subject-driven images for the two subjects. During disentangled finetuning, most finetuning tech- niques are the same, except that we will change the input text prompt to "a S* toy and a V cup", which contains the two subjects the generation results. In the middle, we send the input images to the adentity-irrelevan branch to obtain dlentity-ireeleann features, and generate images using the features with two random seeds. images indeed contain the
are interested in. In the middle and right of the figure, present
We can see that the generated formation. On the right, we provide the subject-driven generation results. The first column is the images containing both subjects, where we use "a S toy and a V* cup on the snowy mountain/on the cobblestone street" the prompt. The second column and the third column about a single subject, where we use "a S* toy on the snowy mountain/on the cobblestone street" and "a V* cup on the snowy mountain/on the cobblestone street", respectively.
in-
The preliminary exploration shows that our proposed method can provide overall satisfying genera- tion results on the multi- -ubjec scenario, showing its potential for broader applications. However, looking carefully at the generated images, we can see that although overall the generated cup is similar the given cup, some visual texture details are a little different. The detail differences may come from that since the identity-relevant branch needs to customize more subjects, and the pixels for each subject become fewer, is hard for the model to notice every detail of each subject. Beyond the scope of this work, we believe customizing multiple subjects in scene is an interesting topic and there are more problems worth exploring in future works.
A.9 PERFORMANCE ON NON-CENTER-LOCATED IMAGES
In the previous examples, we follow previous works Ruiz et al. 2022) to use clear and center-located images of the subject. We also try to see whether our method can still work when the subject only covers fewer pixels and is not center-located. The results are shown in Figure [5] From the results, that the image encoder description and text encoder, the subject can still be generated. Although the generated subject is similar to the given subject, the very few pixels of the subject make it hard precisely preserve the
still learn the identity-irrelen irrelevvntt information. With the text
18

---

Published as a conference paper at ICLR 2024
8 = AS"" toy and a V cup"
AS" toy and V cup
A toy
AV
snowy
on the cobble stone street Subject-Driven Generation Results
Input Images
Identity-irrelevant
Figure 14:
Preliminary exploration on customizing multiple subjects in a scene. The left are images used for the two subjects, and the text P. to finetune them. In the middle, we present the generated images using the identity-irellvant feature as the condition, with two random seeds. On the right, provide the subject-driven text-to-image generation results, where in the first column we generate images about two subjects with "a S toy and a V* cup", in the second column we generate images about the duck toy with "a S toy", and in the third column generate images about the cup with the prompt "a V cup".
details, especially for the second subject. A possible solution tackle the problem in this scenario is that we can first crop the image and then use a hyper-resolution network to get a clear image for the subject, and then conduct finetuning with DisenBooth.
Input Images
Identity imelevant Subject-driven Generation
Input Images
Identity-irreeeant
Subject-driven Generation
A.S Thy under the Eiflel Tower
the beach
A3" toy on the Greet Wall
A.S try
AS try Square
Generation results when the given subject located in the center of the image and covers fewer pixels.
Figure 15:
A. GENERATION WITH HUMAN SUBJECTS
We also compare our proposed Diseh and baselines in generating human subjects. The quali- tative results are shown in Figure where we also indicate the used images for each method. For the finetuning baseline TI and DreamBooth, we use 3 images, and E4T only needs few steps of finetuning with image. ELITE can directly infer without finetuning. For our method, we provide the finetuning results with both 1 image and 3 images. The results show that our method also has clear advantage both preserving the human identity and conforming to the textual prompt.
A.11 MORE GENERATION EXAMPLES ON DREAMBENCH
More generation results of different methods DreamBench shown in Figure 17 Figure 18 and Figure [9 Denoting the position the first generated image of InstructPix2PPxx as row column 1, from the results, observe that:
19

---

Published as a conference paper at ICLR 2024
Input Image
Ti images
Dreamiooth Images
ELITE Non-finetuning
E4T Image
DisenBooth
image
images
"A.S" under the Lifel Tower"
"AS" man with ink painting style'
*A woman front trees with autumn leaves"
"A.S" woman under the Eiffel Tower"
Figure 16: Generation results with human subjects.
InstructPix2Pix is not suitable for subject-driven text-to-image generation. The generated pictures InstructPix2Pix always have the same pose as the input images. Additionally, does not have the idea of the subject, easily making the identity of the subject change. For example, in row 1 column of Figure the color of the backpack is changed to orange. In column of Figure T8 the color of the is changed pink, and in row 1 column 2 of Figure 19 the dog is changed to blue but in fact, we only need a blue house behind the original dog.
TI usually suffers severe identity change of the subject. For example, in Figure 17 the colors the backpacks generated by TI are often different from the input images. In Figure 18 in column 3 and column 4 of row 2, the cans also have different identities from the input images.
Dream suffers from overfitting the identity-irrelevane irrelevant information in the input images. For example, in Figure [8] almost all the images of the generated cans have the same back- ground as the last input image, making it ignore the text prompts such as "on the beach" in row 3 column and "on top fabric" in row 3 column 4. Similar phenomenons are also observed Figure 19 where "on top of a purple rug" in row column and "a purple S* dog" in row 3 column 6 also ignored.
DisenBooth shows satisfactory subject-driven text-to-image generation results, where the subject identity is preserved and the generated images also conform to the text descriptions.
A.12 GENERATION WITH ANIME SUBJECTS
In previous examples, we finetune the subjects on DreamBench. We also use DisenBooth to fine- tune on some anime characters, and the results are shown in Figure 20 The results show that our DisenBooth works well for these anime subjects.
A.13 USER STUDY GUIDANCE
We include the guidance for the two user studies. The first one is for the user preference on which method has the best ability to preserve the subject identity. Below is the detailed guidance.
20

---

Published as a conference paper at ICLR 2024
Images
Hellefoog"
algver porr with
DbanBoott
Figure 17: Generated images of the backpack given different text prompts with different methods.
input images
beach
"as" ca coltiaatonn
"enadS' san"
Intr
Figure 18: Generated images of the can given different text prompts with different methods.
"* the pingle
dagir front ot Zike house" feral
" poliee cLeTE"
pupa dag
vot dog
miibbew roue
DreamBooth
Figure 19: Generated images of the dog given different text prompts with different methods.
21

---

Published as a conference paper at ICLR 2024
Input images of anime girl
S" girl the beach"
"a.5 the
"a.5 with
flying playing sky with angel wings"
Input images of an anime girl
5 girl is the forest"
S" girl under the sakura
5" girl under the Eiftel
"a girl with green clothes"
"s.5" girl is playing with a dog in the snow
anima
S gir with city background"
"aS" girli moonlight"
"aS"
"a S girl with the Tower background"
Input images of an anime girl
*a S" grass
girl
wearing sunglasses with hair"
Figure 20: DisenBooth generated examples on some anime subjects.
Read several reference images of the subject, and keep the subject in mind.
Rank the given four images. The image that has more similar subject the previously given reference images should be ranked top. two or more images satisfy both require- ments, you can give the images with higher quality top rank.
During ranking, ddentity-irrelever factors like the background and pose should are not considered for similarity.
The second one is for the user preference on which method has the best subject-driven text-to-image generation ability. Below is the detailed guidance.
Read several reference images of the subject, and keep the subject in mind.
Read the textual description carefully and then we will show several images generated with the textual description.
Rank the given four images. The image that has a more similar subject as the previously given reference images and conforms better to the text descriptions should be ranked top. If two or more images satisfy both requirements, you can give the images with higher quality top rank.
A.14 LIMITATION DISCUSSION
The limitations of DisenBooth lie in the following two aspects. The first one is that since our Dis- enBooth a finetuning method on pretrained Stable Diffusion, inherits the limitations of the pretrained Stable Diffusion. The second limitation is that since proposed method does not quire any additional supervision, it can only disentangle the subject identity and iereevat information. How to conduct more fine-grad fine-grained disentanglement in the identity iireeeeat informa- tion, e.g., the pose, the background, and the image style, to achieve a more flexible and controllable generation is worth exploring in the future.
22

---

Published as a conference paper at ICLR 2024
A.15 OTHER RELATED WORKS
Disentangled Representation Learning Disentangled representation learning Wang et al. (2022)
aims discover the explainable latent factors behind data, which has been applied in various fields such as computer vision Lee et al. (2018); Gonzalez-Garcin et al. (2018), recommendation Wang
al. 2021); Chen al. 2021); Wang al. (2023). and graph neural networks Zhang et (0033 2022). Learning disentangled representations can not only help to improve the inference explain- ability, but also makes the model more controllable. this paper, since we primarily focus on the subject, it is natural for us to disentangle the identity-relevan and dieenity--rrelevann information for more controllable generation. Particularly, our learned disentangled representations are in a multi- modal fashion, where the identity-relevant information is extracted from the text description and the identity-irelevane information is extracted from the image.
A.16 SOCIETAL IMPACTS
Since our work is based the pretrained text-to-image models, has similar societal impact to these base works. On the one hand, this work has great potential to complement and augment human creativity, promoting related fields like painting, virtual fashion try-on, etc. On the other hand, generative methods can be leveraged to generate fake pictures, which may raise some social or cultural concerns. Therefore, we hope that users can judge these factors for using the proposed method in the correct way.
23


Reviews:
[{"review_id": "k7yMpPej5o", "reviewer": "Reviewer_nfwg", "summary": "The paper addresses a very important limitation in subject-driven T2I models - entangled representations of subject with background/irrelevant information. The key idea of the work is to learn e to learn a textual identity-preserved embedding and a visual identity-irrelevant embedding for each image containing the subject, through two novel disentangled auxiliary objectives. Experiments show superior results compared to the baselines.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is very interesting and the experiment evaluation sufficiently demonstrates the utility of the proposed method. While I have few concerns in the presentation (see Weaknesses), the work could potentially have wide applicability across several applications in Gen AI space. Another attractive aspect of the paper is that the results look great with fine-tuning a small number of parameters compared to the baselines.", "weaknesses": "Despite multiple readings, I struggle to understand concretely the use of identity irrelevant branch in section 4.1 - which is a key proposal of the paper.  I suggest authors to expand and provide more details on this line “However, since we only need the identity-irrelevant information in this embedding, we add an adapter followed by the CLIP image encoder to filter out the identity-relevant information.” How exactly the filtering is achieved in equation (4)? Results in Figure 5 look great and show that the disentanglement is indeed working. I am happy to revise my scores after reading the rebuttal. Another question to the authors - in comparing the results with baselines such as Dreambooth, are the same number of subject images utilized? Did authors also explore fine-tuning the CLIP Text encoder? Curious to know if that would create more issues with overfitting? Overall I liked the work, results are solid - but the proposed method needs more clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5RN3IuyXAc", "reviewer": "Reviewer_9xGd", "summary": "The paper focuses on the problem of entanglement of the global and the local (object-specific) features while editing or personalizing a Diffusion model. The argument put forward is that the current state-of-the-art methods used for text-to-image editing tasks in the diffusion domain are not aware of the identity of the object. This means that while editing the images, the background information might dominate the editing process, ignoring the foreground object, or the identity of the object may be compromised. The paper argues that this is due to the entangled nature of the information used in the denoising process of these diffusion models. To address this, the paper introduces disentangled embedding derived from CLIP-based image and text encoders to make sure the global and object-centric features are extracted and used in a disentangled manner with the diffusion model. The proposed method, DisenBooth,  further designs the novel weak denoising and contrastive embedding auxiliary tuning objectives to achieve the disentanglement. Throughout the paper, identity-preserving edits are shown and compared quantitatively and qualitatively with the competing methods.", "questions": "1) Regarding the choice of LoRA optimization over fine-tuning the entire UNet, how does this affect the quality of the proposed method compared to alternative fine-tuning strategies?\n\n2) While the paper focuses on editing single objects, have the authors explored how the method performs with multiple objects in a scene? What challenges and insights arise when dealing with such complex scenarios, especially in terms of encoding and disentanglement?\n\n3) Concerning the inclusion of masks, could you discuss the potential benefits and drawbacks of explicitly integrating masks into images before encoding? How does this compare to the current masking strategy in the adaptor, and under what circumstances might one approach be more advantageous?\n\n4) The method appears to emphasize objects at the center of an image. Have you tested its performance with objects located in the corners or covering fewer pixels? What are some challenges or failure cases in such scenarios, and how do the image and text encoders adapt to encode identity-preserving information effectively?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper proposes a way to disentangle the identity of the object used for editing or personalization in diffusion models. This has an impact on personalized editing, where the user does not want the identity of the object to be compromised while the editing is performed. To achieve this, the paper employs identity-preserving embeddings and two losses to ensure that the foreground information is disentangled from the background information while denoising the images.\n\n2) The paper shows the results of their method in different scenarios. The supplementary shows results on the anime dataset. The quality of the results is impressive compared to the competing methods.\n\n3) The paper also conducts a thorough ablation study of the components used in the method. The study shows the contribution of loss terms and the mask affect the efficiency of the said method. The paper also shows editing scenarios where only one image is used for the personalization task. The results show that the proposed method is better than the competing method in this scenario as well.", "weaknesses": "1) The paper employs a LoRA optimization strategy for fine-tuning the diffusion model, but it doesn't address how this choice impacts the quality of their approach compared to fine-tuning the entire UNet. It raises questions about whether the proposed loss functions and encoding strategies would still be effective in such a scenario and what impact this might have on result fidelity.\n\n2) The paper primarily concentrates on editing and personalizing single, often centrally located objects. While this is a challenging task, it would be intriguing to see how the method performs when dealing with scenes containing multiple objects. It is particularly important to understand how the encoding and disentanglement processes behave in such complex scenarios.\n\n3) It remains uncertain whether the mask could be explicitly integrated into the images before the encoding steps. How this approach might affect performance and how does it compare to the current masking strategy employed in the adaptor?\n\n4) The method appears to primarily focus on objects placed at the center of an image. What remains unclear is how it performs when objects are located in the corners of the image or occupy fewer pixels. It's essential to investigate potential failure cases in this context and assess whether the image and text encoders can accurately capture identity-preserving information in such scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dqmQth2ArU", "reviewer": "Reviewer_UqfN", "summary": "This paper proposes the DisenBooth for disentangled and identity-preserving subject-driven text-to-image generation. It learns two distinct embeddings to capture the identity-relevant and identity-irrelevant information, respectively, with a weak denoising loss and contrastive embedding loss to facilitate disentanglement. The experimental results show good qualitative performance.", "questions": "See above weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is simple but effective for disentangled finetuning.\n2. The results are good, and extensive ablation experiments are conducted.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. To exclude the identity-irrelevant information such as background, one straightforward way is to filter them using a subject mask, which can be easily obtained by a segmentation model (e.g., SAM). Recent studies like Break-a-scene[1] have also explored masked diffusion loss for disentangling objects and backgrounds. This paper should also compare with these methods.\n2. The proposed method might rely on multiple images to learn the identity-relevant and irrelevant embeddings. When there is only one single image for training, it may be challenging to disentangle them. Although the paper includes experiments on a single image in Section A.3, there is a lack of visual results to verify the effectiveness (like Figure 5), and the number of testing samples is also limited (only 3).\n3. More recent methods should be compared, including Break-a-scene[1], Custom Diffusion[2], and SVDiff[3]. \n\n\n[1] Avrahami, Omri, et al. \"Break-A-Scene: Extracting Multiple Concepts from a Single Image.\" In SIGGRAPH Asia 2023.  \n[2] Kumari, Nupur, et al. \"Multi-concept customization of text-to-image diffusion.\" In CVPR 2023.  \n[3] Han, Ligong, et al. \"Svdiff: Compact parameter space for diffusion fine-tuning.\" In ICCV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KbmhiWbChK", "reviewer": "Reviewer_QDoi", "summary": "The authors propose to disentangle identity-relevant and identity-irrelevant information from the reference set to achieve better identity-preserving performance in personalized text-to-image generation. By guiding the finetuning of the diffusion model denoising process with disentangled textual identity-preserved embedding and visual identity-irrelevant embedding, the personalization results can achieve high fidelity to the target object and flexibility to adapt to the background of any reference image. The disentanglement is constrained by proposed weak denoising and contrastive embedding objectives to ensure a non-trivial solution. Numerical and qualitative performances of the work demonstrate the superiority of the work.", "questions": "- Why ELITE is not included in quantitative comparison?\n- InstructPix2Pix uses instructions to guide image editing. I wonder how it is used for comparison in this paper. \n- A small problem with the structure. Since the finetuning objective consists of several separate losses and two of them are novel ones proposed for disentanglement, the ablation of the disentangled objectives is what readers are most curious about. I think the authors should consider moving Appendix A.1 to the paper section 5.3 if possible.\n- Typo: Figure 5 caption line 1. \"The identity-irrelevant images are generated using the text prompt Ps.\" should be \"The identity-relevant images\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Valid motivation and objective\n- Reasonable thoughts of ideas to address the identity-preserving problem in personalization with disentangling feature embeddings\n- Thorough experiments", "weaknesses": "- Missing reference and comparison [1][2][3]\n- Lack of experimental results on real human figures. Since the focus of this work is identity-preserving, it is of great interest to me to see how it works on human objects. Ideally, the authors should provide a comparison figure like Figure 3 in [1].\n\n[1] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or. Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models. arXiv preprint arXiv:2302.12228 (2023).\n[2] Chen, Wenhu and Hu, Hexiang and Li, Yandong and Ruiz, Nataniel and Jia, Xuhui and Chang, Ming-Wei and Cohen, William W. Subject-driven Text-to-Image Generation via Apprenticeship Learning. NeurIPS 2023.\n[3] Xuhui Jia, Yang Zhao, Kelvin C.K. Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, Yu-Chuan Su. Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models. arXiv preprint arXiv:2304.02642, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)