Subject-driven Text-to-Image Generation via Apprenticeship Learning
Authors: Wenhu Chen
Hexiang Hu
YANDONG LI
Nataniel Ruiz
Xuhui Jia
Ming-Wei Chang
William W. Cohen
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Model
Image Generation
Image Editing
In-Context Learning
TL;DR: We propose the first in-context subject-drive image generation framework, which is much faster and efficient than any other subject-drive image generation models

Abstract: Recent text-to-image generation models like DreamBooth have made remarkable progress in generating highly customized images of a target subject, by fine-tuning an ``expert model'' for a given subject from a few examples.
However, this process is expensive, since a new expert model must be learned for each subject. 
In this paper, we present SuTI, a Subject-driven Text-to-Image generator that replaces subject-specific fine tuning with {in-context} learning.
Given a few demonstrations of a new subject, SuTI can instantly generate novel renditions of the subject in different scenes, without any subject-specific optimization.
SuTI is powered by {apprenticeship learning}, where a single apprentice model is learned from data generated by a massive number of subject-specific expert models. 
Specifically, we mine millions of image clusters from the Internet, each centered around a specific visual subject. We adopt these clusters to train a massive number of expert models, each specializing in a different subject. The apprentice model SuTI then learns to imitate the behavior of these fine-tuned experts. 
SuTI can generate high-quality and customized subject-specific images 20x faster than optimization-based SoTA methods. On the challenging DreamBench and DreamBench-v2, our human evaluation shows that SuTI significantly outperforms existing models like InstructPix2Pix, Textual Inversion, Imagic, Prompt2Prompt, Re-Imagen and DreamBooth.

Subject-driven Text-to-lmage Generation via Apprenticeship Learning
Wenhu Chen
Hexiang Hu
Yandong Li
Nataniel Ruiz R
Xuhui Jia
Ming-We Wei Chang
William W. Cohen
Google Deepmind
Google Research
{wenhuchen, hewiang mingweiiiiie. chang, wcohen wcooon) Gggoogle.com com
unddrr the Lighta
with stiver stiver-tppee
slage with sticktnd bead puth
fancy boot
kickted dackball.
Encodary
Afben
Noval
eating Lee-cream in
- jompping coor cowelk
reeding book with
back steer of wasshing
oanine dog
New Rendition
Ganaralizable
In-Contest
Bouts Jantly Sechiuuh hull bhon brbttiew
with rightrap.
A Anaac soloctul the shaltre of toy stone
waddlng across the Boor the a purey chanes after
toy
Customization
We train single SuTI model to generate novel scenes faithfully reflecting given subjects (unseen in training. specified only by 3-5 in-context text->image demonstrations), without any optimization.
Figure 1:
Abstract
Recent text-to-image generation models like DreamBooth have made remarkable progress in generating highly customized images of a target subject, by fine-tuning an "expert model" for given subject from a few examples. However, this process is expensive, since a new expert model must be learned for each subject. In this paper, we present SuTT,, Subject-driven Text-to-Image generator that replaces subject-specifie fine tuning with in-context learning. Given a few demonstrations of new subject, SuTI can instantly generate novel renditions of the subject in different scenes, without any subject-specife optimization. SuTI powered by apprenticeship learning, where single apprentice model is learned from data generated by a massive number of subject-specific expert models. Specifically, we mine millions of image clusters from the Internet, each centered around specific visual subject. We adopt these clusters to train a massive number expert models, each specializing in a different subject. The apprentice Su'I model then learns imitate the behavior of these fine-tuned experts. SuTI can generate high-quality and customized subject-specifie images 20x faster than pptimization-based SoTA methods. On the challenging DreamBench and Dream Bench-v2, human evaluation shows that SuTT significantly outperforms other existing models.
"Core Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Recent texl-to-image generation models ( have shown great progress in generating highly realistic, accurate, and diverse images from a given text prompt. These models are pre-trained on web-crawled image-text pairs like LAION with autoregressive backend models 3 4 diffusion backend models [5 I. Though achieving unprecedented success in generating highly accurate images, these models are not able to customize to a given subject, like specific dog, shoe, backpack, etc. Therefore, subject-drived text-to-image generation, the task of generating highly customized images with respect to a target subject, has attracted significant attention from the community. Subject-driven image generation is related to text-driven image editing but often needs to perform more sophisticated transformations to source images (e.g., rotating the view, zooming in/out, changing the pose of subject, etc.) so existing image editing methods are generally not suitable for this new task.
Current subject-driven text-to-image generation approaches are slow and expensive. While different approaches like DreamBooth [6), Imagic 777. and Textual Inversion [8] have been proposed, they all require fine-tuning specific models for given subject on one or a few demonstrated examples, which typically takes at least 10-20 minute to specialize the text-to-image model checkpoint for the given subjects. These approaches are cineconnumnng as they require back proppagaggg gradients over the entire model for hundreds or even thousands of steps per customization. Moreover, they are space-consuming as they require storing a subject-specific checkpoint per subject. To avoid the excessive cost, Re-Imagen [9] proposed a rerrieval-augmented text-to-image framework to train a subject-driven generation model in a weakly-supervised fashion. Since the retrieved neighbor images are not guaranteed to contain the same subjects, the model does not perform good as DreamBooth [6] for the task of subject-driven image generation.
To avoid excessive computation and memory costs, we propose to train a single subject-driven text-to-image generation model that can perform on-the-fly subject customization. Our method is dubbed Subject-driven learning algorithm. Unlike standard apprenticeship learning which only focuses on learning from one expert, our apprentice model imitates the behaviors of a massive number of specialized expert models. After such training, SuTI can instantly adapt to unseen subjects and unseen or even compositional descriptions with only 3-5 in-context demonstrations within 30 seconds (on a Cloud TPU v4).
generator (SuTI), which is trained with novel apprenticeship
Specialized Experts
Pal.M
Pal.1
Gompll
Optimiration
Groupby URL
acvesloe
Input Conbext
BUTT
Internet Image Text math
Comdructed Demonstratio
Appreaticcship Learning
Figure 2: Conceptual Diagram of the I Learning Pipeline
Figure 2 presents a conceptual diagram of the learning and data preparation pipeline. We first group the images in WebLI [IO] by their source URL to form tiny image clusters because images from the same URL are likely to contain the same subject. We then performed extensive image-to-image and image-to-text similarity filtering retain image clusters that contain highly similar content. For each subject image cluster, we fine-tuned an expert model to specialize in the given subject. Then, we use the fine-tuned experts to synthesize new images given unseen creative captions proposed by large language models. However, the tuned expert models are not perfect and prone to errors, therefore, we adopt quality validation metric to filter out large portion of degraded outputs. The remaining high-quality images are provided as a training signal to teach the apprentice model SuTI to perform subject-driven image generation with high fidelity. During inference, the trained SuTI can attend to a few in-context demonstrations to synthesize new images on the fly.
We evaluate SuTI various tasks such as subject re-contextualization attribute editing, artistic style transfer, and accessorization. We compare SuTI with existing models on DreamBench (6), which contains diverse subjects from wide categories accompanied by some prompt templates. We compute the CLIP-I/CLIP-T and DINO scores of SuTI's generated images on this dataset and compare them with DreamBooth. The results indicate that SuTI can outperform DreamBooth while having 20x faster inference speed and significantly less memory footprint.
Running on A100 according to public colab: httpss//buugggttgffaae co/sd- dreambooth. library and http:p/s/waggggggoeooooo//oooote

---

Further, we manually created 220 diverse and compositional prompts regarding the subjects in Dream- Bench for human evaluation, which is dubbed the DreamBench-v' v2 dataset. We then comprehensively compare with other baselines like InstructPix2Pix [II] Null-Text Inversion [12] Imagic 77.. Textual Inversion RI, Re-Imagen [9, and DreamBooth [6] on DreamBench-v2. Our human evaluation results indicate that SuTI is 5% higher than DreamBooth and at least 30% better than the other baseline in terms of human evaluation. We conduct detailed fine-grained analysis and found that SuTI's textual alignment is significantly better than DreamBoothn while its subject alignment is slightly better than DreamBooth. However, DreamBooth's 's outputs are still better the photorealism aspect, especially in terms of fine fine-grained detail preservation.
We summarize our contributions in the following aspects:
We introduce the SuTI model, a subject-driven text-to-image generator that performs instant and customized generation for visual subject with few (image, text) exemplars, all in context.
We employ the apprenticeship learning to train one single apprentice SuTI model imitate half a million fine-tuned subject-specific experts on a large-scale seed dataset, leading to generator model that generalizes to unseen subjects and unseen compositional descriptions.
We perform a comprehensive set of automatic and human evaluations show the capability of our model on generating highly faithful and creative images on DreamBench and DreamBench-v2.
To facilitate the reproducibility of our model performance, we release the SuTI model API
Google Cloud Vertex AI model service, under the production name 'Instant tuning" 3
2 Preliminary
In this section, introduce the key concepts and notations about subject-driven image-text data, then discuss the basics of text-to-image diffusion models.
Diffusion Models. Diffusion models 33 are latent variable models, parameterized by θ in the form of (x))) == pe (Ze:T )drrrr where æ1. ET are "noised" latent versions of the input image 2o ~ ((x)). Note that the dimensionality of both latents and the image is the same throughout the entire process, with Øo.7 E Rd and equals the product of <height, width, # of channels>, The process that computes the posterior distribution q(=11:7| (at)) is also called the forward (or diffusion) process, and is implemented as predefined Markov chain that gradually adds Gaussian noise to the data according a schedule By:
T
(( (200) = II (2 g(2x)(2+-1) = (mre; V B
1, 3,T
(1)
Diffusion models are trained to learn the image distribution by reversing the diffusion Markov chain. Theoretically, this reduces to learning to denoise æ, ~ (2o) into Eo, with time re-weighted square error loss see [1]] for the complete proof:
B( (Te, [wre [ c) mo1
where D is the training dataset containing (image, condition) = (zo, c) pairs, the condition normally refers to the input text prompt. In practice, Wy can be simplified according [1
Subject-Driven Text-to-Image Image Generation. Existing subject-driven generation models 6 16 often fine-tune a pre-trained text-to-image diffusion model on set of provided demonstrations C, about a specific subject s. Formally, such demonstration contains a set of text and image pairs C, {(I,,, cx) K- centered around the subject Images Eyg contains images of the same subject s, while c, is short description of images Zy. DreamBooth [6] also requires additional C., which contains images about different subjects of the same category as s for prior preservation. To obtain a customized diffusion model &8. (Ii, c). we need to optimize the following loss function:
arg min R. (eo..)) {Te,t (Te., || (mr, e) 20l12]
The customized diffusion model t (I), c) has shown impressive capabilities to generate highly faithful images of the specified subject s.
Generally available at hwtp:///wwweeeeeeeeooe/////e///e

---

In-Dontex: Demoostestions
Apprenticebli Learning
Imaggen
Bastgle
Exyoci
Atlention
Recodery
kitchon
of dowars
Eincodeer
Bumple
Pine
Sepect 0g
backet correloge handbag
tie sunaet
BufTI
Dirfusion
Optminattio
basket
Supervision
anvelope
landmark
Speoialized Exports
handbag
Figure 3: Overview the apprentice learning pipeline for SuTI. Left part shows customization procedure for expert models, and the right parts shows the SuTI model that imitates the behaviors of differently customized experts, Note that this framework can cope with expert models of arbitary architecture and model fodel.family
Apprenticeship Learning from Subject-specific Experts
Notation. Figure 3
presents the concrete workflow of learning. Our method follows apprenticeship learning [17] with two major component, i.e., the expert diffusion models Øg, (æ;, parameterized by 0g regarding subject
S and apprentice diffusion model (æ c, C2) parameterized by θ. The apprentice model takes an additional set of image-text demonstrations C as input. We use $ to denote the superset of subjects include in the training set.
Dataset. The training set Ds contains collection of (C..Pa)ess. where each entry contains an image-text cluster C, accompanied by an unseen prompt p.. The image-text cluster C, contains a set of 3-10 image-text pairs. The unseen prompt is an imaginary caption proposed by PaLM [88. For example, if e: is 'a photo of berry bowl', then would be an imaginary caption like 'a photo of berry bowl floating on the river'. We describe the dataset construction process to sessetttin 4
Learning. To obtain an expert &Ø, (Iz, c) on a subject s, we fine-tune a pre-trained diffusion model [1] the image cluster C. with the denoising loss
arg min mig {E
[ (3,, c) 221121
where o, ~ (x,) The training is similar to Eqn. 3 except that we do not have negative examples for prior preservation because finding the negative examples from the same class is expensive.
Once an expert model trained, we use it to sample images ys for the unseen text description p. to guide the apprentice SuTI model. We gather the outputs from the massive amount of expert models and then use CLIP filtering to construct a dataset G. Similarly, we fine-tune the apprentice model te (2t, Ps. C.)) with the denoising loss on the pseudo target generated by the expert:
arg min E(y. ....p.... P.. [E,, be (2t, , Ps, C.) - -
(5)
where 22 qa., (,), and the training triples (ys, Ps. C.) are drawn from G.
Algorithm. We formally introduce our learning algorithm the Algorithm[]] To improve the training efficiency, we use distributed training algorithm to accelerate the training process. At each training step, we randomly sample batch { (xx, j of size from the dataset Dz. with B., = (C.., Ps.). We then fine-tune K expert models separately w.r.t. Eqn. 4 in parallel, across K different TPU cores. For every subject s inside the batch B., we use the corresponding expert model 0, to synthesize the image ye given the unseen prompt P.. As not all expert models can generate highly faithful images, we introduce a quality assurance step to validate the synthesized images. Particularly, we measure the quality of an expert's is generation by the delta CLIP score [19] A C., Ps). which is used to decide whether a sample should be included in the dataset G. This ensures the high quality of the text-to-inng -oxttto-image training signal for SuTI. Specifically, the delta CLIP score is computed as the increment CLIP score ys over the demonstrated images I E C.:
A (yy. C. P.) = CLIP P.) max CLIP(x, p.) #EC,
(6)
We then feed G as a training batch to update the parameter O of the apprentice model using Eqn. 5 In all our experiments, we K 400, with each TPU core training an expert model.
Inference. To perform subject-driven text-to-image generation, the trained SuTI takes 3-5 image- text pairs as the demonstration to generate new images based on the given text description. No

---

Algorithm Apprenticeship Learning from Large Crowd of Specialized Expert Models
1: Input: Dataset D3- = [(Cs.Pa) Ps) containing subject image cluster C, and unseen prompt ps
2: Input: Pre-trained diffusion model parameterized by 0
3: Output: Apprentice diffusion model parameterized by θ
4: Initialize buffer G
5: while D3 # do
6: {B,, 1 = Dequeue K), where B., (C,,,
7: Fine-tune K expert models Os On, on {Bs, ) in parallel, based on Eqn. 4
8: for = 1 to K do
9: Sample subject-specific generation ys, with DDPM using To., (2t,, Ps.)
if Ays. C.. ).,)>>) then
11: G = Enqueue (yss
end if
13: end for
14: end while
15: Train Le on the generated demonstration G, based on the Eqn. 5
optimization is needed during inference time. The only overhead of SuTI is the cost of encoding these 3-5 image-text pairs and the attention computation, which is more affordable. Our inference speed is roughly in the same order as the original text-to-image generator
4 Mining and Generating Subject-driven Text-to-Image Demonstrations
In this section, we discuss how we created the seed dataset Ds by mining images and text over the Internet. We construct the seed dataset using the WebLI [10 20 dataset. Particularly, we derive our initial image cluster via subsampling the Episodic WebLI data 20), which grouped Web images from the same URL. Then we filter the clusters ensure high intra-cluster visual similarity, using image matching models. The filtered set of image-text clusters is then denoted {C.).es.
After obtaining the subject-driven image clusters, further prompt large language model [18] to generate a description about the subject, with the goal of creating descriptions of plausible imaginary visual scenes. The generating instances of the descriptions will require skills like subject re-contettullizatio, attribute editing, artistic style transfer, and accessorization. We denote the generated unseen captions Ps. Together with C, this forms the final dataset Dg. More details regarding the grouping and filtering are provided in the Appendix.
The dataset D3 contains total of 2M PC,, pairs. Using the aforementioned delta CLIP score filtering (using a high threshold = 0.02), we remove low-quality synthesized images ys from the expert model, finally obtaining dataset G' with ~50K (C.. effective training pairs for the following apprenticeship learning.
5 Experiment
In this paper, we only train SuTI on the text - 64x64 diffusion model and retain the original 256x256 and 1024x 1024 super-resolution as it is from Imagen ..
Expert Models. The expert model is initialized from the original 2. Imagen 64x64 model. We tune each model on a single TPU core (32 GB) for 500 steps using Adafactor optimizer with a learning rate of le-5, which only takes minutes to finish. We use classifier-free guidance to sample new images, where the guidance weight is set to 30. To avoid excessive memory costs, we use fine- tuned experts to sample pseudo-target images and then write the samples as separate files. SuTI will read these files asynchronously to maximize the training speed. Our expert models have a few distinctions from the DreamBooth [6] 1) we adopt Adafactor instead of Adam optimizer, 2) we not include any class word token like "[DOG] dog' in the prompt. 3) we do not include in-class negatives for prior preservation. Though our expert model weaker than DreamBooth, such design choices significantly reduce time/space costs to enable to train millions of experts with reasonable resources.
5

---

dog eating charry bowl
Two chin-neck vases table
A tiny dog is having a hairout
SuTh
InstructP2PP
Null-Text
Null-PP2P
Re-Imagen
Imagic
Dream/Booth
Figure 4: Comparison with other Image Editing and Image Personalization Models.
Apprentice Model. The apprentice model contains 2.5B parameters, which is 400M parameters larger than the original 2.1B Imagen 64x64 model. The added parameters are coming from the extra attention layers over the demonstrated image-text inputs. We adopt the same architecture as Re-Imagen 9.. where the additional image-text pairs are encoded by re- re-using the UNet DownStack, and the attention layers added to the down UNet DownStack and UpStack different resolutions.
We initialize our model from Imagen's checkpoint. For the additional attention layers, we use random initialization. The apprentice training is performed on 128 Cloud TPU v4 chips. We train the model for a total of 150K steps. We use an Adafactor optimizer with a learning rate of le-4. We use demonstrations during training, while the model can generalize to leverage any number of demonstrations during inference. We show our ablation studies the following section.
Inference. We normally provide demonstration image-text pairs to SuTI during inference. Increas- ing the number of demonstrations does not improve the generation quality much. We use lower classifier- free guidance weight of 15 with DDPM [4] sampling strategy.
5.1 Datasets and Metries
DreamBench. In this paper, we use the DreamBench dataset proposed by DreamBooth [6. The dataset contains 30 subjects like backpacks, stuffed animals, dogs, cats, clocks, etc. These images downloaded from Unsplash. The original dataset contains 25 prompt templates covering different skills like recontextualization, property modification, accessorization, 750 unique prompts generated by the template. We follow the original paper to generate images for each prompt form the 3000 images for robust evaluation. We follow DreamBooth to adopt DINO, CLIP-I to evaluate the subject fidelity, and CLIP-T to evaluate the text fidelity.
In total, there total of
DreanBench-v2. To further increase the difficulty and diversity of DreamBench, we annotate 220 prompts for the 30 subjects in DreamBench as DreamBench-v2. We gradually increase the compositional levels of the prompt to increase the difficulty, like 'back view of [dog]' + 'back view of [dog] watching TV" - 'back view of [dog] watching TV about birds'. This enables us to perform breakdown analysis to understand the model's compositional capabilities.
We use human evaluation to measure the generation quality in DreamBench-v2. Specifically, we aim at measuring the following three aspects: (1) the subject fidelity score s, measures whether the subject being preserved,
(2) the textual fidelity score 8, measures whether it is aligned with the text description, (3) the photorealism score 8p measures whether the image contains artifacts or blurry subjects. These are all binary scores, which are averaged over the entire dataset. We combine them as overall score 5. 3, Sp. which is the most stringent score.

---

5.2 Main Results
Baselines. We provide a comprehensive list of baselines to compare with the proposed SuTI model:
DreamBooth [ a fine-tuning method with space consumption [M] x
[S], where |M] is the
model size and [8] is the number of subjects.
Textual Inversion [ fine-tuning method with space consumption is E
S with the embedding
size of EE, note that [ << M
Null-Text Inversion
12 a fine-tuning method with space consumption is TT)
x
E X
S|
Imagic 77: a fine-tuning- method with largest space consumption among all models as requires training [M] x PP), where [P] is the number of a text prompt P {ps} for the subject set S.
InstructPix2Pix a non-tuning method, which can generate and edit a given image really fast within few seconds. There is no additional space consumption.
Re-Imagen 9 non-tuning method, which will take a few images input and then attend to those retrievals to generate a new image. There is no additional space consumption.
Experimental Results. We show our automatic evaluation results on the DreamBench in Table We can observe that SuTI can perform better or on par with DreamBooth on all of the metrics. Specifically, SuTI outperforms DrcamBooth on the DINO score by 5%, which indicates that our method is better at preserving the subject's visual uppearance. In terms of the CLIP-T score, our method is almost the same DreamBooth, indicating an equívalent capability in terms of textual alignment. These results indicate that SuTI has achieved promising generalization a wide variety of visual subjects, without being trained on the exact instances.
Methods
Backbone
DINO
CLIP-I1
CLIP-T
Real Image (Oracle)
0.774
0.885
DreamBooth
Imagen
0.696
0.812
0.306
DreamBooth
SD 21
0.668
0.803
0.305
Textual Inversion
SD 21
0.569
0.780
0.255
Re-Imagen
Imagen 0.600
0.740
0.270
Ours: SuTI
Imagen
0.741
0.819
0.304
Table 1: Automatic Evaluation the DreamBench.
We further show our human evaluation results on the DreamBench- V2 in Table It shows the related rankings for the additional storage on each subject. As can be seen, SuTI is able to outperform DreamBooth by 5% on the overall score mainly due much higher textual alignment. In contrast, all the other existing baselines getting much lower human evaluation score (< 42%)..
and reported the average inference time measure for inferring
Comparisons. We compare our generation results with other methods in Figure 4 seen, SuTI can generate images highly faithful to the demonstrated subjects. Though SuTI still
As can be
Methods
Backbone
Space
Time
Subject
Text f Photorealism
Overall f
Models requiring test-time tuning
Textual Inversion
8
SD
21
$
30 mins
0.22
0.64
0.90
0.14
Null-Text Inversion 12
Imagen
S$
mins
0.20
0.46
0.70
0.10
Imagic 7
Imagen
SS$$
mins
0.78
0.34
0.68
0.28
DoeamBooth
S$S
mins
0.74
0.53
0.85
0.47
DreamBooth
Imagen
1
S$$
10 mins
0.88
0.82
0.98
0.77
InstructPix2PPx
SD 21
10 sees
0.14
0.46
0.42
0.10
Re-Imagen
Imagen
20
0.70
0.65
0.64
0.42
Ours: SuTI
Imagen
20 sees
0.90
0.90
0.92
0.82
Table 2: Human Evaluation the DreamBench--v2. We report approximated average inference time (averaged over subjects) and the relative rankings of the space cost (more S: more expensive). Methods that do not fine-tune in test-time requires no additional storage (denoted by -).
7

---

missing some local textual (words on the bowl gets blurred) or colorization (dog hair color gets darker), the nuance almost unperceivable for humans. The other baselines like InstructPix2Pix [IT] and Null-Text Inversion [1] are not able to perform very sophisticated transformations. Textual Inversion cannot achieve satisfactory results even with 30 minutes of tuning. Re-Imagen [ though gives reasonable outputs, the subject preservation is much weaker than SuTI. Imagic 7 also generates reasonable outputs, however, its failure rate is still much higher than ours, DreamBooth [6] however generates almost perfect images except for the 'blurry' text on the berry bowl. Through the comparison, we can observe remarkable improvement in the output image quality.
Skillset. We provide SuTI's generation to showcase its ability in re-contextualization, novel view synthesis, rendition, property modification, and accessorization. We demonstrate these different skills in the Appendix Figure 8) In the first row, we show that SuTI is able to synthesize the subjects with different art styles. In the second row, we show that SuTI is able to synthesize the different view angles of the given subject. In the third row, we show that SuTI can modify subjects' facial expressions like 'sad', 'screaming', etc. In the fourth row, we show that SuTI can alter the color of given toy. In the last two rows, we show that SuTI can add different accesorrie (hats, clothes, ett) to the given subjects. Further, we found that SuTI can even compose two skills together to perform highly complex image generation. As depicted in Figuue 5) we show that SuTI can combine re-contextualization with hatttngg/accessoorzaaiooon/yyya
to generate high-quality images.
wolf phodble
playfullyy chastin a fox plushie.
playtally chastag & whimmionl forest.
castine dad
back view of... watching & TV show.
bask viewe wasching
Re-Crolest
Re-Coalest Editing
dlay leagos
aiiiing on a glans
aitiing gleas table. poroclain
duck toy
the wateo
Claudo anyled
Re-Cantest
Bo-Contest Accossrio:
Figure 5: SuTI not only re-contextualizes subjects but also composes multiple transformation, all in-context.
5.3 Model Analysis and Ablation Study
We further conducted set of ablation studies to show factors that impact the performance of SuTI.
Impact of Demonstrations, Figure 6 presents the SuTI's in-context generation with respect to an increasing number of subject-specific image examples. Interestingly, we observe transition in the model's behavior as the number of in-context examples increases. When C2 = 23, SuTI generates images using it prior to the text, similar to traditional text-to-image generation models such as Imagen [.. When |C,l = 1, SuTI behaves similarly to an image editing model, attempting to edit the generation while preserving the foreground subject, and avoiding sophisticated transformation. When |C, = 5, SuTI unlocks the capability of rendering novel pose and shape of the demonstrated subject naturally in the targeted scene. In addition, we also observe that bigger |C,| would result in more robust generation of high text and subject alignment, and better photorealism. We also performed a human evaluation visualizes the results in Figure 6) (right). It shows that as the number of demonstrations increases, the human evaluation score first increases drastically and then gradually converges.
the SuTI's 's generation with respect different numbers of demonstrations and
Quality of the expert dataset matters. We found that the Delta CLIP score critical to ensure the quality of synthesized target images. Such filtering mechanism is highly influential in terms of SuTI's final performance. We evaluated several versions to increase the A threshold from None -> 0.0 - ... + +.025, observe that the human evaluation can steadily increase from 0.54 to 0.82.
8

---

dog
dog
dod
I Overall Score
C,-
82
80
A oacino dog jumping over creek
77
dog reading a book
k=1
k=2 k=3
k=4
k=5
Figure 6: (Left) In-context generation by SuTI model, with an increasing # of demonstrations. (Right) Human evaluation score with with respect to the increasing % of demonstrations.
Without such intensive filtering, the model's overall human score can go very low level (54%). With an increasing A, although the size of the dataset G keeps decreasing from 1.8M to around 500K, the model's generation quality keeps improving until saturation. The empirical study indicates that 0.02 strikes good balance between the quality and quantity of the expert-generated dataset G.
Methods
Inference Time
Subject
Text f
Photorcalism t Overall
DreamBooth
10 secs
0.88
0.82
0.98
0.77
SuTI
20 secs
0.90
0.90
0.92
0.82
Dream-SuTI
15 secs
0.92
0.92
0.94
0.87
Table 3: Quantitative Human evaluation of the Dream-SuTI model on DreamBench-v2.
Further fine-tuning SuTT improves generation quality We note that our model is not exclusive methods that requires further fine-tuning, such DreamBooth [6]. Instead, Su'TI can be bined with DreamBooth [6] naturally to achieve better quality subject-driven generation (dubbed as Dream-SuTT) -STT). Specifically, given K reference images regarding a subject, we can randomly feed one image as the condition and use another differently sampled image as the target output, Through fine-tuning the SuTI model for 500 steps (without any auxiliary loss), the Dream-SuTI model can generate aligned and faithful results for given subject. Table 3 shows comparison of the Dream-Sull against SuTI and DreamBooth, suggesting that Dream SuTI further improves the generation quality. Particularly, it improves the overall score from 0.82 to 0.87, yielding 5% improvement over SuTII and 10% improvement over DreamBooth. Since the fine-tuned Dream- -uTI model already trained on all subject images, only one subject image is needed to present during the inference time, which can further reduce the inference cost.
Reference
Prompt text
DreamBooth
SuTT
Dream SuTTI
A robot standing the street of neon-lit lit city with high-rise buildings.
Figure 7: Comaprison between DreamBooth, SuTT and Dream-SuTT.
To gain better understanding of the quality, we show an example in Figure where we pick failure example from SuTI to investigate whether Dream-SuTI improves it. We observe that the DreamBooth does not have strong text alignment, while SuTT's subject lacks fidelity (the generated robot uses legs instead of wheels). With further fine-tuning subject images, Dream-SuTl is able

---

to generate images not only faithful to the subject but also to the text description. However, we would like to note that such subject-driven fine-tuned model share the same drawback of a typical Dreambooth model, which can no longer generalize well to general distribution objects and hence requiring a copy of model parameter per subject.
Related Work
Text-Guided Image Editing With the surge of diffusion-based models, 22 8 have demonstrated the possibilities to manipulate given image without human intervention. (lended-Diffusiin 23 and SDEdit [24] propose to blend the noise with the imput image to guide the image synthesize process to maintain the original layout. Text2Live (25] generates an edit layer on top of the original image/video input. Coompt-to-Promp (26] and Null-Text Inversion [12] aims at manipulating the attention map in the diffusion model to maintain the layout of the image while changing certain subjects. Imagic [ propose optimization based to achieve significant progress manipulating visual details in given image. InstructPix2Pix [1] propose to distill image editing training pairs synthesized from Prompt-to-Prompl into single diffusion model perform instruction-drrive image editing. Our method resembles InstructPix2Pin in a sense that we are training the model on expert-generated images. However, our synthesized data is generated generated by fine-tuned experts, which mostly natural images. In contrast, the images from InstructPix2Pix are synthetic images. In the experiment section, we comprehensively compare with these existing models show the advantage of our model, especially on more challenging prompts.
Subject-Driven Text-to- Image Generation Subject-Driven Image Generation tackles new chal- lenge, where the model needs to understand the visual subject contained in the demonstrations to synthesize totally new scene. Several GAN-based models [27] 28 pionecred to work personal- izing the image generation model to a particular instance. Later on, DreamBooth [6] and Textual propose optimization-based approach adapt image generation to specific unseen subject. However, these two methods are time and space-consuming which makes them unreal- istic in real-world applications. Another line of work adopt retrieval-augmenter architecture for subject-driven generation including -NN-Diffusion (0), Re- Re-Imagen [9]. however, these methods trained with weakly-supervised data leading much worse faithfulness. In this paper, we aim at developing an apprenticeship learning paradigm to train the image generation model with stronger supervision demonstrated by fine-tuned experts. As result, SuTI can generate customized images about a specified subject, without requiring any test-time fine-tuning, There are some concurrent and related works (31 focusing on specific visual domains such as human faces and / animals. To our best knowledge, SuTI is the first subject-driven text-to-image generator that operates fully in-context, generalizing across various visual domains.
Inversion 8 29
7 Conclusion
Our method SuTI has shown strong capabilities to generate personalized images instantly without test-time optimization. Our human evaluation indicates that SuTI is already better in the overall score than DreamBooth, however, we do identify a few weakness of our model: (1) SuTI's 's generations are less diverse than DreamBooth, and our model is less inclined to transform the subjects' poses or views in the new image. (2) SuTI is less faithful to the low-level visual details than DreamBooth, especially for more complex and often manufactured subjects such as 'robots' or 'rc cars' where the subjects contain highly sophisticated visual details that could be arbitrarily different from the examples inside the training dataset. In the future, we plan to investigate how to further improve these two aspects to make SuTI's s generation more diverse and detail- preaerigre...ing...
10

---

Acknowledgement
We thank Boqing Gong, Kaifeng Chen for reviewing an early version of this paper in depth, with valuable comments and suggestions. We thank Neil Houlsby, Xiao Wang and also the PaLI-X team for providing early access their Episodic WebLI data. We also thank Jason Baldbridge, Andrew Bunner, Nicole Brichtova for discussions and feedback on the project.
Broader Impact
Subject-driven text-to-image generation has wide downstream applications, like adapting certain given subjects into different contexts. Previously, the process was mostly done manually by experts who are specialized in photo creation software. Such manual modification process is time-consuming We hope that our model could shed light on how to automate such process and save huge amount of labors and training. The current model is still highly immature, which can fall into several failure modes as demonstrated in the paper. For example, the model is still prone to certain priors presented in certain subject classes. Some low-level visual details in subjects are not perfectly preserved. However, it could still be used as an intermediate form to help accelerate the creation process. On the flip side, there are risks with such models including misinformation, abuse and bias. See the discussion of broader impacts in 4 for more discussion.
References
[1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic texi-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 2022.
[2] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neurul Information Processing Systems Dutasets and Benchmarks Track, 2022.
[3] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8833 PMLR, 2021.
[4] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong. Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfci Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
[5] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
[6] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR, 2023.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. CVPR, 2023.
[8] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen- An image worth one word: Personalizing -ex-tto-image generation using textual inversion. arXiv preprint arxiv:2208.006/1 2022.
[9] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval-augmenne text-to-image generator. International Conference on Learning Representations, 2023.
[10] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni. Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language- image model. arXiv preprint N.0v:22290000999 2022.
[11] Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instrectpix2pix: Learning to follow image editing instructions. arXiv preprint wrXXi:22/1.09800 2022.
[12] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Crhen-Or. Null text inversion for editing real images using guided ditfusion models. arXiv preprint 0rX14:2211.09994, 2022.
11

---

[13] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thernodynamiis.. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[14] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:684006851, 2020.
[15] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
[16] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Yan Zhu, Multi-concept customization of text-to-image -moge diffusion. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[17] Pieter Abheel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceed- ings of the twenty-first international conference on Machine learning, page 1, 2004.
[18] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. 2022.
[19] Manling Li. Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng. Heng Ji, and Shih-Fu Chang. Clip-event: Connecting text and images with event structures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16420-16429, 2022.
[20] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo. Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang. Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucie, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslce Montgomery, Paulina Pietrzyk, Marvin Ritter, AJ Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lec, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali-x: On scaling up multilingual vision and language model. arXiv preprint XrX25230 18565, 2023.
[21] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[22] Or Patashnik, Zongze Wu, Fli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085-2094, 2021.
[23] Omri Avrahami. Dani Lischinski. and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208- 18218, 2022.
[24] Chenlin Meng, Yutong Hc, Yang Song. Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[25] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XV, pages 707-723. Springer, 2022.
[26] Amir Hertz, Ron Mokady, Jay Tenenhaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint ar 220: 01666, 2022.
[27] Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano. custance-conditioned gan. Advances in Neural Information Processing Systems, 34:27517 -722929 2021.
[28] Yotam Nitzan, Kfir Aberman, Qiurui Hc, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yacl Pritch, and Daniel Cohen-Or. Mystyle: A personalized generative prior. ACM Transactions on Graphics (TOG), -106):1-10, 2022.
[29] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Designing an encoder for fast personalization of text-to-image models. arXiv preprini urXiv:2302.1223 2023.
[30] Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer. Oran Gafni, Eliya Nachmani, and Yaniv Taigman. Knn-diffusion: Image generation via large-scale retrieval. arXiv preprint ar 220202200. 2022.
12

---

[31] Xuhui Jia, Yang Zhao, Kelvin CK Chan. Yandong Li. Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang. and Yu-Chuan Su. Taming encoder for fine-tuning image customization with lext-to-image diffusion models. arXiv preprint arXiv:2304. 2rX0022304.02642, 02642, 2023.
[32] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning, arXiv preprint arXiv:2304.03411 2023.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763, PMLR, 2021.
13

---

A Supplementary Material
A.1 Dataset Construction
To validate the effectiveness, we provide an ablation study to show that higher precision is more important than recall in training the apprentice model. Particularly, when the threshold is set lower number (e.g., 0.01 or 0.015), SuTI becomes less stable.
As our goal is collect images of the same subject, we create initial subject cluster by grouping all (image, alt-text) pairs that come from the same URL (~45M clustrers), and filter the cluster with less than instances (~77.8%% of the clusters). As result, it leaves us with ~1M image clusters. We then apply the pre-trained CLIP VIT-L14 model [33 to filter out 81.1 % of clusters that has the average intra-cluster visual similarity between 0.82 and 0.98 ensure the quality of clusters.
Though the mined clusters already contain (image, alt-text) information, the alt-text's noise level is too high. Therefore, apply the kate-of-the-ang image captioning model [10] to generate descriptive text captions for every image of all image clusters, which forms the data triples of (image, alt-text, caption). However, current image captioning models tend to generate generic descriptions of the visual scene, which often occlude the detailed entity information about the subject. For example, generic captions like 'a pair of red shoes' would greatly decrease the expert model's capability to preserve the subject's 's visual appearance. To increase the specificity of the visual captions, we propose merge the alt-text, which normally contains specific meta information like brands, names, with the -ooeer-generated caption. For example, Given an alt-text of 'duggee talking puppet hey duggee chicco 12m² , and caption of 'a toy the table',
aim to combine them more concrete caption: 'Hey duggee toy the table'. To achieve this, we prompt the pre-trained large language models [18) to read all (alt-text, caption) pairs inside each image cluster, and output short descriptive text about the visual subject. These refined captions with the mined images are used as the image-text cluster C2 w.r.t subject s, which will be used to fine-tune the expert models.
A.2 SuTI Skillset
We demonstrate the complete view of SuTI's skillset in Figure 8 including styled subject generation, multi-view subject rendering, subject expression modification, subject colorization, and subject accessorization.
A.3 Failure Examples
show some failure examples of SuTI. We show several types of failure modes: (1) the model has a strong prior about the subject and hallucinates the visual details based on its prior knowledge. For example, the generation model believes 'teapot' should contain 'lift handle". (2) some artifacts from the demonstration images are being transferred to the generated images. For example, the 'bed' from the demonstration is being brought the generation, (3) the subject's visual appearance is being modified through, mostly influenced by the context, like the 'candle' contains non-existing artifacts when contextualized in the 'toilet', These three failure modes constitute most of the generation errors. (4) The models are not particularly good at handling compositional prompts like the 'bear plushie' and 'sunglasses' example. In the future, we plan to work on how to improve these aspects.
Figure 9
A.4 More Qualitative Examples
We demonstrate more examples from DreamBench-v2 in the following figures:
14

---

duck toy
Pablo Pleasso
Rembrandt
Rene Magriltt
Vincenb van Gogn
dog
Top down view
dog
Depressed
Joyous
Bleepy
Boreaming
A monster toy
Blue
Green
Purple
Pink
Chet
Police
Fire-7ighter outft
Tronman
Superman Angel
Figure 8: SuTI's in-context generation that demonstrates its skill set. Results generated from single model.
First row: rendition of the subject. Sccond row: multi-view synthesis of the subject. Third row: modifying expression the subject. Fourth row: editing color the subject. Fifth row: adding accessories the subject. Subject (image, text) and editing key words are annotated, with detailed template in the Appendix.
15

---

An aged bear plushie pointing to its missing stitches.
candle floating the toilet
A dog Versailles garden.
A black and white teddy bear is wearing a sunglasses
A fancy boot is worn by a Ragdoll
A teapot is placed the floor.
Figure 9: SuTI's failure examples DreamBench-v2.
16

---

canine dog
border collie
duck toy
fancy boot
wolf plushie
back view of [S] watching TV show about birds.
[S] standing lush green field.
[S] floats lazily in the bathtub full of blue bubbles.
[S] cross street.
[S] playfully chasing fox plushie through whimsical forest.
[S] eating an ice- cream bowl.
[S] sniffing backpack the airport.
[S] waddles across the floor [S] under the stage lights. as the a puppy chases after it.
[S] jumping high over creek.
[S] jumping creek on snowy day.
[S] splashing through river wearing detective hat.
[S] sits dusty shelf.
[S] with silver-tipped toes.
[S] reading book.
[S] reading book with a pink glasses on.
[S] wearing goggles sticking its head out of a car window.
[S] on the bed with a nightcap.
[S] with silver-tipped toes kicking football.
[S] sitting on a salad bar.
[S) talking to british shorthair the garden.
[S] herding group of sheep in the thunder storm.
a stack of colorful [S] fill the (S] on the stage with bunny shelves of toy store. sticking its head out.
[S] walking towards lueh jungle landscape, with towering exotic
Figure 10: Visualization of SuTI's generation on the DreamBench-v2 (Part 1).
17

---

A grey sloth plushie
A red monster toy
Pink sunglasses
A poop emoji toy
A clay teapot
[S] climbing tree.
[S] sitting on wing chair.
[S] hang the wall.
[S] clock tower.
[S] glass table.
[S] dangles lazily from backpack.
[S] sitting wing chair with teddy bear.
[S] wooden deck overlooking lake.
[S] under the Tokyo tower.
[S] pouring steaming hot water into teacup.
S reading a paper.
[S] having sushi.
[S] sitting on a river bank facing skycrapers.
[S] talking to red heart emoji tay
sitting glass table, surrounded by delicate porcelain teacups.
wearing T-shirt.
[S] on book cover.
[S) a yellow sunglass case.
wearing big nose funny glasses.
[S] wooden table, together with salmon sushi.
An aged [S]
[S] flying a kite in the desert.
[S] in the microwave oven.
[S] in a hot air ballon in the sunset.
[S] on the floor, surrounded by scattered tea leaves.
00
Figure 11; Visualization of SuTI's generation on the DreamBench-Ve (Part 2).
18

---

A racing car toy
A cartoon devil
A robot toy
A shiny sneaker
[S] driven by the super Mario.
[S] eating banana in a lush tropical jungle.
[S] sitting in comfortable armchair.
[S] in the shoe box
[S] zooms another car toy and arrives the finish line.
[S] playing fencing.
[S] exploring neon-lit city night.
[S] in the shoe box at
luxury boutique store.
[S] on a rallway track.
[S] sitting a desk, typing on [S] chasing curious through sunlit garden.
[S] on treadmill.
multiple keyboards.
[S] on railway track facing train.
[S] playing guitar.
[N] sleeping on bed.
[S] on the roof.
[S] on racing track.
[S] driving a car cruising down scenic coastal road.
[S] on the river bank.
[S] perched the edge rooftop, with a panoramic view
Figure 12; Visualization of SuTI's generation on the DreamBench-v2 (Part 3).
19

---

A Herschel backpaok
Heraobel backpack
Herschel backpack
C2
P3 In Grand Canyon
Herschel backpack
Pg A Herschel backpack
in the
A candle
A candle
A candle
C =
P candle sitting
candle decorated
with nowers.
plushie
bear plushie
C,-
plushies p, in the store.
P in temple.
bear plushle
Figure 13: In-context generation by SuTI model, with an increasing # of demonstration (More examples).
20


Reviews:
[{"review_id": "AEO1QAg8A2", "reviewer": "Reviewer_JnTg", "summary": "In this paper, the authors propose a new model that can perform subject-driven text-to-image generation. Instead of fine tuning a leerte pretrained model on each subject, the authors use apprentice learning to first construct a virtual dataset from a large number of teacher models, each specific to a kind of subjects, and then have the student model learn from the constructed dataset. The authors have also shown strong qualitative and quantitative results and ablation study.", "questions": "Please address the weakness mentioned above. I am happy to change my score if problems related to my concerns are resolved.\nHow long does it take to train the apprentice model?", "limitations": "There is no discussion of the details about the human evaluation (I.e. the instructions given to human and the compensations).", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written and easy to follow. The results seem very promising. The paper tackles an interesting problem that is relevant to the applications.", "weaknesses": "1. This paper could have been impactful if the authors plan to open source or provide a way to reproduce the results. However, from the checklist the authors indicate that they have no plan to open source the model. Given that this model is trained with hundreds of TPUs, unless the authors provide the pretrained model or explicit instructions on how to reproduce the results in a non-cost prohibitive way, I don’t really see a way for the peer researchers to verify the results, nor do I see any real benefits for the practitioners from this paper.\n2. The authors performed human evaluations. However, details about the instructions given to human evaluators and the compensation are not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment #1 “This paper could have been impactful if the authors plan to open source or provide a way to reproduce the results. However, from the checklist the authors indicate that they have no plan to open source the model.”\n\nWe are in full agreement that it would be far better for the community to release the code and/or model, and we are actively working towards doing that.  Although we cannot make guarantees at this point, we do plan to make the model or API publicly available to researchers before paper publication.\n\nWhen the paper was initially submitted, we had not yet established a concrete plan for open sourcing, so we were conservative and marked 'no' in the checklist to avoid potential miscommunication. Please note that according to the NeurIPS 2023 paper checklist guideline (https://neurips.cc/public/guides/PaperChecklist), particularly in the section 6 experiments, it is explicitly stated that “Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).\n\nComment #2 “I don’t really see a way for the peer researchers to verify the results”:\n\nWe respectfully disagree. First of all, we would release all of our test prompts, and generate results, which enables the researchers to do comparison. Secondly, we have included most key technical details in the paper, such that readers can run their experiments to reproduce the results. Finally, as stated above, we are actively working on releasing the model, to enable researchers to reproduce our results. \n\nComment #3 “nor do I see any real benefits for the practitioners from this paper”:\n\nWe believe that our paper provides some deep insight for the research community. For example, we found that fine-tuning on SuTI is better than fine-tuning on the original Imagen model. The subject fidelity improves quite a lot. We plan to put some of these results in the revision. The paper also provides an in-depth description of our novel methodology, which can potentially be replicated with smaller resources on public models if some constraints are relaxed. Moreover, as we mentioned, we are actively investigating the possibility of releasing the SuTI as a model-based model API service.\n\nComment #4 “The authors performed human evaluations. However, details about the instructions given to human evaluators and the compensation are not provided”:\n\nThanks for the comment. We have performed a training session to our human annotators to ensure their evaluation is calibrated, meanwhile the compensation to our annotator is paid hourly instead of per annotation, which incentivizes the annotators to spend more time for each annotation. We will put our annotation guideline and other details to the Appendix in the revision. \n\nIn addition, we also like to mention that the automatic evaluation on DreamBench shows that SuTI outperforms other methods (best on 2 eval metric, and on par with Imagen based DreamBooth on 1 eval metric), which is aligned with the human evaluation.\n\nComment #5 “How long does it take to train the apprentice model?”:\n\nThe training time cost is less than 24 hours on 64 TPUs on Google Cloud Platform (actual duration depends on the version of TPU used. For your reference, training on TPU v4 takes ~10 hours).\n"}, {"review_id": "R1LXmtHiAQ", "reviewer": "Reviewer_6um5", "summary": "The paper proposes an in-context learning method for model customization given personalized objects. The method first collects a large-scale dataset of custom concepts ensuring all images in each custom concept cluster are similar to each other and fine-tunes the model for each concept using Dreambooth. These expert models are then used to get the dataset to train the in-context learning method. It takes a few image text pair of the concept and a new text prompt to generate the image corresponding to the new prompt. The method is based on the Re-Imagen framework. To ensure high quality image text pair dataset from dreambooth models, CLIP based feature similarity threshold is applied. ", "questions": "Minor questions and comments: \n\n1. Does having a variable number of demonstrations instead of 3 have any effect on the final model performance? \n\n2. It would be nice to include some sample descriptive text predicted by the language model for various subjects. Is it usually a class word corresponding to the subject or includes more details regarding the subject? \n\n3. Is there any reason for low photorealism score of SuTI compared to DreamBooth? \n\n4. Line 94. \"to to\" typo\n\n5. Line 116. \"models\" -> \"model\" \n\n6. Line 209. Probably a missing word in the sentence. \"with largest space consumption\" \n\n7. Line 280. \"generated generated\" repeated words. \n\n8. Line 286. “GAN=based” -> “Gan-based” \n\n9. Line 306. \"subjects are contain\" -> \"subjects contain\" \n\n10. Appendix Line 429. \"Subject-Drive\" -> \"Subject-Driven\" \n\n11. Appendix Line 430. \"task of our paper aims to solve\" -> \"task our paper aims to solve\" \n\n", "limitations": "Yes. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The method is one of the first works on in-context learning for model customization. This prevents the time-consuming step of fine-tuning models given any new subject images. Both qualitative and quantitative results show that the method performs on par or better than existing zero-shot or fine-tuning methods.  \n\nThe paper is well-written, easy to understand, and has extensive ablation experiments to validate the importance of different aspects of the method. \n", "weaknesses": "Why is there a need to train the Dreambooth expert models for training the final SuTI model? How does the performance change if the collected dataset itself is used directly to train the final model with N-1 image text pairs for in-context input and 1 sample as the new inference? It would be great to have an analysis regarding that. \n\nOther objects included in the image often take the characteristics of the main subject, e.g., kite in Figure 10 2nd column last row or the british shorthair cat in Fig. 9 1st column last row in the appendix. Does adding a specific characterization in the text prompt for the other subject prevent that? Or is it overfitting in such scenarios? Does having more in-context demonstrations prevent that? \n\nCan it be combined with other editing-based methods like SD-Edit or similar approaches to edit a specific region of the image or combine multiple specific subjects in the same image? E.g., to generate the canine dog of Figure 5 eating the cherry bowl of Figure 4. The editing example shown in Figure 5 changes the whole image instead of just the birds in the TV show.  \n\nAnother point I would like to make is that it would have been great to also show the performance of SuTI with the SD backbone. This is not a weakness per se, and it doesn't affect the final rating. It's excellent work, but it would be helpful to assess the method's performance on such open-source models. Some of the baseline methods shown in the paper are already with the SD backbone. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the highly detailed and constructive feedback!\n\nComment #1  “​​Why is there a need to train the Dreambooth expert models for training the final SuTI model? How does the performance change if the collected dataset itself is used directly to train the final model with N-1 image text pairs for in-context input and 1 sample as the new inference? It would be great to have an analysis regarding that.”\n\nThis is exactly our initial experiments to build SuTI. However, as the images in the same clusters are too similar, often near duplicates (particularly for rigid body objects), so after training, the model falls into a local optimum of copy-pasting the reference image without referring to the text instruction at all. We spent weeks of efforts to increase the diversity within the image cluster to discourage this behavior, however, the issue was not fully resolved. \n\nTherefore, we took the hard route of using DreamBooth + LLM to generate diverse subject images under different context/visual scenes to generate highly different outputs. This proves to be extremely effective in overcoming the copy-paste behavior. We will add more discussion about these failed efforts in the revision.\n\nComment #2 “Other objects included in the image often take the characteristics of the main subject, e.g., kite in Figure 10 2nd column last row or the british shorthair cat in Fig. 9 1st column last row in the appendix. Does adding a specific characterization in the text prompt for the other subject prevent that? Or is it overfitting in such scenarios? Does having more in-context demonstrations prevent that?”\n\nThis is a very common issue from the base text-to-image diffusion model, which sometimes does poorly on understanding language compositionality. We have some empirical evidence to support: (1) “adding more specific characterization” would generally help in some cases, e.g., the kite case can be fixed, but the “british shorthair” won’t be fixed. (2) “adding more in-context examples” does not seem to help much in this case. “More examples” can increase the identity preservation, but does not fully address the compositionality issue. (3) this is not necessarily “overfitting”, rather this compositionality issue is inherited from the original text-to-image generation model (see https://arxiv.org/abs/2212.05032).\n\nComment #3  “Can it be combined with other editing-based methods like SD-Edit or similar approaches to edit a specific region of the image or combine multiple specific subjects in the same image? E.g., to generate the canine dog of Figure 5 eating the cherry bowl of Figure 4. The editing example shown in Figure 5 changes the whole image instead of just the birds in the TV show.” \n“Combine with Region-based in-painting”: Yes, it can definitely be combined with SD-Edit/DiffEdit to only in-paint a specific area of the image. We have some results that we haven't put it in the paper yet but will dosoin the revision.\n“Multiple Subject Composition”: We have tested this composition ability and found that SuTI does generalize to two subjects in some easy cases. But the failure rate is still pretty high, therefore, we did not include this part in the paper. Moving forward, multi-subject image generation will definitely be a priority.\n\nComment #4 “Another point I would like to make is that it would have been great to also show the performance of SuTI with the SD backbone. This is not a weakness per se, and it doesn't affect the final rating. It's excellent work, but it would be helpful to assess the method's performance on such open-source models. Some of the baseline methods shown in the paper are already with the SD backbone.”\n\nWe definitely want to make SuTI accessible to the general public. Since the submission, we have made some good progress toward this direction. Particularly, we are actively investigating the possibility of releasing the SuTI as a model-based model API service.  \n\nComment #5 Response to other individual questions.\n\nQ1: Yes, normally the more the better. But the training will be slower and memory consumption will be higher. We pick 3 as a trade-off between memory consumption vs quality.\nQ2: Sure, we will add more examples in the Appendix. The data contains both: (1) general class word corresponding like “shoes”, (2) more specific subject like “Nike Air … Shoes”. We hope this two modes can help the model ground better.\nQ3: There are two potential reasons: (1) SuTI fits the output of DreamBooth output, which has a slightly different distribution than natural image, especially under large classifier-free guidance. The model might overfit to this new distribution, (2) SuTI only has 2B parameters, while DreamBooth has 2B parameters per subject, which leads to different model capacity. This could cause the photorealism to be different.\nQ4-Q11: We will fix the typos."}, {"review_id": "qvc9Jn0VGs", "reviewer": "Reviewer_F8hk", "summary": "This paper introduces SuTI for the subject-driven text-to-image (T2I) generation method. Numerous expert models are first trained on millions of image clusters collected from the internet, each focuses on a specific visual subject. A dataset is then created, consisting of concept images, target prompts, and corresponding target images. By training the apprentice model on this dataset, SuTI can generate high-quality and customized subject-specific images without the need for test time fine-tuning. Both the qualitative and quantitative experiments are performed with the baselines.", "questions": "- More details about the experiments should be clarified.\n- Can the proposed model perform multiple concept generation, which is also important in concept image generation?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presented method is well-motivated and easy to understand.\n- SuTI demonstrates fast generation and has a broad domain of applicability.\n- The paper includes extensive experimental evaluations with impressive results.", "weaknesses": "Some details of the experiments require clarification:\n\n- The hyperparameters used for training the baselines, such as the number of training iterations and learning rates, should be provided.\n- More information is needed regarding the human evaluation, including the number of questions, the number of evaluated images, and the number of users involved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment #1 “The hyperparameters used for training the baselines, such as the number of training iterations and learning rates, should be provided”:\n\nThanks for the reminder. We mostly use the official colab from these papers and their default hyper-paramters for the baselines. We will add these details to the revision.\n\nComment #2 “More information is needed regarding the human evaluation, including the number of questions, the number of evaluated images, and the number of users involved.”\n\nWe will add our human evaluation guideline into the Appendix in the revision. The evaluation is done by several trained raters on the DreamBenchv2’s 220 images. \n\nComment #3 “Can the proposed model perform multiple concept generation, which is also important in concept image generation?”\n\nWe evaluated SuTI on 2-concept image generation. The model works well on some easy combinations like “dogs [D] besides vase [V]” or “cat [C] behind a bowl [B]”. But it would fail on more difficult combinations (with subject interaction) like “dog [D] wearing shoes [S]”, etc. We will put some of these results in the revision."}, {"review_id": "TZrchMhN4d", "reviewer": "Reviewer_9TJ3", "summary": "The paper presents a novel subject-driven text-to-image generator named SuTI. This model leverages in-context learning as opposed to subject-specific fine-tuning. SuTI is built upon the principles of apprenticeship learning and is capable of generating high-quality, customized, subject-specific images. Remarkably, it achieves this at a speed that is 20 times faster than optimization-based methods.\n\nSuTI has demonstrated superior performance over existing models on benchmark tests such as DreamBench and DreamBench-v2. The paper highlights the recent advancements in text-to-image generation models, which have shown significant progress in generating highly realistic, accurate, and diverse images from given text prompts.", "questions": "1. Dataset Construction: Could you provide more details about the process of constructing the training dataset for SuTI? Specifically, how much time and resources were required to create the dataset? Were there any significant challenges encountered during this process?\n\n2. Model Scalability: How scalable is the SuTI model with respect to the size and diversity of the training dataset? If the dataset were to be expanded or updated with new subjects or scenarios, would this significantly impact the model's performance or the resources required for training?\n\n", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper exhibits several strengths across the dimensions of originality, quality, clarity, and significance:\n\n1. Originality: The paper introduces SuTI, a novel subject-driven text-to-image generator that uses in-context learning instead of subject-specific fine-tuning. This approach is original and innovative, as it deviates from the conventional optimization-based methods, offering a faster and more efficient solution.\n\n2. Quality: The quality of the paper is evident in the rigorous testing and validation of the SuTI model. The model has been benchmarked against existing models on DreamBench and DreamBench-v2, where it has shown superior performance. This demonstrates the robustness and reliability of the model.\n\n3. Clarity: The paper is well-structured and clear in its presentation of the SuTI model. It provides a comprehensive explanation of the model's workings, its applications, and its performance in various tests. The use of visual aids and examples further enhances the clarity of the paper.\n\n4. Significance: The significance of the paper lies in its contribution to the field of text-to-image generation. By introducing a faster and more efficient model, the paper pushes the boundaries of what is currently possible in this field. This could have far-reaching implications for a variety of applications, including content creation, design, and more.", "weaknesses": "One weakness of the paper is the lack of discussion around the cost and complexity of constructing the training dataset. The process of creating a comprehensive and diverse dataset for training a model like SuTI can be a significant undertaking, both in terms of time and resources.\n\nThe paper does not delve into the specifics of this process, leaving readers without a clear understanding of the potential challenges and costs associated with data collection and preparation. This lack of transparency may make it difficult for others to replicate the study or apply the model in different contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment #1 “Dataset Construction: Could you provide more details about the process of constructing the training dataset for SuTI? Specifically, how much time and resources were required to create the dataset?”\n\nSince we can parallelize the DreamBooth training and generation, we set up 100 instances of TPU v4-8  (128G with 4 chips each). Each instance will take one “subject cluster” and then train one DreamBooth for 500 steps to generate the outputs. Once the output is generated, the model will reset to original weight to move on to the next “subject cluster” without saving any checkpoint. The whole dataset construction takes roughly 100 TPU v4-8 for two weeks. \nWe don’t need to store the DreamBooth checkpoint, so the storage cost is negligible.\n\nComment #2 “Were there any significant challenges encountered during this process?”\n\nThis is a very insightful question. Yes, we did encounter some challenges: \n(1) The memory consumption of the original DreamBooth recipe is very high due to the usage of Adam optimizer, which already reaches the memory ceiling of TPU v4-8. Somehow the memory reset will cause an OOM issue. We replaced Adam with Adafactor to resolve this issue, which decreases the memory consumption significantly to enable our pipeline.\n(2) The other difficulty is how to filter the DreamBooth failure outputs. We tested different variants and conducted lots of human study to decide the CLIP-delta threshold. But still, this threshold is not perfectn with lots of false positives and false negatives.  The community is  still in dire need for a better automatic metric to evaluate subject fidelity.\n\nComment #3 “Model Scalability: How scalable is the SuTI model with respect to the size and diversity of the training dataset? If the dataset were to be expanded or updated with new subjects or scenarios, would this significantly impact the model's performance or the resources required for training?”\n\nSo far, we found the model to be quite scalable with respect to the number of subjects and skill sets we added to the training dataset. For now, we have 5-6 skill-sets and millions of subjects in the dataset. We are still exploring newer skillsets right now, like image editing, image inpainting, etc. One possible way to extend the model’s capacity is to use a small adapter layer to increase models’ capacity to handle more diverse inputs.\n"}, {"review_id": "v0tCqNawEg", "reviewer": "Reviewer_HXFy", "summary": "This paper proposes a method for subject-driven text-to-image generation, where a model is tasked to generate novel renditions of a subject given a few images of that subject. Different from previous fine-tuning approaches, this paper trains a model that conditions its generation on the given subject images. To train such a model, this paper first trains a large amount of subject-specific fine-tuned models, and use these fine-tuned models to generate new training data for knowledge distillation. The distilled model achieves good qualitative performance.\n", "questions": "- Is it possible to finetune the apprenticeship model on new subjects?", "limitations": "Yes the authors have addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- It is nice that the apprenticeship model does not require finetuning on new subjects. It is an interesting idea to directly use subject images as the conditional input for generation.\n- The paper is mostly well-written.\n- The learned model achieves good qualitative results.", "weaknesses": "- The training data covers a wide range of subjects. Therefore, it is hard to tell if the apprenticeship model learns to generalize to new subject or not. Have the authors performed de-duplication to ensure that the subjects used for evaluation do not appear in the training set of the expert models?\n- The proposed method requires training of 2M expert models, where each expert is a 2.1B Imagen model. This is extremely expensive both computational-wise and storage-wise. \n- The inference speed of the apprenticeship model seems to be slow, as each demonstration sample needs to pass through the Imagen model. Methods such as DreamBooth do not incur additional inference cost after the finetuning.\n- It would be nice to see some ablation on the importance of the expert model. For each subject cluster, how many synthetic images are used to train the apprenticeship model? What if only the real images are used to train the apprenticeship model?\n- It would be better if the paper could give a more detailed illustration of the apprenticeship model's architecture, instead of referring to the ReImagen paper. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment #1 “The training data covers a wide range of subjects. Therefore, it is hard to tell if the apprenticeship model learns to generalize to new subjects or not. Have the authors performed de-duplication to ensure that the subjects used for evaluation do not appear in the training set of the expert models?”\n\nWe investigated whether the validated concepts appear in the training dataset. We use CLIP score to retrieve nearest cluster images for each of the 30 concepts in DreamBenchv2. We manually check whether these concepts appear in the training set. We found that very few subjects have similar variants in the training set, and these are mostly  “dogs” and “cats”. Most other subjects like “Robot”, “Vase” are very unique and we can’t find even modestly similar ones.\n\nComment #2 “The proposed method requires training of 2M expert models, where each expert is a 2.1B Imagen model. This is extremely expensive both computational-wise and storage-wise.”\n\nOur method is computationally expensive, but since we don’t store any DreamBooth checkpoints, our storage consumption is very low, i.e. only a few hundred Gigabytes. \n\nComment #3 “The inference speed of the apprenticeship model seems to be slow, as each demonstration sample needs to pass through the Imagen model. Methods such as DreamBooth do not incur additional inference cost after the finetuning.”\n\nThere are two solutions to address this issue: \n(1) SuTI is highly compatible with DreamBooth. We can use the tuned SuTI just like DreamBooth by setting #demonstration=0 during train/inference time, which would lead to the exact same performance. If we want to gain better results, we could provide a single demonstration to the subject-tuned-SuTI, which will yield much better subject fidelity than DreamBooth with only marginal inference overhead (15 secs vs 10 secs).\n(2) We can use distillation in https://arxiv.org/abs/2210.03142 to shorten the diffusion steps by 10x or even more, which can decrease the inference time significantly.\n\nComment #4 “It would be nice to see some ablation on the importance of the expert model. For each subject cluster, how many synthetic images are used to train the apprenticeship model? What if only the real images are used to train the apprenticeship model?”\n\nThanks for the suggestion. We will put more ablation studies to the paper revision. \n1. We have experiments to show how the model performance changes w.r.t the number of clusters, and what’s the minimum number of clusters needed to train SuTI.\n2. We also have conducted experiments to see whether we could just use the clustered images (without DreamBooth expert) to train SuTi. Specifically, we use k-1 images as exemplars and the k-th as the target. However, due to the fact that the clustered images are so similar or near duplicate to each other, this training will guide the model to only copy-paste from demonstration. That’s why we will need more diverse outputs from DreamBooth to discourage the copy-paste behavior. We will add more analysis about this discovery into Appendix.\n\nComment #5: “It would be better if the paper could give a more detailed illustration of the apprenticeship model's architecture, instead of referring to the ReImagen paper.”\n\nThanks for the suggestion. We will add detailed model architecture to the Appendix in the revision.\n\nComment #6 “Is it possible to finetune the apprenticeship model on new subjects?”\n\nYes! We found that fine-tuning on SuTI is better than fine-tuning on the original Imagen model by a significant margin, particularly in terms of the subject fidelity. We plan to include some of these results in the revision."}]

Author rebuttal:
We thank all the reviewers for their constructive feedback. Here we want to highlight a few things:

1. First of all, we are advocator of open research and strive to make everything publicly accessible. We plan to make the model or API publicly available to public before paper publication although we can't guarantee the exact date and time at this point.

2. Regarding the the combination of SuTI and DreamBooth, we conducted more experiments and show our results in the attached pdf.

3. Regarding the ablation study to only use the clustered images to train SuTI, i.e. without relying on the synthetic images by DreamBooth. We provide more evidence in the attached pdf.

Meta-review: This work proposes a subject-driven generation model without need for fine-tuning. Concerns are mainly on the training and inference computation cost, reproducibility, availability of the open-source model. After the rebuttal, concerns were mostly addressed and reviewers reached agreement upon acceptance. I agree with this recommendation and would strongly encourage authors to include reviewers’ suggestions during the revision, and make their model available for other researchers and practitioners, either through API or via open-sourcing.
Decision: Accept (poster)