SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines. We provide a comprehensive list of baselines to compare with the proposed SuTI model:
- DreamBooth [6]: a fine-tuning method with space consumption is  $|M| \times |\mathbb{S}|$ , where  $|M|$  is the model size and  $|\mathbb{S}|$  is the number of subjects.
- Textual Inversion [8]: a fine-tuning method with space consumption is  $|E| \times |\mathbb{S}|$  with the embedding size of  $|E|$ , note that  $|E| \ll |M|$ .
- Null-Text Inversion [12]: a fine-tuning method with space consumption is  $|T| \times |E| \times |\mathbb{S}|$ .
- *Imagic [7]: a fine-tuning-based method with largest space consumption among all models as it requires training  $|M| \times |\mathbb{S}| \times |\mathbb{P}|$ , where  $|\mathbb{P}|$  is the number of a text prompt  $\mathbb{P} = \{\pmb{p}_s\}$  for the subject set  $\mathbb{S}$ .
- InstructPix2Pix [11]: a non-tuning method, which can generate and edit a given image really fast within a few seconds. There is no additional space consumption.
- Re-Imagen [9]: a non-tuning method, which will take a few images as input and then attend to those retrievals to generate a new image. There is no additional space consumption.
Experimental Results. We show our automatic evaluation results on the DreamBench in Table 1. We can observe that SuTI can perform better or on par with DreamBooth on all of the metrics. Specifically, SuTI outperforms DreamBooth on the DINO score by  $5\%$ , which indicates that our method is better at preserving the subject's visual appearance. In terms of the CLIP-T score, our method is almost the same as DreamBooth, indicating an equivalent capability in terms of textual alignment. These results indicate that SuTI has achieved promising generalization to a wide variety of visual subjects, without being trained on the exact instances.
Methods	Backbone	DINO ↑	CLIP-I ↑	CLIP-T ↑
Real Image (Oracle)	-	0.774	0.885	-
DreamBooth [6]	Imagen [1]	0.696	0.812	0.306
DreamBooth [6]	SD [21]	0.668	0.803	0.305
Textual Inversion [8]	SD [21]	0.569	0.780	0.255
Re-Imagen [9]	Imagen [1]	0.600	0.740	0.270
Ours: SuTI	Imagen [1]	0.741	0.819	0.304
We further show our human evaluation results on the DreamBench-V2 in Table 2. It shows the related rankings for the additional storage cost and reported the average inference time measure for inferring on each subject. As can be seen, SuTI is able to outperform DreamBooth by  $5\%$  on the overall score mainly due to much higher textual alignment. In contrast, all the other existing baselines are getting much lower human evaluation score  $(< 42\%)$ .
Comparisons. We compare our generation results with other methods in Figure 4. As can be seen, SuTI can generate images highly faithful to the demonstrated subjects. Though SuTI is still
Table 1: Automatic Evaluation on the DreamBench.
Methods	Backbone	Space	Time	Subject ↑	Text ↑	Photorealism ↑	Overall ↑
Models requiring test-time tuning
Textual Inversion [8]	SD [21]	$	30 mins	0.22	0.64	0.90	0.14
Null-Text Inversion [12]	Imagen [1]	$$	5 mins	0.20	0.46	0.70	0.10
Imagic [7]	Imagen [1]	$$$$$	70 mins	0.78	0.34	0.68	0.28
DreamBooth [6]	SD [21]	$$$$	6 mins	0.74	0.53	0.85	0.47
DreamBooth [6]	Imagen [1]	$$$$	10 mins	0.88	0.82	0.98	0.77
InstructPix2Pix [11]	SD [21]	-	10 secs	0.14	0.46	0.42	0.10
Re-Imagen [9]	Imagen [1]	-	20 secs	0.70	0.65	0.64	0.42
Ours: SuTI	Imagen [1]	-	20 secs	0.90	0.90	0.92	0.82
Table 2: Human Evaluation on the DreamBench-v2. We report an approximated average inference time (averaged over subjects) and the relative rankings of the space cost (more $: more expensive). Methods that do not fine-tune in test-time requires no additional storage (denoted by -).
missing some local textual (words on the bowl gets blurred) or colorization (dog hair color gets darker), the nuance is almost unperceivable for humans. The other baselines like InstructPix2Pix [11], and Null-Text Inversion [12] are not able to perform very sophisticated transformations. Textual Inversion [8] cannot achieve satisfactory results even with 30 minutes of tuning. Re-Imagen [9] though gives reasonable outputs, the subject preservation is much weaker than SuTI. Imagic [7] also generates reasonable outputs, however, its failure rate is still much higher than ours. DreamBooth [6] however generates almost perfect images except for the 'blurry' text on the berry bowl. Through the comparison, we can observe remarkable improvement in the output image quality.
Skillset. We provide SuTI's generation to showcase its ability in re-contextualization, novel view synthesis, art rendition, property modification, and accessorization. We demonstrate these different skills in the Appendix Figure 8. In the first row, we show that SuTI is able to synthesize the subjects with different art styles. In the second row, we show that SuTI is able to synthesize the different view angles of the given subject. In the third row, we show that SuTI can modify subjects' facial expressions like 'sad', 'screaming', etc. In the fourth row, we show that SuTI can alter the color of a given toy. In the last two rows, we show that SuTI can add different accessories (hats, clothes, etc) to the given subjects. Further, we found that SuTI can even compose two skills together to perform highly complex image generation. As depicted in Figure 5, we show that SuTI can combine re-contextualization with editing/accessorization/stylization to generate high-quality images.
Figure 5: SuTI not only re-contextualizes subjects but also composes multiple transformation, all in-context.
5.3 Model Analysis and Ablation Study
We further conducted a set of ablation studies to show factors that impact the performance of SuTI.
Impact of # Demonstrations. Figure 6 presents the SuTI's in-context generation with respect to an increasing number of subject-specific image examples. Interestingly, we observe a transition in the model's behavior as the number of in-context examples increases. When  $\mathbb{C}_s = \emptyset$ , SuTI generates images using it prior to the text, similar to traditional text-to-image generation models such as Imagen [1]. When  $|\mathbb{C}_s| = 1$ , SuTI behaves similarly to an image editing model, attempting to edit the generation while preserving the foreground subject, and avoiding sophisticated transformation. When  $|\mathbb{C}_s| = 5$ , SuTI unlocks the capability of rendering novel pose and shape of the demonstrated subject naturally in the targeted scene. In addition, we also observe that a bigger  $|\mathbb{C}_s|$  would result in a more robust generation of high text and subject alignment, and better photorealism. We also performed a human evaluation on the SuTI's generation with respect to different numbers of demonstrations and visualizes the results in Figure 6 (right). It shows that as the number of demonstrations increases, the human evaluation score first increases drastically and then gradually converges.
Quality of the expert dataset matters. We found that the Delta CLIP score is critical to ensure the quality of synthesized target images. Such a filtering mechanism is highly influential in terms of SuTI's final performance. We evaluated several versions to increase the  $\Delta$  threshold from None  $\rightarrow$ $0.0\to \dots \to 0.025$ , we observe that the human evaluation can steadily increase from 0.54 to 0.82.
Figure 6: (Left) In-context generation by SuTI model, with an increasing # of demonstrations. (Right) Human evaluation score with respect to the increasing % of demonstrations.
Without such intensive filtering, the model's overall human score can go to a very low level (54%). With an increasing  $\Delta$ , although the size of the dataset  $G$  keeps decreasing from 1.8M to around 500K, the model's generation quality keeps improving until saturation. The empirical study indicates that  $\Delta = 0.02$  strikes a good balance between the quality and quantity of the expert-generated dataset  $G$ .
Methods	Inference Time	Subject ↑	Text ↑	Photorealism ↑	Overall
DreamBooth	10 secs	0.88	0.82	0.98	0.77
SuTI	20 secs	0.90	0.90	0.92	0.82
Dream-SuTI	15 secs	0.92	0.92	0.94	0.87
Table 3: Quantitative Human evaluation of the Dream-SuTI model on DreamBench-v2.
Further fine-tuning SuTI improves generation quality We note that our model is not exclusive to methods that requires further fine-tuning, such as DreamBooth [6]. Instead, SuTI can be combined with DreamBooth [6] naturally to achieve better quality subject-driven generation (dubbed as Dream-SuTI). Specifically, given  $K$  reference images regarding a subject, we can randomly feed one image as the condition and use another differently sampled image as the target output. Through fine-tuning the SuTI model for 500 steps (without any auxiliary loss), the Dream-SuTI model can generate aligned and faithful results for a given subject. Table 3 shows a comparison of the Dream-SuTI, against SuTI and DreamBooth, suggesting that Dream-SuTI further improves the generation quality. Particularly, it improves the overall score from 0.82 to 0.87, yielding a  $5\%$  improvement over SuTI, and  $10\%$  improvement over DreamBooth. Since the fine-tuned Dream-SuTI model already trained on all subject images, only one subject image is needed to present during the inference time, which can further reduce the inference cost.
Figure 7: Comparison between DreamBooth, SuTI and Dream-SuTI.
To gain better understanding of the quality, we show an example in Figure 7, where we pick a failure example from SuTI to investigate whether Dream-SuTI improves it. We observe that the DreamBooth does not have strong text alignment, while SuTI's subject lacks fidelity (the generated robot uses legs instead of wheels). With further fine-tuning on subject images, Dream-SuTI is able
to generate images not only faithful to the subject but also to the text description. However, we would like to note that such subject-driven fine-tuned model share the same drawback of a typical Dreambooth model, which can no longer generalize well to a general distribution objects and hence requiring a copy of model parameter per subject.
Text-Guided Image Editing With the surge of diffusion-based models, [22, 8] have demonstrated the possibilities to manipulate given image without human intervention. Blended-Diffusion [23] and SDEdit [24] propose to blend the noise with the input image to guide the image synthesize process to maintain the original layout. Text2Live [25] generates an edit layer on top of the original image/video input. Prompt-to-Prompt [26] and Null-Text Inversion [12] aims at manipulating the attention map in the diffusion model to maintain the layout of the image while changing certain subjects. Imagic [7] propose an optimization based to achieve significant progress in manipulating visual details in a given image. InstructPix2Pix [11] propose to distill image editing training pairs synthesized from Prompt-to-Prompt into a single diffusion model to perform instruction-driven image editing. Our method resembles InstructPix2Pix in a sense that we are training the model on expert-generated images. However, our synthesized data is generated generated by fine-tuned experts, which are mostly natural images. In contrast, the images from InstructPix2Pix are synthetic images. In the experiment section, we comprehensively compare with these existing models to show the advantage of our model, especially on more challenging prompts.
Subject-Driven Text-to-Image Generation Subject-Driven Image Generation tackles a new challenge, where the model needs to understand the visual subject contained in the demonstrations to synthesize totally new scene. Several GAN-based models [27, 28] pioneered to work on personalizing the image generation model to a particular instance. Later on, DreamBooth [6] and Textual Inversion [8, 29] propose optimization-based approach to adapt image generation to a specific unseen subject. However, these two methods are time and space-consuming, which makes them unrealistic in real-world applications. Another line of work adopt retrieval-augmented architecture for subject-driven generation including KNN-Diffusion [30], Re-Imagen [9], however, these methods are trained with weakly-supervised data leading to much worse faithfulness. In this paper, we aim at developing an apprenticeship learning paradigm to train the image generation model with stronger supervision demonstrated by fine-tuned experts. As a result, SuTI can generate customized images about a specified subject, without requiring any test-time fine-tuning. There are some concurrent and related works [31, 32] focusing on specific visual domains such as human faces and / or animals. To our best knowledge, SuTI is the first subject-driven text-to-image generator that operates fully in-context, generalizing across various visual domains.
Our method SuTI has shown strong capabilities to generate personalized images instantly without test-time optimization. Our human evaluation indicates that SuTI is already better in the overall score than DreamBooth, however, we do identify a few weakness of our model: (1) SuTI's generations are less diverse than DreamBooth, and our model is less inclined to transform the subjects' poses or views in the new image. (2) SuTI is less faithful to the low-level visual details than DreamBooth, especially for more complex and often manufactured subjects such as 'robots' or 'rc cars' where the subjects contain highly sophisticated visual details that could be arbitrarily different from the examples inside the training dataset. In the future, we plan to investigate how to further improve these two aspects to make SuTI's generation more diverse and detail-preserving.
We thank Boqing Gong, Kaifeng Chen for reviewing an early version of this paper in depth, with valuable comments and suggestions. We thank Neil Houlsby, Xiao Wang and also the PaLI-X team for providing early access to their Episodic WebLI data. We also thank Jason Baldbridge, Andrew Bunner, Nicole Brichtova for discussions and feedback on the project.
Subject-driven text-to-image generation has wide downstream applications, like adapting certain given subjects into different contexts. Previously, the process was mostly done manually by experts who are specialized in photo creation software. Such manual modification process is time-consuming. We hope that our model could shed light on how to automate such a process and save huge amount of labors and training. The current model is still highly immature, which can fall into several failure modes as demonstrated in the paper. For example, the model is still prone to certain priors presented in certain subject classes. Some low-level visual details in subjects are not perfectly preserved. However, it could still be used as an intermediate form to help accelerate the creation process. On the flip side, there are risks with such models including misinformation, abuse and bias. See the discussion of broader impacts in [1, 4] for more discussion.
[1] Chitwan Sahara, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022.
[2] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[3] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[4] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2022.
[5] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[6] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR, 2023.
[7] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imag: Text-based real image editing with diffusion models. CVPR, 2023.
[8] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022.
[9] Wenhu Chen, Hexiang Hu, Chitwan Sahara, and William W Cohen. Re-imagen: Retrieval-augmented text-to-image generator. International Conference on Learning Representations, 2023.
[10] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022.
[11] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022.
[12] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794, 2022.
[31] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arXiv preprint arXiv:2304.02642, 2023.
[32] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411, 2023.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
To validate the effectiveness, we provide an ablation study to show that higher precision is more important than recall in training the apprentice model. Particularly, when the threshold is set to a lower number (e.g., 0.01 or 0.015), SuTI becomes less stable.
As our goal is to collect images of the same subject, we create an initial subject cluster by grouping all (image, alt-text) pairs that come from the same URL ( $\sim$ 45M clusters), and filter the cluster with less than 3 instances ( $\sim$ 77.8% of the clusters). As a result, it leaves us with  $\sim$ 10M image clusters. We then apply the pre-trained CLIP ViT-L14 model [33] to filter out  $81.1\%$  of clusters that has the average intra-cluster visual similarity between 0.82 and 0.98 to ensure the quality of clusters.
Though the mined clusters already contain (image, alt-text) information, the alt-text's noise level is too high. Therefore, we apply the state-of-the-art image captioning model [10] to generate descriptive text captions for every image of all image clusters, which forms the data triples of (image, alt-text, caption). However, current image captioning models tend to generate generic descriptions of the visual scene, which often occlude the detailed entity information about the subject. For example, generic captions like 'a pair of red shoes' would greatly decrease the expert model's capability to preserve the subject's visual appearance. To increase the specificity of the visual captions, we propose to merge the alt-text, which normally contains specific meta information like brands, names, etc with the model-generated caption. For example, Given an alt-text of 'duggee talking puppet hey duggee chicco  $12\mathrm{m}$  and a caption of 'a toy on the table', we aim to combine them as a more concrete caption: 'Hey duggee toy on the table'. To achieve this, we prompt the pre-trained large language models [18] to read all (alt-text, caption) pairs inside each image cluster, and output a short descriptive text about the visual subject. These refined captions with the mined images are used as the image-text cluster  $\mathbb{C}_s$  w.r.t subject  $s$ , which will be used to fine-tune the expert models.
A.2 SuTI Skillset
We demonstrate the complete view of SuTI's skillset in Figure 8, including styled subject generation, multi-view subject rendering, subject expression modification, subject colorization, and subject accessorization.
A.3 Failure Examples
Figure 9 show some failure examples of SuTI. We show several types of failure modes: (1) the model has a strong prior about the subject and hallucinates the visual details based on its prior knowledge. For example, the generation model believes 'teapot' should contain a 'lift handle'. (2) some artifacts from the demonstration images are being transferred to the generated images. For example, the 'bed' from the demonstration is being brought to the generation, (3) the subject's visual appearance is being modified through, mostly influenced by the context, like the 'candle' contains non-existing artifacts when contextualized in the 'toilet'. These three failure modes constitute most of the generation errors. (4) The models are not particularly good at handling compositional prompts like the 'bear plushie' and 'sunglasses' example. In the future, we plan to work on how to improve these aspects.
A.4 More Qualitative Examples
We demonstrate more examples from DreamBench-v2 in the following figures:
A duck toy
A monster toy
Figure 8: SuTI's in-context generation that demonstrates its skill set. Results generated from a single model. First row: art rendition of the subject. Second row: multi-view synthesis of the subject. Third row: modifying expression for the subject. Fourth row: editing the color of the subject. Fifth row: adding accessories to the subject. Subject (image, text) and editing key words are annotated, with detailed template in the Appendix.
Top-down view
Bottom view
Back view
Chef outfit
Police outfit
Nurse outfit
Fire-Fighter outfit
Ironman outfit
Witch outfit
Superman outfit
Angel outfit
An aged bear plushie pointing to its missing stitches.
A candle floating in the toilet
A dog in Versailles garden.
A teapot is placed on the floor.
A black and white teddy bear is wearing a sunglasses
A fancy boot is worn by a Ragdoll
Figure 9: SuTI's failure examples on DreamBench-v2.
canine dog
a back view of [S] watching a TV show about birds.
[S] eating an icecream in a bowl.
[S] jumping over a creek on a snowy day.
[S] reading a book with a pink glasses on.
[S] talking to a British shorthair in the garden.
Figure 10: Visualization of SuTI's generation on the DreamBench-v2 (Part 1).
border collie
[S] standing on a lush green field.
[S] sniffing a backpack in the airport.
[S] splashing through a river wearing a detective hat.
[S] wearing goggles sticking its head out of a car window.
[S] herding a group of sheep in the thunder storm.
duck toy
[S] floats lazily in the bathtub full of blue bubbles.
[S] waddles across the floor as the a puppy chases after it
[S] sits on a dusty shelf.
[S] on the bed with a nightcap.
a stack of colorful [S] fill the shelves of a toy store.
fancy boot
[S] cross a street.
[S] under the stage lights.
[S] with silver-tipped toes.
[S] with silver-tipped toes kicking a football.
[S] on the stage with bunny sticking its head out.
wolf plushie
[S] playfully chasing a fox plushie through a whimsical forest.
[S] jumping high over a creek.
[S] reading a book.
[S] sitting on a salad bar.
[S] walking towards a lush jungle landscape, with overlying trees, exotic plants
A grey sloth plushie
[S] climbing a tree.
A red monster toy
[S] sitting on a wing chair.
Pink sunglasses
[S] hang on the wall.
A poop emoji toy
[S] on a clock tower.
A clay teapot
[S] on a glass table.
[S] dangles lazily from a backpack.
[S] sitting on a wing chair with a teddy bear.
[S] on a wooden deck overlooking a lake.
[S] under the Tokyo tower.