SlowGuess/NeurIPS_2023_OCR huggingface dataset:
a dog, cyberpunk character
ect-driven Image Editing with Attention Control
Figure 5: Qualitative results categorized by generative capabilities. The pre-trained subject representation enables zero-shot subject-driven generation (row #1-2) and efficient, high-fidelity finetuning (row #3-6). Our model can also serve as a foundation text-to-image generation model and leverages on the fly existing techniques developed on latent diffusion models. Combining with ControlNet (row #7-8), BLIP-Diffusion enables controllable generation using both subject and structure conditions; combining with prompt-to-prompt (row #9-10), our model achieves subject-driven image editing by manipulating cross-attention maps.
Vase
cake
vase with weed in the mountain
a burger at the sea
#
attention map
#
editing output
Subject Images
vase
steps 100
Ours
Textual Inversion
steps 100
a vase as a lamp
DreamBooth
steps 300
steps 800
InstructPix2Pix
low guidance
high guidance
robot
dog
steps 120
steps 40
steps 3000
steps 3000
$\therefore m = \frac{3}{11}$
a robot in the jungle
steps 100
a dog jumping on a cobblestone street
steps 800
1
change the background as a jungle
ge the background as a jungle
make the dog jump on a cobblestone street
Methods	DINO	CLIP-I	CLIP-T
Real Images (Oracle)	0.774	0.885	-
Textual Inversion [8]	0.569	0.780	0.255
Re-Imagen [36]	0.600	0.740	0.270
DreamBooth [9]	0.668	0.803	0.305
- 100 fine-tuning steps	0.396	0.698	0.322
- 300 fine-tuning steps	0.500	0.733	0.319
Ours (ZS)	0.594 (±0.004)	0.779 (±0.003)	0.300 (±0.002)
Ours (FT, avg. < 80 steps)	0.670 (±0.004)	0.805 (±0.002)	0.302 (±0.001)
Figure 6: Qualitative comparison between BLIP-Diffusion with text-driven image generation methods [8, 9] and the image editing method [35]. Our model achieves high subject fidelity and prompt relevance, while needing significantly fewer finetuning steps. Example subject images from the dreambooth dataset [9] are shown, which are also used as source images for InstructPix2Pix to edit.
Table 1: Left: Quantitative comparisons on DreamBench. We report average metrics and differences across 10 experiment runs with different set of random seeds, in zero-shot (ZS) and fine-tuning (FT) setups. Right: Alignment metrics in zero-shot (red) and fine-tuning (blue) setups for sample subjects.
existing solutions, BLIP-Diffusion requires much less fine-tuning effort, usually 40-120 steps which are up to x20 times more efficient than previous work [8, 9]. In addition, when combining with ControlNet (row #7-8), our model can achieve simultaneous control over structure and subject. Finally, our model can introduce subject information into image editing pipeline, enable to edit images with specific subject visuals (row #9-10). These applications demonstrate potentials of using BLIP-Diffusion as a foundation text-to-image generation model with multimodal controls.
Comparisons on DreamBooth Dataset. We compare BLIP-Diffusion with methods [8, 9, 35, 36] (details in appendix) on DreamBooth dataset [9], containing 30 subjects, each with 4-7 images. In Figure 6, we show qualitative comparisons. Our model achieves significantly better subject fidelity than Textual Inversion, Re-Imagen and InstructPix2Pix. Compared with DreamBooth, our model exhibits comparable or better generation quality while requiring a significant lower number of fine-tuning iterations, which validates the effectiveness of our pre-trained subject representation.
In Table 1, we follow [9] and report DINO [37], CLIP-I [20] and CLIP-T scores. DINO and CLIP-I scores measure subject alignment and CLIP-T measures image-text alignment (see appendix for detailed description of the metrics). We generate 4 images for each text prompt, amounting in total 3,000 images for all the subjects. We repeat generations with 10 fixed set of random seeds and report average scores. The overall results are consistent with the qualitative findings, where BLIP-Diffusion is superior to Textual Inversion and Re-Imagen while showing comparable performance to DreamBooth while requiring less fine-tuning effort. In particular, our zero-shot generations are better
Ablation Setups	DINO	CLIP-I	CLIP-T
BLIP-Diffusion (250K steps)	0.566	0.773	0.299
- w/o multimodal pre-training	0.521 ↓	0.743 ↓	0.290 ↓
- w/o training text encoder	0.568 ↑	0.782 ↑	0.288 ↓
- w/o subject text	0.565 ↓	0.772 ↓	0.298 ↓
- w/o subject dropping	0.559 ↓	0.766 ↓	0.291 ↓
Figure 7: Attention visualization of subject representation. Leftmost is the generation using prompt "a robot/car" with subject representation injected. The learned subject embeddings capture both local (red) and holistic (blue) subject visuals. Subjects are from the DreamBench [9] dataset.
than fine-tuned Textual Inversion results. Additionally, we show per-subject metrics and observe that fine-tuning significantly improves subject alignment. In the meanwhile, fine-tuning also improves image-text alignment on average. When fine-tuning harms the image-text alignment, it is due to the model overfitting to target inputs thus resulting in generations irrespective of the text prompt. This is in particular an issue when the provided subject images are of limited visual diversity.
Aesthetics Measures. We use LAION aesthetic scorer to measure the visual quality of the generations. The aesthetic scorer predicts a score bounded between 0 and 10 using CLIP ViT-L/14 features, where 10 represents the highest aesthetic score. We calculate the aesthetic scores for 3,000 generations on the DreamBooth dataset and report comparison results with those produced by DreamBooth. As indicated in Table 3, results show that our model shows a clear advantage over DreamBooth models in terms of the aesthetic scores. In addition, results also demonstrate that fine-tuning is beneficial to promote high-quality generations.
Ablation Studies. We conduct ablation studies using 250K subject representation learning steps. Table 2 shows zero-shot evaluation results. Our findings are: (i) it is critical to conduct multimodal representation learning (Section 3.1), which bridges the representation gap between subject embeddings and text prompt embeddings. (ii) freezing text encoder of the diffusion model worsens the interaction between subject embedding and text embedding. This leads to generations copying subject inputs and not respecting the text prompts. Despite leading to higher subject alignment scores, it does not allow text control, falsifying the task of text-to-image generation. (iii) Giving subject text to the multimodal encoder is helpful to inject class-specific visual priors, thereby leading to moderate improvement in metrics. (iv) Pre-training with random subject embedding dropping helps to better preserve the diffusion model's generation ability, thus benefiting the results. We further demonstrate the effect of subject representation learning. The figure (right) shows that both image-text alignment and subject alignment improve with growing pre-training steps of subject representation learning.
Subject Representation Visualization. It is observed that pixels are attracted more to embeddings that describe them [2]. Following this observation, in Figure 7, we visualize the learned subject embeddings using cross-attention maps. The figure shows that the learned embeddings encode
Table 2: Left: Ablation results. Right: Effect of subject representation learning with varying pre-training steps.
Models	Aesthetic Scores
BLIP-Diffusion (fine-tuned)	6.50
BLIP-Diffusion (zero-shot)	6.43
DreamBooth	6.20
Table 3: Aesthetic scores comparison between BLIP-Diffusion models and DreamBooth models. Our models show a quantitative advantage over DreamBooth models.
fine-grained yet different aspects of the subject. For example, certain embeddings (e.g. 0, 3, 4, 10-13) tend to focus on more local features while others (e.g. 1, 14) encode more holistic visuals. This demonstrates the complimentary effect of employing multiple subject embeddings.
Limitations. Our model suffers from common failures of subject-driven generation models, such as incorrect context synthesis, overfitting to training set as detailed in [9]. In addition, it inherits some weakness of the underlying diffusion model, which may fail to understand text prompts and fine-grained composition relations. We show some of such failure examples in Figure 8. Despite the limitations, the proposed technique is generic to harvest future development of diffusion models.
This paper proposed BLIP-Diffusion, a new text-to-image diffusion model with built-in multimodal control capabilities powered by BLIP-2 [12]. The model is pre-trained using a two-stage strategy to learn progressively multimodal subject representation, which facilitates high-fidelity zero-shot and efficient fine-tuned subject-driven generation. BLIP-Diffusion produces better zero-shot generations than fine-tuned models such as Textual Inversion. It also achieves up to 20x fine-tuning speed up than best prior methods with comparable generation quality. In addition, it can work in conjunction with other established techniques, such as ControlNet and prompt-to-prompt, for image generation and editing with simultaneous structure and subject control. We consider BLIP-Diffusion as an important step towards building foundational text-to-image generation model with multimodal control.
We thank colleagues at Salesforce AI Research for support and discussions, and anonymous program committee members for their voluntary reviews and valuable feedback.
[1] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[2] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
[3] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pages 16784-16804. PMLR, 2022.
[4] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[5] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[6] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[7] Chitwan Sahara, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
[8] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022.
[9] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242, 2022.
[30] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[31] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[32] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018.
[33] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568, 2021.
[34] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[35] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022.
[36] Wenhu Chen, Hexiang Hu, Chitwan Sahara, and William W Cohen. Re-imagen: Retrieval-augmented text-to-image generator. arXiv preprint arXiv:2209.14491, 2022.
[37] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[38] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 2011.
[39] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022.
A.1 Broader Impact
Image generation models are susceptible to be used as tools for generating false content or prompting misinformation. Subject-driven generation could be misused as a tool for generating fake image of individuals. To mitigate this issue, our model has been trained on generic objects where person-related subjects have been purposely removed from the training data. This makes the model weaker at generating fake images using person as subject control.
Our model is built using the pre-trained Stable Diffusion model trained on web-scraped datasets. Therefore, our model inherits some of its shortcomings, such as generating biased contents with social stereotypes, or other NSFW contents if used inappropriately. Our model's ability to precisely control the generation subject can help mitigate certain biases. We can use NSFW detectors to block potential inappropriate content from being generated. Nevertheless, we strongly caution against using our model directly in user-facing applications without a careful inspection of the model's output. Proper content moderation and regulation are highly advised to prevent undesirable consequence.
A.2 Failure Cases Analysis
In Figure 8, we outline common failure cases of the model. Our model suffers from issues observed for prior subject-driven generation models as outlined in [9], including incorrect context synthesis, overfitting to training set. In addition, it subsumes some weakness of the underlying diffusion model, such as failing to address text prompts or generating fine-grained composition relations.
(a) Context-appearance entanglement
Subject images
Figure 8: Example failure generations. Subject images used for finetuning are shown on the left.
A backpack on top of a purple rug in a forest.
(b) Failing to address the text prompt.
Subject images
A cube-shaped bear plushie.
(c) Incorrect spatial composition.
Subject images
A sneaker on top of a mirror.
A.3 Competing Methods
We compare BLIP-Diffusion with fine-tuning based [8, 9] and retrieval-augmented [36] subject-driven generation models on the public DreamBench dataset [9]. We also compare qualitatively with the image editing method InstructPix2Pix [35]. We briefly introduce these methods below.
- Textual Inversion [8]: a fine-tuning method which optimizes a placeholder embedding to reconstruct the training set of subject images. It requires 3,000 training steps for learning a new concept, which takes around 30 minutes on an A100 GPU.
- DreamBooth [9]: a fine-tuning method similar to textual inversion. In addition to the placeholder embedding, it also optimizes parameters of the U-Net for a total budget of around 800 steps. We report intermediate results using 100 and 300 fine-tuning steps, while refer to metrics reported by the authors for full model comparison. Fine-tuning DreamBooth on a new concept costs around 6 minutes on an A100 GPU.
- Re-Imagen: a retrieval-augmented model, which takes the subject images as references and attend to them to generate new images. While the model requires no tuning, it significantly underperforms other models. The model is not publicly available, thus we do not have access to qualitative examples for comparison.
- InstructPix2Pix: an image editing model, which takes as input the source image and an editing instruction to generate edited images. Although it does not represent explicitly subjects, it can be used for applications such as subject re-contextualization and property modification. Therefore, we also include it for qualitative comparison. In particular, we experiment with both low (1.0) and high (1.5) image guidance scales, where a low image guidance scale preserves less the subject while promotes the text alignment; a high image guidance scale preserves better the original image yet is more likely to overlook the editing instruction.
A.4 Evaluation Metrics
We adopt metrics proposed in DreamBooth [9] for evaluation, including DINO, CLIP-I and CLIP-T scores. Among them, DINO and CLIP-I scores are used to measure subject fidelity and CLIP-T is used to measure image-text alignment. DINO score is the average pairwise cosine similarity between the ViT-S/16 DINO embeddings of the generated and real images. CLIP-I score is the average pairwise CLIP ViT-B/32 image embeddings of the generated and real images. It is considered that DINO score is the preferred metric for measuring subject fidelity as it is sensitive to the differences between subjects of the same class. CLIP-T score is the average cosine similarity between prompt and image CLIP embeddings.
To better evaluate and compare subject-driven text-to-image models, it is suggested that these metrics should be considered jointly to avoid biased conclusion. For example, a model that naively copies the training set images will produce high DINO and CLIP-I scores with low CLIP-T scores. In the other case, a vanilla text-to-image generation model without subject knowledge, e.g. stable diffusion, will produce high CLIP-T scores with poor subject alignment. Both models are not considered desirable for the subject-driven text-to-image generation task.
A.5 Pre-training Datasets
For multimodal representation learning, we use the same pre-training data as by BLIP-2, totaling 129M images. This includes COCO [30], Visual Genome [31], CC3M [32], CC12M [33], SBU [38] and 115M images from LAION400M [28]. We also employ the synthetic captions created using CapFilt method [29] for web images. We refer interested readers to Section 3.4 in the BLIP-2 paper [12] for details of the data bootstrapping configurations.
For subject representation learning, we use a subset of OpenImage-V6. We filter the data using the annotations provided by the dataset. In particular, we discard a sample if it satisfies one of the following cases: (i) a group of objects of the same class appear in the image; (ii) the image is taken from inside of the subject; (iii) the object is of aspect ratios larger than 2; (iv) objects occupy a too large (0.8) or too small (0.3) area relative to the image; (v) human-related subject, including boy, girl, person, man, mammal, woman, human body, human head, human hair, human arm, human face, human leg, human hand, human foot, human eye, human mouth, human nose, human ear, clothing, suit; (vi) cluttered objects, including tree, plant, houseplant, desk, table, poster and billboard. This results in 292K images for subject representation learning.
A.6 Fine-tuning, Inference and Evaluation on DreamBooth Dataset
For all fine-tuning experiments, we use AdamW [26] optimizer with constant learning rate 5e-6 and no warm-up steps. We use batch size 3, adam beta1 0.9, adam beta2 0.999, adam epsilon 1e-8 and weight decay 0.01. We fine-tune models on a single A100 (40Gb) GPU and select checkpoints manually based on a set of validation prompts. We report the number of iterations for each subject on DreamBench below, on average 76 steps, taking around 40 seconds to complete on a single A100.
For inference, we use PNDM scheduler [39] for 100 denoising steps. We use a fixed guidance scale 7.5 for all experiments.
Table 4: Number of fine-tuning steps for DreamBench subjects.
backpack	110	backpack-dog	110	bear-plushie	110
bowl	40	can	70	candle	80
cat	40	cat2	50	clock	120
colorful-sneaker	80	dog	50	dog2	50
dog3	40	dog5	20	dog6	40
dog7	50	dog8	40	duck-toy	60
fancy-boot	50	grey-sloth-plushie	70	monster-toy	120
pink-sunglasses	90	poop-emoji	90	rc-car	120
red-cartoon	70	robot-toy	110	shiny-sneaker	80
teapot	120	vase	120	wolf-plushie	80
In Table 5 and 6, we report average metrics across 10 experiment runs for each subject in the dataset, in zero-shot and fine-tuning setups, respectively.
A.7 Zero-shot Subject-driven Image Manipulation
Our model is able to extract subject features to guide the generation. In addition to applications of subject-driven generations and editing, we show that such pre-trained subject representation enables intriguing and useful applications of zero-shot image manipulation, including subject interpolation and subject-driven style transfer.
Subject Interpolation. It is also possible to blend two subject representation to generate subjects with a hybrid appearance. This can be achieved by traversing the embedding trajectory between subjects. In Figure 9, we create bilinear interpolations among 4 different subject representations, and render the interpolated subject in a novel context. As the figure shows, the subject appearance blends along the trajectory and fits naturally with the environment. This is useful when multiple subjects are used as reference to guide the generation. For example, subject interpolation can be used in joint with subject-driven style transfer to create hybrid style from multiple guiding subjects.
Table 5: Average metrics for each subject on DreamBench in zero-shot setup.
Subject	backpack	backpack-dog	bear-plushie	berry-bowl	can	candle
DINO	0.452	0.467	0.634	0.750	0.540	0.395
CLIP-I	0.782	0.712	0.739	0.792	0.641	0.710
CLIP-T	0.320	0.310	0.304	0.257	0.314	0.316
Subject	cat	cat2	clock	colorful-sneaker	dog	dog2
DINO	0.760	0.703	0.402	0.680	0.780	0.730
CLIP-I	0.835	0.854	0.735	0.769	0.849	0.831
CLIP-T	0.306	0.286	0.303	0.298	0.310	0.307
Subject	dog3	dog5	dog6	dog7	dog8	duck-toy
DINO	0.558	0.705	0.763	0.656	0.641	0.665
CLIP-I	0.747	0.788	0.867	0.817	0.816	0.840
CLIP-T	0.310	0.313	0.288	0.309	0.307	0.287
Subject	fancy-boot	grey-sloth-plushie	monster-toy	pink-sunglasses	poop-emoji	rc-car
DINO	0.538	0.632	0.490	0.599	0.494	0.569
CLIP-I	0.800	0.755	0.734	0.836	0.689	0.761
CLIP-T	0.291	0.315	0.293	0.308	0.307	0.281
Subject	red-cartoon	robot-toy	shiny-sneaker	teapot	vase	wolf-plushie
DINO	0.697	0.534	0.668	0.451	0.471	0.463
CLIP-I	0.826	0.787	0.759	0.804	0.786	0.737
CLIP-T	0.263	0.315	0.294	0.314	0.262	0.327
Subject-driven Style Transfer. When provided with a subject, the model can encode the appearance style of it and transfer to other subjects. We refer such an application as subject-driven style transfer. In Figure 10 and 11, we generate stylized reference subjects with the aid of edge-guided ControlNet. The styles are hinted by the guiding subjects. Specifically, we feed BLIP-2 with guiding subjects and their category texts, e.g. fire, flower, glass, vase, ball, bread, to extract the subject representation. In this application, guiding subjects serve as alternative of textual prompts to specify styles. This is useful especially when a style is non-trivial to describe by natural languages accurately.