BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing
Authors: Dongxu Li
Junnan Li
Steven Hoi
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion-based Models
Text-to-Image Generation
Image Editing
Vision-and-Language
Multimodal
TL;DR: A new diffusion model achieves zero-shot subject-driven generation/editing and efficient, high-fidelity few-step finetuning for generic subjects.

Abstract: Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we introduce BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. 
Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representation aligned with the text.
Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions. Compared with previous methods such as DreamBooth, our model enables zero-shot subject-driven generation, and efficient fine-tuning for customized subject with up to 20x speedup. We also demonstrate that BLIP-Diffusion can be flexibly combined with existing techniques such as ControlNet and prompt-to-prompt to enable novel subject-driven generation and editing applications. Implementations are available at: https://github.com/salesforce/LAVIS/tree/main/projects/blip-diffusion.

BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing
Dongxu Lit, Junnan Lit, Steven C.H. Hoi
Salesforce Al Research
CCoressponding authors: (li junn 1i shoi sho.sshhsoeeeesorrrcecommm
Zero-Shot Subject-driven Generation
Few-Step Finetuned Subject-driven Generation
Input Imnage
Input Image
in batman suit
happll jumping
painting by Van Gogh
cubc shuped
dog
backpack
in bucket
swimming cating ramen
with golden surfhce
the grand
Controlled driven Generation
Subject-driven Editing via Cross-attention Control
Input
Input Image
source image
SOUrCe
teapot
bird
depth map
on cupboard
attention map
editing oufput bird in the cuter oil painting
Figure 1: Leveraging pre-trained subject representation. BLIP-Diffusion enables subject-driven generation under efficient fine-tuning or zero-shot setups. The model can also serve as a foundation subject-driven text- to-image generation model and supports applications such as controlled generation and image editing, when combined with techniques such as ControlNet [I] and prompt-to-pnont -o-pmmt 22
Abstract
Subject-driven text-to-image generation models create novel renditions of an input subject based text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we intro- duce BLIP-Diffusion, new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre- pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representation aligned with the text. Then we design subject representation learning task which enables a diffusion model leverage such visual representation and generates new subject renditions. Compared with previous methods such DreamBooth, our model enables zero-shot subject-driven generation, and efficient fine-tuning for customized subject with up 20x speedup. We also show that BLIP-Diffusion can flexibly combined with existing techniques such as ControlNet and prompt-to- prompt to enable novel subject-driven generation and editing applications.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Text-to-image generation models have developed significantly and enabled creation of high-quality images based on textual prompts 377 One of their applications is subject-driven generation, which aims to render novel renditions of an input subject while preserving its appearance. The common approach to subject-driven generation [+ is through inverting subject visuals into text embedding space. Specifically, with pretrained text-to-image generation model, placeholder text embedding is optimized to reconstruct a set of subject images. The embedding is then composed into natural language prompts create different subject renditions. One known inefficiency of this approach is that it requires reiterating hundreds (9 100 or thousands [8] tedious fine-tuning steps for each new subject, which hinders it from efficiently scaling to wide range of subjects.
We attribute such inefficiency to the fact that most pre-trained text-to-image models do not natively support multimodal control using both images and texts as control input. As result, it becomes challenging to learn subject representation that aligns with the text space while capturing the subject visuals with high fidelity. To overcome these limitations, subject-driven text-to-image generation model with pre-trained generic subject representation, which enables subject-driven generation in seot or with few-step fine-tuning. Our model builds upon vision-language encoder (i.e. BLIP-2 [2] and a latent diffusion model [6] (i.e. Stable Diffusion). The BLIP-2 encoder takes input the subject image and its category text; it produces text-aligned subject representation as output. We then infix the subject representation in the prompt embedding to guide the latent diffusion model for subject-driven image generation and editing.
introduce BLIP- LLIP-Diffusion, the first
To enable controllable and high-fidelity generation, we propose a new two-stage pre-training strategy to learn generic subject representation. In the first pre-training stage, we perform multimodal representation learning, which enforces BLIP-2 to produce text-aligned visual features based on the input image. In the second pre-training stage, we design subject representation learning task where the diffusion model Icarns to generate novel subject renditions based on the input visual features. To achieve this, we curate pairs of input-target images with the same subject appearing in different contexts. Specifically, we synthesize input images by composing the subject with a random background. During pre-training, we feed the synthetic input image and the subject class label through BLIP-2 to obtain the multimodal embeddings as subject representation. The subject representation is then combined with text prompt to guide the generation of the target image.
Benefiting from the pre-trained subject representation, BLIP-Diffusion achieves promising zero- shot subject-driven generation results and superior fine-tuning efficiency. For example, BLIP- Diffusion takes 40- 120 fine-tuning steps to specialize for a given subject, achieving up to 20x speedup compared to DreamBooth [9]. Furthermore, BLIP-Diffusiee inherits behaviours of the constituent latent diffusion model and can be flexibly extended to support various subject-driven generative applications without further training. Following the prompt-to-prompl [2] approach, BLIP- Diffusion enables editing images with subject-specifi visuals. When combined with ControlNet (I] it enables abbject-driven generation with various additional structure control.
2 Related Work
2.1 Diffusion Models for Text-to-Image Generation
Diffusion models 4
generate images by progressively denoising a random variable drawn from Gaussian distribution. In this work, we are particularly interested in the pre-trained text-to-image latent diffusion models (6. Given a latent variable = and its noisy version 2; obtained by gradually adding noises to 2 for i. steps, latent diffusion models optimize the following objective:
Ea.co-N(o. [|le- co(2,, t)
(1)
which is the squared error between the added noise and the predicted noise cœ( t) by neural model eg at time step I, given e text prompt as condition. During training, the latent variable is obtained by passing the image into pre-trained encoder [16]. For inference, decoder is employed to convert the denoised latent into an image. In addition to text prompts, our model also conditions on subject representation, rendering an image generation architecture with multimodal conditions.

---

BLIP2 Representation Learning Otjectives
Subjoct Prompt Embedding
[B R m
Muttinodal Encoder
Faed Farward
Text Encoder
U-Aeet
Encodder
Food Forword
Forword
Encoder
Muttimodal Encodar
Creas Abard
Teat Pro
Pronet Emb.
Sell Atrention
Self Atnertin
B
"train
Queries Subject
wesrinng
Text
"A train at the station, the train is"
sunglessee.
Input Image
Queries
Input Text
Input Subject Image
Text Prompt Output Subject Image
(4) Multimodial Representation Learning
(b) Subject Representation Learning
Illustration of the two-staged pre-training for learning stage, we follow prior work [12 and pretrain BLIP-2 encoder to obtain text-aligned image representation. Right: the subject representation learning stage, we synthesize input images by composing subjects with random background. The BLIP-2 is then tasked to produce subject prompt embedding, which is later used by the latent diffusion model to generate output subject image. The image encoder remains frozen during pre-training.
Figure 2:
Left: in the multimodal representation
2.2
Subject-driven Text-to-mmage Generation
Given few images of a subject, the task of subject-driven text-to-image generation aims at generating the subject in novel context based text prompts. In the era of diffusion models, Textual Inversion [ proposes to represent visual concepts using placeholder text embedding, and optimize the embedding to reconstruct the subject images. DreamBooth [9] shares a similar methodology while additionally fine-tunes the diffusion model, which leads to better expressiveness and subject fidelity. One known drawback for both methods is their lengthy fine-tuning time for each new subject, which prevents the approaches from easily scaling up. More recent efforts attempt to reduce the required time and effort for fine-tuning. Concurrent our effort, the work [1I 17 [8] pre-train the diffusion model on domain-specific images, such as cat and human face images. These models provide class-specific prior for generation thus being more efficient for fine-tuning. However, they are also constrained to narrow list of subject categories and are not able to easily generalize to generic subjects. The work SuTI [19 proposes a knowledge distillation approach, which learns zero-shot generation from millions of fine-tuned expert models. Their model shows less flexibility in subject poses and is likely be distracted by the background of the input images. In contrast, the pre-trained representation in our model is generic to wide range of subjects, while generalizing efficiently to different subjects.
3 Method
We propose BLIP-Diffusion, the first image diffusion model that features multimodal control through built-in generic pre-trained subject representation. Specifically, we adapt BLIP-2 encoder to extract multimodal subject representation, which is later used together with text prompt to guide generation.
We aim to learn subject representation that captures the subject-specific visual appearances while in the meantime aligns well with the text prompt. To this end, propose two-staged pre-training strategy as shown in Figure First, multimodal representation learning stage produces text-aligned generic image representation. Second, subject representation learning stage prompts the diffusion model with text and subject representation for subject-driven generation. In this section, we delineate the model design and pre-training strategies.
3.1 Multimodal Representation Learning with BLIP-2
We use Stable Diffusion [6] as the latent diffusion model, which relies on CLIP 20 produce prompt embeddings. In order to guide the generation using both text and subject representa- tion as prompt, it is important that the subject embedding and the text embedding are well-aligned to ensure they can cooperate with each other. Inspired by the recent vision- language pre-trained model BLIP-2 [12], which produces high-quality text-aligned visual representation, we decide to adapt it to extract text-aligned subject representation.
text encoder to
Specifically, as shown in Figure we employ two main modules from BLIP-2 learn multimodal representation: frozen pre-trained image encoder to extract generic image features, and multimodal
3

---

Input image
Figure 3: Left: example training image pairs, target images (top) and input images (bottom) with random background. Images are from Openimage dataset (22). Right: comparison between the subject-driven image generation of our model (top), and image variations from DALLE 2 [4) (bottom). We prompt our model with the text prompt "a dog" and the subject representation, while feeding the input image into DALLE-2 for variation generation. Our model learns subject representation with minimal background information encoded, which enahles faithful and flexible control when used together with text prompts.
encoder (i.e. Q-Former) for image-text alignment. The multimodal encoder is transformer that accepts fixed number of learnable query tokens and an input text. The query tokens interact with text through self-attention layers, and interact with frozen image features through cross-attentin layers, and produces text-aligned image features as output. The output is of the same dimension as the number of query tokens. Empirically, we find that the originally implemented 32 output features often overpower the CLIP text embeddings when used in combination for image generation. Therefore, we instead half the number of query tokens and output 16 features.
Following BLIP-2 pre-training, we jointly train three vision-language pre-training objectives, includ- ing an image-text contrastive learning (ITC) loss that aligns the text and image representation by maximizing their mutual information, an soogge-rrondedd text generation (ITG) loss that generates texts for input images, and an image-text matching (ITM) loss that captures fine-grained image- text alignment via binary prediction. We conduct multimodal representation learning on generic image-text paired data, which allows the model to learn diverse set of visual and textual concepts.
3.2 Subject Representation Learning with Stable Diffusion
As the result of the multimodal representation learning, we obtain text-aligned visual representation of the input image. These features capture the generic semantic information of the input image. However, they are not specifically tailored to serve as guidance for the diffusion model. To this end, the subject representation learning stage aims to enable the diffusion model to leverage such visual representation, and generate different renditions of the subjects when combining with text prompts. In particular, we consider two desired properties when injecting the subject representation into a diffusion model. First, we expect the subject representation to well coordinate with the text prompts for the purpose of text-guided subject-driven generation. In this regard, prior methods [ 18 19 do not address the text prompts during training. They are thus not directly suitable to be used for scalable pre-training. Second, the behavior of the underlying diffusion model should idcally be maintained. This allows the subject-driven generation model to take advantage of techniques built on top of the original model on the fly, such as image editing and structure-controlled generation.
Model Architecture. The proposed model architecture is shown in Figure 2p. We connect the output of the 2LIP-2 multimodal encoder to the input of the diffusion model's 's text encoder. During pre- training, the multimodal encoder takes as input a subject image and a text of the subject category, and produces category-awer
subject visual representation. We then transform the subject representation using a feed-forward layer consisting of two linear layers with GELU (1 activation in-between. The projected features are appended to the text prompt token embeddings as soft visual subject prompt. Specifically, when combining the text token and subject embeddings, we use the template "[text prompt], the [subject text] is [subject prompt]". We pass text tokens through the CLIP embedding layer to obtain text token embeddings. We then concatenate subject embeddings and text token embeddings before passing them to the subsequent CLIP model layers. The resultant CLIP embeddings serve as guidance for the diffusion model to generate the output image. The soft visual prompt makes minimal architectural change to the underlying diffusion model, rendering an effective solution to inject subject representation while in the meantime largely inherits the modeling capabilities of the underlying diffusion model.
Subject-generic Pre-training with Prompted Context Generation. We aim to pre-train the model such that it learns to represent generic subjects from the input image. To this end, naive approach

---

Structure Controlld Subjecl-driven Generation with ControiNet
Subject-driven Image Editing with Attention Control
"A dog cute cyberpunk
originnl gerenttion
Noetroliee Uniie
chanccierr
Text Encodey
hooaatvee Denotsing
Input Text Prompt
Encoder
L--Net
(000
Subject Text Image
shaned weights
1
Source Image
BLIP-2
U-Net
skabsboarding Times Squaire
Input Text
"Doo"
Encocer
subject driven generation
Edited Image
Prompt
'Tyy" Subject Image
BUP
BBB
Output Input Condition
and Text
cooperates with ControlNet for structure and subject controllable generation. Our model maintains the modeling capabilities of the underlying diffusion model and requires re-training of the ControlNet parameters. Right: combined with cross-altention control techniques present in prompt-to- prompt, our model can be used for subject-driven image editing. The figure illustrates one denoising step where we mix latent maps generated with and without subject representation.
Figure 4: Left:
is to use the same image as both imput to the multimodal encoder and output to the diffusion model as in [8 (.. However, our preliminary experiments suggest that this leads to trivial solutions where generations are significantly interfered by the background in the inputs, or even models copying the input image as output, rendering generations not respecting the text prompts. On the other hand, while it is possible to collect multiple images of the same subject in different context, and thereby using different images as input and target, such an approach is laborious to scale up to generic subjects.
To address these issues, we propose a new pre-training task for learning subject-generic representation, called prompted context generation, where we curate input- target training pairs by synthesizing images of the subject in random background. The model takes the synthesized subject image as input, and aims to generate the original subject image as output according to a text prompt. Specifically, given an image containing a subject, we first feed the image and category text of the subject to a rext-prompted segmentation model CLIPSeg 23 with confidence thresholding. We then build a trimap by taking the segmentation map with higher confidence as the known foreground, the lower confidence as the uncertain region and the rest as the known background. Given the trimap, we use closed-form matting [24]] 25 to extract the foreground, namely the subject. Then we compose the extracted subject onto a random background image via alpha blending. Finally, we use the synthetic image the input and the original subject image as the output to serve as one training image pair.
As shown in Figure 3 such synthetic pairs effectively separate the foreground subject and the background context, preventing the subject-inrelevany information from being encoded in the subject prompt. In this way. we encourage the diffusion model to consider jointly the subject prompt and the text prompt for generation, leading to pre-trained model that can be faithfully and flexibly controlled by both the subject image and the text prompt.
During pre-training, we freeze the image encoder and jointly train the BLIP-2 multimodal encoder along with the text encoder and U-Net of the latent diffusion model. To better preserve the text-to-image generation capability, we find it beneficial to randomly drop the subject prompt at a 15% probability while using only text prompts guide the diffusion.
original
3.3 Fine-tuning and Controllable Inference
The pre-trained subject representation enables both zero-shot generation and efficient fine-tuning for specific custom subjects. In addition, our model provides high-level visual control while inheriting the modeling capabilities of the underlying diffusion model. This enables us to leverage established image generation and editing techniques on the fly with BLIP- --fffuiff as the foundation generation model. Below we first describe the efficient few-step subject-specifer fine-tuning for custom subject generation. Then we present the extension capabilities of BI.IPDDiffusin by incorporating existing techniques including ControlNet I and prompt-to -orommpp image editing [2].
5

---

Subject-specific Fine-funing and Inference. The pre-trained generic subject representation enables efficient fine-tuning for highly personalized subjects. Given a few subject images and the subject category text, we first use the multimodal encoder to obtain the subject representation individually. We then initialize the subject prompt embedding using the mean subject representation of all the subject images. In this way, we cache the subject prompt embedding without needing forward pass of the multimodal encoder during fine-tuning. The diffusion model is fine-tuned to generate subject images as target by considering the text prompt embedding and the mean subject embedding. We also freeze the text encoder of the diffusion model, which we find helpful to counteract overfitting to target images. We use batch size 3 and constant learning rate of 5e-5 with A [26] optimizer across all the subjects, and generally observe decent results after 40- 10-120 training steps, which takes 20-40 seconds to complete
single A100 GPU.
Structure-controolled Generation with ControlNet. Our model introduces a multimodal condition- ing mechanism for subject-control. In the meanwhile, the architecture is also compatible to integrate with ControlNet ( to achieve simultaneous structure-come crnttour-controolled and subject-controlled generation. Figure 4 illustrates such integration, where we attach the U-Net of the pre-trained ControlNet to that of BLIP-Diffusion via residuals. In this way, the model takes into account the input structure condition, such as edge maps and depth maps, in addition to the subject cues. Since our model inherits the architecture of the original latent diffusion model, we observe satisfying generations using off- the-shelf integration with pre- trained ControlNet without further training.
Subject-driven Editing with Attention Control. Our model combines subject prompt embeddings with text prompt embeddings for multimodal controlled generation. Inspired by prompt-to-prompl (2), our model enables subject-driven image editing by manipulating the cross-attention maps of prompt tokens. In Figure we show such capability where the model edits the original image with subject- specific visuals. For this purpose, we assume the generation process of the original image is known, or can be derived via inversion [13 (7) for real images. To edit the image, we first specify text tokens to edit, for example the token "dog". " Next, we use the cross-attention maps of the specified token to extract automatically mask for regions to edit. In order to preserve the layout and semantics in unedited regions, we keep the attention maps from the original generation while generating new attention maps for the inserted subject embeddings. We mix the denoising latents at each step based on the extracted editing mask. Namely, latents of the unedited regions are from the original generation whereas latents of the edited regions are from the subject-driven generation. In this way, obtain the edited image with subject-specific visuals while also preserving the unedited regions.
4 Experiments
4.1 Pre-training Datasets and Details.
For multimodal representation learning, we follow BLIP-2 12] and pretrain the model on 129M
image-text pairs, including 1 115M image-text pairs from LAION 28 with CapFilt (29 captions,
COCO 300, Visual Genome (31] and Conceptual Captions 32
33
We use ViT from CLIP 20]
as the image encoder, and initialize Q-Former with BERT base 34 As aforementioned, we use 16
queries to learn subject representation. Other training hyperparameters follow [29]
For subject representation learning, we use subset of 292K images from OpenImage-Ve V6 22 each containing a salient subject. We also remove images with human-related subjects. We use BLIP-2 OPT677 to generate captions as text prompts. We obtain a set of 59K background images from the web to synthesize subject inputs. We use Stable Diffusion v1-5 as the foundation diffusion model, We use total batch size 16 with constant learning rate 2e-6 for 500K steps using AdamW (26) optimizer, taking days to finish on 16 A 100 40Gb GPUs More details on hyperparameters and data filtering procedure are included in Appendix for reference.
4.2 Experimental Results
Main Qualitative Results. In Figure 5 we show qualitative generation results of BLIP- -iff-Diifusion. Thanks to the pre-trained subject representation, our model facilitates zero-shot subject-driven gener- ation (row #1), producing meaningful results even for highly customized subjects. The model also enables efficient fine-tuning (row #3-6), demonstrating high-fidelity generation for re-contexulization, artistic stlyization, textual modification, property modification and accessorization. Compared with

---

#1-2
Zero-Shot Subvect-driven Generation
Inpest teug
skatebbored
plushic
paining Van Gogh painted green
in forest
with sunset
wearing lop hat
on rug
Puince Vanselles seen from back
made of lego ai grand cunyon
on the beach
decurated om cup wearing glasses
#3-6
Few-step Fine-tuned Subject-driven Generation
Re-contextualizatin
Artistic Stylization
nubot
the Acmopolis
pencil sketch
Chinese ink painting
at Eillol lower mountain Fuji
a jungle
comic
printed shirt
as book cover
oil pointing
Modification
Property Modification
Accessories
toy
gold scuipture
in police suit
suit
scuplture
sculpture
poper sculpture Structure Controlled Subject-driven Generation
perfume stomizer
tospot
in superman suit iron suit in batman suit
#7-8
Traniig
scapot
bay
teapot
depth
sofa
edges
skateboardingg times
HED edges
pouring milk
row #9-10
Subject-driven Image Editing with Attention Control
Tranniis
clogg
Tosteed 2a
Y250
ViSe With woed
attention map
editing output
map
dog, cyberpunk
editing output
burger the attention map
editing output
Figure 5: Qualitative results categorized by generative capabilities. The pre-trained subject representation enables zero-shot subject-driven generation (row #1-2) and efficient, high-fidelity finetuning (row #3-6). Our model can also serve foundation text-to-image generation model and leverages the fly exsiting techniques developed latent diffusion models. Combining with ControlNet (row #7-8), BL -if--uufusi enables control- lable generation using both subject and structure conditions: combining with prompt-to-promp (row #9-10), our model achieves subject-driven image editing by manipulating cross-attention maps.

---

Subject Images
Ours
Textual Inversion
DreamBooth
InstructPix2Pix
steps 100
steps 3000
steps 100
steps 300
steps 800
low guidance
high guidance
vase
vasc
lamp steps 300
change the vasc into a lamp of the same style
steps 120
steps 3000
steps 100
steps 800
robot
a ro robot jungle
change the background jungle
steps
steps 3000
stops 100
steps 300
steps 800
dog
dog jumping on cobblestonc street
make the dog jump on a cobblestone street
Figure 6: Qualitative comparison between -iff-Difusiion with text-driven image generation methods and image editing method (35). Our model achieves high subject fidelity and prompt relevance, while needing significantly fewer finetuning steps. Example subject images from the dreambooth dataset (9] are shown, which are also used as source images for InstructPix2Pir to edit.
S
Methods
DINO
CLIP-
CLIP-T
0.52
Real Images (Oracle)
0.774
0.885
0.31
Textual Inversion
g
0.569
0.780
0.255
0.30
Re-Imagen 36
0.600
0.29
DreamBooth
0.668
0,803
0.305
100 fine-tuning steps
0.396
0.698
0.322
cuPa Scome
0.2
lacoHoLAn
- 300 fine-tuning steps
0,500
0.733
0.319
0.20
Ours (ZS)
0.594 (1:0.004)
0.779 (=0.003)
0.300 (100.002
0.2
Ours (FT, avg. 80 steps) 0.670 (=:0.004) 0.805 (=0.002) 0.302 (#0.001)
DINO Score (3u Nect Algement)
Table I: Left: Quantitative comparisons on DreamBench. We report average metrics and differences across 10 experiment with different of random seeds, in zero-shot (ZS) and fine-tuniie -uning (FT) setups. Right: Alignment metrics in zero-shot (red) and fine-tuning (blue) setups for sample subjects.
existing solutions, LLI--Diffusion requires much less fine-tuning effort, usually 40- 10-120 steps which are up to x20 times more efficient than previous work [8 9. In addition, when combining with ControlNet (row #7-8), model can achieve simultaneous control over structure and subject. Finally, our model can introduce subject information into image editing pipeline, enable to edit images with specific subject visuals (row #9-10). These applications demonstrate potentials of using BLIP-Diffusion as a foundation eext-to-image generation model with multimodal controls.
Comparisons DreamBooth Dataset. We compare BLIP-Diffusion with methods 9 35 36
(details appendix) on DreamBooth dataset [9, containing 30 subjects, each with 4-7 images. In Figure 6 we show qualitative comparisons. Our model achieves significantly better subject fidelity than Textual Inversion, Re-Imagen and InstructPix2Pixx. Compared with DreamBoothh our model cxhibits comparable or better generation quality while requiring significant lower number of fine-tuning iterations, which validates the effectiveness of our pre-trained subject representation.
In Table we follow 19 and report DINO 37 CLIP-I 20 and CLIP-T scores. DINO and CLIP-I scores measure subject alignment and CLIP-T measures image-text alignment (see appendix for detailed description of the metrics). We generate 4 images for each text prompt, amounting in total 3,000 images for all subjects. We repeat generations with 10 fixed set of random seeds and report average scores. The overall results are consistent with the qualitative findings, where BLIP. Diffusion superior to Textual Inversion and Re-Inn while showing comparable performance to DrothBooth while requiring less fine-tuning effort. In particular, zero-shot generations better

---

0.50
Ablation Setups
DINO
CLIP-1
CLIP-T
2575
00302
2550
BLIP-Diffusion (250K steps)
0.566
0.773
0.299
10.40
w/o multimodal pre-training
0,521 4 0.743 4
0.290 1
0.06
w/o training text encoder
0.568 1
0.782 1
0.288 1
w/o subject text
0.565 4
0.772 4
0.298 L
DIV Score
w/o subject dropping
0.559 +
0.766 +
0.291
6425
CLIPT Score
2,28
Pre-training Steps
Table 2: Left: Ablation results. Right: Effect of subject representation learning with varying pre-training steps.
robot
0
2
6
9
10
12
13
15
Figure 7: Attention visualization of subject representation. Leftmost is the generation using prompt "a robot/car" with subject representation injected. The learned subject embeddings capture both local (red) and holistic (blue) subject visuals. Subjects are from the DreamBench [9] dataset.
than fine-tuned Textual Inversion results. Additionally, we show per-subject metrics and observe that fine-tuning significantly improves subject alignment. In the meanwhile, image-text alignment on average. When fine-tuning harms the image-text alignment, it is due to the model overfitting to target inputs thus resulting in generations irrespective the text prompt. This in particular issue when the provided subject images are of limited visual diversity.
also improves
Aesthetics Measures. We use LAION aesthetic scorer measure the visual quality of the generations. The aesthetic scorer predicts a score bounded between and 10 using CLIP ViT-L/14 features, where 10 represents the highest aesthetic score. We calculate the aesthetic scores for 3,000 generations on the DreamBooth dataset and report comparison results with those produced by DreamBooth. As indicated in Table results show that our model shows a clear advantage over DreamBooth models in terms of the aesthetic scores. In addition, results also demonstrate that fine-tuning is beneficial to promote high-quality generations.
Ablation Studies. We conduct ablation studies using 250K subject representation learning steps. zero-shot evaluation results. Our findings are: (i) is critical conduct multimodal representation learning (Section 3.1). which bridges the representation gap between subject embed- dings and text prompt embeddings. (ii) freezing text encoder of the diffusion model worsens the interaction between subject embedding and text embedding. This leads to generations copying subject inputs and not respecting the text prompts. Despite leading to higher subject alignment scores, does not allow text control, falsifying the task of text-to-image generation. (iii) Giving subject text to the multimodal encoder is helpful to inject class-specific visual priors, thereby leading to moderate improvement in metrics. (iv) Pre-training with random subject embedding dropping helps to better preserve the diffusion model's generation ability, thus benefiting the results. We further demonstrate the effect of subject representation learning. The figure (right) shows that both image-text alignment and subject alignment improve with growing pre-training steps subject representation learning.
Subject Representation Visualization. It is observed that pixels are attracted more to embeddings that deseribe them 2 Following this observation, Figure 7 we visualize the learned subject embeddings using cross-attention maps. The figure shows that the learned embeddings encode
Models
Aesthetic Scores
L.fP-Diffusion (fine-tuned)
6.50
BLIP-Diffusin (zero-shot)
6.43
DreamBooth
6.20
Table 3: Aesthetic scores comparison between BLIP- --fffuusoisi models and DreamBooth models. Our models show a quantitative advantage over DreamBooth models.

---

fine Line-grained yet different aspects of the subject. For example, certain embeddings (e.g. 0, 0,3,4, 10-13) tend focus on more local features while others (e.g. 14) encode more holistic visuals. This demonstrates the complimentary effect of employing multiple subject embeddings.
Limitations. Our model suffers from common failures of subject-driven generation models, such as incorrect context synthesis, overfitting to training set as detailed in [9]. In addition, it inherits some weakness of the underlying diffusion model, which may fail to understand text prompts and fine-grained composition relations. We show some of such failure examples in Figure 8 Despite the limitations, the proposed technique generic to harvest future development of diffusion models.
Conclusion
This paper proposed BLIP-Diffusonn, a new text-to-image diffusion model with built-in multimodal control capabilities powered by BLIP-2 12] The model is pre-trained using a two-stage strategy to learn progressively multimodal subject representation, which facilitates high-fidelity zero-shot and efficient fine-tund sinettuedd subject-driven generation. BLIP-Diffusin Difffusion produces better zero-shot generations than fine-tuned models such as Textual Inversion. It also achieves up to 20x fine-tuning speed up than best prior methods with comparable generation quality. In addition, it can work in conjunction with other established techniques, such as ControlNet and prompt-to- pooppttto-proppt for image generation and editing with simultaneous structure and subject control. We consider BLIP- Diff-ifusioon as an important step towards building foundational text-to-image generation model with multimodal control.
Acknowledgement
We thank colleagues Salesforce Al Research for support and discussions, and anonymous program committee members for their voluntary reviews and valuable feedback.
References
[1] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 2023.
[2] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman. Yael Pritch, and Daniel Cohen Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint urXiv:2208.0164: 2022.
[3] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, 1lya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference Machine Learning, pages PMLR, 2022.
[4] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/23 2022.
[5] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-imape generation. In International Conference on Machine Learning. pages 8821 18 PMLR, 2021.
[6] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10699 2022.
[7] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan. S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to- to- image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 11477, 2022.
[8] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit II Bermano, Gal Chechik, and Daniel Cohen- Or. An image worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01611 2022.
[9] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driee driven generation. arXiv preprint arXiv:2208.122 12242, 2022.
10

---

[10] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-conceptt customization of text-to-image diflusion. arXiv preprint arXiv22112.04488, 2022.
[11] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. Designing an encoder for fast personalization of text-to- image models. arXiv preprint arXiv:2302. 2023.
[12] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping auguage-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 2023.
[13] Jaaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations. 2021.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilisttic models, Advances in Neural Information Processing Systems, 33:6840-688 6851, 2020.
[15] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8780-8794. Curran Associates, Inc., 2021.
[16] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6111 2013.
[17] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingho Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero tin-ttnnig image customization with text-to-image diffusion models. arXiv preprint arXiv:2304.0268 026642 2023.
[18] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without rest-time finetuning. arXiv preprint arXi: 2023.
[19] Wenhu Chen, Hcxiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming- Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning, arXiv preprint arXhv:2304.00//88 2023.
[20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763 8763. PMLR, 2021.
[21] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 2016.
[22] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, al. The open images dataset v4: Unitied image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7):1956 19861181,, 2020.
[23] Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7086-7096, 2022.
[24] Anat Levin, Dani Lischinski, and Yair Weiss. A closed-form solution to natural image matting. IEEE transactions patlern analysis and machine intelligence, 30(2):228-24) 2007.
[25] Thomas Germer, Tobias Uelwer, Stefan Conrad, and Stefan Harmeling. Pymatting: A python library for alpha matting, Journal of Open Source Software, 5(54):2481, 2020.
2020.
[26] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017.
[27] Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. arXiv preprint arXiv:2211. 122446, 2022.
[28] Christoph Schuhmann, Richard Veneu, Romain Bcaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.0211 2021.
[29] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image image pre- training for unified vision- languaee understanding and generation. In International Conference on Machine Learning, pages 12888-18 12900. PMLR, 2022.

---

[30] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 740-755 Springer, 2014.
[31] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32 73, 2017.
[32] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 2556 2565, 2018.
[33] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVE Conference Computer Vision and Puttern Recognition, pages 5558-3566. 2021.
[34] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[35] Tim Brooks, Aleksander Holynski, and Alexii A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 2022.
[36] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval-augmenttd text-to-image generator. arXiv preprint Xiv::22009 14491, 2022.
[37] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-960,, 2021.
[38] Vicente Ordonez, Girish Kulkami, and Tamara Berg, Im2text:
Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 2011.
[39] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022.
A
Appendix
A.1 Broader Impact
Image generation models are susceptible to be used as tools for generating false content or prompting misinformation. Subject-driven generation could be misused as tool for generating fake image of individuals. To mitigate this issue, our model has been trained generic objects where person- related subjects have been purposely removed from the training data. This makes the model weaker at generating fake images using person as subject control.
Our model is built using the pre-trained Stable Diffusion model trained on web-scraped datasets. Therefore, our model inherits some of its shortcomings, such as generating biased contents with social stereotypes, or other NSFW contents if used inappropriately. Our model's ability to precisely control the generation subject can help mitigate certain biases. We can use NSFW detectors to block potential inappropriate content from being generated. Nevertheless, we strongly caution against using our model directly in user-facing applications without careful inspection of the model's 's output. Proper content moderation and regulation are highly advised to prevent undesirable consequence.
A.2 Failure Cases Analysis
In Figure g we outline common failure cases of the model. Our model suffers from issues observed for prior subject-driven generation models as outlined in 9. including incorrect context synthesis, overfitting to training set. In addition, it subsumes some weakness of the underlying diffusion model, such failing to address text prompts or generating fine- fine-grained composition relations.
12

---

Comlext-appearince entanglement
(b) Failing to address the text prompt.
(c) Incorrect spatial compusition.
Subject images
A backpack top a purple rug forest.
Subject images
A cube-shaped bear plushic.
Subjoct images
sneaker top mirror.
Figure Example failure generations. Subject images used for finetuning are shown on the left.
A.3
Competing Methods
We compare BLIP-Diffusion with fine-tuning based 8 and retrieval-an -aggeette [36] subject-driven generation models on the public DreamBench dataset [9] We also compare qualitatively with the image editing method InstructPix2Pix [35 We briefly introduce these methods below.
Textual Inversion [8] fine-tuning method which optimizes placeholder embedding to reconstruct the training set of subject images. It requires 3,000 training steps for learning a concept, which takes around 30 minutes
A100 GPU.
DreamBooth 1 a fine-tuning method similar to textual inversion. In addition to the placeholder embedding, it also optimizes parameters of the U-Net for total budget of around 800 steps. report intermediate results using 100 and 300 fine-tuning steps, while refer to metrics reported by the authors for full model comparison. Fine-tuning DreamBooth on a new concept costs around 6 minutes on an A100 GPU.
Re-Imagen: retrieval-augmennte model, which takes the subject images references and attend to them to generate new images. While the model requires no tuning, it significantly underperforms other models. The model is not publicly available, thus we do not have access to qualitative examples for comparison.
InstructPix2Pir: image editing model, which takes as input the source image and an editing instruction to generate edited images. Although does not represent explicitly subjects, it can be used for applications such subject re-contextualization and property modification. Therefore, we also include it for qualitative comparison. In particular, we experiment with both low (1.0) and high (1.5) image guidance scales, where low image guidance scale preserves less the subject while promotes the text alignment; a high image guidance scale preserves better the original image yet is more likely to overlook the editing instruction.
A.4 Evaluation Metrics
We adopt metrics proposed in DreamBooth for evaluation, including DINO, CLIP-I and CLIP-T scores. Among them, DINO and CLIP-I scores are used to measure subject fidelity and CLIP-T is used to measure image-text alignment. DINO score is the average pairwise cosine similaity between the ViT-S/16 DINO embeddings of the generated and real images. CLIP-I score is the average pairwise CLIP ViT-B/32 image embeddings of the generated and real images. It considered that DINO score is the preferred metric for measuring subject fidelity is sensitive the differences between subjects of the same class. CLIP-T score is the average cosine similarity between prompt and image CLIP embeddings.
To better evaluate and compare subject-driven lext-to-image models, is suggested that these metrics should be considered jointly to avoid biased conclusion. For example, a model that naively copies the training set images will produce high DINO and CLIP-I scores with low CLIP-T scores. In the other case, vanilla text-to-image generation model without subject knowledge, e.g. stable diffusion, will produce high CLIP-T scores with poor subject alignment. Both models are not considered desirable for the subject-driven text-to-image generation task.
13

---

A.5 Pre-training Datasets
For multimodal representation learning, we use the same pre- are-training data as by BLIP-2, totaling
129M images. This includes COCO [30], Visual Genome [3], CC3M [32], CC12M [33 SBU [38]
and 115M images from LAION400M (28]. We also employ the synthetic captions created using
CapFilt method [29 for web images. We refer interested readers to Section 3.4 in the BLIP-2
paper [12
for details of the data bootstrapping configurations.
subset of OpenImage- V6. We filter the data using the annotations provided by the dataset. In particular, we discard a sample if it satisfies one of the following cases: (i) a group of objects of the same class appear in the image; (ii) the image taken from inside of the subject; (iii) the object is of aspect ratios larger than 2; (iv) objects occupy a too large (0.8) or too small (0.3) area relative to the image; (v) human-related subject, including boy. girl, person, man, mammal, woman, human body, human head, human hair, human arm, human face, human leg, human hand, human foot, human eyc, human mouth, human nose, human ear, clothing, suit; (vi) cluttered objects, including tree, plant, houseplant, desk, table, poster and billboard. This results in 292K images for subject representation learning.
For subject representation learning.
A.6 Fine-tuning, Inference and Evaluation on DreamBooth Dataset
For all fine-tuning experiments, we use Adam [26] optimizer with constant learning rate 5c-6 and no warm-up steps. We use batch size 3, adam betal 0.9, adam beta2 0.999, adam epsilon Ic-8 and weight decay 0.01. We fine-tune models on a single A 110 (40Gb) GPU and select checkpoints manually based on set of validation prompts. We report the number of iterations for each subject on DreamBench below, on average 76 steps, taking around 40 seconds to complete on single Á 100.
For inference, we use PNDM scheduler 39 for 100 denoising steps. We use a fixed guidance scale 7.5 for all experiments.
Table 4: Number of fine-tuning steps for DreamBench subjects.
backpack bowl
110
backpack-dog
110
bear-plushie
110
40
can
70
candle
80
cat
40
cat2
50
clock
120
colorful-sneaker
80
dog
50
dog2
50
dog3
40
dog5
20
dogó
40
dog7
50
dog8
40
duck-toy
60
fancy-boot
50
prey-sloth-plushie
70
monster-toy
120
monk-sunglassee
90
poop-emoji
90
rc-car
120
red-cartoon
70
robot-toy
110
shiny-sneaker
80
teapot
120
vase
120
wolf-plushie
80
In Table 5 anddd 6] we report average metrics across 10 experiment runs for each subject in the dataset, in zero-shot and fine-tuning setups, respectively.
A.7 Zero-shot Subject-driven Image Manipulation
Our model is able to extract subject features to guide the generation. In addition to applications of subject-driven generations and editing, we show that such pre-trained subject representation enables intriguing and useful applications of zero-shot image manipulation, including subject interpolation and subject-driven style transfer.
Subject Interpolation. It is also possible to blend two subject representation to generate subjects with a hybrid appearance. This can achieved by traversing the embedding trajectory between subjects. In Figure 9 we create bilinear interpolations among 4 different subject representations, and render interpolated subject a novel context. As the figure shows, the subject appearance blends along the trajectory and fits naturally with the environment. This is useful when multiple subjects are used as reference guide the generation. For example, subject interpolation can be used joint with subject-driven style transfer to create hybrid style from multiple guiding subjects.
14

---

Table Average metrics for each subject on DreamBench in zero-shot setup.
Subject
backpack
backpack-dog
bear-plushic
berry-bowl
can
candle
DINO
0,452
0.467
0.634
0.750
0.540
0.395
CLIP-I
0.782
0.712
0.739
0.792
0.641
0.710
CLIP-T
0.320
0.310
0.304
0.257
0.314
0.316
Subject
cat
cat2
clock
colorful-sneaker
dog
dog2
DINO
0.760
0.703
0.402
0.680
0.780
0.730
CLIP-I
0.835
0,854
0.735
0.769
0.849
0.831
CLIP-T
0.306
0.286
0.303
0.298
0.310
0.307
Subject
dog3
dog5
dog6
dog7
dog8
duck-toy
DINO
0.558
0.705
0.763
0.656
0.641
0.665
CLIP-I 0.747
0.788
0.867
0.817
0.816
0.840
CLIP-T
0.310
0.313
0.288
0.309
0.307
0.287
Subject fancy-boot grey-sloth-plushie monster-toy pink-sunglasses poop-emoji
re-car
DINO 0.538
0.632
0.490
0.599
0.494
0.569
CLIP-I
0.800
0.755
0.734
0.836
0.689
0.761
CLIP-T
0.291
0.315
0.293
0.308
0.307
0.281
Subject red-cartoon
robot-toy
shiny-sneaker
teapot
vase
wolf-plushic
DINO 0.697
0.534
0.668
0.451
0.471
0.463
CLIP-I 0.826
0.787
0.759
0.804
0.786
0.737
CLIP-T 0.263
0.315
0.294
0.314
0.262
0.327
Subject-driven Style Transfer: When provided with a subject, the model can encode the appearance style of and transfer other subjects. We refer such an application subject-driven style transfer. In Figure 10 and 11 we generate stylized reference subjects with the aid of edge-guided ControlNet. feed BLIP-2 with guiding subjects
The styles hinted by the guiding subjects. Specifically, and their category texts, e.g. fire, flower, glass, vase, ball, bread, to extract the subject representation. In this application, guiding subjects serve as alternative of textual prompts specify styles. This is useful especially when a style is non-trivial to describe by natural languages accurately.
A.8 Additional Qualitative Results and Subject Fidelity Showcasing
provide additional qualitative results DreamBench subjects and prompts. We show the reference subject image in the first column. In the rest columns, we provide generated renditions. To showcase subject fidelity and photorealism, purposely mix one genuine subject image in and leave for interested readers to figure out. Read the captions to verify.
In Figure 12
14
15

---

Table Average metrics for each subject on DreamBench fine-tuning setup.
Subject
backpack
backpack-dog
bear-plushie
berry-howl
candle
DINO 0.551
0.639
0.693
0.808
0.618
0.519
CLIP-1
0.839
0.760
0.752
0.829
0.695
0.752
CLIP-T
0.320
0.317
0.307
0.254
0.313
0.311
Subject
cat2
clock
colorful-sneaker
dog
dog2
DINO
0.806
0.747
0.479
0.739
0.821
0.793
CLIP-I
0.869
0.864
0.784
0.805
0.860
0.841
CLIP-T
0.306
0.284
0.305
0.320
0.313
0.307
Subject
dog3
dog5
dog6
dog7
dog8
duck-toy
DINO 0.573
0.727
0.834
0.672
0.723
0.699
CLIP-I
0.751
0,801
0.891
0.823
0.823
0.838
CLIP-T
0.312
0.311
0.280
0.310
0.310
0.284
Subject fancy-boot grey-sloth-plushiee monster-toy pink-sunglasses poop-emoji
re-car
DINO
0.649
0.717
0.566
0.625
0.627
0.651
CLIP-I
0.827
0.780
0.743
0.826
0.784
0.775
CLIP-T
0.299
0.322
0.292
0.312
0.290
0.288
Subject red-cartoon
robot-toy
shiny-sneaker
teapot
vase
wolf-plushic
DINO 0.788
0.626
0.757
0..844
0.628
0.599
CLIP-I
0.882
0,803
0.804
0.819
0.812
0.760
CLIP-T
0.262
0.316
0.297
0.331
0.261
0.325
Figure Zero-shot subject interpolation. interpolate subject representation use same denoising and decoder network for generation. The intermediate subject representation naturally blends the subject appearance, while fitting coherently into the new context.
16

---

BLIP BLIP BLIP
BLIP
BLIP
BLIP
BLIP
BLIP
BLIP
BLIP
BLIP
BUP B BLP BUP
BLIP
BIP
BLIP
BLIP
Figure 10: Zero-shot subject-driven stylization. We show guiding subject images on top. In the rest rows, we show reference subjects and their canny maps on left, and stylized reference subjects by column.

---

BLIP BLIP BLIP
BLI 
BLPP
BLIP BLIP BLIP
BLP BLIP
BLIP
BLI BLP BLI
B
BUP
BLP
BLIP
Figure II: (Cont.) Zero-shot subject-driven stylization. We show guiding subject images on top. In the rest rows, we show reference subjects and their canny maps on left, and stylized reference subjects by column.
18

---

on
on beach
original backpack
cobblestonc street
backpack-dog
top of green grass with sunflowers around it
purple backpack
backpack
snow
on top of a white rug
cuho-shaped
with a wheat field in the background
in the jungle
monster-toy
with city in the background
original toy
on purple in the furest
on the beach
in the jungle
on top of mirror
an original teapot
a teapol
teapot
with tree and top a pink fabric with wheat field the floating top of water background
leaves in the background
Figure 12: Additional qualitative results using DreamBench subjects and prompts. To showcase subject fidelity and photorealism, mix one genuine subject image in the generations for readers to figure out. Zoom-in and read the captions verify.
19

---

a purple rug the an original sunglasses
top pink fabric
jungle
pink-sunglasses
a purple sunglasses
on top of mirror
on cobblestone street
on top white rug
with mountain in the background
in the jungle
purple candle
red candle
candle
with a tree and autumn leaves in the background
on dirt road
original candle
with a wheat field in the background
house background
cube- shuped
firefighter
dog
original dog
wearing black top hat and monocle
in the jungle
wearing Santa hat
Figure 13: (Cont.) Additional qualitative results using DreamBench subjects and prompts. To showcase subject fidelity and photorealism, we mix one genuine subject image the generations for readers figure out. Zoom-in and read the captions to verify.
20

---

an original cartoon
with city the background
on cobblestone street
blue house background
rod-cartoon
with the Eiffel Tower the background
mountain background
on top of green grass with sunflowers around
with a tree and autumn leaves in the background
plushie
original plushic
on top of green grass with ounflowers around
bear-plushic
with the Eiffel Tower in the background
top of mirror
in the jungle
with mountain the background
on white rug
on the beach
in the snow
the jungle
clock
an original clock on top of wooden floor with city the background
with mountain in the background
Figure 14: (Cont.) Additional qualitative results using DreamBench subjects and prompts. To showcase subject tidelity and photorealism, we mix one genuine subject image the generations for readers figure out. Zoom-in and read the captions to verify.
21


Reviews:
[{"review_id": "UB6TmDgUqM", "reviewer": "Reviewer_dpqw", "summary": "The paper addresses the issues of lengthy fine-tuning and preserving the subject fidelity in subject-driven text-to-image generation models. Different from existing models such as Textual Inversion and Dreambooth that invert subject visuals into text embedding space, the paper introduces a new multimodal encoder which is pre-trained to produce visual representation aligned with the text. New subject renditions are then generated using such visual representation. The paper highlights the utility of the proposed approach in zero-shot subject-driven generation and editing applications, and demonstrates 20x speedup in fine-tuning for customized subjects.", "questions": "Please see weaknesses.", "limitations": "Authors adequately addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-motivated and well-written. The key idea of the work to deeply align subject embedding and the text embedding is interesting. The proposed model is compatible with ControlNet and Prompt-to-Prompt, which has potential to unleash several important editing capabilities. The experiment results are solid, with sufficient evaluations.", "weaknesses": "I see the key novelty of the proposed approach in section 3.2 where the paper introduces the subject representation learning. The paper mentions that output of the BLIP-2 multimodal encoder is passed to CLIP Text Encoder by combining the text and subject embeddings. I suggest authors elaborate this to provide more concrete details. We get embeddings as output from CLIP Text Encoder right? How is it possible to combine them before passing as input to CLIP Text Encoder. I might be missing something here, would be great if the paper clarifies this in rebuttal. Also, it is not clear if the paper additionally finetunes the CLIP Text encoder. In lines 164-164, it is mentioned that the Text encoder is also fine-tuned. But wouldn’t that bring language drift issues? Are there any specific strategies used by the paper in fine-tuning text encoder that aid in avoiding the language drift problem? Please clarify. Also, in section 3.2, how many synthetic pairs are used? I am happy to revise my final rating based on the clarifications in the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for confirming the novelty of our approach and the comprehensive results. We address reviewer's question as below.\n\n-----\n**Q1**: “We get embeddings as output from CLIP Text Encoder right? How is it possible to combine them before passing as input to CLIP Text Encoder ”\n\n**A1**: Thanks for the question. As described in Section 3.2, we use subject embeddings from BLIP encoder as soft visual prompts. Specifically, given a text prompt, \nwe first pass text tokens through the CLIP embedding layer, thus obtaining text token embeddings;\nwe then concatenate subject embeddings and the text token embeddings;\nthe combined embeddings are passed to subsequent CLIP layers, i.e. the positional embedding layer, then the self-attention layers.\nWe will revise LN131-136 to make this more clarified.\n\n-----\n**Q2**: “It is mentioned that the text encoder is also fine-tuned. But wouldn’t that bring language drift issues?”\n\n**A2**: As referred to by DreamBooth paper, language drift causes the model to “associate the class name with the specific instance”, due to fine-tuning the CLIP text encoder which overfits to the subject appearance.\n\nDifferent from DreamBooth, our model uses a subject embedding produced by a new BLIP encoder. The CLIP text encoder is tasked to align the subject embedding with the text prompt, rather than learning the subject appearance. Therefore, we reduce the risk of the text encoder overfitting to specific subject instances. Furthermore, we give captions to the text encoder during training to preserve its language capability. \n\nEmpirically, we do not observe the issue of language drift. As a qualitative evidence, in the subject editing application (Figure 5, #9-10), our model generates images well-aligned with the text either with subject embeddings (images after editing), or with text embeddings only (images before editing). This is also evidenced quantitatively by Table 1, which shows that our model produces comparable or better text alignment than prior work. \n\n-----\n**Q3**: “How many synthetic pairs are used”?\n\n**A3**: We use a subset of 292K images from OpenImage-V6, as described in Section 214, Ln-214.\tIn the meanwhile, please kindly find the detailed description of the selection criterion in the supplementary material A.5. We are happy to expand the description in the revised version for better clarity.\n\n-------\n\n**Below, please allow us to reiterate the significance of our work**.\n\n- **BLIP-Diffusion represents a novel approach to subject-driven generation using multimodal encoder**. Previous work (DreamBooth, Textual Inversion) learns subject embeddings via inversion. Our approach using multimodal encoders represents a novel and generic technique that has proved more efficient than inversion. In addition, our approach can also benefit from the advancement of multimodal vision-language foundation models, offering better potentials for stronger subject-driven generative capabilities;\n- **BLIP-Diffusion highlights a new two-staged pre-training strategy for category-generic subject-driven generation**. The multimodal representation learning stage harvests the high-quality text-aligned visual features. The subject representation learning stage includes a novel pre-training task prompted context generation, ensuring the subject visuals and text prompt can well coordinate for generation. Both stages are category-generic and require no domain-specific annotations, which make BLIP-Diffusion stand out from concurrent work.\n- **Zero-shot subject-driven generative capabilities are unprecedented**. Zero-shot generation with highly-customized and category-generic subjects is a challenging task. Such zero-shot capabilities were not available in prior models. We enable this novel capability via the newly introduced subject representation learning stage, which represents a significant advancement in this domain per se.\nBLIP-Diffusion features a foundational architecture that enables versatile applications. Different from existing work, our model’s generative capabilities are showcased in multiple applications, including generation, editing, geometry-guided generation, image manipulation/stylization (see supplementary) and subject interpolation (see supplementary). This demonstrates the flexibility of our model and its potential to serve as a foundation subject-driven generation model.\n- **BLIP-Diffusion demonstrates preferable generation results while offering significant speed-up in finetuning**. Specifically, our model fine-tunes 20x more efficiently than DreamBooth. This effectively reduces fine-tuning time per subject from 5-10 minutes (500-1000 fine-tuning steps) to sub-minute (50-100 fine-tuning steps). This has important implications on applications where fine-tuning efficiency matters, such as multimodal dialogues.\nWe provide quantitative evaluation results on public datasets with category-generic subjects, which validate effectiveness of the model. Our model will be open-sourced for researchers and practitioners for reproducing our results and findings.\n\n-----\n**Q4**: I am happy to revise my final rating based on the clarifications in the rebuttal.\n\n**A4**: We hope the response clarifies the questions. We humbly request the reviewer to re-evaluate the significance of our work based on the response above.\n"}, {"review_id": "VI5tIGsaC5", "reviewer": "Reviewer_ZD8x", "summary": "The paper introduces \"BLIP-Diffusion\", a new subject-driven image generation model that supports multimodal control using subject images and text prompts. The model introduces a pre-trained multimodal encoder to provide subject representation and enables zero-shot subject-driven generation and efficient fine-tuning for customized subjects. This model can be combined with existing techniques to enable novel subject-driven generation and editing applications. ", "questions": "1. Clarification on Zero-Shot Performance: The paper claims that the BLIP-Diffusion model can perform zero-shot rendering of images across various categories of subjects. However, the results presented do not fully substantiate this claim. Could the authors provide more evidence or examples to support this claim?\n\n2. Editability Issue: The paper suggests that using trained background replaced subject images can address the editability issue. However, it seems that this approach might only deal with recontextualization, not subject area editing. Could the authors justify this approach and explain how it addresses the editability issue?\n", "limitations": "The paper's main claim is that the proposed BLIP-Diffusion model can perform zero-shot rendering of images across various categories of subjects. However, the results presented do not fully substantiate this claim. Both qualitatively and quantitatively, the zero-shot results are not as impressive as one might expect. Furthermore, the model's performance seems to heavily rely on fine-tuning. While fine-tuning is a common practice in machine learning, the extent to which the model depends on it raises questions about its practicality and efficiency. The necessity of fine-tuning to achieve good results could be seen as a limitation, especially in scenarios where rapid or on-the-fly generation is required. This dependence on fine-tuning could limit the model's applicability and ease of use in certain contexts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Originality: The paper introduces a novel subject-driven image generation model, BLIP-Diffusion, which supports multimodal control using subject images and text prompts. This model is original in its approach as it combines a pre-trained multimodal encoder for subject representation, enabling zero-shot subject-driven generation and efficient fine-tuning for customized subjects.\n\n2. Quality: The quality of the paper is evident in the detailed explanation of the model and the comprehensive experiments conducted to validate its performance. The paper includes qualitative results that demonstrate the model's capabilities, such as zero-shot subject-driven generation and high-fidelity fine-tuning. The model also shows high subject fidelity and prompt relevance, requiring significantly fewer fine-tuning steps compared to other methods.\n\n3. Clarity: The paper is well-structured and clear in its presentation. The authors provide a thorough explanation of the model, its implementation, and the experiments conducted. The use of figures and tables further enhances the clarity of the paper, providing visual representations of the model's performance and capabilities.\n\n4. Significance: The significance of the paper lies in its contribution to the field of image generation. The BLIP-Diffusion model presents a new approach to subject-driven image generation, offering potential for novel subject-driven generation and editing applications. The model's ability to perform zero-shot subject-driven generation and efficient fine-tuning for customized subjects is a significant advancement in this field.", "weaknesses": "Limited Zero-Shot Performance and Dependence on Fine-Tuning: The paper claims that the proposed BLIP-Diffusion model can perform zero-shot rendering of images across various categories of subjects. However, the results presented do not fully substantiate this claim. Both qualitatively and quantitatively, the zero-shot results are not as impressive as one might expect. Furthermore, the model's performance seems to heavily rely on fine-tuning. While fine-tuning is a common practice in machine learning, the extent to which the model depends on it raises questions about its practicality and efficiency. The necessity of fine-tuning to achieve good results could be seen as a limitation, especially in scenarios where rapid or on-the-fly generation is required. This dependence on fine-tuning could limit the model's applicability and ease of use in certain contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for confirming the originality, quality, clarity and significance of our work. We provide response to reviewer's question below.\n\n-----\n**Q1**: Zero-shot performance and its applicability.\n\n**A1**: As described in Section 3, BLIP-Diffusion is the very first model to unlock the zero-shot subject-driven generation capabilities for generic categories. The significance in its zero-shot capability is summarized below:\n\n1. **Zero-shot subject-driven generative capabilities are unprecedented**. Zero-shot generation with highly-customized and category-generic subjects is a challenging task. Such zero-shot capabilities were not available in prior models. We enable this novel capability via the newly introduced subject representation learning stage, which represents a significant advancement in this domain per se.\n\n2. **Zero-shot subject-driven generation quality outperforms some recent fine-tuning methods**. Quantitatively, Table 1 shows that the zero-shot result outperforms significantly the recent fine-tuning based model Textual Inversion (TI). In particular, BLIP-Diffusion achieves zero-shot DINO score 0.594 (versus 0.569 for TI), and CLIP-T score 0.300 (versus 0.255 for TI). This demonstrates the competitive performance of the zero-shot generation with the prior art. \n\n3. **Whether fine-tuning is needed depends on the application**. Apart from subject-driven generation, BLIP-Diffusion also facilitates various additional zero-shot applications. We kindly refer the reviewer to the supplementary material Figure 3, Figure 4 for more examples, where we show results using BLIP-Diffusion for stylization / image manipulation. Specifically, our model can use the appearance style of an input subject to guide the generation. In this application, despite no fine-tuning being performed, our model produces visually appealing and creative generations. In light of this, we consider that fine-tuning is not an absolute dependency, and its necessity can be determined based on the application.\n\nWhile being able to generate and edit in the zero-shot fashion is an appealing capability, it is important to underscore the novel concept of pre-trained subject representation and the two-staged pre-training strategy.  In this regard, we consider the zero-shot capabilities of BLIP-Diffusion as compelling evidence of the effectiveness of our pre-training approach.\n\n-----\n**Q2**: Fine-tuning limits the ease of use in certain contexts.\n\n**A2**: This is precisely the motivation of introducing pre-trained subject representation into the text-to-image generation model. With such pre-trained representation, BLIP-Diffusion significantly improves the subject-driven fine-tuning efficiency by up to 20 times, compared to the state-of-the-art model DreamBooth. This effectively reduces fine-tuning time per subject from 5-10 minutes (500-1000 fine-tuning steps) to sub-minute (50-100 fine-tuning steps). Whilst potential further improvement is possible, we consider BLIP-Diffusion clearly superior in terms of fine-tuning efficiency than leading solutions.\n\n-----\n**Q3**: Justification of the prompted context generation pre-training task with background replacement.\n\n**A3**: We appreciate the reviewer's question on how background replacement helps subject representation learning. The referred “edibility” requires (1) transferring of subject visuals; (2) guided generation according to text prompts. Our background replacement fulfills both requirements. Even though the pre-training only recontextualizes the subject, the model can generalize to other text prompts and directly edit the subject.  More technically, the prompted context generation task learns a joint subject-text space, which allows any text prompt to interact with the subject visuals.\n\n-----\nWe hope the response addresses reviewer's questions. We will revise the manuscript accordingly for better clarity."}, {"review_id": "4FLXltF90W", "reviewer": "Reviewer_tkfj", "summary": "This paper aims to solve the subject-driven text-to-image generation with a pre-trained subject representation that is derived from a vision-language encoder, the BLIP2 model. The obtained subject representation captures rich information of the visual input while being aligned with the textual space. The text-conditioned diffusion model is pre-trained to produce images based on this subject representation and the text input. The derived diffusion model can produce novel impressive renditions of a given subject with respect to different text prompts, in both zero-shot and few-step finetuned scenarios. Beyond that, the paper presents fancy applications like controlled generations and subject-driven stylization. Compared to the state-of-the-art approaches, the proposed BLIP-Diffusion demonstrates preferable subject-driven generation results both qualitatively and quantitatively while offering significant speed-up.", "questions": "It is suggested to provide more technical details for the style transfer part. Does it need some special prompt design when specifying styles according to the input image?", "limitations": "The limitation is sufficiently discussed in the supplementary material.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper explicitly leverages the text-aligned image feature and the diffusion model trained on this subject representation is able to generate subject renditions with compelling quality. The novel combination with the BLIP2 model forms an effective solution to subject-driven text-to-image generation. The method presented is neat and elegant.\n\nMoreover, I like many technical details authors propose to achieve the final quality.  For example, authors carefully design a synthetic procedure to form the synthetic training pairs to tackle the tendency to produce trivial solutions. Also, it is found that randomly dropping the subject prompt with some probability is beneficial to text-to-image generation.\n\nThe paper demonstrates fancy applications in various applications and presents impressive generation results. the significant speed-up for the subject-driven generation makes it promising in practical usage. The quantitative study further proves the superiority of the method over prior leading approaches. \n\n\n\n\n\n \n", "weaknesses": "- One possible way to explain the method is that this paper uses a detailed text prompt instead of a rough description like \"an image of [V]\" as used in Dreambooth. That is, the features captured from the BLIP2 may not be the key. One simpler baseline is \"Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models\". The paper should mention such baseline and add more discussions regarding this.\n\n- To avoid the distraction of the background, the authors propose to replace the background of the target image with random background. Then why not just remove the background during training?\n\n- One primary usage of subject-driven text-to-image models is to generate images related to humans. However, the paper avoids this scenario by explicitly removing human-related images during training, why is that? It is suggested to have a discussion regarding this, otherwise, it is suspicious that the method does not perform well on portraits. \n\n- While the paper qualitatively measures the fidelity to the input image and the text prompt respectively, there is no measure for image quality. While in the chosen samples of Figure 6 the BLIP-Diffusion shows apparent quality advantage over prior arts, it still needs a quantitative measure to reflect the image quality on a large amount of image results.T", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for confirming the technical depth and empirical values of our work. In the following, we provide response to answer reviewer's questions.\n\n------\n**Q1**:  paper uses a detailed text prompt instead of a rough description like \"an image of [V]\" as used in Dreambooth. the features captured from the BLIP2 may not be the key; discussion on E4T.\n\n**A1**: We use BLIP-2 captions for pre-training. While BLIP-2 captions are more detailed than template prompts “an image of [V]”, they are not sufficient to describe the exact subject visuals. Therefore, having subject embeddings is crucial in our model. \n\nTo provide evidence for this claim, we kindly refer the reviewer to Table 2, which shows that the generation performance improves with better subject embeddings. This highlights the importance of subject embedding on the generation quality. Qualitatively, in Figure 7, we show that subject embeddings learn meaningful global and local subject visuals. This further supports the effectiveness of subject embeddings.\n\nWe are aware of the E4T work and have also referenced it in LN 74. It is worth highlighting that BLIP-Diffusion is category-generic, and can be applied to any customized subjects. This is supported by our quantitative evaluation on the DreamBooth datasets. In contrast, E4T pre-trains the model on domain-specific datasets, e.g. cats, thus cannot generalize easily to out-of-domain subjects. \n\nWe are happy to include an expanded discussion on E4T in the revised version.\n\n-----\n**Q2**: Why not just remove the background during training?\n\n**A2**: We thank the reviewer for this insightful question.\nThis will impose additional constraints on the input subject image to also have an empty background during fine-tuning and inference. As a result, it would most likely require additional procedures for background removal and introduce external reliance, which we consider suboptimal. In contrast, in our setup, the model is able to automatically identify the subject from the input image after pre-training, even with distracting background scenes.\n\n-------\n**Q3**: Application on portraits.\n\n**A3**: We thank the reviewer for raising this issue. While we acknowledge the practical values of portrait generation, we purposely avoid human-related generation for training and evaluation. This is mainly to abide by the code of ethics of the conference (https://neurips.cc/public/EthicsGuidelines), and was also motivated by related NeurIPS work [1]. With our due respect, we humbly request the reviewer to kindly share our concern.\n\nAs such, we would expect the model checkpoint along this submission to not perform well for human portraits. However, we note that the proposed two-staged pre-training strategy is category-generic. Interested readers may opt to resume pre-training on domain-specific datasets to tailor for their own application scenarios.\n\n[1] PASS: An ImageNet replacement for self-supervised pretraining without humans, Yuki M. Asano, Christian Rupprecht, Andrew Zisserman, Andrea Vedaldi\n\n------\n**Q4**: Quantitative image quality measure.\n\n**A4**: We thank the reviewer for the suggestion. We recognize that measuring image quality quantitatively for subject-driven text-to-image generation is an open research question. And to the best of us, few prior work presents relevant results.\nAs such, we propose to use the aesthetic scorer (https://github.com/LAION-AI/aesthetic-predictor) to measure the visual quality of the generated images, which was used to select high-quality image data used for training state-of-the-art diffusion models.\nThe aesthetic scorer predicts an aesthetic score bounded from 0 to 10 using CLIP ViT-L/14 features, where 10 is the highest aesthetic score. To get a better understanding of the score, we reference the description from LAION project page (https://laion.ai/blog/laion-aesthetics/) that out of the LAION 5-billion images: \n\n- 1.2B (24%) images have scores 4.5+;\n- 12M (0.24%) images have scores 6+; \n- 3M (0.06%) images have scores 6.25+; \n- 625K (0.01%) images have scores 6.5+;\n\nWe calculate the aesthetic scores for 3000 images of our model and compare with those generated from DreamBooth. We use prompts and subjects from the DreamBooth datasets. The results are shown as below. \n\n| Setups      | Aesthetic Scores |\n| ----------- | ----------- |\n| BLIP-Diffusion (fine-tuned)    | 6.50       |\n| BLIP-Diffusion (zero-shot)   | 6.43        |\n| DreamBooth   | 6.20       |\n\n**These results show that**:\n- BLIP-Diffusion produces better image quality than DreamBooth. We attribute this to the two-staged pre-training strategy which helps to better align subject embeddings and text embeddings;\n- BLIP-Diffusion after fine-tuning produces top 0.01% quality images when compared with the 5B internet image corpora; while DreamBooth quality is among top 0.24%. This shows a clear quality advantage of our model.\n- Fine-tuning helps produce higher quality images for our model;\n\nWe will include this result into the supplementary material during revision.\n\n------\n\nWe thank the reviewer for appreciating our work and hope the response addresses the questions."}, {"review_id": "QF1kxzMTor", "reviewer": "Reviewer_wCQb", "summary": "This paper proposes  BLIP-Diffusion, a new subject-driven image generation model with multimodal encoder that supports multimodal control which consumes inputs of subject images and text prompts.", "questions": "I am curious about the contribution of the proposed pretraining approach. How's the performance for the naive pretraining strategy (keep the same image as both input to the multimodal encoder and output to the diffusion mode.)?\n\nHow do you get the random background image for generating synthetic image?\n\nFor finetuning compared with Dreambooth, does it take the same time for each timestep?\n\nWhat's the inference time cost of the proposed method compared with standard stable diffusion? ", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The proposed method is novel and enables subject-driven generation under efficient fine-tuning and zero-shot setups.\n\nThe proposed model can serve as a foundation model and combined with previous methods such as prompt-to-prompt and controlnet.\n\nSolid experimental results and ablations.", "weaknesses": "The contribution is a bit lacking. It seems to me that the work is just a combination and the multimodal encoder from CLIP and the t2i model.\n\nThe improvement under efficient fine-tuning setup seems to be incremental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for confirming the novelty of our approach. We address questions below.\n\n-------\n**Q1**: contribution of the work?\n\n**A1**:  We summarize the scientific significance of BLIP-Diffusion as below:\n- **BLIP-Diffusion represents a novel approach to subject-driven generation using multimodal encoder**. Previous work (DreamBooth, Textual Inversion) learns subject embeddings via inversion. Our approach using multimodal encoders represents a novel and generic technique that has proved more efficient than inversion. In addition, our approach can also benefit from the advancement of multimodal vision-language foundation models, offering better potentials for stronger subject-driven generative capabilities;\n- **BLIP-Diffusion highlights a new two-staged pre-training strategy for category-generic subject-driven generation**. The multimodal representation learning stage harvests the high-quality text-aligned visual features. The subject representation learning stage includes a novel pre-training task prompted context generation, ensuring the subject visuals and text prompt can well coordinate for generation. Both stages are category-generic and require no domain-specific annotations, which make BLIP-Diffusion stand out from concurrent work.\n- **Zero-shot subject-driven generative capabilities are unprecedented**. Zero-shot generation with highly-customized and category-generic subjects is a challenging task. Such zero-shot capabilities were not available in prior models. We enable this novel capability via the newly introduced subject representation learning stage, which represents a significant advancement in this domain per se.\nBLIP-Diffusion features a foundational architecture that enables versatile applications. Different from existing work, our model’s generative capabilities are showcased in multiple applications, including generation, editing, geometry-guided generation, image manipulation/stylization (see supplementary) and subject interpolation (see supplementary). This demonstrates the flexibility of our model and its potential to serve as a foundation subject-driven generation model.\n- **BLIP-Diffusion demonstrates preferable generation results while offering significant speed-up in finetuning**. Specifically, our model fine-tunes 20x more efficiently than DreamBooth. This effectively reduces fine-tuning time per subject from 5-10 minutes (500-1000 fine-tuning steps) to sub-minute (50-100 fine-tuning steps). This has important implications on applications where fine-tuning efficiency matters, such as multimodal dialogues.\n- We provide quantitative evaluation results on public datasets with category-generic subjects, which validate effectiveness of the model. Our model will be open-sourced for researchers and practitioners for reproducing our results and findings.\n\n--------\n**Q2**: Performance if using the same image for multimodal encoder and target.\n\n**A2**: As described in Ln 142, this setup leads to trivial solutions where the image is directly copied. In our experiments, the resultant model always reproduces the input image, failing to address text prompts. \n\nThis observation also echoes with the findings as reported by DreamBooth, where the authors propose additional regularization methods to counteract the issue. Their proposed regularization technique, however, does not easily scale up for scalable pre-training. \nIn this regard, with input background replaced, we effectively avoid such optimization shortcuts and enforce the model to learn to condition on both subject visuals and text prompt for the generation.\n\n----\n\n**Q3**: How do you get the random background images?\n\n**A3**: We download 59K images from the royalty-free photo stock website by querying for background and landscape images, as these images usually contain less salient or distracting subjects.\n\n----\n**Q4**: For finetuning compared with DreamBooth, does it take the same time for each timestep?\n\n**A4**: **Yes, our model takes the same time for each iteration as DreamBooth**. \nAs described in section 3.3, we only fine-tune U-Net to specialize for custom subjects. Subject embeddings are pre-computed at the cost of merely one forward pass with negligible time needed, thus no multimodal encoder is needed during fine-tuning.\nTo get a better idea of the wall-clock time cost, on DreamBooth Benchmark, DreamBooth models on average take 6-10 minutes and fine-tune for 600-1000 steps, while our model costs less than a minute for around 80 steps on average.\n\n------\n**Q5**: What's the inference time cost of the proposed method compared with standard stable diffusion?\n\n**A5**: **Inference of BLIP-Diffusion is as efficient as standard stable diffusion models**.\nFor zero-shot inference, only one additional pass of multimodal encoder is needed to initialize the subject embedding. This cost is negligible compared to the iterative denoising step.\nFor inference with fine-tuned model checkpoints, subject embeddings are loaded as part of the model. No additional inference cost is needed.\n\n------\n**Q6**: Improvement under efficient fine-tuning seems incremental.\n\n**A6**: Fine-tuning aims to learn better subject visuals for highly customized subjects. This consequently leads to better subject alignment scores, i.e. DINO, CLIP-I as indicated in Table 1. Such results demonstrate the effectiveness of fine-tuning. \nIn contrast, our fine-tuning setup does not explicitly optimize for prompt understanding, thus showing no significant effect on CLIP-T score, which is naturally expected. While our work does not emphasize on the design of fine-tuning procedure, it may be also possible to integrate better calibrated fine-tuning and inference methods (e.g. [1]). \n\nWe will revise the manuscript to highlight this discussion.\n\n[1] Key-Locked Rank One Editing for Text-to-Image Personalization, Tewel, Yoad and Gal, Rinon and Chechik, Gal and Atzmon, Yuval, SIGGRAPH 2023.\n\n-------\nHope the response addresses the questions."}]

Author rebuttal:

Meta-review: All the reviewers recommend the acceptance of the work. Reviewers appreciated the proposed multi-modal encoder strategy for subject-driven generation. Reviewers raised several clarification questions and also raised concerns regarding the quality of the zero-shot generation results. Authors addressed several of these questions in their responses. The reviewers did raise some valuable concerns and suggestions that should be addressed in the final camera-ready version of the paper, which include adding the relevant rebuttal discussions and revisions in the main paper. The authors are encouraged to make the necessary changes to the best of their ability.
Decision: Accept (poster)