The CLIP Model is Secretly an Image-to-Prompt Converter
Authors: Yuxuan Ding
Chunna Tian
Haoxuan Ding
Lingqiao Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Model
CLIP model
Image Variation
Customized Generation

Abstract: The Stable Diffusion model is a prominent text-to-image generation model that relies on a text prompt as its input, which is encoded using the Contrastive Language-Image Pre-Training (CLIP). However, text prompts have limitations when it comes to incorporating implicit information from reference images. Existing methods have attempted to address this limitation by employing expensive training procedures involving millions of training samples for image-to-image generation. In contrast, this paper demonstrates that the CLIP model, as utilized in Stable Diffusion, inherently possesses the ability to instantaneously convert images into text prompts. Such an image-to-prompt conversion can be achieved by utilizing a linear projection matrix that is calculated in a closed form. Moreover, the paper showcases that this capability can be further enhanced by either utilizing a small amount of similar-domain training data (approximately 100 images) or incorporating several online training steps (around 30 iterations) on the reference images. By leveraging these approaches, the proposed method offers a simple and flexible solution to bridge the gap between images and text prompts. This methodology can be applied to various tasks such as image variation and image editing, facilitating more effective and seamless interaction between images and textual prompts.

The CLIP Model is Secretly an Image-to-Prompp Converter
Yuxuan Ding
Chunna Tian'
School of Electronic Engineering Xidian University Xi'an 710071, China yxding@stu. xidian.di....i edu. en
School of Electronic Engineering Xidian University
Xi'an an 71 710071, China chnattianxxidian edu. cn
Haoxuan Ding Unmanned System Research Institute Northwestern Polytechnical University Xi' an 710072, China
Lingqiao Liu
Australian Institute for Machine Learning
The University of Adelaide
Adelaide 5005, Australia lingqiao www.aaeelulliieee edu. au
haoxuan, ding@mail .wwu edu. cn
Abstract
The Stable Diffusion model is prominent text-to-image generation model that re- lies on a text prompt as input, which encoded using the Contrastive Language- Image Pre- Pre-Training (CLIP). However, text prompts have limitations when it comes to incorporating implicit information from reference images. Existing methods have attempted to address this limitation by employing expensive training pro- cedures involving millions of training samples for image-to-image generation. In contrast, this paper demonstrates that the CLIP model, utilized in Stable Diffusion, inherently possesses the ability to instantaneously convert images into text prompts. Such image-to- torpmppptttppt conversion can be achieved by utilizing linear projection matrix that is calculated in a closed form. Moreover, the paper showcases that this capability amount of similar-domain training data (approximately 100 images) or incorpo- rating several online training steps (around 30 iterations) the reference images. By leveraging these approaches, the proposed method offers a simple and flexible solution to bridge the gap between images and prompts. This methodology can be applied to various tasks such as image variation and image editing, facilitating more effective and seamless interaction between images and textual prompts.
be further enhanced by either utilizing small
Introduction
In recent years, there has been surge of interest in vision-and- -add---an-- research, particularly in the field of text-to-image generation. Prominent models this domain include autoregression models like DALL-E [ and Make-A-Scene 2 well as diffusion models like DALL-E 2 [3] and Stable Diffusion 4 These models have revolutionized the quality of generated images. They leverage text prompts to synthesize images depicting various objects and scenes that align with the given text. Among these models, Stable Diffusion stands out as a significant open-source model. serves foundation for many recent works, including image generation 5 6 8 image editing
10 11 12 13 [4], and more.
"This work done while Yuxuan Ding visiting The University of Adelaide visiting researcher. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

SpaP
Figure 2: Attention map of Stable Diffusion 4 The
Figure I: Demonstration of image vari- ation. The image the left is real reference image, while the four on the right are generated from our method.
bottom row sets attention weights of caption words zero, only keeping the start-/end-token so the caption maps of the bottom are black. Also notice the start- token has strong weights so the map is all white.
However, text prompts have limitations when comes to incorporating unspeakable information from reference images. It becomes challenging generate perfect and detailed prompt when users want to synthesize images related to a picture they have seen. Image variation techniques aim to address this limitation by enabling users to generate multiple variations of an input image, without relying on complex prompts. As illustrated Fig. [ the generated variations closely resemble the reference image, often sharing the same scene objects but with distinct details.
Stable Diffusson Reimagine (SD-R) 4 is a recently proposed image variation algorithm. It achieves this goal by retraining Stable Diffusion 4 where the text encoder is replaced with image encoder to adapt the model for image input. The model is trained using millions of images and over 200,000 GPU-hours, enabling it to effectively generate image variations based reference images.
In this paper. we make significant discovery that allows a more cost-effective conversion approach. We find the CLIP model [5], as utilized in Stable Diffusion, can be repurposed as an effective
converter. This converter can be directly employed or served as a valuable initialization for a data-efficient fine-tuning process. As a result, the expenses associated with constructing or customizing an image-to to--moppp converter can be substantially reduced.
More specifically, our method is built upon a surprising discovery: the control of image generation through text is primarily influenced by the embedding of the end-of-sentence (EOS) token. We found that masking all word tokens, except for the start and end tokens, does not adversely affect the quality of image generation, as illustrated in Figure 2 Simultaneously, during CLIP training, the projection of the end-token embedding is trained to align with the visual embedding. This inherent relationship enables us to derive a closed-fort projection matrix that converts visual embedding into an embedding that is capable of controlling the generation of Stable Diffusion 4. method Stable Diffusion Image-to- to-Prompt Conversion (SD-IPC).
We call this
In addition, we introduce two methods to enhance the quality and flexibility of maaage-to-romme conversion. The first approach involves parameter-efficiet tuning using small amount of data, consisting of only 100 images and requiring just GPU-hour. This method encourages the model to better preserve image information and enables practitioners control the specific content they want to retain when generating new images. The second approach involves customizing the model on reference images using few iterations, ensuring that the generated images are closer specific concepts. While this approach has been explored in previous research, we demonstrate that with the advantageous initialization provided by SD-IPC, the online fine-tuning requires significantly fewer iterations to achieve desirable results.
2 Background and Related Works
2.1 Diffusion Model
Firstly. we present a brief overview of the Stable Diffusion 4 which serves as our underlying model. Diffusion models (DMs) [16
17 18 19 belong to a class of latent variable models. DMs, there exist two Markov chains known as the diffusion process and the reverse process, both having fixed

---

length T. The diffusion process progressively introduces Gaussian noise to the original data (xo) until the signal becomes corrupted (xr). During DMs training, the reverse process is learned, which operates in the opposite direction of the diffusion process. The reverse process can be viewed as denoising procedure, moving from x; to X 1 at each step. After multiple denoising steps, the model obtains instances that closely resemble the real data.
Stable Diffusion 4 is built on the Latent Diffusion Model (LDM) 4 LDM 4 proposed to do diffusion process in a latent space rather than the usual pixel space, significantly reducing the training and inference cost of the diffusion model. The authors proposed utilize VAE compression to get the latent code Zo. which is Xy above. Diffusion process will build on the latents. A U-Net architecture [17] with timestep and text conditions would do the reverse. The text prompt injected into the model with cross-attention layers. We denote Eg (Zz, Ctzt (Ptzt), t) as the output of the U-Net, which the predicted denoising result. Punt is the textual prompt and Cezt (Ptat) the prompt embedding from the text encoder. /. is the timestep. The training objective of DMs is as followed:
E.pee ||e Eo (Ze, Ctat Ctmr (Peze), .))l
where N (0, I) is the noise used to corrupt clean latent variables. During the generation, the latent 2e which starts at random Gaussian noise Zr, will recursively go through a denoising operation until Zo is sampled. Finally, Zo is reconstructed an image by the VAE.
2.2 CLIP Model
The CLIP model [1]] has garnered significant acclaim as a groundbreaking zero-shot model in recent years. Its training process demands optimizing a contrastive loss function using extensive 400-milionn pairs of images and corresponding text descriptions. Through the meticulous training, the model has been able to achieve unparalleled capabilities in zero-shot classification and image-text retrieval.
The model comprises an image encoder CLIP; (+), a text encoder CLIP; (-), visual projection layer W;, and textual projection layer W,. The image encoder encodes input image into a visual embedding fing derived from a special class-token. By applying the visual projection layer, the embedding is projected into the CLIP visual embedding fimg- Similarly, the text encoder processes the input text, yielding sequence of output embeddings fexr for each text token and start token and
end-of-sentent (EOS) )token. The embedding of the EOS token fet of the sentence, is projected into the CLIP textual embedding fine through W.. Formally,
p'o(eos) where t denotes the length
fing = CLIP, (I), fing = W; famg.
(2)
fra CLIP, (s), f. = W, E.2t
(eoe))))
(3)
The training objective of CLIP maximize the cosine similarity between ff and fiig for matched sentence-image pair while minimizing this similarity for unmatched pairs For the simplicity of discussion, we denote the space spanned by fezt as T-space and the space spanned by f as C-space
The CLIP text encoder [15] is directly used in Stable Diffusion to encode text prompts. It encodes a text prompt as sequence of embeddings:
(s(ss)) Eaxt
(cos) Eaat
(eos) part
frue :
frat
(eos) denote the embeddings corresponding to the start-token, the i-th word token and end-token, respectively. From tat
where (D.()))) Eie and fir tu
pl+1.(eos) to feat
(eo,s) are padded tokens.
2.3 Image Variation & Customized Generation
Image Variation. Image variation aims to generate images similar to the reference image but not identical. SD-R [4] is proposed to address this problem, which builds upon the Stable-unCLiP modef The authors fine-tuned the Stable Diffusion model [4] to align with the CLIP visual embedding. In SD-R [4, images can be directly input into the diffusion model through CLIP image encoder. Since the original Stable Diffusion is conditioned on text only, an expensive fine-tuning is required to
http:://whggghttteooo/////llll/p 2-1-nnciip
3

---

accommodate this new input. The process took 200,000 GPU-hours on NVIDIA A100-40GB GPU while our approach only requires GPU-hour on NVIDIA A5000-24GB G
Customized Generation. Recent works such as DreamBooth
Textual Inversion
14 and
Custom Diffusion [3] focus on learning special text prompt to feature specific objects persons from the reference images. For instance, given several photos of a particular cat, these methods use a special-token "(s) cat" to represent the concept and incorporate it with the text prompt. DreamBooth
[ and Custom Diffusion [33 also perform simultancous fine-tuning of diffusion model parameters. However, the fine-tune tue-tunnng process is still somewhat conssmnnn,,,n with Custom Diffusion [T3] requiring nearly 6 minutes on 2 NVIDIA A100 GPUs. In contrast, our fast update SD-IPC only needs
1 minute on 2 A5000 GPUs.
Image Editing. Stable Diffusion 4 is commonly used for image editing tasks. Prompt-to- Prompp 9 and Plug-and-Play [10 utilize attention map bridge to enable concept and style manipulation.
Null-Text Inversion (20 and Pix2Pix-Zero [21] relies on inversion- -hased methods. InstructPix2Pix
22 creates a dataset of paired edited images and fine-tunes Stable Diffusion [4] as an editing model. It's important to highlight that while our primary focus in developing this method was to enhance image variation, can also be employed to generate images based on prompts that combine both textual instructions and accompanying images. Notably, unlike existing approaches that frequently reproduce the layout of the original image in the generated output, our method operates without being confined to replicating the exact layout of the source image.
3 Methodology
3.1 Image-to-Prompt Conversion via Projecting CLIP embedding
By design, the image generation process in the stable diffusion model should be influenced by embeddings of all tokens in a prompt, like Eq. (4). Interestingly, we have discovered that masking word tokens, by setting their attention weights to 0 except for the start-/end-tokeet does not have a negative impact on the quality of generated images. This finding is visually illustrated in Figure 2
On another note, the training objective of CLIP [15] is to match the embeddings fimg and firet
with fige being essentially projection of fizt aforementioned observation, leads us to establish a connection between fing and (e.(e)s) effectively converting the visual embedding a prompt embedding.
pf.(s This inherent relationship. coupled with the
Formally, we assume that after training, CLIP model can induce high cosine similarity between the fing and frzt and we can further make the following upproximation:
iing
ff
with f
W,
(5)
||
kmg
By using Peorr-Penrose -seudo-inverse
23 on 6
(eo(e we obtain an estimate of fiut from fing'
(llss))
fizell
W fing
:= cenort
where, W,+ = (W, w.) WTT.
(6)
tart
Iree
where we empirically observe |f can be well approximated by a constant, e.g., | = 27 and W. can be obtained from the pretrained CLIP model [15). We denote the converted embedding as penvrt and use it to assemble a pseudo-prompt sequence with the following format:
Fentt
1l (006) el.envrt
c6.cn f f.tt
where pl.caurt =... em.enttt Penert In other words, we replace all word-tokens, pad-tokens txst
and end-token in Eq. with the converted fenert, based on the fact that penurt is approximation
of (eo(s))) and masking word-tokens does not influence the generation
*FP16 computing performance of A100 is 77.97 TFLOPS vurse 27.77 TFLOPS of A5000.
'We use singular value decomposition (SVD) to form the pseudo-inverse, singular values which are smaller than 0.3 will be treated as 0.
'Here we keep the pad-tokens, so the index of token is from 0 to 76, where the maximum length of CLIP text input is 77, even they are the same penert they would contribute to the attention weights in cross-atis cross-attention to decrease the weights of start-token.

---

Input Images
SD w/ Text
SDAIPC (Ours)
Figure 3: Image variation results MSCOCO 24 SD w/ Text 4 is generation from the ground- truth text prompts that are not available for variation methods such as SD-R and SD-IPC. SD-IPC is method, notice that SD-IPC does not need any training compared SD-R
This approximation allows immediate conversion of an image to text prompt by directly mapping to (approximately) equivalent prompt. We refer to this method as Stable Diffussin Image-to- Prompt Conversion (SD-IPC), Experimental results in Fig. F demonstrate that SD-IPC effectively captures the semantic information present in the reference image and enables image variation.
Furthermore, we have identified simple yet effective approach to combine both the text prompt and the converted image prompt within our framework. To achieve this, we perform weighted average of the two embeddings. Formally, the process can be described follows:
poomb
penert Eart
+ (oo()
pedst
(.(00)))) Eart
pl,wo
.... Ext ....
pl.comb
comcomb
(8)
where pt.combb Foomb is the combined-token embedding and a hyperparameter control the expression of editing text. Notice that the editing word-token fiw are also in the embedding sequence.
Conditioning on pedit could generate images that match both the visual and textual conditions. We report some editing results in Appendix D.2
3.2 Fine-tuning with TenagettoPrromee Conversion
While the aforementioned SD-IPC method demonstrates reasonable performance, it still faces challenges when comes to real-worle applications due to two main reasons. Firstly, the conversion process in SD-IPC relies approximations, which may not always yield optimal results. Secondly, determining the exact topic or theme of an image introduces ambiguity. As the saying goes, "an image is worth thousand words", but precisely which words? The same reference image can be interpreted diffcrently based on its objects, scenes, styles, or the identities of people depicted within. Therefore, it becomes crucial to have method that allows control of the content we wish to preserve and convert into the prompt. To address these concerns and cater to the needs, we propose partial fine-tuning approach for the CLIP converter derived from Sec.
In proposed approach, we focus on fine-tuning two specific types of parameters. Firstly, we address the optimization of the projection matrix within the cross-atte coos-attention layer of the U-Net in Stable Diffusion [.. This aspect aligns with methodology employed Custom Diffusion [13]. Furthermore, we incorporate deep prompt tuning 25 into the transformer of the CLIP image encoder. Deep prompt tuning (55 introduces learnable tokens within all layers of the transformer while keeping the weights
of other components fixed. More details can be found in Appendix A
5

---

Inpur Images
SD-R
SD-IPC-FT SD-IPC (Ours)
(Ours)
ImageNet Finetuned
CelebA-HQ Finetuned
Places365 Finetuned
Figure 4: Fine-tuned SD-IPC, denoted as SD-IPC-FT, can enhance the image- tele-to-ppre co-prompt conversion quality.
The parameters can be learned by using the following loss:
Eegi [|le Eg (Ze: Cong(a 0)[{] + E2, E., [lle- Eo (Ze, (tac)) t)]]?
(9)
here the first term is Lemort, which is fine-tuning with the image-to-prompl input, and the second term is Lieat, which is the original text-to-image training loss, utilize this regularization to keep the text-to-image generation. In the proposed approach, Csmg (-) refers to the image-to-promp conversion derived from SD-IPC. It encompasses the CLIP image transformer augmented with the newly-introduced learnable prompts in deep prompting and the fixed inverse matrix derived from Eq. (6. During tuning, the inverse projection matrix remains unchanged. z, represents the latent representation of the target image Turget at time step t. The objective function aims to encourage the image-to-pae co-prompt conversion to extract information from Fret that facilitates the recovery of Fusget- There are two possible choices for Zurget (1) Furget can be selected to be the same Teet. (2) Puaget and Jher can different images, but with a shared visual concept that we intend to extract as the prompt. This usually poses stronger supervision to encourage the converter extract information related to the shared theme. The schematic representation of this scheme is illustrated in Appendix C.2
We use images randomly sampled from ImageNet [26). CelebA-HQ [27]. and Places365 28 dataset
encourage the model extract object, identity, and scene information, respectively. Experiments show that merely 100 images and GPU-hour of training are sufficient for achieving satisfied results thanks the good initialization provided by SD-IPC. We call this approach SD-IPC-FT, the results are shown in Fig. Some editing examples are listed in Fig. Fig. and Appendix D.4
3.3 Fast Update for Customized Generation
Existing methods, such as DreamBooth [1] and Custom Diffusion [13], suggest that partially fine- tuning the model on given concept images before generation can be an effective way synthesized images with customized visual concepts, e.g., people with the same identity. Our approach can also benefit from this scheme by performing such online update with SD-IPC. This can be achieved by simply replacing the training images in SD-IPC-FT with reference images and use Lconvrt only. We call this method SD-IPC-CT (CT stands for customized concept tuning). Interestingly, we find that our method can generate customized images with much fewer updates. As comparison, SD-IPC-CT only takes 30-iteration updates with around minute on A5000 GPUs while the Custom Diffusion B3 needs 250 iterations (6 minutes on 2 A100 GPUs). We report customized generation in Fig. 7

---

SD-IPC-FT
SD-IPC-FT (Ours)
SD-IPC-FT (Ours)
Input Images
SD-R
SD-R
SD-R
(Ours)
Without Editing
"On the beach"
"Under the starry sky"
Figure 5: Image editing result with SD-IPC-FT trained with 100 images sampled from ImageNet 26
SD-IPC-FT shows better editing performance than that of SD-R 4
SD-IPC-FT (Ours)
SD-IPC-FT
SD-IPC-FT (Ours)
Input Images
SD-R
SD-R
SD-R
(Ours)
DRMN
Without Editing
'Wearing glasses'
"Weuring hut"
Figure 6: Image editing result with SD-IPC-FT trained with 100 images sampled from CelebA-HH 27 SD-IPC-FT shows better editing performance than that of SD-R [4].
Experiments
4.1 Training Details
Datasets & Evaluations. In previous discussion, we propose three different fine-tuning schemes, using ImageNet [26] for object understanding. CelebA-HQ [27) for portrait understanding, and Places365 28 for scene understanding. The specific training classes or identities have selected for each dataset can be found in Appendix B Each dataset includes 100 images, the test images non-overlap with the training classes. In order to enable customized generation, we choose two objects and two identities as examples, each accompanied by five images. To assess the quality and semantic consistency of our generated outputs, measure FID-Score (29 and CLIP-Score [30..
Architecture & Hyperparameter. We utilize Stable Diffusion v1.4 and CLIP ViT-L/14 models in this paper. We compare our method with the larger Mablle-nCCIIP-smaall mode 10 using CLIP ViT-H/14 and a 1.044/4enen d.02ssnil attention feature. Our method uses DDIM for sampling, while
http://whggggggoooocoooooosollst 1--uccclpppsaa small

---

Training Images
SD-IPC-FT (Ours)
Custom Diffusion
SD-IPC-FT (Ours)
Custom Diffusion
SD-IPC-FT (Ours)
Custom
Diffusion
Without Fditing
"Rainbow color hair"
"Young bog/girl"
Without Editing
"On hed"
"Pink furry"
Figure 7: Customized generation examples. The images at left are training images, they are all from concept identity. We compared our SD-IPC-CT with Custom Diffusion 13] notice that both results are trained by reference images with merely 30 iterations.
A [V] the street.
[V] on cobblestone
A [V] cat floating on top blue house in wheat field in of water.
A A cat with [V] cat with
A purple [V]
the background. the background.
cat.
SD-IPC-CT
(Ours)
DreamBooth
Custom Diffusion
Textual Inversion
Figure 8: Results of DreamBooth ||| benchmark, the training images are listed top-left corner.
Stable-unCLIP uses PNDM 31.. both with 50 sampling steps. SD-IPC-FT is trained for 100, 50, and 100 epochs on ImageNet (26). CelebA-HQ (27). and Places365 (28), respectively. The learning rates for all datasets are le-5 with cosine decay. Customized generation has constant learning rate of 5e-6 for 30-iteration updates. Training is conducted on 2 A5000 GPUs. The editing a is set to 0.9,
4.2 Image Variation Results
SD-IPC. We evaluate image variation on MSCOCO 24 using all 5,000 images in the 2017 7-p--p validation set. Fig. 3 compares text-based generation, SD-R 4 and our SD-IPC. Both SD-R (4

---

Table 1: Accuracy and Retrieval recalls (%) of original
Table 2: FID-Score 29 and CLIP-
CLIP [15] (C-space) and the inverse matrix transfer (T-
Score (30 (%) of the generation
space). Acc@k
the top-k accuracy of ImageNet [26]
results. SD w/ Text 4 means the
val-set, TR @ and IR @k k are top-k text and image retrieval
generation from ground-truth text.
recalls. This is a surprising result that there is almost no
Methods
FIl
CT.Pe-ccore
performance decline.
SD w/ Text [4
23.65
70.15
Emb. Space
Acc@
Acc
TR@1
TRØ5
IR@1
IRG5
SD-R A
19.86
82,59
C-space
71.41
91.78
74.58
92.98
55,54
82.39
SD-IPC (Ours)
24.78
73.57
T spacc
69.48
90.62
71.62
92.06
54.82
82,20
and our method demonstrate image variation, but our approach utilizes an inverse matrix without
training. The FID-Score [29] and CLIP-Score 30 are reported in Tab. 2 Our SD-IPC maintains image quality similar to text-based generation (FID-Score: 24.78 vs. 23.65, CLIP-Score: 73.57 vs.
70.15). Note that SD-R [4] achieves better results due to its advanced backbone model.
Finetuning. In Fig. it is observed that SD-IPC may exhibit inconsistency, for example, the input "teddybear" generates a picture of "kid". This inconsistency could be attributed to fact that SD-IPC fails to discern the semantically related concepts "kid" and "teddybear" However, this issue can be rectified through fine-tuning, as demonstrated in Fig. 4 SD-IPC-FT achieves improved generation quality. Moreover, we find that the editing capability SD-IPC-FT, as illustrated in Fig. E Fig. and Appendix D.4, surpasses that of SD-R [4. and fine-tuning does not impact the editing performance. We incorporated quantitative experiment to validate the superior editing performance of SD-IPC-FT in comparison to SD-R (4. We utilize images from DreamBooth [I1] benchmark and randomly select an editing text for each test image. Editing performance evaluated using CLIP-T score, the quantitative results are presented in Tab. 4 As seen, our method achieves higher CLIP-T score than SD-R [4] Furthermore, we include the training-free SD-IPC for comparison, revealing even SD-IPC slightly outperforms SD-R 4 The role of different target images used in the fine-tuning stage is outlined in Appendix C.2, which showcases how the choice of target images influences the generation result. Additional generation results are presented in Appendix D
4.3 Customized Generation Results
In Fig. we compare our SD-IPC-CT with Custom Diffusion [13 in terms of customized generation. We evaluate customization for two identities ("Obama" and "Chastain") and two objects ("cat" and "tortoise"). The training process only requires 5 images and 30 iterations. The results Fig. 7 reveal that Custom Diffusion [1] struggles to learn the details of the concept with such limited updates, while our SD-IPC-CT demonstrates impressive performance, particularly in the "young boy/girl" editing. However, for rare instances like the "tortoise" example, both methods do not perform well. For quantitative results, we followed DreamBooth [1]] We used DINO and CLIP-I for subject fidelity and CLIP-T for editing performance. Comparing with DreamBooth [11]. Textual Inversion [1], and Custom Diffusion [13,, the results are in Tab. 3 Fig. & and Appendix [D.] DreamBooth [ excels
in DINO/CLIP-I scores but lags in CLIP-T, indicating limited editing performance, evident in visually similar outputs to training images. Textual Inversion [4) and Custom Diffusion [13] have strong CLIP-T but weak DINO/CLIP-I scores, indicating challenges in preserving subject details. Our SD-IPC-CT method strikes a balance between subject identity preservation and editing performance.
Table 3: Results of DreamBooth benchmark and the comparison
Table 4: Superior editing performance of SD-IPC-FT.
with common methods.
Method
DNIO
CLIP-1
CLIPT
Comments
DreamBonth
III
60.11
77.78
25.81
Good Identity, Weak Editing
Method
CLIPT
Textual Inversion
25.11
62.44
Weak Identity, Good Editing
SDAPCC
26.84
Custom Diffusion B3
39.67
68.37
30.90
Weak Identity, Good Editing
FD-ICCFF
28.69
SD-IPC-CT (Ours)
50.25
74.59
28. 14
Good Identity, Good Edlting
SD-R H
26.01
4.4 Ablation Study
Effectiveness of Inverse Projection Matrix. To evaluate the effectiveness of the inverse projection matrix in SD-IPC, we introduce fully-connected layer instead of the inverse matrix in the image-to- prompt conversion, referred to as SD-IPC-FC and SD-IPC-FC(I), where (I) means initialize with

---

Irange
Inpal
SDJPCC
SDJPCC CTC
SPAPC JT
SDIPC-FT
SD-IPC-FC
SI-PPC JFC
Inpur Imoges
SD-IPC-FC SD-IPC-FT
(Ours)
Figure Image variation results of different fine-tuning settings. SD-IPC-FT (C) means only training CLIP prompts, FD-IPC-FT (U) means only training U-Net cross-attention layers, SD-IPC-FC (I) means initializing the FC-layer with the inverse matrix.
Figure 10: Effectiveness of using Eq. 6 for SD-IPC-FT.
our inverse projection matrix. We train the FC models with the same training data as SD-IPC-FT. However, the results in Fig. 10 indicate SD-IPC-FC suffers from overfitting. SD-IPC-FC() slightly alleviates the overfitting but still gets inferior results, shown in Fig. ? This highlights that our SD-IPC-FT benefits from the good initialization of SD-IPC and preserves knowledge in CLIP [15].
Prompt Learning & U-Net Fine-tuning. We perform quantitative tests (text-edited) image variation for the comprehensive ablation studies following the testing in Sec. 4.2 For text-edited variation, we use the editing text as the prompt, such "A [Class Name] with a mountain in the background.". ". We present the results of individual fine-tuning for two components: SD-IPC-FT (C) for CLIP and SD-IPC-FT (U) for the U-Net. Qualitative results are available in Fig. 2 while quantitative results are provided in Tab. 5 and Tab. 6 It demonstrates that fine-tuning each component contributes model adaptation, with the best performance achieved when simultaneously fine-tuning both two parts. Some editing comparisons are in Appendix C.3
Additionally, we investigate the influence of the editing parameter a in Appendix C.1
Table Results of image variation with dif- ferent fine-tuning settings.
Table Results of text-edited image variation with different fine-tuning settings.
Method
DNIO
CLIP-1
CLIP-T
Method
DNIO
CLIP-1
CLIP-T
SD-IPC
44.60
77.44
25.47
SD-IPC
31.09
68.66
26.84
SD-IPC-FT (C)
49.11
76.51
25.82
SD-IPC-FT (C)
29.10
67.03
27.99
SD-IPC-FT (U)
48.53
79.06
26.17
SD-IP--FT (U)
35.21
69.99
28.56
SD-IPC-FT
52.03
79.59
25.90
SD-IPC-FT
40.28
71.97
28.69
4.5 Limitations & Feature Directions
While SD-IPC offers an alternative to SD-R 4 there are remaining challenges. Firstly, the editing text must be contextually appropriate, as using "on the beach" edit a portrait may result in person being the beach but lacking facial features. Secondly, SD-IPC currently does not support multiple image inputs. Another future study is to extend our method to generate a sequence images with consistency, Appendix shows some potential of our method in this direction.
5
Conclusion
This paper reveals that the CLIP model 15 serves as an mmage-to-prompt converter, enabling image variation in text-to-image Stable Diffusion 4 without extensive training. This finding enhances our understanding of the CLIP embedding space, demonstrating that simple inverse matrix can convert visual embeddings into textual prompts. Leveraging this -o-pomppp conversion, our SD-IPC methods achieve impressive image variation and editing capabilities, while also enabling fast adaptation for customized generation. Experimental results also show the potential of our method in more multi-modal tasks. We anticipate that this study will inspire future research exploring the tn-oott prompt pathway in CLiP-based LDMDbased models.
10

---

Acknowledgement
This work was partly supported by the China Scholarship Council under Grant 202006960047 and partly by the National Natural Science Foundation of China (No.62173265), Lingqiao Liu is supported by Centre of Augmented Reasoning.
References
[1] Ramesh, A., M. Pavlov, G. Goh, et al. Zero-shot text-to-image generation. In International Conference Machine Learning, pages 8821-8831 PMLR, 2021.
[2] Gafni, O., A. Polyak, O. Ashual, al. Make-a-scene: Scene- based text-to-image generation with human priors. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 89-106. Springer, 2022.
[3] Ramesh, A., P. Dhariwal, A. Nichol, et al. Hierarchical text-conditional image generation with clip latents. arXiv preprint urXiv:2204.06125 2022.
[4] Rombach, R., A. Blattmann, D. Lorenz, et al. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695. 2022.
[5] Feng, W. X. He, T.-. Fu, et al. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05033 2022.
[6] Liu, N., S. Li, Du, et Compositional visual generation with composable diffusion models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27,
2022, Proceedings, Part XVII, pages 423-439. Springer, 2022.
[7] Chefer, H., Y. Alaluf, Y. Vinker, et al. sitend-and-excit:: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint 13X1v:2301..3826, 2023.
[8] Zhang, L., M. Agrawala. Adding conditional control text-to-image diffusion models. arXiv preprint ar arXiv:2302.05544 2023.
[9] Hertz, A., R. Mokady, J. Tenenbaum, et al. Prompt-to-promp image editing with cross attention control. arXiv preprint arXiv:2208. 016006 2022.
[10] Tumanyan, N., M. Geyer, S. Bagon, et al. Plug-and-play diffusion features for text-driven mmage-to-image translation. arXiv preprint arXiv:2211. 1.221.11255 12572, 2022.
[11] Ruiz, N., Y. Li, V. Jampani, et al. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. 12242, 2022.
[12] Kawar, B., S. Zada, O. Lang, et al. Imagie: Text-based real image editing with diffusion models. arXiv preprint arXiv:22 2..00:2200009276,, 2022.
[13] Kumari, N., B. Zhang, R. Zhang, et al. Multi-concept customization of text-to-image diffusion.
In CVPR. 2023.
[14] Gal, R., Alaluf, Y. Atzmon, et al. An image is worth one word: Personalizing text-to- to-image generation using textual inversion. arXiv preprint arXiv:2208.016 2022.
[15] Radford, A., J. W. Kim, C. Hallacy, et al. Learning transferable visual models from natural International conference on machine learning, pages 8748-8763.
language supervision.
PMLR, 2021.
[16] Sohl-Diekstein, J., E. Weiss, N. Maheswaranathan, et al. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[17] Ho, J., A. Jain, P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, $3:6840-6855 2020.
[18] Dhariwal, P., A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, $4:8780-8794, 2021.
[19] Song, J., C. Meng, S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.

---

[20] Mokady, R., A. Hertz, K. Aberman, et al. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6038-6047. 2023.
[21] Parmar, G., K. Kumar Singh, R. Zhang, et al. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11. 2023.
[22] Brooks, T., A. Holynski, A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 18392-18402. 2023.
[23] Penrose, R. A generalized inverse for matrices. In Mathematical proceedings of the Cambridge philosophical society, vol. 51, pages 406-413, Cambridge University Press, 1955.
[24] Lin, T.-Y., M. Maire, S. Belongie, al. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[25] Jia, M., L. Tang, B.-C. Chen, et al. Visual prompt tuning. In Computer Vision-ECCV 2022:
17th European Conference, Tel Aviv, Israel October 23-27, 2022, Proceedings, Part XXXIII, pages 709-727. Springer, 2022.
[26] Deng. J., W. Dong, R. Socher, et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, Icee, 2009.
[27] Karras, T., T. Aila, Laine, et al. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations. 2018.
[28] Zhou, B., A. Lapedriza, A. Khosla, et al. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 1416):1452-1464 2017.
[29] Heusel, M., H. Ramsauer, Unterthiner, et al. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[30] Hessel, J., A. Holtzman, M. Forbes, et Clipscore: reference- evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528. 2021.
[31] Liu, L., Y. Ren, Z. Lin, et al. Pseudo numerical methods for diffusion models on manifolds, In International Conference on Learning Representations. 2022.
12


Reviews:
[{"review_id": "kAqVlFl8TG", "reviewer": "Reviewer_rWok", "summary": "This paper demonstrates that the CLIP model, used in Stable Diffusion, inherently possesses the ability to convert images into text prompts. They achieve this by utilizing a linear projection matrix calculated in a closed form. Furthermore, the paper shows that this capability can be enhanced by incorporating a small amount of similar-domain training data or performing online training steps on reference images. These approaches offer a simple and flexible solution to bridge the gap between images and text prompts, enabling more effective interaction between the two modalities.", "questions": "1. Since SD-IPC and textual inversion all aims to convert the visual embeddings into the text prompt, detailed comparison with it should be included in this part. From my point of view, SD-IPC is just a simplified textual inversion method.\n2. For Customized Generation, more comparisons should be provided in this paper.", "limitations": "Not applicable.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well organized and the motivation is good. The basic idea of Image-to-Prompt Conversion is simple but interesting. I like the idea of using a closed-form projection matrix to converts the visual embeddings into the semantic prompt which can help control the stable diffusion.\n2. The insights about start-/end-token's attention map for stable diffusion are impressive.\n3. The proposed methods' finetuning time of customized generation is significantly faster than the current methods.", "weaknesses": "1. Although the basic idea of the paper is cool, the visual results of SD-IPC and SD-IPC-FT do not have sufficient preponderance over than previous methods.\n2. This paper only compares proposed methods with Customized Generation and SD-R. I think the authors should also add experiments to compare the SD-IPC-FT with DreamBooth (with and w/o LoRA), textual inversion and Reference-only module for controlnet. \n3. As shown in Table 2, SD-IPC achieves worst FID which means they output the most unrealistic methods. Besides, the CLIP-Score are also not good enough. I think authors should add more explanations of these tables in the SD-IPC part in section 4.2. Aslo, more comparisons should be added in these tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. The common questions are first answered in **General Responses**, then we clarify questions from individual review.\n___\n### Q1: The visual results of SD-IPC and SD-IPC-FT do not have sufficient preponderance over than previous methods.\nSD-IPC-FT surpasses prior methods like SD-R [1] in terms of editing performance. SD-R [1] occasionally struggles with effective editing, as shown in Paper Figure 5, 6, and Rebuttal Figure 3. Quantitative results in **General Responses Q3** reinforce this point, with our method achieving an editing fidelity (CLIP-T) of 28.69, compared to SD-R [1] at 26.01. Notably, even the training-free SD-IPC attains a CLIP-T score of 26.84.\n___\n### Q2: I think the authors should also add experiments to compare the SD-IPC-FT with DreamBooth (with and w/o LoRA), textual inversion and Reference-only module for controlnet.\n\nThank you for pointing out this issue, we have addressed this matter in **General Responses Q2**, where we present the experiments of customized generation. Our SD-IPC-CT is comprehensively compared with DreamBooth [2], Textual Inversion [3], and Custom Diffusion [4]. The visual results are depicted in Rebuttal Figure 1. We will incorporate these results into our final version and provide more visual results in the supplementary material. We acknowledge your interest regarding a comparison with the recently introduced Reference-only controlnet. However, we must clarify that this falls outside the scope of our paper, as the Reference-only controlnet was released in May 2023, coinciding with the time of our submission.\n___\n### Q3: As shown in Table 2, SD-IPC achieves worst FID which means they output the most unrealistic methods. Besides, the CLIP-Score are also not good enough.\nThank you for the comment, it appears that the reviewer misunderstood the purpose of Table 2: Table 2 is used to demonstrate the effectiveness of the training-free image-to-prompt conversion equation, which severs as the initialization of SD-IPC-FT and SD-IPC-CT.\nIn Table 2, we can see that SD-IPC achieves comparable results to SD with ground-truth text (Line. 216). The CLIP score, which compares generated and reference images, quantifies their \"content distance\". It's important for the CLIP score to fall within a reasonable range, excessively high scores may indicate a lack of variation, (For example, if the generated image is identical to the to-be-varied image (no variation effect), it will result in the highest CLIP score) while too low scores suggest fidelity loss in the generated image. When referring to SD with ground-truth text prompt (SD w/ Text), it means generating images using the text-prompt of reference image. Consequently, the generated result effectively encapsulates all desired semantics from the reference image. Our method achieving a similar CLIP score to SD w/ Text demonstrates its ability to generate substantial variation while maintaining sufficient semantics. While our method may exhibit lower image quality than SD-R (partly due to the backbone difference, V1.4 vs. V2.1), our approach still outperforms SD-R in editing performance, as evident in **General Responses Q3**. As shown in Rebuttal Figure 5, our method also works for Stable Diffusion V2.1, we will update our results in the new version.\n___\n### Q4: From my point of view, SD-IPC is just a simplified textual inversion method.\nAs stated in **General Responses Q1**, SD-IPC and SD-IPC-FT are designed for image variation generation, focusing on generating similar semantics as the reference, e.g., similar scene or object. Textual Inversion [3] aims customized image generation, preserving the reference object's identity. Notably, our SD-IPC-CT also enables fast customized image generation. Furthermore, **General Responses Q2** expands more comparisons beyond the results shown in Figure 7 of the current paper.\n___\n### Q5: For Customized Generation, more comparisons should be provided in this paper.\nWe appreciate your suggestion. The experiments related to customized generation have been detailed in **General Responses Q2**. Both quantitative and qualitative results will be incorporated into our final version.\n___\n[1] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Ruiz, Nataniel, et al. \"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3] Gal, Rinon, et al. \"An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[4] Kumari, Nupur, et al. \"Multi-concept customization of text-to-image diffusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n"}, {"review_id": "usBZwdwlzi", "reviewer": "Reviewer_1WWU", "summary": "This paper titled presents a method called Stable Diffusion Image-to-Prompt Conversion (SD-IPC) that leverages the inherent capabilities of the Contrastive Language-Image Pre-Training (CLIP) model to convert images into text prompts for image generation tasks. The authors start from the analysis that the control of image generation through text is primarily influenced by the embedding of the end-of-sentence (EOS) token, and masking other word tokens does not significantly affect the quality of image generation. From this constatation, they derive a closed-form projection matrix that converts visual embeddings into text embeddings and use them to control the Stable Diffusion image generation process.", "questions": "As the authors said, the approximation of the inversion of the textual projection matrix may be a source of suboptimal results. How does this impact the generation process? ", "limitations": "The authors show some limitations in their work including the need for coherence between the editing prompt and the target image or the lack of multiple target images.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": " **Technical soundness** The paper is well written and the explanation of the underlying mechanisms of the CLIP model and its relationship with image generation is made intuitive. \n\n**Simple and clever visual-to-prompt embedding conversion** The paper presents a straightforward yet ingenious approach to convert images into text prompts by inverting the textual projection layer. This conversion allows for image variations and editing, effectively bridging the gap between images and textual prompts. The simplicity and cleverness of this method contribute to its practicality and usability.\n\n**Efficient variation model learning through fine-tuning** The authors propose two methods to enhance the quality and flexibility of image-to-prompt conversion. The first method involves parameter-efficient tuning using a small amount of data, requiring minimal computational resources and time. This approach enables efficient learning of the variation model, making it practical for real-world applications.\n\n**Visually sound generations and variations** The paper showcases visually sound image generations and variations achieved through the proposed SD-IPC method. By leveraging the inherent capabilities of the CLIP model, the generated images align well with the desired concepts and demonstrate high-quality results.", "weaknesses": "The method is only compared with SD-R and the inclusion of additional comparisons with existing methods could have further strengthened the paper in particular for the evaluation of the editing capability of the method. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments. Some common questions are first addressed in the **General Responses**, followed by answers to individual reviews.\n___\n### Q1: Additional comparisons with existing methods could have further strengthened the paper in particular for the evaluation of the editing capability of the method.\nThanks for your suggestion. The experiments of customized generation have been detailed in **General Responses Q2**. Both quantitative and qualitative results will be incorporated into our final version.\n___\n### Q2: As the authors said, the approximation of the inversion of the textual projection matrix may be a source of suboptimal results. How does this impact the generation process?\nWe express our gratitude to the reviewer for raising this important concern. The discrepancy between the projected visual feature and its textual counterpart indeed introduces confusion between concepts. As evident in Paper Figure 4 and Line 219$\\sim$221, an image of a \"teddybear\" was transformed into a picture of a \"kid\", as also demonstrated in Supplementary Material Figure 15, where a \"bear\" became a \"tiger\". The primary driver of this confusion stems from the inherent modality gap between images and text. Our CLIP experiments have revealed the cosine similarity between ground-truth image-text pairs is only around 0.3, which signifies the alignment of features is not perfect. This modality gap also persists after the inverse matrix projection, leading to the perplexing shifts between closely related concepts.\n\nMotivated by this gap, we propose SD-IPC-FT approach to reduce the gap via fine-tuning partial parameters of the SD model.\n___\n"}, {"review_id": "HpNjrtYkwn", "reviewer": "Reviewer_aD4y", "summary": "This paper focuses on the problem of generating images by a reference image w/ or w/o further text guidance. The core of this problem partly lies in how to convert the images to embeddings that can be directly feed into Stable Diffusion model. To this end, the authors propose to leverage CLIP model, the text encoder of which is also part of the stable diffusion model. They compare results with Stable Diffusion Re-imagination, as well as Custom diffusion.", "questions": "see above", "limitations": "I guess two obvious limitation are:\n(1) may only work for text-to-image models that relies on CLIP encoder\n(2) need to get re-trained every time for a new domain.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The strengths are as follows:\n\n- This is a timely interesting topic. How to generate new images by a referring image has been an interesting problem, especially given the recent burst of diffusion models. Compared to previous method such as Dreambooth, this method is light-weight. Authors even provided a version without any fine-tuning, though the performance may not be that good. But the authors also provide a easy-to-finetune pipleline which seems to largely improve the performance\n\n- The training cost is much lower. Compared to SD-R which requires 200,000 GPU-hours, this method here reduced the cost to only 100 images with 1 GPU-hour. Compared to Custom Diffusion which requires 250 iterations of fine-tuning, this paper only required ~30 steps.\n\n- This paper provides the generation capability both w/ or w/o text, in addition to the reference image. This is a nice property.\n\n- The results look reasonable.", "weaknesses": "The first thing to improve might be the writing.\n\n(1) The abstract part can be improved. When I first read the abstract without knowing the main text, I was a bit struggling to understand which exactly problem this paper is particularly trying to solve. Converting image to text? Improving Diffusion? or Image-to-Image generation? This part can be clearer.\n\n(2) There is actually not much contents discussing related work, despite the existence of section 2. I would recommend authors have a separate related section, that discuss prior work sufficiently. For example, the current 2.3 section only discussed SD-R and just mentioned dreambooth, textual inversion, and custom diffusion. But there are more works, such as instructpix2pix, plug-and-play, etc. The similarities and differences can be articulated. Besides, much background contents in section 2 cam actually be cut, e.g., line 81-84.\n\n(3) The formulation in Section 3 can be improved. E.g., where is the 76 comes from? This is not obvious for people unfamiliar with the CLIP details. When you combine text prompt and converted image prompt in equation (8), how do you do that clearly?\n\n(4) you can have a figure illustrating the fine-tuning process in section 3.2\n\nThe next question I have is that, can this method be generally applied to other text-to-image diffusion models. It seems to me that this paper exploits the prior knowledge that stable diffusion used CLIP text encoder, thus you can use text image encoder to convert image to text prompt. This is fine, but would this still work with other approaches such as imagen that relies on T5 model? e.g. dreambooth is agnostic to the text encoder and thus is more generic.\n\nWhile the authors claimed this method is cost effective compared to the the SD-R trained on millions of images and over 200,000 GPU hours, has the author measured the out-of-distribution capability sufficiently? e.g. SD-R may just train once, and generalize to all images, but this method needs to be trained for each domain.\n\nDetailed question:\n\n(1) it seems the way you combine text and image prompts in eq (8) is adhoc, what if the text tokens are longer than 76, then where do you put those $f_{txt}^{comb}$? This seems technically problematic.\n(2) For line 133, you assumed the norm to be 27 as a constant, which seems also problematic to me. This feels not rigorous. One better way to justify this is to visualize the histograms of the norms.\n(3) for imagenet, cele-A, and places fine-tuning using equation (9), how do you get the text, image pair for the $L_{text}$?\n\n\nResults section:\n\n(1) directly compared the SD-IPC-FT w/ text version with SD-R is apple-to-orange. The SD-R just uses image as guidance, you should compare the SD-IPC ( w/ or w/o FT) version that only uses images as well. You can have w/ and w/o text results both there.\n(2) No quantitative comparison between SD-IPC-FT with custom diffusion.\n(3) seems SD-R without text input is still better in Table 2?\n(4) custom diffusion accept multiple concepts, but seems this method is not able to.\n(5) In figure 9, for SD-IPC-FC, what if you initialize the fc layer from the closed form inverse projection, but make it trainable?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestion. The common questions are first answered in **General Responses**. Below please find the responses to specific comments.\n___\n### Q1: The abstract part can be improved.\nWe appreciate your suggestion. We will rewrite the abstract to highlight the relationship of the proposed methods, and the relationship with other methods, as summarized in **General Responses Q1**.\n___\n### Q2: Separate related section, that discuss prior work sufficiently.\nWe will incorporate a discussion in the final version. We have included some quantitative experiments in **General Responses Q2**. It's important to note that, as explained in **General Responses Q1**, our method is primarily focus on image variation (producing similar images to the reference image) and customized image generation. As a result, direct comparisons with image editing techniques such as instructpix2pix, plug-and-play may not be applicable.\n___\n### Q3: Questions about the text token length.\nThe number 76 derives from the CLIP, where the text's maximum length is 77 (token index is from 0 to 76). The sentence would be padded or truncated to length of 77. This is also followed by the Stable Diffusion [1]. In the context of Eq.(8), the combination is computed by adding the end-token embedding with a weight $\\alpha$ to the converted image embedding, as $f_{txt}^{comb} = f_{txt}^{cnvrt} + \\alpha \\cdot f_{txt}^{t,\\left\\langle {eos} \\right\\rangle }$. After the combination, the $f_{txt}^{comb}$ would replace the end-token and all pad-tokens.\n___\n### Q4: Figure illustrating the fine-tuning process.\nWe appreciate your valuable suggestion. The framework of our method has been presented in the supplementary material Section A. In response to your comment, we will improve the figures to provide clearer insights.\n___\n### Q5: Can this method be generally applied to other text-to-image diffusion models?\nOur method is compatible with the diffusion model utilizing CLIP as text encoder. As Stable Diffusion is now the dominating text-to-image generation model, our work has sufficient impact to the related research community. \n___\n### Q6: SD-R may just train once, but this method needs to be trained for each domain.\nOur method shows out-of-distribution generalization in experiment results. Notably, the testing images featured in the paper remain entirely separate from the fine-tuned images. Furthermore, the fine-tuning of SD-IPC-FT on object, scene, and portrait serves to **define what to preserve from the reference image**. This is a **distinct advantage of our method** as image variation inherently has some ambiguity in deciding what to preserve from the reference image. In contrast, SD-R lacks this capability, as shown in Paper Figure 4. SD-R struggles to preserve scene information in the \"Bar\" example (it still focuses on the crowd). In other words, our fine-tuning is **not on individual domains**, but rather to **addressing diverse requirements of preservation**.\n___\n### Q7: Visualize the histograms of the norms.\nThe histograms depicting the norms of embeddings can be found in Rebuttal Figure 4. Specifically, the norms of a randomly chosen set of 1,000 texts ranges from 26.5 to 29.\n___\n### Q8: For imagenet, celeb-A, and places fine-tuning, how do you get the text?\nThe texts of ImageNet and Places365 are class name prompt \"This is a photo of [Class Name].\". For portrait, we utilize the MM-CelebA-HQ dataset which has image captions for each portrait.\n___\n### Q9: Directly compared the SD-IPC-FT w/ text version with SD-R is apple-to-orange.\nWe have indeed compared both SD-IPC-FT **w/ text** and SD-R **w/ text** in Paper Figure 5 and Figure 6, where the samples labeled as \"Without Editing\" showcase results only from images, while the other samples encompass edited text, such as \"On the beach\" and \"Under the starry sky\". It is evident from figures that our SD-IPC-FT surpasses SD-R in terms of editing performance. For a quantitative comparison, refer to **General Responses Q3**. Apart from the \"Wearing a hat\" instance of the male in Paper Figure 6, SD-R exhibits minor edits. This circumstance might mislead the reviewer to think them as SD-R results without editing. \n___\n### Q10: No quantitative comparison between SD-IPC-FT with custom diffusion.\nThank you for your suggestion. We have reported the quantitative comparison in **General Responses Q2** and visual results in Rebuttal Figure 1.\n___\n### Q11: SD-R without text input is still better in Table 2.\nIn Table 2, we aimed to illustrate that the training-free SD-IPC achieves comparable results to SD w/ Text (Line. 216). Notably, excessively high CLIP score may indicate a lack of variation, while too low score suggests fidelity loss in the generated image. The SD w/ Text effectively encapsulates all desired semantics from the reference image. Our method achieves a similar score to SD w/ Text, demonstrating its ability of substantial variation while maintaining sufficient semantics. Our method may exhibit lower image quality than SD-R, due to the backbone difference, V1.4 vs. V2.1, ours still outperforms SD-R in editing performance, as evident in **General Responses Q3**. Also shown in Rebuttal Figure 5, our method works for SD V2.1.\n___\n### Q12: For SD-IPC-FC, initialize the fc layer from the closed form inverse projection.\nFollowing the benchmark in **General Responses Q3**, we evaluate the image variation, where the CLIP-T is between the generated image and a prompt \"This is a photo of [Class Name].\". SD-IPC-FC (I) is to initialize the FC layer with inverse projection. The initialization can alleviate the overfitting, but only with limited effectiveness. Visual results are in Rebuttal Figure 2, quantitative comparisons are as followed:\n| Method | DINO | CLIP-I | CLIP-T |\n| :--- | :---: | :---: | :---: | \nSD-IPC | 44.60 | 77.44 | 25.47 |\nSD-IPC-FT | **52.03** | **79.59** | **25.90** |\nSD-IPC-FC | 24.71 | 64.96 | 23.76 |\nSD-IPC-FC (I) | 46.55 | 76.54 | 25.78 |\n___"}, {"review_id": "LUA8iOZMid", "reviewer": "Reviewer_hEWJ", "summary": "This paper demonstrates that the CLIP model in Stable Diffusion inherently possesses the ability to convert images into text prompts, which can be achieved by utilizing a linear projection matrix that is calculated in a closed form.", "questions": "Please see the weaknesses. I am willing to improve my score if the concerns are addressed.", "limitations": "The limitations have been described in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of image-to-prompt conversion is clear and the proposed closed-form method is concise.\n2. The capability of image-to-prompt conversion can be further enhanced by finetuning the model with a small amount of data and time.\n3. The paper demonstrates various applications of the method.", "weaknesses": "1. The main concern lies in the experiments section:\n(1) The quantitative comparison between SD-IPC and SD-IPC-FT can be given. And the quantitative and qualitative results when separately fine-tuning the two specific types of parameters (cross-attention layers or deep prompts) also can be given.\n(2) Why not SD-IPC adopts the backbone of  SD-R, i.e.,  Stable Diffusion v2.1 ?\n(3) In Sec.4.3, I suggest the paper adopts the dataset and edited prompt used by DreamBooth (https://github.com/google/dreambooth) to make a full comparison. Both qualitative and quantitative results should be presented. And the proposed method can also be compared with more methods (e.g., DreamBooth and Textual Inversion).\n(4) It would be better if the quantitative results of the first ablation experiment are presented.\n2. The motivation for applying deep prompts tuning in SD-IPC-FT needs to be further explained.\n3. The editability of the proposed method may be limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. The common questions are first answered in the **General Responses**, then we clarify specific questions.\n___\n### Q1: The quantitative comparison between SD-IPC and SD-IPC-FT can be given. And the quantitative and qualitative results when separately fine-tuning the two specific types of parameters (cross-attention layers or deep prompts) also can be given.\nWe appreciate your valuable comment. For the comprehensive ablation studies, we perform quantitative tests on image variation and text-edited image variation using images from the benchmark in **General Responses Q3**. DINO and CLIP-I are computed between the generated images and real images. For image variation, we calculate the CLIP-T between the generated image and the prompt \"This is a photo of [Class Name].\". For text-edited image variation, we use the editing text as the prompt, such as \"A [Class Name] with a mountain in the background.\". We report the results of individually fine-tuning the two parts, referred to SD-IPC-FT (C) for the CLIP part and SD-IPC-FT (U) for the U-Net part.\nThe qualitative results of image variation are in Rebuttal Figure 2, here is the quantitative results:\n\n| Method | DINO | CLIP-I | CLIP-T |\n| :--- | :---: | :---: | :---: | \nSD-IPC | 44.60 | 77.44 | 25.47 |\nSD-IPC-FT (C) | 49.11 | 76.51 | 25.82 |\nSD-IPC-FT (U) | 48.53 | 79.06 | **26.17** |\nSD-IPC-FT | **52.03** | **79.59** | 25.90 |\n\nThe qualitative results of text-edited image variation are Rebuttal Figure 3, the quantitative results are below:\n\n| Method | DINO | CLIP-I | CLIP-T |\n| :--- | :---: | :---: | :---: | \nSD-IPC | 31.09 | 68.66 | 26.84 |\nSD-IPC-FT (C) | 29.10 | 67.03 | 27.99 |\nSD-IPC-FT (U) | 35.21 | 69.99 | 28.56 |\nSD-IPC-FT | **40.28** | **71.97** | **28.69** |\n\nAs depicted in the two tables, fine-tuning both the CLIP prompts and U-Net layers contribute to achieving better results.\n___\n### Q2: Why not SD-IPC adopts the backbone of SD-R, i.e., Stable Diffusion v2.1?\nWe appreciate your suggestion. Initially, we followed Custom Diffusion [1], which employs Stable Diffusion v1.4, as a starting point for developing our method. SD-R [2] was made available in March 2023, subsequent to our method's development. In response to your point, we have ascertained that SD-IPC is compatible with Stable Diffusion v2.1 as well, and the variation results can be found in Rebuttal Figure 5.\n___\n### Q3: In Sec.4.3, I suggest the paper adopts the dataset and edited prompt used by DreamBooth to make a full comparison. Both qualitative and quantitative results should be presented.\nThank you for pointing out this issue, we have addressed this matter in **General Responses Q2**, where we present the experiments of customized generation. The qualitative comparisons are in Rebuttal Figure 1. We will incorporate these results into our final version and provide more visual results in the supplementary material.\n___\n### Q4: It would be better if the quantitative results of the first ablation experiment are presented.\nThank you for the constructive comments. The ablation study is reported in **Q1**.\n___\n### Q5: The motivation for applying deep prompts tuning in SD-IPC-FT needs to be further explained.\nGiven our limited training data consisting of only 100 images for fine-tuning the Stable Diffusion [1], directly adjusting every parameter could potentially result in severe overfitting and the loss of previously acquired knowledge. In light of this, a solution emerged from prior research [3], where deep prompt-tuning exhibited to be able to resist overfitting. On the other hand, the fine-tuning aims to address inferior CLIP feature, and to change the focus on different aspects of the feature (object, scene, or portrait). Consequently, we have chosen to adopt this approach. \n\n___\n[1] Kumari, Nupur, et al. \"Multi-concept customization of text-to-image diffusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[3] Zhou, Ziqin, et al. \"Zegclip: Towards adapting clip for zero-shot semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n\n"}]

Author rebuttal:
# General Responses
We thank all reviewers for your thoughtful and detailed feedback, which is of great importance to our work. Here we address some common issues and share some findings that all reviewers might have interests in.
___
### Q1: Clarification of certain facets of the proposed method.
This paper presents a novel finding that allows for the direct transformation of an image into a textual prompt embedding in the CLIP model [1]. Leveraging this finding, we formulate a range of methods with distinct advantages and applications. In the subsequent sections, we highlight the certain facets of our methods:

 Method | Application | Comparisons | Merits | Relationship |
 :--- | :--- | :--- | :--- | :--- |
 SD-IPC | Image Variation | SD-R [2] | Training-free, Demonstrate the finding | Serving as the initiation of SD-IPC-FT and SD-IPC-CT |
 SD-IPC-FT | Image Variation | SD-R [2] | Lightweight training, Able to customize what to preserve from reference images | Tuning SD-IPC offline, Better edit effect than SD-R [2] |
 SD-IPC-CT | Customized Generation | DreamBooth [3], Textual Inversion [4], Custom Diffusion [5] | Fast online training, Better generation quality with few training iterations | Tuning SD-IPC online |
___
### Q2: Results of customized generation benchmark and the comparison with common methods.
Following the benchmark proposed in DreamBooth [3], we employ a dataset comprising 30 subjects belonging to 15 different classes for the fine-tuning. Each subject is represented by 4$\sim$6 images, and will be edited using a total of 25 texts for 4 times, getting a results of 3,000 images. DINO and CLIP-I score are computed for subject fidelity (preservation of subject identity details), CLIP-T score is for prompt fidelity (the editing performance). We compare our SD-IPC-CT with three common customized generation methods: DreamBooth [3], Textual Inversion [4], and Custom Diffusion [5]. The visual results are depicted in Figure 1 of Rebuttal PDF, the quantitative results obtained are as followed:

| Method | DINO | CLIP-I | CLIP-T | Comments |
| :--- | :---: | :---: | :---: | :--- |
| DreamBooth [3] | **60.11** | **77.78** | 25.81 | **Good Identity**, Weak Editing |
| Textual Inversion [4] | 25.11 | 62.44 | 29.53 | Weak Identity, **Good Editing** |
| Custom Diffusion [5] | 39.67 | 68.37 | **30.90** | Weak Identity, **Good Editing** |
| SD-IPC-CT (Ours) | 50.25 | 74.59 | 28.14 | **Good Identity**, **Good Editing** |

The result demonstrates that DreamBooth [3] obtains the highest DINO/CLIP-I scores. However, its CLIP-T score is significantly lower, indicating unsatisfactory editing performance. This is also apparent in the visual results, where DreamBooth [3] generated images closely resemble the training images with minor edits. In contrast, Textual Inversion [4] and Custom Diffusion [5] exhibit strong CLIP-T scores, albeit much lower DINO/CLIP-I scores, highlighting their weakness in preserving subject details. In comparison, our SD-IPC-CT method strikes a good balance between subject identity preservation and editing performance.

___
### Q3: The advantage of the proposed method over SD-R [2].
In the current version of paper, we highlight SD-IPC-FT's advantage over SD-R [2] in Paper Figure 5, Figure 6, demonstrating the superior editing performance of SD-IPC-FT. Here we add a quantitative experiment to valid this claim. Specifically, we fine-tune SD-IPC-FT using 100 ImageNet [6] images, ensuring it retains object-level details from a reference image. For testing, we utilize the total of 158 images from DreamBooth benchmark, featuring 30 subjects. To conduct the text-edited image variation task, we randomly select an editing text from DreamBooth benchmark for each test image. Editing performance is evaluated using CLIP-T score. Importantly, DINO and CLIP-I scores are omitted as evaluation since they indicate similarity, which does not reflect the quality of "variation''. For example, if the generated image is identical to the to-be-varied image (no variation effect), it will result in the highest DINO and CLIP-I score. The visual results are shown in Rebuttal Figure 3 (the SD-IPC-FT (C) for only fine-tuning the CLIP part and SD-IPC-FT (U) for only fine-tuning the U-Net part, this two are for ablation study). The quantitative results are as followed: 

| Method | CLIP-T |
| :--- | :---: | 
SD-IPC | 26.84 |
SD-IPC-FT | **28.69** |
SD-R [2] | 26.01 |

As seen, our method achieving a higher CLIP-T score than SD-R. Furthermore, we include the training-free SD-IPC for comparison, revealing even SD-IPC slightly outperforming SD-R in CLIP-T score.
___
[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

[2] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[3] Ruiz, Nataniel, et al. "Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[4] Gal, Rinon, et al. "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion." The Eleventh International Conference on Learning Representations. 2022.

[5] Kumari, Nupur, et al. "Multi-concept customization of text-to-image diffusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[6] Deng, Jia, et al. "Imagenet: A large-scale hierarchical image database." 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009.

Meta-review: This paper studies how to generate images by a reference image with Stable Diffusion. It proposes to leverage CLIP model to convert images into text prompts by utilizing a linear projection matrix calculated in a closed form. The reviewers in general found the proposed method as technical sound, well motivated, interesting and efficient. The initial concerns were mainly about missing comparisons with existing methods and confusion caused by presentation, which were well addressed in the rebuttal. As a result, all reviewers unanimously agree to accept this submission. In the final version, the authors should include all the clarifications and the additional empirical results provided in the rebuttal.
Decision: Accept (poster)