SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuxuan Ding*
School of Electronic Engineering
Xi'an 710071, China
yxding@stu.xidian.edu.cn
Chunna Tian
chnatian@xidian.edu.cn
Haoxuan Ding
Unmanned System Research Institute
Xi'an 710072, China
haoxuan.ding@mail.nwpu.edu.cn
Lingqiao Liu
The University of Adelaide
Adelaide 5005, Australia
lingqiao.liu@adelaide.edu.au
The Stable Diffusion model is a prominent text-to-image generation model that relies on a text prompt as its input, which is encoded using the Contrastive Language-Image Pre-Training (CLIP). However, text prompts have limitations when it comes to incorporating implicit information from reference images. Existing methods have attempted to address this limitation by employing expensive training procedures involving millions of training samples for image-to-image generation. In contrast, this paper demonstrates that the CLIP model, as utilized in Stable Diffusion, inherently possesses the ability to instantaneously convert images into text prompts. Such an image-to-prompt conversion can be achieved by utilizing a linear projection matrix that is calculated in a closed form. Moreover, the paper showcases that this capability can be further enhanced by either utilizing a small amount of similar-domain training data (approximately 100 images) or incorporating several online training steps (around 30 iterations) on the reference images. By leveraging these approaches, the proposed method offers a simple and flexible solution to bridge the gap between images and text prompts. This methodology can be applied to various tasks such as image variation and image editing, facilitating more effective and seamless interaction between images and textual prompts.
In recent years, there has been a surge of interest in vision-and-language research, particularly in the field of text-to-image generation. Prominent models in this domain include autoregression models like DALL-E [1] and Make-A-Scene [2], as well as diffusion models like DALL-E 2 [3] and Stable Diffusion [4]. These models have revolutionized the quality of generated images. They leverage text prompts to synthesize images depicting various objects and scenes that align with the given text. Among these models, Stable Diffusion [4] stands out as a significant open-source model. It serves as a foundation for many recent works, including image generation [5, 6, 7, 8], image editing [9, 10, 11, 12, 13, 14], and more.
*This work was done while Yuxuan Ding was visiting The University of Adelaide as a visiting researcher.
Figure 1: Demonstration of image variation. The image on the left is a real reference image, while the four on the right are generated from our method.
Figure 2: Attention map of Stable Diffusion [4]. The bottom row sets attention weights of caption words to zero, only keeping the start-/end-token, so the caption maps of the bottom are black. Also notice the start-token has strong weights so the map is all white.
However, text prompts have limitations when it comes to incorporating unspeakable information from reference images. It becomes challenging to generate a perfect and detailed prompt when users want to synthesize images related to a picture they have seen. Image variation techniques aim to address this limitation by enabling users to generate multiple variations of an input image, without relying on complex prompts. As illustrated in Fig. 1, the generated variations closely resemble the reference image, often sharing the same scene or objects but with distinct details.
Stable Diffusion Reimagine (SD-R) [4] is a recently proposed image variation algorithm. It achieves this goal by retraining Stable Diffusion [4], where the text encoder is replaced with an image encoder to adapt the model for image input. The model is trained using millions of images and over 200,000 GPU-hours, enabling it to effectively generate image variations based on reference images.
In this paper, we make a significant discovery that allows a more cost-effective image-to-prompt conversion approach. We find the CLIP model [15], as utilized in Stable Diffusion, can be repurposed as an effective image-to-prompt converter. This converter can be directly employed or served as a valuable initialization for a data-efficient fine-tuning process. As a result, the expenses associated with constructing or customizing an image-to-prompt converter can be substantially reduced.
More specifically, our method is built upon a surprising discovery: the control of image generation through text is primarily influenced by the embedding of the end-of-sentence (EOS) token. We found that masking all word tokens, except for the start and end tokens, does not adversely affect the quality of image generation, as illustrated in Figure 2. Simultaneously, during CLIP training, the projection of the end-token embedding is trained to align with the visual embedding. This inherent relationship enables us to derive a closed-form projection matrix that converts visual embedding into an embedding that is capable of controlling the generation of Stable Diffusion [4]. We call this method Stable Diffusion Image-to-Prompt Conversion (SD-IPC).
In addition, we introduce two methods to enhance the quality and flexibility of image-to-prompt conversion. The first approach involves parameter-efficient tuning using a small amount of data, consisting of only 100 images and requiring just 1 GPU-hour. This method encourages the model to better preserve image information and enables practitioners to control the specific content they want to retain when generating new images. The second approach involves customizing the model on reference images using a few iterations, ensuring that the generated images are closer to specific concepts. While this approach has been explored in previous research, we demonstrate that with the advantageous initialization provided by SD-IPC, the online fine-tuning requires significantly fewer iterations to achieve desirable results.
Firstly, we present a brief overview of the Stable Diffusion [4], which serves as our underlying model. Diffusion models (DMs) [16, 17, 18, 19] belong to a class of latent variable models. In DMs, there exist two Markov chains known as the diffusion process and the reverse process, both having a fixed
<sup>3</sup>https://stability.ai/blog/stable-diffusion-reimagine
length  $T$ . The diffusion process progressively introduces Gaussian noise to the original data  $(\mathbf{x}_0)$  until the signal becomes corrupted  $(\mathbf{x}_T)$ . During DMs training, the reverse process is learned, which operates in the opposite direction of the diffusion process. The reverse process can be viewed as a denoising procedure, moving from  $\mathbf{x}_t$  to  $\mathbf{x}_{t-1}$  at each step. After multiple denoising steps, the model obtains instances that closely resemble the real data.
Stable Diffusion [4] is built on the Latent Diffusion Model (LDM) [4]. LDM [4] proposed to do diffusion process in a latent space rather than the usual pixel space, significantly reducing the training and inference cost of the diffusion model. The authors proposed to utilize a VAE compression to get the latent code  $\mathbf{z}_0$ , which is  $\mathbf{x}_0$  above. Diffusion process will build on the latents. A U-Net architecture [17] with timestep and text conditions would do the reverse. The text prompt is injected into the model with cross-attention layers. We denote  $\epsilon_{\theta}(\mathbf{z}_t, c_{txt}(p_{txt}), t)$  as the output of the U-Net, which is the predicted denoising result.  $p_{txt}$  is the textual prompt and  $c_{txt}(p_{txt})$  is the prompt embedding from the text encoder.  $t$  is the timestep. The training objective of DMs is as followed:
$$
\mathbb {E} _ {\epsilon , \mathbf {z}, p _ {t x t}, t} \left[ \| \epsilon - \epsilon_ {\theta} (\mathbf {z} _ {t}, c _ {t x t} (p _ {t x t}), t) \| _ {2} ^ {2} \right], \tag {1}
$$
where  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  is the noise used to corrupt clean latent variables. During the generation, the latent  $\mathbf{z}_t$ , which starts at a random Gaussian noise  $\mathbf{z}_T$ , will recursively go through a denoising operation until  $\mathbf{z}_0$  is sampled. Finally,  $\mathbf{z}_0$  is reconstructed to an image by the VAE.
2.2 CLIP Model
The CLIP model [15] has garnered significant acclaim as a groundbreaking zero-shot model in recent years. Its training process demands optimizing a contrastive loss function using extensive 400-million pairs of images and corresponding text descriptions. Through the meticulous training, the model has been able to achieve unparalleled capabilities in zero-shot classification and image-text retrieval.
The model comprises an image encoder  $\mathrm{CLIP}_i(\cdot)$ , a text encoder  $\mathrm{CLIP}_t(\cdot)$ , a visual projection layer  $W_{i}$ , and a textual projection layer  $W_{t}$ . The image encoder encodes an input image  $x$  into a visual embedding  $\mathbf{f}_{img}$  derived from a special class-token. By applying the visual projection layer, the embedding is projected into the CLIP visual embedding  $\mathbf{f}_{img}^{c}$ . Similarly, the text encoder processes the input text, yielding a sequence of output embeddings  $\mathbf{f}_{txt}$  for each text token and a start token and end-of-sentence (EOS) token. The embedding of the EOS token  $\mathbf{f}_{txt}^{t,\langle\text{eos}\rangle}$ , where  $t$  denotes the length of the sentence, is projected into the CLIP textual embedding  $\mathbf{f}_{txt}^{c}$  through  $W_{t}$ . Formally,
$$
\mathbf {f} _ {i m g} = \operatorname {C L I P} _ {i} (x), \quad \mathbf {f} _ {i m g} ^ {c} = W _ {i} \cdot \mathbf {f} _ {i m g}, \tag {2}
$$
$$
\mathbf {f} _ {t x t} = \operatorname {C L I P} _ {t} (s), \quad \mathbf {f} _ {t x t} ^ {c} = W _ {t} \cdot \mathbf {f} _ {t x t} ^ {t, \langle e o s \rangle}. \tag {3}
$$
The training objective of CLIP is to maximize the cosine similarity between  $\mathbf{f}_{txt}^{c}$  and  $\mathbf{f}_{img}^{c}$  for matched sentence-image pair while minimizing this similarity for unmatched pairs. For the simplicity of discussion, we denote the space spanned by  $\mathbf{f}_{txt}$  as  $\mathcal{T}$ -space and the space spanned by  $\mathbf{f}_{*}^{c}$  as  $\mathcal{C}$ -space.
The CLIP text encoder [15] is directly used in Stable Diffusion to encode text prompts. It encodes a text prompt as a sequence of embeddings:
$$
\mathbf {f} _ {t x t} := \left[ \mathbf {f} _ {t x t} ^ {0, \langle s o s \rangle}, \mathbf {f} _ {t x t} ^ {1, w _ {0}}, \dots , \mathbf {f} _ {t x t} ^ {t, \langle e o s \rangle}, \dots , \mathbf {f} _ {t x t} ^ {7 6, \langle e o s \rangle} \right] \tag {4}
$$
where  $\mathbf{f}_{txt}^{0,\langle sos\rangle}$ ,  $\mathbf{f}_{txt}^{i,w}$  and  $\mathbf{f}_{txt}^{t,\langle eos\rangle}$  denote the embeddings corresponding to the start-token, the  $i$ -th word token and end-token, respectively. From  $\mathbf{f}_{txt}^{t+1,\langle eos\rangle}$  to  $\mathbf{f}_{txt}^{76,\langle eos\rangle}$  are padded tokens.
2.3 Image Variation & Customized Generation
Image Variation. Image variation aims to generate images similar to the reference image but not identical. SD-R [4] is proposed to address this problem, which builds upon the Stable-unCLIP model<sup>4</sup>. The authors fine-tuned the Stable Diffusion model [4] to align with the CLIP visual embedding. In SD-R [4], images can be directly input into the diffusion model through CLIP image encoder. Since the original Stable Diffusion is conditioned on text only, an expensive fine-tuning is required to
<sup>4</sup>https://huggingface.co/stabilityai/stable-diffusion-2-1-unclip
accommodate this new input. The process took 200,000 GPU-hours on NVIDIA A100-40GB GPU while our approach only requires 1 GPU-hour on NVIDIA A5000-24GB GPU $^5$ .
Customized Generation. Recent works such as DreamBooth [11], Textual Inversion [14], and Custom Diffusion [13] focus on learning a special text prompt to feature specific objects or persons from the reference images. For instance, given several photos of a particular cat, these methods use a special-token " $\langle s \rangle$  cat" to represent the concept and incorporate it with the text prompt. DreamBooth [11] and Custom Diffusion [13] also perform simultaneous fine-tuning of diffusion model parameters. However, the fine-tuning process is still somewhat time-consuming, with Custom Diffusion [13] requiring nearly 6 minutes on 2 NVIDIA A100 GPUs. In contrast, our fast update SD-IPC only needs 1 minute on 2 A5000 GPUs.
Image Editing. Stable Diffusion [4] is commonly used for image editing tasks. Prompt-to-Prompt [9] and Plug-and-Play [10] utilize attention map as a bridge to enable concept and style manipulation. Null-Text Inversion [20] and Pix2Pix-Zero [21] relies on inversion-based methods. InstructPix2Pix [22] creates a dataset of paired edited images and fine-tunes Stable Diffusion [4] as an editing model. It's important to highlight that while our primary focus in developing this method was to enhance image variation, it can also be employed to generate images based on prompts that combine both textual instructions and accompanying images. Notably, unlike existing approaches that frequently reproduce the layout of the original image in the generated output, our method operates without being confined to replicating the exact layout of the source image.
3.1 Image-to-Prompt Conversion via Projecting CLIP embedding
By design, the image generation process in the stable diffusion model should be influenced by embeddings of all tokens in a prompt, like Eq. (4). Interestingly, we have discovered that masking word tokens, by setting their attention weights to 0 except for the start-/end-token, does not have a negative impact on the quality of generated images. This finding is visually illustrated in Figure 2.
On another note, the training objective of CLIP [15] is to match the embeddings  $\mathbf{f}_{img}^{c}$  and  $\mathbf{f}_{txt}^{c}$ , with  $\mathbf{f}_{txt}^{c}$  being essentially a projection of  $\mathbf{f}_{txt}^{t,\langle es\rangle}$ . This inherent relationship, coupled with the aforementioned observation, leads us to establish a connection between  $\mathbf{f}_{img}^{c}$  and  $\mathbf{f}_{txt}^{t,\langle es\rangle}$ , effectively converting the visual embedding to a prompt embedding.
Formally, we assume that after training, CLIP model can induce high cosine similarity between the  $\mathbf{f}_{img}^{c}$  and  $\mathbf{f}_{txt}$  and we can further make the following approximation:
$$
\frac {\mathbf {f} _ {i m g} ^ {c}}{\| \mathbf {f} _ {i m g} ^ {c} \|} \approx \frac {\mathbf {f} _ {t x t} ^ {c}}{\| \mathbf {f} _ {t x t} ^ {c} \|}, \text {w i t h} \mathbf {f} _ {t x t} ^ {c} = W _ {t} \mathbf {f} _ {t x t} ^ {t, \langle e o s \rangle}. \tag {5}
$$
By using Moore-Penrose pseudo-inverse [23] on  $W_{t}^{6}$ , we obtain an estimate of  $\mathbf{f}_{txt}^{t,\langle es\rangle}$  from  $\mathbf{f}_{img}^{c}$ :
$$
\mathbf {f} _ {t x t} ^ {t, \langle e o s \rangle} \approx \frac {\left\| \mathbf {f} _ {t x t} ^ {c} \right\|}{\left\| \mathbf {f} _ {i m g} ^ {c} \right\|} W _ {t} ^ {+} \mathbf {f} _ {i m g} ^ {c} := \mathbf {f} _ {t x t} ^ {c n v r t}, \text {w h e r e ,} W _ {t} ^ {+} = \left(W _ {t} ^ {\top} W _ {t}\right) ^ {- 1} W _ {t} ^ {\top}, \tag {6}
$$
where we empirically observe  $\| \mathbf{f}_{txt}^c\|$  can be well approximated by a constant, e.g.,  $\| \mathbf{f}_{txt}^c\| = 27$  and  $W_{t}$  can be obtained from the pretrained CLIP model [15]. We denote the converted embedding as  $\mathbf{f}_{txt}^{convrt}$  and use it to assemble a pseudo-prompt sequence with the following format:
$$
\tilde {\mathbf {f}} _ {t x t} := \left[ \mathbf {f} _ {t x t} ^ {0, \langle s o s \rangle}, \mathbf {f} _ {t x t} ^ {1, c n v r t}, \dots , \mathbf {f} _ {t x t} ^ {7 6, c n v r t} \right], \tag {7}
$$
where  $\mathbf{f}_{txt}^{1,cnvt} = \dots = \mathbf{f}_{txt}^{76,cnvt} = \mathbf{f}_{txt}^{cnvt}$ . In other words, we replace all word-tokens, pad-tokens and end-token in Eq. (4) with the converted  $\mathbf{f}_{txt}^{cnvt}$ , based on the fact that  $\mathbf{f}_{txt}^{cnvt}$  is an approximation of  $\mathbf{f}_{txt}^{t,\langle es\rangle}$  and masking word-tokens does not influence the generation<sup>7</sup>.
<sup>5</sup>FP16 computing performance of A100 is 77.97 TFLOPS verse 27.77 TFLOPS of A5000.
<sup>6</sup>We use singular value decomposition (SVD) to form the pseudo-inverse, singular values which are smaller than 0.3 will be treated as 0.
Here we keep the pad-tokens, so the index of token is from 0 to 76, where the maximum length of CLIP text input is 77, even they are the same  $\mathbf{f}_{\text{txt}}^{\text{cnvrt}}$ , they would contribute to the attention weights in cross-attention, to decrease the weights of start-token.
Figure 3: Image variation results on MSCOCO [24]. SD w/ Text [4] is generation from the ground-truth text prompts that are not available for variation methods such as SD-R and SD-IPC. SD-IPC is our method, notice that SD-IPC does not need any training compared to SD-R [4].
This approximation allows immediate conversion of an image to a text prompt by directly mapping it to an (approximately) equivalent prompt. We refer to this method as Stable Diffusion Image-to-Prompt Conversion (SD-IPC). Experimental results in Fig. 3 demonstrate that SD-IPC effectively captures the semantic information present in the reference image and enables image variation.
Furthermore, we have identified a simple yet effective approach to combine both the text prompt and the converted image prompt within our framework. To achieve this, we perform a weighted average of the two embeddings. Formally, the process can be described as follows:
$$
\mathbf {f} _ {t x t} ^ {c o m b} = \mathbf {f} _ {t x t} ^ {c n v r t} + \alpha \cdot \mathbf {f} _ {t x t} ^ {t, \langle e o s \rangle}, \quad \tilde {\mathbf {f}} _ {t x t} ^ {e d i t} = \left[ \mathbf {f} _ {t x t} ^ {0, \langle s o s \rangle}, \mathbf {f} _ {t x t} ^ {1, w _ {0}}, \dots , \mathbf {f} _ {t x t} ^ {t, c o m b}, \dots , \mathbf {f} _ {t x t} ^ {7 6, c o m b} \right], \qquad (8)
$$
where  $\mathbf{f}_{text}^{i,comb} = \mathbf{f}_{text}^{comb}$  is the combined-token embedding and  $\alpha$  is a hyperparameter to control the expression of editing text. Notice that the editing word-token  $\mathbf{f}_{text}^{i,w}$  are also in the embedding sequence. Conditioning on  $\tilde{\mathbf{f}}_{text}^{edit}$  could generate images that match both the visual and textual conditions. We report some editing results in Appendix D.2.
3.2 Fine-tuning with Image-to-Prompt Conversion
While the aforementioned SD-IPC method demonstrates reasonable performance, it still faces challenges when it comes to real-world applications due to two main reasons. Firstly, the conversion process in SD-IPC relies on approximations, which may not always yield optimal results. Secondly, determining the exact topic or theme of an image introduces ambiguity. As the saying goes, "an image is worth a thousand words", but precisely which words? The same reference image can be interpreted differently based on its objects, scenes, styles, or the identities of people depicted within. Therefore, it becomes crucial to have a method that allows control of the content we wish to preserve and convert into the prompt. To address these concerns and cater to the needs, we propose a partial fine-tuning approach for the CLIP converter derived from Sec. 3.1.
In proposed approach, we focus on fine-tuning two specific types of parameters. Firstly, we address the optimization of the projection matrix within the cross-attention layer of the U-Net in Stable Diffusion [4]. This aspect aligns with the methodology employed in Custom Diffusion [13]. Furthermore, we incorporate deep prompt tuning [25] into the transformer of the CLIP image encoder. Deep prompt tuning [25] introduces learnable tokens within all layers of the transformer while keeping the weights of other components fixed. More details can be found in Appendix A.
Figure 4: Fine-tuned SD-IPC, denoted as SD-IPC-FT, can enhance the image-to-prompt conversion quality.
The parameters can be learned by using the following loss:
$$
\mathbb {E} _ {\epsilon , \mathbf {z}, x _ {\text {r e f}}, t} \left[ \| \epsilon - \epsilon_ {\theta} (\mathbf {z} _ {t}, c _ {i m g} (x _ {\text {r e f}}), t) \| ^ {2} \right] + \mathbb {E} _ {\epsilon , \mathbf {z}, p _ {t x t}, t} \left[ \| \epsilon - \epsilon_ {\theta} (\mathbf {z} _ {t}, c _ {t x t} (p _ {t x t}), t) \| ^ {2} \right], \tag {9}
$$
Here the first term is  $\mathcal{L}_{cnvrt}$ , which is fine-tuning with the image-to-prompt input, and the second term is  $\mathcal{L}_{text}$ , which is the original text-to-image training loss, we utilize this as a regularization to keep the text-to-image generation. In the proposed approach,  $c_{img}(\cdot)$  refers to the image-to-prompt conversion derived from SD-IPC. It encompasses the CLIP image transformer augmented with the newly-introduced learnable prompts in deep prompting and the fixed inverse matrix derived from Eq. (6). During tuning, the inverse projection matrix remains unchanged.  $\mathbf{z}_t$  represents the latent representation of the target image  $x_{\mathrm{target}}$  at time step  $t$ . The objective function aims to encourage the image-to-prompt conversion to extract information from  $x_{\mathrm{ref}}$  that facilitates the recovery of  $x_{\mathrm{target}}$ . There are two possible choices for  $x_{\mathrm{target}}$ : (1)  $x_{\mathrm{target}}$  can be selected to be the same as  $x_{\mathrm{ref}}$ ; (2)  $x_{\mathrm{target}}$  and  $x_{\mathrm{ref}}$  can be different images, but with a shared visual concept that we intend to extract as the prompt. This usually poses stronger supervision to encourage the converter to extract information related to the shared theme. The schematic representation of this scheme is illustrated in Appendix C.2.
We use images randomly sampled from ImageNet [26], CelebA-HQ [27], and Places365 [28] dataset to encourage the model extract object, identity, and scene information, respectively. Experiments show that merely 100 images and 1 GPU-hour of training are sufficient for achieving satisfied results thanks to the good initialization provided by SD-IPC. We call this approach SD-IPC-FT, the results are shown in Fig. 4. Some editing examples are listed in Fig. 5, Fig. 6, and Appendix D.4.
3.3 Fast Update for Customized Generation
Existing methods, such as DreamBooth [11] and Custom Diffusion [13], suggest that partially fine-tuning the model on given concept images before generation can be an effective way to synthesized images with customized visual concepts, e.g., people with the same identity. Our approach can also benefit from this scheme by performing such an online update with SD-IPC. This can be achieved by simply replacing the training images in SD-IPC-FT with reference images and use  $\mathcal{L}_{\text {convrt }}$  only. We call this method SD-IPC-CT (CT stands for customized concept tuning). Interestingly, we find that our method can generate customized images with much fewer updates. As a comparison, SD-IPC-CT only takes 30-iteration updates with around 1 minute on 2 A5000 GPUs while the Custom Diffusion [13] needs 250 iterations (6 minutes on 2 A100 GPUs). We report customized generation in Fig. 7.
Input Images
SD-IPC-FT (Ours)
SD-R
Figure 5: Image editing result with SD-IPC-FT trained with 100 images sampled from ImageNet [26]. SD-IPC-FT shows better editing performance than that of SD-R [4].
Without Editing
"On the beach"
"Under the starry sky"
y sky"
Figure 6: Image editing result with SD-IPC-FT trained with 100 images sampled from CelebA-HQ [27]. SD-IPC-FT shows better editing performance than that of SD-R [4].
"Wearing glasses"
"Wearing a hat"