SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets & Evaluations. In previous discussion, we propose three different fine-tuning schemes, using ImageNet [26] for object understanding, CelebA-HQ [27] for portrait understanding, and Places365 [28] for scene understanding. The specific training classes or identities we have selected for each dataset can be found in Appendix B. Each dataset includes 100 images, the test images are non-overlap with the training classes. In order to enable customized generation, we choose two objects and two identities as examples, each accompanied by five images. To assess the quality and semantic consistency of our generated outputs, we measure FID-Score [29] and CLIP-Score [30].
Architecture & Hyperparameter. We utilize Stable Diffusion v1.4 $^{8}$  and CLIP ViT-L/14 $^{9}$  models in this paper. We compare our method with the larger Stable-unCLIP-small model $^{10}$  using CLIP ViT-H/14 and a 1,024-dimensional attention feature. Our method uses DDIM [19] for sampling, while
<sup>8</sup>https://huggingface.co/CompVis/stable-diffusion-v1-4
<sup>9</sup>https://huggingface.co/openai/clip-vit-large-patch14
<sup>10</sup>https://huggingface.co/stabilityai/stable-diffusion-2-1-unclip-small
Figure 7: Customized generation examples. The images at left are training images, they are all from one concept or one identity. We compared our SD-IPC-CT with Custom Diffusion [13], notice that both results are trained by 5 reference images with merely 30 iterations.
Figure 8: Results of DreamBooth [11] benchmark, the training images are listed at top-left corner.
Stable-unCLIP uses PNDM [31], both with 50 sampling steps. SD-IPC-FT is trained for 100, 50, and 100 epochs on ImageNet [26], CelebA-HQ [27], and Places365 [28], respectively. The learning rates for all datasets are  $1e - 5$  with cosine decay. Customized generation has a constant learning rate of  $5e - 6$  for 30-iteration updates. Training is conducted on 2 A5000 GPUs. The editing  $\alpha$  is set to 0.9.
4.2 Image Variation Results
SD-IPC. We evaluate image variation on MSCOCO [24] using all 5,000 images in the 2017-split validation set. Fig. 3 compares text-based generation, SD-R [4], and our SD-IPC. Both SD-R [4]
Table 1: Accuracy and Retrieval recalls (\%) of original CLIP [15] ( $\mathcal{C}$ -space) and the inverse matrix transfer ( $\mathcal{T}$ -space). Acc@ $k$  is the top-  $k$  accuracy of ImageNet [26] val-set, TR@ $k$  and IR@ $k$  are top-  $k$  text and image retrieval recalls. This is a surprising result that there is almost no performance decline.
Emb. Space	Acc@1	Acc@5	TR@1	TR@5	IR@1	IR@5
C-space	71.41	91.78	74.58	92.98	55.54	82.39
T-space	69.48	90.62	71.62	92.06	54.82	82.20
Table 2: FID-Score [29] and CLIP-Score [30]  $(\%)$  of the generation results. SD w/ Text [4] means the generation from ground-truth text.
Methods	FID	CLIP-Score
SD w/ Text [4]	23.65	70.15
SD-R [4]	19.86	82.59
SD-IPC (Ours)	24.78	73.57
and our method demonstrate image variation, but our approach utilizes an inverse matrix without training. The FID-Score [29] and CLIP-Score [30] are reported in Tab. 2. Our SD-IPC maintains image quality similar to text-based generation (FID-Score: 24.78 vs. 23.65, CLIP-Score: 73.57 vs. 70.15). Note that SD-R [4] achieves better results due to its advanced backbone model.
Finetuning. In Fig. 4, it is observed that SD-IPC may exhibit inconsistency, for example, the input "teddybear" generates a picture of "kid". This inconsistency could be attributed to fact that SD-IPC fails to discern the semantically related concepts "kid" and "teddybear". However, this issue can be rectified through fine-tuning, as demonstrated in Fig. 4, SD-IPC-FT achieves improved generation quality. Moreover, we find that the editing capability of SD-IPC-FT, as illustrated in Fig. 5, Fig. 6, and Appendix D.4, surpasses that of SD-R [4], and fine-tuning does not impact the editing performance. We incorporated a quantitative experiment to validate the superior editing performance of SD-IPC-FT in comparison to SD-R [4]. We utilize images from DreamBooth [11] benchmark and randomly select an editing text for each test image. Editing performance is evaluated using CLIP-T score, the quantitative results are presented in Tab. 4. As seen, our method achieves a higher CLIP-T score than SD-R [4]. Furthermore, we include the training-free SD-IPC for comparison, revealing even SD-IPC slightly outperforms SD-R [4]. The role of different target images used in the fine-tuning stage is outlined in Appendix C.2, which showcases how the choice of target images influences the generation result. Additional generation results are presented in Appendix D.
4.3 Customized Generation Results
In Fig. 7, we compare our SD-IPC-CT with Custom Diffusion [13] in terms of customized generation. We evaluate customization for two identities ("Obama" and "Chastain") and two objects ("cat" and "tortoise"). The training process only requires 5 images and 30 iterations. The results in Fig. 7 reveal that Custom Diffusion [13] struggles to learn the details of the concept with such limited updates, while our SD-IPC-CT demonstrates impressive performance, particularly in the "young boy/girl" editing. However, for rare instances like the "tortoise" example, both methods do not perform well. For quantitative results, we followed DreamBooth [11]. We used DINO and CLIP-I for subject fidelity and CLIP-T for editing performance. Comparing with DreamBooth [11], Textual Inversion [14], and Custom Diffusion [13], the results are in Tab. 3, Fig. 8, and Appendix D.5. DreamBooth [11] excels in DINO/CLIP-I scores but lags in CLIP-T, indicating limited editing performance, evident in visually similar outputs to training images. Textual Inversion [14] and Custom Diffusion [13] have strong CLIP-T but weak DINO/CLIP-I scores, indicating challenges in preserving subject details. Our SD-IPC-CT method strikes a balance between subject identity preservation and editing performance.
Table 3: Results of DreamBooth [11] benchmark and the comparison with common methods.
Method	DNIO	CLIP-I	CLIP-T	Comments
DreamBooth [11]	60.11	77.78	25.81	Good Identity, Weak Editing
Textual Inversion [14]	25.11	62.44	29.53	Weak Identity, Good Editing
Custom Diffusion [13]	39.67	68.37	30.90	Weak Identity, Good Editing
SD-IPC-CT (Ours)	50.25	74.59	28.14	Good Identity, Good Editing
Table 4: Superior editing performance of SD-IPC-FT.
Method	CLIP-T
SD-IPC	26.84
SD-IPC-FT	28.69
SD-R [4]	26.01
Effectiveness of Inverse Projection Matrix. To evaluate the effectiveness of the inverse projection matrix in SD-IPC, we introduce a fully-connected layer instead of the inverse matrix in the image-to-prompt conversion, referred to as SD-IPC-FC and SD-IPC-FC(I), where (I) means to initialize with
Figure 9: Image variation results of different fine-tuning settings. SD-IPC-FT (C) means only training CLIP prompts, SD-IPC-FT (U) means only training U-Net cross-attention layers, SD-IPC-FC (I) means initializing the FC-layer with the inverse matrix.
Figure 10: Effectiveness of using Eq. 6 for SD-IPC-FT.
our inverse projection matrix. We train the FC models with the same training data as SD-IPC-FT. However, the results in Fig. 10 indicate SD-IPC-FC suffers from overfitting. SD-IPC-FC(I) slightly alleviates the overfitting but still gets inferior results, shown in Fig. 9. This highlights that our SD-IPC-FT benefits from the good initialization of SD-IPC and preserves knowledge in CLIP [15].
Prompt Learning & U-Net Fine-tuning. We perform quantitative tests on (text-edited) image variation for the comprehensive ablation studies following the testing in Sec. 4.2. For text-edited variation, we use the editing text as the prompt, such as "A [Class Name] with a mountain in the background." We present the results of individual fine-tuning for two components: SD-IPC-FT (C) for CLIP and SD-IPC-FT (U) for the U-Net. Qualitative results are available in Fig. 9, while quantitative results are provided in Tab. 5 and Tab. 6. It demonstrates that fine-tuning each component contributes to model adaptation, with the best performance achieved when simultaneously fine-tuning both two parts. Some editing comparisons are in Appendix C.3.
Additionally, we investigate the influence of the editing parameter  $\alpha$  in Appendix C.1.
Table 5: Results of image variation with different fine-tuning settings.
Method	DNIO	CLIP-I	CLIP-T
SD-IPC	44.60	77.44	25.47
SD-IPC-FT (C)	49.11	76.51	25.82
SD-IPC-FT (U)	48.53	79.06	26.17
SD-IPC-FT	52.03	79.59	25.90
Table 6: Results of text-edited image variation with different fine-tuning settings.
Method	DNIO	CLIP-I	CLIP-T
SD-IPC	31.09	68.66	26.84
SD-IPC-FT (C)	29.10	67.03	27.99
SD-IPC-FT (U)	35.21	69.99	28.56
SD-IPC-FT	40.28	71.97	28.69
4.5 Limitations & Feature Directions
While SD-IPC offers an alternative to SD-R [4], there are remaining challenges. Firstly, the editing text must be contextually appropriate, as using "on the beach" to edit a portrait may result in a person being on the beach but lacking facial features. Secondly, SD-IPC currently does not support multiple image inputs. Another future study is to extend our method to generate a sequence of images with consistency. Appendix E shows some potential of our method in this direction.
This paper reveals that the CLIP model [15] serves as an image-to-prompt converter, enabling image variation in text-to-image Stable Diffusion [4] without extensive training. This finding enhances our understanding of the CLIP embedding space, demonstrating that a simple inverse matrix can convert visual embeddings into textual prompts. Leveraging this image-to-prompt conversion, our SD-IPC methods achieve impressive image variation and editing capabilities, while also enabling fast adaptation for customized generation. Experimental results also show the potential of our method in more multi-modal tasks. We anticipate that this study will inspire future research exploring the image-to-prompt pathway in CLIP-based or LDM-based models.
This work was partly supported by the China Scholarship Council under Grant 202006960047 and partly by the National Natural Science Foundation of China (No.62173265). Lingqiao Liu is supported by Centre of Augmented Reasoning.
[20] Mokady, R., A. Hertz, K. Aberman, et al. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038-6047. 2023.
[21] Parmar, G., K. Kumar Singh, R. Zhang, et al. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11. 2023.
[22] Brooks, T., A. Holynski, A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402. 2023.
[23] Penrose, R. A generalized inverse for matrices. In Mathematical proceedings of the Cambridge philosophical society, vol. 51, pages 406-413. Cambridge University Press, 1955.
[24] Lin, T.-Y., M. Maire, S. Belongie, et al. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[25] Jia, M., L. Tang, B.-C. Chen, et al. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII, pages 709-727. Springer, 2022.
[26] Deng, J., W. Dong, R. Socher, et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. IEEE, 2009.
[27] Karras, T., T. Aila, S. Laine, et al. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations. 2018.
[28] Zhou, B., A. Lapedriza, A. Khosla, et al. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1464, 2017.
[29] Heusel, M., H. Ramsauer, T. Unterthiner, et al. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[30] Hessel, J., A. Holtzman, M. Forbes, et al. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528. 2021.
[31] Liu, L., Y. Ren, Z. Lin, et al. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations. 2022.