SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$V_{8}^{*}$  and  $V_{1}^{*}$  enjoying a day at an amusement park
Figure 6: Numerical analysis in terms of the prompt and identity similarity on four prompt types.
Methods	Prompt↑	Identity↑	Detect↑
w/o celeb basis	0.1386	0.2528	69.28%
w/ 350 names	0.2214	0.2023	69.28%
w/o filter	0.1939	0.2037	80.62%
w flatten	0.2026	0.1873	80.39%
p = 64	0.2247	0.1061	76.47%
p = 256	0.1812	0.0656	60.13%
p = 768	0.1380	0.0836	47.06%
w/o F	0.1914	0.1896	55.56%
w/o aug.	0.2083	0.1931	75.16%
Ours (single)	0.2234	0.2186	81.05%
Ours (joint)	0.2545	0.2072	84.78%
Table 2: Ablation studies.
$V_{5}^{*}$
age-, pose pose pose
Figure 7: Our method is not only robust to large age, gender, head pose, and light variants in the inputs, but also has good ability of controlling age, gender, and head pose in the outputs.
reconstruction
age+
pose
forgotten. Although dreambooth [10] and custom diffusion [8] can successfully generate the human and the concept, the generated identities are not the same as the target image.
Besides visual quality, we also perform the numerical comparison between the proposed method and baselines in Tab. 1. From the table, regardless of the over-fitted Textual-Inversion (highest identity and detect scores but lowest prompt score), our method shows a much better performance in terms of identity similarity and the face detection rate and achieves similar text-prompt alignment as Custom Diffusion [8]. We also plot the generated results on four detailed types in Fig. 6, where the proposed method shows the best trade-off. Notice that, the proposed method only contains very few learning-able parameters and optimizes faster.
Moreover, since identity similarity is very subjective, we generate 200 images from different identities to form a user study. In detail, we invite 100 users to rank the generated images from one (worst) to five (best) in terms of visual quality, prompt alignment, and identities, getting 60k opinions in total. The results are also shown in Tab. 1, where the users favor our proposed method.
4.3 Results on Age, Gender, Head Pose, and Expression
We qualitatively evaluate the robustness of our method to input age, gender, head pose, and light variants in Fig. 7. Besides, modifying the text prompts can edit the age, gender, and pose of the results. We further provide the expression controlling results in the appendix, along with the comparisons with Face Composer [51]. Thanks to the identity disentangling ability of the face recognition [14] network, the learned representations are highly relevant to the input identities while irrelevant and insensitive to other attributes like age, pose, expression, etc. Furthermore, the learned representation as an interpolation of celeb basis inherits the good characteristic of those "recognized" celeb names, which corresponds to large amount of image-text paired samples [3] used for pretraining Stable Diffusion [1].
Figure 8: Ablation studies on building celeb basis.
(a) Face encoder.
(b) Data augmentation
(c) Single v.s. joint optimization
Figure 9: Ablation studies on coefficient optimization.
To further evaluate the sub-modules of our proposed method in both building celeb basis (Sec. 4.4.1) and the proposed new personalization method (Sec. 4.4.2), we start from the default settings ('Ours (single)' in Tab. 2) of our method, conducting the ablation study by separately removing each submodule or using a different setting as follows. Due to the space limitation, we give more visual results in the appendix.
4.4.1 Ablation Studies on Celeb Basis
# of names in celeb basis. We evaluate the influence of the names to build a celeb basis. In extreme cases, if there is no name and we directly learn the embedding from the face encoder  $F$  (w/o celeb basis), the model is overfitted to the input image and can not perform the concept combination. With fewer celeb names (w/ 350 names), the generated quality is not good as ours baseline (single) as in Fig. 8a. Besides, the quality of the celeb basis is also important, if we do not filter the names (w/o filter), the performance will also decrease as in Tab. 2.
Flatten basis v.s. first and last name basis. In the main method, we introduce our celeb basis as the first and last name basis since each name embedding does not have the same length. We thus involve a more naive way by flattening all the embeddings to build the basis (w/ flatten). As shown in Fig. 8b and Tab. 2, the generated images of our first and last name basis understand prompts better.
Choice of reduction dimension  $p$ . We also evaluate the influence of the number of coefficients  $p$ . Considering the 768 dimensions of the CLIP text embedding, we vary  $p$  ranging in {64, 256, 512, 768}. Note that  $p = 768$  can be also considered as 'w/o PCA' since the dimension of text embedding is 768. As shown in Tab. 2, the best result is obtained from the baseline choice  $(p = 512)$  and we show the differences in the generated images in the appendix.
4.4.2 Ablation Studies on Coefficients Optimization
W/o face recognition encoder  $F$ . Naively, we can optimize the coefficients  $A_{1}, A_{2}$  of the celeb embeddings from back-propagation directly. However, we find the search space is still large to get satisfied results as shown in Fig. 9a and Tab. 2 (w/o  $F$ ). So we seek help from the pretrained face encoder, which has more discriminative features on the face.
W/o data augmentation. Since there is only one single image as the tuning-able sample, we perform some data augmentations as introduced in Sec. 3.2. However, if we remove these augmentations, the generated face becomes vague and unnatural as shown in Fig. 9b, and the identity of the generated samples is also decreased as shown in Tab. 2.
Input Car
traveling through space
in a comic book
in snow
Input Cat
Figure 10: Through a simple replacement on the encoder and text basis, our method can be extended to more general classes, e.g. car and cat.
Manga drawing
Pointlism painting
in Christmas
Single v.s. joint training. Our method supports joint training of multiple identities using a single MLP, we evaluate the differences between single training/joint testing and joint training/joint testing. As shown in Tab. 2 and Fig. 9c, although training individually can also perform some interactions between the two-person, training the images jointly improve the robustness and reduces the overfitting risk compared with single training, resulting in slightly better results.
We propose a new method to personalize the pre-trained text-to-image model on a specific kind of concept, i.e. the human being, with simply single-shot tuning. Our approach is enabled by defining a basis in the domain of the known celebrity names' embeddings. Then, we can map the facial features from the pre-trained face recognition encoder to reconstruct the coefficients of the new identity. Compared with the previous concept injection method, our method shows stronger concept combination abilities, e.g. better identity preservation, can be trained on various identities at once, and can produce results where the newly-added humans interact with each other. Besides, the proposed method only requires 1024 parameters for each person and can be optimized in under 3 minutes, which is also much more efficient than previous methods.
Limitations. Although our method can successfully generate the images of new identities, it still occurs some limitations. First, the human faces generated by original stable diffusion [1] intrinsically contain some artifacts, causing somewhat unnatural performance of the proposed method (see our appendix). It might be solved by a more powerful pre-trained text-to-image model ( $e.g$ . Imagen [5], IF [52]) since they can generate better facial details. Second, we only focus on human beings currently. It is also interesting to build the basis of other species,  $e.g$ . cars, and cats. To this end, we provide some rough experiments (whose implementation details are introduced in the appendix) of extending our method to cars and cats, as shown in Fig. 10. The results show that our method can also generate text-aligned images with good quality without bells and whistles. But the simple try could be improved through using a stronger encoder or crawling more text to construct basis. Third, although the text alignment scores evaluate the quantitative performance, for the editing ability of age, expression, and pose, we do not have a metric result yet. As supplement, we will use facial feature extractors to quantitatively compare the human identity personalized tasks in the future. Fourth, similar to the inference-only method FastComposer [51], we have also tried to train a model on 50k identities (20 faces per identity without text labels) as an encoder-based method. However, it is hard to converge in a short GPU time within a batch size of eight. We think it might be because we need much more data and computing resources to train a unified or general facial model and we will explore it in the future.
Ethics Consideration and Broader Impacts. We propose a new method for model personalization on the human face, which might be used as a tool for deepfake generation since we can generate not only the human face but also interactions between people. However, this prevalent issue is not limited to this approach alone, as it also exists in other generative models and content manipulation techniques. Besides, our personalization person generation method can also ablate the abilities of the erasing concept methods [53] and other deep fake detection methods [54].
This work was supported in part by the National Natural Science Foundation of China under Grant 61976231 and in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2023A1515012853.
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2021.
[2] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pages 4401-4410, 2019.
[3] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5B: An open large-scale dataset for training next generation image-text models. arXiv:2210.08402, 2022.
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, 2021.
[5] Chitwan Sahara, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NIPS, pages 36479-36494, 2022.
[6] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv:2204.06125, 2022.
[7] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv:2208.01618, 2022.
[8] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In CVPR, pages 1931-1941, 2023.
[9] Ziyi Dong, Pengxu Wei, and Liang Lin. DreamArtist: Towards controllable one-shot text-to-image generation via contrastive prompt-tuning. arXiv:2211.11337, 2022.
[10] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv:2208.12242, 2022.
[11] Rinon Gal, Moab Arar, Yuval Atzmon, AmitH. Bermano, Gal Chechik, and Daniel Cohen-Or. Designing an encoder for fast personalization of text-to-image models. arXiv:2302.12228, 2023.
[12] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3D faces. In SIGGRAPH, pages 187-194, 1999.
[13] Karl Pearson. LIII. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, 1901.
[14] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. ArcFace: Additive angular margin loss for deep face recognition. In CVPR, pages 4690-4699, 2019.
[15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
[16] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
[17] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NIPS, 2017.
[18] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873-12883, 2021.
[19] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In NIPS, pages 1–11, 2019.
[20] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, pages 8821-8831, 2021.
[21] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In NIPS, pages 1-10, 2018.
[22] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv:1410.8516, 2014.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NIPS, pages 6840-6851, 2020.
[24] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502, 2020.
[25] Wenhu Chen, Hexiang Hu, Chitwan Sahara, and William W Cohen. Re-imagen: Retrievalaugmented text-to-image generator. arXiv:2209.14491, 2022.
[26] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, pages 18208-18218, 2022.
[27] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. arXiv:2206.02779, 2022.
[28] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint-by-example: Exemplar-based image editing with diffusion models. arXiv:2211.13227, 2022.
[29] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv:2211.09794, 2022.
[30] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. arXiv:2302.03027, 2023.
[31] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv:2303.09535, 2023.
[32] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv:2208.01626, 2022.
[33] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, pages 8110-8119, 2020.
[34] Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or. Mystyle: A personalized generative prior. ACM Transactions on Graphics, 41(6):1–10, 2022.
[35] Daniel Reich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics, 42(1):1-13, 2022.
[36] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2Stylegan: How to embed images into the stylegan latent space? In CVPR, pages 4432-4441, 2019.
A.1 Evaluation on Faces in a Wider Range
Besides evaluating the results on StyleGAN synthetic faces as in the main paper, we compare our method with state-of-the-art methods on a wide range of human face images, including the real word faces, the faces from different races, and the interactions of the same gender. These face images are from VGGFace2 [55] and collected from the Web.
Single Person in an Image. Fig. 11 shows the evaluation results on a single person's personalization for real identities. Concretely, Row 1-2 shows the ability to generate diverse images of different methods. The baselines Textual Inversion [7], DreamBooth [10], and Custom Diffusion [8] can generate identity-consistent results with the single prompt 'A photo of  $V^{*}$ ', but the light condition and skin texture seem not natural. DreamBooth and Custom Diffusion fail to disentangle the background of the target inputs. The stylization results are shown in Row 3-4, where Textual Inversion preserves the target identities but fails to change the style, DreamBooth, and Custom Diffusion generate the images of the corresponding style but lose the identity information, and our method preserves the identity and completes the stylization as given prompts.
Multiple Persons in an Image. We give the comparison of generating multiple-person interaction in Fig. 12. Similar to the single-person results, the proposed method shows a good concept combination ability among different persons (including the newly added and the original celebrities) and can serve as a novel concept to communicate with other humans.
A.2 Additional Results Comparing with Concurrently Work: FastComposer
Different from the methods [7, 10, 8] that the finetuning time cost for each identity is below 30 minutes, FastComposer [51] needs to be pre-trained on a large human dataset, requiring 150k steps with batch size 128 on 8 NVIDIA A6000 (48GB) GPUs. So as its performance is restricted by the dataset as in their limitation. Differently, our method requires only 400 steps with batch size 2 on a single NVIDIA A100 (40GB) GPU. After the pre-training stage, FastComposer can generate images based on the given single image and prompt like other personalized methods.
To further demonstrate the efficacy of our method, we evaluate the qualitative performance of our method compared with FastComposer. Our method outperforms FastComposer on single action controlling, the interaction between two persons, and expression controlling. The detailed comparisons are as follows.
Action of Single Person. We first evaluate the performance of single persons' personalization. As illustrated in Fig. 13 and Fig. 14, we show the synthesized results under six scenarios in each figure.
In Fig. 13, given the prompt 'A photo of  $V_{1}^{**}$ , FastComposer generates an unnatural result. In the remaining scenarios, the light seems very disharmonious in the images synthesized by FastComposer. Besides, in these results, only the upper chest and head can be generated, where the human body and limbs fail to appear, revealing an over-fitting issue of FastComposer.
In Fig. 14, FastComposer meets the same issue. Although the key objects and face identity are almost consistent with the input prompts, e.g. LEGO toys, fighter jet, and yellow jacket, the results from FastComposer seem to be a rude combination of the objects and face. Besides, their methods have a huge issue with concept forget. Some important prompts like 'sweater' and 'motorbike' are ignored by FastComposer.
We also highlight the faces generated by FastComposer share very similar expressions from the original input image, i.e. dilated pupils, wrinkles on the forehead, and mouth shape, which means this method is overfitted on the input image. Differently, our method generates identity- and prompt-consistent results, appearing natural and realistic.
Interaction between Two Persons. Fig. 15 illustrates the interaction and common action of two persons in a single image. Sharing the similar aforementioned over-fitting problem, FastComposer only generates the face part of the human. As for the actions including 'shaking', 'playing', 'sitting', and 'baking', FastComposer fails to generate the correct behaviors, where the faces usually occupy the most part of the images. The problem of invariant expression also occurs in FastComposer.
Textual Inversioin
DreamBooth
A photo of  $V_{1}^{*}$
A photo of  $V_{2}^{*}$
$V_{3}^{*}$  in the style of Archer
$V_{4}^{*}$
$V_{4}^{*}$  by Andy Warhol
$V_{5}^{*}$  is playing basketball
$V_{6}^{*}$
Figure 11: Single person's personalization result for real identities.
$V_{6}^{*}$  in a Chinese vegetable market
Expression Controlling. Considering that the above FastComposer results struggle to generate the different expressions, we conduct a face expression controlling experiment to validate the abilities of our method and FastComposer. We use the following prompts to change the expression of the target input image:
-  $V^{*}$  is smiling with happiness
-  $V^{*}$  is crying with sadness
-  $V^{*}$  has an anger expression
-  $V^{*}$  looks very surprised, looking at a gift box
-  $V^{*}$  is shaking with fear scared by a candle lighting, scared expression
-  $V^{*}$  has a tired expression, headache, uncomfortable, a frown creased her forehead
As shown in Fig. 16, the proposed method can successfully generate different expressions under the text prompts, which shows the advantage of the proposed method.
A.3 Uncurated Results
We show the uncurated results of our method in Fig. 17, which demonstrate that our method can generate stable and text-aligned images without being cherry-picked.
B.1 Celeb Images Generated by Stable Diffusion
In general, the celeb images generated by Stable Diffusion [1] and those realistic ones collected from Google Images seem very similar, but with a little difference on facial details. As seen in Fig. 18, we compare the real celeb images collected from Google Images with the fake ones synthesized by Stable Diffusion. Overall, the real and fake images closely resemble the same individual. However, certain facial features in the fake images may be exaggerated, which can constrain our method's performance and explain why the celebrity profiles in the outcomes generated by both baseline models and our approach do not precisely match their real counterparts.
B.2 Collecting and Filtering Celeb Names
After crawling about 1,500 celebrity names, to filter out the names that have the ability to generate prompt-consistent identities and interacting with other celebs, we feed three types of prompts, i.e. 'A photo of  $V_{\mathrm{c}}$ ', 'Vc is playing the guitar', and 'Vc talks with Barack Obama', to the Stable Diffusion [1] for synthesizing image results, where  $V_{\mathrm{c}}$  indicates the celeb name (e.g. 'Anne Hathaway'). Fig. 19 shows the examples that satisfy the filtering condition and the ones that fail to generate reasonable results. Consequently, 691 names pass the checking, which can be tokenized and encoded into  $m = 691$  celeb embedding groups.
The statistics of these 691 names are shown in Fig. 20. Specifically, we count the frequency of each celeb name in the LAION-2B dataset text samples. LAION-2B is a subset of LAION-5B [3] where Stable Diffusion [1] is pretrained. After sorting the celeb names according their frequency, we obtain Fig. 20, where the x-axis is the index of the name, the y-axis is the appearing time of this name in the dataset. This plot indicates that our collected celeb names correspond to thousands image-text paired samples in average. There are only 25 names whose frequencies are less than 100 and 173 names whose frequencies are larger than 10000. This frequency distribution shows that the distribution of the celeb names is relatively fair.
B.3 Building Celeb Basis based on PCA
We only keep the first and second embeddings of each celeb embedding group, resulting in the first name embedding set  $C_1$  and the second name embedding set  $C_2$ , where  $C_1, C_2 \in \mathbb{R}^{m \times d}$ . Then, for simplicity, we omit the subscript of  $C_k$ , using  $C$  to indicate any one of  $C_1$  and  $C_2$ .