SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ge Yuan $^{1,2,3,4*}$  Xiaodong Cun $^{4}$  Yong Zhang $^{4}$  Maomao Li $^{4\dagger}$  Chenyang Qi $^{5}$  Xintao Wang $^{4}$  Ying Shan $^{4}$  Huicheng Zheng $^{1,2,3\dagger}$
<sup>1</sup> School of Computer Science and Engineering, Sun Yat-sen University
<sup>3</sup> Guangdong Key Laboratory of Information Security Technology
$^{4}$  Tencent AI Lab  $^{5}$  The Hong Kong University of Science and Technology
Figure 1: Given a single facial photo ( $v1$  or  $v2$ ) as a tunable sample, the proposed method can insert this identity into the trained text-to-image model, e.g., Stable Diffusion [1], where the new person ( $v1$ ) can act like the original concept in the trained model and interact with another newly trained concept ( $v2$ ). Note that the input images are randomly generated from StyleGAN [2].
Exquisite demand exists for customizing the pretrained large text-to-image model, e.g. Stable Diffusion, to generate innovative concepts, such as the users themselves. However, the newly-added concept from previous customization methods often shows weaker combination abilities than the original ones even given several images during training. We thus propose a new personalization method that allows for the seamless integration of a unique individual into the pre-trained diffusion model using just one facial photograph and only 1024 learnable parameters under 3 minutes. So we can effortlessly generate stunning images of this person in any pose or position, interacting with anyone and doing anything imaginable from text prompts. To achieve this, we first analyze and build a well-defined celeb basis from the embedding space of the pre-trained large text encoder. Then, given one facial photo as the target identity, we generate its own embedding by optimizing the
*This work was done when the first author was an intern at Tencent AI Lab.
weight of this basis and locking all other parameters. Empowered by the proposed celeb basis, the new identity in our customized model showcases a better concept combination ability than previous personalization methods. Besides, our model can also learn several new identities at once and interact with each other where the previous customization model fails to. Project page is at: http://celeb-basis.github.io. Code is at: https://github.com/ygtxr1997/CelebBasis.
Vast image-text pairs [3] during training and the powerful language encoder [4] enable the text-to-image models [1, 5, 6] to generate diverse and fantastic images from simple text prompts. Though the generated images are exquisite, they may still fail to satisfy the users' demands since some concepts are not easy to be described by the text prompt [7]. For instance, individuals frequently post self-portraits on social media platforms, where the pre-trained text-to-image models struggle to produce satisfactory pictures of them despite receiving comprehensive instructions. This shortcoming makes these models less attractive to general users.
Recent works [8, 9, 10, 7] solve this problem via efficiently tuning the parameters of the model for personalization usage. For example, these techniques insert the new concepts (e.g. a specific bag, a dog, or a person) into the model by representing them as rarely-used pseudo-words [10] (or text-embeddings [9, 7, 8, 11]) and finetuning the text-to-image model with a few of these samples [10, 7]. After training, these pseudo-words or embeddings can be used to represent the desired concept and can also perform some combination abilities. However, these methods often struggle to generate the text description-aligned image with the same concept class (e.g. the person identities) [8, 10]. For instance, the original Stable Diffusion [1] model can successfully generate the image of the text prompt:" Barack Obama and Anne Hathaway are shaking hands." Nevertheless, in terms of generating two newly-learned individuals, the previous personalized methods [8, 10, 7] fall short of producing the desired identities as depicted in our experiments (Fig. 5).
In this work, we focus on injecting the most specific and widely-existing concept, i.e. the human being, into the diffusion model seamlessly. Drawing inspiration from the remarkable 3DMM [12], which ingeniously represents novel faces through a combination of mean and weight values derived from a clearly defined basis, we build a similar basis to the embeddings of the celebrity names in pretrained Stable Diffusion. In this way, are capable of representing any new person in the trained diffusion model via the basis coefficients.
We first collect a bunch of celebrity names from the Internet and filter them by the pre-trained text-to-image models [1]. By doing so, we obtain 691 well-known names and extract the text embedding by the tokenizer of the CLIP [4]. These celeb names correspond to over 700K samples in LAION datasets [3] (see our appendix for detailed statistics), containing powerful priors. Then, we construct a celeb basis via Principal Component Analysis (PCA [13]). To represent a new person with PCA coefficients, we use a pre-trained face recognition network [14] as the feature extractor of the given photo and learn a series of coefficients to re-weight the celeb basis, so that the new face can be recognized by the pre-trained CLIP transformer encoder. During the process, we only use a single facial photo and fix the denoising UNet and the text encoder of Stable Diffusion to avoid overfitting. After training, we only need to store the 1024 coefficients of the celeb basis to represent the newly-added identity since the basis is shared across the model. Though simple, the concept composition abilities [1] of the trained new individual is well-preserved, as we only reweight the text embeddings of the trained CLIP model and freeze the weights in the diffusion process. Remarkably, the proposed method has the ability to produce a strikingly realistic photo of the injected face in any given location and pose. Moreover, it opens up some new possibilities such as learning multiple new individuals simultaneously and facilitating seamless interaction between these newly generated identities as shown in Fig. 1. The contributions of the paper are listed as follows:
- We propose celeb basis, a basis built from the text embedding space of the celebrities' names in the text-to-image model and verify its abilities, such as interpolation.
- Based on the proposed celeb basis, we design a new personalization method for the text-to-image model, which can remember any new person from a single facial photo using only 1024 learnable coefficients.
- Extensive experiments show our personalized method has more stable concept composition abilities than previous works, including generating better identity-preserved images and interacting with new concepts.
Image Generation and Editing. Given a huge number of images as the training set, deep generative models target to model the distribution of training data and synthesize new realistic images through sampling. Various techniques have been widely explored, including GAN [15, 2], VAE [16, 17], Autoregressive [6, 18, 19, 20], flow [21, 22]. Recently, diffusion models [23, 24] gain increasing popularity for their stronger abilities of text-to-image generation [5, 6, 1]. Conditioned on the text embedding of pre-trained large language models [4], these diffusion models are iterative optimized using a simple denoising loss. During inference, a new image can be generated from sampled Gaussian noise and a text prompt. Although these diffusion models can synthesize high-fidelity images, they have difficulties in generating less common concepts [25] or controlling the identity of generated objects [7]. Current editing methods are still hard to solve this problem, e.g. directly blending the latent of objects [26, 27] to the generated background will show the obvious artifacts and bring difficulties in understanding the scenes correctly [28]. On the other hand, attention-based editing works [29, 30, 31, 32] only change the appearance or motion of local objects, which can not generate diverse new images with the same concept (e.g. human and animal identity) from the referred image.
Model Personalization. Different from text-driven image editing, tuning the model for the specific unseen concept, i.e. personalized model, remembers the new concepts of the reference images and can synthesize totally unseen images of them, e.g. appearance in a new environment, interaction with other concepts in the original stable diffusion. For generative adversarial newworks [2, 33], personalization through GAN inversion has been extensively studied. This progress typically involves finetuning of the generator [34, 35], test-time optimization of the latents [36], or a pre-trained encoder [37]. Given the recent diffusion generative model [1, 5], it is straightforward to adopt previous GAN inversion techniques for the personalization of diffusion models. Dreambooth [10] finetunes all weight of the diffusion model on a set of images with the same identity and marks it as the specific token. Meanwhile, another line of works [7, 9, 38] optimizes the text embedding of special tokens (e.g.  $V^{*}$ ) to map the input image while freezing the diffusion model. Later on, several works [8, 39, 40] combine these two strategies for multi-concept interaction and efficient finetuning taking less storage and time. These methods focus on general concepts in the open domain while struggling to generate interactions between fine-grained concepts, i.e.human beings with specific identities. Since most of the previous works require the tuning in the test time, training inversion encoders are also proposed to generate textual embedding from a single image in the open domain (e.g. UMM [41], ELITE [42], and SuTI [43]), or in human and animal domain (e.g. Taming-Encoder [44], Instant-Booth [45], E4T [11]). However, a general human identity-oriented embedding is difficult to be obtained from a naively optimized encoder, and tuning the Stable Diffusion on larger-scale images often causes the concept forget. In contrast, our method focuses on a better representation of identity embedding in the diffusion model (celeb basis in Sec. 3.1), which significantly eases the process of optimization such that we only need 1024 parameters to represent an identity more correctly as in Sec. 3.2 and stronger concept combination abilities, which can converge in only 3 minutes.
Identity Basis. Representing the human identity via basis is not new in traditional computer vision tasks. e.g. in human face modeling, 3D Morphable Models [12] and its following models [46, 47] scans several humans and represent the shape, expression, identity, and pose as the PCA coefficients [13], so that the new person can be modeled or optimized via the coefficients of the face basis. Similar ideas are also used for face recognition [48], where the faces in the dataset are collected and built on the basis of the decision bound. Inspired by these methods, our approach takes advantage of the learned celebrity names in the pre-trained text-to-image diffusion model, where we build a basis on this celebrity space and generate the new person via a series of learned coefficients.
Our method aims to introduce a new identity to the pre-trained text-to-image model, i.e. Stable Diffusion [1], from a single photo via the optimized coefficients of our self-built celeb basis. So that it can memorize this identity and generate new images of this person in any new pose and interact with other identities via text prompts. To achieve this, we first analyze and build a celeb basis on
Figure 2: The building process of the proposed Celeb Basis. First, we collect about 1,500 celebrity names as the initial collection. Then, we manually filter the initial one to  $m = 691$  names, based on the synthesis quality of text-to-image diffusion model [1] with corresponding name prompt. Later, each filtered name is tokenized and encoded into a celeb embedding group  $g_{i}$ . Finally, we conduct Principle Component Analysis to build a compact orthogonal basis, which is visualized on the right.
the embedding space of the text encoder through the names of the celebrities (Sec. 3.1). Then, we design a face encoder-based method to optimize the coefficients of the celeb basis for text-to-image diffusion model customization (Sec. 3.2).
3.1 Celeb Basis
Preliminary: Text Embeddings in Text-to-Image Diffusion Models. In the text-to-image model, given any text prompts  $u$ , the tokenizer of typical text encoder model  $e_{\mathrm{text}}$ , e.g. BERT [49] and CLIP [4], divides and encodes  $u$  into  $l$  integer tokens by order. Correspondingly, by looking up the dictionary, an embedding group  $g = [v_1, \dots, v_l]$  consisting of  $l$  word embeddings can be obtained, where each embedding  $v_i \in \mathbb{R}^d$ . Then the text transformer  $\tau_{\mathrm{text}}$  in  $e_{\mathrm{text}}$  encodes  $g$  and generates text condition  $\tau_{\mathrm{text}}(g)$ . The condition  $\tau_{\mathrm{text}}(g)$  is fed to the conditional denoising diffusion model  $\epsilon_{\theta}(z_t, t, \tau_{\mathrm{text}}(g))$  and synthesize the output image following an iterative denoising process [23], where  $t$  is the timestamp,  $z_t$  is a noised image or latent at  $t$ . Previous text-to-image model personalization methods [7, 10, 9] have shown the importance of text embedding  $g$  in personalizing semantic concepts. However, in text-to-image models' personalization, they only consider it as an optimization goal [9, 7, 8], instead of improving its representation.
Interpolating Abilities of Text Embeddings. Previous works have shown that text embedding mixups [50] benefit text classification. To verify the interpolation abilities in text-to-image generation, we randomly pick two celebrity names embeddings  $v_{1}$  and  $v_{2}$ , and linearly combine them as  $\hat{v} = \lambda v_{1} + (1 - \lambda)v_{2}$ , where  $0 < \lambda < 1$ . Interestingly, the generated image of the interpolated embedding  $\hat{v}$  also contains a human face as shown in Fig. 3, and all the generated images perform well in acting and interacting with other celebrities. Motivated by the above finding, we build a celeb basis so that each new identity can lie in the space formed by celebrity embeddings.
Figure 3: The interpolated text-embedding of two celebrities is still a human (top row) and it also can perform strong concept combination abilities in the pretrained Stable Diffusion [1] (bottom row).
Build Celeb Basis from Embeddings of the Collected Celebrities. As shown in Fig. 2, we first crawl about 1,500 celebrity names from Wikipedia as the initial collection. Then, we build a manual filter based on the trained text-to-image diffusion model [1] by constructing the prompts of each name and synthesizing images. A satisfied celeb name should have the ability to generate human images
Figure 4: During training (left), we optimize the coefficients of the celeb basis with the help of a fixed face encoder. During inference (right), we combine the learned personalized weights and shared celeb basis to generate images with the input identity.
with prompt-consistent identity and interact with other celebs in synthesized results. Overall, we get  $m = 691$  satisfied celeb names where each name  $u_{i}, i \in \{1, \dots, m\}$  can be tokenized and encoded into a celeb embedding group  $g_{i} = [v_{1}^{i}, \dots, v_{k_{i}}^{i}]$ , notice that the length  $k_{i}$  of each celeb embedding group  $g_{i}$  might not be the same since each name may contain multiple words (or the tokenizer will divide the word into sub-words). To simplify the formula, we compose the nonrepetitive embeddings so that each  $g_{i}$  only contains the first two embeddings (i.e.  $k_{i} = 2$  for all  $m$  celebrities). Using  $C_{1}$  and  $C_{2}$ , i.e.  $C_{k} = [v_{k}^{1}, \dots, v_{k}^{m}]$ , to denote the first and second embeddings of each  $g_{i}$  respectively, we can roughly understand them as the first name and last name embedding sets.
To further build a compact search space, inspired by 3DMM [12] which uses PCA [13] to map high-dimensional scanned 3D face coordinates into a compact lower-dimensional space, for each embedding set  $C_k$ , we calculate its mean  $\overline{C}_k = \frac{1}{m} \sum_{i=1}^{m} v_k^i$  and PCA mapping matrix  $B_k = \mathrm{PCA}(C_k, p)$ , where  $\overline{C}_k \in \mathbb{R}^d$  and  $\mathrm{PCA}(X, p)$  indicates the PCA operation that reduces the second dimension of matrix  $X \in \mathbb{R}^{m \times d}$  into  $p$  ( $p < d$ ) principal components, i.e.  $B_k = [b_k^1, \dots, b_k^p]$ . As shown in Fig. 2, the mean embedding  $\overline{C}_k$  still represents a face and we can get the new face via some coefficients applied to  $B_k$ .
Overall, our celeb basis is defined on two basis  $[\overline{C}_1,B_1]$  and  $[\overline{C}_2,B_2]$  working like the first and last name. We use the corresponding principle components  $A_{1}$  and  $A_{2}$  (where  $A_{k} = [\alpha_{k}^{1},\dots,\alpha_{k}^{p}])$  to represent new identities. Formally, for each new person  $\hat{g}$ , we use two  $p$ -dimensional coefficients of the celeb basis and can be written by:
$$
\hat {g} = \left[ \hat {v} _ {1}, \hat {v} _ {2} \right], \quad \hat {v} _ {k} = \bar {C} _ {k} + \sum_ {x = 1} ^ {p} \alpha_ {k} ^ {x} b _ {k} ^ {x}, \tag {1}
$$
In practice,  $p$  equals 512 as discussed in the ablation experiments.
To control the generated identities, we optimize the coefficients with the help of a face encoder as the personalization method. We introduce it in the below section.
3.2 Stable Diffusion Personalization via Celeb Basis
Fast Coefficients Optimization for Specific Identity. Given a single facial photo, we use the proposed celeb basis to embed the given face image  $x$  of the target identity into the pretrained text-to-image diffusion model as shown in Fig. 4. Since direct optimization is hard to find the optimized weight, we consider using the pre-trained state-of-the-art face recognition models  $F$ , i.e. ArcFace [14], to capture the identity-discriminative patterns. In detail, we adopt the  $F$  to extract 512 dimension face embedding as priors. Then a single-layer MLP followed by an  $L_{2}$ -normalization is used to map the face priors into the modulating coefficients  $A_{1}$  and  $A_{2}$ . Following the Eq. 1, we can obtain the embedding group  $\hat{g}$  of the  $x$  using the pre-defined basis. By representing the text prompt of  $\hat{g}$  as  $V^{*}$ , we can involve  $V^{*}$  to build the training pairs between the text prompt of input face and "A photo of  $V^{*}$ ", "A depiction of  $V^{*}$ ", etc.. Similar to previous works [10, 7, 8], we only use simple diffusion denoising loss [23]:
$$
\mathbb {E} _ {\epsilon \sim N (0, 1), x, t, g} [ \| \epsilon - \epsilon_ {\theta} \left(z _ {t}, t, \tau_ {\text {t e x t}} (g)\right) \| ], \tag {2}
$$
where  $\epsilon$  is the unscaled noise sample,  $g$  denotes the text embeddings containing  $\hat{g}$ . During training, only the weights of MLP need to be optimized, while other modules, including the celeb basis, face
encoder  $F$ , CLIP transformer  $\tau_{\mathrm{text}}$ , and UNet  $\epsilon_{\theta}$  are fixed. Thus, the original composition abilities of the trained text-to-image network are well-preserved, avoiding the forgetting problem. Since we only have a single photo, we use color jitter, random resize, and random shift as data augmentations on the supervision to avoid overfitting. Notice that, we find our augmentation method can work well even though there are no regularization datasets which is important in previous methods [10, 7, 8], showing the strong learning abilities of the proposed methods. Since the proposed method only involves a few parameters, it only takes almost 3 minutes for each individual on an NVIDIA A100 GPU, which is also much faster than previous methods.
Testing. After training, only two groups of coefficients  $A_{1}$  and  $A_{2}$  applied to the principal celeb basis components need to be saved. In practice, the number of principal components of each group is  $p = 512$ , coming to only 1024 parameters and 2-3KB storage consumption for half-precision floatings. Then, users can build the prompt with multiple action description prompts (e.g. "A photo of  $V^{*}$  is playing guitar") to synthesize the satisfied images as described in Fig. 4.
Multiple Identities Joint Optimization. Most previous methods only work on a single new concept [9, 10], Custom Diffusion [8] claim their method can generate the images of multiple new concepts (e.g. the sofa and cat). However, for similar concepts, e.g. the different person, their method might not work well as in our experiments. Besides, their method is still struggling to learn multiple ( $>3$ ) concepts altogether as in their limitation. Differently, we can learn multiple identities (e.g. 10) at once using a shared MLP mapping layer as in Fig. 4. In detail, we simply extend our training images to 10 and jointly train these images using a similar process as single identities. Without a specific design, the proposed method can successfully generate the weight of each identity. After training, our method can perform interactions between each new identity while the previous methods fail to as shown in the experiments. More implementation details are in the appendix.
Datasets. We conduct experiments on the self-collected  $2K$  synthetic facial images generated by StyleGAN [2]. Using synthetic faces as input for assessing the effectiveness of generative models, one can ease the reliance on the dataset foundation of the initial pre-trained text-to-image model. We also perform some experiments on the real photo of the individual, more results and comparisons are shown in the supplemental materials.
Metrics. First, we assess the performance of the generated images by utilizing objective metrics. For instance, we calculate the consistency between the prompt and generated image through CLIP score [4], which is denoted as "Prompt" in tables. Additionally, ensuring identity consistency and clarity of facial features are crucial aspects of our task. Therefore, we evaluate identity similarity using a pretrained face recognition encoder [14] and mark it as "Identity". Furthermore, to demonstrate the rationality behind generation, we also calculate the rate of successful face detection ("Detect") via a pretrained face detector [14]. Lastly, user studies are conducted to evaluate text-image alignment along with identity and photo qualities. The images having the highest text-alignment scores from a single inference batch are used to compare the peak qualitative performance of all the methods.
4.2 Comparing with State-of-the-Art Methods
We compare the proposed method with several well-known state-of-the-art personalization methods for Stable Diffusion [1], including DreamBooth [10], Textual-Inversion [7] and Custom Diffusion [8]. As shown in Fig. 5, given one single image as input, we evaluate the performance of several different types of generation, including the simple stylization, concept combination abilities, and two new concept interactions. Textual inversion tends to overfit the input image so most of the concepts are
Table 1: Quantitative comparisons.
Methods	Objective Metrics↑	User Study↑	#Params↓	Time↓(min)
Prompt	Identity	Detect	Quality	Text	Identity
Textual Inversion [7]	0.1635	0.2958	92.86%	2.23	1.88	2.55	1,536	24
Dreambooth [10]	0.2002	0.0512	54.76%	3.32	3.70	2.75	\( 9.83 \times 10^8 \)	16
Custom Diffusion [8]	0.2608	0.1385	80.39%	3.31	3.55	2.96	\( 5.71 \times 10^7 \)	12
Ours	0.2545	0.2072	84.78%	3.47	4.01	3.37	1,024	3
Textural Inversion
DreamBooth
a photo of  $V_{1}^{*}$
Custom Diffusion
$V_{1}^{*}$
$V_{2}^{*}$
a sand sculpture of  $V_{2}^{*}$
$V_{3}^{*}$
$V_{3}^{*}$  Funko Pop
$V_{4}^{*}$  eating bread in front of the Eiffel Tower
$V_{5}^{*}$
$V_{5}^{*}$  wearing yellow jacket, and driving a motorbike
$V_{6}^{*}$
$V_{6}^{*}$  and Barack Obama cooking together in a kitchen
$V_{7}^{*}$
$V_{7}^{*}$  and Anne Hathaway having a delicate candlelight dinner
$V_{2}^{*}$  shakes hands with  $V_{3}^{*}$
Figure 5: We compare several different abilities between our method and baselines (Textural Inversion [7], Dreambooth [10], and Custom Diffusion [8]).