Inserting Anybody in Diffusion Models via Celeb Basis
Authors: Ge Yuan
Xiaodong Cun
Yong Zhang
Maomao Li
Chenyang Qi
Xintao Wang
Ying Shan
Huicheng Zheng
Venue: NeurIPS 2023 poster (2023)
Keywords: Text-to-Image Synthesis
Personalized Synthesis
Face Embedding
TL;DR: Integration of a unique individual into the diffusion model using just one facial photograph and only 1024 learnable parameters under 3 minutes

Abstract: Exquisite demand exists for customizing the pretrained large text-to-image model, $e.g.$ Stable Diffusion, to generate innovative concepts, such as the users themselves. However, the newly-added concept from previous customization methods often shows weaker combination abilities than the original ones even given several images during training. We thus propose a new personalization method that allows for the seamless integration of a unique individual into the pre-trained diffusion model using just $one\ facial\ photograph$ and only $1024\ learnable\ parameters$ under $3\ minutes$. So we can effortlessly generate stunning images of this person in any pose or position, interacting with anyone and doing anything imaginable from text prompts. To achieve this, we first analyze and build a well-defined celeb basis from the embedding space of the pre-trained large text encoder. Then, given one facial photo as the target identity, we generate its own embedding by optimizing the weight of this basis and locking all other parameters. Empowered by the proposed celeb basis, the new identity in our customized model showcases a better concept combination ability than previous personalization methods. Besides, our model can also learn several new identities at once and interact with each other where the previous customization model fails to. Project page is at: http://celeb-basis.github.io. Code is at: https://github.com/ygtxr1997/CelebBasis.

Inserting Anybody in Diffusion Models via Celeb Basis
Ge Yuan 1,,34 Xiaodong Cun Yong Zhang' Maomao Lift Chenyang Qi Huicheng Zheng
Xintao Wang'
Ying Shan'
School of Computer Science and Engineering, Sun Yat-sen University 2 Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China Guangdong Key Laboratory of Information Security Technology
Tencent AT Lab
The Hong Kong University of Science and Technology
paintig paintnng
Bankey
v1
is plyiigg The guftor
estg breed from of the Effiel Tower
knghh picte araor
weord a 57.11 spoceau in Space
and limrect coola kitchen
baking bertiddy cola.
v2 Single Input
pay
anjy dellcone
anjoy day
Genenrted Image
Figure I: Given a single facial photo (e1 or v2) a tunable sample, the proposed method can insert this identity into the trained text-to-image model, e.g.. Stable Diffusion [1], where the new person (e1) can act like the original concept in the trained model and interact with another newly trained concept (v22. Note that the input images are randomly generated from StyleGAN [2].
Abstract
Exquisite demand exists for customizing the pretrained large text-to-image model, e.g. Stable Diffusion, generate innovative concepts, such the themselves. However, the newly-added concept from previous customization methods often shows weaker combination abilities than the original ones even given several images during training. We thus propose a new personalization method that allows for the seamless integration of unique individual into the pre-trained diffusion model using just one facial photograph and only 1024 learnable parameters under 3 minutes. So can effortlessly generate stunning images of this person in any pose or position, interacting with anyone and doing anything imaginable from text prompts. To achieve this, we first analyze and build well-defined celeb basis from the embedding space of the pre-trained large text encoder. Then, given one facial photo as the target identity, we generate own embedding by optimizing the
"This work was done when author was an intern Tencent Lah. !Coreeponding authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

weight of this basis and locking all other parameters. Empowered by the proposed celeb basis, the new identity in our customized model showcases a better concept combination ability than previous personalization methods. Besides, our model can also learn several new identities at once and interact with each other where the previous customization model fails to. Project page is at: github. io. Code is at: https: https://githii //github.. com http:/xx1999999eee//sstt
1 Introduction
Vast image-text pairs [3] during training and the powerful language encoder [4] enable the text-to- image models [1, 5, 6] to generate diverse and fantastic images from simple text prompts. Though the generated images are exquisite, they may still fail to satisfy the users' demands since some concepts are not easy to be described by the text prompt [7]. For instance, individuals frequently post self-portraits on social media platforms, where the pre-trained text-to-image models struggle to produce satisfactory pictures of them despite receiving comprehensive instructions. This shortcoming makes these models less attractive to general users.
Recent works [8, 10, 7] solve this problem via efficiently tuning the parameters of the model for personalization usage. For example, these techniques insert the new concepts (e.g. a specific bag, dog, person) into the model by representing them as rarely-used pseudo-words [10] (or text- embeddings [9, 7, 8, I1]) and finetuning the text-to-image model with a few of these samples [10, 7]. Afte training. these pseudo-words or embeddings can be used to represent the desired concept and can also perform some combination abilities. However, these methods often struggle to generate image with the same concept class (e.g. the person identities) [8. 10]. For instance, the original Stable Diffusion [1] model can successfully generate the image of the text prompt:" Barack Obama and Anne Hathaway are shaking hands." Nevertheless, in terms of generating two newly-learned individuals, the previous personalized methods [8, 10, 7] fall short of producing the desired identities as depicted in our experiments (Fig. 5).
the text description-aligne
In this work, we focus on injecting the most specific and wixely-existing concept, i.e. the human being, into the diffusion model seamlessly. Drawing inspiration from the remarkable 3DMM [12], which ingeniously represents novel faces through a combination of mean and weight values derived from clearly defined basis, we build similar basis to the embeddings of the celebrity names in pretrained Stable Diffusion. In this way, are capable of representing any new person in the trained diffusion model via the basis coefficients.
We first collect a bunch of celebrity names from the Internet and filter them by the pre-trained text-to-image models [1]. By doing so, obtain 691 well-known names and extract the text embedding by the tokenizer of the CLIP (4]. These celeb names corresponds to over 700K samples in LAION datasets [3] (see our appendix for detailed statistics), containing powerful priors. Then, we construct a celeb basis via Principal Component Analysis (PCA [13]). To represent a new person with PCA coefficients, we use pre-trained face recognition network [14] as the feature extractor of the given photo and learn a series of coefficients to re-weight the celeb basis, so that the new face can be recognized by the pre-trained CLIP transformer encoder. During the process, we only use a single facial photo and fix the denoising UNet and the text encoder of Stable Diffusion to avoid overfitting. After training, we only need to store the 1024 coefficients of the celeb basis to represent the newly-added identity since the basis is shared across the model. Though simple, the concept composition abilities [1] of the trained new individual is well-preserved. as we only reweight the text embeddings of the trained CLIP model and freeze the weights in the diffusion process. Remarkably. the proposed method has the ability to produce strikingly realistic photo of the injected face in any given location and pose. Moreover, it opens up some new possibilities such as learning multiple new individuals simultaneously and facilitating seamless interaction between these newly generated identities shown in Fig. 1. The contributions of the paper are listed as follows:
We propose celeb basis, basis built from the text embedding space of the celebrities names in the text-to-image model and verify its abilities, such as interpolation.
Based on the proposed celeb basis, we design a new personalization method for the text-to- image model, which can remember any new person from a single facial photo using only 1024 learnable coefficients.
2

---

Extensive experiments show our personalized method has more stable concept composition abilities than previous works, including generating better interacting with new concepts.
images and
2 Related Work
Image Generation and Editing, Given huge number of images as the training set, deep generative models target to model the distribution of training data and synthesize new realistic images through sampling. Various techniques have been widely explored, including GAN [15, 21, VAE [16, 17], Autoregressive [6. 18, 19, 20J. flow [21, 22]. Recently, diffusion models [23, 24] gain increasing popularity for their stronger abilities of text-to-image generation [5, 6, 1J. Conditioned embedding of pre-trained large language models [4], these diffusion models are iterative optimized using& simple denoising loss. During inference, new image can be generated from sampled Gaussian noise and a text prompt. Although these diffusion models can synthesize high-fidelity images, they have difficulties in generating less common concepts [25] or controlling the identity of generated objects [7]. Current editing methods are still hard to solve this problem, e-g. directly blending the latent of objects [26, 27] to the generated background will show the obvious artifacts and bring difficulties in understanding the scenes correctly [28]. On the other hand, iveentio-bbsse editing works [29, 30, 31, 32] only change the appearance or motion of local objects, which can not generate diverse new images with the same concept (e.g. human and animal identity) from the referred image.
the text
Model Personalization. Different from text-driven image editing, tuning the model for the specific unseen concept, i.e. personalized model, remembers the new concepts of the reference images and can synthesize totally unseen images of them, e.g. appearance in a new environment, interaction with other concepts in the original stable diffusion. For generative adversarial neworks [2. 33), personalization through GAN inversion has been extensively studied. This progress typically involves finetuming of the generator [34, 35], test-time optimization of the latents [36], or a pre-trained encoder [37]. Given the recent diffusion generative model [1. 5], it is straightforward to adopt previous GAN inversion techniques for the personalization of diffusion models. Dreambooth [10] finetunes all weight of the diffusion model on set of images with the same identity and marks it as the specific token. Meanwhile, another line of works [7, 9, 38] optimizes the text embedding of special tokens (e.g. V*) to map the input image while freezing the diffusion model. Later several works [8, 39, 40] combine these two strategies for multi-concept interaction and efficient finetuning taking less storage and time. These methods focus general concepts in the open domain while struggling generate interactions between fine- tnee-graied concepts, i.e he.hmm beings with specific identities. Since most of the previous works require the tuning in the test time, training inversion encoders are also proposed to generate textual embedding from a single image in the open domain (e.g. UMM [41], ELITE [42], and SuTT [43]), in human and animal domain (e.g. MominngEEncodrr [44], Instant- Instant-Booth [45], E4T [11]). However, a general human identity-oriented embedding is difficult to be obtained from a naively optimized encoder, and tuning the Stable Diffusion larger-scale images often causes the concept forget. In contrast, our method focuses on a better representation of identity embedding in the diffusion model (celeb basis in Sec. 3. which significantly eases the process of optimization such that we only need 1024 parameters to represent an identity more correctly as in Sec. 3.2 and stronger concept combination abilities, which can converge in only 3 minutes.
Identity Basis. Representing the human identity via basis is not new in traditional computer vision tasks. e.g. in human face modeling, 3D Morphable Models [12] and its following models [46, 47] scans several humans and represent the shape, expression, identity, and pose as the PCA coeffi cients [13], so that the new person can be modeled or optimized via the coefficients of the face basis. Similar ideas are also used for face recognition [48], where the faces in the dataset are collected and built on the basis of the decision bound. Inspired by these methods, our approach takes advantage of the learned celebrity names in the pre-trained text-to-image diffusion model, where we build basis on this celebrity space and generate the new person via series of learned coefficients.
3 Method
Our method aims to introduce a new identity to the pre-trained text-to-image model, i.e. Stable Diffusion [1], from single photo via the optimized coefficients of our self-built celeb basis. So that can memorize this identity and generate new images of this person in any new pose and interact with other identities via text prompts. To achieve this, first analyze and build celeb basis on
3

---

Srarchod
Filter by Stable Diffision
diaguo Hetpp Palpppaique Jaquaaye
Embeddings
Celeh
Celebrity
photo
Beyong
Ezorge Corlis
Diflotis
David Brwis
Tom
Setisfled
PCA
Bayans
Can generated human imoge
Jackl Chan
+9
Jockie
Relatively good ideniity
53
Roane Alkindon
Good conceps combination
Ston
0
D
The building process of the proposed Celeb Basis. First, we collect about 1,500 celebrity names as the initial collection. Then, we manually filter the initial one m 691 names, based on the synthesis quality of text-to-image diffusion model [1] with corresponding name prompt. Later, each filtered name is tokenized and encoded into celeh embedding group Ji. Finally, we conduct Principle Component Analysis to build compact orthogonal basis, which is visualized on the right.
Figure 2:
the embedding space of the text encoder through the names of the celebrities (Sec. 3.1). Then, we design face encoder- based method to optimize the coefficients of the celeb basis for text-to-image diffusion model customization (Sec. 3.2).
3.1 Celeb Basis
Preliminary: Text Embeddings in Text-to- Diffusion Models. In the text-to-image model, given any text prompts u, the tokenizer of typical text encoder model étext, e.g. BERT [49] and CLIP [4], divides and encodes into integer tokens by order. Correspondingly. by looking up the dictionary, an embedding group 9 = [v1,-.. 2y] consisting of word embeddings can be obtained, where each embedding v; Rdd Then the text transformer Thext in Étext encodes and generates text condition Teext (g). The condition (g) is fed to the conditional denoising diffusion model <0(ze, L, Ttext (g)) and synthesize the output image following an iterative denoising process [23], where t is the timestamp,
is noised image latent Previous text-to-image model personalization methods [7, 10, 9] have shown the importance text embedding g in personalizing semantic concepts. However, in text-to-image image models' personalization, they only consider it optimization goal [9, 7, 8). instead of improving its representation.
Interpolating A Abilities of Text Embeddings. Previous works have shown that text embedding mixups [50] benefit text classification. To verify the interpolation abilities in text-to-image generation, we randomly pick two celebrity names embeddings
and U2, and linearly combine them as é 1. Interestingly, the generated image of the interpolated embedding é also contains a human face shown in Fig. 3, and all the generated images perform well in acting and interacting with other celebrities. Motivated by the above finding, we build celeb basis so that each new identity can lie in the space formed by celebrity embeddings.
Av (1 (1) where
= Robert Downey
photo of
photo of
A photo of
photo of
9 0.68, 0.42s,
P playing the guitar
@ playing basketball
D cating breads in front of Eiffel Tower
is talking Anme Hathuway
11
Barack Obama
Figure 3: The interpolated fext-embedding of two celebrities is still a human (top row) and also can perform strong concept combination abilities the pretrained Stable Diffusion [1] (bottom row).
Build Celeb Basis from Embeddings the Collected Celebrities. As shown in Fig. 2, we first crawl about 1 1,500 celebrity names from Wikipedia as the initial collection. Then, we build manual filter based on the trained text-to-image diffusion model [1] by constructing the prompts of each name and synthesizing images. A satisfied celeb name should have the ability to generate human images

---

Training
Test
photo
UNet
UNet
A photo of playing guitar
Celob Basis
Text
Loss
Transformrs in CLIP
Transformers
MLP&
Data
Celeb Basis
in CLIP
Nerta
Aug.
wwsigeotts
Figure 4: During training (left), optimize the coefficients of the celeb basis with the help of fixed face encoder. During inference (right), we combine the learned personalized weights and shared celeb basis generate images with the input identity.
with prompt-consistent identity and interact with other celebs in synthesized results. Overall, get m = 691 satisfied celeb names where each name Us E {1...., m..] can be tokenized and encoded into a celeb embedding group gi = [u.; J notice that the length k; of each celeb embedding group might not the same since each name may contain multiple words (or the tokenizer will divide the word into sub-words). To simplify the formula, compose the nonrepetitive embeddings so that each 9: only contains the first two embeddings (i.e. k; = 2 for all m celebrities). Using C1 and C2 i.e. C₁ = [ .... u denote the first and second embeddings of each gi respectively, we can roughly understand them as first name and last name embedding sets.
To further build compact search space, inspired by 3DMM [12] which uses PCA [13] map high-dimensional scanned 3D face coordinates into a compact lower-dimensional space, for each embedding set Cs. we calculate its mean Ck = - oy and PCA mapping matrix B₁ = PCA(Ck,, p). where Cs E id and PCA(X, p) indicates the PCA operation that reduces the second dimension matrix X E mmd into p (p < d) principal components, í.e. B₁ = [b/,.... bi]l As shown in Fig. 2. the mean embedding C₁ still represents face and we can get the new face via some coefficients applied Bx.
Overall, our celeb basis is defined on two basis [C1, Bi] and [C2, B2] working like the first and last name. We use the corresponding principle components A₁ and A2 (where A& = [o]:.... af) to represent new identities. Formally, for each new person ÿ, we use two p-dimensional coefficients of the celeb basis and can be written by:
P
9 = 10 02),
0z = C++
Σ
ag
(1)
In practice, p equals 512 as discussed in the ablation experiments.
To control the generated identities, we optimize the coefficients with the help of a face encoder as the personalization method. We introduce it in the below section.
3.2 Stable Diffusion Personalization via Celeb Basis
Fast Coefficients Optimization for Specific Identity. Given a single facial photo, we use the proposed celeb basis to embed the given face image x of the target identity into the pretrained text-to-image diffusion model as shown in Fig. 4. Since direct optimization hard to find the optimized weight, consider using the pre-trained state-of-the-art face recognition models F. i.e. ArcFace [14], to capture the identiy-diseriminatiee patterns. In detail, we adopt the F to extract 512 dimension face embedding priors. Then single-layer MLP followed by Le-normalization is used to map the face priors into the modulating coefficients A₁ and Ag. Following the Eq. we can obtain the embedding group of the using the pre- refinef basis. By representing the text prompt of ğ as V*, we can involve V* to build the training pairs between the text prompt of input face and "A photo of V*" "A depiction of V*n etc.. Similar to previous works [10, 7, 8), only use simple diffusion denoising loss [23]:
Ne Ille c (ze t, Treet (1)
(2)
where is the unscaled noise sample, y denotes the text embeddings containing 9. During training, only the weights of MLP need be optimized, while other modules, including the celeb basis, face
5

---

encoder F. CLIP transformer Teext, and UNet are fixed. Thus, the original composition abilities of the trained text-to-image network are well-preserved. avoiding the forgetting problem. Since we only have a single photo, we use color jitter, random resize, and random shift as data augmentations on the supervision to avoid overfitting. Notice that, we find our augmentation method can work well even though there are no regularization datasets which is important in previous methods [10, 7, 8), showing the strong learning abilities of the proposed methods. Since the proposed method only involves a few parameters, it only takes almost 3 minutes for each individual on an NVIDIA A100 GPU, which is also much faster than previous methods.
Testing, After training, only two groups of coefficients A1 and Ag applied to the principal celeb basis components need be saved. In practice, the number of principal components of each group is = 512, coming to only 1024 parameters and 2-3KB storage consumption for half-precision floatings. Then, users can build the prompt with multiple action description prompts (e.g. "A photo of V* is playing guitar") to synthesize the satisfied images as described in Fig. 4.
Multiple Identities Joint Optimization. Most previous methods only work on single new con- cept [9, 10], Custom Diffusion [8] claim their method can generate the images of multiple new concepts (e.g. the sofa and cat). However, for similar concepts, e-g. the different person, their method might not work well as in our experiments. Besides, their method is still struggling to learn multi- ple (> 3) concepts altogether as their limitation. Differently, we can learn multiple identities (e.g. 10) at once using a shared MLP mapping layer as in Fig. 4. In detail, we simply extend our training images to 10 and jointly train these images using a similar process as single identities. Without specific design, the proposed method can successfully generate the weight of each identity. After training, our method can perform interactions between each new identity while the previous methods fail to as shown in the experiments. More implementation details are in the appendix.
Experiments
4.1 Datasets and Metries
Datasets. We conduct experiments on the self-collected 2K synthetic facial images generated by StyleGAN [2]. Using synthetic faces as input for assessing the effectiveness of generative models, one can ease the reliance on the dataset foundation of the initial pre-trained texi-to-image model. We also perform some experiments on the real photo of the individual, more results and comparisons are shown in the supplemental materials.
Metrics. First, we assess the performance of the generated images by utilizing objective metrics. For instance, we calculate the consistency between the prompt and generated image through CLIP score [4], which is denoted as "Prompt" in tables. Additionally, ensuring identity consistency and clarity of facial features are crucial aspects of our task. Therefore, we evaluate identity similarity using a pretrained face recognition encoder [14] and mark it as "Identity". Furthermore, to demonstrate the rationality behind generation, we also calculate the rate of successful face detection ("Detect") via a pretrained face detector [14]. Lastly, user studies are conducted to evaluate text-image alignment along with identity and photo qualities. The images having the highest text-alignment scores from a single inference batch are used compare the peak qualitative performance of all the methods.
4.2 Comparing with Hlate-of-the-Art Methods
We compare the proposed method with several well-known ffate-ofthhe-ae personalization methods for Stable Diffusion [1]. including DreamBooth [10]. Textual-Inversion [7] and Custom Diffusion [8]. As shown in Fig. 5, given one single image as input, we evaluate the performance of several different types of generation, including the simple stylization, concept combination abilities, and two new concept interactions. Textual inversion tends to overfit the input image so most of the concepts are
Table 1: Quantitative comparisons.
Objective Metricst
User Studyt
Methods
#Params|
Timel (min)
Prompt
Identity
Detect
Quality
Text
Identity
Textual Inversion [7]
0.1635
0.2958
92.86%
2.23
1.88
2.55
1,536
24
Dreambooth [10]
0.2002
0,0512
54.76%
3.32
3.70
2.75
9.83 x 100
16
Custom Diffusion [8]
0.2608
0.1385
80.39%
3.31
3.55
2.96
5.71 x 10"
12
Ours
0.2545
0.2072
84.78%
3.47
4.01
3.37
1,024
3

---

Turget Image
Textural Inversion
DreamBooth
Custom Diffusson
Ours
Vi
photo of V
V22
sand sculpture V22"
V/
V Funko Pop
V
Vi¹ cating bread in front of the Eiffel Tower
RBE EREINN.
VE
VS wearing yellow jacket, and driving a motorbike
Vi
V/2 and Barack Obama cooking together kitchen
Vi
V und Anne Huthaway having delicate candlelight dinner
V2
V2
VS shakes hands with V/2"
V
and V enjoying day amusement park
Figure 5: We compare several different abilities between our method and baselines (Textural Inver-
sion [7], Dreambooth [10], and Custom Diffusion [8]).

---

0.3
Methods
Prompt
Identity"
Detect
Texaual Inverson
0.30
Decantloot
w/o celeh basis
0.1386
0.2528
69.28%
Phonpp Similarity
Cistomn
w/ 350 names
0.2214
0.2023
69.28%
Ours
w/o filter
0.1939
0.2037
80.62%
w flatten
0.2026
0.1873
80.39%
0.00
0.2247
0.1061
76.47%
Stylinal Pontort
p 256
0.1812
0.0656
60.13%
0.18
Single
With Chich
p 768
0.1380
0.0836
47.06%
w/o F
0.1914
0.1896
55.56%
w/o aug.
0,2083
0.1931
75.16%
0.0
0.10
0.15
0.30
25
0.30
Identiy Similazity
Ours (single)
0.2234
0.2186
81.05%
(joint)
0.2545
0,2072
84.78%
Figure 6: Numerical analysis in terms of the prompt and identity similarity on four prompt types.
Table 2: Ablation studies.
Vi
Eccomstruction
playing buskecball
V.3
eccomstruction
agc+
pusc
age-
age, gender
gender
age-
ogen gender
pose
V2
agen pltee
plsee
posc
V2
2go' proee
Agiiy gendess powe:
posee
Figure 7: Our method not only robust to large age, gender, head pose, and light variants in the inputs, but also has good ability of controlling age, gender, and head pose in the outputs.
forgotten. Although dreambooth [10] and custom diffusion [8] can successfully generate the human and the concept, the generated identities are not the same as the target image.
Besides visual quality, we also perform the numerical comparison between the proposed method and baselines in Tab. 1. From the table, regardless of the over-fitted identity and detect scores but lowest prompt score), our method shows much better performance in terms of identity similarity and the face detection rate and achieves similar text-prompt alignment as Custom Diffsuion [8]. We also plot the generated results on four detailed types in Fig. 6, where the proposed method shows the best trade-off. Notice that, the proposed method only contains very few learning-abe able parameters and optimizes faster.
(highest
Moreover, since identity similarity is very subjective, we generate 200 images from different identities form user study. In detail, we invite 100 users rank the generated images from one (worst) to five (best) in terms of visual quality, prompt alignment, and identities, getting 60k opinions in total. The results also shown Tab. 1, where the favor our proposed method.
4.3 Results on Age, Gender, Head Pose, and Expression
We qualitatively evaluate the robustness of our method to input age, gender, head pose, and light variants in Fig. 7. Besides, modifying the text prompts can edit the age, gender, and pose of the results. We further provide the expression controlling results in the appendix, along with the comparisons with Face Composer [51]. Thanks to the identity disentangling ability of the face recognition [14] network, the learned representations are highly relevant to the input identities while irrelevant and insensitive to other attributes like age, pose, expression, etc. Furthermore, the learned representation as interpolation of celeb basis inherits the good characteristic of those "recognized" celeb names, which corresponds large amount of image-text paired samples [3] used for pretraining Stable Diffusion [1].

---

Input Prompt
Input Prompt
playing bashetball.
is plaping the guitur.
52
(Input Image)
wi 691
(Input Image)
Ours
Flatten basis
(a) of names in celeb basis. Figure Ablation studies on building celeb basis.
(b) First and last name basis v.s. flatten basis.
Isput Protupt
Esput Prompt
V" playing the galten
Vi and V2 Aaking bintdda cake.
playing
Bipput linage)
(iupot Imnege)
(a) Face encoder.
(b) Data augmentation
(c) Single v.s. joint optimization
Figure 9: Ablation studies coefficient optimization.
4.4 Ablation Studies
To further evaluate the sub-modules of our proposed method in both building celeb basis (Sec. 4.4.1) and the proposed new personalization method (Sec. 4.4.2), we start from the default settings ( "Ours (single)' in Tab. 2) of our method, conducting the ablation study by separately removing each submodule or using different setting as follows, Due to the space limitation, we give more visual results in the appendix.
4.4.1 Ablation Studies Celeb Basis
# of names in celeb basis. We evaluate the influence of the names to build celeb basis. In extreme cases, if there is no name and we directly learn the embedding from the face encoder F' (w/o celeb basis), the model is overfitted to the input image and can not perform the concept combination. With fewer celeb names (w/ 350 names). the generated quality is not good ours baseline (single) as in Fig. 8a. Besides, the quality of the celeb basis is also important, if we do not filter the names (w/o filter), the performance will also decrease as in Tab. 2.
Flatten basis v.s. first and last name basis. In the main method, we introduce our celeb basis as the first and last a more naive way by flattening all the embeddings to build the basis (w/ flatten). As shown in Fig. 8b and Tab. 2, the generated images of our first and last name basis understand prompts better.
basis since each
embedding does not have the length. We thus involve
Choice of reduction dimension p. We also evaluate the influence the number of coefficients Con- sidering the 768 dimensions of the CLIP text embedding, vary p ranging in (64, 256, 512, 768). Note that = 768 can be also considered as 'w/o PCA' since the dimension of text embedding is 768. As shown in Tab. 2, the best result is obtained from the baseline choice (p 512) and we show the differences in the generated images in the appendix.
4.4.2 Ablation Studies Coefficients Optimization
W/o face recognition encoder F. Naively, we can optimize the coefficients A₁, Ag of the celeb embeddings from back pockppropagttion directly. However, find the search space is still large to get satisfied results as shown in Fig. 9a and Tab. (w/o F). So we seek help from the pretrained face encoder, which has more discriminative features the face.
W/o data augmentation. Since there only one single image as the tuning-able sample, we perform some data augmentations as introduced in Sec. 3.2. However, if we remove these augmentations, the generated face becomes vague and unnatural as shown Fig. 9b, and the identity of the generated samples is also decreased shown in Tab. 2.

---

Ingut trawelingg through space
on the beach
Irput Cat
jumpping Skown
Manga drawing
the beacch
Pointiion painting
in Cheisttraa
Figure 10: Through a simple replacement on the encoder and text basis, our method can be extended to more general classes, e.g. car and cat.
Single v.s. joint training. Our method supports joint training of multiple identities using a single MLP, we evaluate the differences between single training/joint testing and joint training/joint testing. As shown in Tab. and Fig. 9c, although training individually can also perform some interactions between the two-person, training the images jointly improve the robustness and reduces the over- fitting risk compared with single training, resulting in slightly better results.
Conclusion & Discussion
We propose a new method to personalize the pre-trained text-to-image model on a specific kind of concept, i.e. the human being, with simply single-shot tuning. Our approach is enabled by defining a basis in the domain of the known celebrity names' embeddings. Then, we can map the facial features from the pre-trained face recognition encoder to reconstruct the coefficients of the new identity. Compared with the previous concept injection method, our method shows stronger concept combination abilities, e.g. better identity preservation. can be trained on various identities once, and can produce results where the newly-added humans interact with each other. Besides, the proposed method only requires 1024 parameters for each person and can be optimized in under 3 minutes, which is also much more efficient than previous methods.
Limitations. Although method successfully generate the images of new identities, it still occurs some limitations. First, the human faces generated by original stable diffusion [1] intrinsically contain some artifacts, causing somewhat unnatural performance of the proposed method (see our appendix). It might be solved by a more powerful pre-trained text-to-image model (e.g. Imagen [5], IF [52]) since they can generate better facial details. Second, we only focus on human beings currently. It is also interesting to build the basis of other species, e.g. cars, and cats. To this end, we provide some rough experiments (whose implementation details are introduced in the appendix) of extending our method to cars and cats, as shown in Fig. 10. The results show that our method can also generate text-aligned images with good quality without bells and whistles. But the simple try could be improved through using stronger encoder or craw wawling more text to construct basis. Third, although the text alignment scores evaluate the quantitative performance, for the editing ability of age, expression, and pose, we do not have metric result yet. As supplement, we will use facial feature extractors to quantitatively compare the human identity personalized tasks the future. Fourth, similar to the inference-only method FastComposer [51], we have also tried to train model on 50k identities (20 faces per identity without text labels) as an encoder- based method. However, it is hard to converge in short GPU time within batch size of cight. We think it might be because need much more data and computing resources to train a unified or general facial model and we will explore it in the future.
Ethics Consideration and Broader Impacts. We propose new method for model personalization on the human face, which might be used as tool for deepfake generation since we can generate not only the human face but also interactions between people. However, this prevalent issue not limited to this approach alone, as it also exists in other generative models and content manipulation techniques. Besides, our personalization person generation method can also ablate the abilities the erasing concept methods [53] and other deep fake detection methods [54].
10

---

6 Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 61976231 and in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2023A1515012853.
References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2021.
[2] Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial networks. In CVPR, pages 4401-4410, 2019.
[3] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wight- man, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5B: An open large-scale dataset for training next generation image-text mod- els. arXiv:2210.08402, 2022.
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, pages 8748-8763, 2021.
[5] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NIPS, pages 36479-36494, 2022.
[6] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv:2204.061 2022.
[7] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv:2208.00 01618, 2022.
[8] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and lun-Yan Zhu. Multi- concept customization of text-to-image diffusion. In CVPR, pages 1931-1941, 2023.
[9] Ziyi Dong, Pengxu Wei, and Liang Lin. DreamArtist: Towards controllable one-shot text-to- image generation via contrastive prompt-tuning. arXiv:221..11337 2022.
[10] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dooth:ooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv:2208. 12242, 2022.
[11] Rinon Gal, Moab Arar, Yuval Atzmon, AmitH. Bermano, Gal Chechik, and Daniel Cohen- Or. Designing an encoder for fast personalization of text-to-image models. arXiv:2302. 12228, 2023.
[12] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3D faces. In SIGGRAPH, pages 187-194, 1999.
[13] Karl Pearson. LIII. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, 1901.
[14] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. ArcFace: Additive angular margin loss for deep face recognition. In CVPR, pages
2019.
[15] Ian Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 631114)139-144 2020.
[16] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv: 2013.

---

[17] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NIPS, 2017.
[18] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873- 18873, 2021.
[19] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In NIPS, pages 1-11, 2019.
[20] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, pages 8821-883 2021.
[21] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In NIPS, pages 1-10, 2018.
[22] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. 1:Xivv/410.880 2014.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NIPS, pages 6840-6851, 2020.
[24] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502, 2020.
[25] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval- augmented text-to-image generator. wwXh::229.1449/ 2022.
[26] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, pages 18208-18218, 2022.
[27] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. arXiv:2206.02779, 2022.
[28] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint-by-example Exemplar-based image editing with diffusion models. arXiv:2211.13227,, 2022.
[29] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv:2211.09794, 2022.
[30] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Yan Zhu. Zero-shot image-to-image translation. arXiv:2302.0330 0000000.7 2023.
[31] Chenyang Qi. Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv:2303.0m 09535, 2023.
[32] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv:2208.0162. 2022.
[33] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, pages 81 11081111 19, 2020.
[34] Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or. Mystyle: personalized generative prior. ACM Transactions Graphics, 41(6):1-10, 2022.
[35] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for lasent-based editing of real images. ACM Transactions Graphics, 42(1):1-13, 2022.
[36] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2Stylegan: How to embed images into the stylegan latent space? In CVPR, pages 4432-4441, 2019.
12

---

[37] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or Or. Encoding in style: A stylegan encoder for CVPR, pages 2287-2296, 2021.
translation. In
[38] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv:2210.09276, 2022.
[39] James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual Diffusion: Continual customization of text-to-image diffusion with C-LoRA. arXiv:2304.06027, 2023.
[40] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. 2rX11v2305.01644, 2023.
[41] Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, and Jiaying Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation. arXiv:2303.09319 2023.
[42] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. vrXiv:2302.13845 2023.
[43] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming-Wei Chang, and William W Cohen, Subject-driven text-to-image generation via apprenticeship learning. arXiv:2304.00186 2023.
[44] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. w:Xh::230.00262,, 2023.
[45] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instanthooth: Personalized text-to-image generation without test-time finetuning. arXiv:2304.0344 2023.
[46] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4D scans. Transactions on Graphics, 36(6):14:11194:111 2017.
[47] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao. FaceScape: A large-scale high quality 3D face dataset and detailed riggable 3D face prediction. In CVPR, 2020.
[48] Kwang In Kim, Keechul Jung, and Hang Joon Kim. Face recognition using kernel principal component analysis. IEEE Signal Processing Letters, 9(2):40-42, 2002.
[49] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXivv1810.04805, 2018.
[50] Hongyu Guo. Nonlinear mixup: Out-of-manifold data augmentation for text classification. In 2020.
AAAI, pages
[51] Guangxuan Xiao, Tianwei Yin, William T. Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning-free multi-subject image generation with localized attention. arXiv:2305. 10431, 2023.
[52] Deep Floyd. http:/giihubbcoom//ee/fffffoo///
2023.
[53] Rohit Gandikota, Joanna Materzynska, Jaden Kautt-,, and David Bau. Erasing concepts from diffusion models. a:X1v:2303.07335,, 2023.
[54] Yuchen Luo, Yong Zhang, Junchi Yan, and Wei Liu. Generalizing face forgery detection with high-hrequency features. In CVPR, pages 16317-16326, 2021.
[55] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. VggFace2: A dataset for recognising faces across pose and age. In FC, pages 67-74, 2018.
[56]
13

---

A More Qualitative Results
A.1 Evaluation on Faces in a Wider Range
Besides evaluating the results on StyleGAN synthetic faces as in the main paper, we compare our method with state-of-the-an methods on a wide range of human face images, including the real word faces, the faces from different races, and the interactions of the same gender. These face images are from VGGFace2 [55] and collected from the Web.
Single Person in an Image. Fig. 11 shows the evaluation results on a single person's 's personalization for real identities. Concretely, Row 1-2 shows the ability to generate diverse images of different methods. The baselines Textual Inversion [7]. DreamBooth [10], and Custom Diffusion [8] can generate identity-consiseent results with the single prompt 'A photo of V* but the light condition and skin texture seem not natural. DreamBooth and Custom Diffusion fail to disentangle the background of the target inputs. The stylization results are shown in Row 3-4, where Textual Inversion preserves the target identities but fails to change the style, DreamBooth, and Custom Diffusion generate the images of the corresponding style but lose the identity information, and our method preserves the identity and completes the stylization as given prompts.
Multiple Persons in an Image. We give the comparison of generating multiple-person interaction in Fig. 12. Similar to the single-person results, the proposed method shows good concept combination ability among different persons (including the newly added and the original celebrities) and can serve as novel concept to communicate with other humans.
A.2 Additional Results Comparing with Concurrently Work: FastComposer
Different from the methods [7, 10, 8] that the finetuning time cost for each identity is below 30 minutes, FastComposer [51] needs to be pre-trained on a large human dataset, requiring 150k steps with batch size 128 on 8 NVIDIA A6000 (48GB) GPUs. So as its performance restricted by the dataset as in their limitation. Differently, our method requires only 400 steps with batch size 2 on a single NVIDIA A100 (40GB) GPU, After the pre-training stage, FastComposer can generate images based on the given single image and prompt like other personalized methods.
To further demonstrate the efficacy of our method, evaluate the qualitative performance of our method compared with FastComposer. Our method outperforms FastComposer on single action con- trolling, the interaction between two persons, and expression controlling. The detailed comparisons are as follows.
Action of Single Person. We first evaluate the performance of single persons' personalization. As illustrated Fig. 13 and Fig. 14, we show the synthesized results under six scenarios in each figure.
In Fig. 13, given the prompt 'A photo of Vi"', FastComposer generates an unnatural result. In the remaining scenarios, the light seems very disharmonious in the images synthesized by FastComposer. Besides, in these results, only the upper chest and head can be generated, where the human body and limbs fail to appear, revealing an over-fitting issue of FastComposer.
In Fig. 14, FastComposer meets the same issue. Although the key objects and face identity are almost consistent with the input prompts, e.g. LEGO toys, fighter jet, and yellow jacket, the results from FastComposer seem to be a rude combination of the objects and face. Besides, their methods have a huge issue with concept forget. Some important prompts like 'sweater' and 'motorbike' are ignored by FastComposer.
We also highlight the faces generated by FastComposer share very similar expressions from the original input image, i.c. dilated pupils, wrinkles on the forehead, and mouth shape, which means this method is overfitted on the input image. Differently, our method generates identity- and prompt- consistent results, appearing natural and realistic.
Interaction between Two Persons. Fig. 15 illustrates the interaction and common action of two persons a single image. Sharing the similar aforementioned over-fitting problem, FastComposer only generates the face part of the human. As for the actions including 'shaking', 'playing', 'sitting', and 'baking', , FastComposer fails to generate the correct behaviors, where the faces usually occupy the most part of the images. The problem of invariant expression also occurs in FastComposer.
14

---

Textual Inversioin
Custom Diffusion
Target Image
DreamBooth
Ours
Vi
A photo of Vi
USe
AC
V2
photo of V
V3
Vi in the style of Archer
ER
V
V by Andy Warhol
AEROO 25
18
VS
V is playing basketball
V in Chinese vegetable market
Figure 11: Single person's personalization result for real identities.
Expression Controlling, Considering that the above FastComposer results struggle to generate the different expressions, we conduct face expression controlling experiment to validate the abilities of method and FastComposer. We the following prompts to change the expression of the target input image:
4 is smiling with happiness
y* is crying with sadness
15

---

V has an anger expression
V looks very surprised, looking a gift box
V is shaking with fear scared by a candle lighting, scared expression
V has tired expression, headache, uncomfortable, a frown creased her forehead
As shown in Fig. 16, the proposed method can successfully generate different expressions under the text prompts, which shows the advantage of the proposed method.
A.3 Uncurated Results
We show the uncurated results of our method in Fig. 17, which demonstrate that our method can generate stable and text-aligned images without being cherry-picked.
B
Implementation Details
B.1
Celeb Images Generated by Stable Diffusion
In general, the celeb images generated by Stable Diffusion [1] and those realistic ones collected from Google Images seem very similar, but with little difference on facial details. As seen in Fig. 18, we compare the real celeb images collected from Google Images with the fake ones synthesized by Stable Diffusion. Overall, the real and fake images closely resemble the same individual. However, certain facial features in the fake images may be exaggerated, which can constrain method's 's performance and explain why the celebrity profiles in the outcomes generated by both baseline models and our approach do not precisely match their real counterparts.
B.2 Collecting and Filtering Celeb Names
After crawling about 1,500 celebrity names, to filter out the names that have the ability generate prompt-consistent identities and interacting with other celebs, we feed three types of prompts, i.e. 'A photo of V.', 'V. is playing the guitar', and 'V. talks with Barack Obama', the Stable Diffusion [1] for synthesizing image results, where Ve indicates the celeb name (e.g. 'Anne Hathaway'). Fig. 19 shows the examples that satisfy the filtering condition and the ones that fail generate reasonable results. Consequently, 691 names pass the checking, which can be tokenized and encoded into m 691 celeb embedding groups.
The statistics these 691 names are shown in Fig. 20. Specifically, we count the frequency of each celeb name in the LAION-2B dataset text samples. LAION-2B is subset of LAION-5B [3] where Stable Diffusion [1] is pretrained. After sorting the celeb names according their frequency, we obtain Fig. 20, where the x-axis is the index of the name, the y-axis is the appearing time of this name in the dataset. This plot indicates that our collected celeb names correspond to thousands image-text paired samples in average. There are only 25 names whose frequencies are less than 100 and 173 names whose frequencies are larger than 10000. This frequency distribution shows that the distribution of the celeb names is relatively fair.
B.3 Building Celeb Basis based on PCA
We only keep the first and second embeddings of each celeb embedding group, resulting the first name embedding set C1 and the second name embedding set C2, where C1. C2 E mxd Then, for simplicity, we omit the subscript of Ce, using C indicate any one of C and Cz.
Calculation of Mean. Considering C' may have repetitive embeddings (each row corresponds an embedding vector), for each C, we first remove the duplicate rows that come from the same token to make sure each token only occurs once at most. Then calculate the mean C E RRd across the second dimension of C' as mentioned in the main paper.
PCA. The PCA algorithm has many different coding implementations in practice. In our method, we use Singular Value Decomposition (SVD) to skip the calculation of covariance matrix PCA. Please refer to [13, 56) for more detailed theoretical demonstration and coding techniques. Due to the built celeb basis not optimized, the PCA process only needs to be computed once during the training stage.
16

---

B.4 Training Recipe
We train the MLP with learning rate of 0.005 and batch size of on single NVIDIA A100 GPU. The training augmentation includes horizontal flip, color jitter, and random scaling ranging in 0.1 ~ 1.0. For single identity training, the optimization costs 400 steps, taking identities joint training, we found that training 2,500 steps is enough, taking ~ 18 minutes (averaged about 250 steps and 108 seconds for each identity). The text prompts for training include:
minutes. For 10
A photo of face of V person
A rendering of face of V* person
The photo of a face of V* person
A rendition of a face of V* person
A illustration of a face of V* person
A depiction of a face of V* person
C More Ablation Studies
W/o filtering celebrity names. As mentioned in Section B.2, Stable Diffusion fails to generate correct images from some celeb names. our method, we manually drop these bad samples. If this filtering process canceled, the interaction ability of the learned identities drops lot, as shown in Fig. 21a.
Variants of reduction dimension In our method, the PCA reduction dimension (p < k) controls the degrees of freedom for modulating the variance applied to the celeh basis mean. Considering that k 768 in the CLIP [4] text encoder, we conduct series experiments to study the choice of p. Four values of p are chosen, including 64, 256, 512, 768. Note that = 768 means PCA is not used. Larger p costs more memory storage. As shown in Fig. 21b, chosing p 512 comes to the best identity similarity, which consistent with the quantitative results our main paper.
Variants training images per identity N. Our method focuses on one-shot identity personalization problem (i.e. training from a single photo to represent a newconcept). To verify the influence of training images count per identity. we provide the results in Fig. 22, where the inputs varies in view and lighting condition. shows that when we use more images, the face quality can also be slightly improved, and the synthesized results are already pretty good when == =1 1.
D Extending to More General Classes
Our method have potentials to be extended to non-face classes. Here, we provide some rough experimental results in Fig. 10. For non-face samples, trivially replace the face recognition encoder with a pre-trained ResNet50 (trained on ImageNet-1K) and construct the "car" (or "cat") basis by collecting typical types of cars (or breeds of cats) in phrases, e.g. 'Volkswagen Passat' and 'Mercedes-B Benz A-Class' (or 'Scottish Fold' and 'Abyssinian' for cats). Without other bells and whistles, method generates text-aligned images with good quality of cars and cats. In the future, we would try a better general encoder (to replace ResNet50) and crawl a larger basis for non- non-face classes, extending method for more general subjects using our well-collected basis names.
17

---

Textual Inversioin
Custom Diffusion
Target Image
DreamBooth
Ours
V
V and Rachel McAdams baking birthday cake
V/2
shakes hands with Chadwick Boseman conference
HOPA
V3
V4
A V3 Funko Pop next to V Funko Pop
V5
Vg
Manga drawing of V/ and Vi
V
Vg
VG and V are cooking in kitchen
Vg
Vi
V Vio are sitting on sofa
Figure 12: Multiple persons' personalization result for real identities.
18

---

Wearing the sweater outdoors
A photo of V
Playing the guitar
Wearing a scifi spacesuit in space
Currying vegetables in vegetable market
On a beach
V
FastComposer
Wearing the sweater outdoors
A photo V
Playing the guitar
Wearing a scifi spacesuil in space
Currying vegetables in vegetable market
On a beach
Figure 13: Additional single person's 's personalization results comparing with FastComposer [51].
19

---

Ours
Wearing the sweater outdoors
A photo Vf
Playing the LEGO toys
Wearing a scifi spacesuit space
Wearing yellow jacket on motorbike
Piloting fighter jet
Vi
PastComposer
Wcaring the outdoors
A photo of Vf
Playing the LEGO toys
Wearing a scifi spacesuit space
Wearing yellow jacket on motorbike
Piloting fighter jet
Figure 14: Additional single person's personalization results comparing with FastComposer [51].
20

---

Ours
In a kitchen
In an amusement park
Shaking hands
V
Playing the piano
Sitting sofa
Baking cake
FasiComposer
Vz
In kitchen
In umusement park
Shaking hands
Playing the piano
Sitting on a sofa
Baking a cake
Figure 15: Additional multiple persons' interaction personalization results comparing with FastCom- poser [51].
21

---

Ours
Happy
Sad
Angry
Surprised
Scared
Worried
Target Image
Castoompser
Happy
Sad
Angry
Surprised
Scared
Worried
Figure 16: Comparing with FastComposer [51], our method has better ability of controlling face expression.
22

---

V/¹
Laboratory
V2
wearing the sweater and backpack, outdoors
wearing a scifi spacesuit
Figure Uncurated results in a testing batch.
ES CES
Google Images
P CS
HR GH
CE
Stable Diffusion
Anne
Barack
Robert
Rachel McAdams
Chadwick Boseman
Hathaway
Obama
Downey
Figure 18: Stable Diffusion has the ability generate identity-consian consistent celeb images. But in the aspect of facial details, the celeb images synthesized by Stable Diffusion may exaggerate some facial features compared with the real ones collected from Google Images.
23

---

Satisfied Names
No Used Names
Google Images
A photo of V
1. is playing
the
W
Barack Ohama
V. talks
Benedict
Zooey
Rowan
Cumberbatch
Cate Blanchett
Deschanel
Alkinson
Sheeran
Stephen Chow
Figure 19: We filter out the celeb names that can generate prompt- and identity-consisten images through Stable Diffusion [1]. Left: satisfied names that can generate images share the same identity with Google Images results. Furthermore, these names have the ability to interact with objects or other satisfied celebs (e.g. 'Barack Obama'). Right: Stable Diffusion confuses the face of "Rowan Atkinson' and 'Barack Obama' in four. The gender of 'Sheeran' is mistaken girl in row four. The identity of 'Stephen Chow' has large gap with that of Google Images results.
10*
Frequency
10
10a
10a
10a
0
60
120
180
240
300
360
420
480
540
600
Sorted Index of Celeb Names
Figure 20: The frequency statisties of our used celeb names in LAION-2B [3] (a subset where Stable Diffusion trained).
24

---

Input Prompt
V' is playing basketball.
ATTOW
V* (Input Image)
Ours
w/o filter
(a) The influence of filtering the useless celeb names not.
Input Prompt
V' wearing scifi
spacesuit space.
= 64 (0.25 x)
= 256 (0.5 x)
= 512 (1. x, Ours)
768 (1.5 mux)
V" (Input Image)
(h) The ablation the values of p.
Figure 21: Additional ablation studies on building celeb basis and the choice of reduction dimension p. Note that in our method, the text embeddings are 768-dimensions, which means p 768 is equivalent to the method without PCA.
Training Sumples
N 1=1
5
N
20
Figure 22: Additional ablation studies on the number of training images identity N.
25


Reviews:
[{"review_id": "dw8Rqs62bk", "reviewer": "Reviewer_ABqC", "summary": "The paper proposes a new method to personalize the pre-trained text-to-image model on humans. The propose a novel idea called Celeb basis and it simplifies the problem to reconstructing the coefficients of any new identity. Both quantitative and qualitative results demonstrate the effectiveness of the proposed method on concept composition compared with the baselines. ", "questions": "will the codebase be publicly available and when? ", "limitations": "I think authors have adequately discussed the limitations in terms of the artifacts of the real human faces of the original stable diffusion model and other species. Potential negative impact has also been discussed. ", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "+ It's a very smart idea to utilize Celeb Basis from the text embedding space of the celebrities’ names in the text-to-image model. With the low dimension subspace, the learning problem boils down to very few coefficients, which is simple yet effective. \n\n+ The experimental results speak for the paper itself. It shows more stable concept composition abilities than previous works. \n\nI especially like fig 5 where the motivation of the proposed method is clearly demonstrated and easy to see where the previous methods fall short on. \n", "weaknesses": "N/A ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 1. Responses to R1 (ABqC) \n\n**R1Q1.** ***Will the codebase be publicly available and when?***\n\nWe thank you for your advice. Our codes (including the whole training and testing process) will be publicly available on GitHub as soon as the NIPS notifications are released. We are making the code easier to be used. A project webpage, a GitHub repository, a Google Colab demo, and a HuggingFace workspace will be all publicly available. Additionally, due to our method can generate features compatible with Textual Inversion [7], we will also work on developing a celeb basis as a plug-in extension of commonly used Stable Diffusion [1] WebUI. \n\nWe sincerely expect your future experiences with these resources. And please feel free to give us feedback and advice if you have any problems using these resources.\n"}, {"review_id": "D6Pzhi7Jrz", "reviewer": "Reviewer_zJJ7", "summary": "The paper presents a method to invert input portrait images to latent diffusion model (LDM)'s embedding space and then use the same to create text-based identity preserving compositional portrait images. To enable image inversion, the authors create a novel active appearance model \"Celeb Basis\" (image-only identity PCA) from StyleGAN-2 style frontal images. Later, this \"Celeb Basis\", and a textual embedding are jointly used to train CLIP for identity preserving (ArcFace style loss) image synthesis. At test time, new images of the inverted target image (represented by the Celeb Basis) can be generated based on textual prompts.\n\nThe authors presents ablation studies wrt prompt and identity preservation (and detection) and compare against SOTA methods \"An image is worth one word\" (Textural Inversion), Dreambooth and Multi-concept customization of text-to-image diffusion (Custom diffusion) and achieve good results for prompt-based facial images.\n\nNote, that while other methods are not specific for portrait images, the current Celeb Basis has been built for portrait style images only. ", "questions": "- The data augmentation technique is essentially used color space augmentations. Did the authors consider any geometric augmentations? Since, the image inversion techniques in the StyleGAN space are quite mature (eg HyperStyle), adding augs that change the pose or expression would make the method further robust?\n- How were the results shown in the paper selected? Was it the first image that the method created used or several prompt tries with subtle textual changes made to pick the results shown in the paper?\n- Did the authors consider the use of multi-view images for identity reconstruction?", "limitations": "Novelty: The main novelty of the approach is introduction of celeb-basis, while the rest of the method seems straight forward. Since the results are only shown for a facial model, it scope limits the contribution of the work. Please rename the work from \"Inserting Anybody\" to \"Inserting Anybody's Portrait\". As such control over bodies, etc. is not part of the paper's contribution.\n\nExpression control: While the basis does not model expressions in the 3DMM sense or provides a basis for expressions, the method seems to provide control expression control via prompts (supp. comparison against FastComposer). This seems to be an after thought and it's unclear how well the expressions are enabled by the method. In order to fully understand the expression control capabilities, please use a ground truth dataset (such as https://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html) and provide comparison against ground truth expression vs those generated by prompts. This would also help understand how well the expressions can be re-created. \n\nOverall, I think this is an interesting paper and the results are generally good; the paper's novelty is limited since it enables only identity inversion, and later dependent on CLIP/LDM's capabilities to create pose, expression or semantic variations (e.g. an older V_1), etc. So, there's merit in the contribution. Besides the points/questions mentioned earlier, the authors should expand the limitations, move FastComposer results to the main paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "While the idea of creating an active appearance model, Cootes et al. (Celeb basis) is quite established in the vision community; this paper attempts to \"rig\" (e.g. StyleRig, Tewari et al.) the LDM space to create new prompted facial image given target image (identity preservation only). This is the main contribution of the paper and is novel in that regard, while the idea of controlled prompt-based image synthesis via LDM has been explored before. \n\nThe celeb basis thus enables more control over image synthesis, where the semantic of \"identity\" preservation helps with a more _semantic_ control and results in better compositional results (two target images used together). These compositional results analysis (Fig 5, last two row) are mostly visual since I look at the overall image to argue if it looks more \"believable\".\n\nThe paper is generally easy to follow and extensive ablation studies justify the various modeling decisions made. ", "weaknesses": "- The range of \"Celeb basis\": Since the method used PCA and uses a limited set of image to create the basis, it's limited by the choice of images, which is further limited by the choice of named celebrities in the LDM model itself. For example, it's unclear how well the model will represent certain age, pose and expression ranges.  \n\n- Table 1 shows that while the users prefer the author generated images on Identity and Detect metics, Textual Inversion outperforms them, while that method is not specific to facial images as such. The reason argued is that Textual Inversion outfits. Given that Textual Inversion is a generic method, while the current method is meant for facial images, this result is surprising. \n\n- Comparison against FastComposer: Since most results presented against a concurrent work \"FastComposer\" seem cherry picked, the presented comparison seems biased. For example, the result presented in Fig 1 (FastComposer) seem better than those presented in Supp (Figure 5). Moreover, the current method does not explicitly model expression control in their Celeb basis, and their Fig 6 comparison seems less principled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 2. Responses to R2 (zJJ7)\n\nWe thank you for your careful reading and helpful suggestions. We respond to your concerns below:\n\n**R2W1.** ***Using limited celeb names to represent a person may constrain the ability to control age, pose, and expression.***\n\nPlease see our global response to **GQ1** and **GQ2**, where we have answered in detail.\n\n**R2W2.** ***Textual Inversion (TI) [7] results in the `main paper Table 1` are surprising.***\n\nAs shown in `Figure 5 of the main paper`, TI [7] can only generate images similar to the training image and ignore the additional text prompt. Thus, TI [7] performs better on identity and detection metrics since the output image is very similar to the input one (large detailed face), which is easy to be identified. However, we argue that text-image alignment is critical for our text-driven image synthesizing task. As shown in `our main paper Table 1 and Figure 5`, the synthesizing results of TI [7] are often very similar to the input images, hardly controllable by the input text prompts. The detailed quantitative editing ability comparison in `our main paper Figure 6` also demonstrates this phenomenon.\n\n**R2W3.** ***The comparison against FastComposer [Supp. 5] seems cherry picked and less princinpled.***\n\nWe follow a fair testing protocol (detailed in the later response to **R2Q2**) to compare all the methods.  `Supplement Figure 4-7` compares our method and FastComposer (FC)'s [Supp. 5] the ability to control a single person and compose multiple persons. Concretely in the experiments, for each text input, we run the FC [Supp. 5] online demo eight times and select the best one (with the highest text-image alignment score) as the compared result. \n\nDuring experiments, we found that FC [Supp. 5] has three main weaknesses: \n\n1. hard to generate full body;\n2. hard to make personalized identities interact with each other; \n3. over-consistent facial expression and head pose. \n\nApparently, the results in `Supplement Figure 4-7` demonstrate that our celeb basis outperforms FC [Supp. 5] on these aspects.\n\n**R2Q1.** ***What if adding augs that change the pose or expression?***\n\nThese augmentations may distort the original identity. \n\nFor StyleGAN [2] synthesized faces as input, in our practice, shifting the latent vectors of StyleGAN [2] may change the pose and expression of a face, but also unavoidably brings identity distortion, leading to worse results. Besides, using other face editing methods might improve the performance, but brings massive training overhead. Consequently, without these complex augmentations, our method performs well even using a very simple augmentation scheme (flip, resize, and color jitter).\n\nFor realistic faces, instead of introducing other face editing methods to augment a single image, directly increasing the number of images per person seems to be a better choice, which has been detailed in **GQ3**.\n\n**R2Q2.** ***How were the results shown in the paper selected?***\n\nWe pick the best 1-3 images having the highest text-alignment scores from a single inference batch (with batch size 8, and differently, we use batch size 6 in the `Rebuttal PDF` limited by the NIPS rebuttal due). This testing protocol is used for all the methods including our celeb basis and all the compared methods. The testing protocol can be vised as a peak performance comparison and we will update this information in the revised paper.\n\nWe also list the uncurated samples synthesized through our method in `Rebuttal PDF Figure 3`.\n\n**R2Q3.** ***What if using multi-view images for identity reconstruction?***\n\nPlease see our response to **GQ3**, where we provide the results.\n\n**R2Q4.** ***Rename the title to \"Inserting Anybody's Portraits\".***\n\nWe thank you for your advice and will take it into account to rename a better title for our work. Note that although our method only takes as input face images, the synthesized results contain not only the identity-consistent faces but also half/full bodies, different actions/motions, variant age/pose/expression/gender, and multiple persons' interaction, all driven by text prompts. This can be considered that we successfully insert a given identity (with only a face) into the \"identity\" space of the diffusion model. The newly inserted identity shares the same good editability with those recognized persons (celebrities), as mentioned in **GQ1**.\n\n**R2Q5.** ***Please provide more detailed expression controlling validation results, e.g. based on the Multi-Pie groundtruth expression dataset?***\n\nWe sincerely appreciate your advice. The later response to **R2Q6** will answer to this question in detail.\n\n**R2Q6.** ***Expand the limitations and move FastComposer [Supp. 5] to the main paper.***\n\nThanks for your suggestion. You mentioned that our method has two main limitations: \n* (L1) lacking quantitative metrics above expression controlling; \n* (L2) only supporting identity inversion. \n\nFor (L1), we believe the text alignment metrics already evaluate the quantitative performance of text-to-image methods. And as a supplement to this, we'll use Face Age/Expression/Pose feature extractors to further validate the controlling ability for human identity personalized tasks in the future.\n\nFor (L2), please see our global response to **GQ4**, where we extend our method to \"car\" and \"cat\" classes. We'll take this insight as future work.\n\nWe'll add more detailed limitations of our method and more complement comparisons with the concurrent work FC [Supp. 5] in the final version. "}, {"review_id": "52e7J1qLn7", "reviewer": "Reviewer_MXrx", "summary": "This paper introduces an innovative method for generating highly personalized images utilizing a single reference image and distinct textual prompts.The central idea revolves around identifying a subspace within the text encoder embedding space, which, when combined, can represent any facial images. This is accomplished by applying PCA to the CLIP text embedding of hundreds of celebrity names. Subsequently, a single MLP is optimized to compute the combination weights using the features of a new individual, leveraging the diffusion reconstruction loss. The proposed approach's effectiveness is demonstrated through the results on both single and multiple subject generation tasks. The method significantly outperforms previous baselines in user studies, delivering superior qualitative results with notably enhanced diversity\n\n\n", "questions": "Despite the shortcomings identified above, I appreciate the overall simplicity and effectiveness of the proposed approach. Consequently, I am inclined to recommend acceptance of the paper.", "limitations": "See weakness sections. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The paper's innovative approach to identifying a text embedding subspace stands out. This subspace allows for an effective projection of subject-specific facial data while remaining compact, thereby avoiding overfitting to the reference subject.\n\nS2: The experimental sections of the paper are comprehensive, providing detailed comparisons via quantitative, qualitative, and user studies, along with insightful ablation studies.\n\nS3: The proposed method achieves a good balance between identity preservation and prompt consistency, outperforming popular baselines such as textual inversion and DreamBooth.\n\n\n\n", "weaknesses": "W1: The authors compute the textual inversion subspaces via PCA on textual embedding of celebrity names. However, celebrities might not be a representative sample of ordinary people and could inherently carry biases related to racial and geographical factors. This might bias the subsequent image generation process.\n\nW2: The method employs subject inversion by optimizing weights in the celeb-basis. Nevertheless, the diffusion model remains static, which may result in generated samples occasionally deviating from the reference subject.\n\nW3: The method is still optimization based, and requires costly back propagation, which might be infeasible as the diffusion model gets larger and larger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 3. Responses to R3 (MXrx)\n\nThank you for the careful reading and useful advice. We respond to your questions below:\n\n**R3Q1.** ***Using celebrities to represent ordinary people may bring bias?***\n\nOur global response to **GQ1** explains why limited celebrities can represent any ordinary person. Also as shown in `Rebuttal PDF Figure 1`, our method is robust to the age, head pose, gender, and light conditions with little bias.\n\n**R3Q2.** ***Freezing the diffusion model leads to the deviation?***\n\n\nFinetuning the diffusion model potentially brings the forgetting problem [10,8]: some important concepts may be \"forgotten\" after modifying the diffusion model weights. In our method, due to the good disentangling ability of the face recognition encoder [14] (pre-trained on large-scale face recognition datasets) and powerful priors contained in celeb basis (corresponding to over $700{\\rm{K}}$ samples as shown in `Rebuttal PDF Figure 2`), we believe that learned features have strong ability to precisely represent any given person and generate text/identity-aligned images. The additional results in `Rebuttal PDF Figure 1` and uncurated results in `Rebuttal PDF Figure 3` further demonstrate this.\n\n\n**R3Q3.** ***The optimization-based method is infeasible as the diffusion model getting larger.***\n\nWe agree with this opinion. However, this is indeed a common limitation of most personalization methods as far as we know. Very similar to FC [Supp. 5] which pre-trains a unified human identity feature extractor on text-image paired dataset within batch size 128 and 150k steps on 8 NVIDIA A6000 GPUs, we have also tried to train a model on 50k identities (20 faces per identity without text labels) as an encoder-based method (inference-only). However, it is hard to converge in a short GPU time within a batch size of 8. We think it might be because we need much more data and computing resources to train a unified face representation and we will explore it in the future. \n\nApart from this, we would also like to highlight that our method is still state-of-the-art regarding two types of personalization methods (encoder-based and optimization-based). For encoder-based methods (e.g. FC [Supp. 5]), we utilize the pre-trained knowledge of Stable Diffusion [1] (i.e. celebrity names) as priors, which is still more efficient than training a domain encoder. As for the optimization-based methods (e.g. TI [7], DB [10], CD [8]), our method requires quite low computing cost ($1{\\rm{K}}$ learnable parameters under $3$ minutes) and can be easily extended to a new class (see **GQ4**)."}, {"review_id": "PJOgRrPiMu", "reviewer": "Reviewer_Qr1K", "summary": "The manuscript with title: Inserting Anybody in Diffusion Models via Celeb Basis presents an interesting and novel way to insert identity to pre-trained text-to-image models. The paper is well-written and has clearly delivered the main concept. This approach only requires a single image with few trainable parameters, which opens a new avenue for personalized text-to-image translation.", "questions": "1. My key question is that if we want to apply this framework to a more generalized text-2-image generation, besides face generation, what do we need to do, how important is the celebrity basis. - lets say a simple experiments, what if we input a car image, where we can no longer accurately represented by the basis, what will the results be looking like?", "limitations": "1. The main limitation of this work is that it only focuses on face generation, with good alignment. This limitation may plaque the application of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents an interesting and novel way to perform personalized text-to-image diffusion model with only one image. The key idea behind this is to map the image features to a pre-defined basis from Celebrity dataset, which is novel and works well.\n2. The authors conducted a fairly amount of ablation studies, indicating the roles of each submodules, I appreciate that.\n3. The results demonstrate improved visual quality compared to DreamBooth and Custom diffusion.\n", "weaknesses": "1. One key point that I'm concerned about is how sensitive is the method to different face positions or image quality. To construct the basis, I assume that the dataset faces are well-aligned and more uniform. However, in the real cases, those conditions not necessarily hold, a face can be tilted, not well aligned, however most of the examples in the paper and the supplementary material have the input face with good alignment. I wonder if the authors can share some face generation results with poor alignments. and how to handel this issue.\n\n2. Also I noticed that quite a large amount of the inference results are also from the celebrity dataset (correct me if I got it wrong), would appreciate more results from more personalized images - also curious about the results that are different from the celebrity dataset, for example photos of babies.\n\n3. Lack ablation studies on the number of images.\n\n4. I assume this framework can be generalized to not face only text-to-image generation, as long as we have a good basis, can you elaborate on this? or showing some examples of non-face image generation, for example can this framework be generalized to personalized car generation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 4. Responses to R4 (Qr1K)\n\nThanks for your helpful suggestions. And we respond to your questions as follows:\n\n**R4Q1.** ***The method relies on well aligned faces. How sensitive is the method to different face positions or image quality?***\n\nFace detection and alignment [14] is very commonly used in human face-related projects, e.g. StyleGAN [2], InsightFace [14], GPEN, and a concurrent work FastComposer [Supp. 5]. This preprocessing step is pretty low-cost, usually achieving $30+$ FPS on CPU or $200+$ FPS on GPU and having only about $10{\\rm{M}}$ parameters. Besides, in many practical scenarios, only facial parts of a user can be collected, or even only a single face image. Our method requiring only one face image for each identity is acceptable for real-world application.\n\nBesides, as stated in **GQ1** and **GQ2**, `Rebuttal PDF Figure 1` shows the robustness of our method to different head poses, ages, light conditions, etc. Even given a profile face, our method can generate its frontal face or other head poses just by inputing the corresponding prompts. The robustness and good editability come from the identity disentanglement of our method. In specific, the face recognition encoder [14] pre-trained on large face datasets is robust to gender/age/pose/expression/light/etc. variants of input faces. And the celeb basis space gives good text-driven controllability to the learned representations. \n\n**R4Q2.** ***A large amount inference images are also from the celebrity dataset. Please provide more results on more personalized identities, like babies.***\n\nFirst, we gracefully point out that *\"inference images also from the celebrity dataset\"* is incorrect. To avoid the model directly re-using the celeb embeddings, we use StyleGAN [2] synthesized images (noted as **fake** images) as the personalizing targets to compare our method with other methods. These **fake** input identities are irrelevant to the celebrities, which can be considered as a nobody or ordinary person that Stable Diffusion [1] has NOT seen before.\n\nSecond, our `Rebuttal PDF Figure 1` additionally shows the experimental results on babies, teenagers, and elders.\n\n**R4Q3.** ***No ablation study on #images per person.***\n\nAs detailed in **GQ3**, premised on the fact our method achieves one-shot identity personalization, increasing the number of images per person $N$ from $1$ to $20$ can slightly improve the visual quality of synthesized images. Additionally, the quantitative metrics improve marginally when increasing $N$. The more comprehensive ablation study about using different $N$ will be added to the final version of our paper.\n\n**R4Q4.** ***How to generalize to other non-face samples?***\n\nThe above global response **GQ4** has answered the question. With a simple replacement, we succeed in making our method extended to the \"car\" and \"cat\" personalizing tasks. Inspired by this, we'll continue studying on a more general \"basis\"-based method in future work."}, {"review_id": "cAYTDuLwb1", "reviewer": "Reviewer_RQjb", "summary": "This paper describes a new method for personalizing pre-trained text-to-image models to generate images of unique individuals using just one facial photograph. The method analyzes and builds a celeb basis from the embedding space of the pre-trained text encoder, generating the new identity's embedding by optimizing the weight of this basis and locking all other parameters. The proposed method shows better concept combination ability than previous methods and can learn several new identities at once. The authors plan to release their code.", "questions": "see weakness", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to read.\n2. It is an interesting approach to constructing a celeb basis from the text embedding space and optimzizing its weight to generate unique embeddings. \n3. The results demonstrate the effectiveness of this approach and highlight the potential for generating highly customized images.", "weaknesses": "1. Using a Celeb name basis may limit the expressiveness of the personalized model, as the generated images may be constrained by a predefined set of celebrity names. However, it is unclear what distribution of face, age, and gender was used. This information is crucial to assessing the robustness and generalizability of the proposed method,\n\n2. The paper mentions the importance of using PCA to find the Celeb basis. However, the paper seems to lack an ablation study without using PCA to evaluate the impact of this step on the performance of the personalized model. Without such a comparison, it is difficult to assess the role of PCA in the proposed method and whether it is necessary for achieving the desired results. \n\n3. The generated faces in the proposed method, as shown in Figure 5 for the \"Anne Hathaway\" example, have low quality and distortion. It could be better to provide more analysis.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 5. Responses to R5 (RQjb)\n\nWe sincerely appreciate for your careful reviews. Our responses to your questions are listed as follows:\n\n**R5Q1.** ***Using celeb name basis may constrain the representative ability? What is the distribution of used celeb names?***\n\nOur method is not constrained by the used celerity basis as shown in global response **GQ1**. The personalized results where the faces have large age, pose, expression, gender variants also demonstrate the strong representative ability of celeb basis, as stated in **GQ2**.\n\nThe `Rebuttal PDF Figure 2` shows the distribution of used celeb names. Specifically, we count the frequency of each celeb name (total $691$) in the LAION-2B dataset text samples. LAION-2B is a subset of LAION-5B where Stable Diffusion [1] is pretrained. After sorting the celeb names according their frequency, we obtain `Rebuttal PDF Figure 2`, showing that the most of celeb names appear for $10^3\\sim10^4$ times. Since collecting and validating the detailed gender, age, etc. attributes of our used celeb names is hard to be accomplished in short time, we leave more detailed distribution anlysis as the future work. \n\n**R5Q2.** ***No ablation study of PCA [13]?***\n\nSorry for the unclear details. Actually, we have already ablated the PCA [13] in both `main paper Table 2` and `Supplement Figure 9`, where \"$p=768$\" indicates that there is NO PCA [13] for dimension reducation. Because the dimension of the extracted features from the CLIP text encoder equals to $768$. The existing annotation of $p=768$ in `main paper Table 2` may mislead the reviewer considering it still using a PCA [13] operation. We'll modify the related formulations and declare this more clearly in the paper.\n\n**R5Q3.** ***The synthesized celeb faces in `main paper Figure 5` seem distorted?***\n\nAs clarified in `main paper Section 4.4 Limitation`, we have used Stable Diffusion [1] v1.4 and v1.5 to perform face synthesizing experiments, both of which are hard to generate prefect faces without extra large-scale finetuning. Furthermore, as seen in `Supplement Figure 7 and 8`, the Stable Diffusion [1] synthesized celeb faces may slightly distort from the realistic ones (collected from Google Images). These limitations of Stable Diffusion [1] may lead to the facial distortion of **celebrities** in the output results. That is, given a text prompt containing celeb name, the Stable Diffusion synthesized celebrity may have slight differences with the realistic one or the one in our \"brains\". This appearance does NOT originate from our method. Replacing Stable Diffusion [1] with a stronger diffusion model like Imagen [5], IF [51], SDXL can potentially alleviate this issue, which will be verifed as the future work.\n"}]

Author rebuttal:
# 0. Global Response

We thank all reviewers for their valuable feedback and recognition of the proposed Celeb Basis in terms of the idea novelty, experiment results, and paper written. Below, we address their common concerns.

**GQ1.** ***Can limited celeb names represent any person? (asked by R2,R3,R5)***

Yes, our method uses PCA to find a compact space and fits any given person onto this space through a denoising reconstruction loss [23]. Note that Stable Diffusion [1] pretrained on a very large image-text paired dataset (LAION-5B) including thousands of samples of our used celeb names (see `Figure 2 in our Rebuttal PDF`). Once converged, the learned representation as a linear combination of celeb basis merits good text-editability. In our paper, the results on StyleGAN generated **fake** faces input demonstrate the strong representing ability. Additionally, `Figure 1 in our Rebuttal PDF` shows the robustness of our method against age, pose, gender, light, etc.

**GQ2.** ***The ability of controlling age, pose, expression, etc.? (asked by R2, R4, R5)***

The results in `Supplement Figure 6` along with `Rebuttal PDF Figure 1` show the editability of expression, age, pose, and gender, all driven by text prompts. Thanks to the identity disentangling ability of the face recognition (FR) [14] network, the learned representations are highly relevant to the input identities while irrelevant and insensitive to other attributes like age, pose, expression, etc. Furthermore, the learned representation as an interpolation of celeb basis inherits the good characteristic of those "recognized" celeb names, which corresponds to large amount of image-text paired samples used for pretraining Stable Diffusion [1]. 

**GQ3.** ***What if training with more images per identity? (asked by R2,R4)***

Our method focuses on one-shot identity personalization problem (i.e. training from a single photo to represent a new concept). To verify the influence of training images count $N$ per identity, we provide the results in `Figure 4 of Rebuttal PDF`. It shows that when we use more images, the face quality can also be slightly improved, and the synthesized results are already pretty good when $N=1$. We'll add these results and analyses to the final version.

**GQ4.** ***How to extend to more general classes, like car or cat? (asked by R2,R4)***

Thanks for your suggestion! We also have considered it as a limitation and future work in the `main paper Section 4.4`. Here, we provide some rough experiments. As shown in `Rebuttal PDF Figure 5`, for non-face samples, we trivially replace the FR encoder with a pre-trained ResNet50 (trained on ImageNet-1K) and construct the "car" (or "cat") basis by collecting $\sim100$ typical types of cars (or breeds of cats). Without other bells and whistles, our method generates text-aligned images with good quality of these classes. In the future, we'll try a better general encoder (to replace ResNet50) and crawl a larger basis for non-face classes, extending our method for more general subjects by different well-collected basis names.

**We also provide a `rebuttal PDF` as attached below in this global response part to show more results.**

Meta-review: The paper received good feedback from the reviewers. Overall, the proposed method is smart, simple, and effective. 

The ACs agree to accept the paper to NeurIPS. The authors should consider the discussions during the rebuttal phase in the camera-ready version.
Decision: Accept (poster)