SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Calculation of Mean. Considering  $C$  may have repetitive embeddings (each row corresponds an embedding vector), for each  $C$ , we first remove the duplicate rows that come from the same token to make sure each token only occurs once at most. Then we calculate the mean  $\overline{C} \in \mathbb{R}^d$  across the second dimension of  $C$  as mentioned in the main paper.
PCA. The PCA algorithm has many different coding implementations in practice. In our method, we use Singular Value Decomposition (SVD) to skip the calculation of covariance matrix in PCA. Please refer to [13, 56] for more detailed theoretical demonstration and coding techniques. Due to the built celeb basis is not optimized, the PCA process only needs to be computed once during the training stage.
B.4 Training Recipe
We train the MLP with a learning rate of 0.005 and batch size of 2 on a single NVIDIA A100 GPU. The training augmentation includes horizontal flip, color jitter, and random scaling ranging in  $0.1 \sim 1.0$ . For single identity training, the optimization costs 400 steps, taking  $\sim 3$  minutes. For 10 identities joint training, we found that training 2,500 steps is enough, taking  $\sim 18$  minutes (averaged about 250 steps and 108 seconds for each identity). The text prompts for training include:
- A photo of a face of  $V^{*}$  person
- A rendering of a face of  $V^{*}$  person
- The photo of a face of  $V^{*}$  person
- A rendition of a face of  $V^{*}$  person
- A illustration of a face of  $V^{*}$  person
- A depiction of a face of  $V^{*}$  person
W/o filtering celebrity names. As mentioned in Section B.2, Stable Diffusion fails to generate correct images from some celeb names. In our method, we manually drop these bad samples. If this filtering process canceled, the interaction ability of the learned identities drops a lot, as shown in Fig. 21a.
Variants of reduction dimension  $p$ . In our method, the PCA reduction dimension  $p$  ( $p < k$ ) controls the degrees of freedom for modulating the variance applied to the celeb basis mean. Considering that  $k = 768$  in the CLIP [4] text encoder, we conduct a series of experiments to study the choice of  $p$ . Four values of  $p$  are chosen, including 64, 256, 512, 768. Note that  $p = 768$  means PCA is not used. Larger  $p$  costs more memory storage. As shown in Fig. 21b, choosing  $p = 512$  comes to the best identity similarity, which is consistent with the quantitative results in our main paper.
Variants training images per identity  $N$ . Our method focuses on one-shot identity personalization problem (i.e. training from a single photo to represent a newconcept). To verify the influence of training images count per identity, we provide the results in Fig. 22, where the inputs varies in view and lighting condition. It shows that when we use more images, the face quality can also be slightly improved, and the synthesized results are already pretty good when  $N = 1$ .
D Extending to More General Classes
Our method have potentials to be extended to non-face classes. Here, we provide some rough experimental results in Fig. 10. For non-face samples, we trivially replace the face recognition encoder with a pre-trained ResNet50 (trained on ImageNet-1K) and construct the "car" (or "cat") basis by collecting typical types of cars (or breeds of cats) in phrases, e.g. 'Volkswagen Passat' and 'Mercedes-Benz A-Class' (or 'Scottish Fold' and 'Abyssinian' for cats). Without other bells and whistles, our method generates text-aligned images with good quality of cars and cats. In the future, we would try a better general encoder (to replace ResNet50) and crawl a larger basis for non-face classes, extending our method for more general subjects using our well-collected basis names.
Figure 12: Multiple persons' personalization result for real identities.
$V_{1}^{*}$
A photo of  $V_{1}^{*}$
Playing the guitar
Wearing the sweater outdoors
s
On a beach
Wearing a scifi spacesuit in space
Carrying vegetables in vegetable market
Fassees
A photo of  $V_{1}^{*}$
On a beach
Figure 13: Additional single person's personalization results comparing with FastComposer [51].
Playing the guitar
Wearing a scifi spacesuit in space
Playing the LEGO toys
Ours
Piloting a fighter jet
Wearing the sweater outdoors
Wearing yellow jacket on a motorbike
Fasssss
A photo of  $V_{1}^{*}$
Piloting a fighter jet
Figure 14: Additional single person's personalization results comparing with FastComposer [51].
Wearing yellow jacket on a motorbike
In a kitchen
Shaking hands
Ours
Playing the piano
In an amusement park
Sitting on a sofa
Baking a cake
$V_{2}^{*}$
In a kitchen
Playing the piano
Figure 15: Additional multiple persons' interaction personalization results comparing with FastComposer [51].
Shaking hands
Baking a cake
oos
Happy
Surprised
Sad
Scared
Angry
Worried
FastComposer
Happy
Surprised.
Figure 16: Comparing with FastComposer [51], our method has a better ability of controlling face expression.
in a Chemical Laboratory
wearing the sweater and a backpack, outdoors
wearing a scifi spacesuit
Figure 17: Uncurated results in a testing batch.
Stable Diffusion
Anne
Hathaway
Figure 18: Stable Diffusion has the ability to generate identity-consistent celeb images. But in the aspect of facial details, the celeb images synthesized by Stable Diffusion may exaggerate some facial features compared with the real ones collected from Google Images.
Barack
Obama
Robert
Downey
Rachel
McAdams
Chadwick
Boseman
Figure 19: We filter out the celeb names that can generate prompt- and identity-consistent images through Stable Diffusion [1]. Left: the satisfied names that can generate images share the same identity with Google Images results. Furthermore, these names have the ability to interact with objects or other satisfied celebs (e.g. 'Barack Obama'). Right: Stable Diffusion confuses the face of 'Rowan Atkinson' and 'Barack Obama' in row four. The gender of 'Sheeran' is mistaken as a girl in row four. The identity of 'Stephen Chow' has a large gap with that of Google Images results.
Figure 20: The frequency statistics of our used celeb names in LAION-2B [3] (a subset where Stable Diffusion trained).
$V^{*}$  is playing basketball.
$V^{*}$  (Input Image)
Input Prompt
w/o filter
$V^{*}$  (Input Image)
Figure 21: Additional ablation studies on building celeb basis and the choice of reduction dimension  $p$ . Note that in our method, the text embeddings are 768-dimensions, which means  $p = 768$  is equivalent to the method without PCA.
(a) The influence of filtering the useless celeb names or not.
Input Prompt
$V^{*}$  wearing a scifi spacesuit in space.
$p = 64$
(0.25×)
(b) The ablation on the values of  $p$
$p = 256$
(0.5×)
$p = 512$
(1.0×,Ours)
$p = 768$
(1.5 ×, max)
$N = 1$
Figure 22: Additional ablation studies on the number of training images per identity  $N$ .
$N = 2$
$N = 5$
$N = 10$
$N = 20$