AdaFace: A Versatile Face Encoder for Zero-Shot Diffusion Model Personalization

Authors:   𝑣
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-30 11:49:18

Abstract: Since the advent of diffusion models, personalizing these models -- conditioning them to render novel subjects -- has been widely studied. Recently, several methods propose training a dedicated image encoder on a large variety of subject images. This encoder maps the images to identity embeddings (ID embeddings). During inference, these ID embeddings, combined with conventional prompts, condition a diffusion model to generate new images of the subject. However, such methods often face challenges in achieving a good balance between authenticity and compositionality -- accurately capturing the subject's likeness while effectively integrating them into varied and complex scenes. A primary source for this issue is that the ID embeddings reside in the \emph{image token space} (``image prompts"), which is not fully composable with the text prompt encoded by the CLIP text encoder. In this work, we present AdaFace, an image encoder that maps human faces into the \emph{text prompt space}. After being trained only on 400K face images with 2 GPUs, it achieves high authenticity of the generated subjects and high compositionality with various text prompts. In addition, as the ID embeddings are integrated in a normal text prompt, it is highly compatible with existing pipelines and can be used without modification to generate authentic videos. We showcase the generated images and videos of celebrities under various compositional prompts. The source code is released on an anonymous repository \url{https://github.com/adaface-neurips/adaface}.

Main paper: 
 claims: 
 Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope, as the detailed results, discussions, and conclusions align with and support the initial claims.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
method: 
 Motivated by the advantages of text space face prompts, we propose techniques to distill one or more image-space face encoders into the text space, and further enhance its compositionality. The overall architecture of AdaFace is shown in Figure 3 . The core module of AdaFace is the AdaFace Prompt Inverter, which inverts the image-space ID embeddings to the text space, enabling the integration of the ID embeddings into a standard text prompt. The ID embeddings are then encoded by a CLIP prompt encoder, which coherently composes the semantics of the ID embeddings and the text prompt.The text-level composition also facilitates Composition Distillation (Figure 5 ), which significantly improves the compositionality of the ID embeddings without additional training data. A side-effect of composition distillation is that, when there is spatial misalignment between the subject-single and subject-composition images, the subject features will be gradually contaminated by background features, reducing their authenticity. Accordingly, we propose a Elastic Face Preserving Loss (Figure 6 ), to prevent the subject features from degeneration.The core module of AdaFace is the AdaFace Prompt Inverter, which converts the image-space ID embeddings from a Face2Image model to the text space.The architecture and initialization of the prompt inverter significantly impacts the training efficiency.Compared to other deep learning tasks, the diffusion training is highly stochastic and the gradients have a much lower signal-to-noise ratio. It is highly challenging to train a sizable diffusion component from scratch without high compute budgets and large batch sizes. To achieve efficient learning, we adopt the same architecture as the CLIP text encoder for the AdaFace Prompt Inverter, and initialize it with the pre-trained weights. This ensures that the output embeddings are not very distant from the text space from the beginning of training, and the model learns more signals from the gradients.One may raise the question that since the output of a pre-trained CLIP encoder is in the image space, why it is able to adapt quickly to generate text-space embeddings? We speculate that in CLIP, the semantics of low-level layers and high-level layers are not in totally incompatible spaces, but rather, the high-level semantics enrich the low-level ones. Our hypothesis is corroborated by [Toker et al., 2024] , as well as the community practice of ad-hoc fusing the output embeddings of multiple CLIP text encoder layers 1 . The semantics of layer features gradually transition from the text space to the image space. As a result, during fine-tuning, the skip connections within CLIP will allow the low-level semantics to take shortcut towards the output embeddings, and the high-level layers will gradually learn to enrich the low-level semantics in the text space instead.The training of the prompt inverter is divided into two stages. In the first face distillation stage, aFace2Image model guides the prompt inverter to generate authentic faces in the text prompt space. In the second composition distillation stage, the prompt inverter observes how the original model output responds to compositional prompts, and learns to generate similar responses, so as to allow the text prompts to control the composition of the generated images.The face distillation stage is illustrated in Figure 4 , where the objective is to minimize the difference between the generated images by the original Face2Image model and by the AdaFace Prompt Inverter on the same initial noise. The training objective, namely the face distillation loss, is formulated as a reconstruction loss between the two generated images:EQUATIONwhere G Face2Image and G AdaFace are the Face2Image and the AdaFace Prompt Inverter conditioned U-Nets, respectively, f is a random face drawn from the face space F , z is the initial noise, and θ and θ ′ are the parameters of the AdaFace Prompt Inverter and the Face2Image model, respectively. For some models such as Ada2Face, θ ′ ̸ = θ.In order to sweep the input space {f, z, t} as completely as possible, we adopt a few techniques:Random Gaussian Face Embeddings. Empirically, we observe that almost all random face embeddings result in legitimate face images when processed by the Face2Image model. Therefore, we expand the candidate face space F by including random face embeddings drawn from a Gaussian distribution, alongside the face embeddings extracted from real face images:F = F real ∪ F rand .Multi-Timestep Distillation. We use multiple denoising steps on the same initial noise, and compute the reconstruction loss on all the steps, so that the prompt inverter learns to imitate the Face2Image model's behavior on intermediate noise levels: tokens. This enables the model to better grasp the subtle facial features of the subject, thanks to the increased diversity and richness of the queries and values. Note that the number of keys and output tokens remain unchanged, ensuring that the computational load does not increase drastically.L face = E f ∼F,z1∼N (0,I),t1>•••>t k ∈[1,T ] k i=1 ∥G AdaFace (f, z i , t i |θ) -G Face2Image (f, z i , t i |θ ′ )∥Specifically, when a query projection Q is expanded by N times, we make N identical copies of Q and add Gaussian noises to N -1 of them. The same operation is applied to the value projection V . This is to ensure that the expanded Q ′ and V ′ do not deviate too much from the original Q and V , and the model augments the original features with slightly varied replicas.The attention expansion proves to be particularly beneficial at the lower layers of the prompt inverter.Intuitively, once some information in the features from the upstream Face2Image encoder is lost in the lower layers, it is hard to recover in the higher layers. The mechanism of expanding queries and values creates multiple, slightly varied replicas of the same information, thereby allowing the model to select the most informative copy for preservation and further processing in subsequent layers.This approach is conceptually akin to the role of the excitation operator in a squeeze-and-excitation network [Hu et al., 2018] , which also emphasizes selectively retaining the most significant features.A prevalent issue with existing face encoders is that the subject token tends to dominate the generated images, resulting in degeneration of compositionality. To mitigate this issue, we employ composition distillation (Figure 5 ) to regularize the subject embeddings, ensuring that their semantics are effectively integrated with other tokens, enhancing the overall expression. During this process, the model observes how the original diffusion model adjusts output features to incorporate additional compositional prompts into the output image. The model then imitates these adjustments when encountering similar compositional prompts.For this purpose, four types of prompts are employed to form two contrastive pairs: 1) a "subjectsingle" prompt that only contains the subject, such as "A photo of a [Zendaya]", 2) a "subjectcomposition" prompt such as "A photo of a [Zendaya] in the forest", 3) a "class-single" prompt that only contains a general class, such as "A photo of a woman", and 4) a "class-composition" prompt such as "A photo of a woman in the forest". Ideally, the semantic differences between "A photo of x"and "A photo of x in the forest" should only be relevant to "in the forest", and is independent of x.We represent the semantic differences between two pairs of prompts as their "feature deltas". The training objective is to encourage the feature deltas between the subject-single and subject-composition images to be aligned with the feature deltas between the class-single and class-composition images.In other words, the following equation is expected to hold approximately:EQUATIONwhere subject, class, (subject, compos) and (class, compos) denote the four types of prompts, respectively. (subject, compos) and (class, compos) are randomly drawn from a pool of common compositional prompts consisting of various backgrounds, additional objects, dresses, image styles and lighting conditions. feat(x) refers to relevant features, including 1) the output features from all the cross-attention layers, 2) the attention maps in all the cross-attention layers, and 3) the encoded prompt embeddings by CLIP text encoder. feat(x)feat(y) is the orthogonal subtraction between two feature maps, defined below.We define a compositional delta loss that aligns the feature deltas ∆ i (subject, compos) and ∆ i (class, compos) on the three types of features listed above:EQUATIONin which i indexes the feature type (cross-attention output features, attention maps or CLIP prompt embeddings), and U(C) is a uniform distribution on a set of compositional prompts C.Orthogonal Subtraction. We wish to remove subject-specific features through the feature subtraction "feat(subject, compos)feat(subject)". However, it is commonly observed that the subjectspecific features may have different magnitudes (often smaller under compositional prompts). To mitigate this issue, we propose to use orthogonal subtraction, which is invariant to the scale of the subject-specific features. A relevant idea [Wang et al., 2023] is explored for language model fine-tuning. Specifically, the feature deltas are calculated using the following equation:EQUATIONwhere proj feat(s) (feat(s, c)) is the projection of feat(s, c) onto feat(s), computed as:EQUATIONwith ⟨feat(s, c), feat(s)⟩ being the inner product between the two features. The operation effectively projects feat(s, c) onto the orthogonal complement of feat(s) and then subtracts this projection from feat(s, c). As a result, ∆feat(s, c), the feature delta, is orthogonal to feat(s). This methodology ensures that the deltas remove as much of the subject-specific features as possible, thereby minimizing the influence of the scales of the subject-specific features contained within feat(s, c).Differences with Previous Methods. While previous methods have explored analogous concepts, such as StyleGAN-NADA [Gal et al., 2022b] , which applies similar regularizations in the CLIP prompt embedding space, and PuLID [Guo et al., 2024b] , which introduces similar contrastive regularizations on cross-attention queries, our approach is more comprehensive and effective. Our compositional delta loss encompasses a broader range of relevant features, including the attention maps and output features from cross-attention layers, and the CLIP prompt embeddings. Moreover, we introduce an orthogonal subtraction technique for computing the feature deltas. This technique isolates and extracts composition-specific features, making the distillation more effective.The composition distillation is done on instances with different prompts starting from the same initial noise. This is to encourage the diffusion model to generate images that are compositionally similar [ Zhang et al., 2024] , to achieve more accurate alignment between the image pairs. Despite this effort, spatial misalignment often persists between the images differently prompted. This misalignment can result in delta loss providing erroneous signals from non-facial to facial areas, slowly reducing the authenticity of the generated subjects. For instance, on a noisy input face image, the output image from the subject-single instance is expected to largely retain the same facial contours as the input. However, the output from the subject-composition instance often deviate from the original face contours, due to the introduction of additional compositional elements. An illustrative example provided in the first row of Figure 6 shows how a chef hat in one image spatially aligns with the hair in another, leading to potential contamination in the subject's hair representations.To tackle this challenge, we view the subject-composition image as a "warped" version of the subjectsingle image, and turn to techniques from the Optical Flow literature [Teed and Deng, 2020, Sui et al., 2022] to estimate a matching field. The matching field is used to spatially align the subject features across different images, ensuring them to be consistently maintained after "warping".Specifically, the model takes as input a noisy face image from the training data. The face image is accompanied by a segmentation mask, isolating the face area for matching. We compute the cross attention matrix2 between the queries of a subject-single instance and a subject-composition instance:EQUATIONBy looking up the cross-attention map CA(subj, compos), we can find the pixels best matching a subject-single image pixel in a subject-composition image. The second row in Figure 6 shows the attention maps of four points on the face in the left image. We "soft-warp" the subject-composition features to align with the subject-single features through matrix multiplication, and require the warped features to be close to the facial features in the subject-single image:EQUATION)Here for clarity, feat(subject, compos) is abbreviated as feat(compos). The cosine similarity cos(•, •)is computed on the masked area. The face-preserving loss is computed on each U-Net cross-attention layer. It encourages the subject features in the subject-composition instance to be consistent with those in the subject-single instance, preventing them from being contaminated in the composition distillation process. 3 ExperimentsQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations of the work are discussed in the "Conclusions and Discussion" section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
dataset and training details: 
 We trained AdaFace on a combination of two face datasets: Flickr-Faces-HQ (FFHQ) [Karras et al., 2019] , which comprises 70,000 images, and VGGFace2-HQ [Cao et al., 2018] , which comprises 336,567 images after filtering. Face masks were generated using the BiSeNet face segmentation model [Yu et al., 2018] . The distilled Face2Image model is Ada2Face [Papantoniou et al., 2024] , as it is able to generate authentic and diverse face images. The training employed the Prodigy optimizer [Mishchenko and Defazio, 2024] with d_coef=2 (akin to the learning rate in other optimizers) during face distillation, and d_coef=0.5 during composition distillation. Batch sizes were set to 4 and 3 for the two stages, respectively, with a gradient accumulation of 2. The model was trained with 240,000 iterations in the face distillation stage and 120,000 iterations in the composition distillation stage.During face distillation, the loss reached a plateau twice, resulting in two dynamic expansions of the model capacity. Eventually, the attention layers in the trained prompt inverter were expanded with multipliers of (8x, 8x, 8x, 4x, 4x, ..., 4x) relative to the original CLIP text encoder. This resulted in a total of 2M parameters, in contrast to the 1.2M parameters of the original model.In addition, we collected the images of 23 celebrities, each with 9 10 images, as the evaluated subjects.These celebrities include actors, singers and internet celebrities on Instagram. This dataset will be released along with the code.We compared AdaFace with a few state-of-the-art face encoders, including InstantID [Wang et al., 2024] , ConsistentID [Huang et al., 2024] and PuLID [Guo et al., 2024b] . The input were images from our celebrity-23 dataset.The results presented in Figure 7 demonstrate that AdaFace produces images that not only exhibit high authenticity of the subjects but also show good consistency with the text prompts. In comparison, other models often fall short in generating images that are either less authentic or less compositional.For instance, InstantID tends to produce overly stylized images with significant variability in authenticity across different subjects. PuLID, while generating aesthetically pleasing images, achieves slightly lower authenticity levels compared to AdaFace. Despite also utilizing a text-space approach, Jensen Huang dancing pose among folks in a park, waving hands Yann Lecun in a white apron and chef hat, garnishing a gourmet dish In addition, we plugged AdaFace into AnimateDiff, and generated personalized videos of celebrities under various compositional prompts. The results are shown in Figure 1 . Figure 8 compares with a recent method ID-Animator [He et al., 2024] . AdaFace generated videos with high authenticity and compositionality, while ID-Animator usually produces videos with less authentic subjects.To assess the performance of AdaFace quantitatively, we evaluated a few baseline methods andAdaFace, on the "celebrity-23" images and DreamBench compositional prompts, comparing AdaFace with two baseline methods PuLID and InstantID. First, we measured the face similarity using the cosine similarity between the ArcFace embedding of the generated images and reference images. In addition, the CLIP-Text (CLIP-T) metric determines the consistency of the generated images with the prompts. The DINO and CLIP-I metrics are less indicative and are only for reference. The results, As an ablation study, we list the performance of the AdaFace model without composition distillation.detailedIt can be seen that the face authenticity is slightly reduced after composition distillation, however, the generated images become much more consistent with the prompts.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: For each theoretical result, the paper provides a plenty of experimental support.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
conclusions and discussions: 
 In this work, we present AdaFace, a versatile face encoder that maps human faces into the text prompt space. AdaFace is trained with a low compute budget and achieves high authenticity and compositionality in zero-shot generation of subject images. We demonstrate the effectiveness of AdaFace by showcasing the generated images and videos of celebrities under various compositional prompts. Additionally, our quantitative evaluations further underscore its performance.A notable limitation of AdaFace is that the authenticity of the output face embeddings are constrained by the Face2Image model it distills from. However, this limitation can be addressed by distilling on more powerful Face2Image models and expanding the model capacity. For future work, we would extend the AdaFace method to object images. For instance, applying AdaFace distillation techniques to IP-Adapter [Ye et al., 2023] could enable the generation of both human and object images.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: All the experimental details are clearly stated in the paper and the code will be made publicly available.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: All data and code will be made publicly available.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Please refer to the "Implementation Detail" section in the main paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We evaluated on a diverse set of 30 celebrities, each with around 50 prompts, which is sufficient to reflect the model's performance.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We use 2 A6000 GPUs, each with 48G of memory.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discussed the positive impacts, including its potential use in entertainment and art, video games and virtual reality. Additionally, its potential use for educational purposes in historical recreation, such as recreating faces of historical figures or enhancing documentaries, bringing history to life. We also pointed out potential negative impacts, including privacy violations. There is a risk of creating and using images of individuals without their consent. Moreover, misinformation and deepfakes are among the most concerning impacts, with the creation of deepfake videos that could be used to spread misinformation and manipulate public opinion. We also highlighted security concerns, as the technology could be used to bypass facial recognition systems for fraudulent purposes, posing significant security challenges. The authors will join in the effort for possible mitigation by providing gated release of models.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: The work describes basic safeguards implemented for the responsible release of models, particularly focusing on preventing misuse. We have incorporated filters that specifically exclude NSFW (Not Safe for Work) keywords in the generation prompts, such as 'nude,' 'naked,' 'nsfw,' 'topless,' and 'bare breasts.' This approach helps mitigate the risk of generating inappropriate or sensitive content."Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: , (2) 1 https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/5674The inner product is not scaled to make the matching scores more polarized. 


Decision: This paper studies generating images with personalized human faces using the diffusion model. While the previous method focuses on fine-tuning the diffusion model and leveraging images as conditions, this work attempts to encode the images into text embedding space, allowing the user to combine words like environment description along with the embedded face seamlessly. Overall, reviewers find the paper well-written, and some designs, such as training losses and regulations, in the algorithm are interesting. However, reviewers express major concerns about the experiment parts. Specifically, reviewers find the current experiment results do not reach the performance of existing works (Reviewer 6DX2, Uvbt, jpRS) and lack ablation studies of the proposed components (Reviewer 6DX2, jmG6). During the rebuttal phase, while the authors provide more qualitative results and analysis, the concerns remain without quantitative experiment results on broader test sets, as well as results for ablation studies. Therefore, the reviewers still feel that the responses were insufficient to fully support this paper’s acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes AdaFace, a test-time-tuning-free method for personalized text-to-face-image generation. Previous methods involving face features in the feature space of a face encoder, which is not flexibly composable with natural language for personalized generation. Thus, this paper proposes to map the face features into the features in the text conditioning space. Several techniques are proposed to enhance the performance, like Random Gaussian Face Embeddings, Multi-Timestep Distillation, Dynamic Model Expansion, Composition Distillation, and Elastic Face Preserving Loss. Some experiments demonstrate that the proposed method achieves good visual results.
strength_weakness: strength_weakness: 1. The overall motivation is not novel enough. Finding ways to convert input images into the textual space is a fundamental goal in text-to-image personalization, which has been emphasized in the very first TextualInversion work. Even in the context of tuning-free based methods, the proposed framework is not so novel compared with ELITE [a], which also involves training a mapper from the image space into the textual space. Similar compositional distillation technique has also been explored in SuTI [b].
2. Lack of detailed studies of the proposed components, either qualitatively or quantitively. The authors propose a bag of techniques to improve the performance. Although their motivation is mentioned in the texts, there is no supportive results to illustrate how these techniques work. 
    * There is only one quantitive study in Tab. 1 regarding the compositional distillation. However, there are actually a lot of technical details in the proposed compositional distillation techniques, like the orthogonal subtraction and compositional delta loss, which lack careful experimental analysis against their alternatives. 
    * The proposed face distillation is not well supported. Can we simply train the face encoder with the simple noise prediction loss of diffusion models?
    * The analysis of the proposed Elastic Face Preserving Loss is also missing.
3. ELITE [a] mentions that using multiple token to represent an image may hurt the textual compatibility. It is necessary for the authors to provide a rationale for doing so.
4. How about the method comparing with the popular IP-Adapter (face version) [c]?
5. The overall training pipeline requires multiple stages of training, which is not so elegant.
6. The authors would like to consider merging multiple figures with similar functionalities and structures to one, like Figs. 2, 3, 4 and 5, to leave enough space for necessary experimental results.

[a] ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation, Wei et al., ICCV 2023.

[b] Subject-driven Text-to-Image Generation via Apprenticeship Learning, Chen et al., NeurIPS 2023.

[c] IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models, Ye et al..
questions: questions: Please refer to the weaknesses above.
limitations: limitations: The authors have discussed the limitations of the proposed method.
paper_summary: paper_summary: This paper proposes AdaFace, a method for personalizing text-to-image diffusion models for human faces. At its core, it learns a prompt inverter that maps face embeddings from a pretrained face encoder to the text embedding space of diffusion prompts. It leverages various components including face distillation, composition distillation and elastic face preserving loss to preserve subject identity while attaining good compositionality.
strength_weakness: strength_weakness: *  The quantitative metrics do not show a clear advantage of AdaFace over other existing personalization methods like PuLID. The number of qualitative examples for comparing with those methods is also limited -- just the 5 images per method in Figure 7, and not sufficient to clearly demonstrate that AdaFace outperforms existing methods. It would be helpful to show a larger number of uncurated examples comparing AdaFace and baselines to get a better comparison of their performance.
* The training of the prompt inverter involves a number of components -- such as model expansion, the inclusion of different feature types in composition distillation, orthogonal subtraction and elastic face preserving loss -- but there are no ablation studies on most of them to demonstrate their effects on the performance.
questions: questions: See Weaknesses - 

It would be helpful to see more uncurated qualitative examples for better comparison with other existing approaches. Additionally, are there particular reasons why ConsistentID is not included in the quantitative comparison although it is included in the qualitative comparison?

And although the methods section discussed some conceptual intuitions for the design of the target losses, it would be helpful to use ablation studies to verify their contribution to the performance, and for the compositional delta loss part, show their advantage over previously explored designs.
limitations: limitations: The authors have discussed limitations and societal impact.
paper_summary: paper_summary: This paper proposes AdaFace, a face encoder that maps facial features from the image space to the text space through the AdaFace Prompt Inverter, utilizing the structure and pre-trained weights of the CLIP text encoder for initialization. During the face distillation phase, AdaFace employs random Gaussian face embeddings and multi-timestep distillation, enhancing the model's ability to capture subtle facial details through dynamic model expansion. In the composition distillation phase, AdaFace uses a comparative learning loss, aligning feature increments with orthogonal subtraction, while introducing an elastic face preservation loss to address the misalignment of facial features caused by different prompts.
strength_weakness: strength_weakness: - I tried the demo provided by the authors, and the ID similarity on a few test images was relatively low; it should be far from the state-of-the-art (SOTA) level of ID similarity claimed in the paper.
- The test dataset consists of celebrities, which does not guarantee whether these IDs have appeared in the training set. Furthermore, the number of test samples is too small to be convincing.
- The upper bound of ID fidelity is constrained by the frozen Face2Image model, in this paper, Arc2Face.
- The proposed improvements like Random Gaussian Face Embeddings, Orthogonal Subtraction, etc. are not effectively validated through ablation study.
questions: questions: na
limitations: limitations: yes
paper_summary: paper_summary: This work presents a zero-shot face-generation method based on diffusion models. The proposed method first extracts face features using Face2Vec and trains a network to map these features into the textual space (i.e., the prompt embedding space for diffusion’s text condition). The main difference from existing zero-shot face generation approaches is that, instead of adding conditions in the denoising UNet, the presented method incorporates the condition into the textual embedding. To prevent the subject information from overwhelming the generation (e.g., preserving only the subject ID while ignoring other descriptions), the authors propose a Composition Distillation Loss. This contrastive loss encourages the model to generate descriptions other than the subject information.

The main shortcoming of the paper lies in the experimental validation. There are multiple components proposed, but their effectiveness is not adequately demonstrated. Additionally, both qualitative and quantitative evidence fail to show that the proposed method outperforms SoTA methods.
strength_weakness: strength_weakness: * There is a lack of intuition behind using multi-timestep distillation. This approach can cause accumulated errors, and there is no evidence provided to support the benefits of adopting such a strategy.
* The section on dynamic model expansion is very unclear. It is not specified which part of the model is expanded or if the tokens are simply replicated. While adding Gaussian noise to replicated tokens is mentioned, there is no empirical validation of its effectiveness.
* The qualitative comparison, especially in Figure 7, does not support the claim that the proposed method has advantages over baselines like PuLID. Additionally, the benchmarking results in Table 1 show limited improvement in facial identity preservation, and the text alignment can be inferior to PuLID. Thus, the experiments are not comprehensive enough to convincingly demonstrate that the proposed approach is superior to existing methods like PuLID.
questions: questions: My suggestions to the author are follows:
* Conduct a comprehensive ablation study of the components proposed in the paper, which could include but is not limited to: 1) Justification for placing the learnable part (the conditioning module) in the text encoder rather than the diffusion denoising UNet. 2) Effectiveness of using facial preservation loss, multi-timestep distillation, and dynamic model expansion.

* Design experiments that are more relevant to the proposed methods. The current paper contains many irrelevant experiments/visualizations, such as: Figure 1/8: Applying AnimateDiff and generating video. Figure 2: Illustrating other work's pipeline.
Figure 6: Feature alignment lines and heatmap that do not provide useful information. The compatibility of the proposed method with different backbones/plugins can be mentioned in the paper but should not be the main focus and can be presented in the supplementary materials.

* Polish the language and improve readability (e.g., using tools like ChatGPT). Some content is not presented in a formal paper format, such as Line 222, "each with 9 10 images."
limitations: limitations: The authors have discussed the limitations.

Comments: title: Thanks for your interesting question
Thanks for your response
Thanks for your response
dataset copyright, id theft and carbon footprint
comment: Thank you for taking the time to carefully examine our claim of "seamless integration with video generation pipelines." We appreciate the opportunity to further clarify and differentiate our approach from similar techniques.

1. While methods like DreamBooth, LoRA, and more recently, MagicMe [a], can achieve subject consistency in video generation, these approaches require cumbersome subject-specific fine-tuning, which limits their practicality for long videos featuring multiple subjects. These limitations have partly driven the development of recent zero-shot subject-driven generation methods.

2. More importantly, subject-driven video generation, often referred to as **visual story generation**, aims to create long sequences of videos where specific subjects transition through different scenarios. For example, in StoryDiffusion [b], a video story is described where a man reads a newspaper, drives to a forest, and encounters a tiger. Achieving such complex narratives requires more than just frame-to-frame consistency; it necessitates a mechanism for maintaining subject identity and appearance across diverse and lengthy sequences.

3. Traditional video generation methods can maintain subject and background consistency across **adjacent frames** due to large-scale pre-training on videos, where adjacent frames are typically consistent. This training allows the model to implicitly learn and preserve such consistency. However, when extended to **long sequences**, these models often suffer from subject distortion or semantic drift due to the absence of dedicated subject representations. Our method, while not yet perfect, represents a step towards zero-shot dedicated subject representation learning for visual story generation, addressing these challenges more effectively. In this sense, our method is also significantly different from the layout-based method you mentioned, which primarily focuses on maintaining spatial consistency over a short window of videos, rather than ensuring subject identity over longer sequences.

For a more detailed discussion on the nuances of these techniques, we invite you to refer to the ID-Animator paper [c] (one of our baseline methods), particularly Section 2.3.

We hope this clarification helps in understanding the unique contributions of our work in the context of video generation. We respect your perspective and appreciate your feedback, which has been instrumental in refining our presentation.

    [a] Magic-Me: Identity-Specific Video Customized Diffusion. arXiv:2402.09368.
    [b] StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation. arXiv:2405.01434.
    [c] ID-Animator: Zero-Shot Identity-Preserving Human Video Generation. arXiv:2404.15275.
The reviewer thanks the authors for their response but would like to point out that the claim of "seamless integration with AnimateDiff without fine-tuning" as a contribution of this work is difficult to acknowledge based on the current version of the paper.

If one examines the demos provided by AnimateDiff in their official GitHub repository or GitHub page, it becomes apparent that many works can achieve "seamless integration with AnimateDiff without fine-tuning," such as LoRA or Dreambooth. Even less well-defined conditions, such as layout-based methods (e.g., https://github.com/frank-xwang/InstanceDiffusion/tree/main?tab=readme-ov-file#third-party-implementations), can achieve similar integration. The common factor among these approaches is that they all share the same backbone model as AnimateDiff.

Therefore, the ability to integrate seamlessly with AnimateDiff without fine-tuning seems to be a characteristic of diffusion models that share the same backbone as AnimateDiff, rather than a unique feature of the proposed method. As a result, the reviewer does not agree with the authors' assertion that AnimateDiff integration should be considered a significant contribution of this work.
Thank you for your well-reasoned reply. I'm confident you will respond to the mentioned concerns in a suitable manner. I look forward to reading the new version of your paper.
Thank you for taking the time to revisit our demo and for providing additional feedback.

1. We acknowledge that small-scale subjective evaluations can exhibit high variance, particularly as our method performs better on certain ethnic groups compared to PuLID and InstantID, while underperforming on others. To address this, we plan to scale up the training dataset in the next stage to include a more diverse population, ensuring improved performance across a wider range of subjects.

2. We understand your concerns regarding the absence of ablation studies in our rebuttal. The delay in conducting these studies was due to the first author, who is primarily responsible for the technical implementations and experiments, working on multiple projects in parallel. As a result, the focus of the first author was diverted to other tasks in the period leading up to the rebuttal. However, given the importance of these ablation studies, which has been emphasized by all reviewers, we are committed to prioritizing them moving forward. We will ensure that these studies are completed to make the experimental evaluation more comprehensive.

We respect your decision to maintain your rating and appreciate the valuable feedback you have provided. Your insights have been instrumental, and we are dedicated to addressing the areas you highlighted in future revisions.
Thank you for your response, but my concerns are still not completely addressed.

I revisited the demo provided by the authors and found it to be slightly improved compared to the previous version. However, my conclusion remains the same that AdaFace does not reach the SOTA level in terms of ID similarity measures, such as those achieved by PuLID and InstantID. This led to a question as to why the quantitative comparisons provided by the authors depict AdaFace being on par with PuLID and InstantID. In my first round of reviews, I mentioned that this might be due to limited testing IDs or potential bias, but the authors failed to adequately address this in the rebuttal, i.e., verifying the quantitative metrics on a broader test set. The authors only provided qualitative comparisons of two athletes, one of whom (LeBron James) is a famous basketball player, who could have possibly appeared in the training set.

Additionally, the authors have not supplemented any ablation study in the rebuttal. I disagree with the excuse of insufficient resources or time, given that several months have passed since the submission, which would have been ample time for the authors to prepare the apparently missing ablation studies. The authors promise that these ablations will be incorporated in the next version, but we cannot predict whether these experiments will validate the efficacy of the modules proposed by the authors.

In conclusion, I believe the current paper has issues and is incomplete in terms of experimental evaluation, therefore, I have decided to maintain my score.
Thanks for the response and for including additional qualitative examples. I understand that conducting ablation studies during the rebuttal period might not be feasible. However, ablation studies are a crucial part of a paper, for understanding and validating the necessity of each component of the method. After considering all aspects, I decide to keep my original score.
Thank you for your feedback and for taking the time to review our work.

We appreciate your recognition of the motivations behind "Multi-timestep Distillation" and "Dynamic Model Expansion." However, we respectfully disagree with your assessment that our work "has not shown progress in personalized facial generation."

In our submission, we have demonstrated advancements in personalized facial generation, particularly through seamless integration with the existing video generation pipeline, AnimateDiff without fine-tuning. This plug-and-play ability is a unique advantage of our method. Our empirical results, as presented in the paper, show that our approach can generate facial images comparable to several state-of-the-art methods. Moreover, when integrated with AnimateDiff, it demonstrates significant advantages over the existing subject-driven video generation method, ID-Animator.

While we understand that our contributions may not have been fully aligned with the specific criteria you were looking for, we believe that our work offers meaningful advancements that contribute to the broader field of personalized image and video generation.

We respect your decision to maintain your rating and will take your feedback into account as we continue to refine and improve our work. Your insights have been valuable and we are committed to addressing the areas you highlighted in future revisions.

Thank you once again for your careful consideration.
The reviewer thanks author's feedback. 

The reviewer finds the motivation of "Multi-tiemstep distillation" and "Dynamic model expansion" in author's feedback . However, for a top tier conference like NeurIPS, the reviewer believes the work needs to be 1) technically sound by having justifications for proposed components/ideas and 2) experimentally show superior performance than existing works or comparable performance under a harder scenario. 

Specifically, the reviewer thinks the author has not well justified their proposed methods (see Questions 1) and has not shown progess in the personalized facial generation. As such, the reviewer decides to remain my rating.
Thank you for highlighting these ethical concerns. Your feedback will help us strengthen our paper and contribute to more responsible AI research.

1. **Dataset copyright**. The primary copyright concerns related to generative AI models typically center around the training data. In our experiments, we utilized two well-known face datasets: Flickr-Faces-HQ (released in 2019) and VGGFace2-HQ (released in 2018). These datasets have been publicly available for over five years, and to date, there have been no significant copyright challenges associated with their use. Based on this, we believe that the risk of copyright issues arising from the use of these datasets is minimal. Regarding the evaluation phase, we collected approximately 230 images of 23 celebrities from the internet. It's important to note that our model transforms these input images, generating new images that are semantically distinct (e.g., a "portrait of Taylor Swift" versus a "portrait of Taylor Swift in a superhero costume"). Given the significant transformation and the creation of new, original content, we believe this use case qualifies as "transformative use," which is generally considered "fair use" under copyright law.
2. **Identity theft**. Regarding concerns of identity theft, it's important to note that our face encoder cannot generate subject videos on its own. It must be integrated into a diffusion-based video generation pipeline to create output. To address potential misuse, the generated videos can be easily watermarked by adjusting the settings of the video generation model, ensuring that users are clearly aware these videos are not real. If the watermark is vulnerable to removal through editing, we can implement forensic watermarking techniques to prevent its removal. We will explore this topic in greater detail in the updated version of our paper.
3. **Carbon footprint**. The power of a single A6000 is $0.3 \text{kw/h}$, and the total training time is $2 \text{GPUs} \times 200\text{h}$. Therefore the total carbon emission is

    $0.3 \text{kw/h} \times 2 \times 200\text{h} \times 0.432 \text{kg/kwh} = 51.84 \text{kg}.$
For comparison, the training of Stable Diffusion v1.0 had a total carbon emission of approximately 11,250 kg CO2, about 217 times of ours.

[a]  "Rate-Distortion-Preserving Forensic Watermarking Using Quantization Parameter Variation," in IEEE Access, 2020.
Thanks the authors for the response. I still believe the mentioned ablation studies are necessary. I expect the authors could finish them and add them to the revision. Conditioned on this, I will increase my score to 5.