SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6: Text-driven 3D Generation. These images are generated and conditioned solely on text prompts. We utilize the trained EG3D generators [8] from different data domains.
Appendix Sec. C for more qualitative and quantitative comparisons, and insights into why SDS guidance outperforms CLIP guidance.
Illumination Comparison There have been attempts in 2D to control illumination, either implicitly or explicitly. To produce results compatible in 3D, a straightforward approach is rendering multi-view images from a given NeRF, followed by using the projected lighting direction for each view to render the edited NeRF. However, as shown in Fig. 5, without the constraint of 3D rendering, inconsistent illumination is easily observed, indicating imperfect alignment using 2D methods. With explicit 3D control, FaceDNeRF directly manipulates latent code in the disentangled space, capable of rendering consistent and realistic lighting results.
Migration to Other Data Domain FaceDNeRF is not limited to human faces: the semantically diverse and rich text-guidance diffusion model can be used to directly edit other data domains. As shown in Fig. 1, we perform editing on GANs trained on Cats and Cars, noting the reconstruction loss is also universal to all data domains.
4.3 Text-conditioned 3D Generation
In addition to editing, FaceDNeRF can generate high-quality 3D models given a text prompt. Similar to Dreamfusion [44], we guide the generation process under the iterative supervision of a trained diffusion model. However, their unconstrained optimization with random NeRF initialization lacks the ability to generate realistic results. Fig. 6 shows some generated high-quality examples by our methods, including examples from other data domains.
Latent Regularization Although the latent space is smooth, generation from rare-sampled latent codes may produce unrealistic results. Latent samples around mean latent code tends to give better outputs (truncation trick in StyelGAN2 [27]). In text-conditioned 3D generation, the optimized latent code is more likely to deviate from the latent mean since there is no reconstruction or identity restriction. Thus we add a latent regularization to encourage results around the mean, formulated as:
$$
\mathcal {L} _ {\text {r e g u}} (w, \bar {w}) = \lambda_ {\text {r e g u}} \| w - \bar {w} \| _ {2} ^ {2} \tag {9}
$$
where  $\bar{w}$  is the sample mean of latent space. This regularization is equivalent to constraining the feasible space around the mean latent code (Lagrange Multiplier).
Input image
"A lady with golden hair"
Input image
"An old man"
Input image
"Bob-cut hairstyle"
Figure 7: Results on replacing the EG3D backbone with PanoHead backbone. Top-left sub-image in the respective three sub-figures is the input image, with the input text prompt below each subfigure.
Wearing black pants
Wearing a black dress
Figure 8: Semantics-driven editing on human body by replacing EG3D with EVA3D, a state-of-the-art human body NeRF generator. Each row shows 2 examples with the middle image as input.
Wearing jeans
Wearing a glamorous evening gown
4.4 Generalizability Across Different Backbones & Data Domain
In addition to utilizing EG3D [8] as our backbone, the optimization process of FaceDNeRF can be effectively applied to alternative backbones.
Face Domain Generalization In this instance, we exhibit the implementation of PanoHead [3] as the chosen backbone. Given that PanoHead operates as a generative model for human facial representations, akin to the EG3D, it allows for the smooth integration of FaceDNeRF. The outcomes are illustrated in Fig. 7.
Cross-Domain Generalization Here we demonstrate the extension of our method to the human body domain, which is challenging due to the articulation of human bodies. We employ EVA3D [21], a cutting-edge human body NeRF generator for optimization. Notable artifacts are still present in state-of-the-art methods like EVA3D. As the latent variable deviates from the mean value, the generated result becomes blurry, and the face becomes unrecognizable. Fig. 14 in the appendix
(c)
Editing Text Prompt:
Figure 9: Ablation studies on  $\lambda_{ID}$ ,  $\lambda_{R}$ , and  $\lambda_{D}$ . Input text prompt is "A bald man with a thick mustache". The central images in each row are identical with the setting  $(\lambda_{ID}, \lambda_{R}, \lambda_{D}) = (0.5, 0.4, 3 \times 10^{-5})$ . Each row shows the impact of changing one of the weights associated with a particular loss term.
"A bald man with a thick mustache"
displays generation outcomes by random sampling in the latent space near its mean value, revealing a noticeable lack of 3D consistency and erroneous geometric estimation even near the mean value. Fig. 8 illustrates the editing results. Due to EVA3D's limited generation capacity, we conduct editing on images it generated from randomly sampled latent variables close to the mean value of the latent space. Specifically, we randomly sample two values in the latent space, averaging them to obtain a latent variable value. During each iteration, we calculate the feature loss using the input image and a rendering of the current human body NeRF from the same viewpoint. The diffusion loss and optimization pipeline remain identical to our EG3D version. As depicted in Fig. 8, despite EVA3D's suboptimal generation quality, our editing remains flexible and reasonable. High-quality NeRF generators, inversion methods, and advanced feature or identity preservation techniques are potential future works that could enhance the generalization of our pipeline to other domains.
To assess the impact of the weights of various losses on the generated NeRF, we perform ablation studies on the weights of Identity Loss  $\lambda_{ID}$ , Reconstruction Loss  $\lambda_{R}$ , and Diffusion Loss  $\lambda_{D}$ . The frontal-view renderings of the NeRF results from these studies can be seen in Fig. 9. For a comprehensive analysis, please refer to Sec. D.1 in the appendix. Moreover, we investigated the impact of the editing prompt on editing outcomes. For results and a detailed discussion, please see Sec. D.2 in the appendix.
We propose FaceDNeRF, a new generative method to reconstruct high-quality Face NeRFs from a single image, with semantic prompt editing and relighting capitalizing on recent stable diffusion contributions. We believe our optimization pipeline starts a new approach in semantics-driven editing and relighting given any NeRF GAN, and thus has a good impact and can spawn worthwhile future work in the years to come. Extensive experiments validate our significant improvement over state-of-the-art 3D face reconstruction and editing methods. The proposed FaceDNeRF is readily applicable to many real-world applications such as 3D face manipulation, which, however, might be used unethically. Also, the upper bound of our performance is limited by the chosen GANs or diffusion models. We leverage EG3D [8], which is trained on real-world datasets, thus our generation results are realistic but confined to real-world faces or objects, FaceDNeRF can go beyond faces and can be extended to a generic NeRF generation and editing template, where different 3D generators can replace EG3D [8] to produce NeRFs in various domains with ease.
[71] Zheng, Z., Huang, H., Yu, T., Zhang, H., Guo, Y., Liu, Y., 2022. Structured local radiance fields for human avatar modeling. arXiv:2203.14478.
[72] Zhou, H., Hadap, S., Sunkavalli, K., Jacobs, D.W., 2019. Deep single portrait image relighting, in: International Conference on Computer Vision (ICCV).
[73] Zhou, Z., Tulsiani, S., 2023. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction, in: CVPR.
Our reconstruction loss and identity loss are applied to the ground truth image camera poses. Due to our limited GPU memory, we only render one side view to calculate the diffusion loss and illumination loss at each iteration. The camera rotation angles  $\theta$  and  $\phi$  are randomly sampled from  $\left[\frac{\pi}{2} -\frac{\pi}{12},\frac{\pi}{2} +\frac{\pi}{12}\right]$  and  $\left[\frac{\pi}{2} -\frac{\pi}{12},\frac{\pi}{2} +\frac{\pi}{12}\right]$ , where  $\theta$  and  $\phi$  are the angles of spherical coordinate. We set the optimization iterations for our editing to 500, which takes approximately 10 minutes on a 3090 GPU. We set the weighting  $\mathcal{L}_D$ ,  $\mathcal{L}_{ID}$ ,  $L_{R}$  to be 0.2, 0.2 and  $2\times 10^{-5}$  for most editing cases, which can be finetuned for each editing.
Dataset and Generative Bias We utilize trained checkpoints of EG3D on FFHQ [25], AFHQv2 [11] and ShapeNet [9] for the data domain of face, cat and car respectively. For some editing and generation, we notice the existence of biases in generated results caused by the bias of the training dataset, especially for race, gender, etc.
As a supplement to Fig. 1 and Fig. 3 in our main paper, Fig. 10 shows more results of our FaceDNeRF. All three figures illustrate that given a single face image, our FaceDNeRF can perform semantics-driven NeRF editing on various features, such as expressions, emotions, glasses, hairstyles, races, genders, ages, makeup, beard, mustache, and goatee. Notably, both individual and joint editing of these features can be achieved in high fidelity.
Figure 10: More FaceDNeRF results. The middle column shows the input images, while the left and right halves of the images in the other two columns show the results of text prompts editing on the left and right, respectively. Additionally, the first and last columns showcase the corresponding geometries.
Text-conditioned Generation Comparison Our attempts to compare with Dreamfusion[44] (Implemented on a StableDiffusion) failed since it cannot generate faithful models for human heads. This problem might be caused by various reasons. First, the ambiguity of text prompts (human identity, rendering directions) might result in an inconsistent denoising behavior at each iteration. Also, the randomized and unconstrained NeRF optimization process might collapse. On the contrary, our utilization of latent code and trained 3D generative models ensures successful and high-quality generation.
C More Comparisons
C.1 InterfaceGAN and StyleCLIP
Here we provide supplementary comparisons (both qualitative and quantitative) with InterfaceGAN and StyleCLIP. In Fig. 11, since StyleClip proposed two methods Latent Optimization and Latent Mapper, we compared them separately. We adopt the same text example as Fig.4 in the StyleClip paper. For clarity, we only present the results of our method and the Latent Optimization method. Please refer to the original paper for the results of the Latent Mapper.
The qualitative results demonstrate that our method outperforms the Latent Optimization method in terms of consistency with the text prompt and identity preservation, which are also corroborated in the quantitative results in Tab. 1. It is important to note that the Clip loss alone cannot fully capture the visual perception of distances between images and the semantic meanings of text prompts. In some cases (e.g., Fig.1, 1st-row 3rd-column, 3rd-row 3rd-column, and 4th-row 5th-column), the results of the Latent Optimization method exhibit degradation while the Latent Optimization method achieves the lowest Clip loss. On the other hand, our method consistently produces reasonable results across all cases. Even when compared to the latent mapper method, which requires 10-12 hours of training for a specific text input, our method achieves a comparable level of visual quality. Our method only needs around 10mins inference time and no training is needed for a specific text prompt. (Times are measured on a single NVIDIA GTX 1080ti GPU.)
As shown in Fig. 12, we compare with InterfaceGAN on more attributes. Note that as the latent direction found by the classifier could be entangled, the editing results of InterfaceGAN often exhibit undesired changes in other attributes which leads to identity changes. Tab. 2 quantitatively verifies that our method preserves Identity better while achieving the target editing.
Figure 11: Qualitative comparison of FaceDNeRF with StyleClip (latent Optimization). To mitigate the influence of various weight choices on the final outcomes, we experimented with 40 different combinations encompassing a wide range of weights. We present the results with  $IDLoss \leq 0.05$  and the highest visual consistency with the given text prompt.
Table 1: (a) The Latent Optimization method in StyleClip paper, (b) The Latent Mapper method in StyleClip paper. Ours, (a), and (b) are evaluated on the examples shown in Fig. 11. The values of ID loss are multiplied by 100. The values of Clip loss are multiplied by 10.
Ours	(a)	(b)
ID Loss	2.26±1.26	2.40±1.52	1.68±0.60
Clip Loss	7.49±0.24	6.53±0.17	7.16±0.19
Table 2: ID loss on InterfaceGAN and Our FaceDNeRF. We note that the latent direction for InterfaceGAN be may entangled, which causes more alternation in Identity.
Ours	InterfaceGAN
Young	0.0102	0.0159
Smile	0.0028	0.0105
Mustache	0.0014	0.0148
C.2 SDS in Latent Space
To show the usefulness of using SDS as guidance in GAN latent space, we provide comparison with directly using the original SDS on unconstrained NeRF space. As shown in Fig. 13, the original SDS can only generate a blurry human head, where the process can collapse with less careful prompts. This is because of the over-saturation and low-diversity problems of SDS. There are methods to directly apply diffusion in 3D human heads (Rodin), yet they rely on 3D ground truth data (synthesized), which limits the quality and diversity of 3D head diffusion models, while GAN requires no specific 3D data and achieves high-quality 3D results. Therefore, with trained GAN as a constraint, our SDS can effectively utilize the latent space.
C.3 Using CLIP as Guidance
In this subsection, we will discuss why SDS guidance outperforms CLIP guidance significantly. During our comparison with StyleCLIP, we empirically found that the CLIP loss tends to cause degraded images (i.e., images with low image quality and obvious artifacts, not similar to the original image, not coherent to the text prompts). We suspect several reasons for this phenomenon.
During the optimization process using CLIP guidance, the optimization is based only on the current local gradient estimated by the cosine similarity between text embedding and image embedding, we empirically observe that it may overshoot to degraded images. In the SDS setting, the objective is to find the most probabilistic mode in the image domain conditioned on the text, which is referred to as mode seeking [73]. Also, the t sampled will determine the weight of diffusion guidance, which affects the magnitude of the gradient. This offers a more robust updating process, reducing the possibility of being trapped in a local minimum.
In addition, diffusion models focus on high-quality generation, while CLIP focuses on learning cross-modal representations for understanding the relationship between images and text. However, as discussed in [6], CLIP and many other multi-modal understanding models suffer from 'bag-of-objects' kind of representations, possibly due to the contrastive pre-training settings. They perform well in understanding nouns, but badly in visual language concepts beyond object nouns, like attributes and relations. In our context, the input text prompt involves many non-object concepts, like emotions, colors, and other attributes of faces and decorations (e.g., "blue" hair, a "bald" man, an "Eastern" face, a "manly" woman, an "angry" lady, a woman "wearing" a pair of glasses, etc.). When using CLIP guidance, the rendered image is encoded to calculate cosine similarity with the text embedding, during which spatial and attribute information in the rendered image are lost, with a focus on nouns. Thus, CLIP guidance easily "losses control" over the rendered images and generates degraded images, due to difficulty in understanding non-object concepts. In contrast, noise is injected during the training of diffusion models, which allows the denoiser to incorporate more spatial information during the iteration process. More concepts beyond object nouns are also considered by diffusion models. Therefore, we believe that SDS guidance provides more strict "control" and thus higher quality guidance.
Smile
Young
Mustache
Figure 12: Comparison with InterfaceGAN. From the top, rows are respectively input, InterfaceGAN results and our results. We can observe in our results better identity preservation and more faithful edited result to the desired attributes.
(a) NeRF SDS
(b) GAN SDS
Figure 13: Comparison of SDS on GAN Latent Space with SDS on unconstrained NeRF space. (a) The original SDS operates on unconstrained NeRF space but can only generate blurry human heads which can even fail in some cases. (b) High-quality results generated from GAN latent space guided by SDS.
D.1 Loss Weights
To investigate the different losses on the generated NeRF, we conduct ablation studies on:
(a) Weight of Identity Loss  $\lambda_{ID}$ ;
(b) Weight of Reconstruction Loss  $\lambda_{R}$
(c) Weight of Diffusion Loss  $\lambda_{D}$ .
Fig. 9 shows the frontal-view rendering of the resulting NeRF in our ablation studies. The input text prompt is "A bald man with a thick mustache". The central images in each row are identical, adopting  $(\lambda_{ID}, \lambda_R, \lambda_D) = (0.5, 0.4, 3 \times 10^{-5})$ . In each row, only one weight is changed to show the effect of the corresponding loss term.
Observing the rendering results in row (a) and row (b) in Fig. 9, the results with either larger  $\lambda_{ID}$  or larger  $\lambda_{R}$  look more similar to the input person, indicating that both Identity and Reconstruction Losses help preserve the person's identity, while the two losses operate in different ways.
Identity Loss is calculated using pre-trained ArcFace [14] network as described in Sec. 3.3 in our main paper. ArcFace [14] trains DCNNs for face recognition which maps the face image into a feature with small intra-class but large inter-class distance, which means a person with different expressions, hairstyles, mustache, or subtle wrinkles, should have a similar identity score, thus low
Figure 14: Generation Results using EVA3D by randomly sampling around the latent space's mean value. Despite the latent variable for generation being close to the mean value of the latent space, there is still a noticeable lack of 3D consistency and visible geometric estimation artifacts.
Identity Loss in our case. Therefore, Identity Loss would not encode details such as hairstyles and subtle wrinkles, but help to preserve large-scale identity-specific information, such as the shape of the head, nose, mouth, eyes, eyebrows, and any apparent wrinkles.
Instead, Reconstruction Loss calculated by Eq. (2) in our main paper takes all pixels into consideration, thus trying to preserve all kinds of features indiscriminately. A large value of  $\lambda_{R}$  results in a face NeRF similar to the input image. However, this dilutes the guidance effect from the text prompt's attempt to edit the NeRF by removing hair and adding a mustache in Fig. 9. From the right two images of row (b) in Fig. 9, we observe that the identity still has apparent hair, since the large weight of Reconstruction Loss forces the face NeRF to be as similar as possible to the input image.
This is a trade-off between fidelity to details in the input image versus fidelity to the text prompt. As a result, our joint utilization of Identity Loss and Reconstruction Loss can significantly alleviate the problem, where Identity Loss can preserve the identity information in a less contradictory way in the presence of text guidance.
Results in row (c) of Fig. 9 shows the effect of  $\lambda_{D}$ . A larger value of  $\lambda_{D}$  enhances guidance from the diffusion model, while a smaller value of  $\lambda_{R}$  produces higher-fidelity results to the input image. Users could choose suitable values of  $\lambda_{ID}$ ,  $\lambda_{R}$ , and  $\lambda_{D}$  based on the specific task.
D.2 Editing Prompts
We investigate the influence of the input text prompt in this section. In Fig. 10, and Figure 1 and 3 in our main paper, we mainly show NeRF editing results when the input text prompt is a short sentence or a full description of the expected NeRF, such as "A woman wearing dark red lipstick". While this helps a valid generation that avoids ambiguity in semantics, our FaceDNeRF can also take in a text prompt that only expresses the difference between the input face and the output face, such as "Dark red lipstick", if the input face has no lipstick at all. Here, Fig. 15 shows the ablation results. The right two columns show NeRF editing results when input text prompts are full descriptions of the expected editing results, while the left two columns are results with text prompts only describing what should be different.
In most cases, our FaceDNeRF can generate similar editing results given either type of text prompt with subtle or even no tuning of weights of losses  $\lambda_{ID}$ ,  $\lambda_{R}$ , and  $\lambda_{D}$ , as shown in Fig. 15. However, sometimes ambiguity in semantics interferes editing when the input text prompt is not a full description of the expected output. As shown in Fig. 16, even though our FaceDNeRF can generate an old Elon Musk given "An old man" as the input text prompt, it fails when the input becomes just "Old". We believe this is because "Old" has various meanings depending on its context. Therefore, a single "Old" without any context results in useless guidance from our diffusion model, thus producing poor editing results. "Elderly" and "senior" are two synonyms of "Old" in this context. As shown in Fig. 16. "senior" also fails due to its semantic ambiguity with no context, while "elderly" succeeds because of its specific meaning.
Thus, users of our FaceDNeRF are advised to give a better and complete text prompt to avoid semantics ambiguity.
Figure 15: Ablation study on the text prompt. On the right are results generated with full text prompts. The left ones are produced using simfied prompts. Given text prompt describing the wanted change, FaceDNeRF generates results with desired editing.
Figure 16: Ablation study on synonyms. Here are results generated with various Synonyms, which could affect the results. The contest is important for synonyms to eliminate ambiguity, as the case of "senior" and "A senior man".