FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models
Authors: Hao ZHANG
Tianyuan DAI
Yanbo Xu
Yu-Wing Tai
Chi-Keung Tang
Venue: NeurIPS 2023 poster (2023)
Keywords: NeRF Editing
NeRF Relighting
Face
Diffusion model
3d synthesis
GAN inversion

Abstract: The ability to create high-quality 3D faces from a single image has become increasingly important with wide applications in video conferencing, AR/VR, and advanced video editing in movie industries. In this paper, we propose Face Diffusion NeRF (FaceDNeRF), a new generative method to reconstruct high-quality Face NeRFs from single images, complete with semantic editing and relighting capabilities. FaceDNeRF utilizes high-resolution 3D GAN inversion and expertly trained 2D latent-diffusion model, allowing users to manipulate and construct Face NeRFs in zero-shot learning without the need for explicit 3D data. 
With carefully designed illumination and identity preserving loss, as well as multi-modal pre-training, FaceDNeRF offers users unparalleled control over the editing process enabling them to create and edit face NeRFs using just single-view images, text prompts, and explicit target lighting. The advanced features of FaceDNeRF have been designed to produce more impressive results than existing 2D editing approaches that rely on 2D segmentation maps for editable attributes. Experiments show that our FaceDNeRF achieves exceptionally realistic results and unprecedented flexibility in editing compared with state-of-the-art 3D face reconstruction and editing methods. Our code will be available at https://github.com/BillyXYB/FaceDNeRF.

FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models
Hao Zhang" HKUST hzhangoc connect.ust.hk
Yanbo Xu* CMU, HKUST co.buecomnet..ss..k
Tianyuan Dai Stanford University, HKUST tdaiaa @ connect.ust.hk
Yu-Wing Tai Dartmouth College yuwing @ gmail.com
Chi-Keung Tang HKUST
cktang @cs cs.uutt.kk
Abstract
The ability to create high-quality 3D faces from single image has become in- creasingly important with wide applications in video conferencing, AR/VR, and advanced video editing in movie industries. this paper, we propose Face Diffu- sion NeRF (FaceDNeRF), new generative method to reconstruct high-quality Face NeRFs from single images, complete with semantic editing and relighting capabili- ties. FaceDNeRF utilizes high-resolution 3D GAN inversion and expertly trained 2D -ifent-diffusio model, allowing users to manipulate and construct Face NeRFs in zero-shot learning without the need for explicit 3D data. With carefully designed illumination and identity preserving loss, as well as multi mudti- pre-training, FaceINNRRF offers users unparalleled control over the editing process enabling them to create and edit face NeRFs using just single-view images, text prompts, and explicit target lighting. The advanced features of FaceDNekK have been de- signed to produce more impressive results than existing 2D editing approaches that rely 2D segmentation maps for editable attributes. Experiments show that our FaceDNeRF achieves exceptionally realistic results and unprecedented flexibility in editing compared with state-of-the-at 3D face reconstruction and editing methods. Our code will be available at
com www.ccomm//ii.. BB1llyXBBFFaceDDD.
Introduction
Rich and versatile 3D contents are in high demand entertainment industries such as movie making, computer gaming and emerging applications such Metaverse, which also have high potential in robotics learning as well, where high-quality synthetic data in large amounts are required to enhance generalizability. 3D generative methods such EG3D [ generate high-fidelity NeRF from a single image, but the reconstructed NeRF cannot be easily controlled or edited. Some methods [33 57 use pixel-wise segmentation maps or user-supplied sketches to guide 3D editing. But these methods are hard to scale due to the demanding editing requirement and limited editable attributes. Language regarded as one of the most suitable candidates to provide control signals for 3D editing, especially given the current success of 2D semantics-drig image and language domains are bridged by CLIP, i.e., Contrastive Language- Image Pre-training [45). However, there is still substantial room for improvement in 3D generative and editable models, in terms of usability, edit-ability, and results quality.
editing (28 42), where
'These authors contributed equally to this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Face domain
Other domains
Input
Frontal
Clean geometry
lateral
A man with closed
man
image
man with pair glasses
cut with blue eyes
blue
Figure I: ReFF results. Given a face image, FaceDNeRR can reconstruct, edit, and relit photo-realistic 3D face NeRF using a text prompt and a target light. Our method not limited to human faces, and can be applied to other domains as well. Further results and videos are available in the supplemental materials.
Neural Radiance Field (NeRF), a new paradigm in 3D scene representation, embeds 3D scene in compact fully-connectee neural network and achieves realistic rendering of novel views. NeRF optimized to approximate a continuous and thus differentiable scene representation function, which has quickly become a dominant approach for many relevant downstream tasks including
novel scene composition [29
38
41
66
70), surface reconstruction [31 61 (68) and articulated
3D shape reconstruction [TO
71 due to its differentiability, multi-view consistency, compactness, and fast inference. Since NeRF relies on training from multi-view images, its relighting contingent upon the relighting of these 2D images. Nevertheless, current 2D relighting techniques [5] are unable to ensure view consistency among the relit multi-view images, resulting in decline in the quality of the relit NeRF.
23
43
67
[ 12
Significant attempts have been made recent years to develop semantic-drive NeRF editing models. Pre-trained 2D sanguage-immge models usually leveraged to enable multi-modality in 3D. CLIP- NeRF (60) introduces a disentangled conditional NeRF architecture, where the disentangled latent representations be bridged by mappers trained with CLiP-based matching loss the CLIP embedding, based on which the latent codes are updated for editing based on the input text prompt. Outperforming CLiP-based methods, DreamFusion 44 later adopts Imagen 49 a pre-trained text- to-image diffusion model, as prior to guide the optimization of parameters of randomly initialized NeRF through a novel SDS loss. Magic3D (32) further improves DreamFusion by introducing coarse-to-fine fine optimization scheme with diffusion priors at different resolutions. However, these methods have two main limitations: lack of photorealism and long inference time (reportedly taking 1.5 hours for DreamFusion, and 40 minutes for Magic3D).
We believe the main reason underlying the less realistic results and slow inference of DreamFu- sion 44 and its variants lies in the random and thus unrestricted initialization of NeRF. Leveraging 3D generative models such EG3D training for constrained NeRF generation, may offer a feasible solution. Although combining 21) generative models with CLIP 45 for realistic image generation and editing has been widely adopted approach [55). no significant attempts have been made in 3D realistic context to our knowl- edge. Thus, in this paper, we present FaceDNeRF, which is our knowledge the first work to enable semantic-driven NeRF editing in 3D with high photorealism and versatile relighting from single image, given a text prompt and target light. FaceDNeRF freezes the weights of the EG3D network which can reconstruct photorealistic NeRF from a single image. To enable semantic-driven editing, we adopt stable diffusion 488 guide the optimization through SDS loss introduced in DreamFusion 44. Based on SDS loss, identity loss, and feature loss, the latent vector in EG3D's latent space can be updated. Moreover, the proposed illumination loss allows explicit control over the lighting in view-consistent manner.
H whose generation is restricted by the discriminator during

---

2
Related Work
2D Generation and Editing Research on unconditional or text-driven image generation has made fruitful progress. Generative Adversarial Networks (GANs) [77 contribute to the first revolutionary 2D image generative methods, among which StyleGAN (26] and its variants [24] 27 stand out due to the expressive and well-disentangled latent spaces. StyleGAN-basee -based image editing requires either an
encoder trained to map a given image to the latent space 2
39 or specifying latent update direction
which requires explicit ground-truth annotations another class of generative models that enables text-driven, photorealistic and highly diverse image generation. State-of-the-art text-to-image synthesis methods contribute effective mechanisms to guide samples toward semantics: classifier-free guidance [20] 37 generates images with or without class information during model training; CLIP guidance [3
19
59 65 Diffusson models (54 (56) represent
69 where CLIP 4 trained on 400 million image- maage-textppe pairs spearheaded cross-modal representation learning in modern vision-language tasks. Leveraging the rich joint embedding spaces of CLIP and expressiveness of diffusion models, stable diffusion [48] is arguably the best text-to-image imae model to date, synthesizing images of vast domains and styles based on text prompt. Thus we adopt stable diffusion as the guidance model in our 3D approach.
implicit neural representation, has become the dominating modern approach for 3D generation due to its continuity, differentiability. compactness, and quality of novel-view synthesis over mesh and point cloud. GRAF (50) combines implicit neural rendering with GAN for generalizable NeRF. PiGAN [Z] utilizes SiREN [53] to condition the implicit neural radiance field on the latent space. Although guaranteed with 3D consistency, volumetric rendering requires heavy computation. With limited computation, the image quality of these methods still not comparable those produced by current state of-the----h--rrrr 2D GANs. Thus, many recent approaches adopt hybrid structures. StyleNeRF [88 applies volume rendering in the early feature maps in low resolution, followed by upsampling blocks to generate high-resolution images. However, regularizer based on NeRF is required to ensure 3D consistency during upsampling. Instead of using volume rendering in early layers, EG3D B performs the operation on a relatively high-resolution feature map using hybrid representation for 3D features generated by StyleGAN2 (27) backbone, named tri-plane, which is capable of incorporating more information than an explicit structure such as voxel. StyleSDF 44) shares a similar spirit but uses SiREN (53) for its mapping network, with the mapped result used as the input feature map followed by a style-based generator for upsampling.
3D Generation and Editing NeRF 36
Attempts have been made to generate 3D objects using diffusion models. Rodin 622. Realfusion [35], Set-the- [1] and other diffusion- based 3D reconstruction methods have demonstrated the feasibility of constructing face or other object from single view image. However, the results are not adequately realistic with limited generalizability due to the scarcity of 3D data. is also possible to approximate 3D assets using trained 2D diffusion models, as shown in 44 32 However, the
quality of generated assets is visually worse than the outputs from the utilized 2D models, especially in areas such as the face, where high fidelity is essential criterion.
Illumination Control 3D editable lighting on NeRF is highly desirable feature. Image relighting methods can be roughly categorized into two groups: one involves estimating 3D face information such as 3DMM coefficients (5), albedo, and surface normals, and combining this information with a target lighting condition represented by spherical harmonics (SH) coefficients to generate relit images, such as (22 34 [5). Although guaranteed with 3D) consistency, these methods have limited editability as they cannot easily accommodate changes in the original model, such as wearing glasses or different hairstyles. The other approach involves leveraging 2D/3D generative models to control illumination in the latent space, such as 44] [30]. While this approach is conducive to some editability, illumination can only be implicitly controlled by latent codes, that is, environmental lighting cannot be directly controlled by e.g., SH coefficients or cube mapping. Our method is amenable to both light editing means, bypassing any intrinsic separation which is not necessary in FaceDNeRF.
3 Method
3.1 EG3D and W Space
Our method utilizes trained EG3D generator [8] with its respective latent space. From initial latent code R512 mapping network M maps to w in the space named W, where e R
3

---

Input
optional
Image encoder:
Side
ID encoder:
lady with pair of glasses
Illumination Loss
E W.
Reconstruction and ID Loss
EG3D
W.
Diffusion Loss
Side view
Input Iaagecamees pose
Backpropagate onto Latent Space
Figure 2: FaceDNeRF structure. The Latent 512 scalars are initialized as the mean of sampled W codes. The model computes the Reconstruction Loss and Identity Loss from the input image's 's camera view. From other side views, given a text prompt, the model computes the Diffusion Loss by assessing the discrepancy between the predicted noise and random noise. The Illumination Loss then computed by comparing the estimated SH coefficients of the rendered side-view image to the target SH coefficients. These Losses then utilized update the Latent $12 scalars iteratively through a carefully designed differentiable model via pack-propagation.
During training and generation, the w code will be utilized to modulate all convolution layers as in StyleGAN2 (27]. The generated features will be reshaped into three orthogonal feature plans (Fay, Fyer Fzz), where each of them has the resolution of N x x C. Given camera pose, augmented feature can be rendered using volume rendering in [36]. which will then be up- sampled by the super-resolution blocks. The rendering process acts as an inductive bias that enforces view-consistent results.
The inversion process inverses an input image to the latent space, such that the inverted code w' can faithfully reconstruct the given input. As shown in 2D GAN inversion [46], the quality of reconstruction is better when the inversion is conducted on W space, where its latent codes w E R4x512 are used to separately modulate all L convolution blocks. In addition, the w space preserves the editability [64] (88) Thus, the reconstruction and editing process of our method operates on the W space.
3.2 Formulation
Fig. summarizes our method, which takes single image text prompt and target lighting represented by SH coefficients I as inputs, and generates a reconstructed 3D face NeRF as output. Inspired by high-fidelity face NeRF reconstruction techniques, such EG3D inversion (8). The Reconstruction and Identity Preservation (Sec. 3.3) is employed to guide the facial NeRF reconstruc- tion process. To enable language-guided editing, incorporate Diffusion component (See. 3.4). which is based on the trained text-conditioned latent diffusion model. Notably, our method allows for explicit illumination control, thanks the inclusion of the Illumination component (Sec. 3.5). To generate the target latent code, we randomly sample a value near the mean value of the latent space starting point. Then solve an optimization problem obtain the desired output.
Ln. LID. En and LT.. denote the Reconstruction loss, Identity Loss, Diffusion Loss and Illumination Loss, which will be elaborated in Sec. 3.3 Sec. 3.4 and Sec. 3.5 We solve the following optimization:
arg min AnnCno () wEW+
An An cx), lo,)
(I)

---

where the G(-.-) is the EG3D generator, is the camera pose of the input image (which can be estimated by [16]. C22 is the random side camera pose. L. the target illumination which varies with Cx. Ärp, An. An and Åιz. are weights of the corresponding loss functions. We optimize the w iteratively through gradient descent by haopppropgg the gradient of the objective Eq. (T through these four weighted differentiable loss functions.
3.3 Reconstruction and Identity Preservation
The desired editing should preserve the background and identity of the input image, and hence we design the Reconstruction Loss and Identity Loss. Given input image I and rendered image G((w c). we utilize VGGI6 16 image encoder V(-) [2 extract the image features and construct the
Reconstruction Loss as following:
[n(G(w, c), = [V c)) - (22)1
(2)
For human faces, we adopt the same Identity Loss as 46]
LID (G(,,c), 1 (R(z), R(G(w,))) c)))
where R is the pretrained ArcFace [14] network. VGG16 image encoder and ArcFace LR and LiD only used measure the difference between the input image and the rendered image from input camera's view to avoid the misalignment due to mismatched viewing directions. See our ablation study for insightful analysis of the interaction between the Reconstruction and Identity losses.
3.4 Prompt Editing with Diffusion Model
Although directly optimizing a NeRF by distilling a 2D diffusion model is possible quality of the resulting model is often worse than the used 2D model, especially in the domain of human subject (refer to Appendix C.2). In addition, compared with 3D diffusion models, 3D GANs currently capable of producing high-fidelity 3D NeRFs with smooth latent space. Therefore, perform Score Distillation in the latent space of trained 3D GAN model.
44 32 the
We modify the Score Distillation Sampling (SDS) from DreamFusion 44 our diffusion loss. The loss connects the rendered from sampled camera pose C with the denoising prediction conditioned on the editing prompt y. The denoising process be written as:
Ln(a' = G(w, c.),y) Be(r") [ll e (Ze y, () - [12]
where e(-) is the image encoder that encodes our rendered image x' = G(w, c.) to the latent space of the diffusion model, denoted z. Here 2 is the noisy version of time-step é is the frozen denoising network of the trained diffusion model, where we sample ~ (0.002 0.98) to avoid very high and low noise levels; e (O, I). and its effect on input latent varies with time-step t, which is same as done in Dreamfusion 44
Notably, the camera pose Cs will be resampled randomly in each optimization iteration ensure 3D) view consistency. Unlike DreanFusion, our text prompt is view independent, since we have the identity and reconstruction losses to constrain the optimization process, and that the EG3D generator has underlying 3D information in contrast to randomly initialized NeRF. Therefore, our diffusion loss back propaatoss the latent scalars, where the gradient of LD is given by
Os' Dw
V Lo G(w, c.), & E.,t.
((t)((((Z(; 1) c)
As in Dreamfusion 44] w(t absorbs the coefficient of the forward process, and we ignore the term de, (mesp.d) as the diffusion model frozen.
o2.
3.5 Explicit Illumination Control
As the trained 3D GAN mapped a continuous latent code to 3D-consistent NeRF, direct manipulation of the latent ensures view-consistent illumination editing. We utilize (72) to construct our Illumination Loss, where the hourglass network can be denoted as:
L I Hn(L,
(6)
5

---

Geometry
Edited views
Input
Edited views
Geometry
little child
man with
beard
A lady with pair of glasses
long hair
boy years old
A man
with short blue hair
A hold man with a thick mustache
old
Figure 3: More FaceDNeRF results. The middle column shows input images, while the left and right halves of the images in the other two columns show the results of text prompts editing the left and right, respectively. The first and last columns show the corresponding geometries.
where L, and L; respectively the target SH lighting and the estimated SH lighting of the input image I. and I is the relit image under the target SH lighting. Here, we only use L to compute the La loss with L,, and I will be ignored, i.e., L Hn'(I,) Thus, our Illumination Loss is:
Crr Cx), Le,) =I1 Hn' (Gw, (G(,w..)) cx)) - le,
During the optimization process of w", differentiability is crucial. Given G(w, c.) is differentiable with respect w, the differentiability of Hn' (Gw, cs)) with respect G(w, c.) guarantees the back- pock-propagation onto the w. Therefore, we replace the PyTorch in-place operations and other numpy operations the Hn' model with equivalent differentiable PyTorch tensor operations. As verified by experiments, our modified Hn' achieves the same performance as the original model while being multi-view consistent as we render the output.
3.6 Pivotal Tuning Inversion
The Pivotal Tuning Inversion (PTI) technique [4 is widely employed many GAN inversion processes to overcome the trade-off between distortion and editability (58). After solving the Eq. ( the obtained latent code wi is utilized as pivot to fine-tune the generator using the similar loss function:
arg min
Experiments
This section presents our editing results and compare them with representative methods, emphasizing FacsDNeRF'ss disentanglement capability which is conducive editing control, its flexibility in utilizing a text-guided diffusion model, as well as the multi-view consistency illumination control. Furthermore, our method can migrated other data domains with trained generative models.
4.1 3D Editing from Single Image
Although diffusion models can generate diverse and realistic images in 2D, the lack of large and high-quality 3D datasets limits the performance of current diffusion models. On the other hand, the adversarial learning mechanism together with inductive bias of 3D rendering enables GAN to produce high-quality 3D results. From GAN's smooth latent space, our method finds the suitable latent code whose semantic information matches the editing demand. Fig. [ and Fig. demonstrate the detailed editing results of FaceDNeRF. Please refer to Sec. B of the appendix for more results.

---

smiling man
Young girl, 10
Lady, 30
Input
Old layy 50
Old lody, 70
Qura
Our
Input
InterFaceGAN
Othera
Age-
Age+
StyisCLIP
Figure 4: Editing Comparison with representative methods. We compare with classifier- based InterfaceGAN [51], semantic edited FENeRF [57] and language guided StyleClip [42] with EG3D. Images on the left side are editing results of gender, smile, and "An Eastern face" respectively. The right images age editing comparison with (5T) and our input text prompt "A XX is about YY years old", where XX and YY are depicted above. The comparison illustrates the flexibility and high-fidelity editing capability of FaceDNeRF.
DPR.
FDNeRF (ours):
Figure 5: Illumination Comparison. The images in the first row are relit images generated by DPR [72.. The images in the second row are relit images generated by our method. Upon comparing the four red boxes, our method is observed produce fewer artifacts and more realistic results. Furthermore, the images the third row demonstrate that our method can not only edit face NeRF using text prompts but also allows explicit control of lighting.
4.2 Comparison
Comparison with representative editing methods For effective and easy editing, the underlying latent space should be smooth and semantically meaningful. Compared with the latent space of 2D GANs, disentangling in 3D much harder. InterfaceGAN [5T] seeks a hyper-plane in the latent space with pre-trained classifiers, which makes editing feasible by interpolating latent codes the direction orthogonal to the hyper-plane. However, the assumption that good hyper-plane exists which is well- behaved for linear interpolation only valid when the latent space is highly disentangled. As shown in Fig. 4 the underlying latent space in 3D is not well disentangled. Thus the interpolation (induced by editing) can interfere and affect other irrelevant attributes not to be edited. We also compare with the semantic-bars based method FENeRF [57]. whose editing is performed by manually editing the semantic map. However, this representation limits editable attributes, especially on semantically complex attributes such as age and gender. In addition, we compare with language-guicee StyleClip [22 using the same latent space. Our method achieves better editing quality conditioned on the same text prompt, indicating the superiority of utilizing the diffusion model as guidance. Please refer to

---

man with mustache wearing pair of black glasses
woman with blond hair
A white cat with blue eyes
bald man
A sad kid
A black car
Figure 6: Text-driven 3D Generation. These images are generated and conditioned solely on text prompts. We utilize the trained EG3D generators g from different data domains.
Appendix Sec. for more qualitative and quantitative comparisons, and insights into why SDS guidance outperforms CLIP guidance.
Illumination Comparison There have been attempts in 2D to control illumination, either implicitly or explicitly. To produce results compatible in 3D, a straightforward approach is rendering multi-view images from given NeRF, followed by using the projected lighting direction for each view render the edited NeRF. However, as shown in Fig. 5 without the constraint of 3D rendering, inconsistent illumination is easily observed, indicating imperfect alignment using 2D methods. With explicit 3D control, FaceDNeRF directly manipulates latent code in the disentangled space, capable of rendering consistent and realistic lighting results.
Migration to Other Data Domain FaceDNeRF is not limited to human faces: the semantically diverse and rich eextgguid diffusion model can be used directly edit other data domains. As shown in Fig. D we perform editing on GANs trained on Cats and Cars, noting the reconstruction loss is also universal to all data domains.
4.3 Text-conditioned 3D Generation
In addition to editing, FaceDNeRF can generate high-quality 3D models given a text prompt. Similar Dreamfusion [4]], we guide the generation process under the iterative supervision of trained diffusion model. However, their unconstrained optimization with random NeRF initialization lacks the ability generate realistic results. Fig. 6 shows some generated high-quality examples by our methods, including examples from other data domains.
Latent Regularization Although the latent space is smooth, generation from rare-sampled latent codes may produce unrealistic results. Latent samples around mean latent code tends to give better outputs (truncation trick in StyelGAN2 (77). In text-conditioned 3D generation, the optimized latent code is more likely to deviate from the latent mean since there is no reconstruction or identity restriction. Thus add latent regularization encourage results around the mean, formulated as:
Eregu (. w) Aregul w|
where the sample mean of latent space. This regularization equivalent to constraining the feasible space around the mean latent code (Lagrange Multiplier).

---

Input image
Input image
Input image
"A lady with golden hair"
"An old man"
"Bob-cut hairstyle"
Figure 7: Results on replacing the EG3D backbone with PanoHead backbone. Top-left sub-image in the respective three sub-figures is the input image, with the input text prompt below each subfigure.
Wearing black pants
Input Image
Wearing jeans
Wearing a black dress
wearing glamorous evening gown
Figure 8: Semantics-driven editing human body by replacing EG3D with EVA3D, state-of-the-art
human body NeRF generator. Each row shows 2 examples with the middle image as input.
4.4 Generalizability Across Different Backbones & Data Domain
In addition utilizing EG3D g as backbone, the optimization process of FaceDNeRF can be effectively applied to alternative backbones.
Face Domain Generalization In this instance, exhibit the implementation of PanoHead B as the chosen backbone. Given that PanoHead operates as generative model for human facial representations, akin to the EG3D, it allows for the smooth integration of FaceDNeRF. The outcomes are illustrated in Fig. 7
Cross-Domaiee Generalization Here we demonstrate the extension of our method to the human body domain, which is challenging due to the articulation of human bodies. We employ EVA3D [2]], cutting-edge human body NeRF generator for optimization. Notable artifacts are still present in state-of-the-an methods like EVA3D. As the latent variable deviates from the mean value, the generated result becomes blurry, and the face becomes unrecognizable. Fig. 14 in the appendix

---

(a)
JD 0.25
TD
À 1 0.75
À 10
(b)
R
À = 0.2
R 0.4
A 0.6
0.8
Input Image
(c)
Editing Text Prompt:
* bald man with a thick mustache"
À. =
A = 1.5e-5 2 3e
A D 4,5e 5
60
Figure 9: Ablation studies on ÅID, AR, and Åp. Input text prompt is "A bald man with a thick mustache". The central images each row are identical with the setting (AID. XR. Ap) (0.5, 0.4, "). Each row shows the impact of changing one of the weights associated with particular loss term.
displays generation outcomes by random sampling in the latent space near its mean value, revealing noticeable lack of 3D consistency and erroneous geometric estimation Fig. [ illlusttrssss the editing results. Due to EVA3D's limited generation capacity, we conduct editing on images it generated from randomly sampled latent variables close to the mean value of the latent space. Specifically, we randomly sample two values in the latent space, averaging them to obtain latent variable value. During each iteration, and a rendering of the current human body NeRF from the same viewpoint. The diffusion loss and optimization pipeline remain identical to our EG3D version. As depicted in Fig. 8 despite EVA3D's suboptimal generation quality, our editing remains flexible and reasonable. High-quality NeRF generators, inversion methods, and advanced feature or identity preservation techniques are potential future works that could enhance the generalization of our pipeline to other domains.
the mean value.
calculate the feature loss using the input image
4.5 Ablation Study
To assess the impact the weights of various losses on the generated NeRF, we perform ablation studies the weights of Identity Loss ÅıD, Reconstruction Loss ÅR. and Diffusion Loss Äp- The frontal-view renderings of the NeRF results from these studies can be seen Fig. 9 For comprehensive analysis, please refer to Sec. D.1 in the appendix. Moreover, investigated the impact the editing prompt on editing outcomes. For results and detailed discussion, please see Sec. D.2 in the appendix.
5 Conclusion and Discussion
We propose FaceDNeRF, new generative method reconstruct high-quality Face NeRFs from single image, with semantic prompt editing and relighting capitalizing on recent stable diffusion contributions. We believe our optimization pipeline starts new approach semantics-driven editing and relighting given any NeRF GAN, and thus has a good impact and can spawn worthwhile future work in the years come. Extensive experiments validate our significant improvement over state-of-te thrt 3D face reconstruction and editing methods. The proposed FaceDNeRF is readily applicable to many real-world applications such as 3D face manipulation, which, however, might be used unethically. Also, the upper bound of our performance is limited by the chosen GANs or diffusion models. We leverage EG3D R which trained on real-world datasets, thus our generation results are realistic but confined to real-world faces or objects, FaceDNeRF can go beyond faces and can be extended generic NeRF generation and editing template, where different 3D generators can replace EG3D [8] to produce NeRFs in various domains with ease.
10

---

References
[1] Abdal, R., Zhu, P., Mitra, N..., Wonka, P., 2020. Styleflow: Attrbuute-onddtionee explo- ration of stylegan-generane images using conditional continuous normalizing flows. CoRR abs/2008. 02401.. URL: htwp:htttp://aaatee www.arwww.....aaaa..aooaooooeree wwt.://..rgr//////////ao hwwpg///a////...../// 2008 02401 :
[2] Alaluf, Y., Patashmik, O., Cohen-Or, D., 2021. Only matter of style: Age transformation using
style-based regression model. CoRR abs/2102.02754. URL: https://arxiv. org/arggga...
2102.02754. arxivv2102..00
[3] An, S., Xu, H., Shi, Y., Song. G., Ogras, U.Y., Luo, L.,, 2023. Panohead: Geometry-awan
3d full-head synthesis in 360deg, in: Proceedings of the IEER/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20950-20959.
[4] Bhattad, A., Forsyth, D., 2023. Stylitgan: Prompting stylegan to generate new illumination conditions, in: arXiv.
[5] Blanz, V., Vetter, T., 1999. A morphable model for the synthesis of 3d faces, in: Proceedings of the 26th annual conference on Computer graphics and interactive techniques. pp. 187-194.
[6] Bescante-Bonilley P., Shehada, K., Smith, J.S., Doveh, S., Kim, D., Panda, R., Varol, G., Oliva, A., Ordonez, V., Feris, R., Karlinsky, L., 2023. Going beyond nouns with vision & language models using synthetic data. arxiv: 2303. 17590.
[7] Chan, E., Monteiro, M., Kellnhofer, P., Wu, J., Wetzstein, G., 2020. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis, in: arXiv.
[8] Chan, E.R., Lin, C.Z., Chan, M.A., Nagano, K.. Pan, B., Mcllo, S.D., Gallo, O., Guibas, L.J., Tremblay, J., Khamis, S., Karras, T., Wetzstein, G., 2021. Efficient geometry-aware 3d generative adversarial networks. CoRR abs/21 12.07945. URL: https: //tp///axvvvv/////////eee/eaaavv org abs/
2112.0796
arXiv 2112. 07945
[9] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song. S., Su, H., Xiao, J. Yi, L., Yu, F,, 2015. Shapenet: An information-rich 3d model repository. arxiv: 15122.1111..00 03012
[10] Chen, H., Tretschk, E., Stuyck, T., Kadlecek, P., Kavan, L.,, Vouga, E., Lassner, C., 2022. Virtual elastic objects. CoRR abs/2201 044233. URL. URI: htthhhttt////aiiii org/ wwrggabs//2055 04623
arXiv:2201 04623
[11] Choi, Y., Uh, Y., Yoo, J., Ha, J.W. 2020. Stargan v2: Diverse image synthesis for multiple domains, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[12] Cohen-Bat D., Richardson, E., Metzer, G., Giryes, R., Cohen- Or, D., 2023. Set-the-scene: Global-local training for generating controllable nerf scenes. arXiv:2303: 13450
[13] Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, L., Raff, E., 2022. Vggan-clip:: Open domain image generation and editing with natural language guidance. arXiv: 2204.22204.00 08583
[14] Deng, J., Guo, J., Xue, N., Zafeiriou, S., 2019a. Areface: Additive angular margin loss for deep face recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[15] Deng, Y., Yang, J., Chen, D., Wen, F., Tong, X., 2020. Disentangled and controllable face image generation via 3d Recognition.
learning, in: IEEE Computer Vision and Pattern
[16] Deng. Y.. Yang. J., Xu. S., Chen, D., Jia, Y.. Tong, X., 2019b. Accurate 3d face reconstruction with weakly-superviie lcarning: From single image to image set, in: IEEE Computer Vision and Pattern Recognition Workshops.
[17] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 2014. Generative adversarial networks. arxii: 1406 2661

---

[18] Gu, J., Liu, L., Wang, P., Theobalt, C., 2021. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. CoRR 10.0891, URL: httpss///a.ii....rrrggggo//////
2110.08985) arXiv 2190.
[19]
Härkönen, E., Hertzmann, A., Lehtinen, J., Paris, S., 2020. Ganspace: Discovering interpretable
GAN controls. CoRR abs/2004.02546. URL:
02546
arXiv:2004 02546
[20] Ho, J., Salimans, T., 2022. Classifier-free diffusion guidance, arXiv:2207.1
[21] Hong, F., Chen, Z., LAN, Y., Pan, L., Liu, Z., 2023. EVA3d: Compositional 3d human generation from 2d image collections, in: International Conference on Learning Representations. URL: https://opperevies ntt/ferumt Formmiid gtppmmmiddgg7g900. gU0jjD__2UU
[22] Hou, A., Sarkis, M., Bi, N., Tong, Y., Liu, X., 2022. Face relighting with geometrically consistent shadows, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[23] Jiang, Y., Jiang, S., Sun, G., Su, Z., Guo, Wu, M., Yu, J., Xu, L.. 2022. Neuralhofusion: Neural volumetric rendering under human-object interactions. arXiv: 222iv: 12825
[24] Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, Lehtinen, J., Aila. T., 2021. Alias- free generative adversarial networks. CoRR 12222000.1223,, URL: https://arxxi: org/
abs/2106. 12423 arxiv: 2106. 12423
[25] Karras, T. Laine, S., Aila, T.. 2018a. Flickr faces hg (ffhq)
70k from
stylegan.
CoRR URL: https://ggt /tte:://gitulee
md.
[26] Karras, T. Laine, s., Aila, T., 2018b. A style-based generator architecture for generative adversarial networks. CoRR abs/1812.0492 URL:
arX1v:1812.1 1812. 04948
[27] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T., 2019. Analyzing and improving the image quality of stylegan. CoRR abs/1912.04958. URL: http:://axiiii http://aaxvvv..o org/ abs/1912. 11111.19111...... arxiv: 1912. 04958
[28] Kim, G., Ye, J.C., 2021. Diffusionclip: Text-guided image manipulation using dif- fusion models. CoRR abs/2110.02711.
URL:
02711
arXiv:2110.022 02711
[29] Kundu, A., Genova, K., Yin, X., Fathi, A., Pantofaru, C., Guibas, L.. Tagliasacchi, A., Dellaert, F., Funkhouser, T., 2022. Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation.
[30] Kwak, J.g., Li, Y., Yoon, D., Kim, D., Han, D., Ko, H., 2022. Injecting 3d perception of controllable nerf-gan into stylegan for editable portrait image synthesis, in: European Conference on Computer Vision, Springer, pp. 236-253.
[31] Li, H., Yang. X., Zhai, H., Liu, Y., Bao, H., Zhang, G., 2023. Vox-surf: Voxel-based implicit surface representation. 2208. 10925
[32] Lin, C.H., Gao, J.. Tang, L.. Takikawa, T., Zeng. X., Huang, Kreis, K., Fidler, S., Liu, M. Y.. Lin, T.Y., 2023. Magic3d: High-resolution text-to-3d content creation, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[33] Liu. S., Zhang, X., Zhang, Z., Zhang, R., Zhu, J., Russell, B., 2021. Editing conditional radiance fields. CoRR abs/2105.066 0666666
URL: httpp////aaaiiv.oor////////55 06466
arXiv:2105. 06466
[34] Liu, Y., Shu, Z., Li, Y., Lin, 7., Zhang, R., Kung, S.Y., 2022. 3d-fm gan: Towards 3d-controllable face manipulation, in: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (Eds.), Computer Vision ECCV 2022, Springer Nature Switzerland, Cham. pp. 107-125.
12

---

[35] Melas- Deela-Kyriazzi, L., Rupprecht, C., Laina, I,, Vedaldi, A.. 2023. Realfusion: 360 reconstruction of any object from single image, in: CVPR. URL: httph:ttttt//ttti http:///waxxii...aaaa/////////// wwrg/abs/230// 10663
[36] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng. R., 2020. Nerf: Representing scenes as neural radiance fields for view synthesis, in: ECCV.
[37] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B.. Sutskever, I., Chen, M., 2021. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. CoRR 0tt/2112.107444 arXiv:2112.1074
URL: http:///aaaaa...or///////2..////
[38] Niemeyer, M., Geiger, A., 2020. Giraffe: Representing scenes as compositional generative neural feature fields.
[39] Nitzan, Y., Bermano, A., Li, Y., Cohen-Or, D., 2020. Disentangling in latent space by harness- ing a pretrained generator. CoRR abs/2005.07728. URL: httts::/paaii...rrrrttr httggg/bb/2005555 07728 arXiv: 07728
[40] Or-EI, R., Luo, X., Shan, M., Shechtman, E., Park, JJ., Kennlmachhr-SShizemma. I,, 2021. Stylesdf: High-resolution 3d-consistent image and geometry generation. CoRR abs/2112.11427.
/tt.://ttpr///aoaw..o/////////22 1121122.1442 11427 arXiv:21122 11427
[41] Ost, J., Mannan, F., Thuerey, N., Knodt, J., Heide, F., 2020. Neural scene graphs for dynamic scenes.
[42] Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., Lischinski, D., 2021. Styleclip: Text-driven manipulation of stylegan imagery. 2eXXiv:2003.1220
[43] Peng, S., Dong, J., Wang. Q., Zhang, S., Shuai, Q., Zhou, X., Bao, H., 2021. Animatable neural radiance fields for modeling dynamic human bodies. arXiv:2105, 02872
[44] Poole, B., Jain, A., Barron, J.T., Mildenhall, B., 2022. diffusion. arXiv.
Dreamfusion: Text-to-3d using 2d
[45] Radford, A., Kim, J.W Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, L, 2021. Learning transferable visual models from natural language supervision. CoRR abs/2103.00020. URL: htLL:ttts:///rrrr orgg
2003.00020000
[46] Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., Cohen-Or, D., 2021. Encoding in style: stylegan encoder for image-tu woogee-toiing to-ieage translation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[47] Roich, D., Mokady, R., Bermano, A.H., Cohen- Or, D., 2021. Pivotal tuning for ittent-based editing of real images. ACM Trans. Graph.
[48] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B., 2021. High-resolution image synthesis with latent diffusion models. arXiv: 2112. 10752.
[49] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.,, Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J., Fleet, D.J., Norouzi, M., 2022. Photore- alistic text-to-image diffusion models with deep language understanding. arxiv: 2205 11487
[50] Schwarz, K., Liao, Y., Niemeyer, M.. Geiger, A., 2020. GRAF: generative radiance fields for 3d-aware image synthesis. CoRR abs/2007.00242 URL: htapps://aaxxvv..oo org sitiivv.rrg/bb//20000 2007 02442 arXiv: 02442
[51] Shen, Y., Yang. C., Tang, X., Zhou, B., 2020. Interfacegan: Interpreting the disentangled face representation learned by gans. arXiv: 200.5.2005...... 2005. 09635
[52] Simonyan, K., Zisserman, A., 2015. Very deep convolutional networks for large-scale image recognition. arXiv: 1409.:140...0 1556
13

---

[53] Sitzmann, V., Martel, J.N.P., Bergman, A. W., Lindell, D.B., Wetzstein, G., 2020. Implicit neural representations with periodic activation functions. CoRR 0016/2006.09611. URL: https
warxiv.org/absi. 2006. 09661
arXiv: 2006 09661
[54] Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., Ganguli, S., 2015. Deep unsupervised learning using nonequilibrium thermodynamicc. CoRR abs/1 URL: http://ttp://tttt
org/abs/ 1503. 03585 arXiv: 1503, 03585
[55] Song. K., Han, L., Liu, B., Metaxas, D., Elgammal, A., 2022. Diffusion guided domain adaptation of image generators, arXiv: 2212. 04473
[56] Song, Y., Ermon, S., 2019. Generative modeling by estimating gradients of the data distribution. CoRR 090119997.6000. URL: http:://axiii wrg.aas///////ao/oo//...o.aaeeae 0abb//900 1907 05600
arXiv: 1907 05600
[57] Sun, J., Wang, X., Zhang, Y., Li, X., Zhang, Q.. Liu, Y. Wang J., 2021. Fenerf: Face editing in neural radiance fields. CoRR abs/2111. URL: htt.h:ttts:s/aiiii wwtppaiwww..oa./a.....a.a// attp//bbb////.a////aaa/aa/a..aat 15490
arXiv:2111 15490
[58] Tov, O., Alaluf, Y., Nitzan, Y., Patashník, O., Cohen-Or, D., 2021. Designing an encoder for stylegan image manipulation. arXiv preprint arXiv:2102.02766.
[59] Wang, B., Ponce, C.R., 2021. The geometry of deep generative image models and its applications. CoRR abs/2101.06006. URL: btttp://aaxxiv.rrgabbb//// 06006 2eXiv:2101.0660
[60] Wang. C., Chai, M., He, M., Chen, D., Liao, J., 2021a. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. CoRR abs/2112.05139, URL: httpps///axxi... org/
abs/ 2b12 05139 ar Xiv: 211.0112222 05139
[61] Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W., 202 Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. CoRR abs/2106.10689)
URL: https: pectpa://arxive wwwpziiiiorr//aaa.. abaaa/aaa2210006 10689 arXiv: 10689
[62] Wang, T., Zhang, B., Zhang, T., Gu, S., Bao, J., Baltrusaitis, T., Shen, J., Chen, D., Wen, F, Chen, Q., Guo, B., 2022. Rodin: A generative model for sculpting 3d digital avatars using diffusion. aarXiv: 06135
[63] Weng, C.Y., Curless, B., Srinivasan, P.P., Barron, J.T., Sennemmacher-Shizzemmee I., 2022. Humannerf: Free-viewpoint rendering of moving people from monocular video. arXiv:2203 2211.01127 04127
[64] Wu, Z., Lischinski, D., Shechtman, E., 2020. Stylespace analysis: Disentangled controls for stylegan image generation. arXiv: 2011. 12799.
[65] Xu, Y., Yin, Y., Jiang, L., Wu, Q.. Zheng, C., Loy, C.C., Dai, B., Wu, W., 2022. TransEditor: Transformer-based dual-space GAN for highly controllable facial editing, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[66] Yang. B., Zhang, Y., Xu, Y., Li, Y., Zhou, H., Bao, H., Zhang, G., Cui, Z., 2021. Learning object-compositionaa neural radiance field for editable scene rendering.
[67] Yang, G., Vo, M., Neverova, N., Ramanan, D., Vedaldi, A., Joo, H., 2023. Banmo: Building animatable 3d neural models from many casual videos. ArXiv:2112. 2112.22112.27111
[68] Yariv, L.. Gu, J.. Kasten, Y., Lipman, Y., 2021. Volume rendering of neural im- plicit surfaces. CoRR abs/2106.12052. URL:
12052
arXiv:2106. 12052
[69] Yu, Y., Zhan, F. Wu, R., Zhang, J., Lu, S., Cui, M., Xie, X., Hua, X.S., Miao, C., 2022. Towards counterfactual image manipulation via clip. arXiv:2207 02812
[70] Yuan, W., Lv, 7.,, Schmidt, T., Lovegrove, S., 2021. Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering, pp. 13144-13152.
14

---

[71] Zheng. Z., Huang, H., Yu, T., Zhang, H., Guo, Y., Liu, Y., 2022. Structured local radiance fields for human avatar modeling, arXiv::2003 14478
[72] Zhou, H., Hadap, S., Sunkavalli, K., Jacobs, D. D.W., 2019. Deep single portrait image relighting, in: International Conference on Computer Vision (ICCV).
[73] Zhou, Z., Tulsiani, S., 2023. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction, in: CVPR.
15

---

Appendix
A
Implementation Details
Our reconstruction loss and identity loss are applied to the ground truth image camera poses. Due our limited GPU memory, we only render one side view to calculate the diffusion loss and illumination loss at each iteration. The camera rotation angles 9 and e are randomly sampled from [ 12 12 and 13 12 2 + T where 0 and & are the angles of spherical coordinate. We set the optimization iterations for our editing to 500, which takes approximately 10 minutes on a 3090 GPU. We set the weighting LD, LID, Ln to be 0.2, 0.2 and 2 x 10-6 for most editing cases, which can be finetuned for each editing.
-T
Dataset and Generative Bias We utilize trained checkpoints of EG3D on FFHQ [25]. AFHQv2 [II] and ShapeNet [9] for the data domain of face, and car respectively. For some editing and generation, we notice the existence of biases in generated results caused by the bias the training dataset, especially for race, gender, etc.
B
More Results
As supplement to Fig. 1 and Fig. in main paper, Fig. 10ssoww results of our FaceDNeRF. All three figures illustrate that given single face image, our FaceDNeRR can perform semantics- driven NeRF editing various features, such expressions, emotions, glasses, hairstyles, races, genders, ages, makeup, beard, mustache, and goatee. Notably, both individual and joint editing of these features can be achieved in high fidelity.
Geometry
Edited views
Input
Edited views
Geometry
exotic
makeup
wearing dark red lipstick
bridal makeup
helloween makeup
Big smile
A man with short hair
sleepyy
man with thick gostee
Figure 10: More FaceDNeRF results. The middle column shows the input images, while the left and right halves of the images in the other two columns show the results of text prompts editing on the left and right, respectively. Additionally, the first and last columns showcase the corresponding geometries.
Text-conditioned Generation Comparison Our attempts to compare with Dreamfusion 44 (Implemented on a StableDiffusion) failed since it cannot generate faithful models for human heads. This problem might be caused by various reasons. First. the ambiguity of text prompts (human identity, rendering directions) might result an inconsistent denoising behavior at each iteration. Also, the randomized and unconstrained NeRF optimization process might collapse. On the contrary, our utilization of latent code and trained 3D generative models ensures successful and high-quality generation.
16

---

C More Comparisons
C.1 InterfaceGAN and StyleCIIP
Here we provide supplementary comparisons (both qualitative and quantitative) with InterfaceGAN and StyleCLIP. In Fig. [1 since StyleClip proposed two methods Latent Optimization and Latent Mapper, we compared them separately. We adopt the same text example as Fig.4 4 in the StyleClip paper. For clarity, we only present the results of our method and the Latent Optimization method. Please refer the original paper for the results of the Latent Mapper.
The qualitative results demonstrate that our method outperforms the Latent Optimization method in terms of consistency with the text prompt and identity preservation, which also corroborated in the quantitative results in Tab. T It is important to note that the Clip loss alone cannot fully capture the visual perception of distances between images and the semantic meanings of text prompts. In some cases (c.g., Fig.1, I, 1st-row 3rd-column, 3rd-row 3rd-column, and 4th-row 5th-column), the results of the Latent Optimization method exhibit degradation while the Latent Optimization method achieves the lowest Clip loss. On the other hand, our method consistently produces reasonable results across all cases. Even when compared the latent mapper method, which requires 10-12 hours of training for specific text input, our method achieves comparable level of visual quality, Our method only needs around 10mins inference time and no training needed for speeific text prompt. (Times are measured on single NVIDIA GTX 1080ti GPU.)
we compare with InterfaceGAN on more attributes. Note that as the latent direction found by the classifier could be entangled, the editing results of InterfaceGAN often exhibit undesired changes in other attributes which leads identity changes. Tab. quuatitatively verifies that our method preserves Identity better while achieving the target editing.
As shown Fig. 12
Ours
StyleClip
Ours
StyleClip
Input
"Mohawk hairstyle"
"Bob-cur cut hairstyle"
Figure 11: Qualitative comparison of FaceDNeRF with StyleClip (latent Optimization). To mitigate
the influence of various weight choices the final outcomes, we experimented with 40 different combinations encompassing wide range weights. We present the results with IDLoss 0.05 and the highest visual consistency with the given text prompt.
17

---

Table 1: (a) The Latent Optimization method in StyleClip paper, (b) The Latent Mapper method in
StyleClip paper. Ours, (a), and (b) are evaluated on the examples shown in Fig. 11 The values of ID
loss are multiplied by 100. The values of Clip loss are multiplied by 10.
Ours
(a)
(b)
ID Loss
2.26+11226 7.49:0.24
2.40:11.52
1.68
Clip Loss
6.53
7.16:10.19
Table 2: ID loss on InterfaceGAN and Our FaceDNeRF. We note that the latent direction for InterfaceGAN be may entangled, which causes more alternation in Identity.
Ours
InterfaceGAN
Young
0.0102
0.0159
Smile
0.0028
0.0105
Mustache
0.0014
0.0148
C.2 SDS Latent Space
To show the usefulness of using SDS as guidance in GAN latent space, we provide comparison with directly using the original SDS on unconstrained NeRF space. As shown Fig. B the original SDS can only generate a blurry human head, where the process can collapse with less careful prompts. This because of the over-saturation and low-diversity problems of SDS. There are methods directly apply diffusion in 3D human heads (Rodin), yet they rely on 3D ground truth data (synthesized), which limits the quality and diversity 3D head diffusion models, while GAN requires no specific 3D data and achieves high-quality 3D results. Therefore, with trained GAN as constraint, our SDS can effectively utilize the latent space.
C.3 Using CLIP as Guidance
In this subsection, we will discuss why SDS guidance outperforms CLIP guidance significantly. During our comparison with StyleCLIP, we empirically found that the CLIP loss tends to cause degraded images (i.e., images with low image quality and obvious artifacts, not similar to the original image, not coherent to the text prompts). We suspect several reasons for this phenomenon.
During the optimization process using CLIP guidance, the optimization is based only on the current local gradient estimated by the cosine similarity between text embedding and image embedding, we empirically observe that it may overshoot to degraded images. In the SDS setting, the objective is to find the most probabilistic mode the image domain conditioned on the text, which is referred as mode seeking [73]. Also, the t sampled will determine the weight of diffusion guidance, which affects the magnitude of the gradient. This offers a more robust updating process, reducing the possibility of being trapped in a local minimum.
In addition, diffusion models focus high-quality generation, while CLIP focuses learning cross-modal representations for understanding the relationship between images and text. However, as discussed in (6). CLIP and many other multi-modal understanding models suffer from 'bag-of-objects' kind of representations, possibly due to the contrastive pre-training settings. They perform well in understanding nouns, but badly in visual language concepts beyond object nouns, like attributes and relations. In our context, the input text prompt involves many non-object concepts, like emotions, colors, and other attributes of faces and decorations (c.g., "blue" hair, a "bald" man, an "Eastern" face, a "manly" woman, an "angry" lady, a woman "wearing" a pair of glasses, etc.). When using CLIP guidance, the rendered image is encoded calculate cosine similarity with the text embedding, during which spatial and attribute information in the rendered image are lost, with a focus on nouns. Thus, CLIP guidance easily "losses control" over the rendered images and generates degraded images, due to difficulty in understanding non-object concepts. In contrast, noise is injected during the training of diffusion models, which allows the denoiser to incorporate more spatial information during the iteration process. More concepts beyond object nouns are also considered by diffusion models. Therefore, we believe that SDS guidance provides more strict "control" and thus higher quality guidance.
18

---

Smile
Young
Mustache
Figure 12: Comparison with InterfaceGAN, From the top, rows are respectively input, InterfaceGAN results and results. We can observe in results better identity preservation and more faithful edited result to the desired attributes.
(a) NeRF SDS
(b) GAN SDS
Figure 13: Comparison of SDS on GAN Latent Space with SDS on unconstrained NeRF space. (a) The original SDS operates on unconstrained NeRF space can only generate blurry human heads which can even fail in some cases. (b) High-quality results generated from GAN latent space guided by SDS.
D
Ablation Study
D.1 Loss Weights
To investigate the different losses the generated NeRF, we conduct ablation studies on:
(a) Weight of Identity Loss AYD;
(b) Weight of Reconstruction Loss Ant
(c) Weight of Diffusion Loss Ap-
Fig. shows the frontal-view rendering of the resulting NeRF in our ablation studies. The input text prompt is "A bald man with a thick mustache". The central images in each row are identical, adopting (AιD. AR. Ap) = (0.5, 0.4, 3x 10 5). In each row, only one weight is changed to show the effect of the corresponding loss term
Observing the rendering results in row (a) and row (b) in Fig. the results with either larger \ or larger Åg look more similar to the input person, indicating that both Identity and Reconstruction Losses help preserve the person's identity, while the two losses operate in different ways.
Identity Loss calculated using pre-trained ArcFace 14 network as described in Sec. 3.3 in our main paper. ArcFace [14] trains DCNNs for face recognition which maps the face image into feature with small intra-class but large inter-class distance, which means person with different expressions, hairstyles, mustache, subtle wrinkles, should have a similar identity score, thus low
19

---

Figure 14:
Generation Results using EVA3D by randomly sampling around the latent space's 's mean value. Despite the latent variable for generation being close to the mean value of the latent space, there is still a noticeable lack of 3D consistency and visible geometric estimation artifacts.
Identity Loss in our case. Therefore, Identity Loss would not encode details such as hairstyles and subtle wrinkles, but help to preserve large-scale identity information, such as the shape of the head, nose, mouth, eyes, eyebrows, and any apparent wrinkles.
Instead, Reconstruction Loss calculated by Eq. (2) in our main paper takes all pixels into consideration, thus trying to preserve all kinds of features indiscriminately. A large value of Ag results in face NeRF similar to the input image. However, this dilutes the guidance effect from the text prompt's attempt edit the NeRF by removing hair and adding mustache in Fig. 9 From the right two we observe that the identity still has apparent hair, since the large weight of Reconstruction Loss forces the face NeRF be similar as possible the input image.
images of row (b) in Fig. 9
This is trade-off between fidelity to details in the input image versus fidelity to the text prompt. As result, joint utilization of Identity Loss and Reconstruction Loss can significantly alleviate the problem, where Identity Loss can preserve the identity information a less contradictory way in the presence of text guidance.
Results in row (c) of Fig. Shoows the effect of ÅD. larger value of Ap enhances guidance from the diffusion model, while a smaller value of λR produces higher-fidelity results to the input image. Users could choose suitable value of ÄyD, AR, and Xn based on the specific task.
D.2 Editing Prompts
We investigate the influence of the input text prompt in this section. In Fig. 10 and Figure and 3 in our main paper, we mainly show NeRF editing results when the input text prompt is a short sentence or full description of the expected NeRF, such as "A woman wearing dark red lipstick". While this helps a valid generation that avoids ambiguity in semantics, our FaCeDNeRF can also take in a text prompt that only expresses the difference between the input face and the output face, such "Dark lipstick", the input face has no lipstick at all. Here, Fig. shows the ablation results. The right two columns show NeRF editing results when input text prompts are full descriptions of the expected editing results, while the left two columns are results with text prompts only describing what should be different.
In most cases, our FaceDNeRF can generate similar editing results given either type of text prompt with subtle or even tuning of weights of losses Ayp. An, and Åp. as shown in Fig. 15 However, sometimes ambiguity in semantics interferes editing when the imput text prompt is not a full description of the expected output. As shown in Fig. 16 even though our FaceDNeRF can generate an old Elon Musk given "An old man" as the input text prompt, fails when the input becomes just "Old". We believe this is because "Old" has various meanings depending on its context. Therefore, a single "Old" without any context results in useless guidance from our diffusion model, thus producing poor editing results. "Elderly" and "senior" are two synonyms of "Old" in this context. As shown in Fig. [6] "senior" also fails due to its semantic ambiguity with no context, while "clderly" succeeds because of its specific meaning.
Thus, users of our FaceDNeRF are advised give better and complete text prompt to avoid semantics ambiguity.
20

---

Edited views
Input
Edited views
A woman wearing dark lipstick
Dark red lipstick
A = 0.4
A 0.4
À = 0.5
À R 0.6
À 2e-5
À D 2e-5
with big smile
Big smile
A
A 0.4
R 0.5
À R 0.5
D 2e-5
À D = 2e-5
A bald man with thick mustache
bald, mustache
d 0,4
A 0.5
0.45
0.4 3e-5
D 3e-5
Thick goatee
with thick goatee
ÅID 0.4
A 0.4 0.5
0.5
3e-5
3e-5
Figure 15: Ablation study on the prompt. On the right are results generated with full text prompts. The left ones are produced using simlfied prompts. Given text prompt describing the wanted change, FaceDNeRF generates results with desired editing.
Edited views
Input
Edited views
An old man
Old
A = 0.55
dy 0.55
À = 0.4
À R 0.4
A 2e-5
Ap = 2e-5
elderly
Åι 0.55
Ay 0.55
AR 0.4
R
Àp
2e-5
Ap 2e-5
70-year-old
senior
Åyr 0.55
Ayp 0.55 À R
AR
Àp 2e-5
Figure 16: Ablation study on synonyms. Here are results generated with various Synonyms, which could affect the results. The contest important for synonyms eliminate ambiguity, the case of "senior" and A senior man".
21


Reviews:
[{"review_id": "9o6tJkhjXq", "reviewer": "Reviewer_JGgm", "summary": "This work proposes a 3D generative model-based optimization pipeline for 3D editing from a single image which is a useful and challenging task. The proposed pipeline achieves text and lighting-driven editing by reusing several existing pre-trained models. By carefully designing the losses trade-off, it achieves reasonable results while keeping 3D view consistency.", "questions": "More comparisons and discussions as mentioned in the weakness are required.", "limitations": "The potential negative societal impact should be discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed pipeline for semantic-driven NeRF 3D editing from a single image is composed of several pre-trained models such as EG3D, StableDiffusion, ArcFace, facial SH lighting estimator, and VGG. The editing process is formulated as an optimization problem with several objectives to trade-off the identity preservation, image quality, and editing text/view/lighting consistency. The pipeline achieves reasonable results while keeping 3D view consistency by reusing existing prior knowledge. \n\n- Several engineering modifications have been done to make the whole pipeline work, including differential operation replacement, latent regularization, face view choices. Although the whole system is a bit heavy and the technical novelty is moderate, it can be applied to real photo via GAN inversion and achieves reasonable semantic-driven NeRF 3D editing results with control from text and lighting.\n\n- The supplementary video demonstrates the effectiveness of the proposed method in semantic editing while preserving the identity/view consistency. The paper organization and illustrations are easy to follow.", "weaknesses": "- The comparisons with prior works (e.g. InterfaceGAN, FENeRF, StyleClip) are not enough. There are only a few samples illustrated in Fig. 4. It would be good to see more comparisons and discussions. As claimed that FDNeRF is not limited to human faces, it would be good to show more results on other datasets, especially edited results.\n\n- It is mentioned in line 145 “VGG16 image encoder and ArcFace … due to mismatched viewing directions”. It would be interesting to see if small view changes can benefit the performance since VGG features and ArcFace features are not that pixel-aligned, especially for ArcFace features.\n\n- Clearly boundary can be observed between face and hair, e.g., the edited views of the last row in Figure 3. More discussion and analysis on this issue would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful reviews and suggestions. We are encouraged that you appreciate the three strengths of our work. Following your suggestions, we compared our work with InferfaceGAN and StyleClip quantitatively and qualitatively in Fig.1, Fig.3, Tab.1, and Tab.2. To show the generalizability of our model, we replaced EG3D with PanoHead and EVA3D respectively. \n\nFig.2 demonstrates the results of replacing EG3D with PanoHead which extends EG3D from frontal views to full head views in 360$^\\circ$. Our method can still achieve high-fidelity semantic editing. Since the structure of PanoHead is almost the same as EG3D, structure, loss terms, and other details of PanoHead-based pipeline are almost the same as the EG3D-based pipeline. \n\nFig.5 presents editing examples of articulated human bodies, a very challenging domain. We use EVA3D (ICLR2023 Spotlight), a SOTA human body NeRF generator to replace EG3D while deleting identity loss, which works only on human faces.\n\nFor feature loss, given a human body image, we either estimate the camera pose and compute the feature loss with a rendered image from the same viewpoint in each iteration, or perform inversion to find the corresponding latent vector of the input image, and then compute the feature loss using single or multiple pairs of images rendered from random view directions. While EVA3D authors claimed both high-quality inversion and camera pose estimation can be achieved in the paper, unfortunately, at the time of writing this rebuttal the official codes have not been released. Therefore, for experimental purposes, we perform editing on images generated from EVA3D so that the camera is known. \n\nDuring each iteration, we compute the feature loss using the input image and a rendering of the current human body NeRF from the same viewpoint. Diffusion loss and the optimization pipeline are the same as our EG3D version. As shown in Fig.5, despite the suboptimal generation quality of EVA3D, our editing is still flexible and reasonable. We apologize that we do not have time to implement EVA3D inversion and camera pose estimation. We will release code for more domains after publication. \n\nEG3D contains the geometry constraint of the human head. Thus, if the identity is preserved in one view, images rendered from other views should also preserve the same identity. We did some experiments which show that there is no consistent improvement of adding more constraints on the images rendered from small view changes. Due to the space limit, we do not show the results of the experiments and will add them in our camera-ready paper.\n"}, {"review_id": "xZFPjPLQur", "reviewer": "Reviewer_M2ze", "summary": "This article primarily introduces a method called Face Diffusion NeRF (FDNeRF) for 3D face reconstruction and editing. The key feature is the use of pre-trained 3D GAN and 2D latent diffusion models as priors, allowing users to operate and construct Face NeRF without requiring explicit 3D data. FDNeRF also incorporates carefully designed lighting and identity preservation losses, as well as multimodal pre-training, enabling users to have some control during the editing process. Facial NeRF can be created and edited using only single-view images, text prompts, and explicit target lighting. The article also compares FDNeRF with existing editing methods, evaluating its editability, editing effects, flexibility, and rendering quality through experimental assessments.\n", "questions": "1. What efforts did this work make to ensure identity consistency in cross-domain image editing? Is it still heavily constrained by the field of face reconstruction and editing?\n2. From the results of the article, it seems that lighting remains faithful to facial changes regardless of pose limitations. Is this reasonable? Should lighting be fixed in the environment?\n", "limitations": "The editing effect of FDNeRF is influenced by semantic ambiguities in the input text prompts. It is also limited by GAN, which restricts the controllable range of poses.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "FDNeRF utilizes the EG3D network to reconstruct high-quality and realistic 3D facial models from a single image.\nFDNeRF adopts a stable diffusion model as a guiding model, allowing effective semantically-driven editing of the 3D model based on input textual prompts.\nFDNeRF introduces a novel lighting loss function that enables explicit control of the lighting of the 3D model in a view-consistent manner.\nCompared to other methods, FDNeRF achieves higher computational efficiency and can generate high-quality 3D models in a shorter time frame.", "weaknesses": "FDNeRF primarily focuses on facial reconstruction and utilizes the EG3D network to achieve high-quality 3D reconstruction. This specific network architecture may not perform as effectively for other objects or scenes.\nWhile FDNeRF allows for semantically-driven editing, the level of control over specific facial features or details may be limited. Fine-grained editing capabilities, such as manipulating individual facial landmarks or expressions, may not be as precise or controllable as Fig.3. \nAs with any AI-based system, FDNeRF may be susceptible to biases and inaccuracies. The training data and algorithms used in FDNeRF can inadvertently introduce biases or produce inaccurate results, particularly when dealing with diverse or underrepresented populations.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful reviews and suggestions.\n\nFor your concerns about editing beyond the photorealistic human face domain, our diffusion-guided NeRF generator latent vector optimization pipeline can generalize to more domains by replacing EG3D with other NeRF generators. Fig.2 demonstrates the results of replacing the EG3D with the PanoHead which extends EG3D from frontal views to full head views in 360$^\\circ$. Our method still can achieve high-fidelity semantic editing. Fig.5 presents editing examples of articulated human bodies, a very challenging domain. We use EVA3D (ICLR2023 Spotlight), a SOTA human body NeRF generator to replace EG3D while deleting the identity loss, which works only on human faces. \n\nFor feature loss, given a human body image, we either estimate the camera pose and calculate feature loss with a rendered image from the same viewpoint in each iteration, or perform inversion to find the corresponding latent vector of the input image and compute the feature loss using single or multiple pairs of images rendered from random view directions. While EVA3D authors claimed both high-quality inversion and camera pose estimation can be achieved in the paper, unfortunately, at the time of writing this rebuttal the official codes have not been released. \nTherefore, for experimental purposes, we perform editing on images generated from EVA3D so that the camera is known. \n\n\nDuring each iteration, we compute the feature loss using the input image and a rendering of the current human body NeRF from the same viewpoint. Diffusion loss and the optimization pipeline are the same as our EG3D version. As shown in Fig.5, despite the suboptimal generation quality of EVA3D, our editing is still flexible and reasonable. We apologize that we do not have time to implement EVA3D inversion and camera pose estimation. We will release code for more domains after publication. \n\nRegarding the fine-grained editing capabilities, we acknowledge its advantage of precise control. However, relying only on predefined controlling parameters is suboptimal as semantically complex attributes such as age, gender, and makeup are hard to define (as in the case of FeNeRF and IDE3D). For the purpose of fine-grained control, a natural extension of our method could replace the origin Stable Diffusion by ControlNet, the generation process of which could be controlled by landmarks.  \n\nFor the lighting formulation, we could define a lighting position for the image. Then when viewing the image from different directions, the lighting source should be fixed (Imagine a 3D image captured, its lighting direction is fixed). Also, it is easy to change the lighting direction in our method explicitly as shown in the paper.\n"}, {"review_id": "Y2xaSzZb2M", "reviewer": "Reviewer_1wAz", "summary": "The authors proposes an approach for EG3D-based image editing in this paper. \n\nTo be specific, they utilize modified Score Distillation Sampling (SDS) on a pretrained EG3D along with reconstruction and ID loss for semantic editing (e.g., change age, eyeglasses, and hair) and relighting from a single image. \n\nThe results show good editing fidelity and 3D geometry consistency.", "questions": "- A minor suggestion for \"Seam\" effect of the face. For example, in Figure 3 \"An old man\" case, we can see the \"seam\" artifact along the edge of the faces. This is probably because of the imbalanced camera pose distribution of FFHQ.  A possible solution is proposed by IDE3D [4] by using a density regularization.  \n\n[4] Sun, Jingxiang, et al. \"Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis.\" ACM Transactions on Graphics (ToG) 41.6 (2022): 1-10.", "limitations": "The paper has already discussed possible ethical issues brought by the technique in the paper. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper shows some strengths:\n- The results from a single image show fair identity preservation, editing fidelity and 3D geometry consistency. \n- The proposed method can be migrated to other data domain, e.g., cats and cars. \n", "weaknesses": "There are some weaknesses.\n- Inference time. The proposed method is an optimization-based method, which means this usually requires long time. I'd like to see what the inference time is for FDNeRF and how it compares with other methods, e.g., StyleCLIP, InterfaceGAN, and DPR for relighting. \n- Missing citations and limited novelty. Using SDS score for face editing is not new. For example, [1] uses the SDS score for Style transfer. [2] considers the SDS score as an alternative for multiple applications. \n- Missing experimental results. For face editing, it is common to use the ID similarity before and after editing to evaluate the ID preservation [3]. The paper does not include ID similarity as a metric to evaluate it. \n\n\n[1] Song, Kunpeng, et al. \"Diffusion guided domain adaptation of image generators.\" arXiv preprint arXiv:2212.04473 (2022).\n\n[2] Cohen-Bar, Dana, et al. \"Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenes.\" arXiv preprint arXiv:2303.13450 (2023).\n\n[3] Roich, Daniel, et al. \"Pivotal tuning for latent-based editing of real images.\" ACM Transactions on graphics (TOG) 42.1 (2022): 1-13.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank you for your thoughtful reviews and suggestions. We are encouraged that you appreciate the two strengths of our work.\n\nFollowing your suggestions, we conduct more comparisons and evaluate them quantitatively using ID similarity. The results are given in the pdf file as shown in Fig.1 and Fig.3. For DPR relighting comparison, in the main paper, we render a video with fixed lighting conditions. As a 2D method, DPR fails to preserve consistent lighting when the image is conditioned from different viewpoints. We find that this is quite obvious and do not include these sanity tests in the rebuttal to save space. We will put more comparisons in the supplementary upon publication.\n\nThe inference time for our method is 12 mins (NVIDIA GTX 1080 GPU), compared with 10-12 hour training of the Latent Mapper of StyleCLIP. The inference time of InterfaceGAN is fast (inversion then add latent code with normal vector), yet the quality of editing is limited by the classifier and latent space of GAN (whether it is inherently entangled). The acquisition of both a classifier and the latent boundaries takes hours of training and labeled data (for the classifier). \n\nWe acknowledge the missing citations and we will cite the mentioned papers. Our method differs from \"Diffusion guided domain adaptation of image generators'' which focuses on adapting the generator to new domains, while our method focuses on finding the suitable latent code given the generator. As for \"Set-the-Scene,'' we consider the work as a concurrent work to ours and our tasks are very different. We thank your suggestion of PTI and utilize it in our inversion process of Panohead. We also thank your suggestion on the \"seam effects.''\n\nPlease see the sections in the responses to common concerns for our novelty and motivation.\n\nFinally, we would like to emphasize that our contribution is not only EG3D-based photorealistic semantics-driven editing but also relighting. We believe our optimization pipeline starts a new approach in semantics-driven editing and relighting given any NeRF GAN, and thus has a good impact and can spawn worthwhile future work in the years to come."}, {"review_id": "Ok6nYIwQHW", "reviewer": "Reviewer_4TUt", "summary": "The paper introduces FDNeRF, a generative method that reconstructs Face NeRFs with semantic editing and relighting capabilities. FDNeRF utilizes 3D GAN inversion and a trained 2D latent-diffusion model to enable zero-shot learning, eliminating the need for explicit 3D data. With designed loss functions and multi-modal pre-training, FDNeRF offers extensive control over editing processes using single-view images, text prompts, and target lighting. Experimental results demonstrate the realistic outcomes and editing flexibility of FDNeRF.", "questions": "- Figure 3, in particular, leaves me perplexed. The prompt-guided editing appears to completely alter the original facial attributes, focusing more on generating new features rather than preserving the original identity.\n\n- I found it challenging to assess the effectiveness of the diffusion loss utilized in this model. It would be beneficial to understand the advantages of the SDS loss in comparison to CLIP guidance. A clear demonstration of the superiority of the SDS loss over other techniques would provide valuable insights.\n\n- ", "limitations": "The author should add a figure to illustrate the limitations. ", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "+ It includes a illumination loss using SH and differentiable renderer to preserve the lighting condition. \n+ The illumination loss allows explicit control over the lighting in a view-consistent manner.", "weaknesses": "- I found it challenging to gain valuable insights from this paper as it primarily combines conventional losses, including the identity loss, SDS loss, reconstruction loss, and illustration loss. From a research perspective, the technique presented in this paper offers limited contributions.\n\n- While the paper showcases facial images from various viewpoints, it fails to evaluate its performance on larger poses. The current results are restricted to angles between -30 to 30 degrees, making it difficult to fully comprehend the effectiveness of this approach for 3D reconstruction and editing tasks.\n\n- The paper lacks a comprehensive evaluation of the inversion process. It is crucial to compare the proposed inversion method with state-of-the-art EG3D inversion methods to establish its performance. Additionally, based on the information presented, the inversion method implemented in this paper appears to be inadequate and could benefit from further refinement.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful reviews and suggestions. We are encouraged that you appreciate the two strengths of our work.\n\nAs explained in responses to common concerns, we extended our method to 360 human heads based on PanoHead. We also utilize PTI for the inversion process. The results of Fig. 2 have already incorporated the PTI fine-tuning. \n\nWe have a known trade-off between identity preservation and degrees of editing in many editing frameworks. As quantitatively and qualitatively evaluated in the global response, our method achieves better identity preservation while flexible for attribute-guided editing. The trend to generate new images may be attributed to several reasons: First, the prompts in our editing are simple (which makes editing easier) and carries no identity information. Therefore, the diffusion model will inevitably generate different results, which makes the usage of ID loss and VGG loss crucial. Second, as in the naive SDS case, the SDS loss tends to be over-saturating. The ID-preservation losses ameliorate this problem.\n\nFor the CLIP loss, during our comparison with StyleCLIP, we empirically found that the CLIP loss tends to produce degraded images (i.e., images with low image quality and obvious artifacts,  not similar to the original image, not coherent to to the text prompts such as examples of 1st-row 3rd-column and 4th-row 5th-column in Fig.1). We suspect several reasons for this phenomenon. First, the denoising process of SDS adds noise at each iteration, which reduces the possibility of being trapped in the local minimum. Second, the CLIP loss relies on cosine similarity, which lacks knowledge of how the current updated step will affect the image. Thus, these possible causes may deviate from the original image drastically. On the contrary, the reverse process (denoising) of SDS tries to minimize the score of the current image conditioned on the text prompt at all iterations, which provides more consistent guidance. \n\nWe argue that our choice of framework is not a simple combination of existing losses. Our motivation to utilize SDS loss in the GAN latent space is justified, together with ID loss and VGG loss that preserves identity and illumination loss that controls the lighting condition explicitly, our methods achieve high-fidelity, zero-shot editing, which in this rebuttal can also be migrated to other data domains easily. Fig.2 and Fig.5 demonstrate our optimization pipeline can be easily applied to other backbones like PanoHead and EVA3D.\n\nFinally, we would like to emphasize that our contribution is not only EG3D-based photorealistic semantics-driven editing but also relighting. We believe that our optimization pipeline starts a new approach in semantics-driven editing and relighting given any NeRF GAN, and thus has a good impact and can spawn worthwhile future work in the years to come. \n\n"}]

Author rebuttal:
We thank all reviewers for their insightful comments and appreciation of our work. Below we address the issues. The referred figures and tables are in a separate PDF file. Following the given review sequence, we use R1, R2, R3, R4 to denote reviewer JGgm, M2ze, 1wAz, 4TUt respectively.  


### [R1, R3, R4] More Comparisons and Evaluations.
We conduct both qualitative and quantitative comparisons with InterfaceGAN and StyleCLIP. 


In Fig.1, since StyleClip proposed two methods Latent Optimization and Latent Mapper, we compared them separately. We adopt the same text example as Fig.4 in the StyleClip paper. For clarity, we only present the results of our method and the Latent Optimization method. Please refer to the original paper for the results of the Latent Mapper. 

The qualitative results demonstrate that our method outperforms the Latent Optimization method in terms of consistency with the text prompt and identity preservation, which are also corroborated in the quantitative results in Tab.1. It is important to note that the Clip loss alone cannot fully capture the visual perception of distances between images and the semantic meanings of text prompts. In some cases (e.g., Fig.1, 1st-row 3rd-column, 3rd-row 3rd-column, and 4th-row 5th-column), the results of the Latent Optimization method exhibit degradation while the Latent Optimization method achieves the lowest Clip loss. On the other hand, our method consistently produces reasonable results across all cases. Even when compared to the latent mapper method, which requires 10--12 hours of training for a specific text input, our method achieves a comparable level of visual quality. Our method only needs around 10mins inference time and no training is needed for a specific text prompt. (Times are measured on a single NVIDIA GTX 1080ti GPU.) 

 As shown in Fig.3, we compare with InterfaceGAN on more attributes. Note that as the latent direction found by the classifier could be entangled, the editing results of InterfaceGAN often exhibit undesired changes in other attributes which leads to identity changes. Tab.2 quantitatively verifies that our method preserves Identity better while achieving the target editing. 

### [R2, R4] Extended Head Pose Range.
The range of head pose is limited by the trained EG3D generator as most facial images are in front view. Recent Panohead addresses this problem and enables head generation in 360 degrees. Our method can employ Panohead by replacing the generator, as shown in Fig.2.


### [R3, R4] SDS in Latent Space \& Difference with CLIP
We acknowledge the usage of SDS in other papers. On the other hand, our utilization of SDS loss on the latent space of trained GAN is motivated by high-fidelity 3D editing with language guidance. As shown in Fig.4, the original SDS can only generate a blurry human head, where the process can collapse with less careful prompts. This is because of the over-saturation and low-diversity problems of SDS. There are methods to directly apply diffusion in 3D human heads (Rodin), yet they rely on 3D ground truth data (synthesized), which limits the quality and diversity of 3D head diffusion models, while GAN requires no specific 3D data and achieves high-quality 3D results. Therefore, with trained GAN as a constraint, our SDS can effectively utilize the latent space. 


During our comparison with StyleCLIP, we empirically found that the CLIP loss tends to cause degraded images (i.e., images with low image quality and obvious artifacts, not similar to the original image, not coherent to the text prompts). We suspect several reasons for this phenomenon. First, the denoising process of SDS adds noise at each iteration, which reduces the possibility of being trapped in the local minimum. Second, the CLIP loss relies on cosine similarity, which lacks knowledge of how the current updated step will affect the image. Thus these possible causes can deviate from the original image drastically. On the contrary, the reverse process (denoising) of SDS tries to minimize the score of the current image conditioned on the text prompt at all iterations, which provides more consistent guidance. 

### [R1, R3] Artifacts Near the Edge
We also observe such artifacts and we thank the reviewer for their suggestions. This could also be caused by the relevant face preprocessing and training camera pose of EG3D. When shifted to Panohead, such artifacts thankfully no longer appear.

### [R1, R2] Cross Domain Editing & Generation
We extend our method to the articulated human body domain, as shown in Fig.5, where we replace EG3D in the submission by EVA3D, a human body NeRF generator. The excellent results we obtained during the short NeurIPS rebuttal period indicate that our pipeline can be generalized to various domains and different GAN architectures. 


### [R4] Inversion Ablation
Following your suggestions, we implemented a PTI version of our method. The code of the PTI version will also be released. The results presented in Fig.2 have already incorporated the PTI fine-tuning.  


Meta-review: The paper received mixed reviews initially, which remained still mixed even after discussions. The remaining concern by Reviewer 4TUt after the rebuttal was that there are still issues with the quality of the results regarding identity preservation and disentanglement. While the results are imperfect, it is clear that the proposed method is still an improvement over the compared baselines. Given that the majority of the reviewers are convinced, and that the remaining issue does not seem critical, the AC recommends accepting the paper.
Decision: Accept (poster)