SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hao Zhang*  
HKUST  
hzhangcc@connect.ust.hk
Yanbo Xu* CMU, HKUST yxubu@connect.ust.hk
Tianyuan Dai*  
Stanford University, HKUST  
tdaiaa@connect.ust.hk
Yu-Wing Tai Dartmouth College yuwing@gmail.com
Chi-Keung Tang HKUST cktang@cs.ust.hk
The ability to create high-quality 3D faces from a single image has become increasingly important with wide applications in video conferencing, AR/VR, and advanced video editing in movie industries. In this paper, we propose Face Diffusion NeRF (FaceDNeRF), a new generative method to reconstruct high-quality Face NeRFs from single images, complete with semantic editing and relighting capabilities. FaceDNeRF utilizes high-resolution 3D GAN inversion and expertly trained 2D latent-diffusion model, allowing users to manipulate and construct Face NeRFs in zero-shot learning without the need for explicit 3D data. With carefully designed illumination and identity preserving loss, as well as multi-modal pre-training, FaceDNeRF offers users unparalleled control over the editing process enabling them to create and edit face NeRFs using just single-view images, text prompts, and explicit target lighting. The advanced features of FaceDNeRF have been designed to produce more impressive results than existing 2D editing approaches that rely on 2D segmentation maps for editable attributes. Experiments show that our FaceDNeRF achieves exceptionally realistic results and unprecedented flexibility in editing compared with state-of-the-art 3D face reconstruction and editing methods. Our code will be available at https://github.com/BillyXYB/FaceDNeRF.
Rich and versatile 3D contents are in high demand in entertainment industries such as movie making, computer gaming and emerging applications such as Metaverse, which also have high potential in robotics learning as well, where high-quality synthetic data in large amounts are required to enhance generalizability. 3D generative methods such as EG3D [8] can generate high-fidelity NeRF from a single image, but the reconstructed NeRF cannot be easily controlled or edited. Some methods [33, 57] use pixel-wise segmentation maps or user-supplied sketches to guide 3D editing. But these methods are hard to scale up due to the demanding editing requirement and limited editable attributes. Language is regarded as one of the most suitable candidates to provide control signals for 3D editing, especially given the current success of 2D semantics-driven editing [28, 42], where image and language domains are bridged by CLIP, i.e., Contrastive Language-Image Pre-training [45]. However, there is still substantial room for improvement in 3D generative and editable models, in terms of usability, edit-ability, and results quality.
Figure 1: FaceDNeRF results. Given a face image, FaceDNeRF can reconstruct, edit, and relit a photo-realistic 3D face NeRF using a text prompt and a target light. Our method is not limited to human faces, and can be applied to other domains as well. Further results and videos are available in the supplemental materials.
Neural Radiance Field (NeRF), a new paradigm in 3D scene representation, embeds a 3D scene in a compact fully-connected neural network and achieves realistic rendering of novel views. NeRF is optimized to approximate a continuous and thus differentiable scene representation function, which has quickly become a dominant approach for many relevant downstream tasks including novel scene composition [29, 38, 41, 66, 70], surface reconstruction [31, 61, 68], and articulated 3D shape reconstruction [10, 23, 43, 63, 67, 71] due to its differentiability, multi-view consistency, compactness, and fast inference. Since NeRF relies on training from multi-view images, its relighting is contingent upon the relighting of these 2D images. Nevertheless, current 2D relighting techniques [5, 22, 34, 15] are unable to ensure view consistency among the relit multi-view images, resulting in a decline in the quality of the relit NeRF.
Significant attempts have been made in recent years to develop semantic-driven NeRF editing models. Pre-trained 2D language-image models are usually leveraged to enable multi-modality in 3D. CLIP-NeRF [60] introduces a disentangled conditional NeRF architecture, where the disentangled latent representations can be bridged by two mappers trained with a CLIP-based matching loss to the CLIP embedding, based on which the latent codes are updated for editing based on the input text prompt. Outperforming CLIP-based methods, DreamFusion [44] later adopts Imagen [49], a pre-trained text-to-image diffusion model, as a prior to guide the optimization of parameters of a randomly initialized NeRF through a novel SDS loss. Magic3D [32] further improves DreamFusion by introducing a coarse-to-fine optimization scheme with diffusion priors at different resolutions. However, these methods have two main limitations: lack of photorealism and long inference time (reportedly taking 1.5 hours for DreamFusion, and 40 minutes for Magic3D).
We believe the main reason underlying the less realistic results and slow inference of DreamFusion [44] and its variants lies in the random and thus unrestricted initialization of NeRF. Leveraging 3D generative models such as EG3D [8], whose generation is restricted by the discriminator during training for constrained NeRF generation, may offer a feasible solution. Although combining 2D generative models with CLIP [45] for realistic image generation and editing has been a widely adopted approach [55], no significant attempts have been made in 3D realistic context to our knowledge. Thus, in this paper, we present FaceDNeRF, which is to our knowledge the first work to enable semantic-driven NeRF editing in 3D with high photorealism and versatile relighting from a single image, given a text prompt and target light. FaceDNeRF freezes the weights of the EG3D network which can reconstruct photorealistic NeRF from a single image. To enable semantic-driven editing, we adopt stable diffusion [48] to guide the optimization through SDS loss introduced in DreamFusion [44]. Based on SDS loss, identity loss, and feature loss, the latent vector in EG3D's latent space can be updated. Moreover, the proposed illumination loss allows explicit control over the lighting in a view-consistent manner.
2D Generation and Editing Research on unconditional or text-driven image generation has made fruitful progress. Generative Adversarial Networks (GANs) [17] contribute to the first revolutionary 2D image generative methods, among which StyleGAN [26] and its variants [24, 27] stand out due to the expressive and well-disentangled latent spaces. StyleGAN-based image editing requires either an encoder trained to map a given image to the latent space [2, 39], or specifying latent update direction which requires explicit ground-truth annotations [1, 19, 59, 65]. Diffusion models [54, 56] represent another class of generative models that enables text-driven, photorealistic and highly diverse image generation. State-of-the-art text-to-image synthesis methods contribute effective mechanisms to guide samples toward semantics: classifier-free guidance [20, 37] generates images with or without class information during model training; CLIP guidance [13, 42, 69] where CLIP [45] trained on 400 million image-text pairs spearheaded cross-modal representation learning in modern vision-language tasks. Leveraging the rich joint embedding spaces of CLIP and expressiveness of diffusion models, stable diffusion [48] is arguably the best text-to-image model to date, synthesizing images of vast domains and styles based on a text prompt. Thus we adopt stable diffusion as the guidance model in our 3D approach.
3D Generation and Editing NeRF [36], an implicit neural representation, has become the dominating modern approach for 3D generation due to its continuity, differentiability, compactness, and quality of novel-view synthesis over mesh and point cloud. GRAF [50] combines implicit neural rendering with GAN for generalizable NeRF. PiGAN [7] utilizes SiREN [53] to condition the implicit neural radiance field on the latent space. Although guaranteed with 3D consistency, volumetric rendering requires heavy computation. With limited computation, the image quality of these methods is still not comparable to those produced by current state-of-the-art 2D GANs. Thus, many recent approaches adopt hybrid structures. StyleNeRF [18] applies volume rendering in the early feature maps in low resolution, followed by upsampling blocks to generate high-resolution images. However, a regularizer based on NeRF is required to ensure 3D consistency during upsampling. Instead of using volume rendering in early layers, EG3D [8] performs the operation on a relatively high-resolution feature map using a hybrid representation for 3D features generated by StyleGAN2 [27] backbone, named tri-plane, which is capable of incorporating more information than an explicit structure such as voxel. StyleSDF [40] shares a similar spirit but uses SiREN [53] for its mapping network, with the mapped result used as the input feature map followed by a style-based generator for upsampling.
Attempts have been made to generate 3D objects using diffusion models. Rodin [62], Realfusion [35], Set-the-Scene [12] and other diffusion-based 3D reconstruction methods have demonstrated the feasibility of constructing a face or other object from a single view image. However, the results are not adequately realistic with limited generalizability due to the scarcity of 3D data. It is also possible to approximate 3D assets using trained 2D diffusion models, as shown in [44, 32]. However, the quality of generated assets is visually worse than the outputs from the utilized 2D models, especially in areas such as the face, where high fidelity is an essential criterion.
Illumination Control 3D editable lighting on NeRF is a highly desirable feature. Image relighting methods can be roughly categorized into two groups: one involves estimating 3D face information such as 3DMM coefficients [5], albedo, and surface normals, and combining this information with a target lighting condition represented by spherical harmonics (SH) coefficients to generate relit images, such as [22, 34, 15]. Although guaranteed with 3D consistency, these methods have limited editability as they cannot easily accommodate changes in the original model, such as wearing glasses or different hairstyles. The other approach involves leveraging 2D/3D generative models to control illumination in the latent space, such as [4, 30]. While this approach is conducive to some editability, illumination can only be implicitly controlled by latent codes, that is, environmental lighting cannot be directly controlled by e.g., SH coefficients or cube mapping. Our method is amenable to both light editing means, bypassing any intrinsic separation which is not necessary in FaceDNeRF.
3.1 EG3D and  $\mathcal{W}^+$  Space
Our method utilizes trained EG3D generator [8] with its respective latent space. From initial latent code  $z \in \mathcal{R}^{512}$ , a mapping network  $\mathcal{M}$  maps  $z$  to  $w$  in the space named  $\mathcal{W}$ , where  $w \in \mathcal{R}^{1 \times 512}$ .
Figure 2: FaceDNeRF structure. The Latent 512 scalars are initialized as the mean of sampled  $\mathcal{W}^+$  codes. The model computes the Reconstruction Loss and Identity Loss from the input image's camera view. From other side views, given a text prompt, the model computes the Diffusion Loss by assessing the discrepancy between the predicted noise and random noise. The Illumination Loss is then computed by comparing the estimated SH coefficients of the rendered side-view image to the target SH coefficients. These Losses are then utilized to update the Latent 512 scalars iteratively through a carefully designed differentiable model via back-propagation.
During training and generation, the  $w$  code will be utilized to modulate all convolution layers as in StyleGAN2 [27]. The generated features will be reshaped into three orthogonal feature plans  $(F_{xy}, F_{yz}, F_{xz})$ , where each of them has the resolution of  $N \times N \times C$ . Given a camera pose, an augmented feature can be rendered using volume rendering as in [36], which will then be upsampled by the super-resolution blocks. The rendering process acts as an inductive bias that enforces view-consistent results.
The inversion process inverses an input image to the latent space, such that the inverted code  $w'$  can faithfully reconstruct the given input. As shown in 2D GAN inversion [46], the quality of reconstruction is better when the inversion is conducted on  $\mathcal{W}^+$  space, where its latent codes  $w^+ \in \mathcal{R}^{L \times 512}$  are used to separately modulate all  $L$  convolution blocks. In addition, the  $w^+$  space preserves the editability [64, 58]. Thus, the reconstruction and editing process of our method operates on the  $\mathcal{W}^+$  space.
3.2 Formulation
Fig. 2 summarizes our method, which takes a single image  $x$ , text prompt  $y$ , and target lighting represented by SH coefficients  $l$  as inputs, and generates a reconstructed 3D face NeRF as output. Inspired by high-fidelity face NeRF reconstruction techniques, such as EG3D inversion [8]. The Reconstruction and Identity Preservation (Sec. 3.3) is employed to guide the facial NeRF reconstruction process. To enable language-guided editing, we incorporate a Diffusion component (Sec. 3.4), which is based on the trained text-conditioned latent diffusion model. Notably, our method allows for explicit illumination control, thanks to the inclusion of the Illumination component (Sec. 3.5). To generate the target latent code, we randomly sample a value near the mean value of the latent space as a starting point. Then we solve an optimization problem to obtain the desired output.
$\mathcal{L}_R, \mathcal{L}_{ID}, \mathcal{L}_D$  and  $\mathcal{L}_{IL}$  denote the Reconstruction loss, Identity Loss, Diffusion Loss and Illumination Loss, which will be elaborated in Sec. 3.3, Sec. 3.4 and Sec. 3.5. We solve the following optimization:
$$
\underset {w \in \mathcal {W} +} {\arg \min } \lambda_ {I D} \mathcal {L} _ {I D} (G (w, c), x) + \lambda_ {R} \mathcal {L} _ {R} (G (w, c), x) + \lambda_ {D} \mathcal {L} _ {D} (G (w, c _ {s}), y) + \lambda_ {I L} \mathcal {L} _ {I L} (G (w, c _ {s}), l _ {c _ {s}}). \tag {1}
$$
where the  $G(\cdot, \cdot)$  is the EG3D generator,  $c$  is the camera pose of the input image (which can be estimated by [16]),  $c_s$  is the random side camera pose.  $l_{c_s}$  is the target illumination which varies with  $c_s$ .  $\lambda_{ID}$ ,  $\lambda_R$ ,  $\lambda_D$  and  $\lambda_{IL}$  are weights of the corresponding loss functions. We optimize the  $w$  iteratively through gradient descent by back-propagating the gradient of the objective Eq. (1) through these four weighted differentiable loss functions.
3.3 Reconstruction and Identity Preservation
The desired editing should preserve the background and identity of the input image, and hence we design the Reconstruction Loss and Identity Loss. Given input image  $x$  and rendered image  $G(w,c)$ , we utilize VGG16 image encoder  $V(\cdot)$  [52] to extract the image features and construct the Reconstruction Loss as following:
$$
\mathcal {L} _ {R} (G (w, c), x) = \left| \left| V (G (w, c)) - V (x) \right| \right| _ {2} ^ {2} \tag {2}
$$
For human faces, we adopt the same Identity Loss as [46]:
$$
\mathcal {L} _ {I D} (G (w, c), x) = 1 - \langle R (x), R (G (w, c)) \rangle \tag {3}
$$
where  $R$  is the pretrained ArcFace [14] network. VGG16 image encoder and ArcFace  $\mathcal{L}_R$  and  $\mathcal{L}_{ID}$  are only used to measure the difference between the input image and the rendered image from input camera's view to avoid the misalignment due to mismatched viewing directions. See our ablation study for an insightful analysis of the interaction between the Reconstruction and Identity losses.
3.4 Prompt Editing with Diffusion Model
Although directly optimizing a NeRF by distilling a 2D diffusion model is possible [44, 32], the quality of the resulting model is often worse than the used 2D model, especially in the domain of human subject (refer to Appendix C.2). In addition, compared with 3D diffusion models, 3D GANs currently are capable of producing high-fidelity 3D NeRFs with a smooth latent space. Therefore, we perform Score Distillation in the latent space of trained 3D GAN model.
We modify the Score Distillation Sampling (SDS) from DreamFusion [44] as our diffusion loss. The loss connects the  $x'$  rendered from a sampled camera pose  $c_s$  with the denoising prediction conditioned on the editing prompt  $y$ . The denoising process can be written as:
$$
\mathcal {L} _ {D} (x ^ {\prime} = G (w, c _ {s}), y) = \mathbb {E} _ {\varepsilon (x ^ {\prime}), y, t, \epsilon} \left[ \| \hat {\epsilon_ {\phi}} (\mathbf {z _ {t}}; y, t) - \epsilon \| _ {2} ^ {2} \right] \tag {4}
$$
where  $\varepsilon (\cdot)$  is the image encoder that encodes our rendered image  $x^{\prime} = G(w,c_{s})$  to the latent space of the diffusion model, denoted as  $z$ . Here  $z_{t}$  is the noisy version of  $z$  at time-step  $t$ ,  $\hat{\epsilon}_{\phi}$  is the frozen denoising network of the trained diffusion model, where we sample  $t\sim \mathcal{U}(0.02,0.98)$  to avoid very high and low noise levels;  $\epsilon \sim \mathcal{N}(0,I)$ , and its effect on input latent varies with time-step  $t$ , which is same as done in Dreamfusion [44].
Notably, the camera pose  $c_{s}$  will be resampled randomly in each optimization iteration to ensure 3D view consistency. Unlike DreamFusion, our text prompt  $y$  is view independent, since we have the identity and reconstruction losses to constrain the optimization process, and that the EG3D generator has underlying 3D information in contrast to randomly initialized NeRF. Therefore, our diffusion loss back-propagates to the latent scalars, where the gradient of  $\mathcal{L}_D$  is given by
$$
\nabla_ {w} \mathcal {L} _ {D} \left(x ^ {\prime} = G \left(w, c _ {s}\right), y\right) \triangleq \mathbb {E} _ {\varepsilon , t, \varepsilon} \left[ w (t) \left(\hat {\varepsilon} _ {t} \left(\mathbf {z} _ {\mathbf {t}}; y, t\right) - \epsilon\right) \frac {\partial x ^ {\prime}}{\partial w} \right] \tag {5}
$$
As in Dreamfusion [44],  $w(t)$  absorbs the coefficient of the forward process, and we ignore the term  $\frac{\partial \hat{\varepsilon}_t(\mathbf{z_t}; y, t)}{\partial \mathbf{z_t}}$  as the diffusion model is frozen.
3.5 Explicit Illumination Control
As the trained 3D GAN mapped a continuous latent code to a 3D-consistent NeRF, direct manipulation of the latent ensures view-consistent illumination editing. We utilize [72] to construct our Illumination Loss, where the hourglass network can be denoted as:
$$
\mathbf {L} _ {s} ^ {*}, \mathbf {I} _ {t} ^ {*} = \mathbf {H n} \left(\mathbf {L} _ {t}, \mathbf {I} _ {s}\right) \tag {6}
$$
Figure 3: More FaceDNeRF results. The middle column shows the input images, while the left and right halves of the images in the other two columns show the results of text prompts editing on the left and right, respectively. The first and last columns show the corresponding geometries.
where  $\mathbf{L}_t$  and  $\mathbf{L}_s^*$  are respectively the target SH lighting and the estimated SH lighting of the input image  $\mathbf{I}_s$ , and  $\mathbf{I}_t^*$  is the relit image under the target SH lighting. Here, we only use  $\mathbf{L}_s^*$  to compute the  $L_1$  loss with  $\mathbf{L}_t$ , and  $\mathbf{I}_t^*$  will be ignored, i.e.,  $\mathbf{L}_s^* = \mathbf{H}\mathbf{n}'(\mathbf{I}_s)$ . Thus, our Illumination Loss is:
$$
\mathcal {L} _ {I L} (G (w, c _ {s}), l _ {c _ {s}}) = \left\| \mathbf {H n} ^ {\prime} (G (w, c _ {s})) - l _ {c _ {s}} \right\| _ {1} \tag {7}
$$
During the optimization process of  $w^{*}$ , differentiability is crucial. Given  $G(w, c_{s})$  is differentiable with respect to  $w$ , the differentiability of  $\mathbf{Hn}'(G(w, c_{s}))$  with respect to  $G(w, c_{s})$  guarantees the back-propagation onto the  $w$ . Therefore, we replace the PyTorch in-place operations and other numpy operations in the  $\mathbf{Hn}'$  model with equivalent differentiable PyTorch tensor operations. As verified by experiments, our modified  $\mathbf{Hn}'$  achieves the same performance as the original model while being multi-view consistent as we render the output.
3.6 Pivotal Tuning Inversion
The Pivotal Tuning Inversion (PTI) technique [47] is widely employed in many GAN inversion processes to overcome the trade-off between distortion and editability [58]. After solving the Eq. (1), the obtained latent code  $w_{t}$  is utilized as a pivot to fine-tune the generator  $G$  using the similar loss function:
$$
\underset {G} {\arg \min } \lambda_ {I D} \mathcal {L} _ {I D} (G (w _ {t}, c), x) + \lambda_ {R} \mathcal {L} _ {R} (G (w _ {t}, c), x) + \lambda_ {D} \mathcal {L} _ {D} (G (w _ {t}, c _ {s}), y) + \lambda_ {I L} \mathcal {L} _ {I L} (G (w _ {t}, c _ {s}), l _ {c _ {s}}).
$$
This section presents our editing results and compare them with representative methods, emphasizing FaceDNeRF's disentanglement capability which is conducive to editing control, its flexibility in utilizing a text-guided diffusion model, as well as the multi-view consistency in illumination control. Furthermore, our method can be migrated to other data domains with trained generative models.
4.1 3D Editing from Single Image
Although diffusion models can generate diverse and realistic images in 2D, the lack of large and high-quality 3D datasets limits the performance of current diffusion models. On the other hand, the adversarial learning mechanism together with inductive bias of 3D rendering enables GAN to produce high-quality 3D results. From GAN's smooth latent space, our method finds the suitable latent code whose semantic information matches the editing demand. Fig. 1 and Fig. 3 demonstrate the detailed editing results of FaceDNReF. Please refer to Sec. B of the appendix for more results.
Figure 4: Editing Comparison with representative methods. We compare with classifier-based InterfaceGAN [51], semantic edited FENeRF [57] and language guided StyleClip [42] with EG3D. Images on the left side are editing results of gender, smile, and "An Eastern face" respectively. The right images are age editing comparison with [51], and our input text prompt are "A XX is about YY years old", where XX and YY are depicted above. The comparison illustrates the flexibility and high-fidelity editing capability of FaceDNeRF.
Figure 5: Illumination Comparison. The images in the first row are relit images generated by DPR [72]. The images in the second row are relit images generated by our method. Upon comparing the four red boxes, our method is observed to produce fewer artifacts and more realistic results. Furthermore, the images in the third row demonstrate that our method can not only edit face NeRF using text prompts but also allows explicit control of lighting.
Comparison with representative editing methods For effective and easy editing, the underlying latent space should be smooth and semantically meaningful. Compared with the latent space of 2D GANs, disentangling in 3D is much harder. InterfaceGAN [51] seeks a hyper-plane in the latent space with pre-trained classifiers, which makes editing feasible by interpolating latent codes in the direction orthogonal to the hyper-plane. However, the assumption that a good hyper-plane exists which is well-behaved for linear interpolation is only valid when the latent space is highly disentangled. As shown in Fig. 4, the underlying latent space in 3D is not well disentangled. Thus the interpolation (induced by editing) can interfere and affect other irrelevant attributes not to be edited. We also compare with the semantic-based method FENeRF [57], whose editing is performed by manually editing the semantic map. However, this representation limits editable attributes, especially on semantically complex attributes such as age and gender. In addition, we compare with language-guided StyleClip [42] using the same latent space. Our method achieves better editing quality conditioned on the same text prompt, indicating the superiority of utilizing the diffusion model as guidance. Please refer to