Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
Authors: Keqiang Sun
Shangzhe Wu
Zhaoyang Huang
Ning Zhang
Quan Wang
Hongsheng Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: controllable
neural radiance field
face synthesis

Abstract: Capitalizing on the recent advances in image generation models, existing controllable face image synthesis methods are able to generate high-fidelity images with some levels of controllability, e.g., controlling the shapes, expressions, textures, and poses of the generated face images. However, these methods focus on 2D image generative models, which are prone to producing inconsistent face images under large expression and pose changes. In this paper, we propose a new NeRF-based conditional 3D face synthesis framework, which enables 3D controllability over the generated face images by imposing explicit 3D conditions from 3D face priors. At its core is a conditional Generative Occupancy Field (cGOF) that effectively enforces the shape of the generated face to commit to a given 3D Morphable Model (3DMM) mesh. To achieve accurate control over fine-grained 3D face shapes of the synthesized image, we additionally incorporate a 3D landmark loss as well as a volume warping loss into our synthesis algorithm. Experiments validate the effectiveness of the proposed method, which is able to generate high-fidelity face images and shows more precise 3D controllability than state-of-the-art 2D-based controllable face synthesis methods.

Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
Keqiang Sun'; Shangzhe Wu Zhaoyang Huang',
Ning Zhang', Quan Wang Hongsheng Li
CCUKK MML.b 2xford VGG
Sense'Time Research
*Centre for Perceptual and Interactive Intelligence Limited
kqaundlink ink cuhk edu. hk, szwudrobota uk, hsli@ee. cuhk edu
Abstract
Capitalizing on the recent advances in image generation models, existing con- trollable face image synthesis methods are able to generate high-fidelity images with some levels of controllability, c.g., controlling the shapes, expressions, tex- tures, and poses of the generated face images. However, these methods focus on 2D image generative models, which are prone to producing inconsistent face images under large expression and pose changes. In this paper, we propose a new NeRF-base conditional 3D face synthesis framework, which enables 3D controllability over the generated face images by imposing explicit 3D conditions from face priors. At its core is a conditional Generative Occupancy Field (cGOF) that effectively enforces the shape of the generated face to commit to a given 3D Morphable Model (3DMM) mesh. To achieve accurate control over fine-grained 3D face shapes of the synthesized image, additionally incorporate a 3D landmark loss as well as a volume warping loss into our synthesis algorithm. Experiments validate the effectiveness of the proposed method, which can generate high-fidelin face images and shows more precise 3D controllability than state-of-- the-art 2D- -asd controllable face synthesis methods. Find code and more demo at https://keqiangsun http:///ghaaaauuggggggi..////ppp githubbio/poojjet//cggo
Introduction
Recent success of Generative Adversarial Networks (GANs) 12 has led to tremendous progress in face image synthesis. State-of-the-ar methods, such StyleGAN [99 20 18), capable of generating photo-realistic face images. Apart from photo-realism, being able to control the appearance of the generated images also key in many real-world applications, such face animation, reenactment, and free-viewpoint rendering. Early works on controllable face synthesis rely external attribute annotations to learn attribute guided face image generation model 25 10 53)
However, these attributes, such as "big nose", "chubby" and "smiling" in CelebA dataset 24 can only provide abstract semantic-level guidance on the generation, and the generated faces often lack 3D geometric consistency. Moreover, it often much harder to obtain low-level geometric annotations beyond semantic labels for direct 3D supervision.
Recently, researchers have attempted to incorporate 3D priors from parametric face models, such as 3D Morphable Models (3DMMs) 2 (36. into StyleGAN- synthesis models, allowing for more precise 3D control over the generated images, including facial expressions and head poses 50) 37 Despite their impressive image quality, these models still tend produce 3D inconsistent faces under large expression and pose variations due to the lack of 3D representation, as shown Fig.
Equua Contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

expression pose
pose (b) DiscoFaceGAN (2D+5DMM)
expression
expression
pose
mesh
(a) E4E (2D+Atrribus))
(c) Our Method (3D+3DMM)
Figure 1: Existing controllable face synthesis methods, such as (a) E4E [51] and (b) DiscoFace-
GAN 77. fail preserve consistent geometry in the face images generated with large expression and pose variations, due to the lack of a 3D representation. (c) Our proposed controllable 3D face synthesis method leverages NeRF-based 3D generative model conditioned prior 3DMM, which enables precise, disentangled 3D control over the generated face images.
With the advances differentiable rendering and neural 3D representations, a recent line of work has explored reato-realistic 3D face generation using only 2D image collections as training data [4 4 43 13 35 3 models, such as pi-GAN 44] and StyleNeRF [3], to synthesize high-fidelity 3D consistent faces, by training with 2D images only. However, these models are purely generative and do not support precise 3D control over the generated faces, such as facial expressions.
55 8). Neural Radiance Fields (NeRFs) [29]. in particular, have enabled 3D generative
In this work, we aim connect these two groups of research controllable face synthesis and 3D generative models- and present NeRF-basee based 3D conditional face synthesis model that can generate high-fidelity face images with precise 3D control over the 3D shape, expression, and pose of generated faces, by leveraging a parametric 3DMM face model. Imposing precise 3D conditions on an implicit neural radiance field non-trivial. A naive baseline solution to enforce that the input 3DMM parameters can be reproduced from the generated face image via 3DMM reconstruction model, similar to 77. Unfortunately, this 3DMM parameter reconstruction loss only provides indirect supervisory signals to the underlying NeRF volume, and is insufficient for achieving precise 3D control, as shown in Fig.
We seek to impose explicit 3D condition directly on the NeRF volume. To this end, we propose a conditional Generative Occupancy Field (cGOF). It consists of mesh gesshe volume sampling procedure and complimentary density regularizer that gradually concentrates the volume density around a given mesh. order to achieve fine-grained 3D control, such as expression, we further introduce a 3D landmark loss and volume warping loss.
The main contributions are as follows. We propose novel controllable 3D face synthesis method that learns to generate high-fidelity face images with precise controllability over the 3D shape, expression, and poses, given only collection of 2D images as training data. This is achieved by a novel conditional Generative Occupancy Field representation and set of volumetric losses that effectively condition the generated NeRF volumes on a parametric 3D face model.
2 Related Work
Controllable Generative Adversarial Networks. Generative Adversarial Nets (GANs) 12 gained
popularity over the last decade due to their remarkable image generation ability [4) 17
19
20
Prior
works have studied disentangled representation learning in generative models [4
50
21 44
30 5 6 23). Existing work on controllable facial image synthesis [38 50 49
37 relies 2D
image- based generative models, which does not guarantce 3D consistency in the generated images.
3D-Aware GAN. Another line of work in 3D-Aware generative models has looked into disentangling 3D geometric information from 2D images, and learning to generate various 3D structures, such
as voxels [34]]]]] 58, [1
30 31 26), meshes [47 22), and Neural Radiance Fields (NeRFs) [4] [56] 33 by training on image collections using differentiable rendering. The key idea
13
42
to use discriminator that encourages images rendered from random viewpoints sampled from a prior pose distribution to indistinguishable from real images. Among these works, pi-GAN
has demonstrated high- high-fidelity 3D synthesis results adopting NeRF representation. SofGAN 5

---

Mapping Network
cGOF Generator
D
L gan
R
Lecon
RGBa
Genersted Image
5
Radiance Integration
Fosd
R smooth
Sampler
Rd
Generoted
Landmark Loss
Volume Warping Loss
warp
Lidnk
Dirarp
Liyarp
5DMM landmarial
NoRF Landmarks
Disolrrment Far
z' NeRF
Warped from a' NefF
2 NeRR
Figure 2: Method overview. (Top): Conditional Generative Occupancy Field (cGOF) leverages mesh-guided volume sampler which effectively conditions the generated NeRF on an input 3DMM mesh. It is trained in an adversarial learning framework using only single-view images. (Bottom): The 3D landmark loss encourages the semantically important facial landmarks to follow the input mesh, and the volume warping loss enforces two NeRF volumes generated with different expression codes to be consistent through a warping field induced from the corresponding 3DMM meshes.
ensures viewpoint consistency via the proposed semantic occupancy field but requires 3D meshes with semantic annotations for pre-training. In this work, we build on top of pi-GAN and directly learn from unlabeled image collections with a prior 3DMM.
3D Priors for Face Generation. To learn controllable 3D face synthesis models, researchers have leveraged prior face models. 3D Morphable Models (3DMMs) [36] 2 a widely-used parametric face model, which represents the human face as set of PCA bases derived from 3D scans. Many previous works models to enable controllable 3D face synthesis. In particular, DiscoFaceGAN 77 PIE [49] and StyleRig [50] leverage 3DMM establish losses ensure 3D consistency, but still use 2D image- based StyleGAN generator. HeadNeRF [16] uses a 3D NeRF representation well as 3DMM but trained with reconstruction loss using annotated multi-view datasets. Our method trained on unannotated single-view images only, by directly imposing 3DMM conditions into 3D NeRF volume.
50
37 have proposed to incorporate 3D priors of 3DMM into generative
3 Method
Given a collection of single-view real-world face images V, where each face is assumed to appear only once without multiple views, the goal of this work is learn controllable 3D face synthesis model that can generate face images with desired 3D shape, expression, and pose. To achieve this, we propose a NeRF-based conditional 3D generative model G and incorporate priors from parametric 3D Morphable Model (3DMM) [36]. At the core of our method is a novel conditional Generative Occupancy Field (cGOF) as well as two auxiliary volumetric losses that enable effective 3D control based on the 3DMM, as illustrated in Figure
3.1 Preliminaries on NeRF and pi-GAN
Neural Radiance Field (NeRF). NeRF 29 represents 3D scene as radiance field parametrized by a Multi-Layer Perceptron (MLP) that predicts density value E R and radiance color e E R for every 3D location, given its zyz coordinates and the viewing direction as input. To render a 2D image of the scene from an arbitrary camera viewpoint ç. we cast rays through the pixels into the 3D volume, and evaluate the density values and radiance colors at a number of sampled points along each ray. Specifically, for each pixel, the camera ray r can be expressed as r(() the camera origin, the direction, and defines sample point within near and far bounds l,, and 4, For each 3D sample point x,, query the NeRF network to obtain its o, and radiance color c;;
+ Id, where o denotes

---

(,), = fxx d). Using volume rendering 27 29, the final color of the pixel is given by
N C(r) = T(1 - exp (xp(=o;)))c; where
i-1
((r)=-T(()
T; = exp(
Eo0dg).
(I)
and 6; 4++1 4 is the distance between adjacent samples. To optimize NeRF 3D scene, the photometric loss is computed between the rendered pixels and the ground-truth images captured from dense set of viewpoints.
Generative Radiance Field. Several follow-up works have extended this representation to gen- erative framework 4 13 & [55]. Instead of using multi-view reconstruction loss, they train discriminator that encourages images rendered from randomly sampled viewpoints to be indistin- guishable from real images, allowing the model to be trained with only single-view image collections.
Our method built on top of pi- pi-GAN 4.. generative radiance field model based on SIREN- modulated NeRF representation 45 During training, images are rendered from random viewpoints Ç sampled from a predefined pose distribution Pg (estimated from the training dataset). To model geometry and appearance variations, a random noise code z is sampled from a standard normal distribution and mapped to number frequencies 7/ and phase shifts 3. through a mapping network. These encodings are used to modulate the outputs of each layer in the NeRF MLP f before the sinusoidal activations. The rendered images lg = G(z, {) and the real images randomly sampled from the training dataset are then passed into the discriminator D, and the model is trained using non-saturating GAN loss with RI regularization following (28]:
(()) = Ex-peiEnpe (f(D(G(2, {)))] + Er-en Lf( + (IV
where (u) = - -lg((1 + exp(-u)) and À a hyperparameter balancing the regularization term.
3.2 Controllable 3D Face Synthesis
In order to learn a controllable 3D face synthesis model, we leverage priors of 3DMM 36 which is a parametric morphable face model, where the shape, expression, texture. and other factors of face are modeled by set PCA bases and coefficients goal is to condition the generated face image / on set of 3DMM parameters ă as well as camera pose E, which specifies the desired configuration of the generated face image.
(Zshape: Zexp: Zex Zelse) respectively. Our
Baseline. To enforce such conditioning, following 77, we make use of an off-the-shelf pre-trained and fixed 3DMM reconstruction model It that predicts a of 3DMM parameters from single 2D image 99. This allows us to incorporate 3DMM parameter reconstruction loss that ensures the generated face images commit to the input condition.
Specifically, we sample a random noise z E Rd from a standard normal distribution Pz. which is later passed into the pi-GAN generator G to produce face image Ig. Unlike (7), which additionally trains a set of VAEs to map the noise z to mcaningful 3DMM coefficients Z, we assume that the 3DMM coefficients of all the training images follow normal distribution and simply standardize them to obtain the corresponding normalized noise code T T()) = Ī-1 (ž fi), where and É are the mean and Cholesky decomposition component of covariance matrix of the 3DMM coefficients estimated from all the training images using the 3DMM reconstruction model R.
During training, we predict 3DMM parameters from the generated image using the reconstruc- tion model R, normalize it and minimize reconstruction loss between the predicted normalized parameters and the input noise:
Cream 2 z||1.
where à = ==
(3)
This results in baseline model, where the conditioning is implicitly imposed through the 3DMM parameter reconstruction loss. Example results of this baseline model can be seen in Fig. 5
Challenges. There are several issues with this baseline model. First, the effectiveness of the condi- tioning depends heavily on the accuracy and generalization performance the 3DMM reconstruction model R. Fundamentally, these parameters only describe an extremely abstract, semantic representa- tion of human faces, which is insufficient to govern the fine-grained geometric details in real faces precisely. Since our model generates 3D representation for rendering, more effective solution is

---

cCOFttustt
Weigts
0.02
0.00
0.30
0.95
1.00
1.05
Ray Distance
weight Wi distribution
(Ne(N Twr(our cGOF)
images/depths rendered from different poses
(a) Weight Distribution Comparison
(h) Shape-Texture Ambiguity w/o cGOF
Figure 3: Challenges in imposing 3D condition on NeRF. (a) Without the proposed cGOF (mesh-
guided sampler and density regularizer), directly supervising the over wee-pparamttriized NeRF volume with a depth loss leads to ambiguous surfaces indicated by high variance of the rendering weights Jmn-- (b) Rendering semi-transparent volumes leads poor realism and multi-view consistency.
to impose the conditioning directly on the 3D volume, instead of just on the rendered 2D images. However, the radiance field representation is highly wver-parametrized [57]. The color of pixel is computed by integrating the radiance along the entire camera ray, whereas in many cases in reality, should only reflect the color of the intersection point on an opaque surface. Incorporating losses directly on this peer-paraettrized 3D volume (e.g. a depth loss) tends to hamper its converging to an opaque surface, resulting semi-transparent volume that no longer guarantees multi-view consistency, as shown in Fig. 3
3.3 Conditional Generative Occupancy Field
In order to impose precise 3D control on the NeRF volume, we first identify effective way of enforcing the volume density to converge to the surface of human faces. Inspired by (55), we propose conditional Generative Occupaney Fields (cGOF), which consists of mesh-guided volume sampler (MgS) and a distance-aware volume density regularizer.
The key idea is to use the conditioning 3DMM mesh to explicitly guide the volume sampling procedure. Given a random noise z, we first obtain the corresponding 3DMM coefficients 2 = r 1 (z) by denormalization, which consists of (Zshape, Texps Ziex Zelse). With these coefficients, we can sample a 3D face mesh Mn from the 3DMM A:
Min = (Zushape Zexp) = Mmean + Bshape Zshape + Beep Zeep.
(4)
where Mmaan is the mean shape, and Bshape and Bexp are the deformation bases for shape and expression in 3DMM respectively. During rendering. this mesh will serve as a 3D condition, allowing us to apply importance sampling around the surface region, and suppress the volume densities far away from the surface.
Specifically, for each pixel within the face region, we find its distance im to the input 3DMM mesh Min along the ray direction by rendering depth map, taking into account the perspective projection. This allows us to sample points along the ray r(4) = o + /d within a set of uniformly spaced intervals around the surface intersection point of the ray and the input mesh:
10-1--1 - 16 + //4
1 ]
t; ~ U
fm
+
where i = 1,2,...,.4.. Nustt
(5)
N
where Nuutt is the number of sampled points, and 8 is parameter controlling the "thickness" of the sampling region. We gradually shrink this margin from 0.5 0.05 of the volume range.
Furthermore, we suppress the volume density in the space far away from the given mesh using a distance-aware volume density regularizer. Specifically, we additionally sample sparse set of points along each ray, and denote the absolute distances from these sampling points to the mesh surface as d 14 mmli l...., Nool The distance-aware volume density regularizer Rg is defined as:
Noa
Ra Σe a. [exp (a max (d. 6/2, - 1],
where o; is the volume density evaluated at those sample points, and a an inverse temperature parameter (set to 20), which controls the strength of the penalty response to increased distance Ad,
5

---

Note that this penalty only applies to sample points beyond the surface sampling range 2. The mesh-guided volume sampler together with the volume density regularizer effectively constrains the generated NeRF volume to follow closely the input 3DMM mesh, as shown in Tab.
3.4 Auxiliary Losses for Fine-grained 3D Control
So far, the model is able to generate 3D faces following the overall shape of the conditioning 3DMM mesh. However, since 3DMM is only a low-dimensional PCA model of human faces without capturing the fine-grained shape and texture details of real faces, enforcing the generated 3D faces to fully commit to the coarse 3DMM meshes actually leads to over-smoothed shapes, and does not guarantee precise control over the geometric details, especially under large facial expression changes. To enable precise geometric control, we further introduce two auxiliary loss terms.
3D Landmark Loss encourages the semantically important facial landmarks to closely follow the input condition. Specifically, we identify set of 3DMM vertices that correspond to N₂g = 68 facial landmarks 41 9
Let {1x )== denote the 3D locations of these landmarks on the input 3DMM the generated image, we extract 3D landmarks [ from the estimated 3DMM mesh and project them onto the 2D image to obtain the 2D projections. We then render the depth values of NeRF at these 2D locations using volume rendering and back-project them to 3D to obtain the 3D landmark locations of the generated volume (1)). To avoid the impact of occlusion, we skip the facial contour landmarks (numbered from to 17) when calculating loss. The landmark loss is defined among these three sets of 3D landmarks:
mesh. In order find the same set of landmarks
Mi.
Lidmk -EIi. 1ell +
1
Adopting this 3D landmark loss further encourages the reconstructed 3D faces to align with the input mesh at these semantically important landmark locations, which is particularly helpful in achieving precise expression control, as validated in Tab. 2
Volume Warping Loss enforces that the NeRF volume generated with a different expression code should be consistent with the original volume warped by warping field induced from the two 3DMM meshes. More concretely, let M A (Z.shape, Zesp) be 3DMM mesh randomly sampled during training, M' (Z Zeep)) be mesh of the same face but with a different expression code z'exp' and V and V' be their vertices respectively. We first compute the vertex displacements denoted by AV =/=B V V. and render them into 2D displacement map FAV E Hx where each pixel Flu..) defines the 3D displacement of its intersection point on the mesh M.
For each pixel (u, v) in the image Ig generated from the first NeRF, we obtain the densities and radiance colors (a,, c,) = f(x,, d, z) of the Naurf mesh-guided sample points x; described above. The superscript (u, v) is dropped for simplicity. We warp all the 3D sample points along each ray using the same 3D displacement value rendered the pixel FAv to obtain the warped 3DD points x X; + FAv (assuming that all the points are already close to the mesh surface). We then query the second NeRF generated with Zeep these warped 3D locations x to obtain another set of densities and radiance colors (o,, c)) = fx(, d', z'), and encourage them stay close the original ones.
In addition, we also enforce the images rendered from the two radiance fields to be consistent. We integrate the radiance colors cţ of the warped points with their densities of using Eq. (L). and obtain an image lg (with all pixels). This image Ig rendered from the second radiance field is encouraged to be identical to the original image Ig. The final warping loss is thus composed of three terms:
Nun
Nurt
Swarp
Ba
110. oilli Bo
Σ |cc
cill + 114g 4gl11.
(8)
where Ba, Ae and 31 are the balancing weights.
3.5 Neural Renderer and Overall Training Objective
Due to the computation- and memory-expensive volume rendering process, the quality of the gen- erated images is bound to a small resolution (64 x 64) in training. To improve the image quality without introducing too much computational overhead, we leverage an auxiliary neural renderer

---

Method Name
Venue
DS, T
DS.
DS, 1
FID (512)L
PIE [49]
TOG'20
1.66
15.24
2.65
59.63
StyleRig 50
CVPR'20
1.64
13.03
2.01
56.59
DiscoFaceGAN 7
CVPR '20
5.97
15.70
5.23
76.13
E4E [5]]
SIGGRAPH'21
1.91
8.66
7.08
96.40
Gan-Control
ICCV'21
7.07
7.51
9.33
83.59
HeadNeRF [6
CVPR'22
6.39
5.99
10.26
135.03
Ours w/o Neural Renderer
23.24
29.13
23.45
71.60
Ours w/ Neural Renderer
21.72
27.47
22.82
31.83
Table 1: DS and FID comparison with state-of-the-ee controllable face synthesis methods.
standalone module, which takes in the rendered coarse 2D image and produces higher resolution version of it (512 x 512). We take their pre- pre-trained model on the FFHQ dataset and finetune it on the CelebA-HQ dataset. This neural renderer significantly improves the quality of the generated images without compromising much 3D controllability, as shown in Tab.
introduced in
Similar to RegNeRF 322, further impose smoothness regularizer Ramooth the generated NeRF volume by penalizing the L2 differences neighboring pixels in both the depth maps and normal maps (referred to as Ramonn and R below). The final training objective is thus given by
Agan + Asecon Creonn + Rg + Ayamk Cidmk AwarpCwarp Asmooth smmooth:
where the X's specify the weights for each term.
4 Experiments
Datasets. We train our model on the CelebA 24
dataset, which consists of 200k celebrity face images with various poses, expressions, and lighting conditions. We use the cropped images and do not make use of their facial attribute annotations. The neural renderer module trained on FFHQ dataset, which uses a slightly different cropping scheme. Therefore, we finetune their pre-trained model the CelebA-HQ subset of CelebA containing 30k high-res images.
originally
Implementation Details. We provide implementation details in the supplementary material, including network architectures and hyper- syper-parameters. Here, we highlight few important ones. We build our model on top the implementation pi-GAN [4), which uses a SIREN- based NeRF representation. We sample Naurt 12 points around the 3DMM input mesh and No 12 coarse points for the density regularizer. The final model is trained for 72 hours on 8 GeForce GTX TITAN GPUs.
Metrics. (I) Disentanglement Score (DS). Following Z we evaluate 3D controllability using 3DMM Disentanglement Score. To measure the DS on one disentangled property z; (e.g, shape, expression or pose), randomly sample set of z; from normal distribution while keeping other codes fixed [.j + i, and render images. We re-estimate the 3DMM parameters 2, and { from these rendered images and compute DS, = II / where 02, and 02, denote variance of the reconstructed parameters of the measured property 2, and the rest 2,, respectively. A higher DS score indicates that the model achieves more precise controllability on that property while keeping the others unchanged. (2) Chamfer Distance (CD). We report the bi-directional Chamfer Distance between the generated face surface and the input 3DMM to measure how closely the generated 3D face follows the condition. (3) Landmark Distance (LD). To evaluate the fine- fineggrined controllability of our model, we detect the 3D landmarks in the generated images using [9] and compute the Euclidean distance between the detected landmarks and the extracted from the input 3DMM vertices. (4) Landmark Correlation (I.C). We measure the precision of expression control, by computing the 3D landmark displacements of two faces generated with pair of random expression codes, and reporting the correlation score between these displacements and those obtained from the two 3DMM meshes. (5) Frechet Inception Distance (FID). The quality of the generated face images is measured using Frechet Inception Distance [15] with an ImageNet-pretraing Inception-Ve [48 feature extractor. We align images with landmarks
46 avoid domain shift.
4.1 Qualitative Evaluation
Fig. shows some example images generated by our proposed cGOF model. particular, we visualize the generated faces by varying only one of the controlled properties at time, including
7

---

(a) Varying Expressions
(b) Corresoondigg Normal Maps for (a)
(c) Varying Poses
(d) Varying Identities
Figure 4: Face images generated by the proposed cGOF. The variations in the expression, pose and identity of the faces are highly disentangled and can be precisely controlled.
facial expression, head pose, and to generate high-fidelity 3D faces with precise 3D controllability in a highly disentangled manner, even in the presence of large expression and pose variations. In contrast, existing methods tend to produce inconsistent face images as shown in Fig. []] Please refer to the supplementary materials for more qualitative results and side-by-side comparisons against tftte-of-the-art methods.
shape and texture as in [7]. Our model is able
4.2 Quantitative Evaluation
We compare our method to several state-of-the-art methods. E4E [51] and GAN-Control 45 are purely 2D controllable face image synthesis methods, which are based on facial attribute annotations. DiscoFaceGAN (7]. PIE 49 and StyleRig [50] leverage 3DMM priors but still use 2D image-basee StyleGAN generator. He [16] uses 3D NeRF representation well 3DMM but is trained with a reconstruction loss using annotated datasets, whereas our method is a 3D generative model trained unannotated single-view images only.
compare the controllability accuracy using the Disentanglement Score (see Col to 5 of Tab. []. Our method achieves significantly better disentanglement and more precise control in terms of shapes,

---

Index
Loss
CD
LD 4
LC1
DS.
DS.
DS, T
FID (128)
Lan
1.09
5.04
2.04
2.13
2.54
7.16
18.76
Lreenn (baseline)
0.87
3.85
26.15
3.56
5.03
11.00
21.97
3
(+ Leeepth)*
1.65
6.16
0.06
0.93
1.34
1.09
71.67
MgS
0.29
3.98
27.45
3.55
4.90
9.48
38.91
5
RR
0.31
3.51
51.74
3.56
5.31
15.94
29.62
+ R
0.27
4.72
30.25
3.18
4.43
16.26
31.63
7
Lidmk
0.39
1.86
84.43
16.54
16.65
21.24
56.90
Lwasp
0.26
1.44
89.91
20.47
22.04
22.91
47.18
Redpph
0.27
1.26
92.88
23.24
29.13
23.45
26.64
Table 2:
Ablation Study. Each row adds additional loss term into the objective function upon all the loss terms above it. The last row is our final model. "Note that we do not use the depth loss the final model, which compares the depths rendered from NeRF and 3DMM.
Condition
Egan
+ereco
(+Edepm)*
+MgS
+
horm RR
smooch
+hikk
+
+R
depth smooth
Figure 5: Ablation Study. We analyze the effects of the individual components and loss functions. With the components gradually enabled, the model is able to generate 3D realistic faces precisely following the condition 3DMM. Refer to Sec. 4.3 for more details. "Note that we do not use the depth loss the final model, which compares the depths rendered from NeRF and 3DMM.
expressions, and poses, compared existing methods. Note that since PIE and StyleRig do not have open-sourced code, we compute the score on the 167 generated images provided on the PIE project page We also compare the FID scores in Col of Tab. I Our model achieves plausible image quality without the Neural Renderer, and outperforms the existing methods with the Neural Renderer.
4.3 Ablation Study
We conduct ablation studies validate the effectiveness of each component in our model by adding them one by one, and evaluating the generated images with the four metrics specified above. Tab. 2 and Fig. illustrate the studies and show that all the components improve the precision of 3D controllable image generation. In particular, the mesh-guided volume sampler (MgS) significantly
improves the effectiveness of the 3DMM condition (denoted as '+ MgS' in Tab. 2 The 3D landmark loss Liamk (denoted as '+ Lidmk') ') and the volume warping loss Lwarp (denoted as '+ Lwarp? ') further enhance the control over fine-grained geometric details. We also report FIDs of the outputs of the cGOF with the image resolution of 128 x 128. Overall, although the geometric regularizers lead to slightly degraded image quality (indicated by increased FIDs), they improve the 3D controllability by large margins (indicated by the improvement in other metrics).
Conclusions
We have presented a controllable 3D face synthesis method that can generate high-fidelity face images based on conditional 3DMM mesh. We propose novel conditional Generative Occupancy Field representation and set of 3D losses that effectively impose 3D conditions directly on the generated NeRF volume, which enables much more precise 3D controllability over 3D shape, expression, and pose of the generated faces than state-of-the-arr 2D counterparts. Limitations. The proposed method still lacks precise texture and illumination control over the generated face images. It also relies heavily on the prior 3DMM and does not generate high-fidelity shape details, such as wrinkles and hair. Societal Impacts. The proposed work might be used to generate fake facial images for imposture or
atttps://vcal mpi-inff inf mpg petpeeecss/////E

---

to create synthesized facial images to hide the true identities of cybercriminals. Defensive algorithms for recognizing such synthesized facial images should be accordingly developed.
Acknowledgements
This work is supported in part by Centre for Perceptual and Interactive Intelligence Limited, in part by the General Research Fund through the Research Grants Council of Hong Kong under Grants (Nos. 14204021, 14207319).
We would like thank Yu Deng for providing the evaluation code of the Disentanglement Score, and Jianzhu Guo for providing the pre-trained face reconstruction model. We are also indebted to thank Xingang Pan, Han Zhou, Kwan Yee Lin, Jingtan Piao, and Hang Zhou for their insightful discussions.
10

---

References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embedded images? In
CVPR. pages 8296-8305, 2020.
[2] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH, pages
187 197-19411 1999.
[3] Fric R. Chan, Connor 7. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient
geometry-aware 3D generative adversarial networks. In CVPR, 2022.
[4] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, pages 5799-5809, 2021.
[5] Anpei Chen, Ruiyang Liu, Ling Xie, Zhang Chen, Hao Su, and Jingyi Yu. Sofgan: portrait image generator with dynamic styling. TOG, 2021.
[6] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Neur/PS, pages 2180-2188. 2016.
[7] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrastive learning. In CVPR, pages 5154-5163 2020.
[8] Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. CVPR, 2022.
191 Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face re- construction with weakly-supervised learning: From single image image set. In CVPR Workshop,
2019.
[10] Xing Di and Vishal Patel. Face synthesis from visual attributes via sketch using conditional vaes and gans. arXiv preprint 1801 00077, 2017.
[11] Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shape induction from 2d views of multiple objects.
In 3DV, 2017.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio, Generative adversarial nets. NewrlPS, 27, 2014,
[13] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d aware generator for high-resolution image synthesis. In ICLR, 2022.
[14] Erik Härkönen, Aaron Hertzmann. Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. In NeurlPS. 2020.
[15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhaard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a nash equilibrium. In NeurlPS, 2017.
[16] Yang Hong. Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang. Headnerf: real-time nerf-based parametric head model. CVPR, 2022.
[17] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. Learning category-specific mesh reconstruction from image collections. In ECCV, 2018.
[18] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lchtinen, and Timo Aila.
Alias-free generative adversarial networks. In NewrlPS, 2021.
[19] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial
networks. In CVPR, pages 4401-4410 2019.
[20] Tero Karras, Samuli Laine, Miia Aittala, Janne Hellsten, Jaakko Lchtinen, and Timo Aila. Analyzing and
improving the image quality of StyleGAN. In CVPR, 2020.
[21] Thomas Leinkühler and George Drettakis. Freestylegan: Free-view editable portrait rendering with the camera manifold. TOG, 40(6), dee 2021.
[22] Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In CVPR, 2020.
[23] Zinan Lin, Kiran Thekumparampil, Giulia Fanti, and Sewoong Oh. Infogan-er and modelcentrality: Self-supervised model training and selection for disentangling gans. In JCML. pages 6127-6139, 2020. [24] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, December 2015.
[25]
Yongyi Lu. Yu- Wing Tai, and Chi-Keung Tang. Auribute- guided face generation using conditional cyclegan.
In ECCV, 2018
[26]
Scbastian Lunz, Yingzhen Li, Andrew Fitzgibbon, and Nate Kushman. Inverse graphics gan: Learning to generate 3d shapes from unstructured 24 data. NewrIPS Workshop, 2020.
[27]
Nelson Max. Optical models for direct volume rendering. TVCG, 1(2):991008, 108, 1995.
[28]
Lars Mescheder, Andreas Geiger, and Sehastian Nowozin. Which training methods for gans do actually converge? In ICMI., 2018.
[29]
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[30] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsuper- vised learning of 3d representations from natural images. In ICCV, 2019.
[31] Thu Nguyen- Phuoce Christian Richardt, Long Mai, Yong-Liang Yang. and Niloy Mitra. Blockgan: Learning 3d object-aware scene representations from unlabelled images. NeurIPS, 2020.
11

---

[32]
Michael Niemeyer. Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 2022.
[33]
Michael Niemeyer and Andreas Geiger. Campari: Camera-aware decomposed generative neural radiance
fields. 3DV, 2021.
[34]
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural
feature fields. In CVPR, pages 1 1453-11464, 2021.
[35] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Seaellmachh--Shizemmaa StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation. In CVPR, pages 13503-13513, June 2022.
[36] Pascal Paysan, Reinhard Knothe, Brian Amberg. Sami Romdhani, and Thomas Vetter. A 3d face model for
pose and illumination invariant face recognition. In AVSS, pages 196--00,220 2009.
[37]
Jingtan Piao, Keqiang Sun, Quan Wang, Kwan- Yee Lin, and Hongsheng Li. Inverting generative adversarial renderer for face reconstruction. In CVPR, pages 15619-15228, 2021.
[38] Shengju Qian, Kwan-Yee Lin, Wayne Wu, Yangxiaokang Liu, Quan Wang, Fumin Shen, Chen Qian, and
Ran He. Make face: Towards arbitrary high fidelity face manipulation. In ICCV, pages 1003--1002,, 2019.
[39] Shengju Qian, Keqiang Sun, Wayne Wu, Chen Qian, and Jiaya Jia. Aggregation via separation: Boosting facial landmark detector with semi-supervised style translation. In ICCV. pages 10153-10161 2019. [40] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
[41] Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafciriou, and Maja Pantic. 300 faces tn-the-wild challenge: The first facial landmark localization challenge. In ICCV Workshop, pages 197-403, 2013. [42] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. NewrlPS, 33:20154- 2011--20666 66, 2020.
[43]
Yichun Shi. Divyansh Aggarwal, and Anil K. Jain. Lifting 2d stylegan for 3d-aware face generation. In CVPR, 2021.
[44] Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, and Gerard Medioni. Gan-control: Explicitly controllable
gans. In ICCV. pages 14083-14093, 2021.
[45] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NeurlPS, 2020.
[46] Keqiang Sun, Wayne Wu, Tinghao Liu, Shuo Yang. Quan Wang. Qiang Zhou, Zuochang Ye, and Chen Qian. Fab: A robust facial landmark detection framework for motion-blurred videos. In ICCV, pages 5462-5471, 2019.
[47] Attila Szabó, Givi Meishvili, and Paolo Favaro. Unsupervised generative 3d shape learning from natural images. arXiv preprint arXiv: 2019.
[48] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the
inception architecture for computer vision. In CVPR, June 2016.
[49] Ayush Tewari, Mohamed Elgharib, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt. Pie: Portrait image embedding for semantic control. TOG, 1-1) 2020. [50] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj. Florian Bernard, Sans-PPter Seidel, Patrick Pérez, Michael Zollhofer. and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait images.
In CVPR, pages 6142 6151. 2020.
[51] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. SIGGRAPH. 2021.
[52] Xintao Wang. Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with gencrative facial prior. In CVPR, 2021.
[53] Yaohui Wang. Antitza Dantcheva, and Francois Bremond. From attribute-labels to faces: face generation using a conditional generative adversarial network. In ECCV Workshop, pages 0-0, 2018.
[54] Jiajun Wu, Chengkai Zhang. Tianfan Xue, William T. Freeman, and Joshua B. Tenenbaum. Learning a probabilistic latent space of object shapes via 3D gemerative-adversaria modeling. In NeurlPS, 2016.
[55] Xudong Xu, Xingang Pan, Dahua Lin, and Bo Dai. Generative occupancy fields for 3d surface-aware
image synthesis. NewrlPS. 34, 2021.
[56] Yinghao Xu, Sida Peng, Ceyuan Yang. Yujun Shen, and Bolei Zhou. 3d-aware image synthesis via learning structural and textural representations. In CVPR, 2022.
[57] Kai Zhang, Gernot Riegler. Noah Snavely, and Vladlen Koltun. Nerf++; Analyzing and improving neural radiance fields. arXiv:2010.072 07492, 2020.
[58] Jun Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, and William T. Freeman. Visual object networks: Image generation with disentangled 3D representations. In NewrlPS, 2018
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] You strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Sec.
6
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec. 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sec.
(b) Did you include complete proofs of all theoretical results? [Yes] See Sec.
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] We release the code and instructions at https://kciiangsum http::m.iiiuuu..ooooocccteeeee/e
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. and the supplementary materials.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] See Sec. 4
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] The license of the assets will be released together with our code.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] We are using only public, standard benchmark datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We did not use crowdsourcing or conduct research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "L4et1Oq3FNj", "reviewer": "Reviewer_PM1k", "summary": "The paper attacks the standard but hard problem of starting with a single-view 2D face shot and creating a 3D face model that can take expression and pose as inputs. Their approach is to meld pi-GAN with 3D Morphable Modeling (3DMM) of priors (based on PCA basis faces) and Generative Occupancy Field (GOF).  I believe that they are also proposing the 3D Landmark Loss for this application, and a Volume Warping Loss which is a nice conservation type prior.  ", "questions": "-  \"we assume that the 3DMM coefficients of all the training images follow a normal distribution\".  Surprised that you don't need a conditional covariance matrix for that, does it just not matter much?\n\n- I may have missed it, but I didn't see where or how you chose the different values for lambda's in (9)? \n\n- I did not understand this sentence \"This allows us to incorporate a 3DMM parameter reconstruction loss that ensures the\ngenerated face images to commit to the input condition.\"", "limitations": "Authors note the obvious problem in this field of creating fakes. In fact, it is hard to conceive of when you would want to do this other than trying to fake people out, but while I think this is not a really important problem to work on, I recognize that in general 3D from 2D is a popular set of problems and do not think the paper should be rejected for its choice of problem. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The proposals seem totally reasonable: they bring together solid components and good ideas. \n- The comparisons to state-of-the-art seem fine, and I appreciated the additional results in the Supplemental\n- They show large gains to controllability\n- The ablation study was useful to see \n- I found the paper easy to read and mostly well-written\n\nWeaknesses: \n- This is pretty close to HeadNeRF but with the extreme case of just 1 input view, I would have appreciated more discussion of that\n- While it's nice their loss function in (9) balances many goals, they end up with 6 (5 free) hyperparameters lambda to set\n- Metrics: it would be nice to see a really proper human rater paired comparisons survey done (but yes I see the qualitative results in the Supplemental)\n- They limit the generated images to 64 x 64 during training (and then use another Neural Renderer to map it to 512 x 512... this seemed a bit iffy), and then Table 1 suggests their method without the extra Neural Renderer step). I am not sure I grokked the full ramifications and comparison issues due to this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2aPx5VYnvHo", "reviewer": "Reviewer_jXJ9", "summary": "This paper proposed a new NeRF-based conditional 3D face synthesis framework, which enables the control over head poses and expressions using a parametric 3D face model (3DMM). \n\nThe main contribution of this model is a conditional Generative Occupancy Field (cGOF) that effectively enforces the shape of the generated face to commit to the 3DMM model mesh. Besides that, novel loss terms such as 3D landmark loss and volume warping loss are proposed in the system. \n\nThe authors trained the model on the CelebA dataset and conducted both quantitative and qualitative comparisons to other methods. The results and comparisons showed that the proposed method outperforms the state-of-the-art methods. Meanwhile, the ablation study was also done to show the effectiveness of each proposed component.\n", "questions": "1. For the 3DMM conditioning, the authors mentioned that they assume the 3DMM coefficients of all the training images follow a normal distribution, while it's not very clear what are the representations of the expression components. Is it also a PCA model or FACs based model? The authors cited [37], which didn't mention about expression components. \n\n2. For the 3D landmark loss, the authors mentioned that they didn't consider the facial contour landmarks due to the impact of occlusion. However, in extreme head poses such as 90 degree left/right view, part of the inner facial landmarks are not visible as well. It's not clear how the authors handle these landmarks and be able to generate plausible results for side view head poses. \n\n3. It would also be great if the authors could include the inference time of the proposed framework. ", "limitations": "The authors addressed the limitations of the work that the proposed method couldn't handle precise texture and lighting control over the generated face. And also the model relies on 3DMM model, which could not generate high-fidelity shape details such as wrinkles and hair.\n\nFor societal impact, the authors mentioned that the work might be used to generate fake facial images. Defensive algorithms on recognizing these images should be developed. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper proposed a new NeRF-based 3D face synthesis framework, which allows controlling the large head pose change and expression changes of frontal face images.\n\n2. The core contribution is the usage of the conditional Generative Occupancy Field (cGOF) model. Other terms such as 3D landmark loss and volume warping loss terms all contributed to the effectiveness of the proposed framework.\n \n3. This paper is well structured and easy to follow. The authors clearly explained each term and did ablation study to show the impact of each proposed component.\n\n4. The authors further did quantitative and qualitative evaluations on the proposed method and some state-of-the-art methods such as PIE, StyleRig, DiscoFaceGAN, E2E, HeadNeRF, etc, showing that the proposed method outperforms. \n\nWeaknesses:\n1. I'm a little concerned that the originality of this proposed work is not enough. For example, the 3DMM conditioning idea is similar to the work in [8] and the cGOF idea is inspired from [53]. It would be great to show that how the way the authors model the 3DMM conditioning in a normal distribution outperforms the VAE model in [8].\n\n2. Some technical details are not very clear to me. Please see the following questions. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"]}, "extra_text": {}}, {"review_id": "-fYcHoW1R4h", "reviewer": "Reviewer_r2K4", "summary": "This paper aims the task of controllable 3D face synthesis based on a set of single-view real-world face images. The authors propose a solution based on the framework of pi-GAN and the widely used parametric 3D Morphable Model (3DMM). The key idea is that the conditional Generative Occupancy Field is employed to explicitly guide the volume sampling procedure of the 3DMM mesh. Meanwhile, for fine-grained 3D shape control, the 3D landmark loss and a volume warping loss is used. In the experiments, both qualitative evaluation and quantitative evaluation show the effectiveness of the proposed method. ", "questions": "1.  In section 3.2, “where the shape, expression, pose and other factors of a face are modeled by a set of PCA bases and coefficients z = (z_shape, z_exp, z_tex, z_else)”, here, z_tex denotes the factor of pose or texture? \n\n2. Please give the comparsion results of the propsoed method with the baseline model mentioned in the beginning of section 3.2. ", "limitations": "1. The novolty of this paper is limited. \n2. The presentation of the whole paper is not smooth and well structured. \n3. The experimental results are not sufficient.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The idea of combing pi-GAN and GOF as well as 3DMM for the task of controllable 3D face synthesis shows a certain novelty. \n\nWeaknesses: The novelty of the proposed method is limited. It can not be regarded as a new framework since all the related methods of pi-GAN, GOF and 3DMM are well-known. Meanwhile, the experimental results are not sufficient. The comparison of the proposed method with the Baseline method is missed in Table 1 and Table 2. Moreover, the presentation of the whole paper is not good. For example, in section 3.2, “where the shape, expression, pose and other factors of a face are modeled by a set of PCA bases and coefficients z = (z_shape, z_exp, z_tex, z_else)”, here, z_tex denotes the factor of pose or texture? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept