AniFaceGAN: Animatable 3D-Aware Face Image Generation for Video Avatars
Authors: Yue Wu
Yu Deng
Jiaolong Yang
Fangyun Wei
Qifeng Chen
Xin Tong
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D-Aware GAN
Face Generation
Face Animation
Generative Models

Abstract: Although 2D generative models have made great progress in face image generation and animation, they often suffer from undesirable artifacts such as 3D inconsistency when rendering images from different camera viewpoints. This prevents them from synthesizing video animations indistinguishable from real ones. Recently, 3D-aware GANs extend 2D GANs for explicit disentanglement of camera pose by leveraging 3D scene representations. These methods can well preserve the 3D consistency of the generated images across different views, yet they cannot achieve fine-grained control over other attributes, among which facial expression control is arguably the most useful and desirable for face animation. In this paper, we propose an animatable 3D-aware GAN for multiview consistent face animation generation. The key idea is to decompose the 3D representation of the 3D-aware GAN into a template field and a deformation field, where the former represents different identities with a canonical expression, and the latter characterizes expression variations of each identity. To achieve meaningful control over facial expressions via deformation, we propose a 3D-level imitative learning scheme between the generator and a parametric 3D face model during adversarial training of the 3D-aware GAN. This helps our method achieve high-quality animatable face image generation with strong visual 3D consistency, even though trained with only unstructured 2D images. Extensive experiments demonstrate our superior performance over prior works. Project page: \url{https://yuewuhkust.github.io/AniFaceGAN/

AniFaceGAN: Animatable 3D-Aware Face Image Generation for Video Avatars
Yue Wu Yu Deng?*
Jiaolong Yang
Fangyun Wei
Qifeng Chen'
Xin Tong'
*HKUST
TTinghua University
*Microsoft Research
Abstract
Although 2D generative models have made great progress in face image generation and animation, they often suffer from undesirable artifacts such as 3D inconsis- tency when rendering images from different camera viewpoints. This prevents them from synthesizing video animations indistinguishable from real ones. Recently, 3D-aware GANs extend 2D GANs for explicit disentanglement of camera pose by leveraging 3D scene representations. These methods can well preserve the 3D consistency of the generated images across different views, yet they cannot achieve fine-grained control over other attributes, among which facial expression control is arguably the most useful and desirable for face animation. In this paper, we propose an animatable 3D-aware GAN for multiview consistent face animation generation. The key idea is to decompose the 3D representation of the 3D-awa 3D-aware GAN into template field and a deformation field, where the former represents different identi- ties with canonical expression, and the latter characterizes expression variations of each identity. To achieve meaningful control over facial expressions via defor- mation, propose 3D-level imitative learning scheme between the generator and a parametric 3D face model during adversarial training the 3Dwww GAN. This helps our method achieve high-quality animatable face image generation with strong visual 3D consistency, even though trained with only unstructured 2D images. Extensive experiments demonstrate superior performance over prior works. Project page: https:/ httt:////wwwuuuutttitietoe Atw.:b/./AAAAF/aAAAA
Introduction
Face image synthesis and animation have been longstanding task computer vision and computer graphics with a wide range of applications such as virtual avatars and video conferencing. Remarkable progress has been achieved in recent years with large volume of methods proposed [50] 2 19 57
52] 69, 58 [56 63) I 67 70 46, (7). This progress is hinged on a number of ad in machine learning within which generative adversarial networks (GANs) are arguably the core underpinning.
However, most existing face GANs are based on 2D convolutional neural networks (CNNs) and do not model the underlying 3D facial geometry. When synthesizing faces under different poses and expressions, the results cannot maintain strict 3D consistency. Consequently, these methods can be used in interactive face manipulation but are not suitable for high-quality face video generation and animation. To alleviate this problem, some methods incorporate 3D prioris into the generation process [TI 64 30 20 51 for better 3D rigging over the 2D images. Perhaps the most relevant work to ours is DiscoFaceGAN П. which considers an unconditional and disentangled generative modeling setup as we do. Still, exact 3D consistency cannot be guaranteed by the aforementioned
WWrk done when YW and YD interns Microsoft Research.
f JY is the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Expression control
Pose control
Figure 1:
Random virtual persons generated by our method under different expressions and head poses. Note the high texture consistency across poses and expressions (see more animated examples our accompanying video).
approaches, and their 2D CNN generators often lead to temporal artifacts such as flickering and texture sticking 225 which are undesirable for realistic video avatars.
Recently, number of 3D-aware GANs are proposed by incorporating 3D representations 37
38
14 22 [2 4I to achieve disentangled control of 3D pose. Among them, methods that generate
neural radiance fields (NeRF) 335 have demonstrated striking image synthesis results [TO] 22
Owning to its 3D field modeling and volumetric rendering scheme, the NeRF representation is capable of producing realistic images with strong 3D consistency across different views, rendering it suitable for high-quality 3D scene synthesis. Nevertheless, these methods lack the control over attributes beyond camera pose and thus cannot be directly applied to face animation synthesis tasks.
In this paper, we present AniFaceGAN, an animatable 3D-aware face generation method that is used synthesize realistic face images with controllable pose and expression sequences. Our method is an unconditional generative model that can generate novel, non-existing identities and is trained on unstructured 2D face image collections without any 3D or multiview data. To achieve animation, we leverage separate latent representations for identity and expression in the generator, and attain explicit controllability by incorporating priors of 3D parametric face model. To ensure geometry and texture consistency under expression change, we leverage 3D deformation to derive the desired expressions. Although deformation has been used in some recent NeRF methods [53]
45), these works mostly focused on modeling single dynamic scenes from videos. How to learn deformation in generative setting from unstructured 2D images and how to achieve explicit and accurate expression control through deformation with unsupervised learning remains underexplored.
Our AniFaceGAN generates two 3D fields for face image rendering: template radiance field for modeling the geometry and appearance of generated identity and an expression-driver deformation field for animation. The former based the recent generative radiance manifold (GRAM) approach that shows tfate-of-the-a 3D-aware image generation quality [12]. To learn desired deformation, we incorporate a face morphable model (3DMM) [5] [77 into adversarial training and enforce our deformation field to imitate it under expression variations. In contrast to previous methods such as || which imposes imitations on 2D images, we propose a set 3D-space losses defined on both facial geometry and expression deformation. We train our method on the FFHQ dataset (26) and show that can generate high-quality and 3D consistent images of virtual subjects across different poses and expressions (an example is shown in Fig. I

---

The contributions of our work can be summarized as follows:
We present an animatable 3D-aware face GAN with expression variation disentangled through our proposed expression driven 3D deformation field.
We introduce a set of novel 3D-space imitative losses, which align our expression generation with a prior 3D parametric face model gain accurate and mmenmiinll control of facial expression.
We show that our method can generate realistic face videos with high visual quality. We believe it opens a new door to face video synthesis and photorealistic virtual avatar creation.
2
Related work
Neural implicit functions have been used in numerous to represent 3D scenes in continuous domain. Among them, shows its superiority at modeling complex scenes with detailed appearance and synthesizing multi view images with striet 3D consistency. The original NeRF and most of its
Neural implicit scene representations.
works
42
34
60
59
35
39]
NeRF
35
successors 43
33
45
49 48
40
32 [73 focus on learning scene-specific representation using a set
of posed images or a video sequence of a static dynamic scene. A few methods [55] 10 38 76
22 12 9
have explored the generative modeling task using unstructured 2D images as training data. A very recent method GRAM [12] constrains radiance field learning on set of 2D manifolds and shows promising results for high-quality and multi-view consistent image generation. Our method is also built upon the radiance manifolds representation 12 animation.
for high-quality face image generation and
3D-aware generative model for image synthesis. Generative adversarial networks 21 26
27 are 22
widely used for realistic 2D image synthesis. Recent methods [36 62 37 31
38
10
14
1
72 extend GANs to 3D-aware image synthesis by incorporating 3D knowledge into their generators.
This enables them to learn multiview image generation given only unconstrained 2D image collections as supervisions. For example, HoloGAN [36) utilizes 3D-CNN to generate low-resolution voxel features and projects them 2D feature maps for further neural rendering. GRAF (55) adopts generative radiance field as scene representation and generates images via volumetric rendering [35] GRAM [12] further sparsifies the radiance field set of radiance manifolds and leverages manifold rendering (78 [2] for more realistic image generation. Although 3D-aware GANs are able to control camera viewpoints, they cannot achieve fine- fine-grained control over the generated instances such as shapes and expressions for human faces. This prevents them from being used for multiview face animation generation tasks. In this work, introduce imitative learning of 3DMM into 3D-aware face image generation.
GAN framework to achieve expression-controllan
Face image synthesis with 3D morphable model guidance. 3D Morphable Models (3DMMs) 5
47
play an important role for face image and animation synthesis 666 works pipeline for face reenactment. However, these methods often produce over-smooth textures and
7
28
75
54].
Earlier
66 directly render textured meshes represented by 3DMM using traditional rendering
cannot generate non-face regions due to the limitation of 3DMM. Later works [28 [8]
15
75
S apply
refinement networks on top of the rendered 3DMM images to generate more realistic texture details and fill the missing regions. Nevertheless, these methods still rely on a graphics rendering procedure to synthesize coarse images inference time, which complicates the whole system. Recently, DiscoFaceGAN [1] proposes an cnitativvecontrasttin learning scheme that enforces the generative network to mimic the rendering process of 3DMM. After training, it can directly generate realistic face images and control multiple face attributes in a disentangled manner via the learned 2D GAN, Some concurrent and following works (64 30 20 51] of share similar spirit. However, these methods often encounter 3D-inconsistency issues when controlling camera poses due to the non- physical image rendering process of 2D GANs. Some recent methods [16] 24 incorporate 3DMM knowledge into NeRF's training process to achieve animatable face image synthesis. However, they do not leverage generative training paradigm [21) and require video sequences or multiview images as supervision. By contrast, our method can be trained with only unstructured 2D images.
Face editing and animation with GANs. A large amount of works adopt GANs for face image manipulation and animation synthesis [50 2
19
57
52
69
58
56
63
70
46 54 71 77].
3

---

Target Space Rathance Manifolds
Deformation Field
Temp Space Radiance Manifolds
Deform
Network
Manifold Rendecing
Real Samples
Manitold
coumiaaiooooonnn
Predictor
Lanna
im
Reconst
D
Real/Pake
Radiance
Generator
3DMM
3DMM
(RGBc)
2nd Fezp
Zasp
Ria Pexp
Network Structure
Training Framework
Figure 2: Our proposed framework which consists of a template radiance field and an expression- driven deformation field for animatable 3D-aware face image generation.
However, these methods do not offer a generative modeling of face identities thus are difficult to generate new virtual subjects. They also suffer from 3D-inconsistency and texture flickering issues when changing camera poses due to the use of 2D CNN as image renderer.
3 Approach
Our AniFaceGAN is a generative model trained on a collection of unstructured monocular face images. The generator produces two 3D fields, namely a template radiance field and an expression- driven 3D deformation field. Figure shows the overall pipeline of our method. The inputs to our generator include identity code Zia E Rd: for shapes, expression code Zeap E RR for facial animations, an additional noise E dR controlling other attributes such as appearance, lighting and background, and camera pose 0 E TR We incorporate the 3I) prior of a 3DMM face model [47 into our generative modeling. With 3DMM, the geometry of 3D face represented by a mesh gaN N vertices is parameterized as
S S=S(3,7) 7) S Big + BeEP%
where Bid and Barp are the PCA basis for identity and expression, 33 are the identity and expression coefficients, and the averaged face shape. Our latent spaces of identity and expression are aligned with the 3DMM model to facilitate expression control. During GAN training, we sample z,d and Zezp from prior distributions estimated from real images, which is similar [11]
3D-space imitation learning (to be described later) and
3.1 Template radiance field
The template radiance field determines the geometry and appearance of the generated identity, and defined canonical space with a neutral expression. It is modeled by template radiance field generator G, which takes a 3D point æ, identity code Zid, noise code e, and view direction d as input and output the color e and occupancy of w::
G (x, zid, d) R 46 (c, ai E E..
(2)
In this work, we use the generative radiance manifolds (GRAM) 112 approach generate our template radiance field. GRAM is state-of- ofate-of-the-an art 3D-aware GAN that can generate high-quality images with strong multiview consistency by regularizing radiance learning and sampling on a set of learned surface manifolds. In theory, our template field can be generated by any 3D-aware generative models. We modify the generator of GRAM by changing input to ours and train from scratch within our whole framework.

---

3.2 Expression-driven 3D deformation field
The 3D deformation field is used to generate different expressions by deforming the 3D neural face in the template radiance field. It is represented by a deformation network F.
In practice, F models "inverse" deformation, which deforms 3D point in the target space to the template space. The input to F includes a target-space 3D point æ: and expression code Zexp. and the output is displacement vector representing the 3D offset from the target space template space. Since identity and expression are correlated in 3DMM (see their linear combination in Eq. [T. we also add Zid to the input of F. In summary, F' can be written as:
F (2,, Zid, Zemp) E 1R.+d..5
A
(3)
3.3 Image rendering
Our rendering follows a volume rendering paradigm tailored to the radiance manifold rendering of [12]. For each viewing ray r in the target space, we uniformly sample N points {æ,} along the ray and deform them to the template space by F' as {œ{} where x
+ Aæ, Then we calculate the .M intersection points between the (deformed) ray and template radiance manifolds of GRAM, denoted as { A differentiable intersection calculation method is used in [[2]. Finally,
obtain the color and occupancy of {2}} with G, and composite the output color via the following
rendering equation 40
78 12:
((r) Σ (()))())))())))
( ((())(()))(())))( f=1
(4)
Note that the above process is equivalent to first deforming the radiance manifolds in the template space the target space by "forward" deformation, and then conducting intersection calculation and manifold rendering in the target space. Here we use the inverse deformation defined by F (Eq. in that the radiance manifolds defined on set of implicit surfaces.
3.4 3D-space imitation learning
To ensure that the face shape and expression a generated subject follow those described by 3DMM given input identity and expression codes, employ set of 3D-space losses to enforce the geometry and expression deformation obtained by our generator to imitate the 3DMM model.
Dense geometry imitation. First, we enforce the similarity between the underlying 3D geometry of our generated instance and its corresponding 3DMM face. To achieve this, we first extract depth map of the generated instance in the target space. Specifically, for each viewing ray with sample points {æ:)). its accumulated depth value can be computed via
2(r) Σ T(x))2(x:)
where T'-) is defined in Eq. 4 We use intersections between r and the manifolds the target space as the sample points to calculate the depth, shown in Fig. 2 Then we reproject each pixel to 3D space according to its depth, generating a point cloud S'. We compare S' with a 3D face S(zia. Zexp) obtained by 3DMM via Eq. [ Since 3DMM only represents small facial region, employ directed Chamfer distance 61 to align the 3DMM shape to S²:
(sn() = S
minyes ||x - y112-
XES
3D landmark imitation. We then incorporate a 3D landmark loss function to enforce the deformation field to generate desired expressions. Let / be an image generated by our model with identity code zıd and expression code Zeap- We use a face reconstruction network [13] to extract the 3DMM identity and expression coefficients on denoted as End and Eexp- A 3DMM shape can be reconstructed using 2nd and Sexg- The 3D landmark loss is then defined as
Lim
[| (S(zid Zeap)), )), Jim (Sid 3)xp)) 112
5

---

where fim represents a function of extracting the landmarks. Here we use the landmark points as in [3] except for those on face contour. Note that we define landmark loss 3D space instead of projecting them on 2D image plane for loss calculation as done in [1 because 2D landmarks have larger ambiguity between different expressions.
Deformation imitation. We also encourage the deformation field to follow 3DMM deformation. Specifically, for each 3D point æ in the target space, we first find its nearest point æref on the corresponding 3DMM mesh and compute its deformation to the neutral face according to the 3DMM model defined in Eq. i.e..
A - (S(zad, & zeep) (2ref) - S(zid, ()(ree)) = -Beup (eppee))(Ere)))
(8)
and then employ loss function LADMM:
LDDMM = |Ax ATres 11
(9)
where Ar is the deformation of a obtained via Eq.
Deformation regularizations. We expect the deformation of most points in the target space to be small except for regions influenced by expression change. To this end, we employ a minimal deformation constraint via
Creg
A
(10)
Moreover, to encourage the deformation fields to be smooth and avoid abrupt deformation changes, we impose a simple smoothness constrain on the deformation field:
smooth =
| A(x + =)1
(11)
where & is a small random perturbation.
Adversarial learning. Following [12), we randomly sample real images from the training set, and apply a discriminator to differentiate synthesized images from real images via the same adversarial loss used in [12].
4
Experiments
We train our method on the FFHQ [26] datasef Aataser"[hhich which contains 70K face images. Following [.].
randomly sample latent codes zid. Zezp: and camera pose 0 from estimated prior distributions, and sample e from a normal distribution. In our experiments, the Adam optimizer (29] with 31 = 0 and B2 0.9 is applied for training model. We set the learning rate 2e 5 for the deformation network and the generative radiance manifolds, and 2e for the discriminator. We train our models
on Nvidia Tesla V100 GPUs with a batch size of 32 at the resolution of 128 128. Our model takes approximately three days for training. See the supplement for more details.
4.1 Evaluation on generation quality
To evaluate the image generation quality of our method, compare with two previous face generative models, CONFIG [30] and DiscoFaceGAN [II), which also achieve disentangled control over camera views and expressions of their generated virtual subjects. We further compare with baseline method we call DiscoFaceGRAM,, which naively combines GRAM [12 with DiscoFaceGANN' imitative-contrasttie learning scheme. Specifically. in (II), the GAN model is black box to its imitative-contrastive learning scheme and thus can be replaced by any other GAN model. Thus in DiscoFaceGRAM,, simply substituted the original StyleGAN in [1]] with GRAM used in AniFaceGAN while keeping other parts unchanged, and trained this so-called DiscoFaceGR. AM using the framework of (1T). For DiscoFaceGRAM,
discard the deformation field and directly send Σ,d, Zeep to the template field represented by GRAM. Since we do not control textures and illuminations as in (II). we only apply the imitative loss for 2D landmarks and the contrastive loss for expressions to train this baseline method. The purpose of developing such a baseline is to validate the contribution of the 3D deformation and 3D-level imitation scheme we newly designed in this work. For CONFIG and DiscoFaceGAN, we use the pretrained models released by their authors.
'FFHQ is released under the Creative Commons BY-NC-SA 4,0 license: the buman face images therein were published on Flickr by their authors under licenses that all allow free use for commemmieciall purposes.

---

CONFIG
DiscoFacoGAN
DisMoFaceGRAM
Ours
Figure 3: Visual comparison between our method and others. The quality of the images synthesized by CONFIG and DiscoFaceGRAMr AM is clearly lower than ours. Both DiscoFaceGAN and our method can generate photorealistic images. However, DiscoFaceGAN suffers from inconsistency for different expressions and poses. (See the accompanying video for better visualization and comparison.)
Table Quantitative evaluation of different models on the FFHQ dataset. FID and KID (x 100) are computed with 5K synthesized images and 5K real images. DiscoFaceGRRAM is a baseline method implemented by us (see text for details).
GRAM 12
CONFIG 30 DiscoFaceGAN
DiscoFaceGRAM
Ours
FID KID
19.4
52.6
17.9
23.9
19,9
0.64
3.38
0.79
1.19
0.86
We first show the visual comparison in Fig. 3 CONFIG and DiscoFaceGAN suffer from 3D- inconsistency issues (e.g., see bangs of the hairs) when varying the camera pose as these methods are based on black-box CNN renderers. In DiscoFaceGRAM, the generation process handles expression change without using explicit deformation, and all the losses are imposed on the generated 2D images. We conjecture that such strong 2D-level losses may introduce training, which might have led to its lower image quality. In addition, modeling expressions without deformation also leads to geometry and texture inconsistency (c.g., see the variation of hair bangs and eyeglasses under expression change). By contrast, our method shows strong consistency between generated images of different poses and expressions, thanks to the manifold rendering of our inherent 3D representation and the expression-driven 3D deformation. A more detailed visual comparison terms of consistency is shown Fig. 4 (see Sec. (4.3). DiscoFaceGRAM cannot guarantee full disentanglement between identity and expression (e.g.. see the hair, color of face and eyeglasses while changing expressions) using only image-space imitative and contrastive constraints. Instead, our method physically disentangles the two factors and demonstrates better generation quality and more reasonable expression control.
hurdles for 31D-aware GAN
For quantitative evaluation, we compute the Fréchet inception distance (FID) (23 and Kernel Inception Distances (KID) 4 between 5K randomly synthesized images and 5K randomly sampled real images [12]. As shown in Table 1 our result is slightly worse than GRAM, which is expected since vanilla adversarial training focuses only on image quality while we introduce controllability with additional training losses. The quality of our method is also moderately lower than DiscoFaceGAN, which is also reasonable since our backbone GRAM, as a 3D-aware GAN, still has a quality gap to traditional 2D GANs based on the powerful StyleGAN architecture. Compared with GRAM, our method introduces strong controllability over expression with only slight decrease of image quality. DiscoFaceGAN shows better image quality in terms of FIDD and KID, but sacrifices 3D consistency which is critical for realistic video generation. The large FID&KID gaps between DiscofaceGRAM and our method further validate the effectiveness of our model and loss design.
4.2 Evaluation on factor disentanglement
To further evaluate the disentangled controllability of AniFaceGAN over expression and pose, we calculate the disentanglement score (DS) [I]] of generated images. DS measures that, when
7

---

Ours
DiscoFaceGRAM
DiscoFaceGAN
CONFIG
Figure 4: Comparison of consistency using a visualization method similar to 74 For the first row, we fix the camera location and smoothly change the expression. For the second row, use the same expression and rotate the camera smoothly. We check the texture of a line segment at fixed 2D position. Twisted lines in the first group indicate unwanted texture flickering, and the distortion/noise pattern in the second group indicates 3D inconsistency of the rendered multiview images.
Table Evaluation of factor disentanglement on expression and pose using disentanglement score (DS) [1] and 3D consistency using the multiview reconstruction method of NeuS [68]. The disen- tanglement scores of GRAM Face-vicevdd [7]] is not included as [71] unlike other methods, does not use 3DMM coefficients for expression control.
Disentanglement
3D Consistency
Method
DS
DS DSpose t
PSNRT
SSIM
DiscoFaceGAN
23.84
4.43
33.3
0.925
GRAM PiRenderr
54
13.50
6.06
38.1
0.971
GRAM 12 Facc-vii22vid
38.3
0.970
Ours
24.43
7.29
41.1
0.984
only vary one single factor, if other factors of the generated images are stable. We calculate DS for expression and pose following the same experiment setting as [11] and compare the result with (II) We also compare with another method call GRAM--PRenddeee. which leverages state-of-the-ar face editing method PIRenderer [54] to modify the generated frontal images of GRAM [1] for expression and pose control. As shown in Table PIRenderer [54] cannot well disentangle identity and expression as indicated by the low DS even though it is trained on video data. DiscoFaceGANN achieves better expression disentanglement thanks its contrastive learning scheme, yet it still suffers from detail inconsistency as shown in Fig. 3 Jand Our method utilizes 3D deformation for animation and achieves the best disentanglement of expression. also obtains the highest DS for pose, which validates the advantage of using 3D representation for 3D-consistent image generation.

---

Wo F
With Mim
Ours
Figure 5: Visual comparison on expression control. The first row shows the real images that provide reference expressions, and other are the results from different methods (see text for details).
Table 3: Ablation study on deformation.
Table 4: Ablation study on 3D-space imitation losses.
Method w/o F
Ours
Method wo Cim. w/o Ccs.
w/o LadMM
with 2im
Ours
FID4 28.3
19.9
FIDI
22.4
21.8
19.8
22.1
19.9
4.3
Evaluation consistency
Since DS can only evaluate the factor disentanglement on a semantic level, we conduct further experiments validate the consistency of method under expression and pose change on more detailed level. Following the idea of (74,, we first present the spatiotemporal textures of different methods by smoothly varying expression or camera pose and stacking the texture
fixed horizontal line segment (Fig. 4.. For disentangled expression control, regions not affected by the expression should remain unchanged, leading to a texture image with vertical strips. For 3D-consistent pose variation, the resultant texture image should be similar to the Epipolar Line Images (EP1) [6], which has smoothly tilted strips. As shown Fig. our AniFaceGAN produces desirable texture patterns under expression and pose changes, indicating its strong consistency. DiscoFaceGRAN and DiscoFaceGAN produce vor-vcrtical strips when changing expression, indicating inconsistency of detailed appearance. Under pose change, DiscoFaceGAN and CONFIG lead to distorted texture images which indicate the 3D-inconsistency issue.
We further evaluate the 3D consistency of our method quantitatively and compare with state-of-the-art 2D face editing methods PIRenderer (54] and Face-vid2vid (7l). Following (74), we randomly generate 50 identities with GRAM, and then generate 30 multiview images under fixed set of viewpoints for each identity via different methods. We then train the state-of-the-arr multiview reconstruction method NeuS (68] on them and compare the average PSNR and SSIM scores of the images reconstructed by NeuS. theory. the more consistent the input multiview images are, the higher the reconstruction quality will be. As shown in Table our method has the best performance terms of PSNR and SSIM, indicating its superiority over other alternatives at preserving 3D consistency. Also note that our method is generative modeling method which can synthesize virtual identities, while PIRenderer and Fide-vid2vid cannot generate non-existing subjects solely. The visual results can be found in the supplement.

---

4.4 Ablation study
We conduct ablation studies to evaluate the efficacy of each component of our framework. The results are shown in Table and Fig. [ We train each variant five times using different random seeds method with baseline method without using the deformation field but directly sending Zid and Zezp into the template GRAM (w/ F). As shown, the image generation quality without the deformation field degrades by large margin. The corresponding visual results in Fig. J depict that removing the deformation leads to a loss of disentanglement between identity and expression.
and present the average score. In Table [] we compare
In Table + we validate the efficacy of our proposed 3D-level imitations by removing each loss component during training (w/o L.). We also conduct experiment which replaces the 3D landmark loss with its 2D counterpart (with ( ) by calculating landmark loss on the projected image plane as in [T]. As shown, removing L3DMM yields similar FID score, but empirically find our full model has slightly better visual quality. Removing other loss components all lead to degradation of generated image quality. In addition, replacing our 3D space landmark loss with 2D landmark loss also degrades the image quality and introduces obvious artifacts for certain expressions as shown in Fig. 5 We conjecture this is due to the misalignment between the 2D position distributions of the projected 3DMMs and the faces in real images, which causes conflicts between the 2D landmark imitative loss and the adversarial loss. In contrast, our full model yields high-quality image generation results with accurate control over expression.
4.5 Animatable face video generation
Although our method is trained on unstructured 2D images, it can be used to generate realistic video animations of virtual subjects. We achieve this by generating image sequences with our pre-trained AniFaceGAN using continuous expression codes and camera poses (either handcrafted ones or those extracted from real videos). Thanks to our proposed 3D representation of combined template field and expression-driven deformation field as well as the carefully designed imitative losses, we are able to generate high-quality face videos with strong geometry and appearance consistency under continuous expression and pose changes, which cannot be achieved by previous methods. More details and visual results can be found in the supplement and accompanying video.
5 Conclusions
We have presented animatable 3D-aware face image generation method named AniFaceGAN, which aims to synthesize realistic face images with controllable poses and expressions. To maintain 3D consistency across different expressions, we leverage 3D deformation to derive the desired expressions. For face image rendering, we propose to generate two 3D fields, i.e., a template radiance field and an expression-driven deformation field. A set of novel 3D-space imitation losses proposed to effectively train our model along with adversarial learning. The proposed AniFaceGAN is an unconditional generation model that is trained unstructured 2D face images without any dependency on 3D or multiview data. Experimentally, our method trained on FFHQ dataset can produce high-quality animatable faces with strong visual 3D consistency across different poses and expressions. We hope our method could serve as a strong baseline for realistic face video generation and animation.
6 Broader Impact
This work aims to design an animatable 3D-aware face image generation method for the application of photorealistic virtual avatars. It not intended create content that used mislead or deceive. However, like other related face image generation techniques, it could still potentially be misused for impersonating humans. We condemn any behavior to create misleading or harmful content of a real person. Currently, the images generated by this method contain visual artifacts that can be easily identified. The method is data-driven, and the performance is affected by the biases in the data. So one should be careful about the data collection process when using it.
10

---

References
[1] Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka. Styleflow: Attribate-conditionedd exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics, 40(3):1-21, 2021.
[2] Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. CVAE-GAN: Fine-grained image generation through asymmetric training. In /EEE/CVF International Conference Computer Vision,
pages 7754-4754, 2017.
[3] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti- tiitallaiinn neural radiance fields. In IEEE/CVF
International Conference on Computer Vision, 2021.
[4] Sai Bi, Kalyan Sunkavalli, Federico Perazzi, Eli Shechtman, Vladimir G Kim, and Ravi Ramamoorthi. Deep eg2real: Synthetic-to-ree re-rell translation via image disentanglement. In International Conference on Computer Vision, pages 2730-2739. 2019.
[5] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3D faces. In Annual Conference Computer Graphics and Interactive Techniques, pages 187-194, 1999.
[6] Robert C Bolles, H Harlyn Baker, and David H Marimont. Epipolar-plane image analysis: An approach to determining structure from motion. International Journal of Computer Vision, 1(1):7-55, 1987.
[7] James Booth, Anastasios Roussos, Stefanos 7afeiriou, Allan Ponniah, and David Dunaway. A 3D morphable model learnt from 10,000 faces. In IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 5543-5552, 2016.
[8] Marcal C Bühler, Abhimitra Mcka, Gengyan Li, Thabo Beeler, and Otmar Hilliges. Varitex: Variational neural face textures. In IEEE/CVF International Conference on Computer Vision, pages 13890-13899,
2021.
[9] Fric R Chan, Connor Z. Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas. Jonathan Tremblay. Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In IEEE/CVF International Conference
Computer Vision, 2022.
[10] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In JEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5799-5809, 2021.
[11] Yu Deng, Jiaolong Yang. Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3D imitative-comtrastive contrastive learning. In JEEE/CVF Conference on Computer Vision and Puttern
Recognition. pages 5154-5164 2020.
[12] Yu Deng. Jiaolong Yang. Jianfeng Xiang, and Xin Tong. GRAM: Generative radiance manifolds for 3d-aware image generation. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022. [13] Yu Deng, Jiaolong Yang. Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face recon- struction with supervisd learning: From single image to image set. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[14] Terrance De Miguel Angel Bautista. Nitish Srivastava, Graham W Taylor, and Joshua M Susskind. Unconstrained scene generation with locally conditioned radiance fields. In IEEE/CVF International Conference on Computer Vision, 2021.
[15] Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstcin, Eli Shechtman, Dan B Goldman, Kylc Genova, Zeyu Jin, Christian Theobalt, and Maneesh Agrawala. Text-based editing of talking-head video. ACM Transactions
Graphics, 1814) 2019.
[16] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In /EEF/CVF Conference on Computer Vision and Pattern
Recognition, pages 8649--6658, 2021.
[17] Yue Gao, Fangyun Wei. Jianmin Bao, Shuyang Gu, Dong Chen, Fang Wen, and Zhouhui Lian. High-lidelity and arbitrary face editing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 161 16122,1614, 2021.
[18] Baris Gecer, Binod Bhattarai, Josef Kittler, and Tae-Kyun Kim. supervise adversarial learning to generate photorealistic face images of new identities from 3D morphable model. In European Conference Computer Vision. pages 217-234, 2018
[19] Jiahao Geng. Tianjia Shao, Youyi Zheng, Yanlin Weng, and Kun Zhou. Warp-guided gans for single-photo facial animation. ACM Transactions on Graphics, 3- 2018.
[20] Partha Ghosh, Pravir Singh Gupta. Roy Uziel, Anurag Ranjan, Michael J Black, and Timo Bolkart. Gif: Generative interpretable faces. In International Conference on 3D Vision, pages 868-878, 2020.
[21] Ian Goodfellow, Jean Abadee- Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing
Systems, 2014...
[22] Jiatao Gu, Lingji Liu. Peng Wang, and Christian Theobalt. Stylenerf: style-based 3d-aware generator for high-resolution image synthesis. In International Conference on Learning Representations, 2022. [23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Syxtems, pages 6626-6637, 2017.
[24] Yang Hong. Bo Peng. Haiyao Xiao. Ligang Liu, and Juyong Zhang, Headnerf: A real-time nerf based parametric head model. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
11

---

[25] Tero Karras, Miika Aittala. Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks, Advances in Neural Information Processing Systems, 34, 2021. [26] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4401-4410, 2019. [27] Tero Karras, Samuli Laine. Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In JEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 8110-8119 2020.
[28] Hycongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt. Deep video portraits. ACM
Transactions on Graphics, 37(4): 2018.
[29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[30] Marek Kowalski, Stephan J Garbin, Virginia Estellers, Tadas Baltrusaitis, Matthew Johnson, and Jamie Shotton. Config: Controllable neural face image generation. In European Conference on Computer Vision.
pages 299-315. Springer, 2020.
[31] Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5871-5880, 2020.
[32] Lingjie Liu, Mare Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Transactions on Graphics 4066);1-16, 2021.
[33] Ricardo Brualla Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and
Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In pages [34] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks:
IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7210-7219, 2021.
Learning 3d reconstruction in function space. In JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.
[35] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as ncural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421. Springer, 2020.
[36] Thu Nguyen-Phuoe, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsu- pervised learning of 3D representations from natural images. In IEEE/CVF International Conference on Computer Vision, pages 7598-7597.2 2019.
[37] Thu gguyen-Phhoo, Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra. BlockGAN: Learning 3d object-aware scene representations from unlahelled images. In Advances in Neural Information
Processing Systems, 2020.
[38] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453-11464, 2021.
[39] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3504-3515, 2020.
[40] Michacl Occhsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In IEEE/CVF International Conference on Computer Vision,
2021.
[41] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemellaacher-Shtizemann Stylesdf: High-resolutin 3d-consisten image and geometry generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[42] Jeong Joon Park, Peter Florence. Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 164,17,, 2019.
[43] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla, Deformable neural radiance fields. /EEE/CVP International Conference on Computer Vision, 2021.
[44] Keunhong Park. Utkarsh Sinha. Jonathan T Barron, Sofien Bouaziz, Dan Goldman, Steven M Seitz, and Ricardo -rrtala. Nerfies: Deformable neural radiance fields. In IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[45] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher--inensioonll representation for topologi- cally varying neural radiance fields. ACM Transactions on Graphics. 10 2021.
[46] Or Patashnik. Zongze Wu, Eli Shechtman, Daniel Cohen- Or, and Dani Lischinski. Styleclip: Text-driven manipulation stylegan imagery. In IEFF/CVF International Conference on Computer Vision, pages 2085-2094, 2021.
[47] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3D face model for pose and illumination invariant face recognition. In IEEE International Conference on Advanced Video and Signal Based Surveillance, pages 296-301, 2009.
12

---

[48] Sida Peng, Junting Dong, Qianqian Wang. Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In IEEE/CVE International Conference on Computer Vision, pages 14314-14323 2021.
[49] Sida Peng. Yuanqing Zhang. Yinghao Xu, Qianqian Wang. Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9054-
9063, 2021.
[50] Guim Perarnau, Joost Van de Weijer, Bogdan Raducanu, and Jose M. Alvarez. Invertible conditional gans for image editing. In Advances in Neural Information Processing Systems Workshop Adversarial Training, 2016.
[51] Jingtan Piao. Keqiang Sun, Quan Wang. Kwan-Yee Lin, and Hongsheng Li. Inverting generative adversarial renderer for face reconstruction. IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pages
2021.
[52] Albert Pumarola, Antonio Agudo, Aleix M Martinez, Alberto Sanfeliu, and Francesc Moreno-Nogue facial animation from single image. In European Conference on Computer Vision, pages 818-833, 2018.
GANimation: Anatomically
[53] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguee D-nerf: Neural radiance fields for dynamic scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[54] Yurui Ren, Ge Li. Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In IEEE/CVF International Conference on Computer Vision,
pages 17759-13768. 2021.
[55] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In Advances in Neural Information Processing Systems, 2020.
[56]
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[57] Yujun Shen, Ping Luo, Junjie Yan, Xiaogang Wang. and Xiaoou Tang. FaceID--GAN: Learning a symmetry three-player gan for identity-presert preserving face synthesis. In IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 821-830. 2018.
[58]
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in Neural Information Processing Systems, 32, 2019.
[59]
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit
neural representations with periodic activation functions. Advances Neural information Processing Systems, 33, 2020.
[60] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: continuous 3d-structure-aware neural scene representations. In Advances in Neural Information Processing Systems,
pages 1121-1132, 2019.
[61]
Dmitriy Smirnov, Matthew Fisher, Vladimir G. Kim, Richard Zhang, and Justin Solomon. Deep parametric shape predictions using distance fields. In Conference on Computer Vision and Pattern Recognition, 2020. Attila Szabó, Givi Meishvili, and Paolo Favaro. Unsupervised generative 3d shape learning from natural images. arXiv preprint 1rXiv:1910.00280 0019.
[62]
[63]
Ayush Tewari, Mohamed Elgharib, Florian Beraadd Hans-Peter Scidel, Patrick Pérez, Michael Zollhöfer, and Christian Thcobalt. Pie: Portrait image embedding for semantic control. ACM Transactions on Graphics, 39,4, 2020.
[64] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6142 6651, 2020.
[65] Justus Thics, Michael Zollhöfer, Matthias Nießner, 1 Levi Valgaerts, Mare Stamminger, and Christian Theobalt. Real-time expression transfer for facial reenactment. ACM Transactions on Graphicx, 34(6):183-
1. 2015.
[66]
Justus Thics, Michael Zollhofer, Mare Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment rgb vidcos. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2387-2395, 2016.
[67] Omer Tov,
Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen- Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics, 40(4):1 2021.
[68] Peng Wang, Lingjic Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Advances in Neural Information Processing Systems. 2021.
[69] Ting Chun Wang, Ming- Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot video-to- to to-video synthesis. In Advances in Neural Information Processing Systems, 2019.
[70] Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10039-10049, 2021.
[71] Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot frec-view neural talking- head synthesis for
video conferencing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[72] Ziyu Wang, Yu Deng. Jiaolong Yang. Jingyi Yu, and Xin Tong. Generative deformable radiance fields for disentangled image synthesis of topology-varyee varying objects. In Pacific Graphics, 2022.
13

---

[73] Chung- Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher Shaizeerr-Shiiemma.. Humannerf: Free-viewpoint rendering of moving people from monocular video. In IEEE/CVF International Conference on Computer Vision, 2022.
[74] Jianfeng Xiang, Jiaolong Yang. Yu Deng, and Xin Tong. GRAM-HD: 3d-consistent image generation high resolution with generative radiance manifolds, arXiv preprint arXiv:2206.0725. 2022.
[75] Sicheng Xu, Jiaolong Yang. Dong Chen, Fang Wen, Yu Deng. Yunde Jia, and Xin Tong. Deep 3D portrait from a single image. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 771 7710-7720, 2020.
[76] Xudong Xu, Xingang Pan, Dahua Lin, and Bo Dai. Generative occupancy fields for 3d surface-aware image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
[77] Yanbo Xu, Yueqin Yin, Liming Jiang. Qianyi Wu, Chengyao Zheng, Chen Change Loy, Bo Dai, and Wayne Wu. Transeditor: Transformer-basee based dual-space gan for highly controllable facial editing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[78] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fylfe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM Trunsactions on Graphics, 37(4):1-12, 2018.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "eCBcPxsuudK", "reviewer": "Reviewer_ch4r", "summary": "This paper proposed a framework for face image manipulation. It attempts to achieve fine-grained control over attributes and facial expressions by better preserving 3D consistency. The key contributions include a template implicit field and a 3D deformation field. The reported experimental results show that the proposed method can produce high-quality animatable video avatars with good texture consistency.", "questions": "1. What is the motivation behind AniFaceGAN? In introduction, there is not enough discussion about the motivation. From line 37 to line 38, the transition is not smooth. What weakness does NeRF have and how the proposed AniFaceGAN can do better? If NeRF is already \"producing realistic images with strong 3D consistency\", why do authors bother inventing a new method?\n2. What does Fig. 2 try to express? I had thought this is the overall framework but apparently, it is at most incomplete. For face image manipulation, we expect both input and output are face images and the underlying 3D model serves as prior or regularization terms. The training framework on the right and the network structure on the left do not help each other. It is simply too difficult to decipher how this figure is related to the task of face image manipulation - e.g., what is the difference between the two radiance manifolds? how come the arrow points from target to template (not the other way around)?\n3. Regarding Figs. 3-4, why do authors use different face identities in visual comparison? The same question applies to the video in supplementary materials. When different faces are used to compare different methods, it is difficult to see the difference. \n4. About the comparison with DiscoFaceGAN, there are a few strong claims - e.g., \"DiscoFaceGAN shows better image quality in terms of FID and KID, but sacrifices the 3D-consistency\" (line 218) and \"DiscoFaceGAN suffers from inconsistency for different expressions and poses\" (caption of Fig. 5). I am afraid I cannot see this weakness from either Fig. 5 or supplementary video. Can authors provide further clarification? Note that something obvious to the authors might not be that apparent to others.\n5. How about the computational complexity of AniFaceGAN? There is little discussion about the computational cost. ", "limitations": "I am curious about the performance of AniFaceGAN in some extreme situations - e.g., extreme pose or lighting conditions. Meanwhile, all demo video are very short - it is difficult to evaluate the quality of an animation for such short duration. I am wondering if the authors have video demo of longer period - say >3 seconds.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n+ The concept of 3D-aware face generation is a sensible idea. Ensuring 3D consistency for face image manipulation has shown high texture consistency across poses and expressions.\nWeaknesses:\n- It applied 3DMM expression parameters as a prior for disentanglement of latent variables. Although the experiments in this paper only focuses on 2D face image editing, from the title to the abstract, it easily confuses the reader that they addressed 3D face editing task - e.g., animation usually implies the generation of a 3D avatar from a given face.  The last sentence of the abstract about \"strong visual 3D consistency\" is misleading because there is no 3D reconstruction and editing contribution in this paper indeed.  \n- Regarding the 2D face image generation task, the idea of combining image rendering and 3DMM guided face editing is not novel. Also, from the experimental result view, they are not impressive to argue their contributions. From Figure 3 to Figure 4, their improvements are marginal compared to DiscofaceGAN.  \n- More importantly, the comparisons are not convincing with only two out of dated methods.  Some missing methods will need to be considered for comparison in the experiment:\nTransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editing \nInterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "bx-aKE_KqpR", "reviewer": "Reviewer_PWhM", "summary": "This paper proposes to generate 3DMM controllable 3D-aware faces. They design a generator that produces two 3D fields, namely a template radiance field and an expression-driven 3D deformation field. The expression-driven 3D deformation field is learned with the guidance of 3DMM. Experiments show that this method can produce high-quality 3D consistent controllable faces with delicate control.", "questions": "1. The authors write in the \"Face editing and animation with GANs\" section that \", these methods do not offer a generative modeling of face identities thus are difficult to generate virtual subjects\". More discussions could be made. Why not compare with them by first generating virtual samples and then controlling the results with 2D GANs? Or as the authors have shown inversion plus editing results. The comparisons can also be made.\n\n2. Why would the 3D landmark losses affect the FID? As described,  it is computed on 5K randomly synthesized images. And why replacing the 3D landmark loss in Eq. 8 with the 2D landmark regression loss would greatly increase the FID score? ", "limitations": "1. The whole story in the introduction and title seems to be overclaiming. \n\nWords such as \"Realistic Video Avatars\" and “high-quality animatable video avatars” have appeared in the title and abstract. From the perspective of Realistic Video Avatars, the results can barely meet the need. As all samples are aligned at the same position, the results are far from realistic.\n\nMoreover, if the authors intend to show results in a reenactment setting, it is better that they can show comparisons with results generated in a [GRAM + Face Vid2vid/PIRenderer] manner, and ask users to provide studies on the realness and quality of the results.\n\nThe reviewer's suggestion is to tone down the story and change the title to something like \"Realistic Animatable 3D-Aware Face Image Generation with XXX (Explicitly Deformable Radiance Manifolds)\"\n\n2. The comparisons and discussions are not sufficient.\n\na) The authors write that \"Perhaps the most relevant work to ours is DiscoFaceGAN\". This might be leading readers to believe that only comparing to DiscoFaceGAN is sufficient. Actually, StyleRig is also very relevant to the task of disentangled generative modeling and basically shares the same setting as this paper. The authors should discuss StyleRig (Although they also suffer from texture stitching) and try making a comparison with them.\n\nb) Please also refer to the question part.\n\nc) The numerical comparisons are limited. Why not use the metrics in DiscoFaceGAN on disentanglement for evaluation?\n\n3.  No audio is provided in the supplementary video, thus the reviewer cannot tell whether the lip movements are consistent with the source. Also, there are no 3D results shown, making the 3D consistent claim not strong enough.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++ The demo video of this paper is impressive from the perspective of controllable 3d-aware generative models.\n\n++ The design of one template radiance field and a deformation field is interesting and reasonable.\n\n++ The paper is well-written.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Qd4d1waUc7m", "reviewer": "Reviewer_FGbb", "summary": "This paper proposes a novel pipeline for animatable 3D-aware face image generation. They first propose a template radiance field to obtain the rough face geometry, then an expression-driven deformation field is proposed to animate the facial images based on the change of 3DMM parameters. Qualitative results verify the effectiveness of the proposed pipeline.", "questions": "The questions are elaborated in the weakness section, see above.", "limitations": "They are all properly discussed in the appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. The proposed method makes sense to me. It is natural to predict the non-rigid facial movement by inferring the per-point deformation based on 3DMM parameters. The presented qualitative results are mostly visibly plausible.\n\n2. The paper is well-written with thorough related works.\n\n# Weakness\n\n1. In Eq. 7, the chamfer distance is computed between the canonical 3DMM shape (reconstructed by the input exp and id param) and the predicted shape (inferred from the depth map as indicated by authors). How do the authors guarantee that the two shapes are at the same resolution, i.e., they are comparable for CD calculation. Besides, since the 3DMM only contain meaningful information at the facial regions, how do the authors deal with the neighbor regions like hairs?\n\n2. I am curious about how do the authors deal with the wearings, like the glasses, ear rings, etc. Such small items could not be inferred from the 3DMM prior, so how to guarantee the visual quality and 3D shape of them?\n\n3. Missing some relevant comparison baselines. Some important baselines are not compared, including but not limited to [1, 2, 3, 4]\n\n[1] Pie: Portrait image embedding for semantic control, Tewari et al., TOG 2020.\n\n[2] Stylerig: Rigging stylegan for 3d control over portrait images, Tewari et al., CVPR 2020.\n\n[3] Designing an encoder for stylegan image manipulation, Tov et al., SIGGRAPH 2021.\n\n[4] Headnerf: A real-time nerf-based parametric head model, Hong et al., CVPR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9YJf18i-yE6", "reviewer": "Reviewer_jYf9", "summary": "This paper presented a method for learning an animatable 3D-aware generative model for face images. The method is built on top of the 3D-aware generative model GRAM and introduces 3DMM-guided imitation learning to disentangle the latent space corresponding to identity and expression variations. The method is trained on an image dataset and compared with a few previous baselines, DiscoFaceGAN, CONIFG, etc. The method is quantitatively evaluated with FID and KID, where the performance is comparable to but generally not better than previous approaches. The method also performs qualitative comparisons on mostly randomly sampled images where the authors showed better view consistency than previous work.", "questions": "Please discussion aforementioned questions in the rebuttal.", "limitations": "I believe there need to be more discussions around the limitations of the method in terms of technical details and applications, as mentioned before. The negative societal impact is adequately discussed in my opinion.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has the following strengths:\n\n+ The topic is highly interesting. Controlling the trendy 3D-aware generators is of significant interest in the research communities of machine learning, computer vision, and graphics. Achieving high-quality control of 3D-aware generators is also desirable for many applications.\n\n+ The idea is interesting and has its novelty. DiscoFaceGAN has leveraged imitation learning to learn to sample 2D faces from a 3DMM fashioned latent space. This work adopted a similar idea to the 3D-aware GANs and introduced methods to guide/regularize the 3D representations.\n\n+ Results are encouraging. Thanks to 3D-aware GANs, the proposed method resulted in higher quality view consistency, which is generally not easy to achieve with a 2D GAN architecture.\n\nI'm generally satisfied with the technical components and the novelty of this paper. My main concerns are regarding the quality of results and the evaluation, detailed as follows:\n\n- In achieving facial animation, the paper works well around the mouth region but seems to fail at the eye region: the video-driven animation cannot animate motions around the eyes, such as blinks. This indicates no blinking eyes have been learned in either the 3D deformation modeling or the appearance modeling, or a combination of them. One can blame that the training data has few closed-eye images, but would data augmentation of closed-eye images resolve this problem in the 3D GAN scenario? Could 3DMM guidance be sufficient to model eye animation? \n\n- The DiscoFaceGRAM is a baseline proposed in this method but not well explained. How does it different from the proposed approach? It looks like the DiscoFaceGRAM generated images have artifacts unseen from either DiscoFaceGAN or GRAM, Where are they coming from? Is it due to the architectural design or the training process?\n\n- Quantitative results seem to be not better than previous approaches.\n\n- Almost all comparisons are done with random samples across different approaches, which is not easy to interpret. The best way might be to project images into the latent space and compare the results of editing so there is a common reference. On the other hand, the video results showed some projection-then-edit animations where we can see more artifacts? How were these images/videos produced? It looks like the approach might have limitations dealing with reconstruction/protection compared to 2D StyleGANs. Perhaps more discussion is needed here too.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept