FNeVR: Neural Volume Rendering for Face Animation
Authors: Bohan Zeng
Boyu Liu
Hong Li
Xuhui Liu
Jianzhuang Liu
Dapeng Chen
Wei Peng
Baochang Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: face animation
neural volume rendering
lightweight pose editor
image synthesis

Abstract: Face animation, one of the hottest topics in computer vision, has achieved a promising performance with the help of generative models. However, it remains a critical challenge to generate identity preserving and photo-realistic images due to the sophisticated motion deformation and complex facial detail modeling. To address these problems, we propose a Face Neural Volume Rendering (FNeVR) network to fully explore the potential of 2D motion warping and 3D volume rendering in a unified framework. In FNeVR, we design a 3D Face Volume Rendering (FVR) module to enhance the facial details for image rendering. Specifically, we first extract 3D information with a well designed architecture, and then introduce an orthogonal adaptive ray-sampling module for efficient rendering. We also design a lightweight pose editor, enabling FNeVR to edit the facial pose in a simple yet effective way. Extensive experiments show that our FNeVR obtains the best overall quality and performance on widely used talking-head benchmarks.

FNe Neural Volume Rendering for Face Animation
Bohan Zenghi, Boyu Liu Hong Li', Xuhui Liu",, Jianzhuang Liu?,
Dapeng Chen*, Wei Peng', Baochang Zhang
Institute of Artificial Intelligence, School Automation Science and Electrical Engineering,
Beihang University, Beijing, P.R.China
2Sino-French Engineer School,
Beihang University, Beijing, P.R.China
Huawei Noah's Ark Lab, Shenzhen, P.R.China
'Huawei, Shenzhen, P.R.China
ZZhongguaneun Laboratory, Beijing, P.R.China
"Corresponding author, email: xhliu @ buaa.edu.cn, bezhang @ @buaa.edu.cn Equal contributions
Abstract
Face animation, one of the hottest topics in computer vision, has achieved a promis- ing performance with the help of generative models. However, it remains critical challenge to generate identity preserving and photo-realistic images due to the sophisticated motion deformation and complex facial detail modeling. To address these problems, we propose Face Neural Volume Rendering (FNeVR) network motion warping and 3D volume rendering
to fully explore the potential unified framework. In FNeVR, we design a 3D Face Volume Rendering (FVR) module to enhance the facial details for image rendering. Specifically, we first extract 3D information with well-designed architecture, and then introduce orthogonal adaptive ray-sampinng module for efficient rendering. We also design lightweight pose editor, enabling VN to edit the facial pose in simple yet effective way. Extensive experiments show that our FNeVR obtains the best overall quality and performance widely used talking- head benchmarks. Our code is available"
Introduction
Aiming at animating a still head, face animation has far-reaching applications, such as photography, social media, movie production and virtual assistance, and has become popular research topic in computer vision and computer graphics [10, [4]. It synthesizes a realistic talking-head video by combining the appearance and identity from one source face image with poses and motions extracted from a given driving video, possibly of a different person's identity from the source. With the progress of generative models, especially Generative Adversarial Networks (GANs) and Variational Auto Euto--Encoerrs (VAEs), recent methods have achieved a promising performance in face animation 557 46 12. However, is still challenging task because of the difficulty of guaranteeing the motion transform validity and generating authentic images simultaneously.
Face animation methods can be mainly divided into three categories: model-free, landmark-baseds and 3D sauuctre-based Model-free methods [55] 45 46, 52 7 realize compelling reenactment by mod- eling the relative motion field from the source the driving faces. Notably, First Order Motion Model (FOMM) [46] significantly improves the performance of image animation by using self-supervised keypoints with 2D motion warping. Our studies show that 2D warping is effective in generating
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Source
(b) Driving
(c) FOMM
(<) Face vidAvd
(e) FN:VR
Illustration of face synthesis results. FOMM with 2D warping produces a blurred head profile but with promising pose transformations. Face vid2vid produces clear head profile but with poor facial details. Our FR can combine their merits and generate photo-realistic images.
Figure 1:
high-quality motion transfer. However, it has a limited ability to produce sufficiently realistic images (see Fig. [e)). Landmark-basse methods [2] 15 56 49 57 introduce 2D facial landmarks and obtain considerable progress in face animation, but they are intrinsically visadvantageous in identity preservation. Motivated by the advanced 3D Morphable Models (3DMMs) 3 40; 16 4 5 6 3I],
researchers tend to integrate the 3D geometric prior to condition synthesis and achieve impressive performance to generate realistic images (30 [2]. Nonetheless, these 3D structure based methods often focus introducing 3D facial information in the motion model to improve the authenticity of the results, such as Face vid2vid [51], neglecting the essential advantages of 2D warping in motion transfer. As shown in Fig. dd) Face vid2vid produces clear head profile but with poor facial details.
In this paper, we propose Face Neural Volume Rendering (FNeVR) network for more realistic face animation, by taking the merits of 2D motion warping [46) on facial expression transformation and 3D volume rendering the remarkable properties of neural rendering [33 22 in establishing high fidelity representations, design novel 3D Face Volume Rendering (FVR) module for face animation, which enables our FN to learn more realistic facial details efficiently and thus achieve photo-realistic results. Specifically, the proposed FVR module first adopts convolutional neural networks (CNNs), which we design to extract the 3D color and shape information from the 2D) warped feature with the aid of the 3D reconstruction results. We only apply the 3D reconstruction module supervise the 31) CNNs in the training process without any overhead for inference, which is different from existing 3D structure-based methods [30 12 51] After that, we introduce an orthogonal adaptive ray-sampling module to efficiently calculate the sampled color field and the voxel probability for subsequent image
high-quality image synthesis in unified framework. Inspired by
rendering. Instead of using hierarchical sampling as in [33], our orthogonal adaptive ray-sampling only needs one MLP to process the 3D information directly. Additionally, we design a perceptual loss guarantee the quality of rendering feature maps. All in all, our VN can not only generate more realistic images than 2D-based methods. but also obtain more accurate motion transfer than 3D-based methods (see Fig. [e)). Moreover, we present Lightweight Pose Editing (L.F.) module, which is simple yet effective and only takes Euler angle as the input to efficiently realize facial pose editing. We further design a new loss to supervise the face editing model. The contributions of this paper include:
A Face Neural Volume Rendering (FN network presented to take the merits of 2D motion warping on face expression transformation and 3D volume rendering on high-quality image synthesis in a unified framework for realistic face animation.
A Face Volume Rendering (FVR) module with orthogonal adaptive ray-sampling is proposed to capture facial details effectively and improve animation performance. Additionally, simple yet effective Lightweight Pose Editing (LPE) module is presented only based on the Euler angle.
Extensive experiments are conducted to compare VN with state-of-the-an the-art methods. The results show that our VN obtains the best overall quality and performance on widely used sulking-heade benchmarks.
2 Related Work
complete the deformation of face images by establishing the motion field without extracting prior facial information in advance. Early models (55) can extract identity and pose information, but they may fail to capture the appropriate deformations in some problematic cases. MonkeyNet [45] handles motion transfer by acquiring
Face Animation. Model-free methods (55 45
52
7 51

---

sparse keypoints. Subsequently, the First Order Motion Model (FOMM) [46] significantly improves the performance of face animation with a rigorous first-order mathematical model. While reporting promising synthesized images, most 2D model-free methods have limited ability to model out-of- plane rotations and expressional motions. Face vid2vid [5T] further improves FOMM by introducing 3D representations to model 3D motion field for face generation. Nevertheless, it has a considerable computational cost and still lacks the capability of expression transformation. Landmark-based methods [2] [15 (56 19 49 (57 utilize 2D facial landmarks as conditions for reenactment, However, since 2D facial landmarks cause the injection of identity-related information from the driving face to the generated one, these methods often cannot handle identity preservation well during the generation
process.
48 focus more on 3D structure- based methods by means of the geometrie prior of 3D faces and produce impressive results on subject-agnostic face synthesis. HeadGAN [1] employs the rendered 3D mesh as input to synthesize deformed images, but it does not work well in facial expression transformation. Conditioned on the parameters of the 3D Morphable Model (3DMM) 33 StyleRig 47
In order to address the above issues, many works [28 27 30
and GIF 17 respectively employ pre-trained StyleGAN [25] and StyleGAN2 (26 to warp face images. PIRenderer [41] controls the face motions and predicts a flow field for deformation. Nonetheless, these methods are intrinsically limited to modeling the specific head components such as hair and teeth.
3D Morphable Face Models. The 3D Morphable Model (3DMM) [3] is statistical model that provides 3D face representation for rendering face images by parameter modulation. Recent methods extend 3DMM to more precise modeling of shape, expression, or pose on the basis
40 16 of the original synthesis. Among these models, FLAME [31] additionally models the head pose, joint pose, and geometric expression information, offering more realistic and expressive 3D faces. Benefitting from 3DMM, the face reconstruction methods [ II [54] achieve significant progress. The representative method DECA [33 introduces an encoder to extract the parameters of the face for FLAME and employs another encoder to attain detailed reconstruction. In this paper, we exploit a pre-trained encoder of DECA to extract the parameters required by FLAME and then utilize FLAME to reconstruct a reliable 3D face representation from 2D images.
4
[ 6
3D Volume Rendering, 3D volume rendering is used for rendering 2D projections of 3D objects and scenes. NeRF 33 is breakthrough view synthesis which can produce impressive rendered results based on the input positional information. Its core idea lies in optimizing continuous volumetric scene function using a sparse set of input views. Thanks to its excellent performance, many works integrate NeRF with GANs to directly control the pose of synthesized result and generate impressive multi-view images. GRAF 43 introduces conditional GANs on the basis of NeRF essentially, which improves the quality and 3D consistency of the rendering results. GIRAFFE 388 can complete the rendering of a real-world scene and freely add multiple objects to the same scene. Differently, we introduce lightweight and efficient adaptive volume rendering method, which only requires a single view to enhance the quality of the rendered 2D image.
43 336 37 38
8 18 59 221
3 Method
We formulate the face animation task as a talking- head generation with an editable pose. For a given source image S and driving video {Di. video and N is the total frame number, the objective of our FN is to generate a reoto-realistic head video with the same face identity S and the motion and facial expressions derived from
.Dv). J. where D, denotes the i-th frame in the
DN }- As shown in Fig. 2 the whole framework mainly consists of four modules to achieve this goal: (1) 2D motion estimation, (2) 3D face reconstruction, (3) FVR, and (4) LPE. With S and D; as input, we first utilize the 2D motion estimation module to estimate 2D facial keypoints and generate a motion field to transform the features. We also employ the 3D reconstruction module to build 3D face representations to provide the rendering network with precise and reliable 3D shape information. Then novel FVR module is introduced to fully explore the capacity of the 2D motion module on the challenging expression transformation and generate images with high fidelity. In addition, the LPE module added to extend our model to edit the facial pose efficiently.
3.1 2D Motion Estimation
Despite the trend that most existing methods 51 21
12 focus on studying various estimation
methods of motion fields, extensive experimental results in 46 48 show that the strategy of warping
3

---

Sourcee image
Encoder
Warped feature
odlor
Praler
Motion E Estimation
Keypeints information
IosSedicer
LPE
Generated result
Yaw, Pitch, Roll
Pr
Rendering feature map
Decoder
Orthogonal Ray-sampling
acce
to58,
Reconstruction
Volume Rendering (FVR)
Driving image
3D vertices
Figure 2: Overview of the proposed FNeVR, where @ indicates the channel-wise concatenation. We first extract the warped feature and 3D face reconstruction representation, on which FVR is then performed to obtain the 3D shape and color information for face rendering. The 3D face reconstruction result supervises LPE and FVR by providing pose angles and 3D face vertices.
2D features achieves a consistent motion transfer of facial details. Based on this observation, we utilize the 2D motion estimation module proposed in FOMM [46] complete the warping process. As shown in Fig. 3 a)), we use an Hourglass Network [35 as the keypoint detector to extract = 10 keypoints {Ps.k. Pr, R²} and their Jacobians (Js,s, J,. E <R*x2 of both source and driving images. For each keypoint, we utilize the first-order Taylor expansion to build an affine approximation motion filed according [46]:
Ts= = PSA Js. PD.)),
where z I² indicates a location in the image. Apart from K sparse motion fields, an extra motion field is further introduced to preserve the static background information. To generate the final dense motion field, we aggregates these K + motion fields with related weight masks { (Mg.M], ,M estimated by U-Net used in FOMM. Furthermore, additional occlusion map O indicating regions to be inpainted is also predicted. The process obtaining the dense motion field and the warped feature F.. is:
(s))(()) = Moz +
Σ Mr (s
h=1
Fe. = 0 fmn Ts
where ( is the warping function, F's denotes the source feature and @ is the Hadamard product.
3.2 3D Face Reconstruction
As shown in Fig. b)). exploit the capability of pre-trained encoder 13 to extract the parameters required by FLAME [11 and use the prior face knowledge of FLAME to reconstruct a 3D face representation from 2D images. The reconstruction module is typical encoder-decoder structure, where FLAME serves as the decoder, using linear blend skinning (LBS) and additional expression blend shapes to generate detailed mesh v R&N with N 5023 vertices. Mathematically, given the parameters of shape & R1a pose 0 e 33(++1) (a global rotation vector and K joints' rotation vectors), and expression E 1 FLAME is represented by blend skinning function W as:
17 (Tp v J (3), W),
where T, E RRN depicts head mesh in zero pose, J E R3K denotes the locations of the K joints, and W E RKXN X acts blend weights. The camera parameter e is further estimated to scale and translate the reconstructed vertices to the camera view. Meanwhile, the shape parameter (3 of the source image is assigned to the driving image the reconstruction target.

---

Motion Estimation & Lightweight Pose Editor
3D Face Reconstruction
Source image
Keypolint Detectror
Dense Miniion
arped feature
Source imsge
Encrdkr
B.0.thc
FLAME
-lotsyttter
Driving image
Demse flons
3D vertices
Reditor
Pitch Roll
Mirddade
value 6jacobtan
Driving
image
Occlusion map
(b)
Figure 3: Illustration of Motion Estimation and LPE modules, and 3D Face Reconstruction module.
3.3 Face Volume Rendering (FVR)
We design a new neural rendering framework, Face Volume Rendering (FVR), for face animation as illustrated in the red box in Fig. 2 In this framework, we innovatively employ our designed CNNs effectively extract the 3D color and shape features from the warped feature. We also introduce new loss function to optimize the 3D shape feature with the 3D reconstruction result, which is only utilized during the training process without any overhead for inference. Moreover, we develop a new orthogonal adaptive ray-sampling module to estimate the sampled color field and voxel probability for image rendering in an efficient way. With these novel designs, our FVR module is capable of capturing more realistic facial details than 2D rendering. We also provide more details about our method in the supplementary materials.
3D Feature Extraction. Based on the warped feature Fws we first need to extract the high dimen- sional 3D shape feature Fo and 3D color feature Feotor for subsequent face volume rendering. To this end, we respectively design two CNNs Pa and Pentor (see the supplementary materials), by which richer 3D shape and color features be achieved than [33]
the one hand, Pa calculates F, (F.).. R=*W>D*N, where represents the spatial depth information and N, is the number of channels representing the 3D shape information, which set to in this paper. Observing that the spatial mesh can be well reconstructed by the 3D face reconstruction module, it is thus reliable to utilize the 3D mesh feature F'm derived from to supervise Po to achieve a reasonable result. We maximize the inner product between vectorized Fm and Fo, which leads to matching loss function Lg:
C. = exp(-ar < F. Fm -) 022
(4)
where (1 and a2 are parameters to control the shape
loss function. practice, we empirically set 01 = 10 and (22 = 0.9. Here, we elaborate on computing F,, below. Firstly, we conduct down-sampling operation on the 3D vertices v to get Ndown sampled vertices Udowns which helps to reduce the calculation cost and climinate redundant vertex information. Additionally, leverage the Gaussian function to process the i-th sampled vertex Udown,is forming Gaussian heatmap Fm, with the size of H W x D
Vdoonn 1|
Fm, = exp(
(5)
2a
where a is the variance and result Fm with the size of H x W x D x 1 is calculated by a weighted sum of all the Gaussian heatmaps F as:
0.01. Afterward, the 3D mesh feature of the reconstruction
NAwsn uy Fm.is
F
(6)
where wi is the weight for Fm... Notably, w; is predicted adaptively by an additional CNN. It is worth noting that the 3D reconstruction vertices are not needed during inference, and we can therefore complete the image synthesis in an efficient way.
5

---

On the other hand, we need to extract the 3D color feature from the 2D warped feature Fw. To this end, we use Pootor estimate the 3D color feature Feator with the size of x W D x N Nestor:
Feolor Pootor (Fu) E RR"xxxDD DNooco
where Neotor is the number of channels representing the color information.
Orthogonal Adaptive Ray-Sampling. With the extracted 3D features, we further calculate the sampled color field and the voxel probability, which are required for rendering. Different from single- view task, which casts rays from the camera into the scene in the direction orthogonal to the image plane. Accordingly, we design an orthogonal ray-samp module adaptively estimate the sampled color field Peotor and the voxel probability Pa, which is achieved by a single MI.P-based one-stage process fo as follows:
conventional 3D volume rendering tasks [T8] 43 38), face animation can be considered
Pootor = fo Feolor) E "XWWXDII X H
(8)
where Mcotor is the number of channels representing the color information of Peotor, and the interval sampling number D is the same as the previous spatial channel number D. Note that po is the integral of the volume density with an adaptive interval size along the orthogonal ray. In particular, the selection of the interval size involved in estimating the voxel probability po and adaptively adjusted by the MLP.
It is worth noting that unlike the hierarchical volume sampling as in 33 which requires two-stage network to process multiple views according to the rays of different directions, our orthogonal adaptive ray-sampling only needs one MLP to deal with the assumed ray the angle orthogonal to the image plane. Furthermore, instead of decoupling the facial attributes as geometric-pen prior guided sampling methods (22), we directly process the 3D related features to predict the sampled color field and the voxel probability more efficiently. In short, by simplifying the ray-sampling with an adaptive approach, our one-stage method can significantly increase the rendering speed compared with the common two-stage process. More details about the sampling module can refer to the supplementary materials.
Image Rendering. Specifically, we project the sampled color field Pootor and the estimated voxel probability Po onto 2D generate the rendering feature map F,,. Our rendering method is a modified version of that in (32) according to orthogonal adaptive ray-sampling. Also let be pixel point in the feature map. and j be a sample interval in each pixel i. The rendering method calculates the final color result Fr,; for each pixel i according to the corresponding sampled color field Peotor,i.j and voxel probability Po.i.j as:
F.. Σ Ty (1 exp( (sp(-Po,.s)) Pootorsi.j
(9)
where Po.i. and Peolor.i.j represent the voxel probability and the sampled color field of the j-th interval of the i-th pixel respectively, and T, = exp(_ -Po,i,k) is the transmittance.
Moreover, to ensure the effectiveness of the rendering feature F,, we introduce a two-layer CNN P. and the VGG perceptual loss (24 50) in the training process. Specifically, we directly input F, into the CNN to generate an intermediate image Im, and then compute the perceptual loss LR between Im and the driving frame D; as:
La VGG perceptuul (D:, Im).
(10)
Afterward, concatenate the volume rendering feature map F, and the warped feature F and feed them into our designed rendering decoder which introduces the SPADE layer [39 to obtain the final generated result. This decoder (see the supplementary materials) can enhance the rendering quality while preserving facial details of the input.
3.4 Lightweight Pose Editing (LPE)
Existing face animation methods, which adopt 2D warping motion network 46 well in the feature warping but fail on pose editing. As illustrated in Fig. B a), we introduce a lightweight network (LPE) to add new pose editing function for our framework. Taking the rotation
48 21
work

---

Table 1: Quantitative comparison of same-identity reconstruction on VoxCeleb 34
Method
L1 +
LPIPS
PSNR
SSIM
AKD 4
AED 4
FID 4
Bilayer [66
0.1197
0.4247
15.219
0.3968
12.60
0.0546
219.8
FOMM 46
0.0450
0.1099
23.210
0.7475
1.383
0.0244
11.56
Face vid2vid
0.0485
0.1051
22.642
0.7268
1.616
0.0395
9.142
Face vid2vid-S [51
0.0445
0.0901
23.357
0.7473
1.421
0.0243
9.151
DaGAN 21
0.0462
0.0981
23.263
0.7536
1.441
0.0247
9.660
PIRender 41
0.0566
0.0850
21.040
0.6550
2.186
0.2245
11.88
FNeVR (ours)
0.0404
0.0804
24.292
0.7773
1.254
0.0231
8.443
angle information (yaw E RR. pilch E RR. roll E R!} provided by the 3D face reconstruction module and the keypoint information of the source image input, LPE (Jeditor) estimates the target keypoints dvalue and their Jacobians Jacobian In practice, first formulate yow, pitch and roll as a 3D rotation matrix Trotate E IR3x3 and then introduce an MLP as feditor to compute 6 vvalue and
ò Jacobion under specific rotation angle Orotate:
doaluey JJcobban feditor (frotate: Ps, Js) E R Kx2 x R Kx2x2
(11)
Moreover, we design a loss Cedstor to supervise feditor by aligning 8vatue and Jacobian with the keypoints and the Jacobians the driving frame, respectively:
Leditor d ( Seatue) + (2 5Jacobian),
(12)
where A and X2 are balancing parameters, and L1 is the 4 norm loss. Empirically we set λ = 1 and A2 0.5.
3.5 Training
Our framework is implemented based on self-supervised learning or self-reenactment, where a pair of the source image S and the driving image D, from the speaker in video is used as the input and D, also acts as the ground truth for supervision. We train our FNeVR by minimizing the following loss Lrotal:
Liotal = Lr(Di, (generated) + La(Di, &generated) + LE(PD, JD)+ (R, Tm.) + Lo (Fo, Fm)) Leditor (pp, Jn, Svalue, 83 JJacobann) )
(13)
where I #'gneerated is the output of the whole network FNeVR. Liotal consists of four parts: (1) C. and LG are used to maintain the quality of the generated image: (2) LE is used adjust the 2D motion estimation module to predict reasonable motion fields; (3) L and L. are used to optimize 3D volume rendering module to generate better 3D rendering features; (4) Leditor is used to optimize the face pose editor. For (1) and (2), we employ the same loss functions as in [46), including the perceptual loss Lp, GAN loss LG and equivalence constraint loss LE. For (3) and (4), the losses mainly provide the quality guarantee of the 3D rendering feature F,.
Experiments
In addition to the extensive experiments described in this section, we also provide more results in the supplementary materials.
4.1 Implementation Details
which contains more than 100,000 videos covering 1,251 speakers of different identities, and VoxCeleb2 (9, which contains about 1M videos of different celebrities. We adopt the same preprocessing strategy of cropping faces from the videos and resizing them to 256x256 as in [66]
Datasets. Our evaluation is performed VoxCeleb 34
Training Details. We train FNeVR with self reconstruction the training set, in which pair of source and driving images from the same speaker video are used as input, and the driving image also serves as ground truth for supervision. FNcVR is trained for 100 epochs, repeating the video
7

---

Souvc
Driving
Bllayer
FOMM
FaceviDaid
FaceviDDdds
DaGAN
FNVV
same-identity reconstruction; Bottom: cross-identity reenactment. For both tasks, our FNeVR produce more realistic and fine-grained details than the others. For example, FOMM struggles to strike a balance between clarity and facial details, and Face vid2vid-S and DaGAN may cause distortions within some facial areas (eyes, mouth, and teeth).
Figure 4: Qualitative comparison with SOTA baselines. Top:
Table 2: Quantitative comparision of cross-identity reenactment on VoxCeleb 33 and VoxCeleb2 99.
Table 3: Quantitative comparison of Flops, memory, and efficiency.
Method
VoxCeleb
VoxCeleb2
Method
Flops(G) Params(M)
FPS
FID CSIM
FIDJ CSIM f
Face vid2vid
231.038
125.216
17.790
FOMM
106.9 0.5491
138.1
0.5228
Face vid2vid-S
636.941
173.109
13.219
Face vid2vid-S
106.6 0.6447
148.6 0.6290
DaGAN
75.642
74.660
26.753
DaGAN
110.3 0,5305
139.6 0.4932
FNeVR (ours)
98.23 0.5505
133.9 0.5282
FNeVR (ours)
130.109
61.378
36.568
image set 75 times per epoch. We adopt Adam 229 optimizer with learning rate 1) 2 x 10-4.
3 = 0.5 and 3₂ = 0.9 for each module. Since our model is lightweight, we only use 24GB NVIDIA 3090 GPUs during training.
Evaluation Metrics. The metries include (1) reconstruction faithfulness using L1, PSNR, SSIM
and LPIPS (58), (2) output visual quality using FID (20). (3) Average Keypoint Distance (AKD)
46
(4) Average Euclidean Distance (AED) [46), and (5) identity preservation cosine similarity CSIM
42 P3
4.2
Comparison with State-of-the-Art Methods
Methods. We compare our FNeVR with five state-of-the-arr (SOTA) models: FOMM (46), Face vid2vid [5T], Face vid2vid-S [5I) Bilayer (56). DaGAN (11) and PIRenderer 41). We use the official pre-trained models for Bilayer, FOMM, DaGAN and PIRenderer, and a widely recognized unofficial model for Face vid2vid due to the absence of the official code. Face vid2vid-S is obtained by replacing the decoder of Face vid2vid with a SPADE 39 layer-based decoder. All of these models trained VoxCeleb.
Same-Identity Reconstruction. We conduct a quantitative comparison with SOTA methods on the testing dataset of VoxCeleb when the source image and the driving image belong the same person.

---

Input image
Yaw
Pitch
Roll
Figure 5: Visualization of the pose editing results produced by LPE.
Table 4: Ablation study for same-identity reconstruction on the VoxCeleb testing set.
Configurations
L1 4
LPIPS +
PSNR t
SSIM
AKD 4
AED 4
FID 4
FOMM baseline
0.0450
0.1099
23.210
0.7475
1.383
0.0244
11.56
FOMM + SPADE
0.0426
0.0889
24.006
0.7710
1.316
0.0246
11.13
FOMM FVR
0.0404
0.0875
24.374
0.7783
1.290
0.0233
10.76
FNeVR w/o Fw 0.0460
0.0854
22.653
0.7423
1.367
0.0262
12.07
FNeVR w/o Co 0.0412
0.0869
24.163
0.7705
1.284
0.0239
9.024
FNeVR (ours)
0.0404
0.0804
24.292
0.7773
1.254
0.0231
8.443
As shown in Table I it is evident that our FNeVR outperforms other SOTA methods on all metrics. Especially, FNeVR achieves pronounced improvements on several metrics, such as image quality (PNSR), semantic consistency (AKD), and image authenticity (FID). Fig. a) shows that FNeVR can generate the most realistic images that are most similar to the driving faces.
Cross-Identity Reenactment. We compare the performance of our FNeVR with SOTA methods on the VoxCeleb and VoxCeleb2 testing datasets when the source image and driving video come from different persons. Concretely, we randomly select 10 source images and 14 driving videos from the testing datasets. Table illustrates that FNeVR produces the best overall performance. Although Face vid2vid-S has a better result in terms of CSIM, it consumes several times more computation and memory cost (FLOPs and Parameters) than VN, as indicated Table demonstrating FNe VR's outstanding efficiency in compact computation. Moreover, Fig. b)) visualizes some synthesis results from different methods reenactment and shows that FNeVR can generate more realistic faces whose poses and expressions are closest to the driving faces.
4.3 Ablation Study
We conduct comprehensive experiments on the same-identity reconstruction task to further demon- strate the effectiveness of each module in the proposed FNeVR. Specifically, employ FOMM [46] as the baseline, and construct four comparison models: (1) replacing FOMM's decoder with our designed decoder, (2) inserting FVR into FOMM, (3) FN without F.v inputted into the decoder, and (4) FNeVR without C., The results are shown in Table Both FVR and our designed decoder have significant effectiveness. In particular, FOMM with FVR inserted produces much better results than the baseline. Furthermore, FN without Fw. performs worse than the full FNe VR, indicating that not inputting F. into the decoder will affect the model's ability transfer the facial poses and expressions, and lead to the loss of facial details in the generated results. Furthermore, the results show that if C.p is not introduced, all metrics are worse than the full FNeVR, indicating that is necessary to introduce reliable 3D information. Meanwhile, even without the 3D information, FNeVR still performs well, demonstrating the great effectiveness of the proposed framework. Note that our full model produces the best overall results and surpasses other models lot in several metrics, fully verifying the efficacy of VR.
4.4 Lightweight Pose Editing
With the pose parameters, we conduct qualitative experiment with pose editing visualization in Fig. to further evaluate the effectiveness of the proposed LPE. Specifically, according to the Fuler angle (yaw, pilch, and roll) provided by the 3D face reconstruction result, we separately adjust the rotation effect of the generated results. Fig. 5 shows that I.PE can generate reliable face rotations according to a given Euler angle.

---

5 Conclusion
In this paper, we propose a Face Neural Volume Rendering (FNeVR) network for face animation, which unifies the 2D motion warping and 3D volume rendering in one framework. We follow the line of 2D motion warping and introduce the 3D face reconstruction module to provide reliable 3D shape information for the rendering process. We innovatively develop a Face Volume Rendering (FVR) module to enhance the facial details of the warped feature and generate high high-quality faces. Moreover, we design a Lightweight Pose Editing (LPE) module, which can directly implement pose editing with rotation angles. Extensive experiments illustrate that our FNeVR achieves state-of-the-art performance.
Acknowledgements
This work was supported by "the Fundamental Research Funds for the Central Universities", and the National Natural Science Foundation of China under Grant 62076016, Beijing Natural Science Foundation-Xinnn -iaomi Innovation Joint Fund 1.223024. Besides, we gratefully acknowledge the support of MindSpor CANN (Compute Architecture for Neural Networks) and Ascend Al processor used for this research.
References
[1] Riza Alp Guler. George Trigeorgis. Epameinondas Antonakos, Patrick Snape, Stefanos Zafeiriou, and lasonas Kokkinos. Densereg: Fully convolutional dense shape regression in-the-wild. In CVPR, 2017.
[2] Hadar Averhuch-Flor, Daniel Cohen-Or, Johannes Kopf, and Michael F Cohen. Bringing portraits life.
TOG, 2017.
[3] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH, 1999.
[4]
Timo Bolkart and Stefanie Wuhrer, A groupwise multilinear correspondence optimization for 3d faces. In
ICCV, 2015.
[5] Timo Bolkart and Stefanie Wuhrer. A robust multilinear model learning framework for 3d faces. In CVPR,
2016.
[6] James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway. A 3d morphable model learnt from 10,000 faces. In CVPR, 2016.
[7] Egor Burkov, Igor Pasechnik. Artur Grigorev, and Victor Lempitsky. Neural head reenactment with latent pose descriptors. In CVPR, 2020.
[8] Fric R Chan, Marco Monteiro. Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021.
[9] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxecleb2: Deep speaker recognition. INTER- SPEECH, 2018.
[10] Zhigang Deng and Junyong Noh. Computer facial animation: survey. In Data-Driven 3D Facial Animation. Springer, 2008
[11] Pengfei Dou, Shishir K Shah, and loannis A Kakadiaris. End-to-end 3d face reconstruction with deep
neural networks. CVPR, 2017.
[12] Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriaa Sharmanska. Headgan: One-shot neural head synthesis and editing. In /CCV, 2021.
[13] Yao Feng. Haiwen Feng. Michael J Black, and Timo Bolkart. Learning an animatable detailed 3d face model from in-the-wild images. TOG, 2021.
[14] Yao Feng. Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou. Joint 3d face reconstruction and dense alignment with position map regression network. In ECCV, 2018.
[15] Jiahao Geng. Tianjia Shao, Youyi Zheng, Yanlin Weng, and Kun Zhou. Warp-guided gans for singlc-photo facial animation. TOG, 2018
[16] Thomas Gerig. Andreas ForserForster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönborn, and Thomas Vetter. Morphable face models-an open framework. In FG, 2018.
[17] Partha Ghosh, Pravir Singh Gupta, Roy Uziel, Anurag Ranjan, Michael J Black, and Timo Bolkart. Gif:
Generative interpretable faces. In 3DV, 2020.
[18] Jiatao Gu, Lingjie Liu, Peng Wang. and Christian Theobalt. Stylenerf: style-based 3d-aware generator
for high-resolution image synthesis. arXiv:2110.088
2021.
[19] Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim. Marionette: Few-shot face
reenactment preserving identity unseen targets. In AAAI, 2020.
[20] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium, Asian Journal of Applied Science and Engineering, 2017.
Phtpps//www.mindspor..d
10

---

[21] Fa- Fa-Ting Hong. Longhao Zhang. Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In CVPR, 2022.
[22]
Yang Hong, Bo Peng, Haiyao Xiao. Ligang Liu, and Juyong Zhang. Headnerf: A real-time nerf-based parametric head model. In CVPR, 2022.
[23] Yuge Huang. Yuhan Wang. Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In CVPR, 2020. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
[24]
[25] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[26] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
[27]
Hycongwoo Kim, Mohamed Elgharib. Michael Zollhöfer. Hans-Peter Seidel, Thabo Beeler, Christian
Richardt, and Christian Thcobalt. Neural style-preserving visual dubbing. TOG, 2019.
[28]
Hycongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt. Deep video portraits. TOG, 2018. [29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[30] Mohammad Rami Koujan, Michail Christos Doukas, Anastasios Roussos, and Stefamos Zafeirio..
Head2head: Video-based neural head synthesis. In FG. 2020.
[31] Tianye Li. Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. TOG, 2017.
[32] Nelson Max. Optical models for direct volume rendering. TVCG, 1995.
[33] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik. Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing seenes neural radiance fields for view synthesis. In ECCV, 2020.
[34] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: large-scale speaker identification dataset. INTERSPEECH, 2017.
[35] Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estimation. In
ECCV, 2016.
[36] Thu Nguyen-Phuoe, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologam: Unsuper- vised learning of representations from natural images. In JCCV, 2019.
[37] Thu H Nguyen-Phuut PhPhooe Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra. Blockgan:
Learning 3d abject-aware scene representations from unlabelled images. In NeurIPS, 2020.
[38]
Niemeyer and Andreas Geiger. Giraffe; Representing scenes as compositional generative neural
feature fields. In CVPR, 2021.
[39] Taesung Park, Ming- Yu Liu, Ting- Chun Wang, and Jun- Yan Zhu, Semantic image synthesis with spatially- adaptive normalization. In CVPR, 2019.
[40] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for and illumination invariant face recognition. In AVSS, 2009. [41] Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In ICCV, 2021.
[42] Elad Richardson. Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for image-to-image translation. In CVPR, 2021.
[43] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In NewrlPS, 2020.
[44] Tong Sha. Wei Zhang, Tong Shen, Zhoujun Li, and Tao Mei. Deep person generation: A survey from the perspective of face, pose and cloth synthesis. arXiv:2109.0220 (20000201..... 2021. [45] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sche. Animating arbitrary objects via deep motion transfer. CVPR, 2019.
[46] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In NeurIPS, 2019.
[47] Ayush Tewari, Mohamed Figharih, Gaurav Bharaj. Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait images.
In CVPR, 2020.
[48] Qiulin Wang, Lu Zhang, and Bo Li. Safa: Structure aware face animation. In 3DV, 2021.
[49]
Ting-Chun Wang, Ming-Yu Yu Liu, Andrew Tao, Guilin Liu, Bryan Catanzaro, and Jan Kautz. Few-shot
video-to-video synthesis. In NeurlPS, 2019.
[50] Ting-Chur Wang. Ming-Y. Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-
resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018.
[51] Ting- -Chun Wang. Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In CVPR, 2021.
[52] Yaohui Wang. Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate images via latent space navigation. arXiv22003, 09043, 2022.
[53] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004.
[54] Huawei Wei, Shuang Liang, and Yichen Wei. 3d dense face alignment via graph convolution networks.
905605562, 2019.
[55] Olivia Wiles, A Koepke, and Andrew Zisserman. X2face: A network for controlling face generation using images, audio, and pose codes. In ECCV, 2018.

---

[56]
Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky, Fast bi-layer neural synthesis of one-shot realistic head avatars. In ECCV, 2020.
[57] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky, Few-shot adversarial learning of realistic neural talking head models. In ICCV, 2019.
[58] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechuman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In CVPR, 2018.
[59] Peng Zhou, Lingxi Xic, Bingbing Ni, and Tian. Cips-3d: A 3d-aware generator of gans based on pixel synthesis. 10.092210 2021.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes], please refer to the supplemental
material.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes], please refer to the supplemental material.
(b) Did you include complete proofs of all theoretical results? [Yes], please refer to the supplemental material.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL.? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "JxgB5ywhRGi", "reviewer": "Reviewer_Tdy4", "summary": "This paper proposed a unified framework FNeVR for face animation, which combines 2D motion warping and 3D volume rendering.  Specifically, a 3D Face Volume Rendering module is designed for facial details rendering. Moreover, a pose editor is also incorporated to change the facial pose. The experimental results on popular benchmarks verify the superiority of FNeVR. ", "questions": "(1) It is better to conduct the ablation study to investigate the influence of two hyperparameters in formula (13). \n\n(2) From the presentations (Figure 4(b)) for the cross-identity reenactment experiment, the authors only study the rendering performance when the people in source images and driving videos belong to the same gender. Do other human attributes influence the performance of FNeVR, such as gender, age, haircut, and beard?  \n\n(3) In section 4.4, the pose editor is capable of facial pose editing with a slight Euler angle. Can LPE generate reliable results when a larger Euler angle is given? Of course, the given Euler angle is reasonable for human face rotations.  \n", "limitations": "In the cross-identity reenactment experiment, more experiments could be included to fully demonstrate the validity of the proposed framework, including cross-gender and cross-age reenactment experiments.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The manuscript is well written and clearly clarifies the objective and main idea in the research field. The proposed framework first applies to neural volume rendering for face animation and achieves better performance than SOTA methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7YWNkfM5yx", "reviewer": "Reviewer_n8nw", "summary": "This paper proposes to combine First-order motion, face vid2vid, and face volume rendering into one framework. The prior 3D knowledge from 3DMM is used to help the volume rendering procedure. The SPADE decoder from the open-sourced Face vid2vid is also used for the final prediction of the method. Experiments show that the method improves FOMM and Face vid2vid quantitatively.", "questions": "1. Why exactly is this paper's Orthogonal Adaptive Ray-Sampling better than the hierarchical (two-stage) volume sampling. What is the computational burden difference between the Orthogonal Adaptive Ray-Sampling and NeRF's rendering? And where is the adaptive part?\n\n2. The authors concatenate the volume rendering feature map $F_r$ and the warped feature $F_w$ for the final prediction. What is the performance if the wrapped feature is not sent into the SPADE decoder? Is $F_w$ involved in FOMM+FVR? Is $I_m$ the results of FOMM+FVR?\n\n3. The reviewer assumes that the Flops and memory consumption difference is led by the 3D landmark encoding procedure of Face vid2vid. Thus I am curious about the memory consumption of the models in the ablation studies (FOMM, FOMM + SPADE, FOMM+FVR). According to the paper, FNeVR would have a larger model size than these methods. Particularly, FOMM + SPADE is a combination of open-sourced code, which is the true baseline of the proposed method.\n\nActually, the reviewer encourages the authors to replace the SPADE decoder with a more lightweight one and adopts a variant of FOMM+FVR as the final result of this paper. As can be seen in Table 4, no obvious gain can be achieved by involving the SPADE decoder. Adding it involves more computation and makes the method less elegant.\n\n4. The ablation study without the loss function (4) is not given. To my understanding, this is very crucial in integrating 3D information.\n", "limitations": "I have mixed feelings about this paper. Involving a simple volume rendering into FOMM like reenactment pipeline is intuitive and nice. The whole formulation, including involving 3DMM as weak prior to density estimation is not entirely novel but reasonable. Overall, I think the novelty of this paper is sufficient.\n\n-- However, the video results of this paper are not impressive enough. Only a few videos are involved, which are not sufficient for reviewers to evaluate. No obvious improvements can be seen given the video results. Even re-training FOMM again might lead to the difference. The rotation results are also of poor quality. More importantly, no video comparisons with face vid2vid (S) are given. This is the most important weakness of this paper. \n\nFor now, I am a bit leaning towards rejection, but I will have a look at the rebuttal for the final decision. Why are the results of Face vid2vid not given? The authors should prepare a nicer video for future use of all kinds.\n\n-- The lack of certain analysis and ablations as stated in the Questions part. Moreover, this paper should also consider comparing with other reenact works such as MARR and PIRenderer. \n\n-- No ethical issues are discussed. As a matter of fact, FOMM itself has certain issues.\n\n-- Certain descriptions in this paper are confusing. For example:\n\na) In the abstract the authors write \"this is the first work to formulate the neural volume rendering of face animation with a new architecture design\". To the best of the reviewer's knowledge, every work that involves neural volume rendering into face animation would somehow propose a new architecture. It is not sure what the authors are claiming.\n\nIn the introduction, the authors claim \"this is the first work of neural volume rendering for face animation\", which is inappropriate. NeRF has been applied to face animation in methods such as NerFACE and ADNeRF.\n\nb) The authors use \"Model-free methods\" to indicate methods such as First-order Motion Model. The \"Model\" here might lead to misunderstandings as the authors even write \"Model-free models\". It is actually \"human-designed intermediate structural representation\" free. \n\n-- Certain papers mentioned in the review may not be cited. Please check the references.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++ The idea of involving a simple orthogonal volume rendering for face generation is actually interesting and reasonable.\n\n++ The way of formulating the volume rendering is nice.\n\n++ The 3D prior from 3DMM is nicely involved without interfering with the inference procedure.\n\n++ The quantitative results outperform many previous methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1LVLhWb5sy2", "reviewer": "Reviewer_rGfz", "summary": "This paper proposes a Face Neural Volume Rendering network for identity-preserving and photo-realistic face animation, which unifies the 2D motion warping and 3D volume rendering. Specifically, on top of 2d motion warping, it leverages the 3d face reconstruction for shape information and proposes a face volume rendering module to enhance facial details. It shows good quality and performance on several talking-head benchmarks. ", "questions": "- It will be good to provide speed evaluation at inference, compared to other SOTA methods. \n\n- Is N_\\sigma equal to 1? It is a bit confusing. \n\n- There are multiple papers that use single-pass uniform sampling when volume rendering feature maps or adaptively sample based on the geometric prior (3d face reconstruction in this case). it does not seem to be a novelty as claimed by the paper. Please clarify the contribution and do a comparison against single-stage uniform sampling or geometric-prior guided sampling. \n\n ", "limitations": "- As stated above, for the major contribution of the paper FVR, I do not see a strong design motivation to it given its computational cost. \n\n- It will be good to show failure examples as well. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- It well integrates 2d motion warping field, 3d face reconstruction and 3d volume rendering into a unified framework and shows the effect of each module with an ablation study. However, I think it would be good if the paper could provide more insights into the motivation to the design of each module. For example, the volume rendering is still typically expensive and why here it opts for volume rendering vs rasterization of neural texture (similar to Deferred Neural Rendering). \n\n- Extensive experiments and evaluations. Results are quantitatively and qualitatively good. It also provides analysis why the method is inferior to SOTA on some metrics (CSIM)\n\n- The paper is well presented. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept