FLAME-in-NeRF: Neural control of Radiance Fields for Free View Face Animation
Authors: ShahRukh Athar
Zhixin Shu
Dimitris Samaras
Venue: ICLR 2022 Submitted (2022)
Keywords: Neural Rendering
Facial Reanimation
3D Scene Priors

Abstract: This paper presents a neural rendering method for controllable portrait video synthesis.Recent advances in volumetric neural rendering, such as neural radiance fields (NeRF), have enabled the photorealistic novel view synthesis of static scenes with impressive results. However, modeling dynamic and controllable objects as part of a scene with such scene representations is still challenging. 
In this work, we design a system that enables 1) novel view synthesis for portrait video, of both the human subject and the scene they are in and 2) explicit control of the facial expressions through a low-dimensional expression representation. 
We represent the distribution of human facial expressions using the expression parameters of a 3D Morphable Model (3DMMs) and condition the NeRF volumetric function on them. 
Furthermore, we impose a spatial prior, brought by 3DMM fitting,  to guide the network to learn disentangled control for static scene appearance and dynamic facial actions. We show the effectiveness of our method on free view synthesis of portrait videos with expression controls. To train a scene, our method only requires a short video of a subject captured by a mobile device.

Under review as a conference paper at ICLR 2022
FLAME-in-NERE NEURAL CONTROL OF RADI- ANCE FIELDS FOR FREE VIEW FACE ANIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents neural rendering method for controllable portrait video synthesis. Recent advances volumetric neural rendering, such as neural radiance fields (NeRF), have enabled the photorealistic novel view synthesis of static scenes with impressive results. However, modeling dynamic and controllable objects as part of scene with such scene representations is still challenging. In this work, we design a system that enables 1) novel view synthesis for portrait video, of both the human subject and the scene they in and 2) explicit control of the facial expressions through a low-dimensional expression representation. We represent the distribution of human facial expressions using the expression parameters of a 3D Morphable Model (3DMMs) and condition the NeRF volumetric function on them. Furthermore, we impose spatial prior, brought by 3DMM fitting, guide the network to learn disentangled control for static scene appearance and dynamic facial actions. We show the effectiveness of our method on free view synthesis of portrait videos with expression controls. To train a scene, our method only requires a short video of a subject captured by a mobile device.
Driving
Render
Frame
Expression Conditioned Neural Radiance field Spatial Prior
Training Views
Reanimation novel expressions and views
Figure 1: FLAME-in-NeRF. Our method, FLAME-in-NeRI, models portrait videos (left) using an expression conditioned neural radiance field with a spatial prior (middle). Once trained, FLAME- in-NeRF can reanimate the subject and the scene present in the portrait video with arbitrary facial expressions and novel views.
INTRODUCTION
There is still no fully controllable human head model in natural scenes with arbitrary view synthesis, despite recent increased research. Such a model, in principle, allows for arbitrary control human head pose, facial expression, identity and viewing direction. Earliest attempts towards fully controllable human head model were the form of 3D Morphable Models (3DMMs) (Blanz et al., 1999). 3DMMs PCA-based space independently control face shape, facial expressions and

---

Under review as a conference paper at ICLR 2022
appearance and can be rendered in any view using standard graphics- based rendering techniques such as rasterization or ray-tracing. However, 3DMMs (Blanz et al., 1999) lack the ability to capture fine details of the human head such hair, skin details and accessories such as glasses. Additionally, only the 3D face can be viewed in novel directions, the scene itself cannot, as the mesh only models the human head and nothing else. In contrast, recent methods for novel view synthesis of static and dynamic scenes (Mildenhall et al., 2020; Park et al., 2021; Gafni et al., 2020; Chibane et al., 2021; Li et al., 2021b; Pumarola et al., 2021: Zhang et al., 2020b; Gao et al., 2020; Liu et al., 2020; Zhang et al., 2020; Xian et al., 2021) are able to generate high quality novel views of given captured scene but lack any control of the objects contained within the scene, including that of the human face and its various attributes.
et al., 2020a; Bemana et al., 2020; Martin-Brund
In this paper we introduce FLAME-in-NeRF, method that is capable of arbitrary facial expression control and novel view synthesis. We represent the whole scene as neural radiance field in manner similar to (Mildenhall et al., 2020; Park et al., 2021; Gafni et al., 2020) and lend it explicit expression controls using expression parameters derived from morphable model (Li et al., 2017). A simple way to utilize the expression parameters within a NeRF (Mildenhall et al., 2020), is to concatenate them to the color-MLP as done in Gafni et al. (2020). However, as shown in Fig 2, such naive-concatenation entangles the appearance of the scene to the expression parameters causing significant artefacts during reanimation. Standard volumetric representations, such as those used in vanilla NeRFs (Mildenhall et al., 2020), do not distinguish between various objects of the 3D scene since they lack any object semantics for scene they're 're representing. Consequently, naively concatenating controlling parameters, in this case expression parameters, leads to the appearance being entangled with the controlling parameters. In order avoid this entanglement we, once again, utilize the 3DMM to impose a spatial prior on the 3D scene. Such a prior ensures explicit disentanglement between appearance and expression parameters in parts of the 3D scene where we know the human head is not present, ensuring that the appearance of scene points that do not project on the human face are unaffected by changes in expression. Our model is trained on videos captured using a mobile phone, either by oneself or by someone else. Once trained, FLAME-ing -n-R allows for explicit control of both facial expression and viewing direction while capturing rich details of the scene along with fine details of the human head such as the hair, beard, teeth and accssorris such as glasses. Videos reanimated using our method maintain high fidelity to both the driving video in terms of facial expression manifestation and the original captured scene and human head.
In summary, our contributions in this paper are as follows: 1) We propose a first-of-a-king neural radiance field capable of explicit dynamic control on objects, such as the human face, within a static scene. 2) We experimentally demonstrate expression-appearance entanglement when reanimating portrait videos using standard neural radiance fields. 3) We introduce a spatial ray sampling prior that ensures explicit disentanglement between facial expressions and appearance and significantly improves quality of reanimation. 4) We develop a system capable of simultaneous control of facial expressions and viewing direction trained on short videos captured from a mobile phone.
2 RELATED WORK
FLAMB-in-NeRF is a method for arbitrary facial expression control and novel view synthesis of scenes captured in portrait videos. It is closely related to recent work on neural rendering and novel view synthesis, 3D face modeling, and controllable face generation.
Neural Scene Representations and Novel View Synthesis. FLAME-in-NeRR is related recent advances in neural rendering and novel view synthesis (Mildenhall et al., 2020; Park et al., 2021; Gafni et al., 2020; Yariv et al.. 2020; Chibane et al., 2021; Lassner & ZollhAfffer, 2021; Lombardi et al.. 2021: Li et al., 202 lb; Pumarola et al., 2021; Riegler & Koltun, 2021; Zhang et al., 2020b; Oechsle et al., 2021; Gao et al., 2020; Sitzmann et al., 2019; Liu al., 2020; Zhang et al., 2020a; Bemana et al., 2020; Martin-Brualte et al., 2020; Xian et al., 2021; Wizadwongsa et al., 2021). Notably, Neural Radiance Fields (NeRF) uses a Multi-Layer Perceptron (MLP). F, to learn a volumetric representation of a scene. Given 3D point and the direction from which the point is being viewed, F predicts its color and volume density. For any given camera pose, F is first evaluated densely enough throughout the scene using hierarchical volume sampling (Mildenhall et al., 2020), then volume rendering is used to render the final image. F is trained by minimizing the error between the predicted color of pixel and its ground truth value. While NeRF is able generate high quality and
2

---

Under review as a conference paper at ICLR 2022
Driving Frame
Driving Frane
GA-MB-G--NRRF Render Spatial Pelor
FLAME-Ht-NRRP
Difference Image
Render with Spatial Ray Prlor
Difference Image
Figure 2: Expresion-Appearance Entanglement: Why use the Spatial Ray Prior?: Here we demonstrate the necessity of using Spatial Ray Prior for the reanimation of portrait videos with arbitrary facial expressions and view control. On the left we have a model that does not use a Spatial Ray Prior and on the right, model that does. As results lower quality (e.g. on the lines of the brick wall) than the model with it. Further, the difference images show that, despite keeping the viewing direction constant, the model without the spatial prior changes the background appearance with changing expression. In contrast, the model with the spatial ray prior does do so the prior explicitly disentangles the expression and the appearance in regions of the 3D scene that do not project to the face. (Please watch the accompanying video Supplementary).
be seen, the model without the prior generates
photo-realistic images for novel view synthesis, it only designed for static scene and is unable to represent scene dynamics. Specifically designed for dynamic portrait video synthesis, approach not only models the dynamics of human faces, but also allow specific controls on the facial animation.
Dynamic Neural Scene Representations. Methods such (Li al., 2021b;a; Pumarola al., 2021; Xian et al., 2021) extend NeRF to dynamic scenes by providing as input a time component and along with imposing temporal constraints using scene flow (Li al., 2021b; Xian al., 2021) or by using a canonical frame (Pumarola et al., 2021). Similarly, Nerfies (Park et al., 2021) too work with dynamic scenes by mapping
canonical frame, however assumes that the movement is small. FLAME-in-NeRF, like (Park et al., 2021), models the portrait video by mapping to canonical frame and assumes that the head motion in the video small.
Controllable Face Generation. The advent of adversarial training (Goodfellow et al., 2014), cycle- consistency losses (Zhu et al., 2017) and powerful convolutional architectures (Isola et al., 2017) have made it possible to perform high quality facial expression editing just using 2D images (Shu et al., 2018; 2017; Athar et al., 2020a; Pumarola et al., 2020; Choi et al., 2018; 2020). However, since these methods are restricted images and often trained frontal datasets, their quality degrades as the pose of the input image changes. Methods such as (Kim et al., 2018; Doukas et al., 2021; Koujan al., 2020; Athar et al., 2020b) that use 3DMM reanimate faces. While being able to do so with great detail, they are unable to perform novel view synthesis as they do not model the geometry the whole scene. In (Gafni al., 2020), the authors use neural radiance fields to provide full control on the head. However, they do not model the background and is assumed to be static. In contrast, FLAME-in-NeRE provides full control over the facial expressions the person captured in the portrait video and has to ability to synthesize novel views. However, FLAME-in-NeRF does not provide control over the head-pose and we leave that to future work.
FLAME-iX-NERF
In this section, we describe our method, LLAME-in-NeRL that enables novel view synthesis of dynamic portrait scene and arbitrary control of facial expressions. We model dynamic portrait scene using a Neural Radiance Field with per-point deformation (Park et al., 2021) to allow for slight movement of the head. The deformation mechanism introduced in Park et al. (2021) deforms the
3

---

Under review as a conference paper at ICLR 2022
DBCA [tl]
Landuack Entting [14
Shape Parame
Kendering
Speiil Bay Pray
Exp
Dop Parame:
Volume Rendeetng
Appeureace Code:
Infermaction
F. F:Exp-Defermale NaRtP (Eiy
Code: aq
Figure 3: Overview of training FLAME-in-NeRE. First, we use DECA (Feng et al., 2021) and landmark fitting (Guo et al., 2020) to extract per-frame camera, shape, and expression parameters. Next, these parameters are used render silhouette of the FLAME model geometry. This silhouette used to provide spatial prior on ray sampling where only points that lie rays that intersect the silhouette are affected by the expression parameters. Finally, given the i-th frame, we shoot rays, we sample points along them and input these points to the Deformable NeRF, F, along with the i-th frame's 's expression parameters, deformation code and appearance code to render the final image.
rays of each frame to a canonical frame in order to ensure the rays that intersect are photometrically consistent. Facial expression dynamics controlled by per-frame FLAME expression parameters (Li et al., 2017) derived using (Feng et al., 2021) followed by standard landmarks fitting. In order ensure disentanglement between the view parameters and the expression parameters, spatial prior on ray sampling during training. Specifically, we use a silhouette rendering of the fitted FLAME face model and exclude the expression parameters for all points on rays that do not intersect the silhouette.
adopt
3.1 DEFORMABLE NEURAL RADIANCE FIELDS
neural radiance field (NeRF) is continuous function of, F: (f((),,(d)) + (c(x, d), (x)), that, given 3D point of a scene x and the viewing direction d (i.e the direction of the ray it is on) gives the color = (r, 9, b) and the density a. Here, P is multi-layer perceptron (MLP) and RR3 33+6m the positional encoding (Mildenhall et al., 2020) defined as 7xx)
(x.... sin(2"x), x2
where m is the total number of frequency bands and & E {0, ..., m - 1}. The expected color of íd, where is the camera center and d is the direction of the ray, is given by
camera ray r(t) the standard volumetric rendering equation
["' C(r) C(r)= ! T(t)a(r(t))(((())) d)dt, where T(t) cxp (- o(((ss))sss))
(1)
where, T(t) is the accumulated transmittance along the ray from t, to t. In practice, the integral in Eq. (1) is estimated using hierarchical volume sampling, refer the reader to (Mildenhall al., 2020) for details. Given multiple images of a scene, with their associated camera intrinsics and extrinsics, rays shot through each pixel of each image of the scene using mini-batches. The color of each ray is accumulated via volume rendering using Eq. (1) and the error w.r.t the ground truth pixel color is minimized follows:
min Σ (C) C
(2)
where, p is an indexing variable over all the pixels in the mini-batch, are the parameters of F and CCT is the ground truth pixel color.
NeRF, as defined above, is naturally designed for static scenes where the assumption is that two rays that intersect would have the same color. However, observed in (Park al., 2021), humans rarely
4

---

Under review as a conference paper at ICLR 2022
remain perfectly static during video capture process, more when they are speaking or performing facial expressions, thus vanilla NeRF training fails to model such dynamic scenes. In order to take into account for subtle movement of subjects in the capturing process, Park et al. (2021) proposed the Deformable NeRF architecture. In Park et al. (2021), ), 3D points, x's, captured in the i-th frame the video are deformed to canonical space via deformation function D,
x'. Here, D; is defined as (x, wi) = x' where is a per-frame latent deformation code. In practice, D(x, wi) is modeled using an MLP and its coordinate input also positionally encoded, we choose omit it for brevity. In addition to deformation code, wi, Park et al. (2021) I) also uses per- per-frame appearance code, Øi. thus the final radiance field for the i-th frame is follows:
F (D) wi) r(d), 6.) (c(x, ()),d)
In addition to the parameters of F' each wi and o; are also optimized through stochastic gradient descent. In practice, D(x,w), modeled as dense SE(3) field and we use coarse-to-fine regu- larization on the co-ordinate input, we refer the reader to Park et al. (2021) for details. While the aforementioned methods are able generate novel views (Mildenhall et al., 2020; Park et al., 2021) and handle small movement of objects in the scene (Park et al., 2021), they are still unable to control them.
3.2 EXPRESSION CONTROL IN DEFORMABLE NEURAL RADIANCE FIELDS
TTeMME-in-NeRF models changes of subject's facial expression changes in the color of 3D scene points. To this end, we condition the learning of deformable NeRF on a set of expression parameters provided by FLAME (Li et al., 2017). The expression conditioned deformable NeRF is defined as follows:
F: : (x,,,,, wx)) rdd), di, B.) (c(x, d), (x))
where, B; is the expression parameter of the i-th frame, D is the deformation function, w; and 6, the deformation code and appearance code of the i-th frame respectively. The expected colors of each pixel are then calculated using Eq. (1).
3.3 SPATIAL PRIOR FOR RAY SAMPLING
Ray that luside the FLAME ailboutte
Ray that is outside the FLAME silhoutie
Rén :{0) F: F: (x(D(x ((d), &1, (c(x)
Rout fo)- - F: (r
B,) + (c(x, d))
Figure 4: FLAME induced Prior. FLAME-in NeRF uses silhouette rendering of the FLAME model geometry (an overlay is shown above) to provide a spatial prior on rays shot through the 3D scene. All points that lie rays that intersect the silhouette, shown in red, are affected by the expression parameters. Other points, shown in green, have their expression parameters set to zero and are therefore unaffected by changes in expression.
When the radiance field is modeled as described in Eq. (4), there is nothing that prevents the appearance of point x, that does not project the face, to become dependent on the expression parameters 3r. In Sect 3.4 and Fig 2, we show this phenomena of background expression dependence in practice; a radiance field that changes the appearance of points the background as the view is kept constant but the expression changes. In order to counter this effect, we use a spatial prior on rays that's induced by the fitted FLAME face morphable model. First, we render the FLAME mesh giving binary silhouette image S; for each frame i. Next, we define two sets of points, let Rin X1, X,, be the set of points that lie on rays inside the FLAME silhouette of frame and R. = Xi..., Xm be the set of points that lie rays outside the FLAME silhouette of frame i (as shown in Fig 4), the radiance field now defined as:
F (D((Dxx (x))) (d), 6r, T(x)8)) d(xx))),(xx))
(5)
5

---

Under review as a conference paper at ICLR 2022
where, II(x) = if E R. and otherwise. This ensures that points that do not affect face pixels are not affected by facial expression changes. As can be seen in Fig 2, such a spatial ray prior effectively disentangles the appearance and expression and ensures that the background is unaffected by facial expression parameters.
Face region regularization. Since our method optimizes extrinsic camera parameters w.r. the FLAME 3DMM, we assume the head is static and has an identity mapping to the canonical frame. In order to prevent the deformation network, D. from moving the 3DMM, we penalize any deformation on points sampled from it as follows:
min| ((X - XADMMIl; Vi 6,00%
(6)
where, sp are the parameters of
3.4 ON THE NECESSITY OF A SPATIAL RAY PRIOR
In this section we demonstrate the necessity of the FLAME induced spatial prior on ray sampling as discussed in Sect 3.3. In Fig 2, we show the results of renanimating a portrait video with a constant view directions using our method with and without spatial ray prior. As can be seen in Fig 2, the results of the model without the prior are of lower quality than that of the model with it. Additionally, when we calculate the difference image of the reanimated frames generated by both methods we see that, despite the viewing direction remaining constant, the model without the prior changes the background. In stark contrast, and unsurprisingly. the model with the spatial ray prior does not do so and only makes changes around the face. The prior explicitly disentangles the expression parameters and the appearance regions of the scene which do not project to the face. Therefore, we see that the spatial prior is a necessary ingredient high quality portrait video reanimation.
4 RESULTS
In this section, we show results of facial expression control and novel view synthesis on portrait videos captured using a standard smartphone. Since our work the first to enable both explicit dynamic face controls and novel view synthesis for portrait videos, there is no existing work for an apple-to-apple comparison. We qualitatively and quantitatively compare our method to three baseline approaches that perform similar tasks: (1) Nerfies (Park et al., 2021): sfate-of-the-ann method using NeRF for novel view synthesis on static portrait scene, (2) NerFACE Gafni et al. (2021), a state-of-the-ar method using NeRF for face dynamics control without modeling the entire scene, and (3) First Order Motion Model (FOMM) (Siarohin et al., 2019), a seneral-purponn image reanimation pipeline. We use the appearance code of the first frame for FLAME-ii-NNeRR Nerfies (Park et al.. 2021) and NerFACE (Gafni et al., 2021) to perform the reanimation. For TLAME-in-NNeR and Nerfies (Park et al., 2021), we use the deformation code from the first frame. Full videos of the reanimation can be found in the supplementary material. We strongly urge the readers to check them out to see FLAME-in-NeRF performing at its best.
Training Data Capture and Training details. The training data captured using iPhone XR smartphone for all the experiments in the paper. We ask the subject to enact a wide range of expressions and speech while trying to keep their head still the camera is panned around them. Alternatively, the subject can self-capture a video (a selfie video) as they speak. We calculate the expression and shape parameters of each frames in the videos using DECA (Feng et al., 2021). Next, we compute the extrinsic camera parameters via standard landmark fitting using landmarks predicted by (Guo al., 2020). All training videos are about just 25 seconds long (~ 700 frames). Due to compute restrictions. the video is down-sampled and the models are trained at 256x256 resolution. We use coarse-to- fine regularization (Park al., 2021) to train the deformation network D(x, w:). Please find full details of each experiment in the Section 3. of the supplementary.
4.1 EVALUATION ON VALIDATION DATA
We evaluate FLAME-in-Nelee Nerfies (Park et al., 2021). NerFACE (Gafni et al., 2021) and FOMM (Siarohín al., 2019) held out images. Since FeRR and Nerfies (Park et al., 2021)

---

Under review as a conference paper at ICLR 2022
Driving Frame
FLAME FLAME-in-NeiE Nee- Render
Nerfies Render
Ner FACE CE
Render
Figure 5: Qualitative evaluation by reanimating subject 1: Here we show the results of reanimat- ing Subject I using our method, Nerfies (Park et al., 2021) and NerFACE Gafni et al. (2021) with both expression and view changes. The first row shows the driving frame, the second row shows the results of our method, the third shows the results of Nerfies (Park et al., 2021) and the fourth row shows the results of NerFACE (Gafni et al., 2021). As can be seen columns 1-4, Nerfies (Park al., 2021) is unable to model the facial expressions correctly leading to artefacts on the face while NerFACE (Gafni et al., 2021) is unable to model the 3D scene leading to artefates on the background. We that method, in contrast with Nerfies and NerFACE, generates high-quality reanimation results with high fidelity to the driving expression and consistency across views.
per-frame deformation and appearance code, W; and respectively, cannot perform direct comparison with the ground truth image as it may have a different deformation to the canonical frame than the first frame (which is what deformation of a given validation image to the canonical frame by optimizing the deformation code follows
default for reanimation). Therefore, first find the
min [Cp(w; x, d, (o) CGT
where, (w x, d, 0, ④o) is the predicted color at pixel p generated using Eq. (1) and Eq. (5), do the appearance code of the first frame, are the parameters of as defined in Eq. (5) and CCT is the ground-truth pixel value. Note, we only optimize all other parameters of the radiance field are kept fixed. We optimize Eq. (7) for 2000 epochs which we observe to be more than enough to find the loss plateau. Once the optimization finishes, we report the final MSE, PSNR, LPIPS and Face MSE i. the MSE only over the face region. We use no such optimization with NerFACE (Gafni al., 2021) since it does not have deformation module and FOMM (Siarohin et al., 2019) since is a image-based method. As can be seen in Table 1, our method outperforms Nerfies (Park et al., 2021), NerFACE (Gafni al., 2021) and FOMM (Siarohin et al., 2019) validation images when
7

---

Under review as a conference paper at ICLR 2022
Driving Frame
FLAME- Ne-N Render
Nerfies
Render
Ner CE Render
Figure 6: Qualitative evaluation by reanimating subject 3: Here show the results of reanimat- ing Subject 3 using our method, Nerfies (Park et al., 2021) and NerFACE (Gafni et al., 2021) with both expression and view changes. The first shows the driving frame, the second row shows the results of our method, the third shows the results of Nerfies (Park et al., 2021) and the fourth row shows the results of NerFACE (Gafni al., 2021). As can be seen in columns 1-4, Nerfies is unable to model the facial expressions correctly leading to artefacts on the face while NerFACE is unable to model the scene leading artefates on the background. We see that TTAMF--in-NeRR in contrast with Nerfies and NerFACE, generates high-quality reanimation results with high fidelity to the driving expression and consistency across views.
averaged over all subjects. Since FLAME-in-NeRE, portrait videos with changing facial expressions, Nerfies unable to learn the topological changes of the mouth, often regressing view-dependent changes. NerFACE on the hand, lacking a deformation module and a spatial prior, unable to model the dynamism of the scene and generates significant artefacts in the background 3D scene (see the fourth row of Fig 5 and Fig 6). FOMM (Siarohin et al., 2019), being an image based method, is unable to model novel views. contrast to Nerfies (Park al., 2021), NerFACE Gafni et al. (2021) and FOMM (Siarohin et al., 2019), FLAME-in-NeRF, due to the use of spatial prior along with expression conditioning and deformation module, is able to model both facial expressions and the full 3D scene with high fidelity, thus giving better reconstructions.
Nerfies and NerFACE trained dynamic
'mean' expression (see the third of Fig 5 and Fig 6) with small
4.2 REANIMATION WITH ARBITRARY EXPRESSION CONTROL AND NOVEL VIEW SYNTHESIS
In this section we show results of reanimating Neural Radiance Fields using both FLAME- -M- and Nerfies (Park et al., 2021) using expression parameters as the driving parameters. Per-frame expression parameters from the driving video are extracted using DECA (Feng et al., 2021) and are given input Me. as follows:
8

---

Under review as a conference paper at ICLR 2022
Subject
Method
MSE (4)
PSNR (1)
LPIPS (4)
Face MSE (L)
FLAME-in-Nelee -n--
2.045e-3
26.89
0.185
4.708e-4
Subject 1
Nerfies (Park ct al., 2021)
2.82c-3
25.49
0.205
8.140c-4
NerFACE (Gafni et al., 2021)
8.68e-3
20.61
0.326
1.166e-3
FOMM (Starohin et al., 2019)
2.631e-2
15.79
0.474
4.043e-3
FLAME-In-NeRF
1.255e-3
29.01
0.147
L688e-4
Subject 2
Nerfies (Park al., 2021)
2.05c-3
26.06
0.157
2.806c-4
NerFACE (Gafni et al., 2021)
4.34e-3
23.62
0.233
3.171e-4
FOMM (Siarohin et al., 2019)
3,221c-2
14.91
0,500
6.909c-3
FLAMEEin-NNRF
5.71e-3
22.43
0.223
2.84c-4
Subject 3
Nerfies (Park al., 2021)
4.68e-3
23.29
0.277
7.85e-3
NerFACE (Gafni et al., 2021)
6.347e-3
21.97
0.285
3.38e-4
FOMM (Siarohin et al., 2019)
5.54le-2
12.56
0.476
7.496e-3
FLAME-in-NeRF
2.712e-3
25.67
0.236
4.35e-4
Subject 4
Nerfies (Park ct al., 2021)
5.9c-3
22.29
0.384
1.187c-3
NerFACE (Gafni al., 2021)
4.556e-3
23.41
0.338
5.76e-4
FOMM (Siarohin et al., 2019)
3.758e-2
14.25
0,505
8.054c-3
FLAMME-in-NeRF
2.93e-3
25.97
0.197
3.39e-4
Average
Nerfies (Park al., 2021)
3.862e-3
24.28
0.255
2.532e-3
NerFACE (Gafni et al., 2021)
5.98e-3
22.40
0.295
1.359e-3
FOMM (Siarohin et al., 2019)
3.787e-2
14.37
0.488
6.625e-3
Table 1: Quantitative results of Subject 1,2.3 and 4 on validation data. Our results better than Nerfies (Park et al., 2021), NerFACE (Gafni et al., 2021) and FOMM (Siarohin et al., 2019) on most metrics across all subjects.
Method
MSE (4)
PSNR (1)
LPIPS ())
Face MSE (1)
FLAME-in-NeRF
2.6e-3
25.83
0.223
3.48c-4
FLAME-in-NeRF w/o Spatial Prior
2.83e-3
25.40
0.377
4.75e-4
Table 2: Here we show a quantitative comparison between using the spatial ray prior and not. As can be seen, using the spatial prior gives significantly better results, especially with respect to perceptual quality measured by LPIPS.
F: (x(,,,, r(d), dv, ((()))) (c(x, ox))
where, 33.rive are the expression parameters derived from the driving video. We refer the reader to Eq. (5) for the definitions of the other variables. Since Nerfies (Park et al., 2021) not take as input expression parameters, it's forward pass remains the same as in Eq. (3). As we drive both methods, we also simultaneously change the viewing direction. Fig and Fig shows the results of both FLAME-in-NeRF, Nerfics (Park et al., 2021) and NerFACE (Gafni et al., 2021) with changing the expression parameters and view on two different subjects. As one can see, FLAME-in-NeRE captures the driving expression with high fidelity and is able do regardless of viewing direction and without compromising the reconstruction of the background 3D scene. Nerfies (Park et al., 2021), lacking the explicit conditioning expression parameters, is unable generate images with the correct facial expression. Only small changes in expression can be observed due to view changes. NerFACE (Gafni et al., 2021), lacking a deformation model, cannot model a dynamic scene as can be seen by poor reconstruction of the background in Fig 5 and Fig 6.
5 CONCLUSION
In this paper we have presented FLAME-in-NeRF- novel method capable of arbitrary facial expression control and novel view synthesis for portrait videos. FLAME-in-NeRF uses an expression- conditioned neural radiance field along with spatial prior to generate images with high fidelity to both the subject in the original portrait video and the provided expression parameters, any viewing direction. FLAME-in-NeRR and glasses and reproduce them with high fidelity as the video is driven. However, the problem of controllable human head models with novel view synthesis is far from solved. FLAME-in-NeRF
also able to model details of the subject's face such as hair

---

Under review as a conference paper at ICLR 2022
is unable model large head movements and requires the subject in the portrait video to remain relatively still. We hope to address this in future work. These are exciting times to a part of ML/CV/CG communities neural methods push the state of the art in head control and novel view synthesis with broader impacts in entertainment, education and HCI.
REFERENCES
Athar, Z Shu, and D Samaras. Self-supervised deformation modeling for facial expression editing. In JEEE FG, 2020a.
ShahRukh Athar, Albert Pumarola, Francese Moreno -oggue, and Dimitris Samaras. Facedet3d: Facial expressions with 3d geometric detail prediction. arXiv preprint arXiv:2012.07999, 2020b.
Mojtaba Bemana, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. X-fields: Implicit neural view-, and time-image interpolation. 2020. 2
Volker Blanz, Thomas Vetter, al. A morphable model for the synthesis of 3d faces. 1999. 1,2
Julian Chibane, Aayush Bansal, Veriea Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In CVPR, 2021. 2
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In CVPR, 2018. 3
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
M. Doukas, Mohammad Rami Koujan, V. Sharmanska, A. Roussos, and S. Zafeiriou. HeadZhead++: Deep facial attributes re-targeting. IEEE Transactions on Biometrics, Behavior, and Identity Science, 3:31-43, 2021.
Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning an animatable detailed 3D face model from wh-the-wild images. volume 40, 2021. URL htttps://doi.or///11// 1145/ 3450626. 3459 3499936. 4, 6,
Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction, 2020. 3,
Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In CVPR, June 2021. 6, 7, 9
Chen Gao, Yichang Shih, Wei-Sheng Lai, Chia-Kat Kai Liang. and Jia Ban Huang. Portrait neural radiance fields from a single image. arXiv preprint arXiv:20/2.05900 2020. 2
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Li. Towards fast, accurate and stable 3d dense face alignment. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 4,
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt. Deep video portraits. ACM TOG, 2018. 3
M. Koujan, M. Doukas, A. Roussos, and S. Zafeiriou. HeadZhead: Video-based neural head synthesis. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) (FG), pp. 319-326, Los Alamitos, CA. USA, may 2020. IEEE Computer Society. doi: URL https: http://i.....eoneeeeeeeee wew.cmptters.scctt.............. wowp:/cc.iee...o....oooo
10. FLL.L///GG77882200000000
11
10

---

Under review as a conference paper at ICLR 2022
Christoph Lassner and Michael ZollhAffer. Pulsar: Efficient Sahere-based neural rendering. In CVPR, 2021.
Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4D scans. ACM Transactions Graphics, (Proc. SIGGRAPH Asia), 36(6), 2017. URL https: /ttp:s://dii. wttp///..115/11300000
3130813. 2, 4,5
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, and Zhaoyang Lv. Neural 3d video synthesis, 2021a. 3
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021b. 2,
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. 2020. 2
Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering, 2021.
Ricardo Martin- Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. arXiv:2008. 022668 2020. 2
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. 2020. 2. 4,
Michael Oechsle, Songyou Peng. and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. arXiv preprint arXi::2104.1007 2021. 2
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan Goldman, Steven M. Seitz, and Ricardo Barain-Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021. 2, 4, 5, 6, 7,
Albert Pumarola, Antonio Agudo, Aleix M Martinez, Alberto Sanfeliu, and Francese Moreno -oguer. Ganimation: One-shot anatomically consistent facial animation. International Journal of Computer Vision, 128(3))69--713, 2020.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. CVPR, 2021. 2,
Gemnot Riegler and Vladlen Koltun. Stable view synthesis. CVPR, 2021.
Z. Shu, E. Yumer, S. Hadap, K. Sunkavalli, Shechtman, and Samaras. Neural face editing with intrinsic image disentangling. In CVPR, 2017. 3
Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris Samaras, Nikos Paragios, and lasonas Kokkinos. Deforming autoencoders: Unsupervised disentangling of shape and appearance. In ECCV, 2018. 3
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. 2019. 6, 7,
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-structure-aware neural scene representations. 2019. 2
Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai. and Supasorn Suwajanakom. Nex: Real-time view synthesis with neural basis expansion. In CVPR, 2021. 2
Wengi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021. 3,
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NIPS, 33, 2020. 2
11

---

Under review as a conference paper at ICLR 2022
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arX1v:2010.07492, 2020a. 2
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv:2010.07492, 2020b. 2
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017. 3
12


Reviews:
[{"review_id": "fRKq8QkelJ8", "reviewer": "Reviewer_G8XN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The submitted paper demonstrates the interesting use case of controlling video synthesis with a driving frame/video as input. \n\nWeaknesses:\n1. Missing important baseline/citation: HyperNeRF is an improvement over Nerfies and should be considered as a baseline method: \"HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields\"\n2. Quality of synthesized videos: the synthesized videos are low-resolution, blurry, and there are noticeable floaters around human subjects. I don't consider the quality to be on par with Nerfies or HyperNeRF.\n3. Limited novelty: the proposed method is a relatively straightforward extension of Nerfies but with masks and facial expressions as the latent code.\n4. Limited evaluation: there are only 4 subjects being used as evaluation data", "summary_of_the_paper": "The submitted paper focuses on novel view synthesis from portrait video with controllable facial expressions. The authors separate the modeling of the static background from the dynamic foreground by segmenting out the faces (using landmark detection and fitting). The faces are modeled with Nerfies with expression parameters as latent codes, while the static background is essentially modeled with a standard NeRF without conditioning on per-frame expressions. The proposed method was evaluated on 4 subjects with data collected from a phone. The authors also demonstrate an application of the proposed method where videos can be synthesized with consistent facial expressions as a driving video as input.  ", "summary_of_the_review": "My current recommendation is \"3: reject, not good enough\". There is limited novelty with important baseline missing (HyperNeRF). The synthesized videos are also blurry with artifacts. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "c0nEn0PhJaD", "reviewer": "Reviewer_z2xe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper presents a straightforward but sensible method that benefits the best of both worlds: it utilizes the very strong prior of human faces by using a parametric face model and meanwhile uses the powerful NeRF to handle complex appearance, including that of the background. The quantitative experiments suggest that the proposed method outperforms the baseline methods. Qualitatively, it achieves something that prior methods don't: It allows for expression control that Nerfies don't support, and it handles the background better than NerFACE because it disentangles expressions from backgrounds.\n\nNow the weaknesses:\n\nSince the method relies heavily on how well DECA and landmark fitting work, it should be discussed whether there is any measurement taken to recover from DECA's and/or landmark fitting's mistakes. If I understand correctly, there is no such error recovery scheme in the model design. This seems like a major flaw to me. Just to name one potential failure that DECA may suffer from: The camera extrinsics from DECA may not be accurate, and most of us would agree NeRF-like models rely on good camera poses to yield sharp rendering.\n\nThe video results look rather blurry to me maybe partially because the model was trained at 256x256. I think for the method to be really useful, the authors should explore why such blurriness still exists with the ray sampling masking scheme and the deformation fields already in place. There are also flickering artifacts that \"morph\" the person's head when the expressions are driven. \n\nI believe there is complex dynamics among the expression parameter beta_i, the deformation code omega_i, and the appearance code phi_i: For instance, how does beta_i interact with omega_i? Intuitively, when the expression changes, the deformation field will have to adjust itself to work with the new expression. This dynamics, or rather this entanglement, doesn't sound desirable to me. This deserves at least some discussions, if not experiments.\n\nEquation 6: Why wouldn't the model get into the trivial solution of having a no-op deformation field D? That would give you a perfect zero loss for Equation 6. \n\nIt's strange to have 3.4 ON THE NECESSITY OF A SPATIAL RAY PRIOR as a separate section than 3.3.\n\nThere doesn't seem to be any quantitative ablation study. Although we can see qualitatively the disentanglement, it'd be nice to confirm this with numbers.\n\nFig. 5 shows really similar-quality results between the proposed method and NerFACE in terms of just the reenactment, although I agree the proposed method does perform better overall with fewer artifacts. That said, this observation, together with the fact that the reenacted expressions don't resemble those in the driving frames (e.g., the expressions in Driving Frames 1 & 4 are very different, but they both lead to similar reenacted expressions), casts it into question whether the proposed method controls the expressions well. \n\nMissing references:\n- In the human domain, the authors should cite:\n    Neural Light Transport for Relighting and View Synthesis\n  which does perform view synthesis of human subjects using a \"neural rendering\" approach.\n- Furthermore, this paper:\n    Deep Relightable Textures Volumetric Performance Capture with Neural Rendering\n  should also be cited, since it does handle moving subjects.\n- I think there should be a section on \"Parametric Face Models\" that are dedicated to 3DMM, FLAME, etc. since half of the model name \"Flame-in-NeRF\" lies in that domain. The current Related Work sections focus on just the \"NeRF half.\"\n\nMinor nits:\n- Page 2: \"Lassner & ZollhÃ¶fer\"\n- Page 3, grammatical errors: \"Nerfies (Park et al., 2021) too work\", and then \"frame, however it\"\n- Page 5: \"are able generate\" & \"this phenomena of\"\n- Fig. 4 seems unnecessarily verbose since Eq. 5 already says the same thing very clearly.", "summary_of_the_paper": "\nThis paper proposes a method that marries parametric face models and neural radiance fields to achieve view synthesis of a portrait that provides expression control. Another view of this work is extending Park et al.'s Nerfies to support expression control and driving. \n\nThe paper makes three contributions: Firstly, it proposes a dynamic NeRF model that supports certain explicit control. Secondly, the proposed method disentangles the facial expression from the appearance of the background with a simple masking scheme dubbed as spatial ray sampling prior. Finally, the method enables the simultaneous control of viewpoints and facial expressions.", "summary_of_the_review": "\nThis is a paper that clearly explains a straightforward but sensible approach. I judge it to be slightly below the acceptance bar mainly because (1) the method heavily depends on the success of two non-trivial preprocessing steps and has no way to recover from their mistakes, and (2) the complex dynamics among the expression parameters and deformation fields remains unexplored.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CyS5dcn4EbM", "reviewer": "Reviewer_A3Tv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ Unlike previous works, the proposed framework enables explicit control of the human facial expression while synthesizing an image from a novel viewpoint.\n+ The writing is clear and easy to follow.\n\n\nWeakness:\n- Lack of novelty. The key idea, i.e., combining foreground masks to remove the artifacts from the background, is not new. Separate handling of foreground from background is a common practice for dynamic scene novel view synthesis, and many recent methods do not even require the foreground masks for modeling dynamic scenes (they jointly model the foreground region prediction module, e.g., Tretschk et al. 2021).\n\n- Lack of controllability. 1) The proposed method cannot handle the headpose. While this paper defers this problem to a future work, a previous work (e.g., Gafni et al. ICCV 2021) is already able to control both facial expression and headpose. Why is it not possible to condition the headpose parameters in the NeRF beyond the facial expression similar to [Gafni et al. ICCV 2021]? 2) Even for controlling facial expression, it is highly limited to the mouth region only. From overall qualitative results and demo video, it is not clear the method indeed can handle overall facial expression including eyes, nose, and wrinkle details, and the diversity in mouth shape that the model can deliver is significantly limited. \n\n- Low quality. The results made by the proposed method are of quite low quality. 1)  Low resolution: While many previous works introduce high-quality view synthesis results with high-resolution (512x512 or more), this paper shows low resolution results (256x256) for some reasons. Simply saying the problem of resources is not a convincing argument since many existing works already proved the feasibility of high resolution image synthesis using implicit function. Due to this low-resolution nature, many high-frequency details (e.g., facial wrinkles), which are the key to enabling photorealistic face image synthesis, are washed out. 2)  In many cases, the conditioning facial expressions do not match that from the synthesized image. From the demo video, while mouth opening or closing are somehow synchronized with conditioning videos, there exists a mismatch in the style of the detailed mouth shape.  ", "summary_of_the_paper": "This paper introduces a method for novel view synthesis of a dynamic scene with a person in varying facial expression. To enable the controllability of facial expression, a parametric 3D face model is combined with existing neural radiance field-based approaches. To overcome background artifacts (e.g., distortion) while animating, the spatial face prior is added in a form of binary mask. The method is evaluated on four customized videos. \n", "summary_of_the_review": "While this paper add a controllability in the facial expression for a novel view synthesis by combining the techniques from previous works, the idea is not novel (which shows many overlaps with many existing NeRF based approaches), and the image synthesis quality is not convincing (low resolution without sufficient high-frequency details, notable jitters, and mismatch between the synthesized facial expression and conditioning ones) when compared to how existing methods have demonstrated. Although the proposed method performs better in several customized scenarios, but those testing examples do not cover diverse scenes that can truly validate the performance. Based on the current demonstration, the ability to control the facial expression is highly limited to a few number of mouth shapes (it's more like mouth control, not facial expression control). Considering the novelty and quality in the context of existing methods, the initial rating is reject.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-tRp4Zoqvh", "reviewer": "Reviewer_SS8Y", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n**Strenghts:**:\n- Writing quality. The paper is well-written and is easy to follow.\n- Method. The general idea behind the method - to use a spatial disentanglement mechanism to separate effects of conditioning variables - is reasonable and is definitely worth investigating.\n- Experimental evaluation. Authors provide a comprehensive comparison to very recent baselines, and show that both qualitatively and quantitatively their method performs slightly better.\n\n**Weaknesses:**\n\n- Method limitations. Authors do hint at a very important issue which is common for animatable models - which is entanglement among different factors of variation among the control signals.  However, the scope of the solution that they propose seems quite limited: effectively the only separation that one gets is between the background and foreground. This means that there is no mechanism for the model to separate independent expression parameters, and even the disentanglement between the viewpoint and expression is not fully guaranteed. It would be interesting to see animation on sequences where a single expression parameter of the time is varied while others are being fixed, and similarly fixing expression vector while varying the view to see if there are any spurious correlations, in order to better understand the scope of this issue. \n\n- Related work - geometrical prior. The idea of using an explicit geometrical or spatial prior to improve generalization is not particularly novel, even if one only considers the context of neural radiance fields. In particular, most of the work on neural rendering for bodies (NeuralBody, Neural Actor) use SMPL or similar model, which acts as a 3DMM. Although those works do not model the background, ideas presented in those works will likely generalize better as they also encourage spatial disentanglement between different parts of the articulated object. Very similar ideas can be trivially extended to faces data. Moreover, the assumption that the appearance of the background is independent from the expression is not correct - e.g. shadows depend on those? \n\n- Related work - NerFace. Proposed method is very similar to the existing method - NerFACE. The only difference seems to be in the way the two methods treat background? If that is the main motivation for this work, ideally it should be made more clear. It would also be great if authors describe what would be a difference between their method and simply training a separate NERF model for the background (using pre-computed segmentation masks, and simple ad-hoc blending at inference time). I can see that training a joint model might be beneficial, but ideally this should be demonstrated empirically. \n\n- Experimental evaluation. There are no video materials that compare proposed method to the baselines, only static comparisons. I would also encourage authors to provide failure cases of their method to have a better idea about the limitations. As mentioned earlier, varying independent control variables - view, individual expression variables - can provide a way to qualitatively evaluate how animatable the model actually is.\n\n**Misc:**\n- P5: able to generate\n- P8: artefatcs", "summary_of_the_paper": "This paper introduces a method for creating animatable portraits within a static scene based on dynamic neural radiance fields. \nThe main idea is to fit a 3DMM-like model to a set of portrait frames and use it to rasterize a binary mask which is used to enforce distanglement between the foreground and background when training a dynamic NERF. Both qualitative and quantitative results seem to be convincing.", "summary_of_the_review": "Although qualitative results of the paper seem convincing, the method itself is very similar to an existing one (NerFACE), while the main claimed novelty - the spatial ray prior - seems to be equivalent to training a separate background model with pre-computed segmentation masks. Thus I am leaning more towards rejecting this paper, but I am eager to reconsider my rating if authors address the concerns raised above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received 4 ratings, all below the acceptance threshold. The reviewers expressed concerns around overall novelty of contributions and quality of produced results, and also pointed out lack of comparisons with some prior works and gaps in empirical evaluation. The authors responded to most of these comments, but did not convince the reviewers to upgrade their ratings.
The final recommendation is therefore to reject.
Decision: Reject