Year: 2021

Abstract:
This paper presents a neural rendering method for controllable portrait video synthesis. Recent advances in volumetric neural rendering, such as neural radiance fields (NeRF), have enabled the photorealistic novel view synthesis of static scenes with impressive results. However, modeling dynamic and controllable objects as part of a scene with such scene representations is still challenging. In this work, we design a system that enables 1) novel view synthesis for portrait video, of both the human subject and the scene they are in and 2) explicit control of the facial expressions through a low-dimensional expression representation. We represent the distribution of human facial expressions using the expression parameters of a 3D Morphable Model (3DMMs) and condition the NeRF volumetric function on them. Furthermore, we impose a spatial prior, brought by 3DMM fitting, to guide the network to learn disentangled control for static scene appearance and dynamic facial actions. We show the effectiveness of our method on free view synthesis of portrait videos with expression controls. To train a scene, our method only requires a short video of a subject captured by a mobile device. Training Views Expression Conditioned Neural Radiance field with a Spatial Prior Reanimation with novel expressions and views Render Driving Frame Figure 1: FLAME-in-NeRF. Our method, FLAME-in-NeRF, models portrait videos (left) using an expression conditioned neural radiance field with a spatial prior (middle). Once trained, FLAMEin-NeRF can reanimate the subject and the scene present in the portrait video with arbitrary facial expressions and novel views.

heading: 1 INTRODUCTION
text: There is still no fully controllable human head model in natural scenes with arbitrary view synthesis, despite recent increased research. Such a model, in principle, allows for arbitrary control of human head pose, facial expression, identity and viewing direction. Earliest attempts towards a fully controllable human head model were in the form of 3D Morphable Models (3DMMs) (Blanz et al., 1999). 3DMMs use a PCA-based space to independently control face shape, facial expressions and
appearance and can be rendered in any view using standard graphics-based rendering techniques such as rasterization or ray-tracing. However, 3DMMs (Blanz et al., 1999) lack the ability to capture fine details of the human head such as hair, skin details and accessories such as glasses. Additionally, only the 3D face can be viewed in novel directions, the scene itself cannot, as the mesh only models the human head and nothing else. In contrast, recent methods for novel view synthesis of static and dynamic scenes (Mildenhall et al., 2020; Park et al., 2021; Gafni et al., 2020; Chibane et al., 2021; Li et al., 2021b; Pumarola et al., 2021; Zhang et al., 2020b; Gao et al., 2020; Liu et al., 2020; Zhang et al., 2020a; Bemana et al., 2020; Martin-Brualla et al., 2020; Xian et al., 2021) are able to generate high quality novel views of a given captured scene but lack any control of the objects contained within the scene, including that of the human face and its various attributes.
In this paper we introduce FLAME-in-NeRF, a method that is capable of arbitrary facial expression control and novel view synthesis. We represent the whole scene as a neural radiance field in a manner similar to (Mildenhall et al., 2020; Park et al., 2021; Gafni et al., 2020) and lend it explicit expression controls using expression parameters derived from a morphable model (Li et al., 2017). A simple way to utilize the expression parameters within a NeRF (Mildenhall et al., 2020), is to concatenate them to the color-MLP as done in Gafni et al. (2020). However, as shown in Fig 2, such a naive-concatenation entangles the appearance of the scene to the expression parameters causing significant artefacts during reanimation. Standard volumetric representations, such as those used in vanilla NeRFs (Mildenhall et al., 2020), do not distinguish between various objects of the 3D scene since they lack any object semantics for scene they’re representing. Consequently, naively concatenating controlling parameters, in this case expression parameters, leads to the appearance being entangled with the controlling parameters. In order to avoid this entanglement we, once again, utilize the 3DMM to impose a spatial prior on the 3D scene. Such a prior ensures explicit disentanglement between appearance and expression parameters in parts of the 3D scene where we know the human head is not present, ensuring that the appearance of scene points that do not project on the human face are unaffected by changes in expression. Our model is trained on videos captured using a mobile phone, either by oneself or by someone else. Once trained, FLAME-in-NeRF allows for explicit control of both facial expression and viewing direction while capturing rich details of the scene along with fine details of the human head such as the hair, beard, teeth and accessories such as glasses. Videos reanimated using our method maintain high fidelity to both the driving video in terms of facial expression manifestation and the original captured scene and human head.
In summary, our contributions in this paper are as follows: 1) We propose a first-of-a-kind neural radiance field capable of explicit dynamic control on objects, such as the human face, within a static scene. 2) We experimentally demonstrate expression-appearance entanglement when reanimating portrait videos using standard neural radiance fields. 3) We introduce a spatial ray sampling prior that ensures explicit disentanglement between facial expressions and appearance and significantly improves quality of reanimation. 4) We develop a system capable of simultaneous control of facial expressions and viewing direction trained on short videos captured from a mobile phone.

heading: 2 RELATED WORK
text: FLAME-in-NeRF is a method for arbitrary facial expression control and novel view synthesis of scenes captured in portrait videos. It is closely related to recent work on neural rendering and novel view synthesis, 3D face modeling, and controllable face generation.
Neural Scene Representations and Novel View Synthesis. FLAME-in-NeRF is related to recent advances in neural rendering and novel view synthesis (Mildenhall et al., 2020; Park et al., 2021; Gafni et al., 2020; Yariv et al., 2020; Chibane et al., 2021; Lassner & ZollhÃ¶fer, 2021; Lombardi et al., 2021; Li et al., 2021b; Pumarola et al., 2021; Riegler & Koltun, 2021; Zhang et al., 2020b; Oechsle et al., 2021; Gao et al., 2020; Sitzmann et al., 2019; Liu et al., 2020; Zhang et al., 2020a; Bemana et al., 2020; Martin-Brualla et al., 2020; Xian et al., 2021; Wizadwongsa et al., 2021). Notably, Neural Radiance Fields (NeRF) uses a Multi-Layer Perceptron (MLP), F , to learn a volumetric representation of a scene. Given a 3D point and the direction from which the point is being viewed, F predicts its color and volume density. For any given camera pose, F is first evaluated densely enough throughout the scene using hierarchical volume sampling (Mildenhall et al., 2020), then volume rendering is used to render the final image. F is trained by minimizing the error between the predicted color of a pixel and its ground truth value. While NeRF is able to generate high quality and
photo-realistic images for novel view synthesis, it is only designed for a static scene and is unable to represent scene dynamics. Specifically designed for dynamic portrait video synthesis, our approach not only models the dynamics of human faces, but also allow specific controls on the facial animation.
Dynamic Neural Scene Representations. Methods such as (Li et al., 2021b;a; Pumarola et al., 2021; Xian et al., 2021) extend NeRF to dynamic scenes by providing as input a time component and along with it imposing temporal constraints using scene flow (Li et al., 2021b; Xian et al., 2021) or by using a canonical frame (Pumarola et al., 2021). Similarly, Nerfies (Park et al., 2021) too work with dynamic scenes by mapping to a canonical frame, however it assumes that the movement is small. FLAME-in-NeRF, like (Park et al., 2021), models the portrait video by mapping to a canonical frame and assumes that the head motion in the video is small.
Controllable Face Generation. The advent of adversarial training (Goodfellow et al., 2014), cycleconsistency losses (Zhu et al., 2017) and powerful convolutional architectures (Isola et al., 2017) have made it possible to perform high quality facial expression editing just using 2D images (Shu et al., 2018; 2017; Athar et al., 2020a; Pumarola et al., 2020; Choi et al., 2018; 2020). However, since these methods are restricted to images and are often trained on frontal datasets, their quality degrades as the pose of the input image changes. Methods such as (Kim et al., 2018; Doukas et al., 2021; Koujan et al., 2020; Athar et al., 2020b) that use a 3DMM to reanimate faces. While being able to do so with great detail, they are unable to perform novel view synthesis as they do not model the geometry of the whole scene. In (Gafni et al., 2020), the authors use neural radiance fields to provide full control on the head. However, they do not model the background and it is assumed to be static. In contrast, FLAME-in-NeRF provides full control over the facial expressions of the person captured in the portrait video and has to ability to synthesize novel views. However, FLAME-in-NeRF does not provide control over the head-pose and we leave that to future work.
3 FLAME-in-NERF
In this section, we describe our method, FLAME-in-NeRF, that enables novel view synthesis of dynamic portrait scene and arbitrary control of facial expressions. We model a dynamic portrait scene using a Neural Radiance Field with per-point deformation (Park et al., 2021) to allow for slight movement of the head. The deformation mechanism introduced in Park et al. (2021) deforms the
rays of each frame to a canonical frame in order to ensure the rays that intersect are photometrically consistent. Facial expression dynamics are controlled by per-frame FLAME expression parameters (Li et al., 2017) derived using (Feng et al., 2021) followed by standard landmarks fitting. In order to ensure disentanglement between the view parameters and the expression parameters, we adopt spatial prior on ray sampling during training. Specifically, we use a silhouette rendering of the fitted FLAME face model and exclude the expression parameters for all points on rays that do not intersect the silhouette.

heading: 3.1 DEFORMABLE NEURAL RADIANCE FIELDS
text: A neural radiance field (NeRF) is a continuous function of, F : (γ(x), γ(d))→ (c(x,d), σ(x)), that, given a 3D point of a scene x and the viewing direction d (i.e the direction of the ray it is on) gives the color c = (r, g, b) and the density σ. Here, F is a multi-layer perceptron (MLP) andγ : R3 → R3+6m is the positional encoding (Mildenhall et al., 2020) defined as γ(x) = (x, ..., sin(2kx), cos(2kx), ...) where m is the total number of frequency bands and k ∈ {0, ...,m − 1}. The expected color of a camera ray r(t) = o+ td, where o is the camera center and d is the direction of the ray, is given by the standard volumetric rendering equation
C(r) = ∫ tf tn T (t)σ(r(t))c(r(t),d)dt, where T (t) = exp ( − ∫ t tn σ(r(s))ds ) (1)
where, T (t) is the accumulated transmittance along the ray from tn to t. In practice, the integral in Eq. (1) is estimated using hierarchical volume sampling, we refer the reader to (Mildenhall et al., 2020) for details. Given multiple images of a scene, with their associated camera intrinsics and extrinsics, rays are shot through each pixel of each image of the scene using mini-batches. The color of each ray is accumulated via volume rendering using Eq. (1) and the error w.r.t the ground truth pixel color is minimized as follows:
min θ ∑ p ||Cp(θ)− CGTp || (2)
where, p is an indexing variable over all the pixels in the mini-batch, θ are the parameters of F and CGTp is the ground truth pixel color.
NeRF, as defined above, is naturally designed for static scenes where the assumption is that two rays that intersect would have the same color. However, as observed in (Park et al., 2021), humans rarely
remain perfectly static during a video capture process, more so when they are speaking or performing facial expressions, thus vanilla NeRF training fails to model such dynamic scenes. In order to take into account for subtle movement of subjects in the capturing process, Park et al. (2021) proposed the Deformable NeRF architecture. In Park et al. (2021), 3D points, x’s, captured in the i-th frame of the video are deformed to a canonical space via a deformation function Di : x→ x′. Here, Di is defined as D(x, ωi) = x′ where ωi is a per-frame latent deformation code. In practice, D(x, ωi) is modeled using an MLP and its coordinate input is also positionally encoded, we choose to omit it for brevity. In addition to a deformation code, ωi, Park et al. (2021) also uses a per-frame appearance code, φi, thus the final radiance field for the i-th frame is as follows:
F : (γ(D(x, ωi)), γ(d), φi)→ (c(x,d), σ(x)) (3)
In addition to the parameters of F each ωi and φi are also optimized through stochastic gradient descent. In practice, D(x, ωi) is modeled as a dense SE(3) field and we use coarse-to-fine regularization on the co-ordinate input, we refer the reader to Park et al. (2021) for details. While the aforementioned methods are able generate novel views (Mildenhall et al., 2020; Park et al., 2021) and handle small movement of objects in the scene (Park et al., 2021), they are still unable to control them.

heading: 3.2 EXPRESSION CONTROL IN DEFORMABLE NEURAL RADIANCE FIELDS
text: FLAME-in-NeRF models changes of subject’s facial expression as changes in the color of 3D scene points. To this end, we condition the learning of deformable NeRF on a set of expression parameters provided by FLAME (Li et al., 2017). The expression conditioned deformable NeRF is defined as follows: F : (γ(D(x, ωi)), γ(d), φi, βi)→ (c(x,d), σ(x)) (4) where, βi is the expression parameter of the i-th frame, D is the deformation function, ωi and φi are the deformation code and appearance code of the i-th frame respectively. The expected colors of each pixel are then calculated using Eq. (1).

heading: 3.3 SPATIAL PRIOR FOR RAY SAMPLING
text: When the radiance field is modeled as described in Eq. (4), there is nothing that prevents the appearance of a point x, that does not project on the face, to become dependent on the expression parameters βi. In Sect 3.4 and Fig 2, we show this phenomena of background expression dependence in practice; a radiance field that changes the appearance of points on the background as the view is kept constant but the expression changes. In order to counter this effect, we use a spatial prior on rays that’s induced by the fitted FLAME face morphable model. First, we render the FLAME mesh giving us a binary silhouette image Si for each frame i. Next, we define two sets of points, let Rini = x1, ...,xn be the set of points that lie on rays inside the FLAME silhouette of frame i and Routi = x1, ...,xm be the set of points that lie on rays outside the FLAME silhouette of frame i (as shown in Fig 4), the radiance field is now defined as:
F : (γ(D(x, ωi)), γ(d), φi, I(x)βi)→ (c(x,d), σ(x)) (5)
where, I(x) = 1 if x ∈ Rini and 0 otherwise . This ensures that points that do not affect face pixels are not affected by facial expression changes. As can be seen in Fig 2, such a spatial ray prior effectively disentangles the appearance and expression and ensures that the background is unaffected by facial expression parameters.
Face region regularization. Since our method optimizes extrinsic camera parameters w.r.t the FLAME 3DMM, we assume the head is static and has an identity mapping to the canonical frame. In order to prevent the deformation network, D, from moving the 3DMM, we penalize any deformation on points sampled from it as follows:
min ψ,ωi ||D(x3DMM;ωi)− x3DMM||; ∀i (6)
where, ψ are the parameters of D.

heading: 3.4 ON THE NECESSITY OF A SPATIAL RAY PRIOR
text: In this section we demonstrate the necessity of the FLAME induced spatial prior on ray sampling as discussed in Sect 3.3. In Fig 2, we show the results of renanimating a portrait video with a constant view directions using our method with and without a spatial ray prior. As can be seen in Fig 2, the results of the model without the prior are of lower quality than that of the model with it. Additionally, when we calculate the difference image of the reanimated frames generated by both methods we see that, despite the viewing direction remaining constant, the model without the prior changes the background. In stark contrast, and unsurprisingly, the model with the spatial ray prior does not do so and only makes changes around the face. The prior explicitly disentangles the expression parameters and the appearance in regions of the scene which do not project to the face. Therefore, we see that the spatial prior is a necessary ingredient high quality portrait video reanimation.

heading: 4 RESULTS
text: In this section, we show results of facial expression control and novel view synthesis on portrait videos captured using a standard smartphone. Since our work is the first to enable both explicit dynamic face controls and novel view synthesis for portrait videos, there is no existing work for an apple-to-apple comparison. We qualitatively and quantitatively compare our method to three baseline approaches that perform similar tasks: (1) Nerfies (Park et al., 2021): a state-of-the-art method using NeRF for novel view synthesis on static portrait scene, (2) NerFACE Gafni et al. (2021), a state-of-the-art method using NeRF for face dynamics control without modeling the entire scene, and (3) First Order Motion Model (FOMM) (Siarohin et al., 2019), a general-purpose image reanimation pipeline. We use the appearance code of the first frame for FLAME-in-NeRF, Nerfies (Park et al., 2021) and NerFACE (Gafni et al., 2021) to perform the reanimation. For FLAME-in-NeRF and Nerfies (Park et al., 2021), we use the deformation code from the first frame. Full videos of the reanimation can be found in the supplementary material. We strongly urge the readers to check them out to see FLAME-in-NeRF performing at its best.
Training Data Capture and Training details. The training data was captured using an iPhone XR smartphone for all the experiments in the paper. We ask the subject to enact a wide range of expressions and speech while trying to keep their head still as the camera is panned around them. Alternatively, the subject can self-capture a video (a selfie video) as they speak. We calculate the expression and shape parameters of each frames in the videos using DECA (Feng et al., 2021). Next, we compute the extrinsic camera parameters via standard landmark fitting using landmarks predicted by (Guo et al., 2020). All training videos are about just 25 seconds long (∼ 700 frames). Due to compute restrictions, the video is down-sampled and the models are trained at 256x256 resolution. We use coarse-to-fine regularization (Park et al., 2021) to train the deformation network D(x, ωi). Please find full details of each experiment in the Section 3. of the supplementary.

heading: 4.1 EVALUATION ON VALIDATION DATA
text: We evaluate FLAME-in-NeRF, Nerfies (Park et al., 2021), NerFACE (Gafni et al., 2021) and FOMM (Siarohin et al., 2019) on held out images. Since FLAME-in-NeRF and Nerfies (Park et al., 2021)
use a per-frame deformation and appearance code, ωi and φi respectively, we cannot perform a direct comparison with the ground truth image as it may have a different deformation to the canonical frame than the first frame (which is what we use as default for reanimation). Therefore, we first find the deformation of a given validation image to the canonical frame by optimizing the deformation code as follows
min ω ||Cp(ω;x,d, θ, φ0)− CGTp || (7)
where, Cp(ω;x,d, θ, φ0) is the predicted color at pixel p generated using Eq. (1) and Eq. (5), φ0 is the appearance code of the first frame, θ are the parameters of F as defined in Eq. (5) and CGTp is the ground-truth pixel value. Note, we only optimize ω, all other parameters of the radiance field are kept fixed. We optimize Eq. (7) for 2000 epochs which we observe to be more than enough to find the loss plateau. Once the optimization finishes, we report the final MSE, PSNR, LPIPS and Face MSE i.e the MSE only over the face region. We use no such optimization with NerFACE (Gafni et al., 2021) since it does not have a deformation module and on FOMM (Siarohin et al., 2019) since it is a image-based method. As can be seen in Table 1, our method outperforms Nerfies (Park et al., 2021), NerFACE (Gafni et al., 2021) and FOMM (Siarohin et al., 2019) on validation images when
averaged over all subjects. Since FLAME-in-NeRF, Nerfies and NerFACE are trained on dynamic portrait videos with changing facial expressions, Nerfies is unable to learn the topological changes of the mouth, often regressing to a ‘mean’ expression (see the third row of Fig 5 and Fig 6) with small view-dependent changes. NerFACE on the hand, lacking a deformation module and a spatial prior, is unable to model the dynamism of the scene and generates significant artefacts in the background 3D scene (see the fourth row of Fig 5 and Fig 6). FOMM (Siarohin et al., 2019), being an image based method, is unable to model novel views. In contrast to Nerfies (Park et al., 2021), NerFACE Gafni et al. (2021) and FOMM (Siarohin et al., 2019), FLAME-in-NeRF, due to the use of a spatial prior along with expression conditioning and a deformation module, is able to model both facial expressions and the full 3D scene with high fidelity, thus giving better reconstructions.

heading: 4.2 REANIMATION WITH ARBITRARY EXPRESSION CONTROL AND NOVEL VIEW SYNTHESIS
text: In this section we show results of reanimating Neural Radiance Fields using both FLAME-in-NeRF and Nerfies (Park et al., 2021) using expression parameters as the driving parameters. Per-frame expression parameters from the driving video are extracted using DECA (Feng et al., 2021) and are given as input to FLAME-in-NeRF as follows:
F : ( γ(D(x, ωi)), γ(d), φi, I(x)βDrivei ) → (c(x,d), σ(x)) (8)
where, βDrivei are the expression parameters derived from the driving video. We refer the reader to Eq. (5) for the definitions of the other variables. Since Nerfies (Park et al., 2021) do not take as input expression parameters, it’s forward pass remains the same as in Eq. (3). As we drive both methods, we also simultaneously change the viewing direction. Fig 5 and Fig 6 shows the results of both FLAME-in-NeRF, Nerfies (Park et al., 2021) and NerFACE (Gafni et al., 2021) with changing the expression parameters and view on two different subjects. As one can see, FLAME-in-NeRF captures the driving expression with high fidelity and is able to do so regardless of viewing direction and without compromising the reconstruction of the background 3D scene. Nerfies (Park et al., 2021), lacking the explicit conditioning on expression parameters, is unable to generate images with the correct facial expression. Only small changes in expression can be observed due to view changes. NerFACE (Gafni et al., 2021), lacking a deformation model, cannot model a dynamic scene as can be seen by poor reconstruction of the background in Fig 5 and Fig 6.

heading: 5 CONCLUSION
text: In this paper we have presented FLAME-in-NeRF, a novel method capable of arbitrary facial expression control and novel view synthesis for portrait videos. FLAME-in-NeRF uses an expressionconditioned neural radiance field along with a spatial prior to generate images with high fidelity to both the subject in the original portrait video and the provided expression parameters, in any viewing direction. FLAME-in-NeRF is also able to model details of the subject’s face such as hair and glasses and reproduce them with high fidelity as the video is driven. However, the problem of controllable human head models with novel view synthesis is far from solved. FLAME-in-NeRF
is unable to model large head movements and requires the subject in the portrait video to remain relatively still. We hope to address this in future work. These are exciting times to be a part of ML/CV/CG communities as neural methods push the state of the art in head control and novel view synthesis with broader impacts in entertainment, education and HCI.

Review summaries:
 This paper proposes a method that marries parametric face models and neural radiance fields to achieve view synthesis of a portrait that provides expression control. Another view of this work is extending Park et al.'s Nerfies to support expression control and driving.   The paper makes three contributions: Firstly, it proposes a dynamic NeRF model that supports certain explicit control. Secondly, the proposed method disentangles the facial expression from the appearance of the background with a simple masking scheme dubbed as spatial ray sampling prior. Finally, the method enables the simultaneous control of viewpoints and facial expressions.

The submitted paper focuses on novel view synthesis from portrait video with controllable facial expressions. The authors separate the modeling of the static background from the dynamic foreground by segmenting out the faces (using landmark detection and fitting). The faces are modeled with Nerfies with expression parameters as latent codes, while the static background is essentially modeled with a standard NeRF without conditioning on per-frame expressions. The proposed method was evaluated on 4 subjects with data collected from a phone. The authors also demonstrate an application of the proposed method where videos can be synthesized with consistent facial expressions as a driving video as input.  

This paper introduces a method for creating animatable portraits within a static scene based on dynamic neural radiance fields.  The main idea is to fit a 3DMM-like model to a set of portrait frames and use it to rasterize a binary mask which is used to enforce distanglement between the foreground and background when training a dynamic NERF. Both qualitative and quantitative results seem to be convincing.

This paper introduces a method for novel view synthesis of a dynamic scene with a person in varying facial expression. To enable the controllability of facial expression, a parametric 3D face model is combined with existing neural radiance field-based approaches. To overcome background artifacts (e.g., distortion) while animating, the spatial face prior is added in a form of binary mask. The method is evaluated on four customized videos.