FLAME-in-NeRF: Neural control of Radiance Fields for Free View Face Animation

1 INTRODUCTION . There is still no fully controllable human head model in natural scenes with arbitrary view synthesis , despite recent increased research . Such a model , in principle , allows for arbitrary control of human head pose , facial expression , identity and viewing direction . Earliest attempts towards a fully controllable human head model were in the form of 3D Morphable Models ( 3DMMs ) ( Blanz et al. , 1999 ) . 3DMMs use a PCA-based space to independently control face shape , facial expressions and appearance and can be rendered in any view using standard graphics-based rendering techniques such as rasterization or ray-tracing . However , 3DMMs ( Blanz et al. , 1999 ) lack the ability to capture fine details of the human head such as hair , skin details and accessories such as glasses . Additionally , only the 3D face can be viewed in novel directions , the scene itself can not , as the mesh only models the human head and nothing else . In contrast , recent methods for novel view synthesis of static and dynamic scenes ( Mildenhall et al. , 2020 ; Park et al. , 2021 ; Gafni et al. , 2020 ; Chibane et al. , 2021 ; Li et al. , 2021b ; Pumarola et al. , 2021 ; Zhang et al. , 2020b ; Gao et al. , 2020 ; Liu et al. , 2020 ; Zhang et al. , 2020a ; Bemana et al. , 2020 ; Martin-Brualla et al. , 2020 ; Xian et al. , 2021 ) are able to generate high quality novel views of a given captured scene but lack any control of the objects contained within the scene , including that of the human face and its various attributes . In this paper we introduce FLAME-in-NeRF , a method that is capable of arbitrary facial expression control and novel view synthesis . We represent the whole scene as a neural radiance field in a manner similar to ( Mildenhall et al. , 2020 ; Park et al. , 2021 ; Gafni et al. , 2020 ) and lend it explicit expression controls using expression parameters derived from a morphable model ( Li et al. , 2017 ) . A simple way to utilize the expression parameters within a NeRF ( Mildenhall et al. , 2020 ) , is to concatenate them to the color-MLP as done in Gafni et al . ( 2020 ) . However , as shown in Fig 2 , such a naive-concatenation entangles the appearance of the scene to the expression parameters causing significant artefacts during reanimation . Standard volumetric representations , such as those used in vanilla NeRFs ( Mildenhall et al. , 2020 ) , do not distinguish between various objects of the 3D scene since they lack any object semantics for scene they ’ re representing . Consequently , naively concatenating controlling parameters , in this case expression parameters , leads to the appearance being entangled with the controlling parameters . In order to avoid this entanglement we , once again , utilize the 3DMM to impose a spatial prior on the 3D scene . Such a prior ensures explicit disentanglement between appearance and expression parameters in parts of the 3D scene where we know the human head is not present , ensuring that the appearance of scene points that do not project on the human face are unaffected by changes in expression . Our model is trained on videos captured using a mobile phone , either by oneself or by someone else . Once trained , FLAME-in-NeRF allows for explicit control of both facial expression and viewing direction while capturing rich details of the scene along with fine details of the human head such as the hair , beard , teeth and accessories such as glasses . Videos reanimated using our method maintain high fidelity to both the driving video in terms of facial expression manifestation and the original captured scene and human head . In summary , our contributions in this paper are as follows : 1 ) We propose a first-of-a-kind neural radiance field capable of explicit dynamic control on objects , such as the human face , within a static scene . 2 ) We experimentally demonstrate expression-appearance entanglement when reanimating portrait videos using standard neural radiance fields . 3 ) We introduce a spatial ray sampling prior that ensures explicit disentanglement between facial expressions and appearance and significantly improves quality of reanimation . 4 ) We develop a system capable of simultaneous control of facial expressions and viewing direction trained on short videos captured from a mobile phone . 2 RELATED WORK . FLAME-in-NeRF is a method for arbitrary facial expression control and novel view synthesis of scenes captured in portrait videos . It is closely related to recent work on neural rendering and novel view synthesis , 3D face modeling , and controllable face generation . Neural Scene Representations and Novel View Synthesis . FLAME-in-NeRF is related to recent advances in neural rendering and novel view synthesis ( Mildenhall et al. , 2020 ; Park et al. , 2021 ; Gafni et al. , 2020 ; Yariv et al. , 2020 ; Chibane et al. , 2021 ; Lassner & ZollhÃ¶fer , 2021 ; Lombardi et al. , 2021 ; Li et al. , 2021b ; Pumarola et al. , 2021 ; Riegler & Koltun , 2021 ; Zhang et al. , 2020b ; Oechsle et al. , 2021 ; Gao et al. , 2020 ; Sitzmann et al. , 2019 ; Liu et al. , 2020 ; Zhang et al. , 2020a ; Bemana et al. , 2020 ; Martin-Brualla et al. , 2020 ; Xian et al. , 2021 ; Wizadwongsa et al. , 2021 ) . Notably , Neural Radiance Fields ( NeRF ) uses a Multi-Layer Perceptron ( MLP ) , F , to learn a volumetric representation of a scene . Given a 3D point and the direction from which the point is being viewed , F predicts its color and volume density . For any given camera pose , F is first evaluated densely enough throughout the scene using hierarchical volume sampling ( Mildenhall et al. , 2020 ) , then volume rendering is used to render the final image . F is trained by minimizing the error between the predicted color of a pixel and its ground truth value . While NeRF is able to generate high quality and photo-realistic images for novel view synthesis , it is only designed for a static scene and is unable to represent scene dynamics . Specifically designed for dynamic portrait video synthesis , our approach not only models the dynamics of human faces , but also allow specific controls on the facial animation . Dynamic Neural Scene Representations . Methods such as ( Li et al. , 2021b ; a ; Pumarola et al. , 2021 ; Xian et al. , 2021 ) extend NeRF to dynamic scenes by providing as input a time component and along with it imposing temporal constraints using scene flow ( Li et al. , 2021b ; Xian et al. , 2021 ) or by using a canonical frame ( Pumarola et al. , 2021 ) . Similarly , Nerfies ( Park et al. , 2021 ) too work with dynamic scenes by mapping to a canonical frame , however it assumes that the movement is small . FLAME-in-NeRF , like ( Park et al. , 2021 ) , models the portrait video by mapping to a canonical frame and assumes that the head motion in the video is small . Controllable Face Generation . The advent of adversarial training ( Goodfellow et al. , 2014 ) , cycleconsistency losses ( Zhu et al. , 2017 ) and powerful convolutional architectures ( Isola et al. , 2017 ) have made it possible to perform high quality facial expression editing just using 2D images ( Shu et al. , 2018 ; 2017 ; Athar et al. , 2020a ; Pumarola et al. , 2020 ; Choi et al. , 2018 ; 2020 ) . However , since these methods are restricted to images and are often trained on frontal datasets , their quality degrades as the pose of the input image changes . Methods such as ( Kim et al. , 2018 ; Doukas et al. , 2021 ; Koujan et al. , 2020 ; Athar et al. , 2020b ) that use a 3DMM to reanimate faces . While being able to do so with great detail , they are unable to perform novel view synthesis as they do not model the geometry of the whole scene . In ( Gafni et al. , 2020 ) , the authors use neural radiance fields to provide full control on the head . However , they do not model the background and it is assumed to be static . In contrast , FLAME-in-NeRF provides full control over the facial expressions of the person captured in the portrait video and has to ability to synthesize novel views . However , FLAME-in-NeRF does not provide control over the head-pose and we leave that to future work . 3 FLAME-in-NERF In this section , we describe our method , FLAME-in-NeRF , that enables novel view synthesis of dynamic portrait scene and arbitrary control of facial expressions . We model a dynamic portrait scene using a Neural Radiance Field with per-point deformation ( Park et al. , 2021 ) to allow for slight movement of the head . The deformation mechanism introduced in Park et al . ( 2021 ) deforms the rays of each frame to a canonical frame in order to ensure the rays that intersect are photometrically consistent . Facial expression dynamics are controlled by per-frame FLAME expression parameters ( Li et al. , 2017 ) derived using ( Feng et al. , 2021 ) followed by standard landmarks fitting . In order to ensure disentanglement between the view parameters and the expression parameters , we adopt spatial prior on ray sampling during training . Specifically , we use a silhouette rendering of the fitted FLAME face model and exclude the expression parameters for all points on rays that do not intersect the silhouette . 3.1 DEFORMABLE NEURAL RADIANCE FIELDS . A neural radiance field ( NeRF ) is a continuous function of , F : ( γ ( x ) , γ ( d ) ) → ( c ( x , d ) , σ ( x ) ) , that , given a 3D point of a scene x and the viewing direction d ( i.e the direction of the ray it is on ) gives the color c = ( r , g , b ) and the density σ . Here , F is a multi-layer perceptron ( MLP ) andγ : R3 → R3+6m is the positional encoding ( Mildenhall et al. , 2020 ) defined as γ ( x ) = ( x , ... , sin ( 2kx ) , cos ( 2kx ) , ... ) where m is the total number of frequency bands and k ∈ { 0 , ... , m − 1 } . The expected color of a camera ray r ( t ) = o+ td , where o is the camera center and d is the direction of the ray , is given by the standard volumetric rendering equation C ( r ) = ∫ tf tn T ( t ) σ ( r ( t ) ) c ( r ( t ) , d ) dt , where T ( t ) = exp ( − ∫ t tn σ ( r ( s ) ) ds ) ( 1 ) where , T ( t ) is the accumulated transmittance along the ray from tn to t. In practice , the integral in Eq . ( 1 ) is estimated using hierarchical volume sampling , we refer the reader to ( Mildenhall et al. , 2020 ) for details . Given multiple images of a scene , with their associated camera intrinsics and extrinsics , rays are shot through each pixel of each image of the scene using mini-batches . The color of each ray is accumulated via volume rendering using Eq . ( 1 ) and the error w.r.t the ground truth pixel color is minimized as follows : min θ ∑ p ||Cp ( θ ) − CGTp || ( 2 ) where , p is an indexing variable over all the pixels in the mini-batch , θ are the parameters of F and CGTp is the ground truth pixel color . NeRF , as defined above , is naturally designed for static scenes where the assumption is that two rays that intersect would have the same color . However , as observed in ( Park et al. , 2021 ) , humans rarely remain perfectly static during a video capture process , more so when they are speaking or performing facial expressions , thus vanilla NeRF training fails to model such dynamic scenes . In order to take into account for subtle movement of subjects in the capturing process , Park et al . ( 2021 ) proposed the Deformable NeRF architecture . In Park et al . ( 2021 ) , 3D points , x ’ s , captured in the i-th frame of the video are deformed to a canonical space via a deformation function Di : x→ x′ . Here , Di is defined as D ( x , ωi ) = x′ where ωi is a per-frame latent deformation code . In practice , D ( x , ωi ) is modeled using an MLP and its coordinate input is also positionally encoded , we choose to omit it for brevity . In addition to a deformation code , ωi , Park et al . ( 2021 ) also uses a per-frame appearance code , φi , thus the final radiance field for the i-th frame is as follows : F : ( γ ( D ( x , ωi ) ) , γ ( d ) , φi ) → ( c ( x , d ) , σ ( x ) ) ( 3 ) In addition to the parameters of F each ωi and φi are also optimized through stochastic gradient descent . In practice , D ( x , ωi ) is modeled as a dense SE ( 3 ) field and we use coarse-to-fine regularization on the co-ordinate input , we refer the reader to Park et al . ( 2021 ) for details . While the aforementioned methods are able generate novel views ( Mildenhall et al. , 2020 ; Park et al. , 2021 ) and handle small movement of objects in the scene ( Park et al. , 2021 ) , they are still unable to control them .

Summary: The submitted paper focuses on novel view synthesis from portrait video with controllable facial expressions. The authors separate the modeling of the static background from the dynamic foreground by segmenting out the faces (using landmark detection and fitting). The faces are modeled with Nerfies with expression parameters as latent codes, while the static background is essentially modeled with a standard NeRF without conditioning on per-frame expressions. The proposed method was evaluated on 4 subjects with data collected from a phone. The authors also demonstrate an application of the proposed method where videos can be synthesized with consistent facial expressions as a driving video as input.