FaceDet3D: Facial Expressions with 3D Geometric Detail Hallucination

1 INTRODUCTION . Modelling the geometry of the human face continues to attract great interest in the computer vision and computer graphics communities . Strong PCA-based priors make 3D morphable models ( 3DMMs ) ( Blanz et al. , 1999 ) robust , but at the same time over-regularize them . Thus , they fail to capture fine facial details , such as the wrinkles on the forehead when the eyebrows are raised or bumps on the cheeks when one smiles . Additionally , the lack of diversity in the texture space of most available 3DMMs makes it very hard to generate realistic renderings that capture the large variations of color and texture we observe in human faces . Recent methods ( Tewari et al. , 2019 ; 2018 ; Tran & Liu , 2018 ; Tran et al. , 2019 ; Zhu et al. , 2020 ; Booth et al. , 2017 ; Dou et al. , 2017 ; Jackson et al. , 2017 ; Kim et al. , 2018 ; Genova et al. , 2018 ) address this by learning richer shape and expression spaces using a variety of data modalities such as in-the-wild images ( Tewari et al. , 2018 ; Tran & Liu , 2018 ; Tran et al. , 2019 ; Zhu et al. , 2020 ) and videos ( Tewari et al. , 2019 ) . However , despite using more expressive shape and expression spaces , these models still fail to capture fine details in geometry . Recent methods that accurately estimate facial geometric details from single images ( Chen et al. , 2019 ; Tun Trn et al. , 2018 ; Feng et al. , 2021 ) , while being unable to hallucinate and photo-realistically render novel details under expression change , have nonetheless enabled the large scale annotation of unpaired image data with facial geometric details . Thus , it is now possible to train facial detail hallucination methods using unsupervised adversarial losses ( Choi et al. , 2018 ; Pumarola et al. , 2020 ) . Similarly , Neural Rendering ( Thies et al. , 2019 ) has made it possible to render 3D geometries with photo-realism via the use of high-dimensional Neural Texture Maps ( NTMs ) . However , NTMs are able to store fine details of the output image , causing the rendered details to be completely independent of the geometric details . They do not change even if the geometric details do , making neural rendering unsuitable for rendering facial geometric details . In this paper we introduce FaceDet3D , a generative model that hallucinates facial geometric details for any target expression and renders them realistically . The model is made up of two components : 1 ) A detail hallucination network that infers plausible geometric facial details of a person as their expression changes , from a single image of that person . 2 ) A rendering network that overcomes the aforementioned shortcoming of neural rendering and explicitly conditions the rendered facial details on the details of the 3D face geometry . Thus , the rendered details change only when the facial geometric details change . This conditioning is achieved through the use of the novel Augmented Wrinkle Loss and the Detailed Shading Loss during training . Our method is trained using only a large scale in-the-wild image dataset and a much smaller video dataset captured in controlled conditions , without any 3D data as supervision for the target expression geometry . An exhaustive evaluation shows that once trained , our method is able to generate plausible facial details for any desired anatomically consistent facial expression and render it photo-realistically . 2 RELATED WORK . We next describe the most related methods in facial geometry estimation , geometric facial detail estimation and facial expression editing and reanimation . Geometric Facial Details Estimation . Over the past few years there has been significant improvement in the realism of 3D face geometries estimated from single images . In Richardson et al . ( 2017 ) , a CNN ( CoarseNet ) first regresses a rough geometry of the face , facial details are then estimated using another CNN ( FineNet ) using the coarse depth map and input images . In Sela et al . ( 2017 ) , the regressed correspondence and depth maps are registered onto a template mesh , which is further refined to generate the detailed facial geometry . In Tun Trn et al . ( 2018 ) facial details are modelled with bump maps on top of a 3DMM base . Similarly , in Facial Details Synthesis ( FDS ) ( Chen et al. , 2019 ) , details are represented as vertex displacements of an underlying 3DMM , trained using a combination of 3D data and in-the-wild images . These methods , however , can only estimate the facial details of the expression manifested in an image , but can not predict novel facial details for a different expression , which is the motivation of our method . Geometric Facial Details Animation . In DECA ( Feng et al. , 2021 ) , the authors train a network to regress the detailed geometry using a latent detail code . This code enables transfer of details from a target image to the source image . In contrast , the details generated by our method are directly conditioned on expression parameters and action units ( Ekman & Friesen , 1978 ) . Further , unlike our method , DECA ( Feng et al. , 2021 ) is unable photorealistically render the geometric details , leading to sub-par animation . Facial Expression Editing . The success of image-to-image translation networks ( Isola et al. , 2017 ) , adversarial training ( Goodfellow et al. , 2014 ) and cycle-consistency losses ( Zhu et al. , 2017 ) have led to novel expression editing methods that use large-scale in-the-wild training datasets . In ( Shu et al. , 2017 ) , an unsupervised face model disentangles the input face into albedo , normals and shading . Expressions are then edited by traversals in the disentangled latent space . In Choi et al . ( 2018 ) , expressions are edited via adversarial losses coupled with cycle consistency . In ( Pumarola et al. , 2020 ) , a network edits input images to target expressions , represented as Action Units ( Ekman & Friesen , 1978 ) . Athar et al . ( 2020 ) extends this work by explicitly modeling skin motion followed by texture hallucination . In Choi et al . ( 2020 ) , template images are used for editing . While these methods give photo-realistic results , they are restricted to the 2D image space and can not be used to manipulate 3D viewpoint . 3 FACEDET3D 3.1 PROBLEM FORMULATION Let Ix ∈ RH×W×3 be a face image with an expression x represented by Action Units ( Ekman & Friesen , 1978 ) . Its shape and expression parameters in the Basel Face Model ( BFM ) ( Gerig et al. , 2018 ) space are { αs , αe } . A vertex displacement UV map D ( Ix ) ∈ RHD×WD×3 encodes the facial details of the person shown in Ix with the expression x . We want to : ( 1 ) Generate facial details , represented by a vertex displacement map D ( Iy ) , of the same person with expression y ; ( 2 ) Render an image of the geometry with the generated facial geometric details . We use FDS ( Chen et al. , 2019 ) to extract the texture map , initial detail map and geometry of the input image Ix : T ( Ix ) , D ( Ix ) , αs , αe ← FDS ( Ix ) , ( 1 ) where T ( Ix ) ∈ RHT ×WT×3 is the texture map . Next , we use the detail hallucination network , DetH ( · ) , to hallucinate the plausible facial detail map of the person in Ix for expression y , conditioned on D ( Ix ) , as follows : D̃ ( Iy ) = DetH ( D ( Ix ) , x , y , Age ( Ix ) , FaceID ( Ix ) ) , ( 2 ) where y is the target AU , x is the input AU , Age ( Ix ) are features extracted from an age prediction network and FaceID ( Ix ) is the facial embedding of Ix extracted using ( Schroff et al. , 2015 ) . Note that we do not have access to the ground truth image Iy , thus we hallucinate a plausible detail map of Iy ( i.e D̃ ( Iy ) ) using DetH . Once we have D̃ ( Iy ) , we use it to displace the vertices along their normal direction giving us the detailed geometry . Finally , we render this detailed face geometry using a rendering network R ( · ) : Ĩy = R ( T ( Ix ) , D̃ ( Iy ) , αs , α̂e , y , c , l , γ ) , ( 3 ) where α̂e are the target expression parameters , c are the desired camera parameters , γ is the albedo PCA-space parameters of BFM ( Gerig et al. , 2018 ) , and l are the lighting parameters . 3.2 DETAIL HALLUCINATION . DetH takes the input detail map , D ( Ix ) , the input expression AU x , the shape parameters αs , the face embedding FaceID ( Ix ) , and the age features Age ( Ix ) and extracts features from each of them . There features are concatenated in the channel dimension and passed though the Decoder which receives the target action unit y via Adaptive Instance Normalization ( Huang & Belongie , 2017 ) layers . The Decoder gives as output a hallucination D ( Iy ) H and a mask D ( Iy ) M which are combined together with the input detail map , D ( Ix ) , to give D̃ ( Iy ) as follows : D̃ ( Iy ) =D ( Iy ) M D ( Iy ) H+ ( 19D ( Iy ) M ) D ( Ix ) . ( 4 ) The masking mechanism ensures the preservation of the details that are not meant to be changed with the expression . A schematic of the network is shown in Fig 2 . 3.2.1 TRAINING LOSSES . Due to the lack of a large scale in-the-wild dataset of paired data with expression change or 3D data , we can not perform regression using ground-truth geometric facial details , D ( Iy ) of the image Iy . Therefore , we instead use an adversarial training paradigm along with cycle-consistency losses similar to ( Athar et al. , 2020 ; Pumarola et al. , 2020 ) to hallucinate the plausible facial geometric details D̃ ( Iy ) and to ensure the hallucination ’ s fidelity to the target expression and input features . In order to speed up convergence , we weakly supervise the adversarial training using randomly sampled frames from videos of the MUG ( Aifanti et al. , 2010 ) and the ADFES datasets ( Van Der Schalk et al. , 2011 ) , ensuring that the sampling is sparse enough such that there is significant change in expression with frames sampled from each video . We leave the full exposition of all the standard losses and regularizations to the supplementary . Expression Adversarial Loss . In order to ensure the hallucinated facial geometric details , D̃ ( Iy ) , are consistent with the target expression y , as encoded by AUs , we use an expression discriminator DExp . Given D ( Ix ) of some image Ix manifesting expression x , DExp , outputs the following DExp ( D ( Ix ) ) = { r , x̂ } , ( 5 ) where r is a realism score and x̂ is the predicted AU . For brevity , we will use DExp ( D ( Ix ) ) and DExp ( D ) interchangeably . We use the Non-Saturating adversarial loss ( Goodfellow et al. , 2014 ) along with the R1 gradient penalty ( Mescheder et al. , 2018 ) to train DExp . We use a UNet based discriminator ( Schonfeld et al. , 2020 ) in order to discriminate on pixel level . In addition , DExp is trained to minimize the error of the predicted AU LDExpAU = ED∼PD [ || [ DAUExp ( D ) 9 x||22 ] , ( 6 ) where DAUExp is the AU output head of DExp . The Details Hallucination Network , DetH , in addition to be trained to minimize adversarial loss , is also trained to minimize the expression loss : LDetHAU = E Ix , { y } ||DAUExp ( DetH ( · ) ) 9 y||22 ( 7 ) where DetH ( · ) is to be read as in Eq . ( 2 ) and y is the target AU . Superresolution Losses . The detail maps generated by FDS ( Chen et al. , 2019 ) are of resolution 4096× 4096 and thus can not be used directly for training due to GPU-memory constraints . To get around this , we train DetH on detail maps downsampled to 256× 256 . Simultaneously , we finetune a superresolution network , RCAN ( Zhang et al. , 2018 ) , to super-resolve downsampled 256 × 256 patches of D ( Ix ) by a factor of 4 LRCANSR = L1 ( RCAN ( D ( Ix ) P256 ) , D ( Ix ) P1024 ) , ( 8 ) where D ( Ix ) P1024 is a randomly sampled patch of resolution 1024 × 1024 from the full-resolution detail map D ( Ix ) and D ( Ix ) P256 is its downsampled version . During inference , we use RCAN twice on the predicted detail map D̃ ( Iy ) to upsample it to 4096× 4096 : D̃ ( Iy ) HR = RCAN ( RCAN ( D̃ ( Iy ) ) ) . ( 9 ) In the interest of brevity , we will use D̃ ( Iy ) in lieu of D̃ ( Iy ) HR in the remainder of this text .

Summary: 1. This paper proposes a new method FaceDet3D that can generate geometric facial details from a single image given any desired target expression. The method contains two components: first pass the input image to the Detail Hallucination Network to infer the geometric facial details of the subject as there expression changes; then a rendering network is used to generate the detailed rendering result using the 3D face geometry information.  2. The two component models are trained separately using a large scale in-the-wild images and a small video dataset. During the training of the rendering network, the novel Augmented Wrinkle Loss and Detailed Shading Loss were used.  3. The authors further evaluated the method and compared it to DECA to show that the proposed method could generate plausible facial details for any desired expressions and could render the result photo-realistically. 4. Ablation studies were also done to prove the effectiveness of the components in the proposed method.