FaceDet3D: Facial Expressions with 3D Geometric Detail Hallucination
Authors: ShahRukh Athar
Albert Pumarola
Francesc Moreno-noguer
Dimitris Samaras
Venue: ICLR 2022 Submitted (2022)

Abstract: Facial Expressions induce a variety of high-level details on the 3D face geometry. For example, a smile causes the wrinkling of cheeks or the formation of dimples, while being angry often causes wrinkling of the forehead. Morphable Models (3DMMs) of the human face fail to capture such fine details in their PCA-based representations  and consequently cannot generate such details when  used to edit expressions. In this work, we introduce FaceDet3D, a method that generates - from a single image - geometric facial details that are consistent with any desired target expression.  The facial details are represented as a vertex displacement map and used then by a Neural Renderer to photo-realistically render novel images of any single image in any desired expression and view. 

Under review as a conference paper at ICLR 2022
FACEDET3D: FACIAL EXPRESSIONS WITH 3D GEO- METRIC DETAIL HALLUCINATION
Anonymous authors Paper under double-blind review
ABSTRACT
Facial Expressions induce variety of high-level details on the 3D face geometry. For example, smile causes the wrinkling of cheeks or the formation of dimples, while being angry often causes wrinkling of the forehead. Morphable Models (3DMMs) the human face fail to capture such fine details in their PCA-based representations and consequently cannot generate such details when used to edit expressions. In this work, we introduce FaceDet3D, a method that generates from single image- geometric facial details that consistent with any desired target expression. The facial details are represented as a vertex displacement map and used then by a Neural Renderer photo- realssrralis render novel images of any single image in any desired expression and view.
Iapat
Recomstuction Happy
Reconsesccedd Focial Geomeley
Predicted Focial Detail
Render of Recoustracted Pacial Geometry
Remler of Predicted Facial Detail
Shatling
Figure 1: Facial detail hallucination and rendering. Given a single input image, a target expression (in this case 'Happy'), ) and initial detailed geometry extracted using FDS (Chen et al., 2019) (shown in column 'Reconstruction' and row 'Detailed Shading') as input, our method hallucinates facial geometric details consistent with the target expression. The hallucinated details are added to the smooth proxy geometry (marked as 'Proxy Shading', also extracted using FDS (Chen et al., 2019)), to give detailed geometry with facial details consistent with the target expression (in column 'Happy' and row 'Detailed Shading'). ). The detailed geometry is then rendered using Neural Rendering to give the final image (first row of the column labelled 'Happy"). ) zoom-in of one of the predicted facial details and its render is shown in column 'Predicted Facial Detail' and 'Render of Predicted Facial Detail' respectively. (Electronic zoom recommended)
INTRODUCTION
Modelling the geometry of the human face continues to attract great interest in the computer vision and computer graphics communities. Strong PCA-based priors make 3D morphable models (3DMMs) (Blanz et al., 1999) robust, but the same time over-regularize them. Thus, they fail to capture fine facial details, such as the wrinkles on the forehead when the eyebrows are raised or bumps on the cheeks when one smiles. Additionally, the lack of diversity in the texture space of most available 3DMMs makes it very hard to generate realistic renderings that capture the large variations of color and texture we observe in human faces. Recent methods (Tewari et al., 2019; 2018; Tran & Liu, 2018; Tran et al., 2019; Zhu et al., 2020; Booth et al., 2017: Dou et al., 2017; Jackson et al., 2017; Kim et al., 2018; Genova et al., 2018) address this by learning richer shape and expression spaces using variety of data modalities such in-the-wild images (Tewari al., 2018; Tran & Liu, 2018; Tran et al., 2019; Zhu et al., 2020) and videos (Tewari et al., 2019). However, despite using more expressive shape and expression spaces, these models still fail capture fine details in geometry.

---

Under review as a conference paper at ICLR 2022
Recent methods that accurately estimate facial geometric details from single images (Chen et al., 2019; Tun T'n et al., 2018; Feng et al., 2021), while being unable to hallucinate and photo-realisticlly render novel details under expression change, have nonetheless enabled the large scale annotation of unpaired image data with facial geometric details. Thus, it is now possible to train facial detail hallucination methods using unsupervised adversarial losses (Choi et al., 2018; Pumarola et al., 2020). Similarly, Neural Rendering (Thies et al., 2019) has made it possible to render 3D geometries with photo-realism via the use of high-dimensional Neural Texture Maps (NTMs). However, NTMs are able to store fine details of the output image, causing the rendered details to be completely independent of the geometric details. They do not change even if the geometric details do, making neural rendering unsuitable for rendering facial geometric details.
In this paper introduce Facelet33D, a generative model that hallucinates facial geometric details for any target expression and renders them realistically. The model is made up of two components: 1) A detail hallucination network that infers plausible geometric facial details of person as their expression changes, from a single image of that person. 2) A rendering network that overcomes the aforementioned shortcoming of neural rendering and explicitly conditions the rendered facial details on the details of the 3D face geometry. Thus, the rendered details change only when the facial geometric details change. This conditioning achieved through the use of the novel Augmented Wrinkle Loss and the Detailed Shading Loss during training.
Our method is trained using only a large scale in-the-wild image dataset and a much smaller video dataset captured in controlled conditions, without any 3D data as supervision for the target expression geometry. An exhaustive evaluation shows that once trained, our method is able to generate plausible facial details for any desired anatomically consistent facial expression and render it
RELATED WORK
We next describe the most related methods in facial geometry estimation, geometric facial detail estimation and facial expression editing and reanimation.
Geometric Facial Details Estimation. Over the past few years there has been significant improve- ment in the realism of 3D face geometries estimated from single images. In Richardson et al. (2017), CNN (CoarseNet) first regresses rough geometry of the face, facial details are then estimated using another CNN (FineNet) using the coarse depth map and input images. In Sela et al. (2017), the regressed correspondence and depth maps registered onto a template mesh, which is further refined to generate the detailed facial geometry. In Tun Trn et al. (2018) facial details are modelled with bump maps on top of a 3DMM base. Similarly, in Facial Details Synthesis (CDe) et al., 2019), details are represented as vertex displacements of an underlying 3DMM, trained using a combination of 3D data and in-the-wild images. These methods, however, can only estimate the facial details of the expression manifested in an image, but cannot predict novel facial details for a different expression, which the motivation of our method. Geometric Facial Details Animation. In DECA (Feng et al., 2021), the authors train a network to regress the detailed geometry using latent detail code. This code enables transfer of details from a target image to the source image. In contrast, the details generated by our method are directly conditioned on expression parameters and action units (Ekman & Friesen, 1978). Further, unlike our method, DECA (Feng et al., 2021) is unable photorealisticaly render the geometric details, leading to sub-par animation.
Facial Expression Editing. The success of image-to-image translation networks (Isola et al., 2017), adversarial training (Goodfellow et al., 2014) and sycle-consistency losses (Zhu et al., 2017) have led to novel expression editing methods that use large-scale in-the-wild training datasets. In (Shu et al., 2017), an unsupervised face model disentangles the input face into albedo, normals and shading. Expressions are then edited by traversals in the disentangled latent space. In Choi et al. (2018), expressions are edited via adversarial losses coupled with cycle consistency. In (Pumarola et al., 2020), network edits input images to target expressions, represented as Action Units (Ekman & Friesen, 1978). Athar et al. (2020) extends this work by explicitly modeling skin motion followed by texture hallucination. In Choi et al. (2020), template images are used for editing. While these methods give photo-realistic results, they are restricted to the 2D image space and cannot be used to manipulate 3D viewpoint.
2

---

Under review as a conference paper at ICLR 2022
3 FACEDET3D
3.1 PROBLEM FORMULATION
Let Ix E R be face image with expression x represented by Action Units (Ek- man & Friesen, 1978). Its shape and expres- sion parameters in the Basel Face Model (BFM) (Gerig al., 2018) space are {ak, (Xe). A vertex displacement UV map D[Ix) E RRD x x3 encodes the facial details of the person shown in Ix with the expression
AdiaNN
DII,Y
BiL,
Masking
DII
We want to: (I) Generate facial details, represented by a vertex displacement map (Iyy), , of the same person with expression y; (2) Render an image of the geometry with the generated facial geometric details.
Figure 2: Detail Hallucination Network. Given tar- get expression y, Detff outputs detail hallucination D(Iy)* )" and a detail mask D(Iy) )M The detail hallu- cination is combined with the input detail map D(Ix) using the detail mask to give the final hallucinated facial geometric detail map (Iyy).
We use FDS (Chen et al., 2019) to extract the texture map, initial detail map and geometry of the input image Ix:
T(Ix,, D(I)), as ae FDS(
where T(x)) E " x3 is the texture map.
Next, we use the detail hallucination network, (-). to hallucinate the plausible facial detail map of the person in Ix for expression y, conditioned on D(Ix), follows:
D(Ly)= = (D(x) x, Age FaceiD(Ix))
(2)
where y is the target AU, x is the input AU, Age(Ix) are features extracted from age prediction network and FacelD(Ix)) ) is the facial embedding of I extracted using (Schroff et al., 2015). Note that we do not have access to the ground truth image Iy. thus hallucinate plausible detail map of Iy (i.e ))I,)) using DetH. Once we have D(yy), we use to displace the vertices along their normal direction giving us the detailed geometry. Finally, render this detailed face geometry using a rendering network R()):
ï, = (T(I), ass &e,y &.
where de are the target expression parameters, e are the desired camera parameters, y is the albedo PCA-space parameters of BFMM(erig et al., 2018), and I are the lighting parameters.
3.2 DETAIL. HALL.UCINATION
DetH takes the input detail map, (Ix)), the input expression AU x, the shape parameters Oss, the face embedding FacelI((Ix). and the age features Age(Ix) and extracts features from each of them. There features are concatenated in the channel dimension and passed though the Decoder which receives the target action unit via Adaptive Instance Normalization (Huang & Belongie, 2017) layers. The Decoder gives as output a hallucination D(Iy)* and a mask D(Iy) which are combined together with the input detail map, D(Ix), to give D(I,) as follows:
D(Ly)= Mo P(I,)"+(L D)L)) )(L)))
(4)
The masking mechanism ensures the preservation of the details that not meant to be changed with the expression. A schematic of the network is shown in Fig 2.
3.2.1 TRAINING LOSSES
Due to the lack large scale in-the-wild dataset paired data with expression change or 3D data, we cannot perform regression using ground-tree -tuth geometric facial details, D(Iy) of the image Iy. Therefore, we instead use an adversarial training paradigm along with sycle-consistency losses similar to (Athar et al., 2020; Pumarola et al., 2020) to hallucinate the plausible facial geometric details DIIy) and to ensure the hallucination's 's fidelity to the target expression and input features. In order to speed up convergence, we weakly supervise the adversarial training using randomly sampled frames from videos of the MUG (Aifanti et al., 2010) and the ADFES datasets (Van Der Schalk et al.,
3

---

Under review as a conference paper at ICLR 2022
Tex2 Im
Rasterized Proxy UV-Map
T&R
WV-Raster(Gp.C
Sampled Powsy NTM
Reader le decaalle
AdaiN
Lit-Img Lmg
Rensberar g
T(L)
NTM R #xRxw
Iy
Sampling
Sumpled Detailed NTM
Detailed Ronder
Details Runderer
fw
UV-Raster c)) Rosterized Detalied Map
AtbaNN
iP
y
The Rendering Network. The Rendering Network, R, first predicts a Neural Texture Map NTM from the given input texture map T using NTexs The NTM is then rasterized using both the proxy (geometry w/o details) and the detailed geometry and input into an image rendering network Tex2Im Im. Tex2Im generates rendering the details Iς and low-resolution image i containing only detail-invariant image texture. They added together in the final rendered image Iy.
Figure 3:
2011), ensuring that the sampling is sparse enough such that there is significant change in expression with frames sampled from each video. We leave the full exposition of all the standard losses and regularizations to the supplementary.
Expression Adversarial Loss. In order to ensure the hallucinated facial geometric details, D(Iy), are consistent with the target expression y. as encoded by AUs, we use an expression discriminator DExp- Given (Ix)) of some image Ix manifesting expression x, Disps outputs the following
(D(L()((,)) = {r,*}.
(5)
where r is a realism score and & is the predicted AU. For brevity, we will use DEsp (D(x)) and DEsp (D) interchangeably. We use the Non-Saturating adversarial loss (Goodfellow et al., 2014) along with the RI gradient penalty (Mescheder et al., 2018) to train D We use a UNet based discriminator (Schonfeld et al., 2020) in order to discriminate pixel level. In addition, Dexp is trained minimize the error of the predicted AU
Dew AU
= EDDPO [I] (D)
x x
(6)
where Disp is the AU output head of DEsp. The Details Hallucination Network, DetH in addition to be trained to minimize adversarial loss, is also trained to minimize the expression loss:
Deetf
E Le.yy)
(DA - y112
where (-) is to be read as in Eq. (2) and y is the target AU. Superresolution Losses. The detail maps generated by FDS (Chen et al., 2019) are of resolution 4096 4096 and thus cannot be used directly for training due to GPU-memory constraints. To get around this, we train DetF( on detail maps downsampled to 256 x 256. Simultaneously, we finetune superresolution network, RCAN (Zhang al., 2018), super-resolve downsampled 256 256 patches of D(Ix) by a factor of 4
CRCAN = L1 (RCAN (D(I)) P(z))
(8)
where ) is randomly sampled patch of resolution 1024 x 1024 from the full-resolution detail map D(Ix) and D(Ix) is downsampled version. During inference, we use RCAN twice on the predicted detail map D(Iy) to upsample it to 4096 x 4096:
)(I,)* RCAN (RCAN (D(I,))
(9)
In the interest of brevity, will D(I,) in lieu of ) in the remainder of this text.
4

---

Under review as a conference paper at ICLR 2022
Topert
Rocuns
Anger
Disgust
Happy
Surpriso
Disgust
Datailee
Sbading
Input
Anger
Dieguet
Happy Surprise
Disgust
Happy
Detalled
Sharling
Inpot
Recuns
Anger
Diaguet
Happe Surprisn
Happy
Surprise
Shadlng
Figure 4: Expression Change. Here we show the results of detail hallucination and rendering the expression changes. The first column the input image, the second column the reconstruction (the detailed shading under column 'Recons' generated by (Chen al., 2019)) and subsequent columns are results of the hallucinated details and their renders by our method under different expressions. The first image row is the output rendering and the second is the shading of the detailed geometry. As one can DetHf is able to generate realistic details depending the expression being manifested and R. is able to faithfully render them to the image space. We zoom-in on subset of details the final column for greater clarity, (Please in high resolution)
3.3 RENDERING NETWORK
The rendering network, R, consists of two subnetworks: (1) The Neural Texture prediction network NTex, and (2) The image rendering network Tex2Im.
Neural Texture Prediction. MTex, predicts the Neural Textures given the texture map T(Ix) of image Ix:
NTM ATex, (T(L)): NTM FxHxW
(10)
where NTM the predicted neural texture map with channels. Unlike (Thies al., 2019), where the NTM is identity specific, Alex, can be used on any regardless of identitym to generate its corresponding neural texture map. Through training, Afeex,, learns extract the appropriate high-dimensional texture features from T(Ix) such that NTM can be used to generate realistic render of the person in I in any desired expression and view.
Image rendering. The image rendering network, Tex2Im, consists of two branches, the low-res image renderer ge and the detail renderer 9w. The low-res image renderer generates identity-specific image textures that are invariant to the predicted geometric details, such as the skin-tone or eye color. The detail renderer 9N renders the facial geometric details obtained from the detail hallucination network, DetH, and adds them to the low-res image generated by go. The separation of the image rendering network into two branches allows each branch focus on its respective task and leads to high-quality renderings.
The low-res image renderer, ge. takes as input the NTM sampled using the UV map rasterized by the proxy geometry, Gp {0 x D(Iy), Ois, de), i.e the geometry without any details, along with the shaded albedo also rasterized by Gp:
j ge (Sample(NTM, UV Rastter(rp..))
(11)
where c are the camera parameters, y the target AU, y is a vector of the coefficients of the albedo PCA-space of the BFM (Gerig et al., 2018), and are the lighting parameters. Since go only inputs dependent on Gp it generates the image textures that are invariant to details predicted by DetH.
5

---

Under review as a conference paper at ICLR 2022
The detail renderer, 9w, takes imput the NTM sampled using the UV map rasterized by the detailed geometry, GD (D(Iy): ) as die) along with the shaded albedo also rasterized by GD:
ry go (SampleNNTM, UV UV-Raster(Gp. c)), v,7,1)
(12)
where с are the camera parameters, y is the target AU, 7 is a vector of the coefficients of the albedo PCA-space of the BFM (Gerig et al., 2018), and / are the lighting parameters. The final output image is calculated as:
f - i +1?
(13)
Since all of the detail invariant textures are already generated by ge in I the detail renderer, 9w can solely focus on realistically rendering the details hallucinated by the details hallucination network DetH.
3.3.1 TRAINING LOSSES
The renderings generated by R must: 1) faithfully render the geometric details onto the RGB space and 2) be realistic. Neural Rendering (Thies et al., 2019) is designed to address 2) as the high- dimensional neural texture map is able store the fine details of the output texture. Consequently, the details on the rendered image become entirely conditional on the input texture map, ((xx), and ignore the detailed geometry Gp. This significantly hurts 1) causing the details on the rendered image to remain unchanged even if the facial geometric details change due to changes in facial expression. In order ensure the output renderings satisfy 2) we use, along with the branched architecture discussed in Sect 3.3, an Augmented Wrinkle Loss (AugW) and the Detailed Shading Loss (DSL) to ensure the geometric details are faithfully rendered onto the output image. Additionally. R. also trained with a photometric loss and an Expression Adversarial Loss in order to maintain photometric and expression consistency. We leave the full exposition of all the standard losses and regularizations to the supplementary.
Augmented Wrinkle Loss (AugW). In order to enforce the rendering of geometric details onto the rendered image we add 'fake' wrinkles Given the the detailed geometry of Ix, Gp {D(Ix), Cks, ae). geometry with 'fake' details
image Ix and force R. to generate the same.
G (I(L)) ai2 ) using the geometric details from some random image I and the lighting of Ix, the 'fake" wrinkles are added as follows:
Shading(I.) I-sph(GD, (GD,1); Shading (I.) Lsph 1)
L
(14)
I' Shading (I.) x
Shading(L)
where Lsph is the spherical harmonic lighting function and are the coefficients of the first 9 spherical harmonics. The artificially wrinkled image 1 is now re-rendered using R.
i = ((x(1x) G,,,,
(15)
Laugw LapLoss (.)
where, LapLoss is the Laplacian Loss (Ling & Okada, 2006). In order to faithfully reconstruct I, R. is forced to rely on the detailed geometry Gip, since the input texture map T(Ix), and consequently the neural texture map, contain no information about the 'fake" wrinkles.
Detailed Shading Loss (DSL). In addition to the Augmented Wrinkle Loss, we also try to predict the shading of the detailed facial geometry from the output rendering Ix
Shading(ix) = fa(ix)
(16)
LDSl- = (Saddon(Sha Shading"
where fo is small convolutional network (CNN) with only two layers and the shading Shading" is calculated as in Eq. (14). We calculate this loss only over the skin region. Since, Jo small CNN with limited representational capacity, the details must be quite visible on the rendered image I in order for them to be picked up by fo generate an accurate shading (fa
Expression Adversarial Loss. In order to ensure that the rendered output conforms to the target ex- pression expression adversarial loss. Given rendered image, İ,- R(T(L)) Gr,

---

Under review as a conference paper at ICLR 2022
Method Name
FID (L)
FacelD (1)
FaceDet31)
25,47
1.02c -3 + 4.14e
DECA
192.21
3.04e + 6.6e
Table Quantitative Comparison with DECA based on FID Score and FaceID distance.
manifesting the expression encoded by AU an expression discriminator, D outputs
(Ix) = {r,*} Exp
(17)
where is realism score and & is the predicted AU. We use the Non-Saturating adversarial loss (Goodfellow et al., 2014) along with the R1 gradient penalty (Mescheder et al., 2018) to train DEsp-
In addition, Desp is trained to minimize the predicted AU enror
AU
Ding
EI,=
[ILD
RGB.AU AU (Ix)
x1
(18)
where Desp DRGB,AU the AU output head of RGB The Rendering Network, R, in addition to be trained minimize adversarial loss, is also trained to minimize the AU loss
AU
EL
Dxp
RGB Exp
(R(()) y11..
(19)
where R(+) is to be read as in Eq. (3).
Happy
Toput
Recomstruction
Original View
Detalled Shading
Angry
Input
Recunstruction.
Original View
View
View
View
View
View 5
Detailed
Shading
Figure 5: View Consistency, In this figure we demonstrate the consistency of the details rendered by R, subset of the hallucinated details from DeLH are marked with blue rectangles. As can be seen, R, renders the details in consistent manner across views. (Please view in high resolution)
RESULTS
We train the detail hallucination network, DetH,, and the rendering network R, on 9,000 images from the FFHQ dataset (Karras al., 2019). Additionally, 3,000 frames were sampled from the MUG (Aifanti et al., 2010) and the ADFES (Van Der Schalk et al., 2011) datasets to speed-up training of the detail hallucination network DetH(. Due to memory constraints, DetT( and R. are trained independently. Upon publication will release the code. Below, show the FaceDet3D's results on expression change, results on view consistency, a comparison with DECA (Feng et al., 2021) and Ablation studies.
7

---

Under review as a conference paper at ICLR 2022
Expression change. We first demonstrate the detail hallucination and rendering results with expression change. Fig 4 shows the result of changing the expression of some input image to variety of expressions. The first column of Fig 4 is the input image, the second column shows the image reconstructed using the de- tail map (Ixx) predicted by FDS (Chen et al.,
Input
FneeDet3D FaceDet3D
DECA Goo
DECA
Goo
Rend
Rend
Happy
Angry Diggust
2019). In the subsequent columns D(Ix) is used as input to generate D(Iy)) where y (Anger, Disgust, Happy, Surprise). The first row shows the final rendered image, i.e the out- put of R and the second row shows the shaded geometry with the hallucinated details, i. Gp. The hallucinated details and their corresponding rendering are marked out with dashed red and blue rectangles. We zoom in the red rectangles of the last column. As seen in the second row of Fig 4, the details hallucinated are consistent with the manifested expression. For example, 'Anger' and 'Disgust' (3rd and 4th column of Fig 4 re- spectively) show consistent wrinkling around the forehead and the nose while 'Happy' (col- umn 6 Fig 4) shows consistent wrinkling around the cheeks. Zooming in the last column of Fig 4 highlights the realism of the hallucinated details produced by the rendering network, R. Further, in Fig 7 we show the utility of halluci- the expression changes. The first row of Fig 7 shows the image rendered with the hallucinated details from DeLH as the expression changes. The second row of images shows the images rendered without details, this is done by setting the hallucinated detail map to zero. As seen by comparing the skin appearance marked with the red rectangles, in the first row the skin changes realistically as the expression of the person changes due to the changing facial geometric details, while the skin in the second row remains unchanged, significantly hurting realism. In the supplementary we show further results expression animation and encourage the reader to inspect them.
Figure 6: Qualitative Comparison with DECA (Feng et al., 2021) The row label shows the expres- sion being animated, the first column is the input im- age, the second is the detailed geometry hallucinated by FaceDet3D for the expression being animated, the 3rd column is the render generated by FaceDet3D of its hallucinated detailed geometry. the 4th column is the de- tailed geometry generated by DECA for the expression being animated and the 5th column is render generated by DECA. As can be seen by comparing columns 2 and 4, the details generated by FaceDet3D are significantly consistent with the expression than those gener- ated by DECA, whose details are more generic (see text for details). Further, a comparison between the 3rd and the fifth column of the figure show that the renders of FaceDet3D are significnatly more realistic than those of DECA. (Please view high resolution)
nating details
View Consistency. In Fig we show the consistency of the details rendered by R, across various views in novel expressions. While the underlying face model ensures that the detailed geometry is consistent in any view, there is no guarantee its rendering generated by R. would be too. The first column of Fig 5 is the input image, the second column is the reconstruction of the input in the original expression and view and the third column shows the input image manifesting a novel expression in the view of the input image. The subsequent columns show the input image rendered with the target expression in novel views. The blue rectangles around the details in the rendered image show they are rendered with high fidelity to the shaded geometry and therefore look consistent across a variety of views. Fig 5 shows that R. is able to maintain a close to one-to-one correspondence while rendering geometric details to the image space.
Input
Boonnatructian
Dagus
Happy
Bender
ofihint
shalling
Figure 7: Details No Details. We com- pare the results of changing the expression both with and without hallucinating details. The text in the input image (on the chin) is a watermark. (Please view in high resolution)
Comparison with DECA (Feng et al., 2021): In Fig 6 we provide qualitative comparison of FaceDet3D and DECA. The row label is the expression being animated, the first column is the input image, the second is the detailed geometry hallucinated by FaceDet3D for the expression being animated, the
8

---

Under review as a conference paper at ICLR 2022
Rendering Type
FID (J)
FacelD (4)
NR Double Branch (Ours)
25.47
1.02c 4.14c -1
NR Single Branch (ablation)
40.78
2.2e + 3.
Ordinary Rendering
48.18
2.7e 3 + 6.5e
Table 2: FID Score and FacelD distance.
3rd column is the render generated by F of its hallucinated detailed geometry. the fourth column is the detailed geometry generated by DECA for the expression being animated and the fifth column is render generated by DECA. Note that DECA's results have the full head because they use the FLAME 3DMM while we use the BFM 3DMM.
By comparing the details in the second and the fourth column of Fig one can see that FaceDet3D generates more concentrated and higher quality details. For example, DECA always generates wrinkles on the forehead regardless of the expression being animated, thus the details generated for 'Happy' (row 1) are inconsistent with the expression. Similarly, a comparison between the third and fifth column of Fig 6 shows that the renders FaceDet3D are significantly more realistic than those of DECA. In Table 1, we see that the FacelD distance (using FaceNet (Schroff' et al., expression edits, of the renders of FaceDet3D is
2015)) and the FID score, across a variety significantly lower than the renders of DECA. Further, the blue boxes shown in columns 2 and 3 of Fig 6 show that the hallucinated details are faithfully and photo-realisticaly reproduced the render.
Ablation Studies. We examine the utility of the Aug- mented Wrinkle Loss (AugW) and the Detailed Shad- ing Loss (DSL) in rendering the facial geometric details to the image space. Fig shows the results training R. with and without the AugW and DP losses. As seen by compar- ing the results in rows 2 and of Fig 8, without those losses R fails to render the hallucinated geometric facial details. Next, we ablate using a single network in Tex2Im ('NR Single Branch') versus using two networks as described in Sect 3.3 ('NR Double Branch') )) versus using Ordinary Rendering using the provided texture space of BFM et al., 2018) 8). We calculate the FII) score (Heusel et al., 2017), and the FacelD distance using FaceNet (Schroff et al. 2015) across variety of expression edits. As can be seen in Table 2, using two networks, one that outputs the detail-invariant textures and the other that renders the details, gives better results as compared to using a single network. It not only produces measured by the FID score) but also more closely pre- serves the identity across a variety of expression edits. As it is expected, ordinary rendering performs the worst as its PCA space cannot capture the rich details of the human face across a variety of input identities and expression edits.
Input
Recvmttraation
Diaguet
Happy
DSL
w/oo AugW
and >
Decalled Shadligg
realistic images (as
Figure 8: Ablating Aug and DSL. We show that without AugW and DSL details are not rendered to the image space. (Please view in high resolution)
5 CONCLUSION AND FUTURE WORK
We have presented FaceDet3D, a method for hallucinating, from a single image, plausible facial geo- metric details the expression changes and render them photo-realistcclly. The details hallucination network is trained using adversarial losses and weak supervision without any ground-truth 3D data. The rendering network constrained using the Augmented Wrinkle Loss and Detailed Shading Loss, forcing it to use cues from the detailed geometry to render the details, ensuring their fidelity and consistency across a variety of expressions and views. The detail hallucination network relies on the detail map estimated by FDS (Chen et al., 2019) input to predict the plausible details of the target expression, therefore it cannot handle occlusions such as glasses or make-up very well FDS (Chen et al., 2019) fails in those conditions. In future work, plan to incorporate explicit disentanglement of lighting in order to enable greater control over the final rendering along with explicit modelling occlusions and joint training of DetH and R.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Niki Aifanti, Christos Papachristou, and Anastasios Delopoulos. The mug facial expression database. WIAMIS, pp. 1-4, 2010. 3,7
Athar, Shu, Samaras. Self-supervised deformation modeling for facial expression editing. In IEEE FG, 2020. 2,3
Volker Blanz, Thomas Vetter, et al. A morphable model for the synthesis of 3d faces. 1999.
James Booth, Epameinondas Antomakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, and Stefanos Zafeiriou. 3d face morphable models" in-the-wild". In CVPR. IEEE, 2017.
Aupei Chen, Zhang Chen, Guli Zhang, Kenny Mitchell, and Jingyi Yu. Photo-realistic facial details synthesis from single image. In ICCV, 2019. 2, 3, 5,
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In CVPR, 2018. 2
Jung-Woo
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
Pengfei Dou, Shishir K Shah, and loannis A Kakadiaris. End-to-end 3d face reconstruction with deep neural networks. In CVPR, 2017.
Ekman and W Friesen. Facial action coding system: A technique for the measurement of facial movement. In Consulting Psychologists Prexs, 1978. 2,
Yao Feng, Haiwen Feng. Michael J. Black, and Timo Bolkart. Learning an animatable detailed 3D face model from in-the-wild images. 2021. URL hwt.h:ttt///o/o oorg/10.... /00 444////450255 0.40069666349966.. 2, 7,8
Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William Freeman. Unsuper- vised training for 3d morphable model regression. In CVPR, 2018.
Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Berhhard Egger, Marcel Luthi, Sandro Schönbom, and Thomas Vetter. Morphable face models-an open framework. In JEEE FG, 2018. 3,5, 6,
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NewrlPS, 2014. 2, 7,
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017. 3
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. CVPR, 2017.
Aaron S Jackson, Adrian Bulat, Vasileios Argyriou, and Georgios Tzimiropoulos, Large pose 3d face reconstruc- tion from a single image direct volumetric enn regression. /CCV, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 7
Hyeongwoo Kim, Michael Zollhöfer, Ayush Tewari, Justus Thies, Christian Richardt, Christian Theobalt. Inversefacenet: Deep monocular inverse face rendering. In CVPR, 2018.
Haibin Ling and Kazunori Okada. Diffusson distance for histogram comparison. In CVPR, 2006. 6
Lars Mescheder, Andreas Geiger, and Sehastian Nowozin. Which training methods for gans actually converge? 2018. 4,
Albert Pumarola, Antonio Agudo, Aleix M Martinez, Alberto Sanfeliu, and Francese Moreno-Noguer. Ganima- tion: One-shot anatomically consistent facial animation. International Journal of Computer Vision, 128(3):
698 -13,713, 2020. 2,
Flad Richardson, Matan Sela, Roy Or-El, and R. Kimmel, Face reconstruction from single image. In CVPR,
2017. 2
10

---

Under review as a conference paper at ICLR 2022
Edgar Schonfeld. Bernt Schiele, and Anna Khoreva. A u-net based discriminator for generative adversarial networks. In CVPR, 2020.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015. 3,9
Matan Sela, Elad Richardson, and Ron Kimmel. Unrestricted facial geometry reconstruction using image-to-
image translation. ICCV, 2017. 2
Z. Shu, E. Yumer, S. Hadap, K. Sunkavalli, E. Shechtman, and D. Samaras. Neural face editing with intrinsic image disentangling. In CVPR, 2017. 2
Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hycongwoo Kim, Patrick Pérez, and Christian Theobalt. supervised multi-level face model learning for monocular reconstruction over 250 hz. In CVPR, 2018.
Ayush Tewari, Florian Bernard, Pablo Garrido, Gaurav Bharaj. Mohamed Elgharib, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer, and Christian Theobalt. Fml: Face model learning from videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10812 10822, 2019.
Justus Thies, M. Zollhöfer, and M. Nießner. Deferred neural rendering. ACM Transactions on Graphics (TOG). 2019. 2, 6
Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable model. In IEEE Computer Vision and Pattenn Recognition (CVPR), Salt Lake City, UT, June 2018.
Luan Tran, Feng Liu, and Xiaoming Liu. Towards high-fidelity nonlinear 3d face morphable model. In In Proceeding of IEEE Computer Vision and Pattern Recognition, Long Beach, CA, June 2019.
Anh Tun Tm, Tal Hassner, lacopo Masi, Eran Paz, Yuval Nirkin, and Gérard Medioni. Extreme 3d face reconstruction: Seeing through occlusions. In CVPR, 2018. 2
Job Van Der Schalk, Skyler T Hawk, Agneta H Fischer, and Bertjan Doosje. Moving faces, looking places: validation of the amsterdam dynamic facial expression set (adfes). Emotion, 11(4))907, 2011.
3,7
Yulun Zhang, Kunpeng Li, Kai Li. Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, 2018. 4
Jun-Yan Zhu, Tacsung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017. 2
Wenbin Zhu, Hsiang Tao Wu, Zeyu Chen, Noranart Vesdapunt, and Baoyuan Wang. Redarreinforeed differentiable attribute for 3d face reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.


Reviews:
[{"review_id": "mNBPec33oL", "reviewer": "Reviewer_6mwj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pos:\n+ The method produces faces of higher quality when analyses visually (based on the provided examples).\n+ While straight-forward, the proposes losses are sound and appear to work well.\n\nCons:\n- The novelty and scope appears limited and incremental. Perhaps the author could better explain&detail this aspect?\n- Little to no comparison against current state-of-the-art on 3D face reconstuction.\n- The evaluation metrics used are weak in the context of 3D face reconstruction. While the FID score is a good proxy metric for generated images in general it's unclear how representative it is in this scenario. Furthermore the more established point-to-point errors are missing.\n- The problem is quite niche, ideally one will want an approach than can in general synthesize high-fidelity models and texture. How can this method be adapted to a more general case? \n- What is the reason for using the HQ 4096x4096 data if the images are downsampled to 256x256? Is there a benefit over using 256px directly?\n- What is the impact of each proposes loss?", "summary_of_the_paper": "The paper proposes to improve the quality of the 3D reconstructed faces by taking into account the facial expression. Using a set of generative losses some of the details are recovered or hallucinated. The visual examples provided confirm the improvements, however better numerical evaluations and comparisons are required.", "summary_of_the_review": "The novelty and scope appear limited and the evaluation is incomplete. The paper could be improved by comparison against state-of-the-art and the use of more standard-metrics. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZCWWO2Kreq5", "reviewer": "Reviewer_eEoU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The motivation is very clear.\n* The methodology is well written.\n* The use of Age feature and FaceID feature is reasonable and novel\n* The idea of separating rendering into two branches is interesting.\n* AugW uses the augmented image with wrinkles to enforce the rendering of details.\n\nWeaknesses:\n* Lack of ablation study: 1) the “weakly supervise” mentioned in 3.2.1 is not mentioned in the ablation study. 2) The impact of Age and FaceID features is not studied in the ablation. 3) Is the mask necessary in Detail Hallucination Network? 4) the quantitative result from model w/o AugW and model w/o DSL (SEPARATELY) is not provided. \n\n* Based on Figure 5, 6 and 7, I still observe that quite a few details (i.e., wrinkles) that appear in the input are missing in your generated image. It suggests your model still fails to capture the important and significant details when there are substantial wrinkles on the face.\n\n* Performance study should be conditioned on (broken down according to) the key factors, such as age or emotions.  Obviously, the amount of deformation in the face (wrinkles) will increase with age.  Similarly, certain emotions/AUs result in more fine grained deformation than others.  \n\n* Comparison with DECA seems to be not fair, as you mentioned, DECA uses different morphable models, the reason may be that input is different.\n\n* There is only one compared baseline, DECA. More compared methods could be added.\n\n* It would be better to separate the contributions of the Detail Hallucination Network and Rendering Network. (e.g. Detail Hallucination Network+existing Rendering method)\n", "summary_of_the_paper": "They propose FaceDet3D to generate facial expressions with details. To do this, they propose a Detail Hallucination Network to generate the target detail map from the source detail map along with Expression Adversarial Loss [1] and Superresolution Losses. Then they propose a Rendering Network. In addition to Photometric Loss and Expression Adversarial Loss [1],  they add Augmented Wrinkle Loss and Detailed Shading Loss. The result is an image with target expression containing details (mainly wrinkles), comparison with DECA (a previous work) shows they improve FID and FaceID by large margin.\n\n[1] Conditional Image Synthesis With Auxiliary Classifier GANs\n", "summary_of_the_review": "While the authors design two GANs to generate the detail map and final image along with several loss terms, the authors still need to do a thorough and fair ablation study (how important is the “weakly supervise” mentioned in both Section 3.2.1 and Conclusion? How important is AugW and DSL separately?) and compare with fair previous works (How about comparing with models using the same input?).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "LsBH8h91a-6", "reviewer": "Reviewer_S1aQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is based on the Basel Face model and much depend upon initial texture and displacement map from Chen et al. The displacement map is trained for the AU but the AUs get changed for normal animation too (like when a person speak in Neutal emotion).Moreover, it also changes for a same person due to the degree of emotion. Because the Network is learning through the adversarial way through the correct generation of AU from the trained displacement map, the network can learn to deform without considering the person specific characteristics and average out the high frequency details. For example, in Fig 4, the last character’s initial face is in happy mood and the deformation on cheek from Chen et al. is quite good in rendering. But that deformation is not so prominent when this method is rendering happy given that initial frame and seems averaged out. But the AUs could be correct. The rendering seems losing the degree of deformation. To show the efficacy of detail displacement map, it would be better to see the result on a face where many wrinkles are present (like an aged person face) like the Fig 1 of Chen et al. Also, if we get a comparison what Chen et al. produce for an expressive face for a video frame (t) and what this method produces given only the AU for that face after initialising with Chen et al. output from an earlier frame (say t-5), then the method’s effectiveness for producing deformation with only AU  can be conceived better. \nRegarding the rendering, the concept of producing overall texture followed by addition of detailed texture is almost similar to DECA but the method is different here which utilises the power of neural texture map. As the displacement map is changing the normal of the face and Basel face has the linear albedo subspace, utilization of these two to render a face with many wrinkles(due to emotion) without using NTM can be shown in a figure to qualitatively see the effectiveness of NTM.\nThe comparison with DECA shows the clear advantage of this method. It would be nice to comment about the utility of FLAME vs Basel face model. FLAME’s PCA component and BASEL’s PCA components are different and hence the identity preservation of the DECA can get deteriorate. Is this improvement due to the method or the choice of the underlying face model? Similarly the expression subspace for FLAME is also different and the impact of that will be in the DECA’s rendering. So it may be better for a fair comparison of the method to state the equivalence or the impact of choice of the underlying 3D face model.\n", "summary_of_the_paper": "This paper proposes a pipeline to induce geometrical deformations due to expressions on a person’s 3D face. The existing works either uses single image based 3D reconstruction that manifest the expression present in image or learn a neural representation of the deformation (latent code) which can be used to transfer the deformation due to expression from one face to other. This paper uses direct expression parameters and action units instead of a latent code and predict the deformation on 3D face for an expression instead of manifesting it from image (unlike reconstruction methods).\nTo this end, the authors use an existing robust single image based 3D face deformation method (Chen et al.) to initialise the texture map and geometry as a displacement map of a 3D face (using Basel Face Model) from an image.  Because there is no ground truth of the new expression, they train it using adversarial loss. They produce the deformation using the Action Units(AU) as input and tries to produce those AU from the output displacement map.\nFor rendering, the method uses the Neural Texture which is trained with the texture map produced by the Chen et al. The rendering has two components. One is the coarse level rendering which utilises the shape and expression parameters of the Basel Face model and the detail rendering consisting of the information of the displacement map due to the AU obtained through this method. Both are combined to get the final appearance. \nThey compared the result with DECA (Siggraph 21) which is a FLAME 3D model based method and shows that the identity, shape and expression rendering is better in the proposed method.\n", "summary_of_the_review": "The pipeline of the paper can work, but the impact of the method and the novelty is not very clear as it depends heavily on Chen et al. and to some extent like DECA’s principle in rendering. As mentioned in the main review, the dependency of the Chen et al. needs to be critically evaluated as an initialization. Also, the choice of underlying 3D face model and impact of it in the outcome is also needed to understand the method’s utility in the comparisons. In the rendering, it seems the high frequency is getting lost. A few more results with more wrinkles in face would be better to judge the effectiveness of the rendering given a good initialisation from Chen et al. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G5NgQiUwgiA", "reviewer": "Reviewer_L964", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper proposed a two-stage pipeline that could generate the facial details of any target expressions and render it realistically using just one input image.\n2. A detail hallucination network was proposed and trained. \n3. The Augmented Wrinkle Loss and Detailed Shading Loss were proposed to train the rendering network.\n4. The authors evaluated the method quantitatively and qualitatively, also compared it with the DECA method. \n5. The authors further showed that the rendered results are view consistent, proving the effectiveness of the rendering network. \n6. Ablation studies are done to show the effectiveness of the two novel losses.\nWeaknesses:\n1. The readability of this paper can still be improved. For example, a pipeline figure would help in explaining the overall picture of the proposed method; alpha_s is not in the input of Equation (2) although it appears in Fig 2. \n2. The figures in this paper could be improved to show the details clearly. For example, in Fig 5, it seems that the lighting/shading of the two subjects' results are quite different, making it hard to see the facial details. The same problem also exists for Fig 4&6. Also,I would suggest the authors to show the zoom-in details in the rectangles as in Fig 1 rather than ask the readers to zoom in for details.\n3. The proposed method in this paper is not well-supported by experiments. First, it seems that the image quality of the rendered view is much more blurry than the input image and also the previous work [Chen et al. 2019]. Second, since the proposed method is built on [Chen et al. 2019], it would be fair to include this work for comparison as well reconstructed on the expression of the input image. Lastly, there're no evaluations of the usage of the age prediction network and FaceID embeddings. It would be interesting to know that age information make the proposed method outperforms the others in generating results for people across different ages, etc. ", "summary_of_the_paper": "1. This paper proposes a new method FaceDet3D that can generate geometric facial details from a single image given any desired target expression. The method contains two components: first pass the input image to the Detail Hallucination Network to infer the geometric facial details of the subject as there expression changes; then a rendering network is used to generate the detailed rendering result using the 3D face geometry information. \n2. The two component models are trained separately using a large scale in-the-wild images and a small video dataset. During the training of the rendering network, the novel Augmented Wrinkle Loss and Detailed Shading Loss were used. \n3. The authors further evaluated the method and compared it to DECA to show that the proposed method could generate plausible facial details for any desired expressions and could render the result photo-realistically.\n4. Ablation studies were also done to prove the effectiveness of the components in the proposed method. ", "summary_of_the_review": "This paper proposed a pipeline FaceDet3D that can generate plausible geometric facial details from a single image given any desired target expression. Novel losses were proposed and the model was trained and evaluated to show its effectiveness. \nHowever, I think this paper is still under the bar of acceptance as the experiments are not very convincing. Also, the readability of the paper could still be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers raised a number of major concerns including a poor readability of the presented materials, incremental novelty of the presented and, most importantly, insufficient and unconvincing ablation and experimental evaluation studies presented. The authors’ rebuttal failed to address all reviewers’ questions and failed to alleviate reviewers’ concerns. The authors explain that due to the lack of time they could not complete all experimental studies. A major revision of the paper is needed before the paper can be accepted for publication. Hence, I cannot suggest this paper for presentation at ICLR.
Decision: Reject