Semi-supervised 3D Face Reconstruction with Nonlinear Disentangled Representations
Authors: Zhongpai Gao
Juyong Zhang
Yudong Guo
Chao Ma
Guangtao Zhai
Xiaokang Yang
Venue:  (2020)
Keywords: 3D face reconstruction
semi-supervised learning
disentangled representation
inverse rendering
graph convolutional networks

Abstract: Recovering 3D geometry shape, albedo and lighting from a single image has wide applications in many areas, which is also a typical ill-posed problem. In order to eliminate the ambiguity, face prior knowledge like linear 3D morphable models (3DMM) learned from limited scan data are often adopted to the reconstruction process. However, methods based on linear parametric models cannot generalize well for facial images in the wild with various ages, ethnicity, expressions, poses, and lightings. Recent methods aim to learn a nonlinear parametric model using convolutional neural networks (CNN) to regress the face shape and texture directly. However, the models were only trained on a dataset that is generated from a linear 3DMM. Moreover, the identity and expression representations are entangled in these models, which hurdles many facial editing applications. In this paper, we train our model with adversarial loss in a semi-supervised manner on hybrid batches of unlabeled and labeled face images to exploit the value of large amounts of unlabeled face images from unconstrained photo collections. A novel center loss is introduced to make sure that different facial images from the same person have the same identity shape and albedo. Besides, our proposed model disentangles identity, expression, pose, and lighting representations, which improves the overall reconstruction performance and facilitates facial editing applications, e.g., expression transfer. Comprehensive experiments demonstrate that our model produces high-quality reconstruction compared to state-of-the-art methods and is robust to various expression, pose, and lighting conditions.


Under review as a conference paper at ICLR 2020
SEMI-SUPERVISED 3D FACE RECONSTRUCTION WITH NONLINEAR DISENTANGLED REPRESENTATIONS
Anonymous authors Paper under double-blind review
Input image
Shape, lighting, albedo
Reconstructed face
Expression transfer
Our network decomposes input image into shape, lighting, and albedo with four disentangled representations: identity, expression, pose, and lighting, which allows expression transfer between different face images.
ABSTRACT
Recovering 3D geometry shape, albedo and lighting from a single image has wide applications in many areas, which is also a typical ill-posed problem. In order to eliminate the ambiguity, face prior knowledge like linear 3D morphable models (3DMM) Iearned from limited sean data are often adopted to the reconstruction process. However, methods based on linear parametric models cannot general- ize well for facial images in the wild with various ages, ethnicity, expressions, poses, and lightings. Recent methods aim to learn nonlinear parametric model using convolutional neural networks (CNN) to regress the face shape and texture directly. However, the models were only trained on a dataset that is generated from a linear 3DMM. Moreover, the identity and expression representations are entangled these models, which hurdles many facial editing applications. In this paper, we train our model with adversarial loss in a semi-supervised manner on hybrid batches of unlabeled and labeled face images to exploit the value of large amounts of unlabeled face images from unconstrained photo collections. A novel center loss is introduced to make sure that different facial images from the same person have the same identity shape and albedo. Besides, our proposed model dis- entangles identity, expression, pose, and lighting representations, which improves the overall reconstruction performance and facilitates facial editing applications, e.g., expression transfer. Comprehensive experiments demonstrate that our model produces high-quality reconstruction compared state-of-the-ar methods and is robust to various expression, pose, and lighting conditions.
INTRODUCTION
3D face reconstruction from 2D images enables many exciting applications, such as face recognition
[laazz& & Vetter 2003 Paysan et al. 2009 Liu et al. 2018), face puppetry (Cao et al.) 2014), face reenactment (Thies et al. |l.[[11..00 2016 Garrido et al. 2015). virtual make-up (Li et al. 2015), etc. However, 3D face shape and texture inference from 2D images, especially from single image, an ill-posed problem since some 3D information is lost after the imaging process. 3D morphable model (3DMM)
(Blanz & Vetter 1999] learned from collection of 3D face scans is often adopted strong

---

Under review as a conference paper at ICLR 2020
prior assumption for this problem. 3DMM is linear combination of bases to provide statistical parametric representation of 3D faces. Given a 2D image, the conventional approach is to search maulysi--b--sytheee
optimization (Levine & Yu, 2009 Booth et al., 2018). Specifically, a 3D face is generated through inverse rendering to match the 2D image by optimizing the shape, albedo (i.e., texture separated from illumination conditions), pose, and lighting parameters. However, such 3DMM optimization- based methods are usually time- consuming due to high optimization complexity and suffer from local optima solutions.
for the corresponding 3DMM parameters through
Regressing 3DMM parameters using convolution neural network (CNN) shows remarkable success in 3D face reconstruction (Richardson et al. 2016 Zhu et al. 2019, Genova et al., 2018 Wu al. 2019). However. these methods cannot go beyond but only search for a solution in the restricted linear low-dimensional subspace of 3DMM. Linear statistical models have limitations to construct 3D face shapes and textures. First, facial variations are nonlinear in the real world, e.g., various ethnic groups, ages, facial expressions, and skin colors. Second, order to model highly variable 31) face, a large amount of 3D face scans are needed for training. The most popular 3DMM (Xiangyu Zhu et al. 2015) was built by merging Basel Face Model (BFM) (Paysan et al., 2009) with only 200 subjects in neutral expressions and Face Warehhoee (Cao et al. 2014) with 150 subjects in 20 different expressions, which is not able to fully capture the variability of human faces. A large scale facial model (LSFM) was constructed by Booth al. (2016) from around 10,000 distinet facial identities but only in neutral expressions. Tewari et al. (2018). Tran et al. (2018), and Guo al. (2119) further proposed 3D face models composed of two networks: a coarse-scale linear 3DMM network and a fine-scale corrective network. Even though the finle-scale corrective model can generate more details, 3D face reconstruction will fail if the foundation face shape generated by the linear 3DMM network is not good enough.
Recently, Tran & Liu (2018) and Tran et al. (2019) proposed encoder-decoder networks to regress the face shape and texture directly. The nonlinear networks have higher representation power com- pared a linear model and are able to reconstruct high-fidelity facial texture. However, the nonlinear models were only trained on the 300W-LP dataset (Zhu et al., 2016) that is generated from linear 3DMM with a face profiling technique. The models were further fine-tuned in a self-supervised manner the same dataset. However, since most of the face images were synthesised based on the linear 3DMM, self-supervised training to reconstruct high-fidelity texture using inverse rendering makes limited contributions to the face shape reconstruction. Besides, in these methods, the face albedo and face shape are decoded from a albedo parameter and shape parameter separately with- out considering the facial identity. In fact, across one's different face images, the face albedo and identity shape should only depend on the facial identity, i.e., sharing the same identity represen- tation. Learning albedo and shape parameter separately is difficult disentangle the face albedo from lightings and occlusions. Especially, when the albedo decoder network has high representation power, the albedo decoder may reconstruct high-fidelity face albedo but without aligning with the face shape and fails to contribute to the face shape reconstruction. At last, the identity and expres- sion representations are entangled in these methods and many applications, such as face recognition, face animation, and face reenactment, are not feasible.
In this paper, we propose a novel encoder-decoder architecture using inverse rendering that com- bines computer vision and computer graphics techniques. The vision system (i.e., encoder network) decomposes an input 2D face image into disentangled and sematic representations: identity code, expression code, pose code, and lighting code. The graphics system renders back a face image to match the input image based on the decoder networks that regress the 3D face shape and albedo from the extracted representations. Combining computer vision and computer graphics techniques pro- vides a unique opportunity to leverage the vast amounts of readily available unlabelled face images from unconstrained photo collections through self-supervised learning.
Since 3D face reconstruction from a 2D image is ambiguous and posepd self-supervised learning with unlabelled data through inverse learning is not sufficient. In this paper, we train the network in a semi-supervised manner on hybrid batches of large amounts of unlabeled face images and relatively small amounts of labelled face images that are generated from a linear 3DMM with optimization- based methods. Moreover, following the idea of generative adversarial networks (GAN) (Goodfel-) low et al., 2014), a discriminator network is used to ensure the reconstructed face shape not too far away from the distribution of human face. Semi-supervised adversarial training not only prevents our model from generating unrealistic 3D face shape but also fully exploits the value of unlabeled face images without being constrained by the pre-existing linear 3DMM.
2

---

Under review as a conference paper at ICLR 2020
graph convolutional network (GCN) (Defferrard et al. 2016 Kipf & Welling 2017) instead of fully connected layers with activation or CNN used in Tran & Liu (2018) and Tran et al. 2019). A 3D face shape is usually modeled as mesh that is defined by a collection of vertices, edges, and faces and is considered as an unstructured graph. Modeling graph convolutions on 3D meshes be memory efficient and allows for processing high resolution 3D structures, GCN-based methods to reconstruct 3D face shapes outperforms other state-of-the-art methods (Ranjan et al. 2018, Jiang et al., 2019 Bouritsas et al., 2019). To recover the 3D face albedo, we first use GCN network that has the same architecture with the shape decoder to learn an cotmmiinttio--nndeeeeeee
To reconstruct the 3D face shape,
face albedo. Then we apply CNN-based decoder network that has skip connections with the encoder network (Ronneberger al. 2015) and patchGAN (Shrivastaval al. 2017) to improve the details of the facial texture.
We apply a face recognition loss and a center loss (Wen et al. 2016) to extract the identity represen- tation (i.e., facial identity) from one's unconstrained multiple face images. The center loss used to ensure the identity representation's compactness for each person and separability for different people, so that the identity representation is disentangled from the pose, lighting, and expression representations. In order to further disentangle the identity and expression representations, pairwise training approaches are adopted. Given a pair of labelled face data, we keep the identity codes and interchange the expression codes of 3DMM generate new 3D shapes supervision. Comprehen- sive evaluation experiments show that the proposed method achieves state-of-the-an performance in 3D face reconstruction and can easily be used for the applications of face recognition and facial expression transfer. The main contributions of this paper are summarized below:
We propose an efficient semi-supervised and adversarial training process to fully exploit the value of unlabelled face data and beyond the limitation of linear 3DMM.
We design a novel framework to exact nonlinear disentangled representations from a face image with the help of face recognition losses and shape pairwise loss.
Extensive experiments show that our model achieves state-of-the-ard performance in face reconstruction.
BACKGROUND
This section describes some background information related to our work, including face represen- tations in conventional linear 3DMM, face rendering process, and graph convolution used in face shape reconstruction.
Linear 3DMM We first recap the conventional linear 3DMM. As described in Chu et al. 2014), the linear 3DMM constructed from facial scans via PCA can be expressed as:
s + Ana + Aezy
(1)
where 8 E R3N x1 is a 3D face shape with N vertices, 8 E x is the mean shape, Aid E R is the first K principle components trained facial scans with neutral expression and and R is the identity parameter, Aezp E 3R x is the first L principle components trained on the offset between neutral scans and expression scans and Ceep E Mx1 is the expression parameter.
The texture of 3D face can also be modeled via PCA as:
ビ=ビ-
Ater
(2)
where & E R is a 3D face texture, E RR3Nx1 is the mean texture, Atez E 3R is the first M principle components trained on facial textures and Qtez R Rx111 is the texture parameter.
Rendering process The 3D face modeled by 3DMM is projected onto a image plane with weak perspective projection:
S2D = f Pr R 8 t2D,
1
0 0
where 82D 2 is the face shape located on the image plane after projection, Pr
0
is the orthographic projection matrix, R is the rotation matrix constructed from Euler angles (i.e., pitch, yaw, and roll), t2p = [t,,,t]] [ is the translation vector on the image plane, and f is the scale factor.
3

---

Under review as a conference paper at ICLR 2020
Following Guo al. (2019), assume the face is Lambertian surface and the global illumina- tion is approximated using the spherical harmonics (SH) basis function. The first three bands of SHs are used for the illumination model. Y E R is the illumination parameter for the RGB channels' SH illumination coefficient. Thus, the rendering process depends on the parameter set X (Oad, Geup, Otexr Stexr,itchh, yaw, roll, f, 72
Spectral graph convolution As presented by |Ranjan et et. (0188), we use spectral graph convolution reconstruct 3D face shapes. The shape of 3D face is described as triangular mesh M = (V, A), where V E Raxs denotes the n vertices in the Euclidean space, A E {0, 1 is the sparse adjacency matrix representing the edge connections. The non-normalized graph Laplacian is defined as L D 2-D-4 A, where the degree matrix D is diagonal matrix with D.. = E, Ar,-- Spectral graph convolution is defined on the graph Fourier transform domain, whose bases are the eigenvectors of the Laplacian matrix. An efficient solution for spectral graph convolution is formulating mesh filtering with kernel using recursive Chebyshev polynomial,
Pin K
X Xout.
0
(4)
where X Xout, is the jth feature of the output Xout RRxF and Xin, is the 8th feature of the input Xin E RixPin, c.g., the input mesh vertices V has Fin. = features corresponding to the I, is the scaled Laplacian. T* E R is the Chebyshev polynomial of order & that is computed recursively as Ta (x) = 2xThx- - (I)) with To and Ti r. The parameter 0 IR Fin x F xK is the trainable Chebyshev coefficients.
3D vertex position. L = 2LL/mmar
METHOD
We design an encoder-decoder architecture that allows ene-to-end semi-supervised adversarial train- ing to extract disentangled semantic representations of single image, as shown in Figure D We adopt inverse rendering technique that utilizes parameterized illumination model and differentiable renderer to render back the input face image under varying identity, expression, pose, and lighting conditions. Our model is trained on hybrid batches of unlabeled face images from CelebA (Liu et al. 2015) and labeled face images from 300W-LP (Zhu et al. 2016
Recog loss
Symm
Joss
CCN Albedo decoder
Recon
Loss
Identity
CNN Encoder
GCN
Shape decoder
layer
Expression
Shp loss
Pair loss
Av loss
.mk
Smih
Pose
loss
loss
Lighting
Figure 1: Framework overview. The encoder network takes an input face image and extracts four disentangled representations: identity code (Cız), expression code (Ceap), pose code (Cpose), and lighting code (Ctge). The albedo decoder network reconstructs the face albedo from the identity code. The shape decoder network reconstructs the face shape from the combination of the identity code and expression code. The rendering layer takes the face albedo, face shape, pose, and lighting to render back the face image. Multiple losses are applied on our network. Losses in gray rectangles are only used on labeled face images and in green rectangles are used on all face images.
4

---

Under review as a conference paper at ICLR 2020
3.1 ENCODER-DECODER NETWORK
Encoder As shown in Figure I the encoder network is a multi-task learning network, which takes face image as input and extracts its identity, expression, pose, and lighting representations. A pre- trained ResNet-50 network is used as the backbone of the encoder network. The ResNet-50 network followed by four branches of fully connected layers with outputs of 128-D identity code (cua), 64-D expression code (Ceup). 6-D pose code (Cpose), and 27-D lighting code (Cigt).
Shape decoder The shape decoder network is graph convolutional network modified from the COMA architecture (Ranjan et al. (011)) with an extra graph convolutional layer and up-sampling layer at the beginning. We concatenate the identity code and expression code extracted from the encoder network, i.e., a 192-D vector, as the input of the shape decoder network. The output of the shape decoder the corresponding 3D face shape in the standard position (i.e., without any translations rotations). We denote FC(d) fully connected layer, 1 the number of vertices after the last down- wwmn-sampinn layer, GC'(k w) a graph convolutional layer with k kernel size and w filters, and U Sp) a up-sampling layer by factor of p. respectively. The shape decoder network listed follows: FC * 256) + (2) + GC 256) - (S( + GC(6, 128) - US(4) > GC(6, 64) - US) GC 32) US(4) - GC(6, 16) - GC(6,3).
Albedo decoder The albedo decoder network is also a graph convolutional network and has the same architecture the shape decoder. The albedo decoder takes only the identity code input since the albedo of a face should be independent of the expression, pose, lighting, and occlusions. Importantly, hair, glasses, microphones, and other facial occlusions should not be included in the albedo since one's 's facial albedo should be consistent across his different photos even with different hair styles, glasses, etc. We apply face segmentation by Nirkin et al. (2018) eliminate the effect of facial occlusions. Note that, we did not consider aging. injury, or other factors that may affect one's s face albedo.
After the lighting representation is learned, we change the GCN-based albedo decoder network to CNN network that has skip connections with the encoder network to improve the details of the facial texture. The architecture of the encoder and CNN-based albedo decoder with skip connections is similar to U-Net (Ronneberger et al., 2015). Moreover, we apply a patchGAN (Shrivastava et al, 2017) to further make the facial texture more realistic.
3.2 LOss FUNCTIONS
Our network is trained with a multi-task loss that enable us to regress the 3D face shape and albedo end-to-end. The loss function combines face recognition loss, face reconstruction loss, pairwise shape loss, adversary loss, and other regularization.
Face recognition loss In order extract the identity code that only represents the photo's facial identity, we apply face recognition loss as follows:
(5) where Leoft is the softmax loss that classify each photo to a specific identity class, Leenter is the center loss to improve the discriminative power of the deeply learned identity code (Wen et al. 2016), and Acenter is used for balancing the two loss functions. Face recognition loss is essential to leam the facial identity without being influenced by other factors such facial expressions, poses, lightings, occlusions, etc.
Lrecag = Lsoft AventerLeenterr.
Face reconstruction loss The rendering layer renders back an image to compared with the input image. The face reconstruction loss is formulated as
Lreeon = M @ (I - 1 Lggdt,oooor)
(6)
where @ is the element wise Hadamard product, I is the input image, Î is the rendered image, and M is the mask obtained by Nirkin et al. (2018) to climinate the effect of facial occlusions such as hair, glasses, and microphone. Morcover, image gradient difference loss (GDL) (Mathieu et al. 2015), denoted as Lodlocolor, is applied to recover more details in the reconstruction.
Sparse landmark loss We add sparse landmark loss to help learn the face pose and achieve better face reconstruction. The sparse landmark loss is defined as
Link
|| I:, L) U1 bmak:
5

---

Under review as a conference paper at ICLR 2020
where 82D is the projected face shape from network, L is the vertex indexes of the 68 landmarks in the 3D face shape, U is considered as the ground truth of the corresponding sparse 2D landmarks on the input image and is obtained by Bulatt & Tzimiropoulos (2017). The idea of GDL is also applied on the sparse landmarks, denoted as Lyddt,imk. which describes the distance of two different landmarks should also be close to the corresponding distance in ground truth. Especially, it is important for the distances of the upper eyelids to the lower eyelids and the upper lip to the lower lip that represent the conditions of eye's 's opening and mouth's 's opening, respectively.
Shape loss In order to prevent the network from either generating unrealistic 3D face shapes or being under the constrain of a linear 3DMM, we train our network semi-supervised manner on hybrid batches of unlabeled and labeled face images. For the labeled face images, we choose 300W-LP dataset that contains 122,450 images with fitted 3DMM shapes across large poses and was created by /Zhu et al. (2016) with face profiling technique. The BFM template that has 53,215 vertices is used for the fitted 3DMM shapes. The 3DMM parameters Ceepp and Ceep are provided to calculate each of the fitted 3DMM shapes, presented in Eq. (I). In this paper, we remove the neck and cars of the BFM model to create our own face shape template with 37,202 vertices. The shape loss for the 300W-LP dataset is formulated
(8) where s s + Aid + Aerp is considered as the ground truth of the face shape, is the 3D face shape reconstructed by our network, and T is the vertex indexes of our face template in the BFM model.
skkp = || s|::7]ll:
Pairwise shape loss To further disentangle the identity code and expression code, we train the 300W-LP dataset in pairwise manner. Given an input image, the corresponding 3DMM parameters Keep and aeep are provided. For pair of input images, IA and Lg, interchange the expression parameters Ceerp.A and Cerp. B to get the 3D face shape of A's identity with B's expression. The pairwise shape loss for the 300W-LP dataset is expressed as
(9) where Sshape (.) is the shape decoder, [Cad,A, Gerp.Bl means concatenation of A's identity code and B's expression code from the encoder network, and AA.B 8 + A + AempOezp is the 3DMM shape of A's identity parameter with B's expression parameter.
Lasrr = (ishapp(Ccc.... Cezp. Bl) - sABlE 71l1.
Shape smooth loss Laplacian regularization is used on the shape vertex to help remove undesired noise of 3D face shapes. Conventional Laplacian smoothing assumes all the vertices satisfy the equation X; M.. ΣjEM, X,. where X, is the ith vertex and M, is the vertex indexes of the first order neighbors of X, However, some vertices, like on the edges, in the nostrils, at the eye corners, etc, do not satisfy the Laplacian equation. We calculate the difference of each vertex with the mean of its first order neighbors bo be close to the corresponding difference of the shape template,
M Lemth E(3 iEN
1
Σ
8,)
(3)
Σ 3,)1
(10)
[M
jEM,
jeM:
where @ is our face shape template cropped from the BFM model.
Albedo symmetry loss Facial symmetry is strong prior for face albedo learning. which helps to disentangle facial expression, lighting, and occlusions from the face albedo. The albedo symmetry loss is defined as
Leyma 1|A - flip( A)
(11)
where A is the output face albedo of the GCN-based albedo decoder and flip(.) is an operation of flipping face albedos left and right.
Adversarial loss Semi-supervised learning is not sufficient to generate realistic 3D face shape for the unlabeled face images. Following the idea of generative adversarial network (GAN), an adversarial loss is used to train the encoder-decoder network and a discriminator network alternatively based on WGAN-div (Wu al. (018)). The discriminator network D is GCN-based encoder network and is used to discriminate the fake shapes (i.c., shapes reconstructed from our network) and real shapes (i.e., shapes sampled from the linear 3DMM), so that the reconstructed face shapes will not

---

Under review as a conference paper at ICLR 2020
be too far away from the distribution of the linear 3DMM. The min-max optimization problem can be written as
min max E [D(8)] G D 8~P9
E [[s]: 7i) - A E [v)1 =|.77~
(12)
&-PF.
where Lode - (88) is the adversarial loss, 8, s|; T] are the fake and real face shapes satisfying the probability measures Fg. pr, and Pu is the distribution obtained by sampling uniformly along straight lines between points from the real and fake face shape distributions.
EXPERIMENTS
In this section, we first conduct ablation tests to demonstrate the effectiveness of the framework design (Section 4.1). We then evaluate our method by comparing reconstruction error against 3D face scans with state-of-the- expression transfer based on the disentangled representations of our model (Section
approaches (Section 4.2) At last, we present the application of
We train our model on hybrid batches of unlabeled face images from CelebA dataset (Liu et al. 2015) and labeled face images from 300W-LP dataset (Zhu et al., 2016). MICC Florence dataset (Bagdanov al. 2011) and AFLW2000-3D dataset (Zhu et al.j 2016)) are selected for the quantita- tive and qualitative evaluations. The face region of the BFM model is cropped the 3D face mesh template (i.e., 37202 out of the 53215 vertices). The model and the discriminators are optimized using Adam optimizer with learning rate of 0.0001 and RMSprop optimizer with learning rate of 0.00005, respectively.
4.1 ABLATION STUDY
Shape reconstruction We study the effects of shape smooth loss and adversarial loss on the quality of shape reconstruction, shown in Figure 2 Since face model is not constrained by pre- existing linear 3DMM, the face meshes can potentially be deformed to any shapes. The conventional smoothing loss causes abnormal effects on the edges and nostrils of face shapes. The vertices the mouth's 's inner edge distance away from their neighbors. The nostrils are prone to be flat or even sticking out of the nose. This is because the vertices the edges and nostrils the Laplacian regularization which forces each vertex locates at the mean of its first order neighbors. When the model is trained without the adversarial loss, the forehead and two sides of face meshes are shrunk and eyebrows extrude out. The adversarial loss can make sure the face shapes generated by our model will not be too far away from the shape distribution of human face, while which unknown and pre-created linear 3DMM is used in this paper.
satisfied with
Conventional smooth loss
Without Adv Loss
Input
Full model
Shape ablation test showing failures caused by changing to conventional Laplacian smoothing loss and removing the adversarial loss.
Figure 2:
Texture reconstruction Figure shows the effects the albedo symmetric loss with facial mask. We consider the albedo symmetric loss and facial mask together because the facial occlusions should be masked out first in order to apply the albedo symmetric loss. The facial mask with albedo sym- metric loss is crucial for lighting representation learning. Otherwise, the shade and lighting may be confounded with facial occlusions. Especially, when the representation power of the albedo decoder high, e.g., CNN-based albedo decoder with skip connections to the encoder, the model will fail to
7

---

Under review as a conference paper at ICLR 2020
learn the lighting though the generated texture looks very close to the input image, shown in the last column of Figure 3 However, without learning the lighting, reconstructing high fidelity texture makes limited contributions to the face shape reconstruction because the high fidelity texture may not align with the face shape and looks odd when changing to a different pose. Facial mask with the albedo symmetric loss helps disentangle the lighting from the albedo. When the lighting is learned, CNN-based albedo decoder with skip connections to the encoder is used to improve the detail of facial albedo.
GCN albedo Without mask +symm loss
GCN albedoo With mask +sym loss
CNN+bridge With mask +sy mm loss
+bridee Without mask +ask++ymmm loss
Input
Figure 3: Texture ablation test showing failures of lighting caused by removing facial mask (i.e., mask the facial occlusions) and albedo symmetric loss. We denote facial mask with albedo symmetric loss as mask+symm loss, GCN-based albedo decoder as GCN albedo, and CNN-based albedo decoder with skip connections as Crideridge.
4.2 COMPARISONS TO THE STATE-OF-THE-AN
We evaluate our model quantitatively on the MICC Florence dataset (Bagdanov et al. 2011), which contains the ground truth scans of 53 subjects in neutral expressions. Each subject recorded in three videos: Cooperative, Indoor, and Outdoor with increasingly challenging conditions. Follow- ing the setting in Wu et al. (2019),
left, frontal, and right view of each subject are selected from the Cooperative and Indoor videos. The predicted 3D face shape is obtained by averaging over the 3D face shapes reconstructed from the left, frontal, and right view. The evaluation matric follows Genova al. (2018) where cropped the face region of 95mm around the nose tip of the ground truth scan to calculate the point-to-plant to--lane L2 errors with the predicted face shape.
Cooperative Mean Std.
Indoor
Subject
Ours
Wu19
Tran17
Method
Mean Std.
Tuan
Tran et al.
1.397 0.290
L.381 0.322
(2017)
No.
1.12:0933
Tewari et al. (2017)
1.370 0.321
1,286 0.266
Genova et al. 2018)
1.372 0.353
1,260 0.310
Wu al. (2019)
1.220 0.247
1,228 0.236
Ours
1.163 0.295
1.238 0.302
No. 22
1.30 .99
1,303el m
1.65e1.199
Table
Mean error comparison on MICC dataset
Figure Examples of error map comparison
0.776
0.688A0.532
0.523401775
0,99220038
Input
Reconstraciio
Lighting
Shape
Emror mag
Input
Reconsinccion
Lighting
Shape
Emor map
Figure 5: Examples with different lightings and poses of subject No. 05 from the MICC dataset. A and B are from the video of Cooperative. C and D are from the videos of Outdoor and Indoor, respectively.
8

---

Under review as a conference paper at ICLR 2020
Table shows that the proposed method outperforms other single-view reconstruction methods. Compared to the multi-view reconstruction method (Wu et al. 2019). we achieve better results in the Cooperative condition and have slightly worse results in the Indoor condition. Figure presents two examples (i.e., subject No. 53, and subject No. 22) of detailed error maps. Figure 5 shows the reconstruction results of face images from the same subject (No. 05) in the Cooperative, Indoor, and Outdoor videos with different lightings and poses. The reconstruction errors are small across different conditions.
We further evaluate our model qualitatively on the AFLW2000-3D datasets Zhu et al. 2016) Tewari al. (2018) and Tran et al. (2018) both proposed two-stage models: a coarse-scafe lin- ear model and fine-scale corrective model. Even though the fine-scale corrective model is able to add more details on top of the linear model, the reconstructed face shape will fail when the foun- dation face shape generated in the first stage is not good enough. The foundation face shape is restricted by the linear 3DMM and cannot generalize well in the wild conditions with true diversity of poses, expressions, lightings, and occlusions. As shown in Fig. 6 the face shape reconstructed by our model has better alignment with the input face image and looks more realistic from the frontal view. Moreover, compared with Tewari et al. (2018), the proposed method reconstruct the facial texture in more detail.
Overly
Shape
Albedo
Lightngg
Frontal shape
Ours
[Tewa118]
Ours
Tewan 18
Tran18
Figure 6: 3D reconstruction comparisons with Tewari et al. (2018) and Tran et al. 2018
Tran al. 2019) proposed nonlinear 3DMM and is the most related work our work. The face shape and albedo are reconstructed from eNsed decoders and have higher representation power compared linear 3DMM. However, the model was trained on 300W-LP dataset. Even with higher representation power, the nonlinear model is limited to fit the 300W-LP dataset generated from linear 3DMM. Moreover, the identity and expression of face shape are entangled, resulting in poor performance shapes reconstructed by Tran et al. (2019) tend to have smaller mouth opening and some artifacts are introduced to the face shapes and textures in challenging conditions. The proposed model achieves better performance across various conditions: exaggerated expressions, large poses, diverse lighting, and different occlusions presented in the figures.
face images with diverse expressions. As shown in Figure 7 the face
Overlay
Shape
Ouus
Tran99
Figure 7: 3D reconstruction comparisons with Tran et al. 2019
4.3 APPLICATIONS
Disentangled representations of our model not only can improve the performance of face reconstruc- tion, but also can facilitate many facial editing applications, such face recognition, face puppetry,

---

Under review as a conference paper at ICLR 2020
face replacement, face reenactment, expression transfer, and forth. Figure g demonstrates the function of expression transfer between different face images. We keep the face image's ' identity representation and replace the pose, lighting, and expression representations from another face im- age to generate a realistic new face image with the same identity but another face's pose, lighting, and expression. When we apply the expression transfer different images of the same person, the results are consistent after the expression transfer, demonstrating high robustness of our model.
Expression transfer between different people
Expression transfer between the same
Figure 8: Expression transfer between different face images. The left side is the expression transfer between different people and right side is the expression transfer between the same persian.
CONCLUSION
This paper proposes an encoder-decoder architecture to reconstruct 3D face from a single image with disentangled representations: identity, expression, pose, and lighting. We develop an effective semi-supervised training scheme to fully exploit the value of large amount of unlabeled face images from unconstrained photo collections. An adversarial loss is applied to prevent model from generating unrealistic 3D faces. We evaluate our model quantitatively and qualitatively. Our model outperforms the state-of-the-art single-view reconstruction methods and can effectively disentangle identity, expression, pose, and lighting features.
REFERENCES
Andrew D Bagdanov, Alberto Del Bimbo, and lacopo Masi. The florence 2d/3d hybrid face dataset. In Proceedings of the 201 joint ACM workshop on Human gesture and behavior understanding, pp. 79-80. ACM, 2011.
V. Blanz and T. Vetter. Face recognition based fitting a 3d morphable model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1563)1003311774 Sep. 2003. ISSN 0162-8828. doi: 10. 10.1109/1 1099TPAMM..000.122222 12279988.....
Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Proceed- ings of the 26th Anmual Conference on Computer Graphics and Interactive Techniques, SIG- GRAPH '99, pp. 187-194, New York, NY, USA, 1999. ACM Press/Addison-Wesley Publishing URL
Co. ISBN 0- 20-2001-45000 485600-5. doi:
1145 2145/311535.225 311556
Booth, A. Roussos, S. Zafeiriou, A. Ponniahy, and D. Dunaway. A 3d morphable model learnt from 10,000 faces. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5543-5552, June 2016. doi: 10.1109/CVP..201..550
J. Booth, A. Roussos, E. Ververas, E. Antonakos, S. Ploumpis, Y. Panagakis, and S. Zafeiriou. 3d reconstruction of in-the-wild faces in images and videos. IEEE Transactions on Pattern Analysis and Machine :0011) 2018.2832138.
40(11):0633))) 2652, Nov 2018. ISSN 0162-8825 doi: 10.1109/TPAMI.
Giorgos Bouritsas, Sergiy Bokhnyak, Stylianos Ploumpis, Michael Bronstein, and Stefanos Zafeiriou. Neural 3d morphable models: Spiral convolutional networks for 3d shape representa- tion learning and generation. In The IEEE International Conference on Computer Vision (ICCV), 2019.
10

---

Under review as a conference paper at ICLR 2020
Adrian Bulat and Georgios Tzimiropoulos. How far we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In The IEEE International Conference Computer Viston (ICCV), Oct 2017.
C. Cao. Y. Weng, S. Zhou, Y. Tong, and K. Zhou. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics, 20(3):413- 425, March 2014. ISSN 1077-2626. doi: 10.1 0.11099/VCG:2203.24499
Chen Cao, Qiming Hou, and Kun Zhou. Displaced dynamic expression regression for real- time facial tracking and animation. ACM Trans. Graph., 33(4))43:--43:10, July 2014. ISSN 0730-0301. doi: 10.1 11442660199726612222 2601097. 2601204
URL
Baptiste Chu, Sami Romdhani, and Liming Chen. 3d-aided face recognition robust to expression and pose variations. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2014.
Michaäl Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 3844-3852. Curran Associates, Inc., 2016.
P. Garrido, L. Valgaerts, H. Sarmadi, I. Steiner, K. Varanasi, P. Prez, and C. Theobalt. Vdub: Modify- ing face video of actors for plausible visual alignment to dubbed audio track. Computer Graph- ics Forum, 34(2): 193 2015. doi: 10. 11 11111/// 11ll......22... 12552. Utthttpps: http::::////iineleeeeeeee wiley. www. doii http//bs/11100 www.aa/.........aa..aaaaaaeaeeee egf. 12552
Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T. Free- man. Unsupervised training for 3d morphable model regression. In The IEEE Conference Computer Vision and Puttenn Recognition (CVPR), June 2018
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems 27, pp. 2672-2680 Curran Associates, Inc., 2014. URL http:://appen nips. ctppscccppppe///ccec 5423 generative htts://aapeess2eeeeerrrrrrreee
adversarial nets .dd.
Y. Guo, j. zhang, Cai, B. Jiang, and J. Zheng. Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(6))129411307, June 2019. ISSN 0162-8828. doi: 10.1109/TPAMI. 2018.2837742.
Zi- Zang Jiang, Qianyi Wu, Keyu Chen, and Juyong Zhang. Disentangled representation learning for 3d face shape. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR), 2017.
Martin D. Levine and Yingfeng (Chris) Yu. State-of-the-art of 3d facial reconstruction meth- ods for face recognition based on single 2d training image per person. Pattern Recog-
nition Letters, 30(10)9908 913, 2009. ISSN 0167-8655. doi: https://doiorr//0..100
2patrec.2000.03.011 URL http:// wttpww.wciiee..cccmmee... 011/0167855090055
wtt.///m/cccceeeeeee com./seience//rticle/
Chen Li, Kun Zhou, and Stephen Lin. Simulating makeup through physics-based manipulation of intrinsic image layers. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
Feng Liu, Ronghang Zhu, Dan Zeng, Qijun Zhao, and Xiaoming Liu. Disentangling features in 3d face shapes for joint face reconstruction and recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) December 2015.
11

---

Under review as a conference paper at ICLR 2020
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error, arXiv preprint arXiv: S.1i::5511.05400 2015.
Yuval Nirkin, Iacopo Masi, Anh Tuan Tran, Tal Hassner, Medioni, and Gérard Medioni. On face segmentation, face swapping, and face perception. In IEEE Conference on Automatic Face and Gesture Recognition, 2018.
P. Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vetter. A 3d face model for pose and illu- mination invariant face recognition. In 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 296-301, Sep. 2009. doi: 10.1109/AVSS.200 .58.
Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black. Generating 3d faces using convolutional mesh autoencoders. In The European Conference on Computer Vision (ECCV), September 2018
E. Richardson, M. Sela, and R. Kimmel. 3d face reconstruction by learning from synthetic data. In 2016 Fourth International Conference on 3D Vision (3DV), pp. 4604464066 469, Oct 2016. doi: 10.1109/ 3DV.2016.56.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejan- dro F. Frangi (eds.), Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, pp. 234-241, Cham, 2015. Springer International Publishing. ISBN 978-3-319-24574-4,
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt. Mofa: Model-based deep convolutional face autoencoder for unsuper- vised monocular reconstruction. In The IEEE International Conference Computer Vision (ICCV) Workshops, Oct 2017.
Ayush Tewari, Michael Zollhfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Prez, and Christian Theobalt. Self-supervised multi-level face model learning for monocular recon- struction at over 250 hz. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Justus Thies, Michael Zollhofer, Mare Stamminger, Christian Theobalt, and Matthias Niessner. Face2face: Real-time face capture and reenactment of rgb videos. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Anh Tuan Tran, Tal Hassner, Iacopo Masi, Eran Paz, Yuval Nirkin, and Gérard Medioni. Extreme 3D face reconstruction: Seeing through occlusions. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable model. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Luan Tran, Feng Liu, and Xiaoming Liu. Towards high-fidelity nonlinear 3d face morphable model. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Anh Tuan Tran, Tal Hassner, lacopo Masi, and Gerard Medioni. Regressing robust and discrimina- tive 3d morphable models with very deep neural network. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 499-515, Cham, 2016. Springer International Publishing.
Fanzi Wu, Linchao Bao, Yajing Chen, Yonggen Ling, Yibing Song, Songnan Li. King Ngi Ngan, and Wei Liu. Myf-net: Multi-view 3d face morphable model regression. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
12

---

Under review as a conference paper at ICLR 2020
Jiqing Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, and Luc Van Gool. Wasserstein diver- gence for gans. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp. 673-688, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01128-1.
Xiangyu Zhu, Z. Lei, Junjie Yan, D. Yi, and S. Z. Li. High-fidelity pose and expression normaliza- tion for face recognition in the wild. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 787-796, June 2015. doi: 10.1109/CVP..2055.2287999
X. Zhu, X. Liu, Z. Lei, and S. Z. Li. Face alignment in full pose range: A 3d total solution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(1):88-92, Jan 2019. ISSN 0162- 8828. doi: 10.1 10111//AA..211..27/555
Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z. Li. Face alignment across large poses: A 3d solution. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2016.
13


Reviews:
[{"review_id": "rkx9eFq6tH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nThis paper introduces a model for image-based facial 3D reconstruction. The proposed model is an encoder-decoder architecture that is trained in semi-supervised way to map images to sets of vectors representing identity (which encodes albedo and geometry), pose, expression and lighting. The encoder is a standard image CNN, whereas the decoders for geometry and albedo rely on spectral graph CNNs (similar to e.g. COMA, Ranjan’18). \nThe main contribution of the work with respect to the existing methods is the use of additional loss terms that enable semi-supervised training and learning somewhat more disentangled representations. Authors report quantitative results on MICC Florence, with marginal improvements over the baselines (the choice of the baselines is reasonable).\n\nDecision:\nThe overall architecture is very similar to existing works such as COMA (Ranjan’18) and (Tran’19), including the specific architecture for geometry decoders, and thus the contributions are primarily in the newly added loss terms. \nI also find the promise of “disentangled” representation a bit over-stated, as the albedo and base geometry still seem to be encoded in the same “identity” vector (see related question below).\nThe numerical improvements seem fairly modest with respect to (Tran’19). In addition, there is no numerical ablation study that would demonstrate the actual utility of the main contributions (such as adversarial loss): there are qualitative results but they are not very convincing. \nThus, the final rating “weak reject”.\n\nAdditional comments / typos:\n\n* I am not fully following the argument about sharing identity for albedo and shape on p2: “albedo and face shape are decoded ...”. Would it not be more beneficial to have a fully decoupled representation between the albedo and the facial geometry? I do not see how albedo information would be useful for encoding face geometry and vise-versa. \n* Authors claim that one of the main drawbacks e.g. of (Train’19) is the fact that they train on data generated from linear 3DMM. This is indeed the case, but it does not seem like here the authors fully overcome this issue: they do have additional weakly-supervised data, but they still strongly rely on linear 3DMM supervision (p6, “pairwise shape loss”, “adversarial loss”), and do not seem to provide experimental evidence that the model will work without it.\n* In particular, the “adversarial training” actually corresponds to learning the distribution of the linear 3DMM. Would it not mean that ultimately the model will be limited to learning only linear ? Could you please elaborate on this?\n* p3: “allows ene-to-end … training”\n* p3: “framework to exact … representations“.\n* p8: “evaluation matric”\n\nUpdate:\nAuthors did not provide any response, thus I keep my rating.\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Skx902J2FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an encoder-decoder architecture to reconstruct 3D face from a single image with disentangled representations: identity, expression, pose, and lighting. The authors develop a semi-supervised training scheme to fully exploit the value of large amount of unlabeled face images from unconstrained photo collections. Experimental results on MICC Florence and AFLW2000-3D verify the efficacy of the proposed method.\n\nThe presentation and writing are clear. The problem solved in this paper aligns with real applications.\n\nMy concerns regarding this paper are as below.\n1) What are the training computational complexity and testing time cost of the proposed method? Since speed is very important for real applications.\n2) The datasets used for evaluation are quite old. More experiments on more recent challenging benchmarks are needed to verify the superiority of the proposed method, e.g., IJB-B/C, etc.\n3) Some related works are missing and need to be discussed, e.g., Joint 3D Face Reconstruction and Dense Face Alignment from A Single Image with 2D-Assisted Self-Supervised Learning [Tu et al., 2019], 3D-Aided Dual-Agent GANs for Unconstrained Face Recognition [Zhao et al., T-PAMI 2018], 3D-Aided Deep Pose-Invariant Face Recognition [Zhao et al., IJCAI 2018], etc.\n4) Format of references should be consistent.\n\nBased on my above comments, I give the rate of WR. If the authors could solve my concerns in rebuttal, I would like to further adjust my rate.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HklBjUQHtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a semi-supervised and adversarial training process to exploit the value of unlabeled faces and overcome the limitation of a linear 3DMM and the nonlinear models proposed early (Tran & Liu (2018), Tran et al (2019)). This approach designs a framework to exact nonlinear disentangled representations from a face image with the help of loss functions including face recognition loss, shape pairwise loss and adversarial loss. This framework's contribution is demonstrated with experiments which show this model achieves state-of-the-art performance in face reconstruction.\n\nThis paper should be rejected because:\n(1) the experiments are not representative enough and the results are controversial,\n(2) this paper does not clearly demonstrate how they exploit the value of the unlabeled training images,\n(3) the creative progress of this paper is not typical compared to the early nonlinear model (Tran & Liu (2018), Tran et al (2019)).\n\n\nMain argument: \n\nThe experiments do not provide convincing evidence of the correctness of the proposed approach or its utility compared to existing approaches. The results are not representative enough and missing many details:\n1) What is the ratio of unlabeled training images and labeled training images?\n2) Why only show the results of Cooperative and Indoor situation? \n3) Why the standard deviation of the result in Cooperative situation is higher than the early models?\n4) Why the mean value of the result in Indoor situation is higher?\n5) Why so few situations and datasets your experiments run on?\n6) How did you initialize the parameters and the weights?\n7) How about the time-consuming and memory-consuming of your model?\n\nThe paper does not demonstrate the difference and progress between its model and the early nonlinear model clearly (Tran & Liu (2018), Tran et al (2019)). This paper points out that they fully exploit the value of unlabeled face data, but there are few evidences in this paper to support that. And it also points out the time-consuming problem of early models, but there are no experiment results show how efficient its model is.\n\nThe loss functions are also not convincing enough:\n1) How to choose or initialize the value of lambda center in the Face recognition loss?\n2) Have you demonstrated the solution you used in Shape smooth loss which aims to solve the vertices do not satisfy the Laplacian equation?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a semi-supervised method for reconstructing 3D faces from images via a disentangled representation. The method builds on previous work by Tran et al (2018, 2019). While some results presented in the paper show that this method works well, all reviewers agree that the authors should have provided more experimental evidence to convincingly demonstrate the benefits of their method. The reviewers are also unconvinced by how computationally expensive this method is or by the contributions of the unlabelled data to the performance of the proposed model. Given that the authors did not address the reviewers’ concerns, and for the reasons stated above, I recommend rejecting this paper.
Decision: Reject