Title: SEMI-SUPERVISED 3D FACE RECONSTRUCTION WITH NONLINEAR DISENTANGLED REPRESENTATIONS

Abstract: Shape, lighting, albedo Reconstructed face Expression transfer
Our network decomposes an input image into shape, lighting, and albedo with four disentangled representations: identity, expression, pose, and lighting, which allows expression transfer between different face images.

1. INTRODUCTION:

3D face reconstruction from 2D images enables many exciting applications, such as face recognition (Blanz & Vetter, 2003; Paysan et al., 2009; Liu et al., 2018) , face puppetry (Cao et al., 2014) , face reenactment (Thies et al., 2016; Garrido et al., 2015) , virtual make-up (Li et al., 2015) , etc. However, 3D face shape and texture inference from 2D images, especially from a single image, is an ill-posed problem since some 3D information is lost after the imaging process. 3D morphable model (3DMM) (Blanz & Vetter, 1999) learned from a collection of 3D face scans is often adopted as a strong prior assumption for this problem. 3DMM is a linear combination of bases to provide statistical parametric representation of 3D faces. Given a 2D image, the conventional approach is to search for the corresponding 3DMM parameters through analysis-by-synthesis optimization (Levine & Yu, 2009; Booth et al., 2018) . Specifically, a 3D face is generated through inverse rendering to match the 2D image by optimizing the shape, albedo (i.e., texture separated from illumination conditions), pose, and lighting parameters. However, such 3DMM optimization-based methods are usually timeconsuming due to high optimization complexity and suffer from local optima solutions.

Regressing 3DMM parameters using convolution neural network (CNN) shows remarkable success in 3D face reconstruction (Richardson et al., 2016; Zhu et al., 2019; Genova et al., 2018; Wu et al., 2019) . However, these methods cannot go beyond but only search for a solution in the restricted linear low-dimensional subspace of 3DMM. Linear statistical models have limitations to construct 3D face shapes and textures. First, facial variations are nonlinear in the real world, e.g., various ethnic groups, ages, facial expressions, and skin colors. Second, in order to model highly variable 3D face, a large amount of 3D face scans are needed for training. The most popular 3DMM (Xiangyu Zhu et al., 2015) was built by merging Basel Face Model (BFM) (Paysan et al., 2009) with only 200 subjects in neutral expressions and FaceWarehouse (Cao et al., 2014) with 150 subjects in 20 different expressions, which is not able to fully capture the variability of human faces. A large scale facial model (LSFM) was constructed by Booth et al. (2016) from around 10,000 distinct facial identities but only in neutral expressions. Tewari et al. (2018) , Tran et al. (2018) , and Guo et al. (2019) further proposed 3D face models composed of two networks: a coarse-scale linear 3DMM network and a fine-scale corrective network. Even though the finle-scale corrective model can generate more details, 3D face reconstruction will fail if the foundation face shape generated by the linear 3DMM network is not good enough.

Recently, Tran & Liu (2018) and Tran et al. (2019) proposed encoder-decoder networks to regress the face shape and texture directly. The nonlinear networks have higher representation power compared to a linear model and are able to reconstruct high-fidelity facial texture. However, the nonlinear models were only trained on the 300W-LP dataset (Zhu et al., 2016) that is generated from a linear 3DMM with a face profiling technique. The models were further fine-tuned in a self-supervised manner on the same dataset. However, since most of the face images were synthesised based on the linear 3DMM, self-supervised training to reconstruct high-fidelity texture using inverse rendering makes limited contributions to the face shape reconstruction. Besides, in these methods, the face albedo and face shape are decoded from a albedo parameter and shape parameter separately without considering the facial identity. In fact, across one's different face images, the face albedo and identity shape should only depend on the facial identity, i.e., sharing the same identity representation. Learning albedo and shape parameter separately is difficult to disentangle the face albedo from lightings and occlusions. Especially, when the albedo decoder network has high representation power, the albedo decoder may reconstruct high-fidelity face albedo but without aligning with the face shape and fails to contribute to the face shape reconstruction. At last, the identity and expression representations are entangled in these methods and many applications, such as face recognition, face animation, and face reenactment, are not feasible.

In this paper, we propose a novel encoder-decoder architecture using inverse rendering that combines computer vision and computer graphics techniques. The vision system (i.e., encoder network) decomposes an input 2D face image into disentangled and sematic representations: identity code, expression code, pose code, and lighting code. The graphics system renders back a face image to match the input image based on the decoder networks that regress the 3D face shape and albedo from the extracted representations. Combining computer vision and computer graphics techniques provides a unique opportunity to leverage the vast amounts of readily available unlabelled face images from unconstrained photo collections through self-supervised learning.

Since 3D face reconstruction from a 2D image is ambiguous and ill-posed, self-supervised learning with unlabelled data through inverse learning is not sufficient. In this paper, we train the network in a semi-supervised manner on hybrid batches of large amounts of unlabeled face images and relatively small amounts of labelled face images that are generated from a linear 3DMM with optimizationbased methods. Moreover, following the idea of generative adversarial networks (GAN) (Goodfellow et al., 2014) , a discriminator network is used to ensure the reconstructed face shape is not too far away from the distribution of human face. Semi-supervised adversarial training not only prevents our model from generating unrealistic 3D face shape but also fully exploits the value of unlabeled face images without being constrained by the pre-existing linear 3DMM.

To reconstruct the 3D face shape, we use graph convolutional network (GCN) (Defferrard et al., 2016; Kipf & Welling, 2017) instead of fully connected layers with activation or CNN used in Tran & Liu (2018) and Tran et al. (2019) . A 3D face shape is usually modeled as a mesh that is defined by a collection of vertices, edges, and faces and is considered as an unstructured graph. Modeling graph convolutions on 3D meshes can be memory efficient and allows for processing high resolution 3D structures. GCN-based methods to reconstruct 3D face shapes outperforms other state-of-the-art methods (Ranjan et al., 2018; Jiang et al., 2019; Bouritsas et al., 2019) . To recover the 3D face albedo, we first use a GCN network that has the same architecture with the shape decoder to learn an illumination-independent face albedo. Then we apply a CNN-based decoder network that has skip connections with the encoder network (Ronneberger et al., 2015) and a patchGAN (Shrivastava et al., 2017) to improve the details of the facial texture.

We apply a face recognition loss and a center loss (Wen et al., 2016) to extract the identity representation (i.e., facial identity) from one's unconstrained multiple face images. The center loss is used to ensure the identity representation's compactness for each person and separability for different people, so that the identity representation is disentangled from the pose, lighting, and expression representations. In order to further disentangle the identity and expression representations, pairwise training approaches are adopted. Given a pair of labelled face data, we keep the identity codes and interchange the expression codes of 3DMM to generate new 3D shapes as supervision. Comprehensive evaluation experiments show that the proposed method achieves state-of-the-art performance in 3D face reconstruction and can easily be used for the applications of face recognition and facial expression transfer. The main contributions of this paper are summarized below:

• We propose an efficient semi-supervised and adversarial training process to fully exploit the value of unlabelled face data and go beyond the limitation of a linear 3DMM. • We design a novel framework to exact nonlinear disentangled representations from a face image with the help of face recognition losses and shape pairwise loss.

• Extensive experiments show that our model achieves state-of-the-art performance in face reconstruction.

2. BACKGROUND:

This section describes some background information related to our work, including face representations in conventional linear 3DMM, face rendering process, and graph convolution used in face shape reconstruction.

Linear 3DMM We first recap the conventional linear 3DMM. As described in Chu et al. (2014) , the linear 3DMM constructed from facial scans via PCA can be expressed as:

EQUATION

where

s ∈ R 3N ×1 is a 3D face shape with N vertices, s ∈ R 3N ×1 is the mean shape, A id ∈ R 3N ×K

is the first K principle components trained on facial scans with neutral expression and α id ∈ R K×1 is the identity parameter, A exp ∈ R 3N ×L is the first L principle components trained on the offset between neutral scans and expression scans and α exp ∈ R M ×1 is the expression parameter.

The texture of 3D face can also be modeled via PCA as:

EQUATION

where t ∈ R 3N ×1 is a 3D face texture, t ∈ R 3N ×1 is the mean texture, A tex ∈ R 3N ×M is the first M principle components trained on facial textures and α tex ∈ R M ×1 is the texture parameter.

Rendering process The 3D face modeled by 3DMM is projected onto a image plane with weak perspective projection:

EQUATION

where s 2D ∈ R 2×N is the face shape located on the image plane after projection, P r = 1 0 0 0 1 0 is the orthographic projection matrix, R is the rotation matrix constructed from Euler angles (i.e., pitch, yaw, and roll), t 2D = [t x , t y ] is the translation vector on the image plane, and f is the scale factor.

Following Guo et al. (2019) , we assume the face is Lambertian surface and the global illumination is approximated using the spherical harmonics (SH) basis function. The first three bands of SHs are used for the illumination model. γ ∈ R 27×1 is the illumination parameter for the RGB channels' SH illumination coefficient. Thus, the rendering process depends on the parameter set χ = {α id , α exp , α tex , pitch, yaw, roll, f, t 2D , γ}.

Spectral graph convolution As presented by Ranjan et al. (2018) , we use spectral graph convolution to reconstruct 3D face shapes. The shape of a 3D face is described as a triangular mesh M = (V, A), where V ∈ R n×3 denotes the n vertices in the Euclidean space, A ∈ {0, 1} n×n is the sparse adjacency matrix representing the edge connections. The non-normalized graph Laplacian is defined as L-D -A, where the degree matrix D is a diagonal matrix with D i,i = j A i,j . Spectral graph convolution is defined on the graph Fourier transform domain, whose bases are the eigenvectors of the Laplacian matrix. An efficient solution for spectral graph convolution is formulating mesh filtering with a kernel using a recursive Chebyshev polynomial,

EQUATION

where X out,j is the j th feature of the output X out ∈ R n×Fout and X in,i is the i th feature of the input X in ∈ R n×Fin , e.g., the input mesh vertices V has F in = 3 features corresponding to the 3D vertex position. L = 2L/λ max -I n is the scaled Laplacian. T k ∈ R n×n is the Chebyshev polynomial of order k that is computed recursively as

T k (x) = 2xT k-1 (x) -T k-1 (x) with T 0 = 1 and T 1 = x.

The parameter θ ∈ R Fin×Fout×K is the trainable Chebyshev coefficients.

3. METHOD:

We design an encoder-decoder architecture that allows ene-to-end semi-supervised adversarial training to extract disentangled semantic representations of a single image, as shown in Figure 1 . We adopt inverse rendering technique that utilizes parameterized illumination model and differentiable renderer to render back the input face image under varying identity, expression, pose, and lighting conditions. Our model is trained on hybrid batches of unlabeled face images from CelebA (Liu et al., 2015) and labeled face images from 300W-LP (Zhu et al., 2016) . 

3.1. ENCODER-DECODER NETWORK:

Encoder As shown in Figure 1 , the encoder network is a multi-task learning network, which takes a face image as input and extracts its identity, expression, pose, and lighting representations. A pretrained ResNet-50 network is used as the backbone of the encoder network. The ResNet-50 network is followed by four branches of fully connected layers with outputs of 128-D identity code (c id ), 64-D expression code (c exp ), 6-D pose code (c pose ), and 27-D lighting code (c lgt ).

Shape decoder:

The shape decoder network is a graph convolutional network modified from the COMA architecture (Ranjan et al., 2018) with an extra graph convolutional layer and up-sampling layer at the beginning. We concatenate the identity code and expression code extracted from the encoder network, i.e., a 192-D vector, as the input of the shape decoder network. The output of the shape decoder is the corresponding 3D face shape in the standard position (i.e., without any translations or rotations). We denote as F C(d) a fully connected layer, l the number of vertices after the last down-sampling layer, GC(k, w) a graph convolutional layer with k kernel size and w filters, and U S(p) a up-sampling layer by a factor of p, respectively. The shape decoder network is listed follows: F C(l * 256) → U S(2) → GC(6, 256) → U S(4) → GC(6, 128) → U S(4) → GC(6, 64) → U S(4) → GC(6, 32) → U S(4) → GC(6, 16) → GC(6, 3).

Albedo decoder:

The albedo decoder network is also a graph convolutional network and has the same architecture as the shape decoder. The albedo decoder takes only the identity code as input since the albedo of a face should be independent of the expression, pose, lighting, and occlusions. Importantly, hair, glasses, microphones, and other facial occlusions should not be included in the albedo since one's facial albedo should be consistent across his different photos even with different hair styles, glasses, etc. We apply face segmentation by Nirkin et al. (2018) to eliminate the effect of facial occlusions. Note that, we did not consider aging, injury, or other factors that may affect one's face albedo.

After the lighting representation is learned, we change the GCN-based albedo decoder network to a CNN network that has skip connections with the encoder network to improve the details of the facial texture. The architecture of the encoder and CNN-based albedo decoder with skip connections is similar to U-Net (Ronneberger et al., 2015) . Moreover, we apply a patchGAN (Shrivastava et al., 2017) to further make the facial texture more realistic.

3.2. LOSS FUNCTIONS:

Our network is trained with a multi-task loss that enable us to regress the 3D face shape and albedo end-to-end. The loss function combines face recognition loss, face reconstruction loss, pairwise shape loss, adversary loss, and other regularization.

Face recognition loss In order to extract the identity code that only represents the photo's facial identity, we apply face recognition loss as follows:

EQUATION

where L sof t is the softmax loss that classify each photo to a specific identity class, L center is the center loss to improve the discriminative power of the deeply learned identity code (Wen et al., 2016) , and λ center is used for balancing the two loss functions. Face recognition loss is essential to learn the facial identity without being influenced by other factors such as facial expressions, poses, lightings, occlusions, etc.

Face reconstruction loss:

The rendering layer renders back an image to compared with the input image. The face reconstruction loss is formulated as

EQUATION

where is the element-wise Hadamard product, I is the input image, Î is the rendered image, and M is the mask obtained by Nirkin et al. (2018) to eliminate the effect of facial occlusions such as hair, glasses, and microphone. Moreover, image gradient difference loss (GDL) (Mathieu et al., 2015) , denoted as L gdl,color , is applied to recover more details in the reconstruction.

Sparse landmark loss:

We add sparse landmark loss to help learn the face pose and achieve better face reconstruction. The sparse landmark loss is defined as

EQUATION

where ŝ2D is the projected face shape from our network, L is the vertex indexes of the 68 landmarks in the 3D face shape, U is considered as the ground truth of the corresponding sparse 2D landmarks on the input image and is obtained by Bulat & Tzimiropoulos (2017) . The idea of GDL is also applied on the sparse landmarks, denoted as L gdl,lmk , which describes the distance of two different landmarks should also be close to the corresponding distance in ground truth. Especially, it is important for the distances of the upper eyelids to the lower eyelids and the upper lip to the lower lip that represent the conditions of eye's opening and mouth's opening, respectively.

Shape loss:

In order to prevent the network from either generating unrealistic 3D face shapes or being under the constrain of a linear 3DMM, we train our network in a semi-supervised manner on hybrid batches of unlabeled and labeled face images. For the labeled face images, we choose 300W-LP dataset that contains 122,450 images with fitted 3DMM shapes across large poses and was created by Zhu et al. (2016) with face profiling technique. The BFM template that has 53,215 vertices is used for the fitted 3DMM shapes. The 3DMM parameters α exp and α exp are provided to calculate each of the fitted 3DMM shapes, as presented in Eq. ( 1). In this paper, we remove the neck and ears of the BFM model to create our own face shape template with 37,202 vertices. The shape loss for the 300W-LP dataset is formulated as

EQUATION

where s = s + A id α id + A exp α exp is considered as the ground truth of the face shape, ŝ is the 3D face shape reconstructed by our network, and T is the vertex indexes of our face template in the BFM model.

Pairwise shape loss:

To further disentangle the identity code and expression code, we train the 300W-LP dataset in pairwise manner. Given an input image, the corresponding 3DMM parameters α exp and α exp are provided. For a pair of input images, I A and I B , we interchange the expression parameters α exp,A and α exp,B to get the 3D face shape of A's identity with B's expression. The pairwise shape loss for the 300W-LP dataset is expressed as

EQUATION

where f shape (•) is the shape decoder, [c id,A , c exp,B ] means concatenation of A's identity code and B's expression code from the encoder network, and s A,B = s + A id α id,A + A exp α exp,B is the 3DMM shape of A's identity parameter with B's expression parameter.

Shape smooth loss Laplacian regularization is used on the shape vertex to help remove undesired noise of 3D face shapes. Conventional Laplacian smoothing assumes all the vertices satisfy the equation

X i = 1 |Mi| j∈Mi X j

, where X i is the ith vertex and M i is the vertex indexes of the first order neighbors of X i . However, some vertices, like on the edges, in the nostrils, at the eye corners, etc, do not satisfy the Laplacian equation. We calculate the difference of each vertex with the mean of its first order neighbors bo be close to the corresponding difference of the shape template,

EQUATION

)

where s is our face shape template cropped from the BFM model.

Albedo symmetry loss Facial symmetry is a strong prior for face albedo learning, which helps to disentangle facial expression, lighting, and occlusions from the face albedo. The albedo symmetry loss is defined as

EQUATION

)

where A is the output face albedo of the GCN-based albedo decoder and f lip(•) is an operation of flipping face albedos left and right.

Adversarial loss Semi-supervised learning is not sufficient to generate realistic 3D face shape for the unlabeled face images. Following the idea of generative adversarial network (GAN), an adversarial loss is used to train the encoder-decoder network and a discriminator network alternatively based on WGAN-div (Wu et al., 2018) . The discriminator network D is a GCN-based encoder network and is used to discriminate the fake shapes (i.e., shapes reconstructed from our network) and real shapes (i.e., shapes sampled from the linear 3DMM), so that the reconstructed face shapes will not be too far away from the distribution of the linear 3DMM. The min-max optimization problem can be written as

EQUATION

where L adv = -D(ŝ) is the adversarial loss, ŝ, s[:, T ] are the fake and real face shapes satisfying the probability measures P g , P r , and P u is the distribution obtained by sampling uniformly along straight lines between points from the real and fake face shape distributions.

4. EXPERIMENTS:

In this section, we first conduct ablation tests to demonstrate the effectiveness of the framework design (Section 4.1). We then evaluate our method by comparing reconstruction error against 3D face scans with state-of-the-art approaches (Section 4.2). At last, we present the application of expression transfer based on the disentangled representations of our model (Section 4.3).

We train our model on hybrid batches of unlabeled face images from CelebA dataset (Liu et al., 2015) and labeled face images from 300W-LP dataset (Zhu et al., 2016) . MICC Florence dataset (Bagdanov et al., 2011) and AFLW2000-3D dataset (Zhu et al., 2016) are selected for the quantitative and qualitative evaluations. The face region of the BFM model is cropped as the 3D face mesh template (i.e., 37202 out of the 53215 vertices). The model and the discriminators are optimized using Adam optimizer with a learning rate of 0.0001 and RMSprop optimizer with a learning rate of 0.00005, respectively.

4.1. ABLATION STUDY:



Shape reconstruction:

We study the effects of shape smooth loss and adversarial loss on the quality of shape reconstruction, as shown in Figure 2 . Since our face model is not constrained by a preexisting linear 3DMM, the face meshes can potentially be deformed to any shapes. The conventional smoothing loss causes abnormal effects on the edges and nostrils of face shapes. The vertices on the mouth's inner edge distance away from their neighbors. The nostrils are prone to be flat or even sticking out of the nose. This is because the vertices on the edges and nostrils are not satisfied with the Laplacian regularization which forces each vertex locates at the mean of its first order neighbors. When the model is trained without the adversarial loss, the forehead and two sides of face meshes are shrunk and eyebrows extrude out. The adversarial loss can make sure the face shapes generated by our model will not be too far away from the shape distribution of human face, while which is unknown and a pre-created linear 3DMM is used in this paper. Texture reconstruction Figure 3 shows the effects the albedo symmetric loss with facial mask. We consider the albedo symmetric loss and facial mask together because the facial occlusions should be masked out first in order to apply the albedo symmetric loss. The facial mask with albedo symmetric loss is crucial for lighting representation learning. Otherwise, the shade and lighting may be confounded with facial occlusions. Especially, when the representation power of the albedo decoder is high, e.g., CNN-based albedo decoder with skip connections to the encoder, the model will fail to learn the lighting even though the generated texture looks very close to the input image, as shown in the last column of Figure 3 . However, without learning the lighting, reconstructing high fidelity texture makes limited contributions to the face shape reconstruction because the high fidelity texture may not align with the face shape and looks odd when changing to a different pose. Facial mask with the albedo symmetric loss helps disentangle the lighting from the albedo. When the lighting is learned, a CNN-based albedo decoder with skip connections to the encoder is used to improve the detail of facial albedo. 

4.2. COMPARISONS TO THE STATE-OF-THE-ART:

We evaluate our model quantitatively on the MICC Florence dataset (Bagdanov et al., 2011) , which contains the ground truth scans of 53 subjects in neutral expressions. Each subject is recorded in three videos: Cooperative, Indoor, and Outdoor with increasingly challenging conditions. Following the setting in Wu et al. (2019) , the left, frontal, and right view of each subject are selected from the Cooperative and Indoor videos. The predicted 3D face shape is obtained by averaging over the 3D face shapes reconstructed from the left, frontal, and right view. The evaluation matric follows Genova et al. (2018) where we cropped the face region of 95mm around the nose tip of the ground truth scan to calculate the point-to-plane L2 errors with the predicted face shape. Table 1 shows that the proposed method outperforms other single-view reconstruction methods.

Method:

Compared to the multi-view reconstruction method (Wu et al., 2019) , we achieve better results in the Cooperative condition and have slightly worse results in the Indoor condition. Figure 4 presents two examples (i.e., subject No. 53, and subject No. 22) of detailed error maps. Figure 5 shows the reconstruction results of face images from the same subject (No. 05) in the Cooperative, Indoor, and Outdoor videos with different lightings and poses. The reconstruction errors are small across different conditions.

We further evaluate our model qualitatively on the AFLW2000-3D datasets (Zhu et al., 2016) . 

5. CONCLUSION:

This paper proposes an encoder-decoder architecture to reconstruct 3D face from a single image with disentangled representations: identity, expression, pose, and lighting. We develop an effective semi-supervised training scheme to fully exploit the value of large amount of unlabeled face images from unconstrained photo collections. An adversarial loss is applied to prevent our model from generating unrealistic 3D faces. We evaluate our model quantitatively and qualitatively. Our model outperforms the state-of-the-art single-view reconstruction methods and can effectively disentangle identity, expression, pose, and lighting features.

:



Figures and Tables:

Figure 1: Framework overview. The encoder network takes an input face image and extracts four disentangled representations: identity code (c id ), expression code (c exp ), pose code (c pose ), and lighting code (c lgt ). The albedo decoder network reconstructs the face albedo from the identity code. The shape decoder network reconstructs the face shape from the combination of the identity code and expression code. The rendering layer takes the face albedo, face shape, pose, and lighting to render back the face image. Multiple losses are applied on our network. Losses in gray rectangles are only used on labeled face images and in green rectangles are used on all face images.

Figure 2: Shape ablation test showing failures caused by changing to conventional Laplacian smoothing loss and removing the adversarial loss.

Figure 3: Texture ablation test showing failures of lighting caused by removing facial mask (i.e., mask out the facial occlusions) and albedo symmetric loss. We denote facial mask with albedo symmetric loss as mask+symm loss, GCN-based albedo decoder as GCN albedo, and CNN-based albedo decoder with skip connections as CNN+bridge.

Figure 4: Examples of error map comparison

Figure 6: 3D reconstruction comparisons with Tewari et al. (2018) and Tran et al. (2018)

Figure 8: Expression transfer between different face images. The left side is the expression transfer between different people and right side is the expression transfer between the same persian.

Mean error comparison on the MICC dataset

Table content: <table><tr><td/><td>Cooperative</td><td>Indoor</td><td>Subject</td><td>Ours</td><td>Wu19</td><td>Tran17</td></tr><tr><td/><td>Mean Std.</td><td>Mean Std.</td><td/><td/><td/></tr><tr><td>Tuan Tran et al.</td><td>1.397 0.290</td><td>1.381 0.322</td><td/><td/><td/></tr><tr><td>(2017)</td><td/><td/><td>No. 53</td><td colspan="3">0.783±0.701 1.12±0.953 1.95±1.623</td></tr><tr><td>Tewari et al. (2017)</td><td>1.370 0.321</td><td>1.286 0.266</td><td/><td/><td/></tr><tr><td>Genova et al. (2018)</td><td>1.372 0.353</td><td>1.260 0.310</td><td/><td/><td/></tr><tr><td>Wu et al. (2019)</td><td>1.220 0.247</td><td>1.228 0.236</td><td/><td/><td/></tr><tr><td>Ours</td><td>1.163 0.295</td><td>1.238 0.302</td><td>No. 22</td><td colspan="3">1.399±1.096 1.333±1.038 1.65±1.199</td></tr></table>

Peer review (iclr2020):
Overview: This paper introduces a model for image-based facial 3D reconstruction. The proposed model is an encoder-decoder architecture that is trained in semi-supervised way to map images to sets of vectors representing identity (which encodes albedo and geometry), pose, expression and lighting. The encoder is a standard image CNN, whereas the decoders for geometry and albedo rely on spectral graph CNNs (similar to e.g. COMA, Ranjan’18). The main contribution of the work with respect to the existing methods is the use of additional loss terms that enable semi-supervised training and learning somewhat more disentangled representations. Authors report quantitative results on MICC Florence, with marginal improvements over the baselines (the choice of the baselines is reasonable). Decision: The overall architecture is very similar to existing works such as COMA (Ranjan’18) and (Tran’19), including the specific architecture for geometry decoders, and thus the contributions are primarily in the newly added loss terms. I also find the promise of “disentangled” representation a bit over-stated, as the albedo and base geometry still seem to be encoded in the same “identity” vector (see related question below). The numerical improvements seem fairly modest with respect to (Tran’19). In addition, there is no numerical ablation study that would demonstrate the actual utility of the main contributions (such as adversarial loss): there are qualitative results but they are not very convincing. Thus, the final rating “weak reject”. Additional comments / typos: * I am not fully following the argument about sharing identity for albedo and shape on p2: “albedo and face shape are decoded ...”. Would it not be more beneficial to have a fully decoupled representation between the albedo and the facial geometry? I do not see how albedo information would be useful for encoding face geometry and vise-versa. * Authors claim that one of the main drawbacks e.g. of (Train’19) is the fact that they train on data generated from linear 3DMM. This is indeed the case, but it does not seem like here the authors fully overcome this issue: they do have additional weakly-supervised data, but they still strongly rely on linear 3DMM supervision (p6, “pairwise shape loss”, “adversarial loss”), and do not seem to provide experimental evidence that the model will work without it. * In particular, the “adversarial training” actually corresponds to learning the distribution of the linear 3DMM. Would it not mean that ultimately the model will be limited to learning only linear ? Could you please elaborate on this? * p3: “allows ene-to-end … training” * p3: “framework to exact … representations“. * p8: “evaluation matric” Update: Authors did not provide any response, thus I keep my rating.