Robust 3D face reconstruction from single noisy depth image through semantic consistency

IET Computer VisionVolume 15, Issue 6 p. 393-404 ORIGINAL RESEARCH PAPEROpen Access Robust 3D face reconstruction from single noisy depth image through semantic consistency Peixin Li, Key Laboratory of Machine Perception (MOE), Department of Machine Intelligence, Peking University, Beijing, ChinaSearch for more papers by this authorYuru Pei, Corresponding Author peiyuru@cis.pku.edu.cn orcid.org/0000-0001-8520-3509 Key Laboratory of Machine Perception (MOE), Department of Machine Intelligence, Peking University, Beijing, China Correspondence Science Building No. 2, Peking University, NO.5 Yiheyuan Road, Haidian District, Beijing, 100871, China. Email: peiyuru@cis.pku.edu.cn.Search for more papers by this authorYicheng Zhong, Key Laboratory of Machine Perception (MOE), Department of Machine Intelligence, Peking University, Beijing, ChinaSearch for more papers by this authorYuke Guo, School of Computer and Information Engineering, Luoyang Institute of Science and Technology, Luoyang, ChinaSearch for more papers by this authorHongbin Zha, Key Laboratory of Machine Perception (MOE), Department of Machine Intelligence, Peking University, Beijing, ChinaSearch for more papers by this author Peixin Li, Key Laboratory of Machine Perception (MOE), Department of Machine Intelligence, Peking University, Beijing, ChinaSearch for more papers by this authorYuru Pei, Corresponding Author peiyuru@cis.pku.edu.cn orcid.org/0000-0001-8520-3509 Key Laboratory of Machine Perception (MOE), Department of Machine Intelligence, Peking University, Beijing, China Correspondence Science Building No. 2, Peking University, NO.5 Yiheyuan Road, Haidian District, Beijing, 100871, China. Email: peiyuru@cis.pku.edu.cn.Search for more papers by this authorYicheng Zhong, Key Laboratory of Machine Perception (MOE), Department of Machine Intelligence, Peking University, Beijing, ChinaSearch for more papers by this authorYuke Guo, School of Computer and Information Engineering, Luoyang Institute of Science and Technology, Luoyang, ChinaSearch for more papers by this authorHongbin Zha, Key Laboratory of Machine Perception (MOE), Department of Machine Intelligence, Peking University, Beijing, ChinaSearch for more papers by this author First published: 04 March 2021 https://doi.org/10.1049/cvi2.12024AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract This paper addresses the 3D face reconstruction and semantic annotation from a single-view noisy depth image. A deep neural network-based coarse-to-fine framework is presented to take advantage of 3D morphable model (3DMM) regression and per-vertex geometry refinement. The low-dimensional subspace coefficients of the 3DMM initialize the global facial geometry, being prone to be over-smooth because of the low-pass characteristics of the shape subspace. The proposed geometry refinement subnetwork predicts per-vertex displacements to enrich local details, which is learned from unlabelled noisy depth images based on the registration-like loss. In order to guarantee the semantic correspondence between the resultant 3D face and the depth image, a semantic consistency constraint is introduced to adapt an annotation model learned from the synthetic data to real noisy depth images. The resultant depth annotations are required to be consistent with the label propagation from the coarse and refined parametric 3D faces. The proposed coarse-to-fine reconstruction scheme and the semantic consistency constraint are evaluated on the depth-based 3D face reconstruction and semantic annotation. The series of experiments demonstrate that the proposed approach achieves the performance improvements over compared methods regarding 3D face reconstruction and depth image annotation. 1 INTRODUCTION Generating a complete and high-quality 3D model from a single-view image is a fundamental problem in the computer vision community. Image-based 3D face estimation has a wide variety of applications in animation and virtual reality [1], face recognition [2], mask designs [3, 4], human–machine interactions [5], and virtual driving [5]. This article studies the problem of 3D face reconstruction from a single noisy depth image. Most existing techniques depend on RGB images to provide structure clues, such as boundaries of the eyes and the mouth, in 3D face estimation. However, the illumination variations caused by the uncontrolled lighting in the outdoor scenarios result in poor reconstruction performances. Depth images, on the other hand, are unaffected by illumination variations, and accommodate partial geometric information with per-pixel depth, facilitating 3D reconstruction. There is significant progress in depth image-based 3D facial structure tracking and shape estimation [6-9]. However, the quality of reconstructed 3D face regarding deformable facial structures and geometric details remains to be unsatisfactory. Single-view depth-based 3D face reconstruction is a challenging issue for three reasons. First, the quality of depth images captured by the consumer depth camera is limited due to depth missing and device-specific noise. The facial geometry is incomplete due to self-occlusions in the single-view image. The region with normal nearly perpendicular to the view direction is filled with unpleasant holes. Second, it is difficult to handle the non-rigid expressive deformations when textural constraints are unavailable. For instance, 3D reconstruction of the wide-open mouth from the noisy depth image is ambiguous due to the absence of semantic textures. Third, the unsupervised model learning is desirable since it is tedious and time-consuming to annotate noisy depth images. Traditional techniques, such as the Poisson surface reconstruction [10] and the bilateral filter [11], have been applied to hole filling and denoising of partial meshes. However, the mesh completion by such methods lacks high-level geometric details. The data-driven method is promising to perform the ill-posed 3D face reconstruction from a single-view image. The random forest-based regression model maps the depth image to 3D head poses [6, 7]. Three-dimensional face reconstruction recently takes advantage of the deep neural network (DNN) and exhibits promising performances [12, 13]. The DNN enables the feature learning and constructs a non-linear function to map a 2D image to 3D face shape parameters. Borghi et al. [5] proposed the DNN-based Poseidon model and realized the depth image-based 3D face estimation. The 3D morphable model (3DMM) [14] is a widely used deformable facial representation, taking advantage of a small set of orthogonal bases to model facial shapes and textures [15-17]. However, the 3DMM is limited to model geometric details by using the subspace spanned by a few principal components. The resultant face of the 3DMM model is smooth and lack of person-specific geometric details. The prior landmark pairs facilitate the semantic correspondence between the 3D shape and the 2D image [13, 18-20]. In order to model fine-grained facial details, the coarse-to-fine scheme utilizes the shape-from-shading (SFS) techniques to generate geometric details from RGB images. In contrast, the shading or the textures are unavailable in the depth image to build the semantic-consistent 3D shapes. We propose a novel data-driven approach to predict a complete 3D face with per-pixel semantic labels from a single-view noisy depth image, overcoming the need for texture constraints and prior annotations of the depth images. A cascaded DNN is utilized to build the coarse-to-fine reconstruction framework, which leverages both low-dimensional linear 3DMM and high-dimensional dense per-vertex displacement field (DF). The coarse-level 3DMM regression module decouples facial parameters regarding the pose, expression, and identity. The dense per-vertex DF is recovered to accommodate local facial geometric details. In order to ensure semantic correspondence between the reconstructed 3D face and the depth image, we present an end-to-end semantic annotation subnetwork and require the resultant semantic annotations, including fiducial landmarks and segmentation, to be consistent with label propagation from the coarse and refined 3D faces. In evaluations, the proposed approach exhibits face reconstruction and per-pixel semantic annotation from a single-view depth image, producing performance improvements over compared state-of-the-art methods. Experiments demonstrate that it is promising to apply the per-vertex geometry refinement and semantic consistency constraint to high-quality 3D face reconstruction from a noisy depth image. This article is an extension of our preliminary work [21] by adding: We elaborately describe a DNN-based coarse-to-fine framework to reconstruct high-quality facial geometries, which exploits the high-dimensional DF to enrich the high-frequency geometric details to the 3DMM face and overcomes the need to annotate noisy depth images. We discuss the semantic consistency scheme for structure correspondence between the reconstructed 3D face and the depth image, as well as the adaptation of the depth annotation model for noisy depth images. We conduct an extensive series of experiments to study the effects of the coarse-to-fine scheme and the semantic consistency constraints and compare the proposed approach with state-of-the-art methods. 2 RELATED WORKS 2.1 Depth completion and 3D reconstruction Shape completion and reconstruction from partial geometric data are well-studied in the computer graphics and geometry processing community. Traditional methods rely on energy optimization-based fitting for hole filling. The surface reconstruction techniques, such as the Poisson surface reconstruction [10, 22, 23], the least-squares meshes [24], and the Laplacian mesh [25], are capable of fitting surface to the input point clouds. The non-rigid registration, such as the thin-plate splines-based deformation, fits high-quality models in the existing dataset to partial scans [26-29]. Deep learning techniques are popular in recent 3D shape analysis and completion [30-32]. The 3D convolutional deep belief network-based ShapeNets [33] realized shape completion and repair. Hierarchical octree-based 3D CNNs [34-36] have been used to produce high-resolution models exploiting the sparse input data and the space partition. Dilated convolutions [37] with increasing spatial extents were used for semantic inference and volume completion without losing resolution [38]. Yan et al. [39] proposed an end-to-end trainable network for depth map denoising and SFS refinement using hyper-column architectures [40]. A coarse-to-fine completion strategy captured both local details along with global structures and inferred semantic labels [41]. We also perform the DNN-based shape completion and semantic annotation in a cascaded manner. Our model does not require prior annotation of noisy depth images. Instead, we utilize the unsupervised registration with deformable parametric face and the semantic consistency constraints for 3D face reconstruction and depth annotation. 2.2 3DMM-based approach The 3DMM is popularly used in 3D face reconstruction. Similar to other statistical facial models, such as the AAM [42] and the multi-linear model [43], the 3DMM is known for its compact and low-dimensional representation of facial shape and textures. Traditional methods rely on computationally intensive optimization for facial parameters. Landmark-based alignments were used in the 3DMM-based fitting [18, 44, 45]. Although the 3DMM approach realizes the expressive face reconstruction, the resultant model is smooth and lacks the facial geometric details, such as wrinkles and folds, considering the linear subspace spanned by limited principal components. The 3DMM is often used in the first phase of the coarse-to-fine 3D face estimation framework, where photometric consistency constraints and the SFS techniques were used to recover fine details [18]. Our work employs the 3DMM-based method for 3D face reconstruction from a single depth image. Unlike the above 3D face estimation from a colour image, we present a semantic consistency constraint without using the textural constraints. Moreover, the proposed method enriches the 3DMM face with local geometry details by the geometry refinement module learned from unlabelled noisy depth images. 2.3 Deep learning-based 3D face estimation The DNN has become the mainstream in the machine learning community and is widely used in 3D face reconstruction with superior performances. The DNN-based regressor realized the facial parameter estimation of statistical facial models, such as the 3DMM [12, 13, 45-47], avoiding online iterative optimization. The synthetic facial images with ground-truth parameters were used to train the neural networks for the non-linear mapping from the input image to face parameters [15]. The expert-designed and differentiable renderer enabled the unsupervised network learning from unlabelled real images [12, 46]. The PRNet [48] and the FGNet [47] used the CNN-based regressor to infer the UV map from a 2D image. The CNN-based regressor decoupled facial shape, expression, reflectance, and illuminations using a 3DMM and a learned corrective space [13]. The non-linear 3DMM utilized the multi-layer perceptron (MLP) for shape and CNN-based texture prediction in the UV space [49]. Semantic segmentation [48], fiducial points, or landmarks [13] facilitate 3D face reconstruction. Aside from the facial surface reconstruction, the VRN [50] constructed multi-task architecture for both the 3D facial volume and sparse facial landmarks without using the 3DMM. Aside from the 3DMM-based coarse geometry estimation, the DNN also predicts a per-pixel depth map [51], bump map [52], and displacements [53, 54] for the high-level facial detail by employing the SFS objectives [55]. The dense multi-level photometric loss [13] was used in the coarse and fine alignments. Fine details can be encoded in high-resolution displacement maps, which are learned through a hybrid network of the image-to-image translation network [56] and super-resolution network [54]. Zhong et al. [57] utilized the bidirectional CycleGAN-based generator for denoising and noise simulation. They generated the 3DMM regressors in both the synthetic and noisy depth image domains, where a multi-level shape consistency constraint was introduced to avoid the shape distortions in the CycleGAN-based generators. The adaptive weighting has been used for depth-based face reconstruction [58]. However, the fine-level facial geometric details and the semantic correspondence has not been addressed in the above work. We propose a cascaded network for a coarse 3DMM-based reconstruction and the per-vertex displacements for high-quality face reconstruction from a single-view depth image. Moreover, the proposed framework ensures the semantic correspondence between the resultant 3D face and the depth image by utilizing the depth annotation. 3 METHODS Given a noisy depth image, our goal is to reconstruct the high-quality complete 3D face and per-pixel semantic annotation, as shown in Figure 1. The proposed framework consists of three subnetworks: the coarse-level 3DMM regression subnetwork, the fine-level geometry refinement subnetwork, and the semantic annotation subnetwork. The proposed framework is learned from both the synthetic depth image dataset I a with paired face codes and annotations, as well as the unlabelled noisy depth image dataset I u captured by the consumer depth camera Kinect. FIGURE 1Open in figure viewerPowerPoint Overview of the proposed coarse-to-fine 3D face estimation and per-pixel semantic annotation from a single-view depth image. We introduce a semantic consistency constraint to enforce the structure correspondence between the 3D face and the depth image The proposed approach takes advantage of a coarse-to-fine reconstruction scheme, integrating a 3DMM regression module and a vertex-wise geometric refinement module. The 3DMM model decomposes the input depth image into the head pose, the expression, and the identity, producing a 3D face using a limited number of coefficients in a linear subspace. The geometry refinement module predicts per-vertex displacements to enrich the 3DMM face with geometric details. In order to learn such reconstruction from unlabelled noisy depth images, the reconstructed 3D face is required to fit the depth image as in the non-rigid registration. Moreover, we present a semantic consistency scheme to enforce the structure correspondence between the reconstructed 3D face and the depth image by virtue of an annotation subnetwork. 3.1 3DMM regression First, the proposed system takes one depth image I ∈ ℝ n p with np pixels as an input and outputs the 3DMM facial parameters. The residual convolutional neural network [59] is used to parameterize the regression model and maps the depth image I to 3DMM subspace coefficients of the facial identity α s ∈ ℝ n s , the expression α e ∈ ℝ n e , and the head pose α p ∈ ℝ 6 . The subspace coefficients uniquely determine a 3D face Mc(Vc, E) with vertices V c ∈ ℝ 3 n v and edges E ∈ ℝ 2 n q . nv and nq denote the number of vertexes and triangle edges, respectively. V c = ϕ ( V ¯ + [ α s α e ] ⋅ [ Φ s Φ e ] ′ , α p ) . (1) ϕ performs rigid transformation determined by α p . V ¯ denotes the neutral 3D face. Φ s ∈ ℝ 3 n v × n s refers to the shape bases as the first ns dominant principal components learned from the Basel Face database [60]. The expressive bases Φ e ∈ ℝ 3 n v × n e are learned from the FaceWarehouse database [61]. ns and ne are set to 80 and 64 respectively in experiments. The 3D facial mesh has approx. 36k vertices and 70k triangles. The 3DMM regression network predicts the low-dimensional coefficients in the linear subspace to determine a 3D face. However, the shape and expression subspaces are spanned by limited principal components. The resultant 3DMM face is over-smooth without local geometric variations, such as wrinkles and folds, due to the lack of high-frequency information. 3.2 Per-vertex geometry refinement The proposed geometry refinement module takes the 3DMM face Mc as the input and predicts per-vertex 3D displacements field U ∈ ℝ 3 n v . The resulting refined face Mf (Vf, E) has the same topology as Mc and the updated vertex: Vf = Vc + U. The per-vertex displacements accommodate local non-rigid deformation to enrich facial geometric details. The refinement module is built similar to the PointNet [62], where the cascaded MLP-based network embeds the point cloud of the 3DMM face to extract global facial features. The features are further used to estimate the per-vertex displacements. Here we utilize the multi-scale features of depth images to enhance the input point cloud. The feature vectors of the corresponding pixels extracted in the 3DMM regression module are concatenated with the 3D vertex coordinates as the input of the geometry refinement module, as shown in Figure 1. 3.3 Reconstruction loss Both the synthetic depth image dataset I a with paired 3DMM coefficients and the un-annotated noisy depth image dataset I u captured by the consumer depth camera are used to learn the 3D face reconstruction network, including the 3DMM regression module and the geometry refinement module. The loss function of the 3D face reconstruction is defined as follows: L r e c = λ 1 ∑ I ∈ I a δ α + ∑ κ ∈ { c , f } ∑ I ∈ I u δ p ( θ ( I ) , V κ ) + λ 2 δ s m o . (2) δ α measures the difference between the predicted 3DMM subspace coefficients α and ground truth α ∗ regarding the synthetic data. δ α = ∑ κ ∈ { p , s , e } ∥ α κ − α κ ∗ σ κ ∥ 2 2 . (3) σ p , σ s , and σ e denote code variances of the poses, the identities, and the expressions, respectively. δ p evaluates the point cloud distance between the reconstructed 3D face and the depth image using the nearest neighbour searching scheme. δ p ( θ ( I ) , V ) = ∑ p ∈ θ ( I ) min p ′ ∈ V ∥ p − p ′ ∥ 2 2 , (4)where θ ( I ) refers to the point cloud of the depth image. By minimizing the reconstruction loss, both the 3DMM face Vc and the refined face Vf are required to fit the depth image. The smoothness term δ s m o is defined as the displacement variations of neighbouring vertexes, and δ s m o = ∑ u i ∈ U ∑ ( v i , v j ) ∈ E ∥ u i − u j ∥ 2 2 . (5) λ 1 and λ 2 are hyper-parameters. The 3DMM face uses limited subspace coefficients to control the entire facial shape. In contrast, the geometry refinement module enriches the deformation power by using the high-dimensional DF. Note that the proposed approach does not require annotation of noisy depth images by enforcing fitting of the 3D face and the depth image. However, there is no guarantee of semantic correspondence between the reconstructed face and the depth image. 3.4 Semantic annotation The semantic annotation module predicts per-pixel labels Y(Yl, Yr) of the input depth images, including the landmark labels Y l ∈ ℝ n p × n l of nl landmarks and feature segmentation labels Y r ∈ ℝ n p × n r of nr regions of interest. The predicted depth image annotation is further utilized to ensure the semantic correspondence between the 3D face and the depth image. The convolutional U-net [63] is used to parameterize the semantic annotation functions concerning the landmark detection and the segmentation. The semantic annotation network is learned from the synthetic depth images I a with paired per-pixel labels. The semantic annotation loss Lsem is defined as follows: L s e m = λ 3 δ l ( y l , y l ∗ ) + δ c e ( y r , y r ∗ ) . (6) δ l evaluates the l2-norm-based difference between the predicted landmark heat maps and the ground truth, and δ l ( y l , y l ∗ ) = ∑ i = 1 n l ∥ y l , i − y l , i ∗ ∥ 2 2 . The cross-entropy based metric δ c e ( y r , y r ∗ ) = − ∑ j = 1 n r y r , j log y r , j ∗ . yl and yr denote the predicted landmark heat map and the labelling probability of facial regions. y∗ refers to the ground truth annotation. λ 3 is a hyper-parameter. The annotation model learned from the synthetic data is limited to handle the noisy depth images, considering apparent appearance gaps between the synthetic depth images and those captured by the consumer depth camera. This study presents a semantic consistency scheme to adapt the annotation network to noisy depth images. 3.5 Semantic consistency We introduce a semantic consistency constraint to enforce the semantic correspondence of the 3D face and the depth image, together with the domain adaptation of the annotation network. The reconstructed 3D face has the same topology as the 3DMM parametric face, and the per-vertex annotation of the parametric face can be transferred to the reconstructed 3D face. Given the 3DMM face Mc and the refined face Mf obtained by the face reconstruction networks, the parametric face's semantic annotations are transferred to the depth images using the nearest neighbour searching scheme. The predicted landmarks and segmentations by the annotation network are required to be consistent with those obtained by the label transfer from the reconstructed 3D face. Moreover, the corresponding facial features on the 3D face and the depth image are required to be closely registered. For instance, the eye regions on the depth image are expected to be overlapping with those on the reconstructed 3D face. The semantic consistency loss L c o n = ∑ κ ∈ { c , f } { λ 4 δ p ( R θ ( I ) , R V κ ) + λ 5 δ l ( y l , y l κ ′ ) + δ c e ( y r , y r κ ′ ) } . (7) R V and R θ ( I ) denote feature regions on the reconstructed 3D face V and the point cloud θ ( I ) of the input depth image. We measure the l2-norm and the cross entropy-based annotation difference between the attribute transfer-based label maps, including the landmark map y l ′ and the segmentation map y r ′ , and those obtained by the annotation network. λ 4 and λ 5 are hyper-parameters. The cascaded 3D reconstruction network realizes the non-rigid registration between the 3D face and the depth map with semantic correspondence by minimizing L c o n . The semantic consistency constraints enhance facial feature reconstruction and extend the generalization capacity of the annotation network. The annotation network initialized using the synthetic data is adapted to predict the per-pixel labelling of noisy depth images. 3.6 Loss The overall loss function of the proposed 3D face reconstruction and semantic annotation network is defined as follows: L = γ r e c L r e c + γ s e m L s e m + γ c o n L c o n . (8) The hyper-parameters γ r e c , γ s e m , and γ c o n are used to balance the 3D face reconstruction, the per-pixel annotation, and semantic consistency constraints, respectively. 4 EXPERIMENTS We evaluate the proposed cascaded 3D reconstruction and semantic annotation network on noisy depth images. Figure 2 shows the results of the 3D face reconstruction from single-view depth images with various head poses. We conduct the ablation studies on the coarse-to-fine scheme and the semantic consistency constraints and compare it with state-of-the-art methods quantitatively and qualitatively. FIGURE 2Open in figure viewerPowerPoint 3D face reconstruction from a single depth image. From top to bottom: input depth image, reconstructed 3D face, overlapping of 3D faces with depth images, and per-pixel labelling of depth images Dataset. The consumer depth camera Kinect is used to acquire 20k depth images from 10 subjects with a resolution of 640 × 480, where 18k depth images are used for training and the remaining for testing by random splitting. The depth images are cropped to the resolution of 150 × 150 with the face-centred. The synthetic dataset consists of 40k depth images by depth rendering of the 3D morphable faces with the randomly sampled pose, expression, and identity parameters. We also use popular depth image datasets of the BIWI [7] and the ICT-3DHP [64] to evaluate the 3D head pose estimation. The BIWI dataset consists of 15k frames in 24 sequences of 20 subjects. The ICT-3DHP consists of 14k depth images of 10 sequences. The 11th and the 12th sequences of the BIWI, and the 9th sequence of the ICT-3DHP are used as the testing data, and the remaining sequences are used as the training data in pose estimation experiments. 4.1 Dataset The proposed network is learned from both the synthetic depth images and real noisy ones. The coarse-level 3DMM regression subnetwork is built upon the 34-layer residual network [59], which is initialized by using synthetic depth images with paired facial parameters. Similarly, the U-net [63]-based semantic annotation subnetwork [63] is initialized using the synthetic data with paired per-pixel labels. The unlabelled noisy depth images are used to learn the coarse-to-fine 3D face reconstruction networks with the semantic consistency constraints. The hyper-parameters λ 1 , λ 2 , λ 3 , λ 4 , and λ 5 are set to 0.5, 5, 0.1, 0.2, and 0.1, respectively. γ r e c , γ s e m , and γ c o n in the overall loss are set to 1, 0.5, and 2 in experiments. The framework is implemented by the TensorFlow. The network is optimized using the ADAM algorithm. β 1 = 0.9 and β 2 = 0.999 . The learning rate is set to 1e − 4 initially for the first 30 epochs. Then, the learning rate is divided by two after every 5 epochs. The mini-batch size is set to eight. The training of the network is on a NVIDIA 2080ti GPU and takes 40 h with 60 epochs. The 3D face estimation and semantic annotation from a depth image take 0.025 and 0.01 s, respectively. 4.2 Metric We evaluate the 3D face reconstruction, the depth annotation, and the head pose estimation. The face reconstruction accuracy (RA) is computed using the point distance between the 3D face V and the point cloud θ ( I ) of the depth image. R A = 1 | θ ( I ) | ∑ p ∈ θ ( I ) ∥ p − v ∥ 2 , (9)where v ∈ V is the nearest neighbour of p on the depth plane. The landmark accuracy (LA) and segmentation accuracy (SA) are computed as: L A = 1 n l ∑ i = 1 n l ∥ v l , i − v l , i ∗ ∥ 2 , a n d S A = 2 | R ∩ R ∗ | | R | + | R ∗ | . (10)vl and v l ∗ denote the predicted and manually labelled landmark coordinates. The Dice similarity coefficients (DSC) are used to measure the segmentation accuracy SA when given the predicted region of interest R and the manually labelled R ∗ . The pose estimation accuracy P A = | a − a ∗ | , measuring the absolute deviation of the predicted Euler angle a and the ground-tr