Robust landmark‐free head pose estimation by learning to crop and background augmentation

IET Image ProcessingVolume 14, Issue 11 p. 2553-2560 Research ArticleFree Access Robust landmark-free head pose estimation by learning to crop and background augmentation Aoru Xue, School of Computer Engineering and Science, Shanghai University, People's Republic of ChinaSearch for more papers by this authorKai Sheng, School of Computer Engineering and Science, Shanghai University, People's Republic of ChinaSearch for more papers by this authorSongming Dai, orcid.org/0000-0002-2048-0748 School of Computer Engineering and Science, Shanghai University, People's Republic of ChinaSearch for more papers by this authorXiaoqiang Li, Corresponding Author xqli@shu.edu.cn orcid.org/0000-0001-7243-2783 School of Computer Engineering and Science, Shanghai University, People's Republic of China Shanghai Institute for Advanced Communication and Data Science, Shanghai University, People's Republic of ChinaSearch for more papers by this author Aoru Xue, School of Computer Engineering and Science, Shanghai University, People's Republic of ChinaSearch for more papers by this authorKai Sheng, School of Computer Engineering and Science, Shanghai University, People's Republic of ChinaSearch for more papers by this authorSongming Dai, orcid.org/0000-0002-2048-0748 School of Computer Engineering and Science, Shanghai University, People's Republic of ChinaSearch for more papers by this authorXiaoqiang Li, Corresponding Author xqli@shu.edu.cn orcid.org/0000-0001-7243-2783 School of Computer Engineering and Science, Shanghai University, People's Republic of China Shanghai Institute for Advanced Communication and Data Science, Shanghai University, People's Republic of ChinaSearch for more papers by this author First published: 20 August 2020 https://doi.org/10.1049/iet-ipr.2019.1369AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract It is well known that the performance of head pose estimation is greatly affected by the bounding box margin of the face and its background. Traditionally, researchers will manually choose a suitable bounding box margin to strike a balance between ensuring sufficient information and minimising background noise. However, head pose estimation is still worse when the background is complex in reality or when the box margin changes slightly. To make estimation results more robust, the authors propose two methods to improve it: (i) a convolutional cropping module that can learn to crop the input image to an attentional area for head pose regression. (ii) Background augmentation that can make the network more robust to the background noise. Rather than using the face landmarking to calculate head pose angles, they use another convolutional neural network to regress the head pose angles, which is independent of the landmark detection results. They evaluate the method on BIWI and AFLW2000 dataset and experimental results show that their approach outperforms many other methods. Besides, they evaluate the method on Pointing′04 dataset using head pose accuracy. Furthermore, the approach is more robust and has a lower variance in realistic scenarios. 1 Introduction Head pose estimation (HPE) is the process of inferring the orientation of a human head (includes three Euler angles called pitch, yaw, and roll) from a given image. The head pose is described by three degrees of freedom of a person's head, which is the orientation in relation to the camera coordinate system [1]. It can not only be served for many tasks as face alignment [2], face recognition [3], facial expression recognition [4] or human attention recognition [5] but also can be used in many fields such as human–computer interaction [6], driver attention recognition [7], and augmented reality [8]. HPE is a daunting challenge task in computer vision. In realistic scenarios, the head pose estimation method should satisfy a set of constraints: the method should be capable of performing in real-time [9], which means the method should have a small computational cost; it should be able to work for any person no matter how old they are or who they are [9]; it should be robust to handle different situations, e.g. when the same person is in a different background, the method should output similar results. The methods of head pose estimation have been an area of intense investigation. In the early stage, the dominant way is using landmark-based methods [10, 11] for head pose estimation. These landmark-based methods [10, 11] use a convolutional neural network (CNN) to detect face landmarks and then obtain the head pose angles from the correspondence between 2D points and a pre-defined 3D model. They are not robust enough because of their heavily depending on landmark detection. Machine learning-based methods proposed in [12, 13] use facial segmentation and conditional random field (CRF) for face analysis task, which contains head pose estimation. The results of the head pose estimation depend on the accuracy of face segmentation. Recent advances in the CNN have led to the idea that it is possible to achieve the goal without (w/o) using face landmark. A simple idea is to use depth information so that the estimated head pose is more precise by RGB-D data. The methods proposed in [14, 15] use depth information to estimate head pose but these methods lack practicalities and always need high computational costs. To reduce computational costs, the light-weight method such as Hopenet [16] proposed a practical model that gets a good balance between accuracy and computation costs. FSA-Net [17] proposes a more light-weight structure for feature aggregation. Using stage-wise regression such as SSR-Net [18], FSA-Net [17] achieves the state-of-the-art (SOTA) results. However, from the actual test situation, the predicting bounding box (shown in Fig. 1) of a face detector may have slight deviations and the person may come in a variety of backgrounds. The existing HPE methods are not robust to these changes, which limit these methods in some way. For example, Fig. 2 shows that the SOTA method is not robust to the change of bounding box margin and Fig. 3 shows that some methods are not robust to background changing. Fig. 1Open in figure viewerPowerPoint Bounding box margin of an example. In our experiment, we choose the margin size from 0 to 50 to test the robustness of a head pose estimation method Fig. 2Open in figure viewerPowerPoint Result of SOTA network (FSA-Net). The values predicted by the SOTA network change significantly with the change of bounding box size on all three axes, so it can be considered that the SOTA network is not robust to the change of bounding box margin Fig. 3Open in figure viewerPowerPoint Network output in different backgrounds. The top row is the output of SSR-Net with squeeze-and-excitation w/o background augmentation and the bottom row is the same model using background augmentation. The top row indicates that the network is not robust in different backgrounds and the bottom row using the background augmentation approach alleviates the problem. For example, A1 in the top row has totally different head pose angles (the offset of pitch and yaw between A1 and A2 is about ). However, in the bottom row, their offset In this study, we propose a cropping convolutional module (CCM) that learns to crop face automatically for head pose estimation and a novel background augmentation method. CCM consists of a bunch of convolutional structures and regresses the appropriate bounding box coordinates. The image is cropped using the obtained coordinates and head pose is regressed by a regression network with the cropped image as input. CCM is robust to bounding box margin by learning to crop attentional area. By training CCM and head pose estimation network alternately, CCM can learn the best cropping position and the head pose estimation network can also learn to regress the head pose accurately. To make the prediction more robust, a pre-trained Mask R-CNN [19] is used to crop the human foreground and the foreground is redrawn to a cropped COCO [20] background image. Background augmentation forces the model to learn to distinguish foreground human face and background. Comparing with other methods which crop the human face with manually set small box [21], the proposed approach exploits more complex background information so that the network can learn better facial feature and get more precise predictions. We conduct plentiful experiments on Pointing′04, BIWI [22] and AFLW2000 [23] datasets and experimental results indicate that the proposed approach outperforms many other methods (includes SOTA). The main contributions of the study are summarised as follows: (1) We propose a CCM module that learns to crop attentional area for head pose estimation. By training CCM and head pose estimation networks alternately, CCM can learn the best cropping position and the head pose estimation network can also learn to regress the head pose accurately. (2) We propose a background augmentation approach for head pose estimation, which is for increasing the robustness against the complex background of a model. The background augmentation approach not only reduces the mean absolute error (MAE) of head pose estimation but also does not add any extra computational costs. To the best of our knowledge, the approach is the first data augmentation method that improves the robustness against the complex background of a model in head pose estimation. 2 Related work Landmark-based methods The dominant way of head pose estimation is using the facial landmarks in the early stage. Nabati and Behrad [24] used facial feature points and tracking algorithm and calculate 3D rotation and translation between the web camera and 3D head pose in a video processing task. Methods such as Dlib [10] and FAN [11] obtain the head pose angles from establishing a correspondence between the key points and the 3D face model. Abate et al. [25] exploited a quad-tree-based representation of facial landmarks. By using binary search in a pre-defined model array, it outputs the corresponding head pose angles. However, they all have the same problem that they depend heavily on the accuracy of facial landmarks or facial feature points and are not robust when people have facial decorations. Landmark-free methods Head pose information can be learned from a depth image and there are many methods proposed to predict head pose using depth information. Venturelli et al. [14] use the depth image to estimate head pose with Siamese CNN. No additional information such as facial landmarks, nose tip position, skin colour, and so on are taken into account, this method can work well on BIWI [22] dataset. However, the input of depth images will increase computational costs, which is inconvenient for practical use. A light-weight method such as Hopenet [16] uses ResNet18 [26] as its backbone and regard it as a combination of classification and regression problems. They predict head pose by grouping the head pose angles to 66 bins and predict their bin index. By calculating the expectation of predictions, the bin index is mapped to head pose angles. Then they sum up the classification loss and regression loss with a weight of for backpropagation. The authors of [12, 13] propose a method that uses face segmentation based on CRFs for head pose estimation. Face segmentation depends on the quality of the image and the simplicity of the background. The method performs badly in poor conditioned pictures and complex backgrounds. The method uses MAE and pose estimation accuracy to measure their results. FSA-Net [17] designs a structure that learns a fine-grained structure mapping for spatially grouping features before feature aggregation. FSA-Net [17] regress head pose via soft stagewise regression using the aggregated features. It is more robust than any of the previous works and achieves SOTA results. Motivation Spatial transformer networks (STNs) [27] propose a robust method for spatial affine transformations to help networks learn the invariant features in the spatial domain. The spatial affine transformations include translation, scaling, shearing, and rotation. The STN [27] applies the affine transformation by regressing six parameters of the transformation matrix. Fu et al. [28] propose a recurrent attention CNN (RA-CNN) for fine-grained image recognition. RA-CNN [28] recursively learns discriminative region attention and region-based feature representation at multiple scales in a mutually reinforced way. Both the methods can be used to learn a pattern of cropping in many computer vision fields. However, neither of them seems to work fine in head pose estimation. In our experiments, the end-to-end training of them cannot make the network converge to an ideal result. This may be due to the following reasons: (i) the head pose regression (HPR) is easily overfitted and so end-to-end training with these methods converge to a locally optimal solution. (ii) the STN [27] changes the head pose for the image during training because it learns to rotate and shear. (iii) The regression problem is more complicated than the classification problem, and the head pose regression (HPR) network may not provide a good gradient for the coordinate regression network. The proposed CCM can solve these problems perfectly. 3 Method In this section, we describe our CCM method and background augmentation approach. The overview of the method with CCM is shown in Fig. 4. Fig. 4Open in figure viewerPowerPoint Overview of the proposed method with CCM. CCM and HPR net update parameters by training alternately. The HPR net can be any CNN structure 3.1 Problem formulation The problem is to estimate three Euler angles from an input image containing portraits. The targets are defined as a vector . denotes the head pose (e.g. means pitch). So our task is to find a function F so that it predicts given every image x. The aim is to minimise the mean-square error (MSE) between ground truth and predictions (see (1)) (1)where is the prediction of the network of the image and is the ground-truth head pose angles. N is the batch size. In this study, the part whose angle is or is neglected. 3.2 Cropping convolutional module (CCM) CCM is designed to crop the attentional face automatically. The CCM has two steps: (i) regress the coordinates of the bounding box (left-top and right-bottom, four parameters in total). (ii) Crop the original image using the regressed points. The coordinates regression network can be any regression network that maps the input image to four outputs. The cropping operation is an affine transformation. As we have mentioned in Section 2, neither STN [27] nor RA-CNN [28] seems to work fine in head pose estimation due to the three reasons: (i) the HPR is easily overfitted and so end-to-end training with these methods converge to a locally optimal solution. (ii) STN [27] changes the head pose of an image during training because it learns to rotate and shear. (iii) The regression problem is more complicated than the classification problem and the HPR network may not provide a good gradient for the coordinate regression network. The proposed CCM can solve these problems perfectly. End-to-end training requires that cropping operations be derivable, so we need to define the gradient of cropping operations. However, end-to-end training lacks the sample amplification brought by random cropping, which makes the network easy to overfit. Since cropping is exactly what we need the network to learn in training. Therefore, we choose to make the cropping network and HPR network train alternately. Based on the above discussion, CCM can be implemented [27] with four parameters (2)where is the input image with C channels, W is the width and H is the height. f is a convolutional regression network. Note that the bounding box coordinates are normalised between 0 and 1. So the affine transformation can be defined as (3)where are the source coordinates of the input image and are the target coordinates of the output image. and are the width and height of the predicted bounding box. The transformation matrix above combines the translation transformation and scaling transformation, which has the same effect as cropping and scaling. Following [27], the output feature map V using bilinear sampling is defined as (4) where is the value of input image in (m,n) of channel c. is the output value for pixel at location . So, the derivatives of is (5) and similar to (5) for , , and . In an implementation, the network structure is a stack of convolutional structures. In the last layer of the network, we use the fully connected layer and softmax layer to get the network output between 0 and 1. The visualisation of CCM cropping is shown in Fig. 5. Fig. 5Open in figure viewerPowerPoint CCM cropped images. The first and third rows are original images w/o cropping and the second and fourth rows are the corresponding CCM cropped images 3.3 HPR net HPR net is the main part of head pose estimation. In our experiment, we have tried SSR-Net [18], HopeNet [16], and FSA-Net [17]. The general structure of the HPR network is shown in Fig. 4. 3.4 Training alternately Different from STN [27], we do not need the gradient of input feature because it does not help to coordinate regression. Apart from that, training the network end-to-end will make the HPR network overfit to the training dataset. To allow CCM to learn the attentional cropping area from the original image and at the same time to allow HPR network generalise as much as possible, training the HPR network with both random cropped image and CCM cropped image is necessary. So, the training trick is as follows: (1) Train the HPR network with a random cropped image and CCM cropped image. (2) Freeze the parameters of the HPR network and train the CCM with the original image. (3) Repeat (1) and (2) until CCM and HPR network converge. The HPR network makes full use of randomly cropped images in the training set, which makes the HPR network has good generalisation performance. In fact, training directly with random cropped images can also generalise to a good result, but the result is not as good as the training alternately with two models. Training alternately enables CCM to learn attentional cropping while the HPR network has good generalisation performance. 3.5 Background augmentation There is always a problem that the network may output different head pose angles with the same person in different backgrounds. As we think, due to the simplicity of the background of the training dataset, the network is not able to capture useful information for head pose estimation in complex background. To make it pay more attention to learning the face feature rather than the background, we propose the background augmentation method and force the network to obtain facial features for more accurate prediction. We use Mask R-CNN [19] to get the mask corresponding to the foreground of the image. The mask is used to replace the original image with a background dataset where the mask value is 0, and if not, remains as it is. The approach works when loading a batch of data, which is like an augmentation method. For this reason, our approach is easy to implement and does not increase any computational complexity. The process of background augmentation is shown in Fig. 6. Since the Mask R-CNN [19] does not work well on some certain images, we have a solution for that to make sure the training set is clean. We only use those pictures that can be detected by a face detector and in our experiment we just use Multi-task Convolutional Neural Network(MTCNN) [29] as the face detector. The entire training dataset first goes through Mask R-CNN [19] to get the face mask. Then the original image's background is replaced with a completely black background. The mask is judged by the face detector by detecting the augmented image. So once the augmented image cannot be detected by the face detector, we replace its mask with the mask whose pixel values are 1. Fig. 6Open in figure viewerPowerPoint Flow chart of background augmentation. Given an input image, the proposed method obtains its mask using Mask R-CNN. The background is randomly chosen from the COCO dataset. The original image's background is replaced by the background image with the mask In reality, we may encounter a situation that more than one person appears in the picture. For this situation, it is better not to change the strategy for an image with only a person's face. Since in a real scenario, we are not able to crop an image with only a person's face when more than two faces overlap very much. Therefore, the network will learn real-world knowledge from training data. Owing to the lack of face landmarking, our model is insensitive to blinking and closed eyes and glasses. 4 Datasets and experiments 4.1 Datasets We use the 300W-LP [23], AFLW2000 [23], Pointing′04, and BIWI [22] datasets through all of the experiments. For background augmentation, the COCO [20] dataset is used as background candidate. The 300W-LP [23] dataset is synthetically generated from the 300W dataset by predicting its depth to get its profile views with 3D rotation. It has 61,225 face images in total. Owing to its image edge distortion (see Fig. 7), the network may predict head pose angles by learning the image edge distortion information. Fig. 7Open in figure viewerPowerPoint Overview of the 300W-LP dataset. The network may overfit because of the edge distortion The background augmentation approach solves this by replacing its background edge with a real-world background image. Face location is labelled in the dataset so we use it for training. The AFLW2000 [23] is a dataset with 68 landmarks re-annotated using a 3D model. There are 2000 images in this dataset and it is an in-the-wild dataset. The BIWI [22] dataset contains 24 videos of 20 subjects with roughly 15,000 frames. It is recorded with a Kinect while turning their heads around freely. For each frame, depth and red–green–blue (RGB) images are provided, together with ground truth in the form of the 3D location of the head and its rotation angles. Pointing′04 is a dataset with challenging nature and much variety in face poses. Besides, in this dataset, some of the subjects have facial hair (mustaches and beard) and some are wearing eyeglasses. The dataset uses and between two consecutive poses in the horizontal and vertical orientation, respectively. The total number of datasets is 2790. Since datasets in [12, 30–32] were used to evaluate results, we decide to use this dataset as well. COCO [20] is a large-scale object detection, segmentation, and captioning dataset. COCO [20] has several types of labels such as object segmentation, recognition in context, superpixel stuff segmentation, 330 K (220 K labelled), 1.5 million object instances, 80 object categories, 91 stuff categories, five captions per image and 250,000 people with landmarks. For us, this dataset has a huge amount of complex background information for background data enhancement. 4.2 Experimental setting The experiments are conducted on a machine with an NVIDIA RTX2070 and an Intel i7 CPU. The background augmentation approach is used in some experiments. For the network, the input size is , which can greatly reduce the amount of calculation w/o reducing the accuracy. For training CCM, we initialise the fully connected layer's bias to and weights to zeros. In this way, the cropped image is close to the original image at the beginning of training. We set the learning rate of CCM to 0.003. The HPR network's settings are the same as the corresponding paper. 4.3 Experiments There are are parts to our experiments. (1) Results on benchmark dataset for background augmentation. We train a lot of HPR networks on 300W-LP [23] dataset and test on AFLW2000 [23] and BIWI [22] datasets. A lot of methods are trained on that so that we can compare it with those in this dataset. (2) Results on AFLW2000 [23] for CCM. The training dataset is 300W-LP [23]. (3) Effectiveness of CCM. We randomly select a picture and select different margin sizes to test the robustness of the network output. (4) Comparison with machine learning-based methods. (5) Variance of predicted values with (w) and w/o background augmentation. We carried out research on the robustness against the complex background of the proposed background augmentation approach. 4.3.1 Results on benchmark dataset for background augmentation We first explore the effectiveness of background augmentation and results are shown in Tables 1 and 2. FAN [11] and Dlib [10] are both landmark-based methods and they use face landmarks and pre-defined 3D model to predict head pose. 3DDFA [33] is a CNN-based model that predicts the 3D model from an RGB image. Hopenet [16] is a landmark-free model using a multi-loss approach. SSR-Net [18] is a coarse-to-fine model used to predict human age but also can be used to predict head pose. FSA-Net [17] is the SOTA model with a carefully designed feature aggregation structure and attention structure. Table 1. MAE of yaw, pitch and roll in AFLW2000 dataset using different methods Method MAE (yaw) MAE (pitch) MAE (roll) MAE FAN [11] 6.36 12.30 8.71 9.12 Dlib [10] 13.153 23.633 10.545 15.77 3DDFA [33] 5.40 8.530 8.25 7.39 Hopenet(=2) [16] 6.47 6.56 5.44 6.16 BG-Hopenet(=2) 6.13 6.44 5.41 5.99 SSR-Net 5.24 7.19 5.94 6.12 BG-SSR-Net 5.18 7.04 5.82 6.01 FSA-Net() [17] 4.82 6.19 4.76 5.25 BG-FSA-Net() 4.84 6.11 4.74 5.23 Mean error of yaw, pitch and roll. Bold values indicate the best results of the current experimental group. Table 2. MAE of yaw, pitch and roll in BIWI dataset using different methods Method MAE (yaw) MAE (pitch) MAE (roll) MAE FAN [11] 8.53 7.48 7.63 7.89 Dlib [10] 16.8 13.8 6.19 12.2 3DDFA [33] 36.2 12.3 8.78 19.1 hGLLiM [21] 6.06 7.65 5.62 6.44 Hopenet(=2) [16] 5.17 6.98 3.39 5.18 BG-Hopenet(=2) 5.02 6.56 3.44 5.00 SSR-Net 4.49 6.31 3.61 4.80 BG-SSR-Net 4.44 6.32 3.55 4.77 FSA-Net(w/o) [17] 4.56 5.15 2.94 4.22 BG-FSA-Net(w/o) 4.61 5.00 2.93 4.18 Mean error of yaw, pitch and roll. Bold values indicate the best results of the current experimental group. The prefix 'BG' in Tables 1 and 2 represents that the background augmentation method is used. Tables 1 and 2 prove two things. Landmark-free methods generally outperform landmark-based methods in results since side-face is hard to recognise (accurate face key points) and the landmark-free method can mine more information, which is helpful for head pose estimation. The proposed background augmentation works fine in both AFLW2000 [23] and BIWI [22]. Background augmentation provides sufficient samples for training, avoids the influence caused by edge distortions, and improves the generalisation performance of various landmark-based models. 4.3.2 Results on benchmark dataset for CCM To prove the CCM module can improve the average performance of head pose estimation, we train our model with CCM in 300W-LP [23] and test in AFLW2000 [23]. Except for the CCM's hyper-parameters, the other network's hyper-parameters for the HPR net are consistent with the original papers. Note that we change the input size of Hopenet [16] to 64 for a fairer comparison. The prefix 'CCM' in Table 3 represents that the CCM method is used through training. Table 3. MAE in AFLW2000 dataset w and w/o CCM Module MAE (yaw) MAE (pitch) MAE (roll) MAE Hopenet( =1) [16] 6.92 6.64 5.67 6.41 CCM-Hopenet( =1) 5.44 7.42 6.06 6.30 SSR-Net 5.14 7.09 5.89 6.01 CCM-SSR-Net 4.70 6.50 4.95 5.38 FSA-Net() [17] 4.82 6.19 4.76 5.25 CCM-FSA-Net() 4.45 6.32 4.63 5.13 Mean error of yaw, pitch, and roll Bold values indicate the best results of the current experimental group. All of the methods w/o CCM are trained with random cropping transformation for generalisation as the original paper. Table 3 indicates training alternately with CCM improves results. Methods with CCM reduce the MAE of almost all poses. The reason is that during training, CCM learns to crop the attentional face and provide the best input for the HPR network. In the meanwhile, the HPR network makes use of both random cropped images and CCM cropped images for generalisation. Although CCM and background augmentation can be used at the same time, there is no significant improvement in results compared to using CCM alone, since CCM makes head pose estimation generalise better than background augmentation. 4.3.3 Effectiveness of CCM To prove the proposed CCM really learns to provide the HPR network the most suitable cropped image and is robust to crop margin, we randomly choose an image in the AFLW2000 dataset, different bounding box margin is tested. Figs. 8, 9–10 show the predicted results using our CCM and the original method. In the experiment, we randomly choose an image in different bounding box margins. The bounding box margin ranges from 0 to 50. For example, if the width and height of the image is , the image size ranges from 412 to 512. Using the proposed CCM, the range of the network's output is smaller in change, so the proposed CCM c