Data augmentation with occluded facial features for age and gender estimation

IET BiometricsVolume 10, Issue 6 p. 640-653 ORIGINAL RESEARCH PAPEROpen Access Data augmentation with occluded facial features for age and gender estimation Lu En Lin, Lu En Lin Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, ChinaSearch for more papers by this authorChang Hong Lin, Corresponding Author Chang Hong Lin chlin@mail.ntust.edu.tw Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, China Correspondence Chang Hong Lin, Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, No. 43, Keelung Rd., Da'an Dist., Taipei, Taiwan, China. Email: chlin@mail.ntust.edu.twSearch for more papers by this author Lu En Lin, Lu En Lin Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, ChinaSearch for more papers by this authorChang Hong Lin, Corresponding Author Chang Hong Lin chlin@mail.ntust.edu.tw Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, China Correspondence Chang Hong Lin, Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, No. 43, Keelung Rd., Da'an Dist., Taipei, Taiwan, China. Email: chlin@mail.ntust.edu.twSearch for more papers by this author First published: 26 April 2021 https://doi.org/10.1049/bme2.12030AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Here, the feature occlusion, a data augmentation method that simulates real-life challenges on the main features of the human face for age and gender recognition is proposed. Previous methods achieved promising results on constrained data sets with strict environmental settings, but the results on unconstrained data sets are still far from perfect. The proposed method adopted three simple occlusion techniques, blackout, random brightness, and blur, and each simulates a different kind of challenge that would be encountered in real-world applications. A modified cross-entropy loss that gives less penalty to the age predictions that land on the adjacent classes of the ground truth class is also proposed. The effectiveness of our proposed method is verified by implementing the augmentation method and modified cross-entropy loss on two different convolution neural networks, the slightly modified AdienceNet and the slightly modified VGG16, to perform age and gender classification. The proposed augmentation system improves the age and gender classification accuracy of the slightly modified AdienceNet network by 6.62% and 6.53% on the Adience data set, respectively. The proposed augmentation system also improves the age and gender classification accuracy of the slightly modified VGG16 network by 6.20% and 6.31% on the Adience data set, respectively. 1 INTRODUCTION In recent years, facial analysis tasks have become a very popular research topic. Among these tasks, age and gender classification provides basic and essential information about the human face. The information obtained from age and gender classification can be valuable to a wide variety of applications, such as automatic surveillance [1], targeted advertising [2] and so on. In some previous studies, [3-6] showed promising results in constrained environment settings. Even with the promising results in [3-6], there is still room for improvements in real-life and unconstrained situations. There are challenges with real-life and unconstrained situations, such as the subject's face being obstructed, images with low resolution, blurry images, extreme lighting conditions or subjects with missing distinctive features, that even humans cannot identify easily. Deep learning methods have proved a lot of success compared to the traditional machine learning methods in age and gender classification tasks. The most recent state-of-the-art methods [8-11] all use convolutional neural networks (CNN). When training a CNN, the training data plays a significant role. With more training data, CNN can achieve better generalization. However, data sets with correctly labelled data are very scarce and difficult to collect. When training data is limited, overfitting becomes a critical issue. To deal with the overfitting issue, several types of strategies can be adopted, such as weight regularization [11], dropout [12] layers, and data augmentations [13-15] and so on. There are some common methods used in practice. Data augmentation is a way to increase the number of training data without collection of new data. Deep learning methods have proved a lot of success compared to traditional machine learning methods in the age and gender recognition tasks, as most recent state-of-the-art methods [7, 8] in these tasks utilize a CNN. In CNN, the training data plays an important role. With more training data, the network is easier to achieve a better generalization. However, correctly labelled data can be scarce and difficult to collect for age and gender recognition using only faces. In a limited training images situation, overfitting is prone to happen. The proposed method aims to augment images by simulating the real-life challenges described earlier, obstruction, extreme lighting, and low resolution, in the region around the facial feature of the human face. This creates a more diverse training data for the network to adapt to the real-life environments, and improving the generalization and robustness of the network. A modified cross-entropy loss is also proposed; this modified cross-entropy loss gives less penalty to the predictions that lands on adjacent classes of the ground truth classes. By doing so, the network learns to predict closer to the ground truth, yielding better results. Our augmentation method applies three simple image processing techniques, blackout, random brightness, and blur, on random areas of the input facial image, each simulating different kinds of real-life challenges that could happen in real-life situations. This augmentation method is tested on the Adience database [11] and implemented on two networks, the simpler AdienceNet [10] and the more complex VGG16 [12], to show that our augmentation method works on both simple and complex networks. With the outbreak of COVID-19 in 2019, wearing a face mask is required in many public places. Wearing a face mask occludes the two main features of the human face, the nose and the mouth. Without these two features, the CNN must rely on other feature of the human face to perform age and gender classification, which is what the proposed method aims to achieve. For this study, the main contributions are as follows: (1) We proposed an image augmentation method to improve the generalization and robustness of CNNs by simulating real-life situations. (2) We also proposed a modified cross-entropy loss function, which gives less penalty to the age predictions that land on adjacent age classes of the ground truth age class. (3) Our proposed augmentation method was implemented and tested on two different networks, the slightly modified AdienceNet [16] and the slightly modified VGG16 [17], to show the viability of our augmentation method despite the complexity of the network. (4) Our proposed augmentation method uses simple image processing techniques, making it easy to implement and consumes very little computing resources. Based on our contributions listed above, the proposed augmentation system improves the age and gender classification accuracy of the slightly modified AdienceNet [16] network by 6.62% and 6.53% on the Adience data set [18], respectively. The proposed augmentation system also improves the age and gender classification accuracy of the slightly modified VGG16 [17] network by 6.20% and 6.31% on the Adience data set [18], respectively. The proposed modified cross-entropy loss improves age and gender classification accuracy of the slightly modified AdienceNet [16] network by 0.98% and 2.53% on the Adience data set [18], respectively. The proposed modified cross-entropy loss also improves the age and gender classification accuracy of the slightly modified VGG16 [17] network by 0.96% and 1.37% on the Adience data set [18], respectively. The rest of this study is organized as follows, Section 2 reviews some preceding works on gender and age recognition tasks and data augmentation methods, Section 3 describes the proposed augmentation method; Section 4 describes the test results on different implementations, and analyses the effectiveness of the proposed feature occlusion and the proposed modified cross-entropy loss; and Section 5 concludes the study by summarizing the proposed method, and discusses potential future works. 2 RELATED WORKS Many feature extraction methods have been used to learn important information from facial images, and these features provide meaningful attributes for machine learning classifiers to learn and perform age and gender estimation tasks. Tapia and Perez [19] used local binary patterns [20] to extract features and performs gender estimation with feature selection based on mutual information. Luu et al. [3] combined support vector machines (SVMs) [21] and active appearance models [22] to perform age estimation tasks, utilizing the shape and texture of the faces. El Dib and El-Saban [23] combined active shape models [24] and Gabor features [25] on SVM [21] to perform age estimation tasks. Hu et al. [5] utilized the gradient information by extracting local directional pattern features [26] on facial images to perform age and gender estimation. All of the methods mentioned above were tested on constrained data sets with under control conditions, such as FERET [27], FG-NET [28] and MORPH [29]. Even though these methods showed promising results, there is still room for improvement in unconstrained environments. The results achieved on the constrained environments are promising, but the results of the unconstrained environments are far from the requirements to perform in real-life situations. For unconstrained environments, Wei et al. [30] proposed a dynamic image-to-class warping to perform occluded face detection and occluded face recovery. Yang et al. [31] proposed the nuclear norm-based matrix regression, which used the minimal nuclear norm of the residual representation image as a criterion in the matrix regression model. Song et al. [32] proposed a pairwise differential siamese network that uses a mask learning strategy to find and discard the corrupted feature elements for facial recognition. All of the methods mentioned above uses reconstruction, dictionary, filter and matching based methods to perform facial recognition. All the methods showed promising results, but only facial recognition is performed. On the other hand, deep learning-based methods have shown better performance in both unconstrained and constrained cases than traditional machine learning methods, but deep learning methods require more computing resources. With the rise of deep learning, CNNs have been extensively used in recent age and gender estimation tasks. One of the major advantages of CNNs is that the feature extraction processes are managed automatically by the convolution layers. Through the numerous training passes, the convolution layers learn to obtain meaningful information from the images by updating the weights of the network in each training pass. Levi et al. [16] proposed a simple convolutional network, the AdienceNet [16], to perform age and gender classification tasks. Lapuschkin et al. [33] implemented age and gender classification tasks on multiple popular CNNs, such as CaffeNet [34], GoogleNet [35], and VGG16 [17], with different facial alignment settings and pre-trained model weights. Chen et al. [36] used a deep convolutional neural network (DCNN) that consists of three modules, the classifier, the regressor, and the error-correcting module, to perform age estimation tasks. Wolfsharr et al. [37] proposed a hybrid machine learning system that extracts features with a pre-trained CNN and classifies the gender with an SVM [21]. Rodriguez et al. [8] approached the age and gender classification tasks by using an additional CNN utilizing attention mechanism [38], feeding small patches of images with higher attention into the classification network instead of the entire face images. Rothe et al. [9] proposed deep expectation, which used VGG16 [17] as the backbone and expected value formulation for age regression to perform age estimation. Even though the above-mentioned methods produced recognizable performance, none of the mentioned methods performs age and gender estimation in one single model. Levi et al. [16], Lapuschkin et al. [33] and Rodriguez et al. [8] perform age and gender estimation with two separate models. Performing age and gender estimation together in one model, can save training time and computing resources. The performance can still be improved by implementing different training strategies, such as training losses, regularizations, and data augmentations. Wang and Perez [14] compared the performance between generating augmented data with generative adversarial networks [39] and using traditional transformations, which consists of shifting, scaling, rotation, flipping, distortion and hue shading of the training images. They also proposed a network that attempts to learn the optimal augmentation techniques to be used for a given data set. Zoph et al. [13] identified 22 data augmentation methods that are beneficial to object detection tasks. DeVries and Taylor [40] proposed cutout, a simple regularisation technique that randomly masks out square regions of input images during training, and it can be interpreted as an extension of dropout [12] in the input space, and showed performance improvements in object detection tasks. All of the above-mentioned methods provided positive improvements to the performance of the networks, but most of these methods were implemented and tested on object detection and recognition tasks. The proposed method was designed for facial analysis tasks and aimed to simulate real-life situations that could happen to a human face, such as obstruction, blur and extreme lighting. 3 PROPOSED METHOD In this section, the proposed method, feature occlusion, the slightly modified AdienceNet [16], the slightly modified VGG16 [17], the modified cross-entropy loss and each of the component are discussed in details. 3.1 Feature detection The purpose of the proposed method is to improve the generalization and robustness of convolutional neural networks by augmenting facial images to simulate real-life and unconstrained situations. These situations include facial features being blurred, obstructed or missing. For example, wearing sunglasses blocks the eyes and part of the nose, wearing a facial mask blocks the mouth and the nose or extreme lighting that cause features to disappear. To create real-life and unconstrained situations, we perform occlusions on the three main features, eyes, nose and mouth, of human faces. To locate these features, we perform face detection on the input facial images, and face alignment is also performed to improve the quality of input images. Detailed information on face detection and face alignment used in the proposed method is described in Sections 3.1.1 and 3.1.2, respectively. 3.1.1 Face detection and alignment To perform occlusion on facial features, we have to find the facial features first. We extract the facial region and facial landmarks using the MTCNN [41] face detection tool. The MTCNN [41] is a multitask cascaded CNN unifying facial detection and alignment tasks proposed by Zhang et al. The MTCNN [41] pipeline consists of three stages, including two networks and three different tasks. The first stage uses a proposal network to find the candidates of facial regions, and these candidates are fed into the second stage's refine network to reject false candidates. Finally, the third stage identifies and locates facial region and five facial landmarks. The accuracy of the MTCNN [41] is very high, in the 18,622 images of the Adience data set [16], the MTCNN found all but except for 34 faces. Most of these images are extremely low in quality or having the major facial features heavily obstructed. In some cases, humans cannot even estimate the age of the subject. For the heavily obstructed images, they are still used for training, but they will not participate in the feature occlusion process. An example of facial region and landmark detection of MTCNN [41] and low-quality images are shown in Figure 1. FIGURE 1Open in figure viewerPowerPoint (a) Example of facial landmarks extracted by the MTCNN [41] on an image from the Adience data set [18] and (b) low-quality images where MTCNN failed to detect facial features After the features are detected, an affine transformation is performed on the original image using the OpenCV [42] image processing library to align the facial image. 3.2 Feature occlusion After facial landmarks are obtained and facial images have been aligned, feature occlusion is performed. Section 3.2.1 describes how feature regions are selected, Sections 3.2.2–3.2.4-3.2.2–3.2.4 describe how each type of occlusion is performed on the selected feature regions. 3.2.1 Region selection As described in Section 3.1.1, we detected five facial landmark coordinates, left eye, right eye, nose, left mouth corner and right mouth corner, from the input facial image. We use these coordinates to determine the region of occlusion. There are five different regions selected in the proposed method, left eye, right eye, nose, both-eye and mouth regions. The five selected regions are shown in Figure 2. FIGURE 2Open in figure viewerPowerPoint Five selected regions of the proposed method on images from the Adience data set [18]: (a) left eye, (b) right eye, (c) both-eye, (d) nose and (e) mouth To determine the region size, we use the distance between two eyes to calculate o, the offset that is used to transform landmark coordinates into rectangular regions. The calculation of o is computed with Equation (1): o = ( Re x ' − L e x ' ) / γ (1)where γ is the parameter that controls the size of the occluded region, which is set to three in our experiments; R e x ′ and L e x ′ are the x coordinates of the left and right eye. For the left eye, right eye and both-eye regions, we calculate the selected region by expanding the coordinates into four different coordinates, creating a square. The expansion process of the left eye coordinates into four coordinates of the region is shown in Equation (2): L e 0 = ( L e x ′ − o , L e y ′ − o ) L e 1 = ( L e x ′ − o , L e y ′ + o ) L e 2 = ( L e x ′ + o , L e y ′ − o ) L e 3 = ( L e x ′ + o , L e y ′ + o ) (2)where L e 0 , L e 1 , L e 2 and L e 3 are the four coordinates of the selected region for the left eye, L e x ′ and L e y ′ are the left eye coordinates from the aligned MTCNN [41] facial image, the expansion process is the same for the right eye and nose region. For the mouth region, the left mouth and right mouth corner coordinates are used for the expansion; we expand the left mouth and right mouth corners into four coordinates, creating a rectangle. The expansion process for the mouth region is shown in Equation (3): m 0 = ( L m x ′ − o , L m y ′ − o ) m 1 = ( L m x ′ − o , R m y ′ + o ) m 2 = ( R m x ′ + o , L m y ′ − o ) m 3 = ( R m x ′ + o , R m y ′ + o ) (3) 3.2.2 Random brightness Extreme lighting conditions not only affect the whole input facial image but also affect certain regional areas of the input facial image. For example, heavy sunlight can cause bright spot on facial images, inadequate or blockage of light source can cause strong shadows casting on facial images, and the angle of the face when the image is taken can cause uneven light distribution on the facial image. This can cause a significant difference in contrast and light intensity to neighbouring regions on the facial image. When these extreme lighting cases happen on the main facial features, it can confuse the network and decrease the classification accuracy. Figure 3 shows some examples of extreme lighting cases from the Adience data set [18], the top row shows where extreme lighting cases affecting the whole facial image, and the bottom row shows where extreme lighting cases affecting only partial region of the facial image. FIGURE 3Open in figure viewerPowerPoint Examples of extreme lighting images on both the whole region (top row) and partial region (bottom row) We adjust the brightness of the input facial image by combining two methods, multiplying the input facial image with a scale and adding or subtracting with a bias. The computation of random brightness is shown in Equation (4): R h , w ′ = α ⋅ R h , w + β (4)where R h , w and R h , w ′ represent the region of height h and width w before and after adjusting brightness, pixel values in R h , w ′ were clipped between 0 and 255 to prevent the values from getting out of boundary. α and β are the randomly generated scale and bias, which controls the brightening and darkening of the region, as shown in Equation (5): { 1.0 < α < 1.6 , 0 < β < 16 i f brightening 0.8 < α < 1.0 , − 64 < β < 0 i f darkening (5) We set α to between 0.8 and 1.0 for darkening because when α is smaller than 0.8, the darkening results does not look normal, and the β value is also set to the higher negative to compensate for the small range of α. The results of random brightness performed on image from the Adience data set [18] with different α, β and bias are shown in Figure 4. FIGURE 4Open in figure viewerPowerPoint Results of random brightness performed on the nose region: (a) α = 1.45 , β = 10 and (b) α = 0.85 , β = − 30 3.2.3 Blackout Blackout is the second occlusion technique used in the proposed method, the goal of blackout is to recreate real-life situations where the facial features of the subject are obstructed. This can happen when the subject is wearing glasses, sunglasses, masks, making hand movements, or being behind objects. Figure 5 shows some examples of obstructed faces from the Adience data set [18]. FIGURE 5Open in figure viewerPowerPoint Examples of obstructed subject faces We perform blackout by setting the selected region's pixel values to zero, which is black in colour, similar to the method used in Cutout [40]. Cutout [40] is a regularization technique that applies a fixed-size zero-mask to a random location of each input image, which can be interpreted as applying a spatial dropout before the input space. Blacking out facial features forces the network to focus more on the minor details of the face instead of the main facial features. By doing so, we can improve the robustness of the network when the main features of the face are obstructed. Figure 6 shows the results of blackout on an image from the Adience data set [18]. FIGURE 6Open in figure viewerPowerPoint Results of blackout on (a) nose (b) mouth and (c) both-eye region 3.2.4 Blur For the last occlusion, we use the blur occlusion. The main goal of the blur occlusion is to stimulate low resolution or blurry images. Low-resolution image is caused by the upscaling of the input image because the size of the face is smaller than the input size of the network. While the image is taken, movement causes the image to be blurry or improper focus of the camera lens can also lead to blurry images. Figure 7 shows examples of low-resolution and blurry images from the Adience data set [18]. FIGURE 7Open in figure viewerPowerPoint Examples of low-resolution and blurry images from the Adience data set [18]. Top row shows the upscaled images and bottom row shows the blurry images The blurred occlusion is applied to the selected region to achieve the blur effect, and the process of blurring the selected region is shown in Equations (6) and (7): R h , w ′ = R h , w ∗ K (6) K = 1 k w ⋅ k h ⋅ [ 1 1 1 1 1 1 ⋯ 1 1 1 1 ⋮ ⋱ ⋮ 1 1 1 ⋯ 1 1 ] (7)where R h , w and R h , w ′ represent the region of height h and width w before and after applying blur occlusion, * symbol denotes the convolution operator, and K is the two-dimensional filter kernel of height k h and width k w , k h and k w were both set to 15 in our experiments. Figure 8 shows the result of blur occlusion on an image from the Adience data set [7]. FIGURE 8Open in figure viewerPowerPoint The result of the blur occlusion 3.3 Data augmentation Due to the limited amount of training data in the Adience data set [7], we performed data augmentation to prevent the possibility of overfitting. After feature occlusion is performed, we obtained an occluded image with the desired height and width of 256 × 256 pixels, and we cropped from the centre and four corners of the occluded image, yielding five images with the size of 228 × 228 pixels. Horizontal flip is then applied to the cropped images, resulting in ten times the amount of original data in the Adience data set. Figure 9 shows the results of data augmentation on an image from the Adience data set [7]. FIGURE 9Open in figure viewerPowerPoint Results of data augmentation 3.4 Convolutional neural network To verify the effectiveness of feature occlusion, modified cross-entropy loss and data augmentation on general CNN architectures, we trained and tested on two different networks. The first network is a slightly modified AdienceNet, originally proposed in [16]. The AdienceNet [16] is a neural network with three convolutional layers and three fully connected layers. It is a shallow neural network with around 8.5 million parameters. The second network is slightly modified VGG16 [17]. VGG16 [17] is a deep CNN with much more parameters compared to the AdienceNet [16]. With more parameters, the VGG16 [17] is capable of learning more complex features, but it requires longer training time. According to [33], VGG16 [17] performs very well in age and gender classification tasks, and it can achieve better scores than other popular CNNs such as CaffeNet [34], GoogleNet [35] and AdienceNet [16]. Detailed information of the slightly modified AdienceNet [16] and the slightly modified VGG16 [17] is described in Sections 3.4.1 and 3.4.2. 3.4.1 Modified AdienceNet [16] We modified the AdienceNet [16] to best suit our experiment. We changed the last layer of the network from one fully connected layer to two fully connected layers. Of the two fully connected layers, the first one has an output of two for gender prediction and the second one has an output of eight for age prediction. This modification was made so the loss can be calculated separately. The architecture of the slightly modified AdienceNet [16] is shown in Figure 10. FIGURE 10Open in figure viewerPowerPoint Architecture of modified AdienceNet [16]. The red box shows the modifications made The modified AdienceNet [16] takes in input images with a size of 228 × 228 × 3 pixels, and then the image is fed into the following convolutional layers: (1) Conv1 with 96 filters of the kernel size 7 × 7, stride of 4. (2) Conv2 with 256 filters of the kernel size 5 × 5, stride of 1. (3) Conv3 with 384 filters of the kernel size 3 × 3, stride of 1. Each convolutional layer is followed by a rectified linear operator (ReLU) [43], a batch normalization layer [44], and a max-pooling layer with the kernel size of 3 × 3 and stride of two. The output of Conv3 is then fed into four fully connected layers: (1) FC4 with 512 outputs, followed by the ReLU [43] activation and a dropout layer. (2) FC5 with 512 outputs, also followed by the ReLU [43] activation and a dropout layer. (3) FC6 with 8 outputs, which represent the eight age classes. (4) FC7 with 2 outputs, which represent the two gender classes. 3.4.2 Modified VGG16 [17] VGG16 [17] is a deep network proposed by Simonyan et al. with 13 convolutional layers and 3 fully connected layers. We also modified the VGG16 [17] to best suit our experiment. We changed the last layer of the VGG16 [17] from one fully connected layer to two fully connected layers. Of the two fully connected layers, the first one has an output of two for gender prediction and the second one has an output of eight for age prediction. The network is fed with images with the size of 224 × 224 × 3 pixels, followed by five groups of convolution layers: (1) Conv1 and Conv2 with 64 filters of kernel size 3 × 3, stride of 1. (2) Conv3 and Conv4 with 128 filters of kernel size 3 × 3, stride of 1. (3) Conv5, Conv6 and Conv7 with 256 filters of kernel size 3 × 3, stride of 1. (4) Conv8, Conv9 and Conv10 with 512 filters of kernel size 3 × 3, stride of 1. (5) Conv11, Conv12, and Conv13 with 512 filters of kernel size 3 × 3, stride of 1.