Deep learning for face recognition on mobile devices

IET BiometricsVolume 9, Issue 3 p. 109-117 Research ArticleFree Access Deep learning for face recognition on mobile devices Belén Ríos-Sánchez, Corresponding Author Belén Ríos-Sánchez brios@cedint.upm.es orcid.org/0000-0003-3666-5330 Group of Biometrics, Biosignals and Security, Universidad Politécnica de Madrid, Edif. CeDInt-UPM, Campus de Montegancedo, 28223 Pozuelo de Alarcón, Madrid, SpainSearch for more papers by this authorDavid Costa-da Silva, David Costa-da Silva Group of Biometrics, Biosignals and Security, Universidad Politécnica de Madrid, Edif. CeDInt-UPM, Campus de Montegancedo, 28223 Pozuelo de Alarcón, Madrid, SpainSearch for more papers by this authorNatalia Martín-Yuste, Natalia Martín-Yuste Group of Biometrics, Biosignals and Security, Universidad Politécnica de Madrid, Edif. CeDInt-UPM, Campus de Montegancedo, 28223 Pozuelo de Alarcón, Madrid, SpainSearch for more papers by this authorCarmen Sánchez-Ávila, Carmen Sánchez-Ávila Group of Biometrics, Biosignals and Security, Universidad Politécnica de Madrid, Edif. CeDInt-UPM, Campus de Montegancedo, 28223 Pozuelo de Alarcón, Madrid, SpainSearch for more papers by this author Belén Ríos-Sánchez, Corresponding Author Belén Ríos-Sánchez brios@cedint.upm.es orcid.org/0000-0003-3666-5330 Group of Biometrics, Biosignals and Security, Universidad Politécnica de Madrid, Edif. CeDInt-UPM, Campus de Montegancedo, 28223 Pozuelo de Alarcón, Madrid, SpainSearch for more papers by this authorDavid Costa-da Silva, David Costa-da Silva Group of Biometrics, Biosignals and Security, Universidad Politécnica de Madrid, Edif. CeDInt-UPM, Campus de Montegancedo, 28223 Pozuelo de Alarcón, Madrid, SpainSearch for more papers by this authorNatalia Martín-Yuste, Natalia Martín-Yuste Group of Biometrics, Biosignals and Security, Universidad Politécnica de Madrid, Edif. CeDInt-UPM, Campus de Montegancedo, 28223 Pozuelo de Alarcón, Madrid, SpainSearch for more papers by this authorCarmen Sánchez-Ávila, Carmen Sánchez-Ávila Group of Biometrics, Biosignals and Security, Universidad Politécnica de Madrid, Edif. CeDInt-UPM, Campus de Montegancedo, 28223 Pozuelo de Alarcón, Madrid, SpainSearch for more papers by this author First published: 25 February 2020 https://doi.org/10.1049/iet-bmt.2019.0093Citations: 4AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Mobility implies a great variability of capturing conditions, which is not easy to control and directly affects to face detection and the extraction of facial features. Deep learning solutions seem to be the most interesting choice for automatic face recognition, but they are highly dependent on the model generated during the training stage. In addition, the size of the models makes it difficult for their integration into applications oriented to mobile devices, particularly when the model must be embedded. In this work, a small-size deep-learning model was trained for face recognition on low capacity devices and evaluated in terms of accuracy, size and timings to provide quantitative data. This evaluation is aimed to cover as many scenarios as possible, so different databases were employed, including public and private datasets specifically oriented to recreate the complexity of mobile scenarios. Also, publicly available models and traditional approaches were included in the evaluation to carry out a fair comparison. Moreover, given the relevance of template matching and face detection stages, the assessment is complemented with different classifiers and detectors. Finally, a JAVA-Android implementation of the system was developed and evaluated to obtain performance data of the whole system integrated on a mobile phone. 1 Introduction The expansion of mobile devices and the improvements on their frontal cameras have contributed to the actual tendency of using facial biometrics to guarantee secure access to mobile devices [1, 2], applications or services that require to verify the identity of the users [3]. In addition, given the fact that the face plays a key role in daily human interactions and it is used by most the people to identify each other, it is one of the traits included in official identity documents both, at national and international levels. In this context, considerable efforts are made by government agencies in numerous countries not only with the aim of increasing civilian security but also to allow a more fluent and comfortable border crossing [4–6]. On the other hand, face biometrics is becoming increasingly popular because it is perceived as relatively secure and comfortable by the users [7], decreasing the security perception as the user gains experience in biometrics. Also, [7] shows that the most concerning factors regarding face biometrics include the performance under different capturing conditions, the possibility of forging the system with photographs or videos, and privacy concerns derived from the capture and storage of personal images. In fact, facial biometrics is specially sensitive to capturing conditions such as lighting variations, capturing angle, different facial expressions and changes in the appearance of the users (beard, glasses and other accessories). This variability during the capture is particularly difficult to control in the case of mobile applications and directly affects to face detection and feature extraction stages. For this reason, a great deal of research was conducted in the field of automatic face recognition during the last decades, yielding into a great number of algorithms and performance studies [8, 9], most of them focused on face detection and the extraction of facial features. Facial feature extraction methods were traditionally divided into geometrical and holistic approaches, but schemes changed since the presentation of FaceNet [10], a solution for facial recognition based on deep learning. In fact, there is a strong tendency towards deep-learning solutions for facial recognition [11] nowadays because they have shown great potential for this task, surpassing by far previous holistic and geometric approaches and have marked a turning point in face recognition. However, these solutions are highly dependant on the model generated during the training phase: the deeper the model, the better the results, but also the model size increases. For instance, the 90 MB FaceNet model [10] achieves a classification accuracy of 99.63% ± 0.09 on the Labelled Faces in the Wild (LFW) dataset [12], while the reduced model provided by OpenFace (30 MB) [13], which follows the same network architecture but reducing the number of parameters, obtains 92.92% ± 1.34 classification rate on the LFW dataset. The size of the models makes it difficult for their integration into applications oriented to mobile devices, whose storage and processing capabilities are limited, and especially in those cases that require the model to be embedded into the application files. In this work, a small size deep-learning model for face recognition is presented, which was specially designed to be integrated into mobile devices. It is assessed in terms of accuracy, size and timing with the aim of evaluating their applicability to different scenarios, paying special attention to the variability shown by images captured using mobile devices. This analysis is also intended to provide quantitative data, comparable to past, present and future methods. Accordingly, the evaluation also includes two publicly available deep-learning based models and two traditional approaches, holistic and geometrical. To ensure the comparability of the results, a well-defined protocol that was relying on the guidelines outlined by the ISO/IEC 19795 standard [14, 15] is followed during the evaluation and many public datasets or images are involved. Also, some private datasets were also used to cover more complex scenarios like those that could happen when mobile devices are used. Feature extraction is a key stage, but also face detection and comparison stages are quite relevant. A correct location of the face within the image, preferably without face surroundings such as hair, clothes or background, is very important because it directly affects posterior steps. The main challenges at this stage are derived from the influence of the input devices and the variability of the environmental conditions during the capture. On the other hand, a feature matching method that properly fits the feature extraction method is necessary. Given their relevance and the lack of works focused on these stages, three face detectors and three different classifiers were also evaluated. Finally, a JAVA-Android implementation of the system was developed and integrated on an LG Nexus 5 device. A complete evaluation of the whole system regarding performance and resource consumption was carried out. The remaining of this paper is organised as follows. First, the methods involved in the evaluation are presented in Section 2. Then, Section 3 presents the obtained results together with a description of the evaluation protocol and the involved databases. Finally, conclusions are provided in Section 4. 2 Methods 2.1 Face detection and alignment The face detectors compared in this work are presented down below: Viola-Jones detector [16]: A supervised learning method conceived for rapid and accurate visual object detection that is the most popular face detector so far. The learning algorithm is based on AdaBoost, and reduces the original set of visual features to a small number, including the most representative ones, resulting in extremely efficient classifiers. These classifiers are combined in a cascade of increasing complexity, which allows them to quickly discard background regions in the first stages while analysing in detail the object-like regions. In addition, a representation of the images that allows computing the features used by the detector very quickly is introduced. In particular, the implementation provided by OpenCV library [39] is used in this comparative. HOG feature-based detector: A supervised learning method relying on Histogram of Oriented Gradients (HOG) features combined with a linear classifier, an image pyramid, and a sliding window detection scheme. This detector slides a linear classifier over a HOG pyramid as discussed in [17], but using the version of HOG features from [18]. This evaluation makes use of the implementation provided by Dlib [19]. Cascaded convolutional networks-based detector [20]: A cascaded architecture of three carefully designed deep convolutional networks feed with an image pyramid to estimate face position and landmarks location in a coarse-to-fine manner. It exploits the inherent correlation between detection and alignment to increase their performance. Concretely, the implementation provided by FaceNet [10] is employed in this analysis. These facial detectors are able to detect all the faces present in the image. Since this work is focused on a single face, the biggest face within the image is selected, assuming that the user whose identity is validated is closer to the camera. Given the sensitivity of most feature extraction methods to the position of the face elements within the images, specially holistic and model-based approaches, detected faces need to be aligned to a common reference framework. To these ends, some reference points belonging to eyes, nose and mouth are detected and transformed into a fixed position. Finally, all the images are resized to a common size of 160 × 160 px to fulfil the requirements of deep learning approaches and to improve the image comparison performance. 2.2 Feature extraction 2.2.1 Deep learning Three different deep-learning solutions for facial feature extraction based on GoogleNet [21] are compared in this work. This deep neural network architecture introduced the concept of the inception module. These modules perform convolutions with different filter sizes in parallel. Filter sizes are limited to 1 × 1, 3 × 3 and 5 × 5, and the ratio of 3 × 3 and 5 × 5 convolutions should increase in the higher layers. The outputs of the filters in each module are concatenated, forming the input of the next layer. In particular, the version of the architecture including dimensionality reduction, was used to construct the feature extraction models. These models directly learn a mapping from face images to a compact Euclidean space where distances directly correspond to a measure of face similarity. FaceNet [10]: FaceNet employs a deep convolutional network trained to directly optimise the embedding itself using a triplet-based loss function based on Large Margin Nearest Neighbour [22] and a private dataset containing 100–200 M images. As a result, FaceNet surpasses state-of-the-art face recognition performance using only 128-bytes per face, getting a classification accuracy of 99.63% ±0.09 on the Labelled Faces in the Wild (LFW) dataset [12]. Details about the methodology, system architecture and network structure can be found in [10]. The TensorFlow implementation provided in [23] was used in this comparative. Table 1. gb2s_Model2 network definition Type Output size #1 × 1 #3 × 3 reduce #3 × 3 #5 × 5 reduce #5 × 5 Pool proj conv1 (7 × 7 × 3,2) 48 × 48 × 64 — — — — — — max pool + norm 24 × 24 × 64 — — — — — m 3 × 3,2 inception (2) 24 × 24 × 192 — 64 192 — — — norm + max pool 12 × 12 × 192 — — — — — m 3 × 3,2 inception (3a) 12 × 12 × 256 64 96 128 16 32 m, 32p inception (3b) 12 × 12 × 320 64 96 128 32 64 , 64p inception (3c) 6 × 6 × 640 — 128 256,2 32 64,2 m 3 × 3,2 inception (4a) 6 × 6 × 312 128 48 96 16 32 ,56p inception (4e) 6 × 6 × 504 — 80 128,2 32 64,2 m 3 × 3,2 inception (5a) 3 × 3 × 256 96 32 128 — — , 32p inception (5b) 3 × 3 × 256 96 32 128 — — m, 32p avg pool 256 — — — — — — linear 128 — — — — — — normalisation 128 — — — — — — OpenFace [13]: OpenFace was inspired by FaceNet and DeepFace [24] but oriented to mobile scenarios where a real-time face recognition system adapts depending on the context. Accordingly, the system was specially designed to provide high accuracy with little training and low prediction times. OpenFace uses a modified version of FaceNet's nn4 network after a preprocessing stage where the images are aligned. This version reduces the number of parameters and the number of images in the training dataset, which is composed of 500 k images coming from two public face datasets, CASIA-WebFace [25] and FaceScrub [26]. Details about the system architecture, network structure and implementation can be found in [13, 27]. Among the four models offered by OpenFace, the nn4.small2 was selected for this comparative as it gets the best performance (92.92% ± 1.34 on the LFW dataset). gb2s_Model: OpenFace was developed as a general-purpose face recognition library oriented to mobile scenarios. It was aimed to bridge the accuracy gap between publicly available and private face recognition systems encouraged by FaceNet. Since it is oriented to mobile applications and considerably reduces the size of the model (30 MB) compared to FaceNet's (90 MB). However, 30 MB could be still too large for being embedded in some mobile applications. For this reason, a smaller model was trained in this work, which will be referred to as gb2s_Model hereafter, and its size is 12.5 MB. Table 1 presents the architecture of our network. Similarly to FaceNet and OpenFace, our model also was trained using triplets. These triplets consist of two matching and a non-matching roughly aligned face patches and the loss aims to separate the positive pair from the negative by a squared L2 distance margin. This way, the faces of the same person have small distances and the faces of different people have large distances. The selection of triplets is very important, and given their good results, we followed the same strategy as FaceNet. It is composed of a negative exemplar mining component, which ensures consistently increasing difficulty of triplets as the network trains, and a hard-positive mining component that encourages spherical clusters for the embeddings of a single person to improve clustering accuracy. As a result, the network provides an embedding on the unit hypersphere and Euclidean distance represents similarity. A subset of the LFW database was used to train this model. 2.2.2 Geometrical facial features Geometrical relations between some representative facial points, commonly known as Landmarks, can be used to compose the biometric template of the user. Concretely, distances and angles between these points were used in this work. Accurate positioning of the landmarks is needed before calculating the biometric features. It is a crucial step, since distances and angles strongly depend on these locations. To this end, DLib C + + library [19] is used to compute 68 Landmarks from a facial image (Fig. 1). An initial analysis revealed that some points are superfluous and others cannot be detected accurately depending on the pose of the face, so they were discarded (blue points in Fig. 1), giving as a result a final set of 19 Landmarks (orange points in Fig. 1). Figure 1Open in figure viewerPowerPoint 68 Landmarks provided by DLIB. Orange points are selected to compute facial features (distances and angles) Starting from these 19 points, there are 171 possible distances, but many of them are not meaningful. To decide which of them are the most relevant for face recognition purposes, an empirical but exhaustive analysis was conducted. As a result, the distances between the following points were selected: 18-22, 18-37, 18-40, 18-27, 18-32, 18-36, 18-49, 23-18, 23-22, 23-40, 23-43, 23-46, 23-36, 23-9, 37-22, 37-27, 37-40, 37-43, 37-46, 37-32, 37-36, 37-49, 37-55, 43-27, 43-55, 32-40, 32-36, 32-43, 32-49, 32-55,32-9, 49-22, 49-23, 49-27, 49-46, 49-36, 49-9, 28-36, 28-32, 28-55, 28-58, 28-49, 52-58, 52-27, 52-23, 52-22, 52-18. These distances are depicted in Fig. 2a. Figure 2Open in figure viewerPowerPoint Geometrical facial features (a) Distances, (b) Angles from eyes (top-left), nose (top-right), mouth (bottom-left) and chin (bottom-right) To complete the biometric template the angles between some points are computed. They are gathered by facial elements (eyes, nose, mouth and chin) as depicted in Fig. 2b, and corresponds to the segment pairs: (18-40, 40-22), (32-40, 40-36), (27-43, 43-27), (32-43, 43-36), (37-34, 34-46), (18-34, 34-27), (22-34, 34-23), (49-34, 34-55), (18-49, 49-27), (37-49, 49-46), (9-49, 49-55), (18-55, 55-27), (37-55, 55-46), (49-55, 55-9), (49-9, 9-55), (18-9, 9-27), (37-9, 9-46), (32-9, 9-36). 2.2.3 Uniform local binary patterns A robust, accurate, efficient, simple, compact and highly informative texture descriptor that presents high applicability to many computer vision an image processing applications, including biometrics. It was presented on [28] as an extension of the local binary pattern (LBP) descriptor [29]. LBP descriptor calculates a code for every pixel in the image by comparing the intensities of the pixel and its neighbours. These LBP codes contain information about the local features of the texture of the image. The uniform binary patterns (LBPU) appear more frequently in the texture description. The histogram of the calculated LBP codes is typically used as a feature vector, but in the case of the LBPU, a different label is given to each uniform binary pattern and another one is assigned to remaining binary patterns to compute the histogram. Accordingly, a shorter pattern but representative enough of the distribution of the local features of the image is obtained. In addition, with the aim to add some global information, the image is divided into sub-regions and a histogram is calculated for each sub-image. Finally, these histograms are concatenated to compose the biometric feature vector that describes the face. Since the final feature vector contains 97,200 elements, dimensionality reduction could be required by some classifiers. In that case, principal component analysis (PCA) [30] will be applied. 2.3 Matching The classifiers included in the evaluation are briefly described down below: Distance-based classifier (DBC): One of the most extended methods for the comparison of biometric features because of its simplicity and efficiency, and whose non-supervised nature makes it suitable for mobile applications. It provides a numeric value as a result, which represents the difference between two feature vectors. Accordingly, the compared vectors are considered as belonging to the same person if the computed distance is lower than a previously established threshold. Due to deep learning approaches evaluated in this work map the images to a compact Euclidean space, Euclidean distance has been applied in this work. Support vector machines (SVMs): Supervised classifier that is able to analyse the biometric features provided during the training phase to learn a model. Particularly, the classification process is based on the search of the hyperplanes, which better differentiate the biometric features belonging to an individual from those corresponding to other people. Accordingly, following the one-against-all strategy, an SVM per person is created with the aim to differentiate each individual from the others. The numeric value provided as a result represents the similarity between two feature vectors. Consequently, a feature vector is considered as belonging to the claimed user if the obtained metric is higher than a previously established threshold. Given the fact that results depend on both the environmental conditions and the population to which the images belong, this method requires a representative enough database during the training phase. Neural networks (NNs): Learning structures composed of a series of interconnected computing units called neurons. They are able to approximate non-linear output signals (classification results) from input signals (feature vectors). The relevance of each connection is defined by a parameter called weight. The weights are adapted during the training stage as a result of a supervised learning algorithm. In this work, a two-layer feed-forward network with sigmoid hidden neurons and linear output neurons is applied. The network was trained with the Levenberg–Marquardt backpropagation algorithm [31]. 3 Evaluation 3.1 Databases A large number of images coming from different databases, public and private, were used in this evaluation. The database selection aims to cover as most realistic cases of use as possible. Accordingly, the images present different degrees of difficulty regarding pose, scale, background, lighting conditions, appearance, accessories and expressions. These databases are explained in Table 2. Since the model must be able to generalise the knowledge and be capable of recognising new people, such as those potential users of the mobile devices, databases used to train the facial recognition model (neither other public datasets that contains the same users) took part in the evaluation of the system. BioID [32]: A face dataset that was recorded for research purposes in order to compare different face detection methods under realistic conditions. As a result, the images exhibit a large variety of illumination, background and face sizes. The dataset is composed of 1521 grey level images with resolution 384 × 286 px coming from 23 users and recorded in a frontal position. In addition, manually set eye positions are provided together with the images. Table 2. Databases overview Database Access #Users #Images Image size Colour Variations BioID public 23 1521 384 × 286 grey B, L, S EUCFI public 395 7900 180 × 200 colour B, S, E, A ORL public 40 400 92 × 112 grey L, T, E, A Ext. Yale B public 28 16,128 640 × 480 grey L, P PrintAttack public 38 1400 320 × 240 colour L gb2sTablet private 60 16,593 320 × 240 colour B, S, A public 60 4220 640 × 480 colour B, S, A gb2s_Selfies private 26 262 — colour B, L, T, E, A gb2s_IDCards private 26 261 — grey L, T A, B, E, L, P, S and T stands for accessories, background, expressions, lighting, pose, scale and time, respectively. EUCFI [33]: The Essex University Collection of Face Images (EUCFI, so-called Face Recognition Data), provided in order to encourage comparative research. It contains four different datasets with different complexity levels regarding background, scale and facial expressions. Lighting conditions are artificial, a mixture of tungsten and fluorescent overhead, and some people wear glasses or have beards. This collection is composed of 7900 images coming from 395 individuals. The resolution of the images is 180 × 200 px or 196 × 196 depending on the dataset. ORL [34]: A database of faces recorded under controlled environmental conditions at a lab. It is composed of 10 images of each of 40 individuals recorded against a dark homogeneous background with the subjects in an upright, frontal position. For some people, the images were taken at different time, varying the lighting, facial expressions (eyelid aperture, smile) and the use of glasses. The size of each image is 92 × 112 px with 256 grey levels per pixel. Extended Yale B [35]: The extended Yale Face Database B was captured for research purposes and it is specially focused on illumination and pose variability. It contains 16,128 greyscale images of size 640 × 480 px coming from 28 individuals under 9 poses and 64 illumination conditions. Facial marker points are also available together with the images. Print-Attack [36]: A protocolled subset of the Replay Attack Dataset [37]. The Print-Attack Replay Database was created to evaluate the response of systems against spoofing attacks. It consists of 200 colour video clips of printed-photo attack attempts against 50 individuals, under different lighting conditions. It also contains 200 real-access attempt videos from the same people. In this evaluation, only frames coming from the real-access videos were employed. These images have a resolution of 320 × 240 px. They also provide face locations automatically annotated. gb2sTablet_Face_Dataset: A set of 250 frames coming from the gb2sTablet_Face_Dataset which is part of the proprietary gb2sTablet_Database. It contains images from 60 people captured in an indoor environment using artificial lighting and simple but uncontrolled backgrounds. Images were recorded in a frontal position but without restrictions about the distance to the camera, appearance or accessories. The size of the images is 320 × 240 px. gb2sμMOD_Face_Dataset [38]: Composed of 4220 images from the visual spectrum gb2sμMOD Face Dataset, which is part of the gb2sμMOD database. It contains images from 60 contributors captured in three different sessions with different backgrounds and lighting conditions. The first and second sessions include images with a simple background and controlled lighting conditions while the third session presents different real-life backgrounds and varied lighting conditions. Images were recorded in an almost frontal position but without restrictions about the distance to the camera, beard or hairstyle. On the contrary, those users who wore glasses in the first session were imposed to wear them in the second session but giving them the freedom to choose in the third one. Videos were recorded at a resolution of 640 × 480 px using an e-CAM51_USB camera. gb2s_Selfies: A proprietary database-oriented to emulate real daily-life scenarios. Accordingly, this dataset shows different illumination conditions and backgrounds, possible presence and absence of glasses, hear variations, or expressions. 26 individuals participated in the database creation and 10 images per person were recorded in 10 different sessions, one image per day. Each person captured images of his/her face using the frontal camera of his/her own mobile phone the more frontal as possible (Selfie position). gb2s_IDCards: A private database-oriented to evaluate security applications that require an official document to verify the identity of the users. In this case, the same 26 individuals participated in the database creation and 10 images per person were recorded in 10 different sessions, one image per day. Each person captured images of his/her ID card using the back camera of his/her own mobile phone. 3.2 Evaluation protocol To ensure a correct performance measurement of the different combinations of feature extraction and matching methods, as well as to guarantee a fair comparison between them and hopefully, new developments, a well-defined evaluation methodology was followed. It is based on the definitions suggested by the ISO/IEC 19795 standard [14, 15] and it is composed of three parts: (i) Dataset Organisation. To train, configure and evaluate the system, the images are separated into train, development and test subsets. As the system may be able to generalise the knowledge and be capable of recognising new people, such as those potential users of the mobile devices, the images in the training subset come from different databases than development and test subsets. As commented previously, our model was trained using a subset of Development and train