Securable networked scheme with face authentication

IET BiometricsVolume 11, Issue 2 p. 97-108 ORIGINAL RESEARCH PAPEROpen Access Securable networked scheme with face authentication Da-You Huang, Da-You Huang Department of Electrical Engineering, National Chung Hsing University, Taichung, TaiwanSearch for more papers by this authorChun-Liang Lin, Corresponding Author Chun-Liang Lin chunlin@dragon.nchu.edu.tw orcid.org/0000-0002-8520-3450 Department of Electrical Engineering, National Chung Hsing University, Taichung, Taiwan Development Center of Sustainable Agriculture, National Chung Hsing University, Taichung, Taiwan Correspondence Chun-Liang Lin, Department of Electrical Engineering, National Chung Hsing University, Taichung 402, Taiwan. Email: chunlin@dragon.nchu.edu.twSearch for more papers by this authorYang-Yi Chen, Yang-Yi Chen Department of Electrical Engineering, National Chung Hsing University, Taichung, Taiwan Development Center of Sustainable Agriculture, National Chung Hsing University, Taichung, TaiwanSearch for more papers by this author Da-You Huang, Da-You Huang Department of Electrical Engineering, National Chung Hsing University, Taichung, TaiwanSearch for more papers by this authorChun-Liang Lin, Corresponding Author Chun-Liang Lin chunlin@dragon.nchu.edu.tw orcid.org/0000-0002-8520-3450 Department of Electrical Engineering, National Chung Hsing University, Taichung, Taiwan Development Center of Sustainable Agriculture, National Chung Hsing University, Taichung, Taiwan Correspondence Chun-Liang Lin, Department of Electrical Engineering, National Chung Hsing University, Taichung 402, Taiwan. Email: chunlin@dragon.nchu.edu.twSearch for more papers by this authorYang-Yi Chen, Yang-Yi Chen Department of Electrical Engineering, National Chung Hsing University, Taichung, Taiwan Development Center of Sustainable Agriculture, National Chung Hsing University, Taichung, TaiwanSearch for more papers by this author First published: 09 September 2021 https://doi.org/10.1049/bme2.12056AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Recently, facial recognition has been extensively adopted in various fields. Wide applications are associated with a large amount of data transmission so that edge computing is inspired accordingly. In this research task, the major goal of edge computing is to handover a part of the computing work to the terminal equipment; the server only needs to process the results of final return. The IoT configuration proposed includes a perception layer, a transmission layer, and an application layer to fulfil a complete IoT system. In the perception layer, the facial authentication mechanism is adopted. This system is equipped with a highly robust anti-spoofing function, which can avoid criminal access from photos or electronic screens. Finally, the IoT transmission system is realised as the transmission layer. Combined with such a transmission mechanism, one can distribute user facial features to user's electronic devices instead of storing it in the server. This not only saves storage resources and transmission costs, but also allows users to complete data transmission and face authentication easily. 1 INTRODUCTION The Internet of Things (IOT) generally includes a perception layer, a transmission layer, and an application layer. With regard to the features, the well-developed technologies at the transmission layer include RFID, Wi-Fi, ZigBee, Lora, and Bluetooth. Focussing on the IoT system with face recognition as authentication in [1], which uses local binary patterns (LBP), principal analysis (PCA), and Sparse representation classification (SRC) to implement an IOT based facial recognition system. In [2], the authors use local gradient patterns (LGP) and binary histograms of oriented gradients (BHOG) for feature extraction and Raspberry Pi as an IoT computing system. The authors of [3] used passive infrared sensors, LBP features, and k-nearest neighbours (KNN) algorithm and realised them in Raspberry Pi for face recognition. Biometrics has become a popular trend for identity verification while co-operating with different kinds of access control. Classification of biometrics can be roughly divided into physiological characteristics and behavioural characteristics. The commonly adopted identification (ID) systems apply one of the following major features: fingerprint, face, and iris because of their identifiability. In practice, identification is divided into two modes: 'authentication' and 'identification'. Authentication is kind of a one-to-one comparison. First, the user's biometrics is obtained by a capturing device. The system then compares the personal characteristic features already stored in the memory. In contrast, ID is a one-to-many comparison. The face recognition problem has long been investigated [4, 5]. However, traditional methods are known to be sensitive to variations such as facial expression, angle, environmental illumination, and object pose variation, which usually occur in an unconstrained scene [6]. Recently, the prominent convolutional neural network (CNN) approaches are commonly adopted because they are more robust to these changes. Classic CNN architectures include: Le-Net, Visual Geometry Group (VGG) [7], GoogleNet [8], ResNet [9], and more. The upgraded approaches have recently been proposed to deal with the advanced problems such as the limitation of ID performance degradation due to ageing or growth effects such as hair growth, sun effects, skin ageing, diseases, weight changes and many others [10, 11]. However, there are disadvantages using CNN for face recognition, that is the amount of calculation is extremely large, the cost of computing equipment is high and the versatile face forgery. Moreover, most of the traditional approaches do not possess the capability of anti-spoofing. Ref. [12] proposes a face anti-spoofing R-CNN (FARCNN), which can detect original face image as a three-way classification for real face, fake face and background. However, the bounding box is directly output through FARCNN, thus it might be difficult to locate small face regions. Based on RetinaNet and feature pyramid network (RPN), [13] creates a neural network architecture, which exhibits excellent performance on the small face detection. However, inference time is relatively slow due to the heavy-weight backbone network. The increasing use of face recognition has brought a critical issue of anti-spoofing. Common face recognition deceptive attacks are photo printed face photos, video such as recorded face video and 3D mask such as the 3D printed human head model, etc. Popular anti-counterfeiting methods include HSV + LBP features, YCbCr + LPQ features, HOOF + LBP features [14], and HOG features [15]+SVM classification. The specific feature diagrams are shown as in Figures 1 and 2. FIGURE 1Open in figure viewerPowerPoint Histograms of oriented gradients feature FIGURE 2Open in figure viewerPowerPoint Local binary patterns feature So far, many researchers have made distinguished progress in the study of facial anti-spoofing: [16] proposed a texture analysis method and used Fourier transform for conversion. The research team of [17] proposed a micro-texture analysis method and analysed it using LBP. Texture analysis presents better results when the environment is nearly invariant. However, in complicated environments, quality robustness of texture information is relatively poor. The main cause of performance instability is due to high sensitivity of the RGB cameras to light. In [18], the authors merged key facial features and constructed three-dimensional depth information to judge whether it is alive from facial images taken from different angles. While it is effective, it is quite time consuming and might be difficult to apply in practice due to its complexity. In [19], an NIR-RGB camera was adopted and supporting vector machine (SVM) was used as the classifier. It pointed out that the electronic visual displays and photos provided by NIRs camera are uneasy to image. After the rise of deep learning, researchers found that CNN can extract features efficiently than those of traditional methods, however, most methods are based on the pre-trained neural networks. It is possible to extract face features, however, unwanted features might be extracted as well. In addition, the pre-trained classic network usually has a deeper depth. In the face anti-spoofing application, richness of the training set is usually insufficient. Thus, it is practically unnecessary for such a deep network. What is needed here is a responsive and computational effective network, we build a network by proposing appropriate loss functions to yield output multiple indicators. We built two models based on Mobilenetv2 [20] with specific loss functions developed for the current purpose. For the face authentication and anti-spoofing models, additive angular margin loss [21], fine-tuned centre loss [22], and CrossEntropy [23] are referred. The angular margin loss [21] and the fine-tuned centre loss [22] are combined to a novel loss function. The combination of these models and the specific loss function significantly improves the accuracy of the model. The two modules provide multiple indicators to determine object identity and prevent from fraud, thereby improving robustness of identification performance of the entire architecture. For applications, the IoT transmission technology based AxiLink developed by our team is established to realise the complete system. The system is developed to reduce data transmission amount, minimise construction cost, and reduce data storage room accompanied with the user-friendly installation process. The system enables devices to intelligently connect with others without a pre-setting process, and data can be transmitted to the server in near real-time. The object data recorded in the database can also be used for indoor positioning of the user. 2 PHYSIOLOGICAL AUTHENTICATION DESIGN In the experimental IoT system considered in this work, we adopt Axlink, developed by our research team, as the transmission tool. The face verification and anti-spoofing models are based on the CNN. They have different depths and are trained with different loss functions. To reduce the number of parameters and calculations, inverted residual blocks are widely adopted in the model. The face detection uses Dlib and multi-task cascaded convolutional network (MTCNN) [24]. 2.1 Establishment of AxiLink Axilink is a measure of implanting an nRF24L01 wireless module in portable devices and using hardware to automatically perform data transmission without the need of extra paring processes. It possesses the advantage of a self-organising network structure and object positioning. Therefore, users need not set up or pair the connected devices in advance. The whole scheme includes Identity Module, Device, Cell, Node, and Reviver (shown in Figure 3). Functional descriptions are explained below. Identity Module: It is used to link the user's identity. There are two-stage user identity authentication (explained in Subsection 2.6). Device: Axlink communication module is implanted in specific devices such as those shown in Figure 4. When users with Identity Module activate the device, the measured data are transmitted to the Cell via the communication module automatically without the process of paring. Cell: It automatically detects multiple identity modules and devices as shown in Figure 5 and transmit various information of the user and devices to the Node. There is a priority queue arranged inside the system that prevents data loss and error during transmission. Node: LoRa is adopted as the core chip which receives data sent from Cell. The data includes user ID, device ID and Cell ID. In the open field, the transmission distance might be up to 3∼10 km. Lora's transmission is limited inside buildings or the areas suffered from strong electromagnetic interference. To resolve the issue, the AxiLink possesses a mode called concatenation. When the Cell cannot transmit data to the last Node, it will transmit data to the nearest Node. The Node will then transfer data to another Cell as a relay until data can be transmitted to the last Node. Figure 6 illustrates concatenation structure of the AxiLink. FIGURE 3Open in figure viewerPowerPoint The proposed AxiLink transmission scheme FIGURE 4Open in figure viewerPowerPoint Axilink communication module implanted in various portable devices FIGURE 5Open in figure viewerPowerPoint The cell detecting multiple identity modules and devices Receiver: It not only receives data from Node, but also from the Identity Module and the device itself, and it transmits all the information received at the Gateway via USB ports as displayed in Figure 7. FIGURE 6Open in figure viewerPowerPoint Concatenation structure of the AxiLink FIGURE 7Open in figure viewerPowerPoint The communication pathways 2.2 Gateway The gateway collects data generated by various sources via appropriate links. In this experiment, it also serves as the core computing unit, which is responsible for processing and parsing data received. The process is illustrated in Figure 8. FIGURE 8Open in figure viewerPowerPoint Operational flow of gateway data analysis In the experiment, data received by the receiver possesses device information about 80 digital bytes in total, including User ID, Cell ID, Node ID and two characteristic features. Data protocol is shown in Figure 9. FIGURE 9Open in figure viewerPowerPoint Data transmission protocol 2.3 Design of the face authentication system In addition to cost saving, this also prevents malicious deception. Figure 10 illustrates the operational chart of face authentication. FIGURE 10Open in figure viewerPowerPoint Operational flow chart of the face authentication system The basis for whether any objects nearby areprocessed by the AxiLink. The identity device continues to drop signals after detecting the Cell nearby. After entering the face authentication process, the image is processed by MTCNN to extract the object's face features. The system then compares them with the features received from the AxiLink. The angle of discrimination and Euclidean distance corresponding to various feature vectors are calculated and used as the evaluation criteria. 2.4 Design of the anti-spoofing system As illustrated in Figure 11, we get the image processed via an MTCNN from the main programme. In [19], it was mentioned that near-infrared light might interfere images of electronic visual displays and photos. Here, we use RGB and NIR images for the purpose of processing anti-spoofing. FIGURE 11Open in figure viewerPowerPoint Operational flow of the anti-spoofing process In the experiments, RGB and NIR photos pass through the anti-spoofing module, regardless of whether there are other objects nearby. The ID system uses cosine similarity and Euclidean distance to judge legality of the test objects. The threshold is set to be 0.9-0.95 after testing 10,000 pairs of positive and negative samples. For the illegal objects, a classifier is further used. 2.5 IoT system integration There are six steps to complete data transmission in our system. The demonstrative virtual field including ID device, gateway, cell, node, and server is illustrated in Figure 12. FIGURE 12Open in figure viewerPowerPoint The demonstrative virtual field Figure 13 describes the complete operational flow. The whole system consists of four parts: main programme, AxiLink data reception, face authentication, and anti-spoofing process. The main process is constantly running by using MTCNN. The NIR camera serves as the first layer against fraud. AxiLink data reception is a parallel running programme, which is used to continuously receive AxiLink signals and store them to the gateway's temporary memory after analysis. Face authentication and anti-spoofing are used for authentication and the second layer of anti-spoofing. FIGURE 13Open in figure viewerPowerPoint The complete operational flow The overall system operates on the basis of face feature capture and data stored in the personal identity device is illustrated in Figure 14. For face recognition shown in Figure 15, when the system detects a human face, it automatically communicates with the personal identity device through the IoT and reads out user data from the device's storage space including user ID, face features and network data. Advantages of our proposed approach compared with other IoT systems with facial ID access are listed in Table 1. FIGURE 14Open in figure viewerPowerPoint Facial feature training process and data storage FIGURE 15Open in figure viewerPowerPoint Face authentication process with internet of things TABLE 1. Comparison of the IoT systems with facial ID access Architecture ID type Characteristic data depository Recognition time Access to the IoT system Our approach Module ID, face feature, anti-spoofing Portable devices 1∼2 s Multiple IDs access at any time [27] Face feature Terminal or serve NA Image based authentication [28] [29] 3 DESIGN ALGORISM OF FACE AUTHENTICATION AND ANTI-SPOOFING In the experiment, the embedded system runs three neural networks simultaneously. Figure 16 shows details of the inverted residual block. To minimise the parameter numbers, we refer to the inverted residual block of Mobilenetv2. On this basis, two deep neural networks are reconstructed. FIGURE 16Open in figure viewerPowerPoint Process of the inverted residual block 3.1 Construction of face authentication module To construct the face authentication model, inverted residual blocks are used extensively. At the first stage of face training, a total of 5 × 5 convolution operations are adopted to extract rough features. It has a larger field of vision than that of 3 × 3 case, and the number of parameters is much lower than that of 7 × 7 case. Next, they are added to the inverted residual blocks to extract more detailed features. Critical parameters of the inverted residual blocks set for use in the experiment are listed in Table 2. TABLE 2. Parameters for experiments of the inverted residual blocks Parameter number Input Expansion Output Stride Inverted Residual-1 3 16 16 2 Inverted Residual-2 3 72 24 2 Inverted Residual-3 3 88 24 1 Inverted Residual-4 5 96 40 2 Inverted Residual-5 5 240 40 1 Inverted Residual-6 5 240 40 1 Inverted Residual-7 5 120 48 1 Inverted Residual-8 5 144 48 1 Inverted Residual-9 5 288 96 2 Inverted Residual-10 5 576 96 1 Inverted Residual-11 5 576 96 1 After conducting the inverted residual blocks, we obtain a set of feature maps of b a t c h s i z e × 576 × 7 × 7 . At this time, we use Pointwise Convolution to amplify the dimension of this feature and use Adaptive Average Pooling to fuse this feature. Finally, it is directed to a classifier. The classifier includes AdaptiveAvgPooling, Linear, BatchNormalize, and h_swish. Assuming the categories of classes to be c, the classifier is O d ∈ ℜ c × 1 . Figure 17 illustrates the operational architecture of face authentication. During training, we have two sets of output features, the first one is extracted from AdaptiveAvgPooling, and the second one is the classifier output. They are matched with the individual loss functions to achieve multi-index outputs. In practice, one may use the output of Type 1 instead of the Type 2 output, since the former needs less parameters and simpler calculation. FIGURE 17Open in figure viewerPowerPoint Simplified architecture of face authentication 3.2 Construction of anti-spoofing module For the function of face anti-spoofing, the number of chosen parameters of the inverted residual blocks is smaller than that of the face authentication model, and the depth is relatively shallow. A total of 5 × 5 convolutions are utilised in our experiment to extract rough features, which are then added to the inverted residual blocks to extract more subtle features. Parameters of inverted residual blocks are listed in Table 3. TABLE 3. Parameters of the inverted residual blocks Parameter of the inverted residual blocks Input Expansion Output Stride Inverted Residual-1 3 16 16 2 Inverted Residual-2 3 72 24 2 Inverted Residual-3 3 88 24 2 Inverted Residual-4 5 96 40 2 After passing the inverted residual blocks, we have a set of feature maps of b a t c h s i z e × 96 × 7 × 7 . Next, we use Adaptive Max polling to fuse this feature as the final output. This is the first version. Different loss functions are considered in this research, there are two variants. The second one uses a classifier to generate the classification results. The classifier includes AdaptiveAvgPooling, Linear, BatchNormalize, h_swish, and dropout. The classifier is of O a ∈ ℜ 3 × 1 . Figure 18 describes a simplified architecture of the anti-spoofing module. In these two versions, Type 1 is superior in accuracy with lower computing cost, since it uses cosine similarity and Euclidean distance to make judgement without using classifier, while Type 2 is used for the case of illegal object recognition. FIGURE 18Open in figure viewerPowerPoint Architecture of the anti-spoofing module 3.3 Loss functions When training the neural networks, we combine two loss functions into one. This loss function is referred to as Additional Angle Margin Loss [21] and Centre Loss [22]. It not only calculates Euclidean distance of a feature vector, but also considers the angle between two feature vectors. To explain further, assume that the feature vector extracted from the middle layer is x d . In the following theoretical framework, F m is the feature matrix with dimension D m and batch size n , y is the centre vector of each category and C is the centre matrix of all categories with dimension D m and the number of categories being m : F m = [ x d 1 M x d n ] = [ x d 1,1 L x d 1 , D m M O M x d n , 1 L x d n , D m ] n × D m , C = [ y 1 M y m ] = [ y 1,1 L y 1 , D m M O M y m , 1 L y m , D m ] m × D m (1) Next, we subtract the corresponding C y i from F m and take 2-norm to calculate the loss value. From which one gets the distance loss named L d defined as. L d = ∑ j = 1 n ‖ x d j − C y j ‖ 2 (2)where C y i is the centre of class y i , n is the batch size and x d j is the feature vector corresponding to C y j . The gradient of L d with respect to x d is given by ∇ x d L d = ∑ i = 1 n x d i − C y i ‖ x d i − C y i ‖ 2 (3) After calculating L d , we get the second output from the classifier of our module, which is used to calculate the Additional Angle Margin Loss [14]. The CrossEntropy proposed in [16] is adopted as the evaluation criterion as follows L C E ( x , c l a s s ) = − log ( e ( x [ c l a s s ] ) ∑ j e ( x [ j ] ) ) (4)where x [ c l a s s ] and x [ j ] denote, respectively, the class and the jth feature vector. Assume that the output feature vector from classifier is x a and F o is the feature matrix with dimension D o and batch size n . We set up a weighting matrix W according to the number of categories and the dimension of the input vector: F o = [ x a 1 M x a n ] = [ x a 1,1 L x a 1 , D o M O M x a n , 1 L x a n , D o ] n × D o , W = [ w 1 M w m ] = [ w 1,1 L w 1 , D o M O M w m , 1 L w m , D o ] m × D o (5) Next, we normalise W and F o separately and proceed to calculate cos ( θ j ) corresponding to W where θ j is the angle between the respective features. To maximise the angle between different categories, r is directly added to the corresponding θ j . After calculating cos ( θ j + r ) , we substitute the results to L C E defined by Equation (4) to calculate the final loss value. In the following expression, the loss function L a denotes the batch number and s is a hyperparameter used to determine the sphere radius, which is set to be 32 in this experiment, r is set to be 0.5, n is batch size and m is the number of categories: L a = − 1 n ∑ j = 1 n log e s ( cos ( θ j + r ) ) e s ( cos ( θ j + r ) ) + ∑ i = 1 , i ≠ j m e s ( cos θ i ) (6) The total loss function is given by the weighted combination of L a and L d as L a d = L a + L d = − 1 − β n ∑ j = 1 n log e s ( cos ( θ j + r ) ) e s ( cos ( θ j + r ) ) + ∑ i = 1 , i ≠ j m e s ( cos θ i ) + β ∑ j = 1 n ‖ x d j − C y j ‖ 2 (7)where β is a hyperparameter used to weight the two loss functions. It is suggested that the value of β is less than 0.2 to avoid L d affecting the relative importance of L a . 4 RESULTS AND VERIFICATION For experimental verification, a functional IoT system with indoor positioning, anti-spoofing function, and face authentication is completely implemented in this research task. In the training phase, VGG data set and the publicly available NUAA photo-imposter database [25] were adopted to train face authentication and anti-spoofing models. In the real-world testing phase, the photos of different angles taken from 10 users were collected. Figures 19 and 20 present the training results of the face authentication model. Figure 21 presents a demonstrative sample of face feature extraction. We use 100,000 images taken from 300 people for training the loss function L a d . After performing 400 iterations, the ID accuracy is up to 90.2%. FIGURE 19Open in figure viewerPowerPoint Training and validated accuracy of face authentication model FIGURE 20Open in figure viewerPowerPoint Training loss of face authentication model FIGURE 21Open in figure viewerPowerPoint Face feature extraction Figures 22 and 23 show the classification of the face authentication model down to two dimensions after conducting t-SNE [26] to generate 10 categories. The total loss L a d defined by Equation (7) is used to enhance discrimination among various categories. FIGURE 22Open in figure viewerPowerPoint Training result of classification FIGURE 23Open in figure viewerPowerPoint Validation of classification Figures 24 and 25 show the training results of the anti-spoofing module. We train this model with the NUAA data set and the self-built data set, which includes a total of 18,000 images in three types: real human, photo intruder and electronic screen intruder. The loss L a d is not adopted here, since the convergences of angle margin and Euclidean distance are not distinguishable when the number of iterations is small. FIGURE 24Open in figure viewerPowerPoint Training and validation accuracy FIGURE 25Open in figure viewerPowerPoint Training loss Figure 26 shows the feature extraction samples of NIR pictures of the anti-spoofing module in different stages. These photos were all taken from a binocular camera to catch NIR and RGB images simultaneously. The shooting environment is at the laboratory illuminated by fluorescent lamps and each image contained only one colourful positive face. In which, 'phone' refers to the screen of the electronic device, 'picture' is a general photo of a positive face, and 'true' denotes a real human. It can be seen from this figure that this model is sensitive to the edges of human faces. Because infrared rays may interfere image features taken from mobile phones and photos, the model can effectively distinguish the difference between positive and negative samples to improve ID accuracy. FIGURE 26Open in figure viewerPowerPoint Feature extraction samples Figures 27 and 28 show, respectively, the training results and validation of Type1 anti-spoofing module down to three dimensions after conducting t-SNE that visualises high-dimensional data by giving each datapoint a location in a two or three-dimensional map. This classifier is used for non-object identification. If it is a legal one, the system directly uses cosine similarity as the basis for judgement. FIGURE 27Open in figure viewerPowerPoint Anti-spoofing module training result FIGURE 28Open in figure viewerPowerPoint Result of validation Table 4 lists ID accuracy of face authentication. This experiment compares two models with different sizes and the values of three loss functions. A larger model achieves higher accuracy, however, there will be more parameters to determine. A small model with L a d requires a lower-cost model on the devices. TABLE 4. Face authentication accuracy Face authentication accuracy Parameters