Augmented SRC for face recognition under quality distortions

IET BiometricsVolume 8, Issue 6 p. 431-442 Research ArticleFree Access Augmented SRC for face recognition under quality distortions Jinane Mounsef, Corresponding Author Jinane Mounsef jmounsef@asu.edu School of Electrical, Computer & Energy Engineering, Arizona State University, Tempe, AZ, USASearch for more papers by this authorLina Karam, Lina Karam School of Electrical, Computer & Energy Engineering, Arizona State University, Tempe, AZ, USASearch for more papers by this author Jinane Mounsef, Corresponding Author Jinane Mounsef jmounsef@asu.edu School of Electrical, Computer & Energy Engineering, Arizona State University, Tempe, AZ, USASearch for more papers by this authorLina Karam, Lina Karam School of Electrical, Computer & Energy Engineering, Arizona State University, Tempe, AZ, USASearch for more papers by this author First published: 03 October 2019 https://doi.org/10.1049/iet-bmt.2018.5242Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In the last two decades, numerous methods have been developed to offer a formulation to the face recognition problem under scene-dependent conditions. However, these methods have not considered image quality degradations resulting from capture, processing, and transmission such as blur and occlusion due to packet loss, under the same scene variations. Although deep neural networks are achieving state-of-the-art results on face recognition, the existing networks are susceptible to quality distortions. In this work, the authors propose an augmented sparse representation classifier (SRC) framework to improve the performance of the conventional SRC in the presence of Gaussian blur, camera shake blur, and block occlusions, while preserving its robustness to scene-dependent variations. In their evaluation of the SRC framework, they present a feature sparsity concentration and classification index that is capable of assessing the quality of features in terms of recognition accuracy as well as class-based sparsity concentration. For this purpose, they consider three main types of features including image raw pixels, histogram of oriented gradients and deep learning visual geometry group (VGG) Face. The obtained performance results show that the proposed method outperforms state-of-the-art sparse-based and blur-invariant methods. 1 Introduction Face recognition is still receiving a lot of attention because of its high relevance to biometrics, information security, law enforcement needs, and surveillance systems, to name a few. One important challenge that limits the effectiveness of face recognition technology is image quality [1]. In real-world environments, the acquired face image quality varies due to lens resolution, focus, distance, noise, illumination, storage, and transmission, to name a few. This may greatly affect the performance of face recognition algorithms. While face recognition has already achieved a very good performance over large-scale galleries that include traditional scene-dependent distortions such as pose variations, extreme ambient illumination [2–7], and partial occlusions due to obstacles and disguise [7–12], there still exist many more challenges related to variations in image quality such as blur and block occlusions due to packet loss. These types of distortions, which result from capture, processing, and transmission are commonly present in images and videos acquired by surveillance cameras and increasingly by mobile handheld devices or transmitted over Internet protocol and wireless networks. In such scenarios, it is very likely that the captured/transmitted image contains a distorted or a blurred face. Moreover, while transmitting compressed face images over lossy packet networks, it is possible that one or more of the packets will not reach the destination. This type of data packet loss during transmission results in partial occluding blocks, thereby hiding major facial features. Other common reasons for occlusion are scene-dependent obstacles and disguise accessories that cover one or more random regions in the face. Table 1 lists some of the main factors for image quality degradation during capture and transmission. Table 1. Main factors in capture and transmission for image quality degradation Blur Block occlusion out of focus transmission errors/loss motion (camera/subject) low device performance shallow depth of field software issues turbulent medium (fog/rain) faulty hardware In face recognition, the sparse representation classifier (SRC) method [8] has proved that it could overcome challenging scene-dependent variations including illumination changes, random pixel corruption, and small-size occlusion/disguise. The SRC method assumes that a test image can be represented by a linear combination of sample images from the same subject, which form the basis elements of an overcomplete dictionary, via -minimisation. In the case of low level of occlusion, the error corrupts only a fraction of the image pixels and is, therefore, sparse. Such a case can be handled within the SRC framework, where the component of the test image arising due to occlusion is naturally separated from the component arising from the identity of the test subject. Wright et al. [8] show that the choice of features to represent the samples is not important in the SRC framework when the dimension of feature space is sufficiently large and, therefore, they simply use raw image pixels as features. Wright et al. [8] further indicate that, when dealing with occlusion, no features are more robust than the raw image pixels. Although the SRC shows promising results for clean images and images affected by sparse noise (small-size occlusion/disguise), we show in this work that under non-sparse image quality distortions that commonly occur under real-world conditions (e.g. blur and relatively large-size occlusions), the choice of features becomes important. Numerous sparse representation-based classification methods followed later with the sole aim of improving the robustness of the SRC method to appearance variations such as the extended SRC (ESRC) [10], robust sparse coding (RSC) [9], and structurally incoherent low-rank matrix decomposition [12]. Although the above methods show promising results with images affected by sparse noise, they have not considered acquisition and transmission distortions such as blur and non-sparse noise (relatively large-size occlusions), as part of their design. Furthermore, these methods require that the training images be well-aligned for reconstruction purposes. However, the alignment methods involve a priori knowledge of facial landmarks, which become inaccurate when the face quality is degraded with blur and non-sparse noise. Blur-invariant methods have been proposed to reduce the sensitivity of images to blur distortion. Several authors proposed different blur invariants that were mainly based on image moments without the need of image restoration. Flusser et al. [13] and Flusser and Suk [14] proposed a system of blur invariants, which are recursive functions of geometric moments of the image, and proved their invariance under a convolution with arbitrary centrosymmetric kernel. Flusser and Zitová [15] developed further the concept of blur invariants by introducing combined invariants to convolution and rotation that were successfully used in satellite image registration. Zhang et al. [16] and Suk and Flusser [17] proposed combined invariants to convolution and affine transform. These invariants were used in different non-face recognition applications such as aircraft silhouette recognition and sign language recognition. Other researchers developed blur invariants that are based on orthogonal moments instead of geometric moments. Legendre moments [18–21], Zernike moments [22–24], and Chebyshev moments [25] are some of the examples. Some other authors proposed blur invariants in the Fourier domain. Ojansivu and Heikilla [26, 27] and Tang et al. [28] used blur-invariant properties of the Fourier transform phase for image registration and matching. Later, Pedone et al. [29, 30] generalised the same idea. The popular method of local phase quantisation (LPQ) [31] also belongs to the same category. Although the Gaussian kernel is a special case of symmetric kernels, these blur-invariant methods do not work well with Gaussian blur (GB). Few attempts to derive invariants to GB have been reported. Xiao et al. [32] derived invariants to GB but did not use the Gaussian form explicitly. Instead, they used the circular symmetry property. Gopalan et al. [33] derived blur invariants but did not make any assumption on the parametric shape of the kernel. Zhang et al. [34, 35] derived a blur-invariant distance that is specifically designed for GB. Although the blur invariants are not explicitly defined, the invariant distance measure was used for object classification. Flusser et al. [36] developed new invariants to GB for face recognition based on the same concept of Zhang's distance [34]. These invariants are based on projection operators in the Fourier domain and on image moments in the image domain. The authors showed that the proposed invariants outperformed the blur-invariant Zhang's distance [34, 35] and the LPQ blur-invariant features [31]. Nevertheless, the method in [36] is not a good design for blurred face recognition as the authors have assumed the use of the same images for training and testing with the latter being blurred. On the other hand, Vageeswaran et al. [37] proposed a blur-robust algorithm robust direct recognition of blurred faces (rDRBF) for unconstrained blurred face images, which showed a significant improvement over the LPQ algorithm of [31] for large levels of GB. The method in [37] solves a convex -norm problem to create artificially blurred versions of the gallery images, where the blurred probe image is matched to them. A major constraint of the latter method is that it applies blur estimation for every gallery image, which is computationally complex. More recently, deep learning methods, also known as deep neural networks (DNNs), have achieved high accuracies in face recognition such as DeepFace [38], DeepID [39], VGG-Face [40], and FaceNet [41]. Despite their excellent performance, 'deep' methods are susceptible to visual distortions. Dodge and Karam [42] provide an evaluation of state-of-the-art DNN models for image classification under quality distortions, where they show that the performance of these DNNs is significantly reduced in the presence of blur distortion. Grm et al. [43] show equally in their work that high levels of blur have a detrimental effect on DNNs such as AlexNet [44], VGG-Face [40], GoogleNet [45], and SqueezeNet [46]. Fernandez studies in [47] the performance of state-of-the-art DNNs on objects that have been occluded by different block sizes. The author shows that the performance of these networks decreases sharply in the presence of an increasing occlusion size. The main contribution of our work is in proposing a solution to face recognition under quality distortions such as blur and contiguous block occlusions. Since the SRC performs well under traditional scene-dependent variations, our method consists of improving the SRC framework in the presence of such visual quality distortions. No previous work has evaluated the impact of blur on the SRC classifier. Therefore, as part of this work, we explore the effect of blur and occlusion on the SRC and propose an improved augmented SRC-based framework (ASRC) that is more robust to blur and occlusion for any selected feature by accounting for these distortions as part of the dictionary construction. We conduct rigorous experimentation on three face recognition benchmarks, namely the Olivetti research laboratory (ORL) [48], extended Yale B [49], and labelled faces in the wild (LFW) [50] after adding GB, realistic camera shake blur (CSB), and block occlusions separately to the images at different distortion levels. A preliminary version of the proposed ASRC framework appeared in [51]. In this paper, we have improved on our earlier work by expanding the ASRC structure to account for block occlusions in addition to blur. We also propose a feature selection method to assess the quality of features in order to provide the ASRC with a sparse representation while enhancing the fidelity of the reconstruction. In addition, the proposed ASRC is evaluated for robustness against unseen realistic camera shake distortions [52]. The obtained results show that the proposed ASRC performs better than state-of-the-art sparse representation-based classification methods [9, 10, 12] including the standard SRC [8] and blur-invariant methods [35–37]. The rest of this paper is organised as follows. Section 2 briefly introduces the SRC classifier and its limitations in the presence of visual distortions. Section 3 presents the proposed ASRC method and feature quality assessment. The experimental setup and results are presented in Section 4. Finally, Section 5 concludes with some final remarks. 2 Related work As our method is based on the concept of SRC, we briefly introduce the latter method in this section to facilitate the description of our proposed method. Then, we describe the main limitations of the SRC framework in the presence of visual distortions such as blur and occlusion due to acquisition and transmission errors/loss. 2.1 Sparse representation classifier Consider a set of training samples corresponding to M object classes with samples { in theith object class, 1 ≤ i ≤ M. Each training sample is normalised to have unit -norm. The dictionary D is formed by inserting the training samples of all M object classes as entries, resulting in (1) Given a sufficient number of training samples in the ith object class, a test sample vector y from the same class can be represented as a sparse linear combination of the training samples in D as follows: (2) where is the sparse coefficient vector and is the number of training samples in D. The sparse vector is computed by solving the following constrained -norm minimisation problem: (3) where and denote the -norm and -norm, respectively, and is a sparsity coefficient. Once (2) is solved, the representation residual for the ith object class is computed as follows: (4) where is a vector whose non-zero entries are the entries in that are associated with the ith class. The test sample y will be assigned to the class corresponding to the minimum representation residual (5) 2.2 SRC limitations The SRC framework described in Section 2.1 shows that a test sample y can be sufficiently represented using only the training samples from the same class as shown in (2) if the representation is naturally sparse. The sparser the recovered is, the easier will it be to accurately determine the identity of the test sample y. Wright et al. [8] indicate that the choice of features is not critical enough to affect the sparsity of the representation as long as the feature dimension surpasses a threshold that is predicted by the theory of compressive sensing. Thus, the SRC framework in [8] was implemented using raw image pixels as features. However, under unconstrained variations and large distortions, the raw pixels are not informative enough to recover the sparse representation. In this work, it is shown that features that are more resilient to scene-dependent variations [38–40, 44, 53–55] help in keeping the framework classes distinct, and can result in a significant performance gain under quality distortions. Wright et al. [8] considered in their work the effect of occlusion by modifying the model described in (2) to explicitly account for additive noise w (6) where . A fraction of w's entries are non-zero corresponding to the occluded pixels in y. We can rewrite (6) as [8] (7) where and . has at most non-zero entries, where is the number of non-zero entries of for the ith class and is the number of non-zero entries of w. As stated in [8], we might recover as the sparsest solution for (7) by solving the -norm minimisation problem (3) as long as the fraction of occlusion is <33% of the image size d (i.e. ). This implies that if the occlusion covers a large portion of the image (more than 33% coverage), the SRC framework cannot provide the sparsest solution for (7), and thus, its performance decreases. 3 Augmented SRC We first describe the different distortion types that we consider in this work and derive their effect on sparse representations. We then propose a modification to SRC, namely ASRC, to further enhance the classification performance under these types of distortions. 3.1 Effect of distortions on sparse representations Let be a raw-pixel test sample that can be sparsely represented in terms of the training samples in as given in (2). If is the ith atom entry vector of D and is the ith entry element of , then (2) can then be rewritten as (8) Let be an operation representing the distortion applied to y and whose output is , the distorted version of y, given by (9) If the operation is linear, can be expressed as (10) where . Let be the dictionary whose atom vectors are . From (8) and (10), it follows that if y is sparsely represented in terms of the elements in D, the distorted version will be sparsely represented in terms of the elements in . Furthermore, y and will have the same sparse coefficient vector . To illustrate the distortions that can be described by a linear operation, we consider the cases of blur and block occlusions. 3.1.1 Effect of blur The blurring process can be represented using a point spread function (PSF). A special popular case is a Gaussian PSF, which can be expressed as (11) where is the standard deviation and () denotes the pixel location at which the blur PSF is defined. The GB PSF is commonly used to account for defocus blur in addition to blur due to atmospheric turbulence. For convenience, we will refer to the lexicographically ordered as h, where the blur level is controlled by the variance , and to * as a two-dimensional (2D) convolution followed by a lexicographic ordering. The blur distortion can thus be represented using the linear operation . Consequently, using (10), the blurred image can be expressed as (12) The SRC framework described in Section 2.1 shows that a test sample y can be sufficiently represented using only the training samples from the same class as in (2) if the representation is naturally sparse. Under the condition that the considered blur level preserves sufficient separation between the M classes to keep these distinct (12) indicates that if the clean image y is sparsely represented in terms of the clean training samples in the dictionary D, then blurring the atoms in D at the same blur level as the blurred test sample will maintain the sparsity of the representation. 3.1.2 Effect of block occlusions In many practical face recognition scenarios, the test image y could be partially occluded by one or several blocks due to the loss of data packets during the transmission of compressed images over lossy packet networks. Face recognition is particularly affected if the blocks are obstructing major facial features. In [8], the SRC framework is designed to uniformly handle sparse occlusions by representing such occlusions with an additive operation as in (6). In this work, to investigate the resilience of the SRC framework to occlusions including those covering more than 33% of the image size, we represent these as a linear pointwise vector multiplication operation. Let be a raw-pixel test sample that can be sparsely represented in terms of the training samples in as given in (8). Let be a corruption function such that and where denotes a pixel location. A single-block occlusion of size k, such that it is limited between and in the u direction and between and in the v-direction, is modelled by considering an occlusion kernel given by: for , , , and elsewhere. For example, a black occlusion occurs if . can also be constructed to represent multiple-block occlusions as the union of two or more single-block occlusions. can also be generalised to represent random pixel corruptions by allowing the corruption to occur at random non-contiguous values. For convenience, we will refer to the lexicographically ordered as b and to × as a 2D pointwise multiplication followed by a lexicographic ordering. As such, the block occlusion distortion can be represented by a linear operation . From (10), it follows that the occluded image can be expressed as: (13) Under the assumption that the considered occlusion preserves sufficient separation between the M classes to keep these distinct (13) indicates that if the clean image y is sparsely represented in terms of the clean training samples in the dictionary D, then adding occlusion to the atoms in D at the same block position and with the same size k as the occluded test sample will maintain the sparsity of the representation. It is worth mentioning that representing the atoms using features that are more resilient to blur or occlusion, as compared with raw pixels, will help in keeping the classes distinct at higher blur/occlusion levels. 3.2 Proposed method: ASRC under blur and occlusion In the following, we present in details our ASRC method, which aims at improving the performance of the original SRC [8] under blur and block occlusions. The discussion here applies as well to other types of distortions that can be described or approximated using linear operations. From (12) and (13), it can be seen that the distorted raw image test sample can be sparsely represented in terms of the distorted atoms , if the clean test sample y is sparsely represented in terms of the atoms and if the classes are kept distinct after applying the operation to the dictionary atoms. This brings the importance of representing the atoms with features that are more resilient to blur/occlusion distortion as compared with raw pixels, as discussed in more details later in this work. For the blur distortion, in order to accommodate various blur levels, we propose to augment the dictionary D with training samples that have been blurred with GB kernels of increasing variance and size (including the absence of blur). For the occlusion distortion, and in order to accommodate various occlusion positions, we propose to augment the dictionary D with training samples, which have been occluded at different occlusion positions with a specified occlusion size (including the absence of occlusion). In either case, we opt to assign the group of atoms that correspond to a specific distortion level (blur) or distortion position (occlusion), for a particular identity i, , to a separate object class j, , in the resulting dictionary that we will be denoted by . For simplicity, in the remainder of this section, we denote both the blur level and the occlusion position by distortion level , for short. Let us assume that the original dictionary D consisting of the clean training samples is given by (14) where , is the set of training samples for the ith object class. Let be a linear operation representing the distortion at level . Also let . The proposed dictionary is given by (15) where corresponds to the number of represented distortion levels and , , is given by (16) In (16), \ and correspond to the integer division and modulo operations, respectively, and corresponds to the absence of distortion. We apply the SRC using the proposed dictionary to classify test samples based on the class-wise minimum reconstruction error as in (5). Nevertheless, as the subject belongs to either one of the possible levels (including the absence) of distortion, the subject identity i is obtained as follows: (17) where is computed using (5), except that the residual is now computed with respect to the new constructed dictionary and M is replaced by . Algorithm 1 (Fig. 1) summarises the procedure of the ASRC method for the blur and occlusion distortions. Figure 1Open in figure viewerPowerPoint Algorithm 1: the ASRC algorithm for blur and occlusion 3.3 Effect of feature space The choice of feature space is important to properly separate object classes in a more discriminative space. Feature extraction methods are mainly subdivided into three main groups. The subspace learning group, which includes raw image values, Eigenfaces [56], Fisherfaces [57], and Laplacianfaces [58], is a conventional class of linear features, where training samples from a single class are modelled to lie on a linear subspace. This type of feature has been widely used with classifiers such as nearest neighbour [59], nearest space [60], and linear support vector machines [61]. The subspace learning methods have been replaced later by the hand-crafted non-linear approaches, which use the local orientation information. These features have proven to work well on face recognition applications in constrained environments. They include features such as local binary patterns [54], LPQ [31], histogram of oriented gradients (HoG) [53], and Fisher vectors [55]. More recently, DNN-based models have achieved high performance in terms of accuracy in the face recognition domain. They consist of layers of convolutional filters, where the weights of the filters can be learnt using a gradient descent-based optimisation procedure. Deep features extracted from DNNs such as DeepFace [38], DeepID [39], VGG-Face [40], and FaceNet [41] are discriminative enough to recognise clean face images that have been captured in unconstrained environments. In [8], Wright et al. designed and tested the SRC framework on subspace learning features for ease of presentation. Wright et al. [8] argue that, if the sparsity is well-harnessed, the choice of the features is not important, as long as the dimension of the feature exceeds a specific bound [8]. In our work, we provide a performance evaluation of the SRC and ASRC frameworks by selecting different feature types. For this purpose, we consider hand-crafted features (HoG), as well as deep learning features (VGG-Face) for two main reasons. The hand-crafted features are effective at detecting local information, namely contour and texture, specifically in constrained environments. The deep learning features are robust in unconstrained environments, as the DNN layers have configurable parameters that can be learnt from the processed data across the layers. In Section 4.4, our experimental results verify that the choice of these two types of features affect the performances of the SRC and ASRC frameworks by improving their recognition accuracy rates compared with raw image features. A feature transformation maps data from the image space of dimension D to a high-dimensional feature space of dimension m, where . Consequently, SRC is performed by solving the following constrained -norm minimisation problem: (18) where and D is defined as in (14). The feature-domain ASRC framework is modelled using (18), where D is replaced with , such that , and where is defined as in (15). 3.4 Feature selection and proposed feature sparsity concentration and classification index To assess the quality of features in improving the sparsity of the representation, we propose a method of feature selection based on the class-based sparsity concentration as well as the classification accuracy. The validity of a sample depends on the sparsity of its coefficient vector , which can be measured using the sparsity concentration index (SCI) [8]. A valid test image has a sparse representation with non-zero entries concentrated mostly on one subject, whereas an invalid image has a sparse representation with coefficients spread among a wide range of multiple subjects. The SCI index of a coefficient vector is defined as a measure of how concentrated the coefficients are on a single class in the dataset and are computed as follows [8]: (19) where M is the total number of distinct classes. For a solution as found in (3), varies between 0 and 1, where higher values correspond to sparser representations. As in [8], we choose a threshold and accept a sparse representation as valid if . The most discriminative feature should not only ameliorate the sparsity of the representation but it must also improve the representation accuracy. The residual is used to measure the fidelity of the representation by measuring the similarity between the test sample and each individual class i as in (4). Given an application with a validation dataset, we aim to rank the features based on their ability to increase the sparsity of the proposed framework representation while improving its identification performance. For this purpose, we propose a feature sparsity concentration and classification index (FSCCI) for feature selection. We propose to capture the number of valid images that are correctly classified [true positive rate (TPR)] and the number of valid images that are incorrectly classified [false positive rate (FPR)] for each threshold . For a given validation dataset with N total