Single sample per person face recognition algorithm based on the robust prototype dictionary and robust variation dictionary construction

IET Image ProcessingVolume 16, Issue 3 p. 742-754 ORIGINAL RESEARCH PAPEROpen Access Single sample per person face recognition algorithm based on the robust prototype dictionary and robust variation dictionary construction Shan Xue, Shan Xue orcid.org/0000-0001-7071-4058 Faculty of Computer Science, Xi'an Aeronautical University, Xi'an, China School of Mechatronic Engineering, Xi'an Technological University, Xi'an, ChinaSearch for more papers by this authorHai-Peng Ren, Corresponding Author Hai-Peng Ren renhaipeng@xaut.edu.cn orcid.org/0000-0003-3834-5103 School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China Correspondence Hai-Peng Ren, School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China. Email: renhaipeng@xaut.edu.cnSearch for more papers by this author Shan Xue, Shan Xue orcid.org/0000-0001-7071-4058 Faculty of Computer Science, Xi'an Aeronautical University, Xi'an, China School of Mechatronic Engineering, Xi'an Technological University, Xi'an, ChinaSearch for more papers by this authorHai-Peng Ren, Corresponding Author Hai-Peng Ren renhaipeng@xaut.edu.cn orcid.org/0000-0003-3834-5103 School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China Correspondence Hai-Peng Ren, School of Mechatronic Engineering, Xi'an Technological University, Xi'an, China. Email: renhaipeng@xaut.edu.cnSearch for more papers by this author First published: 26 November 2021 https://doi.org/10.1049/ipr2.12381AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Single sample per person (SSPP) face recognition uses only a single face image of each subject in the gallery set to recognize the probe sample. Since single sample cannot provide intra-class variation information, the matching accuracy of the gallery faces with the faces captured in unconstrained video is usually low. Recently, in order to improve the accuracy, the sparse representation-based classification (SRC) technology has been extended to generic learning method, which uses prototype and variation dictionary (P+V) model for face recognition. Because the inter-class scatter between atoms in prototype dictionary is not big enough, and the intra-class scatter in constructed variation dictionary (such as posture and expression) is not rich enough, the robustness of P+V model for SSPP face recognition is poor. To solve this problem, a robust prototype dictionary and robust variation dictionary construction (RPRV) method is proposed. First, a set of atoms is obtained by dictionary learning method using gallery images and generic images. Second, some effective atoms are selected by the proposed function index method. Finally, the robust prototype dictionary (RP) and the robust variation dictionary (RV) are represented linearly using these effective atoms, respectively. The face recognition is performed according to the proposed RP+RV model. Experiment results using public datasets show that the proposed RPRV has strong robustness for the face captured under the unconstrained environment. Comparison results show that the proposed RPRV method outperform state-of-the-art SRC-based methods for SSPP face recognition. 1 INTRODUCTION Face recognition is an important research field in computer science and many application scenarios over the past decades [1-6]. In many practical face recognition systems, for example, law enforcement, entrance monitoring, intelligent video surveillance, criminal identification, the gallery set is typically designed using only one single sample per person (SSPP) [7] because of the limited storage and privacy policy. What makes the situation even worse is that the probe samples are usually captured with video surveillance cameras under unconstrained conditions [8, 9]. Therefore, SSPP face recognition in this unconstrained environment is very challenging task due to, first, the strong intra-class variations such as illumination [10-12], pose [12, 13], facial expression [14-16] and occlusion [17, 18], and second, the lack of gallery samples. Up to date, four main kinds of methods for SSPP face recognition are reported, including virtual sample generation [19-21], classifier [22-24], image partitioning [25-29] and generic learning [30-40]. The virtual sample generation method uses the singular value decomposition (SVD) [19], pose and illumination variability [20], the lower-upper decomposition [21] etc. to generate multiple virtual samples from a single gallery image of the subject, so that the conventional techniques using multiple samples per subject can be applied for SSPP face recognition. Although the recognition results of these algorithms can be improved to a certain extent compared with the original single sample, the extra information generated through virtual samples is quite limited. Therefore, the performance improvement is quite limited. The classifier method uses training, determination etc. to make the distance between the same class of frontal gallery sample and the probe sample close to the distance between the different class. Plichoskia et al. [22] proposed an SSPP method based on differential evolution algorithm, applied the differential evolution method to the face recognition framework for the first time, which has good performance under constrained environment. Pang et al. [23] proposed a iterative dynamic generic learning (IDGL) method to handle SSPP problem with a contaminated biometric enrolment database, which has good performance under unconstrained environment. Zhang et al. [24] proposed an SSPP method based on dissimilarity-based nearest neighbor classifier, which is effective on the AR dataset. However, the classifier training process of gallery class is time-consuming. In order to recognize in real time, this method is usually run offline after training. These recognition system cannot change the gallery class immediately under given circumstances. The image partitioning method divides the single gallery image into several overlapping/nonoverlapping image patches, and each patch is classified independently. Then, the classification results of the probe image patches are obtained, and the final classification result of the probe image is obtained by voting or other methods. Zhu et al. [25] extended the collaborative representation based classification (CRC) [26] to image partition based method, that is, PCRC, by integrating the CRC outputs of all partitioned patches. First, the recognition results of image patches are obtained by CRC method, then the recognition result of one scale is obtained by voting method, finally, under the multi-scale, the recognition result is obtained by the weighted combination. Furthermore, Yan et al. [27] and Zhang et al. [28] proposed the manifold learning methods, first, the feature is extracted for each image patch, then the graph structure of each image is obtained using its patch features, and finally the recognition result is obtained by matching the graph structure. Adjabi et al. [29] proposed an original method based on multi-block color-binarized statistical image features to handle SSPP problem, which method has faster running speed. These methods use the similaritiy of patches in the same position to identify a probe patch. However, when there is pose variations or misalignments in the probe sample, the patches of the same position of the probe and gallery sample may not match well. The generic learning method assumes that similar intra-class variations could be shared by different individuals. First, the intra-class variation information is obtained by an auxiliary generic set; this variation information is represented as the difference of gallery samples and probe samples [30, 31]. Second, the sparse coefficients between gallery samples and probe sample are obtained. Finally, the identity of the probe sample is obtained by the sparse coefficients. The generic learning method generally outperforms the other two kinds of methods. Sparse representation classification (SRC) [32] has better face recognition performance when each person has more than one sample in the gallery set. However, it is not suitable for SSPP because of the single gallery sample. Recently, SRC has been extended to generic learning method by using prototype and variation dictionary (P+V) model for SSPP. Deng et al. [33] proposed an extended sparse representation classification (ESRC) method, by using an auxiliary intra-class variation dictionary to represent the difference between the gallery and probe images. The atoms in variation dictionary typically represent intra-class difference of samples computed from the generic set. Motivated by ESRC, Yang et al. [34] proposed the sparse variation dictionary learning (SVDL) model, a projection from the gallery set to the generic set is learned, and the sparse variation dictionary is obtained by the projection using the generic set. By further extending SVDL, Ji et al. [35] proposed a collaborative probabilistic labels (CPL) method. First, the probabilistic labels are constructed by label propagation using the gallery samples and the generic samples. Then, the adaptive variation dictionary is obtained using the group sparse coding method. Finally, the probe sample is recognized using its corresponding adaptive variation dictionary and probabilistic labels. Moreover, Deng et al. [36] presented a superposed SRC based P+V model, namely, a probe sample is considered as the sum of the prototype and the intra-class variation. In their P+V model, the prototype is constituted by the sample in the gallery set since the gallery sample is assumed to be standard and variation-free. Yang et al. [37] proposed joint and collaborative representation with local adaptive convolution feature (JCR-ACF) based on the P+V model. The local region is obtained by densely sampling and sparsely detecting facial points method, and deep features are learned by convolutional neural networks (CNN) using the images of the local region. Then probe image of local region is recognized using the deep features and a generic set. Zhu et al. [38] proposed a local generic representation (LGR) based framework for SSPP face recognition. A local prototype dictionary is built by the neighboring patches from the gallery set, while an intra-class variation dictionary is built by using a generic set to predict the facial variation. Gao et al. [39] proposed a semi-supervised sparse representation based classification (S 3 $^{3}$ RC), the identity of the gallery sample of the minimum deviation with probe sample is the recognition result. Gao et al. [40] proposed a synergistic generic learning (SGL) method under the P+V model framework. Although SGL achieves a promising performance for SSPP face recognition, there still exist two problems. First, the neutral samples in prototype dictionary are difficult to identify, since the training dataset in unconstrained environment does not contain neutral samples of each subject. Therefore, SGL will not be well fit for this unconstrained case. Second, the variation information provided by the constructed variation dictionary is limited, resulting in the limited performance improvement. SSPP face recognition usually contains only a single positive neutral face image in the gallery set, and the probe samples in the unconstrained environment are affected by illumination changes, expression changes, occlusion and other factors. For the SSPP, the probe sample is difficult to match with the gallery samples because the diversity of samples in the gallery set does not exist. The prototype dictionary plus variation dictionary (P+V) model [36] can provide a solution for this problem. In [36], each generic sample or the difference between each generic sample and the mean of all the generic samples of this class is regarded as an atom, all atoms are constructed as the variation dictionary, this dictionary represents variation information of the probe sample. At the same time, the single sample in the gallery set is regarded as an atom in the prototype dictionary, and this prototype dictionary represents the class information. The probe sample is reconstructed using the prototype dictionary and the variation dictionary, and the sparse coefficients of the atoms are obtained. Finally, the coefficients of the atoms in the variation dictionary represents the variation degree of the probe sample, the predicted class of the probe sample is obtained by using the coefficients of the atoms in the prototype dictionary. In P+V model, the single gallery sample shares the variation information in the auxiliary generic set when the probe sample is reconstructed, although it solves the SSPP problem of not enough diversity of samples in the gallery set to some extent, there are still two problems for the construction of variation dictionary in P+V model: (1) The constructed variation dictionary contains some inter-class information, which makes the recognition degrade; (2) The prototype dictionary constructed using only the gallery samples cannot improve the inter-class discriminative ability, which limits the recognition accuracy improvement. To deal with the above two problems, under the framework of the P+V model [36], we propose a robust prototype dictionary and robust variation dictionary construction (RPRV) method to handle the SSPP face recognition problem in unconstrained environment. First, aiming at the problem that atoms in the variation dictionary have some inter-class scatter information, we propose a robust variation dictionary construction method to reduce the inter-class scatter. A ratio of inter-class scatter to intra-class scatter is defined first, and the atoms with the smaller ratio are selected to construct robust variation dictionary, which reduces the inter-class scatter in the variation dictionary. Second, aiming at the problem that the discriminative ability of atoms in prototype dictionary is insufficient, we propose a robust prototype dictionary construction method. A function of inter-class scatter is defined, then, the atoms with bigger function value are selected to construct prototype dictionary, which improves the inter-class discrimination ability. Using RPRV, we derive a robust method for SSPP to improve the recognition accuracy. The major contributions of the proposed method are listed below: 1) A ratio index is proposed for the variation dictionary atoms selection, which helps to reduce the inter-class discriminative ability in the variation dictionary; 2) A function index is defined for the prototype dictionary atoms selection, which improves the inter-class discriminative ability in the prototype dictionary; 3) The proposed variation dictionary and prototype dictionary construction method is more robust in P+V SSPP recognition framework so as to achieve better recognition accuracy as compared to existing methods. The rest of the paper is organized as follows: The proposed algorithm for SSPP face recognition is given in Section 2. The experimental results on the constrained datasets and unconstrained datasets, and the comparison with existing methods are given in Section 3. The conclusions are given in Section 4. 2 THE PROPOSED METHOD The sketch map of the RPRV construction method is shown in Figure 1. gallery images and generic images are combined for dictionary learning in order to construct the robust prototype dictionary (RP) and robust variation dictionary (RV). The robust prototype dictionary and robust variation dictionary are used to linearly represent a reconstructed sample with the same label as the prototype used. The label with the smallest residual error between the reconstructed sample and the probe sample is the recognition result. In this method, the enhanced robust prototype dictionary and robust variation dictionary help to improve the SSPP face recognition performance. FIGURE 1Open in figure viewerPowerPoint The sketch map of the RPRV method For the convenience of description, let y ∈ ℜ d $ {{\bf y}} \in {\Re ^d}$ be a probe sample, where d $ d$ denotes the dimension of a sample. Let G = [ g 1 , g 2 , … , g m ] ∈ ℜ d × m $ {{\bf G}} = [{{{\bf g}}_{{\bf 1}}},{{{\bf g}}_{{\bf 2}}}, {\ldots} ,{{{\bf g}}_m}] \in {\Re ^{d \times m}}$ be the gallery data matrix, where m $ m$ denotes the number of the gallery subjects, each subject has only one sample. The auxiliary generic data matrix is defined as A = [ A 1 , A 2 , … , A C ] ∈ ℜ d × n $ {{\bf A}} = [{{{\bf A}}_1},{{{\bf A}}_2}, {\ldots} ,{{{\bf A}}_C}] \in {\Re ^{d \times n}}$ , where C $ C$ denotes the number of the generic subjects, n $ n$ denotes the number of the generic samples, and A i ∈ ℜ d × n i $ {{{\bf A}}_i} \in {\Re ^{d \times {n_i}}}$ denotes that subject i $ i$ contains n i $ {n_i}$ samples, ∑ i = 1 C n i = n $ \sum \nolimits _{i = 1}^C {{n_i} = n}$ , A i ∈ Ω i $ {{{\bf A}}_i} \in {\Omega _i}$ , Ω i $ {\Omega _i}$ denotes class i $ i$ in the data matrix. The variation matrix is constructed by the generic samples and their class centroids as follows: V ̂ = [ A 1 − c 1 e 1 T , A 2 − c 2 e 2 T , … , A c − c c e c T ] , \begin{equation} {{\hat{\bf V}}} = [{{{\bf A}}_1} - {{{\bf c}}_1}{{{\bf e}}_1}^T,{{{\bf A}}_2} - {{{\bf c}}_2}{{{\bf e}}_2}^T,{\ldots} ,{{{\bf A}}_c} - {{{\bf c}}_c}{{{\bf e}}_c}^T], \end{equation} (1)where e i = [ 1 , … , 1 ] T ∈ ℜ n i × 1 $ {{{\bf e}}_i} = {[1, {\ldots} ,1]^T} \in {\Re ^{{n_i} \times 1}}$ , c i = A i e i / n i $ {{{\bf c}}_i} = {{{\bf A}}_i}{{{\bf e}}_i}/{n_i}$ denotes the geometric centroid of class Ω i ${\Omega _i}$ . Let V i = A i − c i e i T $ {{{\bf V}}_i} = {{{\bf A}}_i} - {{{\bf c}}_i}{{{\bf e}}_i}^T$ be the variation matrix of class Ω i $ {\Omega _i}$ . 2.1 Dictionary learning Gallery data matrix G ∈ ℜ d × m ${{\bf G}} \in {\Re ^{d \times m}}$ and variation matrix V ̂ ∈ ℜ d × n ${{\hat{\bf V}}} \in {\Re ^{d \times n}}$ are merged into the matrix X ∈ ℜ d × ( m + n ) ${{\bf X}} \in {\Re ^{d \times ({m + n} )}}$ using the following Equation (2) X = G , V ̂ . \begin{equation} {{\bf X}} = {\left[ {{{\bf G}},{{\hat{\bf V}}}} \right]}. \end{equation} (2) The dictionary D $ {{\bf D}}$ is learned from the matrix X $ {{\bf X}}$ X = DW + E , \begin{equation} {{\bf X}} = {{\bf DW}} + {{\bf E}}, \end{equation} (3)where D ∈ ℜ d × k ${{\bf D}} \in {\Re ^{d \times k}}$ denotes variation dictionary, k $ k$ is the number of atoms in the dictionary ( k ≤ ( m + n ) ≪ d ) $ (k \le ({m + n} ) \ll d )$ , W = [ w 1 , w 2 , … , w ( m + n ) ] ∈ ℜ k × ( m + n ) ${{\bf W}} = [ {{\bf w}}_{1},{{\bf w}}_{2} ,{\ldots} ,{{\bf w}}_{(m+n)} ]\in \Re ^{k\times (m+n)}$ is the sparse coefficient matrix, and E ∈ ℜ d × ( m + n ) $ {{\bf E}} \in {\Re ^{d \times ({m + n} )}}$ denotes the residual matrix. In this step, due to the high dimension of each column of the matrix X $ {{\bf X}}$ , it is time consuming to learn a dictionary D $ {{\bf D}}$ directly. Therefore, we use the skinny singular value decomposition (skinny SVD) method [41] to reduce dimensions as given by X = U r ∑ r H ′ r , \begin{equation} {{\bf X}} = {{{\bf U}}_{{\bf r}}}{\sum _{{\bf r}}}{{{{\bf H^{\prime }}}}_{{\bf r}}}, \end{equation} (4)where ∑ r = diag ( σ 1 , σ 2 , … , σ r ) $ {\sum _{{\bf r}}} = {\rm {diag}}({\sigma _1},{\sigma _2}, {\ldots} ,{\sigma _r})$ is a diagonal matrix with r $ r$ positive singular values in a descending order, U r ∈ ℜ d × r $ {{{\bf U}}_{{\bf r}}} \in {\Re ^{d \times r}}$ and H r ∈ ℜ ( m + n ) × r $ {{{\bf H}}_{{\bf r}}} \in {\Re ^{(m+n) \times r}}$ are column-wise orthogonal matrices satisfying U ′ r U r = H ′ r H r = I $ {{{{\bf U^{\prime }}}}_{{\bf r}}}{{{\bf U}}_{{\bf r}}} = {{{{\bf H^{\prime }}}}_{{\bf r}}}{{{\bf H}}_{{\bf r}}} = {{\bf I}}$ . Let X ∼ = ∑ r H ′ r ∈ ℜ r × ( m + n ) $ {{\tilde{\bf X}}} = {\sum _{{\bf r}}}{{{{\bf H^{\prime }}}}_{{\bf r}}} \in {\Re ^{r \times (m+n)}}$ ( r ≤ ( m + n ) ≪ d ) $ (r \le (m+n) \ll d)$ be a matrix with the reduced dimension and we learn dictionary D ∼ $ {{{\tilde{\bf D}}}}$ to represent X ∼ $ {{\tilde{\bf X}}}$ under sparse constraints. Specifically, the following objective function is used for minimization problem given by min D ∼ , W X ∼ − D ∼ W F 2 + γ W 1 s . t . d ∼ ′ i d ∼ i = 1 , ∀ i . \begin{equation} \def\eqcellsep{&}\begin{array}{l} \mathop {{\rm {min}}}\limits _{{{\tilde{\bf D}}}, {{\bf W}}} {\left\Vert {{{\tilde{\bf X}}} - {{\tilde{\bf D}W}}} \right\Vert} _F^2 + \gamma {{\left\Vert {{\bf W}} \right\Vert} _1} \\[7pt] \def\eqcellsep{&}\begin{array}{*{20}{c}}s.t. &\quad {{{{{\tilde{\bf d}^{\prime }}}}_i}{{{{\tilde{\bf d}}}}_i} = 1} \\[7pt] \end{array} ,\forall i. \end{array} \end{equation} (5) Equation (5) is a joint optimization problem of the dictionary D ∼ ${{\tilde{\bf D}}}$ and the coefficient matrix W ∼ ${{\tilde{\bf W}}}$ . Although it is not jointly convex with respect to D ∼ $ {{{\tilde{\bf D}}}}$ and W ∼ $ {{\tilde{\bf W}}}$ , it is convex with respect to each of them when the other one is fixed. Therefore, the coefficient matrix W ∼ $ {{\tilde{\bf W}}}$ is optimized using the basis pursuit denoising (BPDN)-homotopy algorithm in [42], when the dictionary D ∼ ${{\tilde{\bf D}}}$ is fixed. Then the dictionary D ∼ ${{\tilde{\bf D}}}$ is optimized using metaface learning algorithm in [43], when the coefficient matrix W ${{\bf W}}$ is fixed. J D ∼ = arg min D ∼ X ∼ − D ∼ W ∼ F 2 s . t . d ∼ j T d ∼ j = 1 , ∀ j , \begin{equation} \def\eqcellsep{&}\begin{array}{*{20}{c}} {J_{{{\tilde{\bf D}}}}} = \arg \mathop {\min }\limits _{{{\tilde{\bf D}}}} {\left\lbrace {{\left\Vert {{{\tilde{\bf X}}} - {{\tilde{\bf D}\tilde{W}}}} \right\Vert} _F^2} \right\rbrace} &\quad {s.t.{{{\tilde{\bf d}}}_j}^T{{{\tilde{\bf d}}}_j} = 1,\forall j,} \end{array} \end{equation} (6)where dictionary matrix D ∼ = [ d ∼ 1 , d ∼ 2 , … , d ∼ k ] ${{\tilde{\bf D}}} = [{{{\tilde{\bf d}}}_1},{{{\tilde{\bf d}}}_2}, {\ldots} ,{{{\tilde{\bf d}}}_k}]$ , d ∼ j ∈ ℜ r × 1 ${{{\tilde{\bf d}}}_j} \in {\Re ^{r \times 1}}$ , and coefficient matrix W ∼ = [ w ∼ 1 ; w ∼ 2 ; … , w ∼ k ] ∈ ℜ k × ( m + n ) $ {{\tilde{\bf W}}} = [ {{\tilde{\bf w}}}_{1}; {{\tilde{\bf w}}}_{2}; {\ldots} ,{{\tilde{\bf w}}}_{k} ]\in \Re ^{k\times (m+n)}$ , w ∼ j ∈ ℜ 1 × ( m + n ) ${{{\tilde{\bf w}}}_j} \in {\Re ^{1 \times {(m+n)}}}$ . In this phase, we update d ∼ j $ {{{\tilde{\bf d}}}_j}$ one by one by fixing the other columns in D ∼ $ {{{\tilde{\bf D}}}}$ , then Equation (6) is transformed into J d ∼ j = arg min d ∼ j Y − d ∼ j w ∼ j F 2 s . t . d ∼ j T d ∼ j = 1 , ∀ j , \begin{equation} \def\eqcellsep{&}\begin{array}{*{20}{c}}{J_{{{{\tilde{\bf d}}}_j}}} = \arg \mathop {\min }\limits _{{{{\tilde{\bf d}}}_j}} {\left\Vert {{{\bf Y}} - {{{\tilde{\bf d}}}_j}{{{\tilde{\bf w}}}_j}} \right\Vert} _F^2 &\quad {s.t.{{{\tilde{\bf d}}}_{{\bf j}}}^T{{{\tilde{\bf d}}}_{{\bf j}}} = 1,\forall j,} \end{array} \end{equation} (7)where Y = X ∼ − ∑ i ≠ j d ∼ i w ∼ i $ {{\bf Y}} = {{\tilde{\bf X}}} - \sum \limits _{i \ne j} {{{{\tilde{\bf d}}}_i}{{{\tilde{\bf w}}}_i}}$ . Then, let J d j = 0 $ {J_{{{{\bf d}}_j}}} = 0$ , we get the solution as follows: d ∼ j = Y w ∼ j T / Y w ∼ j T 2 . \begin{equation} {{{\tilde{\bf d}}}_j} = {{{\bf Y}\tilde{\bf w}_j^T \big/{{\left\Vert {\bf Y}\tilde{\bf w}_j^T \right\Vert} }}_2}. \end{equation} (8) After obtaining D ∼ $ {{{\tilde{\bf D}}}}$ and W $ {{\bf W}}$ , the dictionary D $ {{\bf D}}$ and the sparse coefficient matrix Z $ {{\bf Z}}$ in Equation (3) can be obtained by D = U r D ∼ W = W ∼ . \begin{equation} {\left\lbrace { \def\eqcellsep{&}\begin{array}{*{20}{c}} {{\bf D}} = {{{\bf U}}_{{\bf r}}}{{\tilde{\bf D}}} \\[7pt] {{{\bf W}} = {{\tilde{\bf W}}}} \end{array} } \right.}. \end{equation} (9) The summary dictionary learning algorithm is shown in Algorithm 1. ALGORITHM 1. Dictionary Learning algorithm Input: gallery data matrix G = [ g 1 , g 2 , … , g m ] ∈ ℜ d × m $ {{\bf G}} = [{{{\bf g}}_1},{{{\bf g}}_2}, {\ldots} ,{{{\bf g}}_m}] \in {\Re ^{d \times m}}$ generic data matrix V ̂ = [ v 1 , v 2 , … , v n ] ∈ ℜ d × n $ {{\hat{\bf V}}} = [{{{\bf v}}_1},{{{\bf v}}_2}, {\ldots} ,{{{\bf v}}_n}] \in {\Re ^{d \times n}}$ 1: Gallery data matrix G ${{\bf G}}$ and variation matrix V ̂ ${{\hat{\bf V}}}$ are merged into the matrix X ∈ ℜ d × ( m + n ) ${{\bf X}} \in {\Re ^{d \times ({m + n} )}}$ 2: the dimension of samples is reduced by skinny SVD X = U r ∑ r H ′ r ${{\bf X}} = {{{\bf U}}_{{\bf r}}}{\sum _{{\bf r}}}{{{\bf H^{\prime }}}_{{\bf r}}}$ , low-dimension samples matrix X ∼ = ∑ r H ′ r ∈ ℜ r × ( m + n ) ${{\tilde{\bf X}}} = {\sum _{{\bf r}}}{{{\bf H^{\prime }}}_{{\bf r}}} \in {\Re ^{r \times (m+n)}}$ is obtained 3: dictionary learning X ∼ = D ∼ W ∼ ${{\tilde{\bf X}}} = {{\tilde{\bf D}\tilde{W}}}$ 4: D ∼ ${{{\tilde{\bf D}}}}$ is initialized as a random unit vector per column 5: while do 6: Fix D ∼ ${{{\tilde{\bf D}}}}$ , update W ∼ ${{\tilde{\bf W}}}$ by BPDN homotopy method 7: Fix W ∼ ${{\tilde{\bf W}}}$ , update D ∼ ${{{\tilde{\bf D}}}}$ 8: for j = 1 $j=1$ to k $k$ do 9: Fix d ∼ i , i ≠ j ${{{\tilde{\bf d}}}_i},i \ne j$ , update d ∼ j ${{{\tilde{\bf d}}}_j}$ using Equation (8) 10: end for 11: end while 12: D = U r D ∼ ${{\bf D}} = {{{\bf U}}_r}{{\tilde{\bf D}}}$ , W = W ∼ ${{\bf W}} = {{\tilde{\bf W}}}$ Output: dictionary matrix D ${{\bf D}}$ coefficient matrix W ${{\bf W}}$ 2.2 Dictionary construction The prototype dictionary and variation dictionary are constructed in this subsection. First, the coefficient matrix W ${{\bf W}}$ in Section 2.1 can be written as W = [ P , Q ] ${{\bf W}} = [ {{{\bf P}},{{\bf Q}}} ]$ , where P = [ p 1 , p 2 , … , p m ] ∈ ℜ k × m , Q = [ q 1 , q 2 , … , q n ] ∈ ℜ k × n ${{\bf P}} = [ {{{{\bf p}}_1},{{{\bf p}}_2}, {\ldots} ,{{{\bf p}}_m}} ] \in {\Re ^{k \times m}},{{\bf Q}} = [ {{{{\bf q}}_1},{{{\bf q}}_2}, {\ldots} ,{{{\bf q}}_n}} ] \in {\Re ^{k \times n}}$ . For variation matrix V ̂ ∈ ℜ d × n ${{\hat{\bf V}}} \in {\Re ^{d \times n}}$ , let v i , i = 1 , 2 , … , n ${{{\bf v}}_i},i = 1,2, {\ldots} ,n$ be column i $i$ of the matrix V ̂ ${{{\hat{\bf V}}}}$ , v i ${{{\bf v}}_i}$ can be represented as follows: v i ≈ D q i = ∑ j = 1 k d j q i , j . \begin{equation} {{{\bf v}}_i} \approx {{\bf D}}{{{\bf q}}_i} = \sum \nolimits _{j = 1}^k {{{{\bf d}}_j}{q_{i,j}}} . \end{equation} (10) Let v ∼ i , j = d j q i , j ${{{\tilde{\bf v}}}_{i,j}} = {{{\bf d}}_j}{q_{i,j}}$ , Equation (10) can be written as the summation of k $k$ components, given by v i ≈ ∑ j = 1 k v ∼ i , j . \begin{equation} {{{\bf v}}_i} \approx \sum \nolimits _{j = 1}^k {{{{\tilde{\bf v}}}_{i,j}}}. \end{equation} (11) Then we use v ∼ i , j ${{{\tilde{\bf v}}}_{i,j}}$ and its label information to calculate the fisher ratio of each column d j ${{{\bf d}}_j}$ , the bigger the fisher ratio of column d j ${{{\bf d}}_j}$ , the more discriminative to separate the class of samples. We denote the column mean of v ∼ i , j , i = 1 , 2 , … , n ${{{\tilde{\bf v}}}_{i,j}},i = 1,2, {\ldots} ,n$ by v ¯ j ${{{{\bar{\bf v}}}}_j}$ and the column mean of v i , j , i = 1 , 2 , … , n c ${{{\bf v}}_{i,j}},i = 1,2, {\ldots} ,{n_c}$ belonging to class Ω c ${\Omega _{\rm {c}}}$ by v ¯ j c ${{\bar{\bf v}}}_j^c$ , where Ω c , c = 1 , 2 , … , C ${\Omega _{\rm {c}}},c = 1,2, {\ldots} ,C$ denotes the class of the samples v i ${{{\bf v}}_i}$ , n c ${n_c}$ is the number of samples v i ${{{\bf v}}_i}$ belonging to class Ω c ${\Omega _{\rm {c}}}$ . The fisher ratio [44] of feature v i , j ${{{\bf v}}_{i,j}}$ is written as follows: f v ( j ) = σ b σ w = ∑ c = 1 C ( v ¯ j c − v ¯ j ) 2 ∑ c = 1 C ∑ v i ∈ Ω c ( v ∼ i , j − v ¯ j c ) 2 , \begin{equation} {f_v}(j) = \frac{{{\sigma _b}}}{{{\sigma _w}}} = \frac{{\sum \nolimits _{c = 1}^C {{{({{\bar{\bf v}}}_j^c - {{{{\bar{\bf v}}}}_j})}^2}} }}{{\sum \nolimits _{c = 1}^C {\sum \nolimits _{{{{\bf v}}_i} \in {\Omega _c}} {{{({{{{\tilde{\bf v}}}}_{i,j}} - {{\bar{\bf v}}}_j^c)}^2}} } }}, \end{equation} (12)where σ b ${\sigma _b}$ denotes inter-class scatter, σ w ${\sigma _w}$ denotes intra-class scatter. Each gallery sample can be represented as follows: g i ≈ D ·