# CoLR: Classification-Oriented Local Representation for Image Recognition

Authors: Tan Guo; Lei Zhang; Xiaoheng Tan; Liu Yang; Zhiwei Guo; Fupeng Wei
Journal: Complexity
 (2019)

---

## Abstract

Naïve sparse representation has stability problem due to its unsupervised nature, which is not preferred for classification tasks. For this problem, this paper presents a novel representation learning method named classification-oriented local representation (CoLR) for image recognition. The core idea of CoLR is to find the most relevant training classes and samples with test sample by taking the merits of class-wise sparseness weighting, sample locality, and label prior. The proposed representation strategy can not only promote a classification-oriented representation, but also boost a locality adaptive representation within the selected training classes. The CoLR model is efficiently solved by Augmented Lagrange Multiplier (ALM) scheme based on a variable splitting strategy. Then, the performance of the proposed model is evaluated on benchmark face datasets and deep object features. Specifically, the deep features of the object dataset are obtained by a well-trained convolutional neural network (CNN) with five convolutional layers and three fully connected layers on the challenging ImageNet. Extensive experiments verify the superiority of CoLR in comparison with some state-of-the-art models.


---

## Body

## 1. Introduction

Image recognition is a fundamental problem in pattern recognition community. Industrial prospects and research interests in image recognition have been motivated by a wide range of real-world applications [1–3]. There are mainly two components for a common image recognition system: (1) robust and discriminant feature learning [4–7], such as Gabor wavelet-based features [8] and local binary pattern (LBP) [9]. The linear holistic appearance-based approaches, such as principal component analysis (PCA) [10] and linear discriminative analysis (LDA) [11], have greatly advanced image recognition technology. In addition, nonlinear subspace methods, such as Kernel PCA [12], use kernel tricks to map original data into a high-dimensional space to make data separable. The second component is (2) classifier construction, for example, Nearest Neighbor (NN) [13], Nearest Feature Line (NFL) [14], and Nearest Feature Plane (NFP) [15]. These classifiers, known as representation-based models, concern how to identify the query image based on the linear combination of training samples [16].According to the labels of the training samples used to represent test sample, representation-based approaches could be divided into within-class representation-based methods and across-class representation-based methods [16]. Within-class representation-based methods evaluate the relation between query sample and the training samples of each individual class separately by class. NN is the simplest nonparametric within-class representation-based classification method. It classified the test sample by searching for its NN in training dataset. As a simple extension of NN, NFL classifies the test sample according to the nearest feature line of every two samples in each training class. NFP further uses three independent training samples to represent the test sample. Classifiers using more training samples to represent test image have also been proposed, such as the nearest subspace (NS) classifiers [17], which represent the test sample by all the training samples of each class.Recent research has demonstrated that sparse coding (or sparse representation) is a powerful image representation model. The most typical across-class representation-based method is sparse representation-based classification (SRC) method [16], where test samples are first sparsely coded over all the training samples based on the l1-norm minimization of representation coefficients, and then performs classification according to the representation results. SRC is robust to occlusion, illumination, and noise. Many related works have been developed to improve SRC, such as kernel based SRC [18], robust sparse coding [19], Gabor feature based SRC [20], and sparse dense hybrid representation [21]. SRC assumes that samples from a single subject lie on a subspace and restricts the representation of test sample to be sparse by the regularization of sparsity-inducingl1-norm. This endows SRC with discriminative ability. Although SRC has shown excellent results, its working mechanism has been heat-debated. Zhanget al. [22] argued that all the samples should give a contribution to represent the test sample and proposed a collaborative representation (CR) based method for classification with l2-norm constraint. Yanget al. [23] indicated that l0-optimizer can achieve sparsity only, whereas l1-optimizer in SRC can achieve closeness as well as sparsity. It is closeness that guarantees the effectiveness of the l1-optimizer based SRC. Akhtaret al. [24] suggested that sparseness explicitly contributes to improved classification; hence it should not be completely ignored for computational gains. More recently, the discriminant nature of collaborative representation was analyzed in [25].Ideally, the representation of a test example should focus on the training data from the identical class with the assumption of independence of sample subspaces, and such a representation is also discriminative for classification. Nevertheless, the assumption cannot always be guaranteed due to the similarity of patterns between different classes. That is, samples from different classes may have high correlation, and SRC tends to randomly select a single representative sample from the high-correlation training samples [26–28]. Meanwhile, SRC might select quite different training sample with test sample to favor sparsity. This trait is undesirable as it will produce different representations for similar images and thus destroy the classification performance.One feasible way for the limitation is to go beyond sparsity and take into account additional information about the underlying structure of the solution. A desired representation for a test sample should have a group structure with the significant representation coefficients focusing on the correct subject [29]. For this purpose, Huanget al. [30] proposed a class specific sparse representation for classification (CSSRC) to seek certain group sparsity structure by harnessing the label information of training data. CSSRC could be solved as a classical group lasso (GL) problem attaining the purpose of sparsity at the group level by minimizing the l1-norm across classes and l2-norm within each class. The minimization of l1-norm across classes can help find the correct subject. However, the dense representation within the selected class may prevent the attained result from the desired solution as the optimal representation of a test sample over training samples of the correct subject may not necessarily be dense [31]. The class-wise sparse representation (CSR) method is proposed in [31] to seek an optimum representation of the test image by minimizing the class-wise sparsity of the training data. However, the class-wise sparsity regularization treats different training classes equally and does not take the relation between test sample and each training class into consideration, which may restrict the discrimination of obtained representation.Furthermore, Yuet al. [32] empirically observed that sparse coding results tend to be local, and the nonzero coefficients are often assigned to training samples nearby the represented sample. They theoretically pointed out that under certain assumption locality is more essential than sparsity, as locality must lead to sparsity but not necessarily vice versa. Based on the observation, Wanget al. [33] developed a locality-constrained linear coding (LLC) scheme to find a locality-constrained representation of test sample by its nearest neighbors. Luet al. [34] proposed a weighted sparse representation-based classification method, where both data locality and linearity were considered. Several two-phase sparse representation-based methods have been proposed to conduct coarse-to-fine classification and achieved good performance [35–38]. These methods actually illustrate the effect of “locality” in representation-based classification methods and illustrate the principal that samples closer to the test sample should have more significance in representing it. By considering both data locality and label information of training data, Chaoet al. [39] proposed a locality and group sensitive sparse representation (LGSR). Nevertheless, the group sparsity penalized term was directly added to the data locality constraint term, which may disarrange the group sparse structure of the solution. In [40], a weighted group sparse representation classification (WGSRC) is developed by minimizing the weighted mixed-norm (l2,1-norm) regularized reconstruction error with respect to training samples. However, the l2-norm reconstruction error measurement might not handle real world contamination well [31]. Table 1 tabulates a brief comparison of several related state-of-the-art methods. Besides, some dictionary learning methods have been developed to learn a dictionary from the training samples to replace the original training samples for representation learning [41–49]. The K-SVD algorithm is one of the typical dictionary learning algorithms [41]. K-SVD is a generalized k-means clustering algorithm. However, the K-SVD algorithm is not suitable for classification tasks, because it only requires that the learned dictionary should well reconstruct the training samples. To enhance the discriminative ability of learned dictionary, Zhang and Li studied a discriminative K-SVD (D-KSVD) algorithm [42]. Jianget al. [43] proposed a label consistent K-SVD (LC-KSVD) dictionary learning algorithm, which associated the label information with the atoms to improve the classification performance. Liet al. [44] developed a discriminative dictionary learning algorithm termed as locality constrained and label embedding dictionary learning (LCLE-DL) algorithm. More recently, a mechanism-based structured analysis discriminative dictionary learning (ADDL) framework is developed to seamlessly integrate analysis discriminative dictionary learning, analysis representation, and analysis classifier training into a unified model [45]. A discriminative block-diagonal low-rank representation (BDLRR) method is proposed to learn discriminative data representation by imposing an effective structure in the low rank representation framework [50].Table 1
A comparison between several representation-based learning methods.
Methods Sparse Sample local Representation residual measure Sample sparse Group sparse Class-wise sparse SRC [16] √ × × × l 2-norm  LLC [32] × × × √ l 2-norm  LGSR [39] × √ × √ l 2-norm  CSSRC [30] × √ × × l 2-norm  WGSRC [40] × √ × √ l 2-norm  CSR [31] × × √ × l 1-norm  CoLR × × √ √ l 1-normOne of the key problems for representation-based classification method is to fully utilize the prior information of data distribution. This paper aims to learn discriminative classification-oriented local representation for image recognition with the guiding of prior information induced from the observed data. The overview of the proposed method is shown in Figure1. The idea is that the training samples and classes near the test sample should make more contribution in representing it. In view of this, different training class is distinguished with different weights according to the distance between the test sample and each training class. Furthermore, weights are introduced into each class by utilizing a locality adaptor penalizing the training samples far away from the test sample. In sum, we consider three principals, i.e., class-wise sparseness, data locality, and class weighting. Class information is utilized to seek the minimum number of training classes in representing test sample. An l1-norm based loss function is utilized, by which the obtained representation can less over-fit the outliers. With these principals, the developed model can not only encourage class-wise sparseness, but also boost locality sensibility within the selected training classes. Thus, the obtained representation is more discriminative with both the most relevant training classes and samples highlighted, which is desired for classification. Several contributions of the paper are listed as follows.Figure 1
Overview of the proposed method. CoLR adopts a coarse-to-fine strategy to search for the most relevant training samples and classes with test sample. Nonneighbor training classes and samples are implicitly excluded by imposing bigger punishment (in red dotted line). This way, the test sample is guided to be represented by its nearby training classes and samples for a more discriminative classification-oriented local representation. This is the discriminability origin of CoLR.( 1 ) By taking the advantages of data locality and label priors, the proposed CoLR model can learn discriminative classification-oriented local representation uncovering the underlying membership of test samples for classification.( 2 ) An efficient optimization algorithm is developed to solve CoLR model based on a variable splitting strategy and the Augmented Lagrangian Multiplier (ALM) method.( 3 ) The performance of CoLR is verified on various benchmark face databases and deep CNN features, and promising results have been achieved in comparison with some state-of-the-art models.The remainder of this paper is organized as follows. Section2 introduces some typical across-class representation-based classification algorithms. The CoLR model is proposed and described in Section 3. Experiments on benchmark databases are presented to evaluate the proposed method in comparison with other popular methods in Section 4. Finally, the conclusion is made in Section 5.


## 2. Related Works

In closed-universe image recognition scenario, the training samples and their class labels are usually provided. Assume there aren training face samples from C distinguished classes with ni training samples in class i, the training sample matrix A=[A1,A2…AC]∈Rm×n, Ai=[ai,1,ai,2…ai,ni]∈Rm×ni(i=1,2…C). Each column in A is an m-dimensional vector through column concatenation of the training sample. n=∑i=1Cni is the total number of training samples. The task is to assign the given test sample y∈Rm with the correct class label. In this section, we will review two typical works.


### 2.1. Sparse Representation for Classification

Sparse representation based classification (SRC) is based on the concept that patterns from the same class lie on a linear subspace [54]. The method parsimoniously searches for the most similar training sample in training set to represent the test sample. If a new coming test sample y comes from the class i, it will approximately lie on the linear span of the training samples of this class as (1)y≈AixiAnd the other samples in the training set cannot reconstruct the test sample as faithful as the ones from the identical class of the test sample. In practice, the test sample is represented using all the samples of training setA. (2)y≈Axwhere x=[0…0,xi,1,xi,2…xi,ni,0…0]T is a coefficient vector whose entries are ideally zero except those associated with i-th class. For a sparse and discriminative solution, SRC aims to solve the l0-minimization problem as(3)x∗=arg⁡minxx0s.t.y-Ax2≤εwhere ε is a given tolerance and ∙0 denotes the l0-norm, which indicates the number of nonzero entries in the representation vector. However, the minimization of the l0-norm is an NP-hard problem. Donoho proved that “for most large underdetermined systems of linear equations the minimal l1-norm solution is also the sparsest solution” [55]. Thus, the solution of (3) is equivalent to the following l1-norm minimization problem:(4)x∗=arg⁡minxx1s.t.y-Ax2≤εOnce the coding vector is obtained, the identification of the test sample is performed by checking which training class leads to the minimal reconstruction residual as follows:(5)identityy=arg⁡mini⁡y-Aixi2where xi is the coding vector corresponding to training class Ai. A sparse representation x of y over A is naturally discriminative to indicate the identity of y.


### 2.2. Group LASSO for Classification

Using thel1-norm to regularize the representation coefficient as in (4) may lead to an unstable solution because the training samples exhibit strong correlations. In the circumstance, SRC is known to have stability problem. It might represent a test sample by training data from distinct subjects, which is undesirable for classification. As a remedy for this problem, structured representation methods are proposed to take advantage of label priors. The training samples with the same label are defined as a group. An l2,1-norm regularization term is used as the group sparsity constraint, which is also known as group lasso [56]. The formulation of group lasso is defined as (6)x∗=arg⁡minx∑i=1Cxi2s.t.y-Ax2≤εwhere xi is the coding vector corresponding to training class Ai. This constraint enforces nonzeros coefficients to occur at a few specific groups, while those within the same group can be dense once that group is selected. However, the dense representation within the class may adversely affect the correct class selection during the minimization process of group lasso [30].


## 2.1. Sparse Representation for Classification

Sparse representation based classification (SRC) is based on the concept that patterns from the same class lie on a linear subspace [54]. The method parsimoniously searches for the most similar training sample in training set to represent the test sample. If a new coming test sample y comes from the class i, it will approximately lie on the linear span of the training samples of this class as (1)y≈AixiAnd the other samples in the training set cannot reconstruct the test sample as faithful as the ones from the identical class of the test sample. In practice, the test sample is represented using all the samples of training setA. (2)y≈Axwhere x=[0…0,xi,1,xi,2…xi,ni,0…0]T is a coefficient vector whose entries are ideally zero except those associated with i-th class. For a sparse and discriminative solution, SRC aims to solve the l0-minimization problem as(3)x∗=arg⁡minxx0s.t.y-Ax2≤εwhere ε is a given tolerance and ∙0 denotes the l0-norm, which indicates the number of nonzero entries in the representation vector. However, the minimization of the l0-norm is an NP-hard problem. Donoho proved that “for most large underdetermined systems of linear equations the minimal l1-norm solution is also the sparsest solution” [55]. Thus, the solution of (3) is equivalent to the following l1-norm minimization problem:(4)x∗=arg⁡minxx1s.t.y-Ax2≤εOnce the coding vector is obtained, the identification of the test sample is performed by checking which training class leads to the minimal reconstruction residual as follows:(5)identityy=arg⁡mini⁡y-Aixi2where xi is the coding vector corresponding to training class Ai. A sparse representation x of y over A is naturally discriminative to indicate the identity of y.


## 2.2. Group LASSO for Classification

Using thel1-norm to regularize the representation coefficient as in (4) may lead to an unstable solution because the training samples exhibit strong correlations. In the circumstance, SRC is known to have stability problem. It might represent a test sample by training data from distinct subjects, which is undesirable for classification. As a remedy for this problem, structured representation methods are proposed to take advantage of label priors. The training samples with the same label are defined as a group. An l2,1-norm regularization term is used as the group sparsity constraint, which is also known as group lasso [56]. The formulation of group lasso is defined as (6)x∗=arg⁡minx∑i=1Cxi2s.t.y-Ax2≤εwhere xi is the coding vector corresponding to training class Ai. This constraint enforces nonzeros coefficients to occur at a few specific groups, while those within the same group can be dense once that group is selected. However, the dense representation within the class may adversely affect the correct class selection during the minimization process of group lasso [30].


## 3. The Proposed Classification-Oriented Local Representation Model

### 3.1. Model Formulation

Searching classes instead of individual samples can alleviate the problem of random selection of highly correlated data in SRC. By taking both weighted class-wise sparseness across classes and sample locality within each class into consideration, the following optimization problem is proposed.(7)minx12y-Ax1+λ∑i=1Cwizis.t.zi=di⨀xi2where x∈Rn is representation coefficient of test sample y over training dataset A. The first term y-Ax1 is the representation residual of the test sample y measured byl1-norm, and the second term indicates the weighted class-wise sparseness regularization term. Specifically, the l2-norm of locality sensitive coefficients within each class is weighted to search for the minimal number of classes out of C training classes by introducing a new class indicator variable z=z1,z2,…,zCT. wi (i=1,2…C) is the weight of samples of class i in representing test sample y.di is a vector that gives different freedom for each training sample associated with class i. λ>0 is a tradeoff parameter between the representation residual and representation coefficient.We aim to represent test sampley not only by its neighbors but also by highly relevant training classes. As illustrated in (7), unlike pursuing a group sparse representation in Rn like group lasso, we directly search for a weighted class-wise sparse representation in RC by taking the relation between test sample and each training class into consideration. Afterwards, the dissimilar training samples are penalized by incorporating a locality adaptor. wis vary from class to class to boost or inhibit the corresponding training class in representing y, and a large wi would make the corresponding zi shrink to be small. l1-norm is used to measure the representation residual, which is proved to be robust to handle outliers, and the obtained representation of test sample would less over-fit the outliers. In model (7), ⨀ denotes the element-wise multiplication, and di=[di,1,di,2…di,ni] (i=1,2…C) performs as a locality adaptor that gives different freedom for each training sample aij (the j-th training sample of class i) proportional to its similarity with test sample y. (8)dij=exp⁡disty,aijσ1⁡where disty,aij is the Euclidean distance between test sample y and training sample ai,j. σ1 is used to adjust the weight decay speed for the locality adaptor. A larger dij indicates a farther distance between y and aij. In (7), wi is used to evaluate the distance between the test sample with training class i. The linear regression model is utilized to measure the distance between test sample and each training class. The test sample is firstly represented as a linear combination of the training samples in each class.(9)y≈AixiThe reconstructed test sampley~i over class Ai is (10)y~i=AiAiTAi-1AiTyThe following metric is utilized to measure the distance between test sampley and training class i:(11)wi=exp⁡y-y~i2σ2In (11), wi indicates the distance from y to the subspace generated by Ai. A larger wi means class i is further from y and should make less contribution to represent it. σ2 is a bandwidth parameter and used for adjusting the weight decay. Figure 2 intuitively illustrates the motivation. Figure 2(a) is a test sample from Class 1. Figure 2(b) shows the normalized distance, i.e., wi (i=1,2…C) in (7), between test sample and each training class. Figure 2(c) shows the normalized distance, i.e., di (i=1,2…n) in (7), between test sample and each training sample. Figure 2(d) shows the final representation of test sample (a) over the training set. As the test sample is from Class 1, so the punishment on the representation coefficient of training samples in Class 1 is relatively small. For other training samples and classes, the punishment is bigger. With these constrains, the test sample is guided be represented by the nearby training samples from Class 1 as shown in Figure 2(d), which is quite discriminative for classification.Figure 2
An example for proposed method on 228 training samples of 38 subjects from Extended Yale B database [51]. (a) A test sample from Class 1; (b) the weight on each training class; (c) the weight on each training sample; (d) the representation coefficients obtained by our method.
(a) (b) (c) (d)


### 3.2. Model Optimization

As presented above, the optimization problem of developed model is expressed as follows:(12)minx12y-Ax1+λ∑i=1Cwizis.tzi=di⨀xi2The second term in (12) is the representation coefficient regularization. ∑i=1Cwizi=w1z1+…+wCzC. The term can be written in the form of l1-norm with a new variable v. v1=w1d1⨀x12+…+wCdC⨀xC2 with vi=widi⨀xi2 (i=1,2…C). Model (12) can be rewritten as(13)minx12y-Ax1+λv1s.t.vi=widi⨀xi2where v=v1,v2,…,vCT. We introduce a vector ri=[ri1,ri2,…,rini]T  i=1,2…C, rij=widij, j=1,2…,ni, and then we have (14)vi=ri⨀xi2Furthermore,(15)vi=Rixi2where Ri∈Rni×ni is a diagonal matrix for the i-th class with ri as its diagonal elements, and R is given by (16)R≔R1⋯⋮⋱⋮⋯RCLete=y-Ax. We obtain the following equivalent optimization problem with the diagonal weight matrix R by introducing auxiliary variable u∈RC:(17)mine,x,z,u12e1+λv1s.t.e=y-Ax,u=Rx,vi=ui2where v=v1,v2,…,vCT. Denote u~=[u12,u22,…,uC2]T∈RC, and ui=Rixi∈Rni is a subvector of u with elements associated with class i. (17) is a constrained optimization problem which could be solved by the Augmented Lagrangian Multiplier (ALM) method [57, 58]. Its corresponding ALM function is given as (18)Lμe,x,v,u=12e1+λv1+αTy-Ax-e+βTRx-u+γTu~-v+μ2y-Ax-e22+Rx-u22+u~-v22where α∈Rm, β∈Rn, γ∈RC are vectors of Lagrange multipliers and μ>0 is the penalty parameter. Instead of optimizing all arguments simultaneously, as e,x,v,u are separable, we solve them individually and iteratively. In the (k+1)-th iteration, the updating schemes are as follows.Step 1 (updatee).
We updatee by solving the following subproblem with x, z, and u fixed:(19)ek+1=arg⁡mine⁡12e1+αkTy-Axk-e+μk2y-Axk-e22=S1/2μky-Axk+αkμkwhere Sϵ(∙), ϵ>0, is the soft-thresholding (shrinkage) operator defined component-wise as (20)Sϵθi=sign⁡θi∙max⁡θi-ϵ,0Step 2 (updatex).
We updatex by solving the following subproblem with e, v, and u fixed:(21)xk+1=arg⁡minx⁡αkTy-Ax-ek+1+βkTRx-uk+μk2y-Ax-ek+122+Rx-uk22=arg⁡minx⁡μk2xTATA+RTRx-μky-ek+1+αkμkTA+uk-βkμkTRx
This is a convex quadratic problem. Hence the problem reduces to solving the following linear system:(22)μkATA+RTRx=μkATy-ek+1+αkμk+RTuk-βkμk
The close-form solution ofx is obtained as(23)xk+1=ATA+RTR-1ATy-ek+1+αkμk+RTuk-βkμkStep 3 (updatev).
We updatev by solving the following subproblem with x, e, and u fixed:(24)vk+1=arg⁡minv⁡λv1+γkTu~k-v+μk2u~k-v22=arg⁡minv⁡λμkv1+12v-u~k+γkμk22which can also be solved via the soft-thresholding operator as in (20).Step 4 (updateu).
We updateu by solving the following subproblem with e, v, and x fixed:(25)uk+1=arg⁡minu⁡βkTRxk+1-u+γkTu~-vk+1+μk2Rxk+1-u22+u~-vk+122With some manipulation, we have (26)uk+1=arg⁡minu⁡∑i=1Cγki-μkvk+1iui2+μkui-Rixk+1i+βki/μk222which has a closed form solution by one-dimensional shrinkage (or soft-thresholding) formula [59](27)uk+1i=max⁡qki2-γki-μkvk+1i2μk,0qkiqki2.i=1,2…C.where qki≔Rixk+1i+βki/μk/2Step 5.
Update the Lagrangian multipliersα, β, γ and the parameter μ as follows:(28)αk+1=αk+μky-Axk+1-ek+1βk+1=βk+μkRxk+1-uk+1γk+1=γk+μku~k+1-vk+1μk+1=min⁡ρμk,μmaxNote that the subproblems fore, x, u, v are all convex problems. They both have closed-form solutions. For the competence of presentation, the detailed optimization procedure is outlined in Algorithm 1. The convergence property of Algorithm 1 can be guaranteed by the existing ADM theory [56]. From the experimental perspective, the developed optimization algorithm exhibits good convergence property as shown in Figure 3. In our experiments, the iteration number is empirically less than 100 under the given settings. Once the optimal solution x is achieved, the label of test sample y can be obtained as(29)i∗=arg⁡mini⁡y-Aixi1xi1,i=1,2…Cwhere xi is the coding vector associated with training class Ai.Algorithm 1:SolvingCoLR model based on ALM.
Input: training set A, test sample y, diagonal matrix R, parameter λ. 1:Initialize:x0=0, z0=0, u0=0, α0=0, β0=0, γ0=0, μ 0 = 10 - 3, μmax=108, ρ=1.1, ε=10-6. 2: While  not converged  do 3: Fix other variable and update eby solving problem (19). 4: Fix other variable and update x by solving problem (21). 5: Fix other variable and update vby solving problem (24). 6: Fix other variable and update u by solving problem (25). 7: Update the multipliers and parameters by (28) 8: Check the convergence conditions: y-Ax-e∞<ε,Rx-u∞<ε,andv-u~∞<ε 9:End While Output: x, eFigure 3
An illustration for the convergence property of Algorithm1. (a) is the curve of objective function value versus iteration number. (b)-(d) are the curves of 3 convergence conditions in optimization versus iteration number: (b) y-Ax-e∞; (c) v-u~∞; (d) Rx-u∞.
(a) (b) (c) (d)


## 3.1. Model Formulation

Searching classes instead of individual samples can alleviate the problem of random selection of highly correlated data in SRC. By taking both weighted class-wise sparseness across classes and sample locality within each class into consideration, the following optimization problem is proposed.(7)minx12y-Ax1+λ∑i=1Cwizis.t.zi=di⨀xi2where x∈Rn is representation coefficient of test sample y over training dataset A. The first term y-Ax1 is the representation residual of the test sample y measured byl1-norm, and the second term indicates the weighted class-wise sparseness regularization term. Specifically, the l2-norm of locality sensitive coefficients within each class is weighted to search for the minimal number of classes out of C training classes by introducing a new class indicator variable z=z1,z2,…,zCT. wi (i=1,2…C) is the weight of samples of class i in representing test sample y.di is a vector that gives different freedom for each training sample associated with class i. λ>0 is a tradeoff parameter between the representation residual and representation coefficient.We aim to represent test sampley not only by its neighbors but also by highly relevant training classes. As illustrated in (7), unlike pursuing a group sparse representation in Rn like group lasso, we directly search for a weighted class-wise sparse representation in RC by taking the relation between test sample and each training class into consideration. Afterwards, the dissimilar training samples are penalized by incorporating a locality adaptor. wis vary from class to class to boost or inhibit the corresponding training class in representing y, and a large wi would make the corresponding zi shrink to be small. l1-norm is used to measure the representation residual, which is proved to be robust to handle outliers, and the obtained representation of test sample would less over-fit the outliers. In model (7), ⨀ denotes the element-wise multiplication, and di=[di,1,di,2…di,ni] (i=1,2…C) performs as a locality adaptor that gives different freedom for each training sample aij (the j-th training sample of class i) proportional to its similarity with test sample y. (8)dij=exp⁡disty,aijσ1⁡where disty,aij is the Euclidean distance between test sample y and training sample ai,j. σ1 is used to adjust the weight decay speed for the locality adaptor. A larger dij indicates a farther distance between y and aij. In (7), wi is used to evaluate the distance between the test sample with training class i. The linear regression model is utilized to measure the distance between test sample and each training class. The test sample is firstly represented as a linear combination of the training samples in each class.(9)y≈AixiThe reconstructed test sampley~i over class Ai is (10)y~i=AiAiTAi-1AiTyThe following metric is utilized to measure the distance between test sampley and training class i:(11)wi=exp⁡y-y~i2σ2In (11), wi indicates the distance from y to the subspace generated by Ai. A larger wi means class i is further from y and should make less contribution to represent it. σ2 is a bandwidth parameter and used for adjusting the weight decay. Figure 2 intuitively illustrates the motivation. Figure 2(a) is a test sample from Class 1. Figure 2(b) shows the normalized distance, i.e., wi (i=1,2…C) in (7), between test sample and each training class. Figure 2(c) shows the normalized distance, i.e., di (i=1,2…n) in (7), between test sample and each training sample. Figure 2(d) shows the final representation of test sample (a) over the training set. As the test sample is from Class 1, so the punishment on the representation coefficient of training samples in Class 1 is relatively small. For other training samples and classes, the punishment is bigger. With these constrains, the test sample is guided be represented by the nearby training samples from Class 1 as shown in Figure 2(d), which is quite discriminative for classification.Figure 2
An example for proposed method on 228 training samples of 38 subjects from Extended Yale B database [51]. (a) A test sample from Class 1; (b) the weight on each training class; (c) the weight on each training sample; (d) the representation coefficients obtained by our method.
(a) (b) (c) (d)


## 3.2. Model Optimization

As presented above, the optimization problem of developed model is expressed as follows:(12)minx12y-Ax1+λ∑i=1Cwizis.tzi=di⨀xi2The second term in (12) is the representation coefficient regularization. ∑i=1Cwizi=w1z1+…+wCzC. The term can be written in the form of l1-norm with a new variable v. v1=w1d1⨀x12+…+wCdC⨀xC2 with vi=widi⨀xi2 (i=1,2…C). Model (12) can be rewritten as(13)minx12y-Ax1+λv1s.t.vi=widi⨀xi2where v=v1,v2,…,vCT. We introduce a vector ri=[ri1,ri2,…,rini]T  i=1,2…C, rij=widij, j=1,2…,ni, and then we have (14)vi=ri⨀xi2Furthermore,(15)vi=Rixi2where Ri∈Rni×ni is a diagonal matrix for the i-th class with ri as its diagonal elements, and R is given by (16)R≔R1⋯⋮⋱⋮⋯RCLete=y-Ax. We obtain the following equivalent optimization problem with the diagonal weight matrix R by introducing auxiliary variable u∈RC:(17)mine,x,z,u12e1+λv1s.t.e=y-Ax,u=Rx,vi=ui2where v=v1,v2,…,vCT. Denote u~=[u12,u22,…,uC2]T∈RC, and ui=Rixi∈Rni is a subvector of u with elements associated with class i. (17) is a constrained optimization problem which could be solved by the Augmented Lagrangian Multiplier (ALM) method [57, 58]. Its corresponding ALM function is given as (18)Lμe,x,v,u=12e1+λv1+αTy-Ax-e+βTRx-u+γTu~-v+μ2y-Ax-e22+Rx-u22+u~-v22where α∈Rm, β∈Rn, γ∈RC are vectors of Lagrange multipliers and μ>0 is the penalty parameter. Instead of optimizing all arguments simultaneously, as e,x,v,u are separable, we solve them individually and iteratively. In the (k+1)-th iteration, the updating schemes are as follows.Step 1 (updatee).
We updatee by solving the following subproblem with x, z, and u fixed:(19)ek+1=arg⁡mine⁡12e1+αkTy-Axk-e+μk2y-Axk-e22=S1/2μky-Axk+αkμkwhere Sϵ(∙), ϵ>0, is the soft-thresholding (shrinkage) operator defined component-wise as (20)Sϵθi=sign⁡θi∙max⁡θi-ϵ,0Step 2 (updatex).
We updatex by solving the following subproblem with e, v, and u fixed:(21)xk+1=arg⁡minx⁡αkTy-Ax-ek+1+βkTRx-uk+μk2y-Ax-ek+122+Rx-uk22=arg⁡minx⁡μk2xTATA+RTRx-μky-ek+1+αkμkTA+uk-βkμkTRx
This is a convex quadratic problem. Hence the problem reduces to solving the following linear system:(22)μkATA+RTRx=μkATy-ek+1+αkμk+RTuk-βkμk
The close-form solution ofx is obtained as(23)xk+1=ATA+RTR-1ATy-ek+1+αkμk+RTuk-βkμkStep 3 (updatev).
We updatev by solving the following subproblem with x, e, and u fixed:(24)vk+1=arg⁡minv⁡λv1+γkTu~k-v+μk2u~k-v22=arg⁡minv⁡λμkv1+12v-u~k+γkμk22which can also be solved via the soft-thresholding operator as in (20).Step 4 (updateu).
We updateu by solving the following subproblem with e, v, and x fixed:(25)uk+1=arg⁡minu⁡βkTRxk+1-u+γkTu~-vk+1+μk2Rxk+1-u22+u~-vk+122With some manipulation, we have (26)uk+1=arg⁡minu⁡∑i=1Cγki-μkvk+1iui2+μkui-Rixk+1i+βki/μk222which has a closed form solution by one-dimensional shrinkage (or soft-thresholding) formula [59](27)uk+1i=max⁡qki2-γki-μkvk+1i2μk,0qkiqki2.i=1,2…C.where qki≔Rixk+1i+βki/μk/2Step 5.
Update the Lagrangian multipliersα, β, γ and the parameter μ as follows:(28)αk+1=αk+μky-Axk+1-ek+1βk+1=βk+μkRxk+1-uk+1γk+1=γk+μku~k+1-vk+1μk+1=min⁡ρμk,μmaxNote that the subproblems fore, x, u, v are all convex problems. They both have closed-form solutions. For the competence of presentation, the detailed optimization procedure is outlined in Algorithm 1. The convergence property of Algorithm 1 can be guaranteed by the existing ADM theory [56]. From the experimental perspective, the developed optimization algorithm exhibits good convergence property as shown in Figure 3. In our experiments, the iteration number is empirically less than 100 under the given settings. Once the optimal solution x is achieved, the label of test sample y can be obtained as(29)i∗=arg⁡mini⁡y-Aixi1xi1,i=1,2…Cwhere xi is the coding vector associated with training class Ai.Algorithm 1:SolvingCoLR model based on ALM.
Input: training set A, test sample y, diagonal matrix R, parameter λ. 1:Initialize:x0=0, z0=0, u0=0, α0=0, β0=0, γ0=0, μ 0 = 10 - 3, μmax=108, ρ=1.1, ε=10-6. 2: While  not converged  do 3: Fix other variable and update eby solving problem (19). 4: Fix other variable and update x by solving problem (21). 5: Fix other variable and update vby solving problem (24). 6: Fix other variable and update u by solving problem (25). 7: Update the multipliers and parameters by (28) 8: Check the convergence conditions: y-Ax-e∞<ε,Rx-u∞<ε,andv-u~∞<ε 9:End While Output: x, eFigure 3
An illustration for the convergence property of Algorithm1. (a) is the curve of objective function value versus iteration number. (b)-(d) are the curves of 3 convergence conditions in optimization versus iteration number: (b) y-Ax-e∞; (c) v-u~∞; (d) Rx-u∞.
(a) (b) (c) (d)


## 4. Experiments

### 4.1. Parameter Setting

In this section, we will study the key parameters in proposed CoLR model and give some parameter setting suggestions to use it for classification tasks. There are 3 parameters, i.e.,λ, σ1, and σ2 in proposed CoLR model. Among them, λ is used to keep balance between the reconstruction error and the level of weighted class-wise sparseness of the representation coefficient. σ1 and σ2 are the bandwidth parameters in sample and class distance metrics. When λ is larger, fewer training samples would be selected. Empirically speaking, a relatively small λ is preferred to keep the balance between representation residual and the representation coefficient. For CoLR, we search for the optimal λ in the range of 10-6,10-5,10-4,10-3,10-2,10-1,1,10 to obtain the best recognition accuracy. The optimal σ1 and σ2 are searched in the set of 1/32,1/16,1/8,1/4,1/2,1,10. Figure 4 shows recognition accuracy versus different parameters on CMU PIE database [52]. The parameters influence each other. When the value of λ is assigned to be a small positive value, our method could achieve promising performance. Figure 4(a) shows that when the value of parameter λ varies between 10-5 and 10-3, CoLR performs relatively well and the curve of recognition rate tends to be smooth for different values of σ1 and σ2. It can be seen that our method is robust to the values of parameters. When λ is set as 10-4, Figure 4(b) shows that the recognition rates are generally high when the value of σ2 is in the range of 1/4,10. When the value of σ1 is constant, the recognition rate tends to increase as σ2 increases. And when the value of σ2 is constant, the recognition rate tends to increase as σ1 decreases. On the basis of the analysis above, the optimal values of λ, σ1, and σ2 are suggested to be assigned from the range of 10-5,10-3, 1/32,1/8, and 1/4,10, respectively. With these parameters, stable and satisfactory performance of CoLR model can be expected.Figure 4
Face recognition rates of CoLR under different parameter settings on CMU PIE dataset [52]. (a) λ versus σ1 and σ2; (b) σ1 versus σ2 with λ=0.0001.
(a) (b)


### 4.2. Experimental Results on Face Recognition

In this section, our approach is compared with several related state-of-the-art approaches, including nearest neighbor (NN), linear regression classification (LRC) [17], collaborative representation based classification (CRC) [22], sparse representation for classification (SRC) [16], locality constrained linear coding for classification (LLC) [32], weighted group sparse representation classification (WGSRC) [40], class-wise sparse representation (CSR) [31], and discriminative block-diagonal low-rank representation (BDLRR) [50]. Some dictionary learning-based classification methods are also compared, including LC-KSVD [43], LCLE-DL [44], and dictionary pair learning (DPL) [48]. LLC can be seen as an extended version of SRC exploiting data locality instead of sparsity constraint for improved representation coding and adopts a reconstruction-based classification rule. WGSRC classify a test sample by minimizing the weightedl2,1-norm regularized reconstruction error with respect to training images. CSR seeks an optimum representation of the query image by minimizing the class-wise sparsity of the training data. The parameter settings of other methods follow the references. Experiments were conducted on 5 face datasets, including the ORL [3], Extended Yale B [51], CMU PIE [52], and AR [53]. A description of the 4 datasets is shown in Table 2.Table 2
Description of the datasets used in the experiments.
Database # Samples # Dimension # Classes ORL [3] 400 1,024 40  Extended Yale B [51] 2,414 1,024 38  CMU PIE [52] 1,680 1,024 68  AR [53] 1,400 1,260 100


#### 4.2.1. Experiments on the ORL Database

The ORL face database consists of 400 face images from 40 individuals with 10 images per person. The images were taken at different times, lighting variation, facial expressions (open/closed eyes, smiling/not smiling), and facial details (glassed/no glassed) against a dark homogeneous background. In the experiments, each image in ORL database is manually cropped and resized to 32×32. Figure5(a) shows some sample images from one subject. Two experimental settings are considered. Firstly, a random subset withp(= 3, 4, 5, 6) images of each individual is selected for training and the rest for testing. For each experimental scenario, we first apply PCA as preprocessing step to reduce the dimension of original data to 100. We run the programs 10 times and calculate the recognition rates as well as the standard deviations, which are reported in Table 3. Besides, we evaluate the performance of different methods with different feature dimensions, i.e., 30, 60, 90, and 120. The experimental results are listed in Table 4. The best results are highlighted in the italic face font.Table 3
Recognition rates on the ORL dataset with different number of training samples.
Methods # training samples per subject on ORL dataset 3 4 5 6 NN 0.758±0.031 0.819±0.024 0.863±0.014 0.875±0.020  LRC [17] 0.810±0.034 0.885±0.027 0.916±0.020 0.946±0.018  CRC [22] 0.851±0.025 0.916±0.015 0.936±0.011 0.942±0.025  SRC [16] 0.882±0.030 0.933±0.016 0.943±0.011 0.954±0.016  LLC [32] 0.847±0.027 0.893±0.024 0.899±0.020 0.909±0.021  WGSRC [40] 0.881±0.025 0.932±0.020 0.946±0.012 0.958±0.016  LC-KSVD2 [43] 0.857±0.026 0.902±0.015 0.913±0.015 0.932±0.010  LCLE-DL [44] 0.870±0.017 0.913±0.015 0.927±0.020 0.939±0.011  CSR [31] 0.859±0.024 0.924±0.020 0.942±0.011 0.955±0.019  DPL [48] 0.870±0.025 0.912±0.024 0.926±0.022 0.939±0.023  BDLRR [50] 0.879±0.017 0.921±0.021 0.932±0.021 0.949±0.016  CoLR 0.882±0.034 0.940±0.018 0.952±0.011 0.965±0.015Table 4
Recognition rates on the ORL dataset under different feature dimensions.
Methods Feature dimensions (5 samples per subject) 30 60 90 NN 0.845±0.020 0.853±0.040 0.859±0.009  LRC [17] 0.901±0.018 0.911±0.027 0.927±0.018  CRC [22] 0.857±0.021 0.914±0.025 0.939±0.012  SRC [16] 0.898±0.019 0.932±0.018 0.949±0.015  LLC [32] 0.844±0.020 0.885±0.026 0.913±0.020  WGSRC [40] 0.894±0.016 0.932±0.029 0.943±0.015  LC-KSVD2 [43] 0.851±0.026 0.876±0.025 0.924±0.020  LCLE-DL [44] 0.858±0.029 0.876±0.032 0.926±0.020  CSR [31] 0.894±0.011 0.929±0.022 0.943±0.015  DPL [48] 0.896±0.033 0.908±0.041 0.926±0.029  BDLRR [50] 0.917±0.019 0.926±0.026 0.934±0.023  CoLR 0.923±0.015 0.937±0.019 0.956±0.016Figure 5
Some facial images used in our experiments: (a) the ORL dataset; (b) the Extended Yale B dataset; (c) the CMU PIE dataset; (d) the AR dataset.
(a) (b) (c) (d)


#### 4.2.2. Experiments on the Extended Yale B Database

Extended Yale B face database contains about 2414 frontal face images of 38 persons and around 64 near frontal images under different illuminations per person. In this experiment, we simply use the cropped images and resize them to 32×32 pixels. Figure5(b) shows some example images of one subject. A random subset withp (=6, 8, 12, 16) images per individual is taken with labels to form the training set, and the remaining samples are used for testing. The experiment is repeated 10 times, and the comparison results are shown in Table 5. We then evaluate the performance of different algorithms under different feature dimensions with 16 samples per subject as the training set. The FR recognition rates under different dimensions are shown in Table 6, where the best results are highlighted in the italic face font.Table 5
Recognition rates on the Extended Yale B dataset with different number of training samples.
Methods # training samples per subject 6 8 12 16 NN 0.417±0.012 0.482±0.011 0.582±0.014 0.655±0.008  LRC [17] 0.686±0.015 0.774±0.011 0.858±0.006 0.901±0.007  CRC [22] 0.847±0.012 0.892±0.009 0.930±0.005 0.949±0.004  SRC [16] 0.811±0.011 0.863±0.010 0.915±0.009 0.943±0.006  LLC [32] 0.822±0.009 0.871±0.008 0.918±0.006 0.943±0.007  WGSRC [40] 0.820±0.009 0.871±0.010 0.921±0.005 0.946±0.007  LC-KSVD2 [43] 0.796±0.014 0.857±0.009 0.897±0.008 0.922±0.008  LCLE-DL [44] 0.802±0.015 0.855±0.009 0.897±0.010 0.921±0.007  CSR [31] 0.828±0.009 0.877±0.007 0.917±0.008 0.937±0.007  DPL [48] 0.810±0.012 0.857±0.006 0.909±0.008 0.930±0.004  BDLRR [50] 0.792±0.014 0.843±0.007 0.909±0.009 0.935±0.007  CoLR 0.860±0.008 0.906±0.009 0.942±0.005 0.957±0.005Table 6
Recognition rates on the Extended Yale Bdataset under different feature dimensions.
Methods Feature dimensions (16 samples per subject) 50 100 150 NN 0.497±0.009 0.593±0.009 0.624±0.013  LRC [17] 0.866±0.007 0.885±0.009 0.898±0.007  CRC [22] 0.794±0.015 0.893±0.013 0.923±0.007  SRC [16] 0.860±0.008 0.907±0.011 0.923±0.007  LLC [32] 0.741±0.018 0.864±0.013 0.902±0.007  WGSRC [40] 0.822±0.009 0.896±0.013 0.920±0.006  LC-KSVD2 [43] 0.656±0.010 0.865±0.006 0.907±0.007  LCLE-DL [44] 0.671±0.010 0.873±0.006 0.909±0.008  CSR [31] 0.839±0.009 0.900±0.013 0.922±0.008  DPL [48] 0.832±0.015 0.901±0.009 0.921±0.005  BDLRR [50] 0.871±0.011 0.904±0.009 0.930±0.007  CoLR 0.847±0.007 0.908±0.012 0.929±0.006


#### 4.2.3. Experiments on the CMU PIE Database

The CMU PIE database contains over 40, 000 face images of 68 individuals. Images of each individual were acquired across 13 different poses, under 43 different illumination conditions, and with 4 different expressions. Here we use a near frontal pose subset, namely, C07, for experiments, which contains 1629 images of 68 individuals. Each individual has about 24 images. All images are manually cropped and resized to 32×32 pixel in our experiment. Several sample images from the dataset are shown in Figure5(c). A random subset withp (= 2, 4, 6, 8) images of each individual is selected as training dataset, and the rest is used for testing. For each givenp, we independently run all the methods 10 times and report the recognition rates as well as the standard deviations in Table 7. Table 8 lists the recognition rates and corresponding standard deviations of different comparing methods with different dimensions of features. In the experiment scenarios, 6 images were randomly chosen from each subject for training set, and the remaining samples were used for testing. Similarly, PCA is utilized to calculate a low dimensional subspace to reduce the dimensionality of original face data, and the reduced dimensionalities are set as 50, 100, 150, and 200.Table 7
Recognition rates on the CMU PIE dataset with different number of training samples.
Methods # training samples per subject 2 4 6 8 NN 0.363±0.015 0.550±0.013 0.678±0.013 0.759±0.014  LRC [17] 0.513±0.025 0.815±0.013 0.900±0.014 0.925±0.010  CRC [22] 0.810±0.018 0.916±0.007 0.933±0.005 0.939±0.005  SRC [16] 0.782±0.019 0.906±0.009 0.938±0.005 0.948±0.008  LLC [32] 0.789±0.021 0.912±0.008 0.935±0.006 0.945±0.006  WGSRC [40] 0.791±0.019 0.913±0.008 0.939±0.006 0.951±0.007  LC-KSVD2 [43] 0.764±0.022 0.892±0.009 0.921±0.007 0.936±0.007  LCLE-DL [44] 0.773±0.023 0.894±0.008 0.926±0.007 0.937±0.007  CSR [31] 0.806±0.019 0.915±0.008 0.938±0.006 0.946±0.007  DPL [48] 0.760±0.016 0.893±0.009 0.930±0.008 0.941±0.006  BDLRR [50] 0.737±0.015 0.885±0.010 0.932±0.005 0.947±0.008  CoLR 0.820±0.020 0.920±0.007 0.942±0.006 0.947±0.006Table 8
Recognition rates on the CMU PIE dataset under different feature dimensions.
Methods Feature dimensions (6 samples per subject) 50 100 150 200 NN 0.604±0.019 0.649±0.018 0.669±0.016 0.681±0.018  LRC [17] 0.893±0.014 0.899±0.009 0.898±0.009 0.903±0.008  CRC [22] 0.868±0.014 0.925±0.006 0.930±0.007 0.937±0.004  SRC [16] 0.903±0.013 0.930±0.005 0.931±0.006 0.939±0.005  LLC [32] 0.838±0.015 0.917±0.009 0.929±0.006 0.938±0.005  WGSRC [40] 0.892±0.015 0.930±0.006 0.934±0.007 0.942±0.004  LC-KSVD2 [43] 0.828±0.011 0.918±0.010 0.930±0.005 0.934±0.007  LCLE-DL [44] 0.834±0.012 0.922±0.006 0.930±0.008 0.937±0.005  CSR [31] 0.891±0.014 0.916±0.008 0.920±0.008 0.929±0.005  DPL [48] 0.886±0.007 0.922±0.006 0.934±0.009 0.937±0.006  BDLRR [50] 0.907±0.007 0.928±0.005 0.933±0.011 0.941±0.009  CoLR 0.907±0.013 0.931±0.004 0.935±0.007 0.935±0.004


#### 4.2.4. Experiments on the AR Database

In this experiment, a subset of AR database that contains 50 males and 50 females with 6 illumination and 8 expression variations in two sessions is used. 7 images with only illumination and expression changes from session 1 are used as training set, and 7 images from session 2 are used as testing set. Figure5(d) shows some sample images from one person. The recognition rates of different methods on this recognition task are shown in Table 9.Table 9
Recognition rates on the AR dataset under different feature dimensions.
Methods Feature dimensions (7 samples per subject) 50 100 150 200 NN 0.651 0.689 0.694 0.694  LRC [17] 0.683 0.720 0.734 0.736  CRC [22] 0.789 0.876 0.904 0.930  SRC [16] 0.816 0.887 0.899 0.919  LLC [32] 0.784 0.869 0.889 0.906  WGSRC [40] 0.814 0.883 0.903 0.916  LC-KSVD2 [43] 0.669 0.793 0.829 0.846  LCLE-DL [44] 0.694 0.817 0.863 0.883  CSR [31] 0.824 0.884 0.897 0.917  DPL [48] 0.786 0.854 0.877 0.890  BDLRR [50] 0.831 0.881 0.886 0.915  CoLR 0.824 0.897 0.906 0.923


#### 4.2.5. Experiments on the Deep CNN Features

Deep learning with a convolutional neural network has been proved to be very effective in feature extraction and representation of images. In this experiment, we would like to test the performance of different representation-based classifiers on deep CNN features. The structures of CNN for training on the ImageNet with 1000 categories are the same as the proposed CNN in [60]. The basic structure of the adopted is illustrated in Figure 6, which includes 5 convolutional layers and 3 fully connected layers. For further details of the CNN training architecture and features one can refer to [60, 61]. The CNN outputs of the 6-th (f6) and 7-th (f7) fully connected layers are used as inputs of different representation-based classifiers for classification, respectively. We adopt the deep convolutional activated features (DeCAF) from [61] for experiments. The well-trained network parameters shown in Figure 6 are used for deep representation of the 4DA (domain adaptation) dataset. The 4DA dataset includes four domains such as Caltech 256 (C), Amazon (A), Webcam (W), and DSLR (D) sampled from different sources, in which 10 object classes are selected. Example images are shown in Figure 7.Figure 6
Diagram of the training and testing protocol in the experiment.Figure 7
Examples of object images from 4 sources: Amazon (1st row), DSLR (2nd row), Webcam (3rd row), and Caltech (4th row).Single-domain recognition task is considered in this experiment. For each dataset with different CNN-layer features, 20, 8, 8, and 8 samples per class are randomly selected for training from Amazon, DSLR, Webcam, and Caltech domains, respectively, and the remaining are used as test samples for each domain. PCA is further applied to both training and test sets to preserve 99% principle components of data for computational efficiency. The average recognition accuracy for each method is reported. Recognition rates of each method on 4DA datasets usingf6 and f7 CNN-layer features are shown in Tables 10 and 11, respectively.Table 10
Recognition rates of each method on 4DA datasets usingf6 CNN-layer features.
Methods Datasets Amazon DLSR Webcam Caltech NN 0.923±0.012 0.979±0.021 0.972±0.020 0.817±0.014  LRC [17] 0.936±0.009 0.977±0.019 0.977±0.018 0.856±0.010  CRC [22] 0.896±0.021 0.978±0.018 0.976±0.016 0.810±0.038  SRC [16] 0.935±0.006 0.975±0.022 0.977±0.018 0.852±0.012  LLC [32] 0.936±0.007 0.975±0.022 0.978±0.018 0.853±0.013  WGSRC [40] 0.936±0.007 0.975±0.022 0.977±0.018 0.852±0.012  LC-KSVD2 [43] 0.933±0.007 0.975±0.022 0.974±0.013 0.848±0.012  LCLE-DL [44] 0.933±0.006 0.977±0.020 0.975±0.012 0.847±0.012  CSR [31] 0.924±0.006 0.975±0.021 0.972±0.013 0.659±0.012  DPL [48] 0.936±0.007 0.981±0.009 0.978±0.020 0.851±0.020  BDLRR [50] 0.936±0.006 0.973±0.011 0.972±0.023 0.859±0.017  CoLR 0.933±0.009 0.982±0.020 0.979±0.014 0.831±0.014Table 11
Recognition rates of each method on 4DA datasets usingf7 CNN-layer features.
Methods Datasets Amazon DLSR Webcam Caltech NN 0.928±0.011 0.971±0.017 0.964±0.010 0.828±0.012  LRC [17] 0.937±0.011 0.973±0.014 0.976±0.014 0.863±0.008  CRC [22] 0.914±0.011 0.958±0.018 0.972±0.016 0.825±0.011  SRC [16] 0.937±0.010 0.971±0.015 0.975±0.013 0.860±0.008  LLC [32] 0.936±0.010 0.971±0.015 0.974±0.013 0.861±0.009  WGSRC [40] 0.936±0.010 0.971±0.015 0.975±0.013 0.861±0.009  LC-KSVD2 [43] 0.935±0.009 0.978±0.013 0.975±0.014 0.854±0.008  LCLE-DL [44] 0.936±0.009 0.973±0.013 0.973±0.015 0.856±0.006  CSR [31] 0.933±0.009 0.971±0.016 0.974±0.011 0.817±0.013  DPL [48] 0.937±0.008 0.971±0.021 0.964±0.018 0.850±0.019  BDLRR [50] 0.933±0.005 0.969±0.020 0.963±0.017 0.858±0.014  CoLR 0.933±0.011 0.975±0.011 0.976±0.014 0.848±0.006Based on the experimental results above, some conclusions can be reached as follows.( 1 ) In general, across-class representation-based methods such as SRC and CRC perform better than within-class representation-based methods, e.g., NN and LRC. The reason can be attributed to the collaborative representation characteristic of the across-class representation-based methods. This is due to the fact that face recognition is often an undersampled classification problem. As a result, within-class representation-based methods might be incapable of providing enough representation ability for query sample because of limited within-class samples, which will restrict their performance.( 2 ) As for across-class representation-based methods, training samples compete with each other to win their share in representation learning with sparseness constraint, which will make the learned representation more discriminative for classification. As a result, sparseness-based methods tend to outperform nonsparse methods, such as CRC. CRC has closed-form solution with lower computational cost. One possible future research direction is to further improve its discriminant ability for classification with computational advantage inherited.( 3 ) Supervised representation learning methods, such as WGSRC, GSC, and CSR, utilize label information for representation learning, which will make the obtained representation more suitable and discriminative for classification tasks. Comparatively speaking, the proposed CoLR model outperforms comparing methods in most of the cases under considered experimental scenarios. This is partly because CoLR imposes bigger punishment on these training classes and samples that are likely to be far away from the query sample. The method actually adopts a coarse-to-fine strategy to search for the most relevant training samples and classes with query sample and implicitly excludes the nonneighbor training classes and samples. With the strategy, more discriminative classification-oriented and locality adaptive representation can be learnt, which tends to be more efficient and adaptive for classification tasks as the experimental results show. As a classification method, CoLR can be combined with different kinds of feature for classification. The performance of CoLR can be further enhanced using discriminative deep CNN features.( 4 ) The emphasis of this paper is on learning a discriminative representation for classification on given dictionary. Thus, CoLR model directly adopts the original training samples as the dictionary to learn classification-oriented representation. Experimental results show that CoLR could yield excellent performance in comparison with dictionary learning methods, which show that the representation learning strategy of CoLR is efficient in exploring the inherent discrimination information of training samples for classification. Many references have suggested that a dictionary learnt from the original training data seems to be more discriminative and compact than original training data. The experimental results have shown the effectiveness and efficiency of CoLR for image recognition. Future work will consider the possible applications of CoLR by unifying dictionary learning methods.


## 4.1. Parameter Setting

In this section, we will study the key parameters in proposed CoLR model and give some parameter setting suggestions to use it for classification tasks. There are 3 parameters, i.e.,λ, σ1, and σ2 in proposed CoLR model. Among them, λ is used to keep balance between the reconstruction error and the level of weighted class-wise sparseness of the representation coefficient. σ1 and σ2 are the bandwidth parameters in sample and class distance metrics. When λ is larger, fewer training samples would be selected. Empirically speaking, a relatively small λ is preferred to keep the balance between representation residual and the representation coefficient. For CoLR, we search for the optimal λ in the range of 10-6,10-5,10-4,10-3,10-2,10-1,1,10 to obtain the best recognition accuracy. The optimal σ1 and σ2 are searched in the set of 1/32,1/16,1/8,1/4,1/2,1,10. Figure 4 shows recognition accuracy versus different parameters on CMU PIE database [52]. The parameters influence each other. When the value of λ is assigned to be a small positive value, our method could achieve promising performance. Figure 4(a) shows that when the value of parameter λ varies between 10-5 and 10-3, CoLR performs relatively well and the curve of recognition rate tends to be smooth for different values of σ1 and σ2. It can be seen that our method is robust to the values of parameters. When λ is set as 10-4, Figure 4(b) shows that the recognition rates are generally high when the value of σ2 is in the range of 1/4,10. When the value of σ1 is constant, the recognition rate tends to increase as σ2 increases. And when the value of σ2 is constant, the recognition rate tends to increase as σ1 decreases. On the basis of the analysis above, the optimal values of λ, σ1, and σ2 are suggested to be assigned from the range of 10-5,10-3, 1/32,1/8, and 1/4,10, respectively. With these parameters, stable and satisfactory performance of CoLR model can be expected.Figure 4
Face recognition rates of CoLR under different parameter settings on CMU PIE dataset [52]. (a) λ versus σ1 and σ2; (b) σ1 versus σ2 with λ=0.0001.
(a) (b)


## 4.2. Experimental Results on Face Recognition

In this section, our approach is compared with several related state-of-the-art approaches, including nearest neighbor (NN), linear regression classification (LRC) [17], collaborative representation based classification (CRC) [22], sparse representation for classification (SRC) [16], locality constrained linear coding for classification (LLC) [32], weighted group sparse representation classification (WGSRC) [40], class-wise sparse representation (CSR) [31], and discriminative block-diagonal low-rank representation (BDLRR) [50]. Some dictionary learning-based classification methods are also compared, including LC-KSVD [43], LCLE-DL [44], and dictionary pair learning (DPL) [48]. LLC can be seen as an extended version of SRC exploiting data locality instead of sparsity constraint for improved representation coding and adopts a reconstruction-based classification rule. WGSRC classify a test sample by minimizing the weightedl2,1-norm regularized reconstruction error with respect to training images. CSR seeks an optimum representation of the query image by minimizing the class-wise sparsity of the training data. The parameter settings of other methods follow the references. Experiments were conducted on 5 face datasets, including the ORL [3], Extended Yale B [51], CMU PIE [52], and AR [53]. A description of the 4 datasets is shown in Table 2.Table 2
Description of the datasets used in the experiments.
Database # Samples # Dimension # Classes ORL [3] 400 1,024 40  Extended Yale B [51] 2,414 1,024 38  CMU PIE [52] 1,680 1,024 68  AR [53] 1,400 1,260 100


### 4.2.1. Experiments on the ORL Database

The ORL face database consists of 400 face images from 40 individuals with 10 images per person. The images were taken at different times, lighting variation, facial expressions (open/closed eyes, smiling/not smiling), and facial details (glassed/no glassed) against a dark homogeneous background. In the experiments, each image in ORL database is manually cropped and resized to 32×32. Figure5(a) shows some sample images from one subject. Two experimental settings are considered. Firstly, a random subset withp(= 3, 4, 5, 6) images of each individual is selected for training and the rest for testing. For each experimental scenario, we first apply PCA as preprocessing step to reduce the dimension of original data to 100. We run the programs 10 times and calculate the recognition rates as well as the standard deviations, which are reported in Table 3. Besides, we evaluate the performance of different methods with different feature dimensions, i.e., 30, 60, 90, and 120. The experimental results are listed in Table 4. The best results are highlighted in the italic face font.Table 3
Recognition rates on the ORL dataset with different number of training samples.
Methods # training samples per subject on ORL dataset 3 4 5 6 NN 0.758±0.031 0.819±0.024 0.863±0.014 0.875±0.020  LRC [17] 0.810±0.034 0.885±0.027 0.916±0.020 0.946±0.018  CRC [22] 0.851±0.025 0.916±0.015 0.936±0.011 0.942±0.025  SRC [16] 0.882±0.030 0.933±0.016 0.943±0.011 0.954±0.016  LLC [32] 0.847±0.027 0.893±0.024 0.899±0.020 0.909±0.021  WGSRC [40] 0.881±0.025 0.932±0.020 0.946±0.012 0.958±0.016  LC-KSVD2 [43] 0.857±0.026 0.902±0.015 0.913±0.015 0.932±0.010  LCLE-DL [44] 0.870±0.017 0.913±0.015 0.927±0.020 0.939±0.011  CSR [31] 0.859±0.024 0.924±0.020 0.942±0.011 0.955±0.019  DPL [48] 0.870±0.025 0.912±0.024 0.926±0.022 0.939±0.023  BDLRR [50] 0.879±0.017 0.921±0.021 0.932±0.021 0.949±0.016  CoLR 0.882±0.034 0.940±0.018 0.952±0.011 0.965±0.015Table 4
Recognition rates on the ORL dataset under different feature dimensions.
Methods Feature dimensions (5 samples per subject) 30 60 90 NN 0.845±0.020 0.853±0.040 0.859±0.009  LRC [17] 0.901±0.018 0.911±0.027 0.927±0.018  CRC [22] 0.857±0.021 0.914±0.025 0.939±0.012  SRC [16] 0.898±0.019 0.932±0.018 0.949±0.015  LLC [32] 0.844±0.020 0.885±0.026 0.913±0.020  WGSRC [40] 0.894±0.016 0.932±0.029 0.943±0.015  LC-KSVD2 [43] 0.851±0.026 0.876±0.025 0.924±0.020  LCLE-DL [44] 0.858±0.029 0.876±0.032 0.926±0.020  CSR [31] 0.894±0.011 0.929±0.022 0.943±0.015  DPL [48] 0.896±0.033 0.908±0.041 0.926±0.029  BDLRR [50] 0.917±0.019 0.926±0.026 0.934±0.023  CoLR 0.923±0.015 0.937±0.019 0.956±0.016Figure 5
Some facial images used in our experiments: (a) the ORL dataset; (b) the Extended Yale B dataset; (c) the CMU PIE dataset; (d) the AR dataset.
(a) (b) (c) (d)


### 4.2.2. Experiments on the Extended Yale B Database

Extended Yale B face database contains about 2414 frontal face images of 38 persons and around 64 near frontal images under different illuminations per person. In this experiment, we simply use the cropped images and resize them to 32×32 pixels. Figure5(b) shows some example images of one subject. A random subset withp (=6, 8, 12, 16) images per individual is taken with labels to form the training set, and the remaining samples are used for testing. The experiment is repeated 10 times, and the comparison results are shown in Table 5. We then evaluate the performance of different algorithms under different feature dimensions with 16 samples per subject as the training set. The FR recognition rates under different dimensions are shown in Table 6, where the best results are highlighted in the italic face font.Table 5
Recognition rates on the Extended Yale B dataset with different number of training samples.
Methods # training samples per subject 6 8 12 16 NN 0.417±0.012 0.482±0.011 0.582±0.014 0.655±0.008  LRC [17] 0.686±0.015 0.774±0.011 0.858±0.006 0.901±0.007  CRC [22] 0.847±0.012 0.892±0.009 0.930±0.005 0.949±0.004  SRC [16] 0.811±0.011 0.863±0.010 0.915±0.009 0.943±0.006  LLC [32] 0.822±0.009 0.871±0.008 0.918±0.006 0.943±0.007  WGSRC [40] 0.820±0.009 0.871±0.010 0.921±0.005 0.946±0.007  LC-KSVD2 [43] 0.796±0.014 0.857±0.009 0.897±0.008 0.922±0.008  LCLE-DL [44] 0.802±0.015 0.855±0.009 0.897±0.010 0.921±0.007  CSR [31] 0.828±0.009 0.877±0.007 0.917±0.008 0.937±0.007  DPL [48] 0.810±0.012 0.857±0.006 0.909±0.008 0.930±0.004  BDLRR [50] 0.792±0.014 0.843±0.007 0.909±0.009 0.935±0.007  CoLR 0.860±0.008 0.906±0.009 0.942±0.005 0.957±0.005Table 6
Recognition rates on the Extended Yale Bdataset under different feature dimensions.
Methods Feature dimensions (16 samples per subject) 50 100 150 NN 0.497±0.009 0.593±0.009 0.624±0.013  LRC [17] 0.866±0.007 0.885±0.009 0.898±0.007  CRC [22] 0.794±0.015 0.893±0.013 0.923±0.007  SRC [16] 0.860±0.008 0.907±0.011 0.923±0.007  LLC [32] 0.741±0.018 0.864±0.013 0.902±0.007  WGSRC [40] 0.822±0.009 0.896±0.013 0.920±0.006  LC-KSVD2 [43] 0.656±0.010 0.865±0.006 0.907±0.007  LCLE-DL [44] 0.671±0.010 0.873±0.006 0.909±0.008  CSR [31] 0.839±0.009 0.900±0.013 0.922±0.008  DPL [48] 0.832±0.015 0.901±0.009 0.921±0.005  BDLRR [50] 0.871±0.011 0.904±0.009 0.930±0.007  CoLR 0.847±0.007 0.908±0.012 0.929±0.006


### 4.2.3. Experiments on the CMU PIE Database

The CMU PIE database contains over 40, 000 face images of 68 individuals. Images of each individual were acquired across 13 different poses, under 43 different illumination conditions, and with 4 different expressions. Here we use a near frontal pose subset, namely, C07, for experiments, which contains 1629 images of 68 individuals. Each individual has about 24 images. All images are manually cropped and resized to 32×32 pixel in our experiment. Several sample images from the dataset are shown in Figure5(c). A random subset withp (= 2, 4, 6, 8) images of each individual is selected as training dataset, and the rest is used for testing. For each givenp, we independently run all the methods 10 times and report the recognition rates as well as the standard deviations in Table 7. Table 8 lists the recognition rates and corresponding standard deviations of different comparing methods with different dimensions of features. In the experiment scenarios, 6 images were randomly chosen from each subject for training set, and the remaining samples were used for testing. Similarly, PCA is utilized to calculate a low dimensional subspace to reduce the dimensionality of original face data, and the reduced dimensionalities are set as 50, 100, 150, and 200.Table 7
Recognition rates on the CMU PIE dataset with different number of training samples.
Methods # training samples per subject 2 4 6 8 NN 0.363±0.015 0.550±0.013 0.678±0.013 0.759±0.014  LRC [17] 0.513±0.025 0.815±0.013 0.900±0.014 0.925±0.010  CRC [22] 0.810±0.018 0.916±0.007 0.933±0.005 0.939±0.005  SRC [16] 0.782±0.019 0.906±0.009 0.938±0.005 0.948±0.008  LLC [32] 0.789±0.021 0.912±0.008 0.935±0.006 0.945±0.006  WGSRC [40] 0.791±0.019 0.913±0.008 0.939±0.006 0.951±0.007  LC-KSVD2 [43] 0.764±0.022 0.892±0.009 0.921±0.007 0.936±0.007  LCLE-DL [44] 0.773±0.023 0.894±0.008 0.926±0.007 0.937±0.007  CSR [31] 0.806±0.019 0.915±0.008 0.938±0.006 0.946±0.007  DPL [48] 0.760±0.016 0.893±0.009 0.930±0.008 0.941±0.006  BDLRR [50] 0.737±0.015 0.885±0.010 0.932±0.005 0.947±0.008  CoLR 0.820±0.020 0.920±0.007 0.942±0.006 0.947±0.006Table 8
Recognition rates on the CMU PIE dataset under different feature dimensions.
Methods Feature dimensions (6 samples per subject) 50 100 150 200 NN 0.604±0.019 0.649±0.018 0.669±0.016 0.681±0.018  LRC [17] 0.893±0.014 0.899±0.009 0.898±0.009 0.903±0.008  CRC [22] 0.868±0.014 0.925±0.006 0.930±0.007 0.937±0.004  SRC [16] 0.903±0.013 0.930±0.005 0.931±0.006 0.939±0.005  LLC [32] 0.838±0.015 0.917±0.009 0.929±0.006 0.938±0.005  WGSRC [40] 0.892±0.015 0.930±0.006 0.934±0.007 0.942±0.004  LC-KSVD2 [43] 0.828±0.011 0.918±0.010 0.930±0.005 0.934±0.007  LCLE-DL [44] 0.834±0.012 0.922±0.006 0.930±0.008 0.937±0.005  CSR [31] 0.891±0.014 0.916±0.008 0.920±0.008 0.929±0.005  DPL [48] 0.886±0.007 0.922±0.006 0.934±0.009 0.937±0.006  BDLRR [50] 0.907±0.007 0.928±0.005 0.933±0.011 0.941±0.009  CoLR 0.907±0.013 0.931±0.004 0.935±0.007 0.935±0.004


### 4.2.4. Experiments on the AR Database

In this experiment, a subset of AR database that contains 50 males and 50 females with 6 illumination and 8 expression variations in two sessions is used. 7 images with only illumination and expression changes from session 1 are used as training set, and 7 images from session 2 are used as testing set. Figure5(d) shows some sample images from one person. The recognition rates of different methods on this recognition task are shown in Table 9.Table 9
Recognition rates on the AR dataset under different feature dimensions.
Methods Feature dimensions (7 samples per subject) 50 100 150 200 NN 0.651 0.689 0.694 0.694  LRC [17] 0.683 0.720 0.734 0.736  CRC [22] 0.789 0.876 0.904 0.930  SRC [16] 0.816 0.887 0.899 0.919  LLC [32] 0.784 0.869 0.889 0.906  WGSRC [40] 0.814 0.883 0.903 0.916  LC-KSVD2 [43] 0.669 0.793 0.829 0.846  LCLE-DL [44] 0.694 0.817 0.863 0.883  CSR [31] 0.824 0.884 0.897 0.917  DPL [48] 0.786 0.854 0.877 0.890  BDLRR [50] 0.831 0.881 0.886 0.915  CoLR 0.824 0.897 0.906 0.923


### 4.2.5. Experiments on the Deep CNN Features

Deep learning with a convolutional neural network has been proved to be very effective in feature extraction and representation of images. In this experiment, we would like to test the performance of different representation-based classifiers on deep CNN features. The structures of CNN for training on the ImageNet with 1000 categories are the same as the proposed CNN in [60]. The basic structure of the adopted is illustrated in Figure 6, which includes 5 convolutional layers and 3 fully connected layers. For further details of the CNN training architecture and features one can refer to [60, 61]. The CNN outputs of the 6-th (f6) and 7-th (f7) fully connected layers are used as inputs of different representation-based classifiers for classification, respectively. We adopt the deep convolutional activated features (DeCAF) from [61] for experiments. The well-trained network parameters shown in Figure 6 are used for deep representation of the 4DA (domain adaptation) dataset. The 4DA dataset includes four domains such as Caltech 256 (C), Amazon (A), Webcam (W), and DSLR (D) sampled from different sources, in which 10 object classes are selected. Example images are shown in Figure 7.Figure 6
Diagram of the training and testing protocol in the experiment.Figure 7
Examples of object images from 4 sources: Amazon (1st row), DSLR (2nd row), Webcam (3rd row), and Caltech (4th row).Single-domain recognition task is considered in this experiment. For each dataset with different CNN-layer features, 20, 8, 8, and 8 samples per class are randomly selected for training from Amazon, DSLR, Webcam, and Caltech domains, respectively, and the remaining are used as test samples for each domain. PCA is further applied to both training and test sets to preserve 99% principle components of data for computational efficiency. The average recognition accuracy for each method is reported. Recognition rates of each method on 4DA datasets usingf6 and f7 CNN-layer features are shown in Tables 10 and 11, respectively.Table 10
Recognition rates of each method on 4DA datasets usingf6 CNN-layer features.
Methods Datasets Amazon DLSR Webcam Caltech NN 0.923±0.012 0.979±0.021 0.972±0.020 0.817±0.014  LRC [17] 0.936±0.009 0.977±0.019 0.977±0.018 0.856±0.010  CRC [22] 0.896±0.021 0.978±0.018 0.976±0.016 0.810±0.038  SRC [16] 0.935±0.006 0.975±0.022 0.977±0.018 0.852±0.012  LLC [32] 0.936±0.007 0.975±0.022 0.978±0.018 0.853±0.013  WGSRC [40] 0.936±0.007 0.975±0.022 0.977±0.018 0.852±0.012  LC-KSVD2 [43] 0.933±0.007 0.975±0.022 0.974±0.013 0.848±0.012  LCLE-DL [44] 0.933±0.006 0.977±0.020 0.975±0.012 0.847±0.012  CSR [31] 0.924±0.006 0.975±0.021 0.972±0.013 0.659±0.012  DPL [48] 0.936±0.007 0.981±0.009 0.978±0.020 0.851±0.020  BDLRR [50] 0.936±0.006 0.973±0.011 0.972±0.023 0.859±0.017  CoLR 0.933±0.009 0.982±0.020 0.979±0.014 0.831±0.014Table 11
Recognition rates of each method on 4DA datasets usingf7 CNN-layer features.
Methods Datasets Amazon DLSR Webcam Caltech NN 0.928±0.011 0.971±0.017 0.964±0.010 0.828±0.012  LRC [17] 0.937±0.011 0.973±0.014 0.976±0.014 0.863±0.008  CRC [22] 0.914±0.011 0.958±0.018 0.972±0.016 0.825±0.011  SRC [16] 0.937±0.010 0.971±0.015 0.975±0.013 0.860±0.008  LLC [32] 0.936±0.010 0.971±0.015 0.974±0.013 0.861±0.009  WGSRC [40] 0.936±0.010 0.971±0.015 0.975±0.013 0.861±0.009  LC-KSVD2 [43] 0.935±0.009 0.978±0.013 0.975±0.014 0.854±0.008  LCLE-DL [44] 0.936±0.009 0.973±0.013 0.973±0.015 0.856±0.006  CSR [31] 0.933±0.009 0.971±0.016 0.974±0.011 0.817±0.013  DPL [48] 0.937±0.008 0.971±0.021 0.964±0.018 0.850±0.019  BDLRR [50] 0.933±0.005 0.969±0.020 0.963±0.017 0.858±0.014  CoLR 0.933±0.011 0.975±0.011 0.976±0.014 0.848±0.006Based on the experimental results above, some conclusions can be reached as follows.( 1 ) In general, across-class representation-based methods such as SRC and CRC perform better than within-class representation-based methods, e.g., NN and LRC. The reason can be attributed to the collaborative representation characteristic of the across-class representation-based methods. This is due to the fact that face recognition is often an undersampled classification problem. As a result, within-class representation-based methods might be incapable of providing enough representation ability for query sample because of limited within-class samples, which will restrict their performance.( 2 ) As for across-class representation-based methods, training samples compete with each other to win their share in representation learning with sparseness constraint, which will make the learned representation more discriminative for classification. As a result, sparseness-based methods tend to outperform nonsparse methods, such as CRC. CRC has closed-form solution with lower computational cost. One possible future research direction is to further improve its discriminant ability for classification with computational advantage inherited.( 3 ) Supervised representation learning methods, such as WGSRC, GSC, and CSR, utilize label information for representation learning, which will make the obtained representation more suitable and discriminative for classification tasks. Comparatively speaking, the proposed CoLR model outperforms comparing methods in most of the cases under considered experimental scenarios. This is partly because CoLR imposes bigger punishment on these training classes and samples that are likely to be far away from the query sample. The method actually adopts a coarse-to-fine strategy to search for the most relevant training samples and classes with query sample and implicitly excludes the nonneighbor training classes and samples. With the strategy, more discriminative classification-oriented and locality adaptive representation can be learnt, which tends to be more efficient and adaptive for classification tasks as the experimental results show. As a classification method, CoLR can be combined with different kinds of feature for classification. The performance of CoLR can be further enhanced using discriminative deep CNN features.( 4 ) The emphasis of this paper is on learning a discriminative representation for classification on given dictionary. Thus, CoLR model directly adopts the original training samples as the dictionary to learn classification-oriented representation. Experimental results show that CoLR could yield excellent performance in comparison with dictionary learning methods, which show that the representation learning strategy of CoLR is efficient in exploring the inherent discrimination information of training samples for classification. Many references have suggested that a dictionary learnt from the original training data seems to be more discriminative and compact than original training data. The experimental results have shown the effectiveness and efficiency of CoLR for image recognition. Future work will consider the possible applications of CoLR by unifying dictionary learning methods.


## 4.2.1. Experiments on the ORL Database

The ORL face database consists of 400 face images from 40 individuals with 10 images per person. The images were taken at different times, lighting variation, facial expressions (open/closed eyes, smiling/not smiling), and facial details (glassed/no glassed) against a dark homogeneous background. In the experiments, each image in ORL database is manually cropped and resized to 32×32. Figure5(a) shows some sample images from one subject. Two experimental settings are considered. Firstly, a random subset withp(= 3, 4, 5, 6) images of each individual is selected for training and the rest for testing. For each experimental scenario, we first apply PCA as preprocessing step to reduce the dimension of original data to 100. We run the programs 10 times and calculate the recognition rates as well as the standard deviations, which are reported in Table 3. Besides, we evaluate the performance of different methods with different feature dimensions, i.e., 30, 60, 90, and 120. The experimental results are listed in Table 4. The best results are highlighted in the italic face font.Table 3
Recognition rates on the ORL dataset with different number of training samples.
Methods # training samples per subject on ORL dataset 3 4 5 6 NN 0.758±0.031 0.819±0.024 0.863±0.014 0.875±0.020  LRC [17] 0.810±0.034 0.885±0.027 0.916±0.020 0.946±0.018  CRC [22] 0.851±0.025 0.916±0.015 0.936±0.011 0.942±0.025  SRC [16] 0.882±0.030 0.933±0.016 0.943±0.011 0.954±0.016  LLC [32] 0.847±0.027 0.893±0.024 0.899±0.020 0.909±0.021  WGSRC [40] 0.881±0.025 0.932±0.020 0.946±0.012 0.958±0.016  LC-KSVD2 [43] 0.857±0.026 0.902±0.015 0.913±0.015 0.932±0.010  LCLE-DL [44] 0.870±0.017 0.913±0.015 0.927±0.020 0.939±0.011  CSR [31] 0.859±0.024 0.924±0.020 0.942±0.011 0.955±0.019  DPL [48] 0.870±0.025 0.912±0.024 0.926±0.022 0.939±0.023  BDLRR [50] 0.879±0.017 0.921±0.021 0.932±0.021 0.949±0.016  CoLR 0.882±0.034 0.940±0.018 0.952±0.011 0.965±0.015Table 4
Recognition rates on the ORL dataset under different feature dimensions.
Methods Feature dimensions (5 samples per subject) 30 60 90 NN 0.845±0.020 0.853±0.040 0.859±0.009  LRC [17] 0.901±0.018 0.911±0.027 0.927±0.018  CRC [22] 0.857±0.021 0.914±0.025 0.939±0.012  SRC [16] 0.898±0.019 0.932±0.018 0.949±0.015  LLC [32] 0.844±0.020 0.885±0.026 0.913±0.020  WGSRC [40] 0.894±0.016 0.932±0.029 0.943±0.015  LC-KSVD2 [43] 0.851±0.026 0.876±0.025 0.924±0.020  LCLE-DL [44] 0.858±0.029 0.876±0.032 0.926±0.020  CSR [31] 0.894±0.011 0.929±0.022 0.943±0.015  DPL [48] 0.896±0.033 0.908±0.041 0.926±0.029  BDLRR [50] 0.917±0.019 0.926±0.026 0.934±0.023  CoLR 0.923±0.015 0.937±0.019 0.956±0.016Figure 5
Some facial images used in our experiments: (a) the ORL dataset; (b) the Extended Yale B dataset; (c) the CMU PIE dataset; (d) the AR dataset.
(a) (b) (c) (d)


## 4.2.2. Experiments on the Extended Yale B Database

Extended Yale B face database contains about 2414 frontal face images of 38 persons and around 64 near frontal images under different illuminations per person. In this experiment, we simply use the cropped images and resize them to 32×32 pixels. Figure5(b) shows some example images of one subject. A random subset withp (=6, 8, 12, 16) images per individual is taken with labels to form the training set, and the remaining samples are used for testing. The experiment is repeated 10 times, and the comparison results are shown in Table 5. We then evaluate the performance of different algorithms under different feature dimensions with 16 samples per subject as the training set. The FR recognition rates under different dimensions are shown in Table 6, where the best results are highlighted in the italic face font.Table 5
Recognition rates on the Extended Yale B dataset with different number of training samples.
Methods # training samples per subject 6 8 12 16 NN 0.417±0.012 0.482±0.011 0.582±0.014 0.655±0.008  LRC [17] 0.686±0.015 0.774±0.011 0.858±0.006 0.901±0.007  CRC [22] 0.847±0.012 0.892±0.009 0.930±0.005 0.949±0.004  SRC [16] 0.811±0.011 0.863±0.010 0.915±0.009 0.943±0.006  LLC [32] 0.822±0.009 0.871±0.008 0.918±0.006 0.943±0.007  WGSRC [40] 0.820±0.009 0.871±0.010 0.921±0.005 0.946±0.007  LC-KSVD2 [43] 0.796±0.014 0.857±0.009 0.897±0.008 0.922±0.008  LCLE-DL [44] 0.802±0.015 0.855±0.009 0.897±0.010 0.921±0.007  CSR [31] 0.828±0.009 0.877±0.007 0.917±0.008 0.937±0.007  DPL [48] 0.810±0.012 0.857±0.006 0.909±0.008 0.930±0.004  BDLRR [50] 0.792±0.014 0.843±0.007 0.909±0.009 0.935±0.007  CoLR 0.860±0.008 0.906±0.009 0.942±0.005 0.957±0.005Table 6
Recognition rates on the Extended Yale Bdataset under different feature dimensions.
Methods Feature dimensions (16 samples per subject) 50 100 150 NN 0.497±0.009 0.593±0.009 0.624±0.013  LRC [17] 0.866±0.007 0.885±0.009 0.898±0.007  CRC [22] 0.794±0.015 0.893±0.013 0.923±0.007  SRC [16] 0.860±0.008 0.907±0.011 0.923±0.007  LLC [32] 0.741±0.018 0.864±0.013 0.902±0.007  WGSRC [40] 0.822±0.009 0.896±0.013 0.920±0.006  LC-KSVD2 [43] 0.656±0.010 0.865±0.006 0.907±0.007  LCLE-DL [44] 0.671±0.010 0.873±0.006 0.909±0.008  CSR [31] 0.839±0.009 0.900±0.013 0.922±0.008  DPL [48] 0.832±0.015 0.901±0.009 0.921±0.005  BDLRR [50] 0.871±0.011 0.904±0.009 0.930±0.007  CoLR 0.847±0.007 0.908±0.012 0.929±0.006


## 4.2.3. Experiments on the CMU PIE Database

The CMU PIE database contains over 40, 000 face images of 68 individuals. Images of each individual were acquired across 13 different poses, under 43 different illumination conditions, and with 4 different expressions. Here we use a near frontal pose subset, namely, C07, for experiments, which contains 1629 images of 68 individuals. Each individual has about 24 images. All images are manually cropped and resized to 32×32 pixel in our experiment. Several sample images from the dataset are shown in Figure5(c). A random subset withp (= 2, 4, 6, 8) images of each individual is selected as training dataset, and the rest is used for testing. For each givenp, we independently run all the methods 10 times and report the recognition rates as well as the standard deviations in Table 7. Table 8 lists the recognition rates and corresponding standard deviations of different comparing methods with different dimensions of features. In the experiment scenarios, 6 images were randomly chosen from each subject for training set, and the remaining samples were used for testing. Similarly, PCA is utilized to calculate a low dimensional subspace to reduce the dimensionality of original face data, and the reduced dimensionalities are set as 50, 100, 150, and 200.Table 7
Recognition rates on the CMU PIE dataset with different number of training samples.
Methods # training samples per subject 2 4 6 8 NN 0.363±0.015 0.550±0.013 0.678±0.013 0.759±0.014  LRC [17] 0.513±0.025 0.815±0.013 0.900±0.014 0.925±0.010  CRC [22] 0.810±0.018 0.916±0.007 0.933±0.005 0.939±0.005  SRC [16] 0.782±0.019 0.906±0.009 0.938±0.005 0.948±0.008  LLC [32] 0.789±0.021 0.912±0.008 0.935±0.006 0.945±0.006  WGSRC [40] 0.791±0.019 0.913±0.008 0.939±0.006 0.951±0.007  LC-KSVD2 [43] 0.764±0.022 0.892±0.009 0.921±0.007 0.936±0.007  LCLE-DL [44] 0.773±0.023 0.894±0.008 0.926±0.007 0.937±0.007  CSR [31] 0.806±0.019 0.915±0.008 0.938±0.006 0.946±0.007  DPL [48] 0.760±0.016 0.893±0.009 0.930±0.008 0.941±0.006  BDLRR [50] 0.737±0.015 0.885±0.010 0.932±0.005 0.947±0.008  CoLR 0.820±0.020 0.920±0.007 0.942±0.006 0.947±0.006Table 8
Recognition rates on the CMU PIE dataset under different feature dimensions.
Methods Feature dimensions (6 samples per subject) 50 100 150 200 NN 0.604±0.019 0.649±0.018 0.669±0.016 0.681±0.018  LRC [17] 0.893±0.014 0.899±0.009 0.898±0.009 0.903±0.008  CRC [22] 0.868±0.014 0.925±0.006 0.930±0.007 0.937±0.004  SRC [16] 0.903±0.013 0.930±0.005 0.931±0.006 0.939±0.005  LLC [32] 0.838±0.015 0.917±0.009 0.929±0.006 0.938±0.005  WGSRC [40] 0.892±0.015 0.930±0.006 0.934±0.007 0.942±0.004  LC-KSVD2 [43] 0.828±0.011 0.918±0.010 0.930±0.005 0.934±0.007  LCLE-DL [44] 0.834±0.012 0.922±0.006 0.930±0.008 0.937±0.005  CSR [31] 0.891±0.014 0.916±0.008 0.920±0.008 0.929±0.005  DPL [48] 0.886±0.007 0.922±0.006 0.934±0.009 0.937±0.006  BDLRR [50] 0.907±0.007 0.928±0.005 0.933±0.011 0.941±0.009  CoLR 0.907±0.013 0.931±0.004 0.935±0.007 0.935±0.004


## 4.2.4. Experiments on the AR Database

In this experiment, a subset of AR database that contains 50 males and 50 females with 6 illumination and 8 expression variations in two sessions is used. 7 images with only illumination and expression changes from session 1 are used as training set, and 7 images from session 2 are used as testing set. Figure5(d) shows some sample images from one person. The recognition rates of different methods on this recognition task are shown in Table 9.Table 9
Recognition rates on the AR dataset under different feature dimensions.
Methods Feature dimensions (7 samples per subject) 50 100 150 200 NN 0.651 0.689 0.694 0.694  LRC [17] 0.683 0.720 0.734 0.736  CRC [22] 0.789 0.876 0.904 0.930  SRC [16] 0.816 0.887 0.899 0.919  LLC [32] 0.784 0.869 0.889 0.906  WGSRC [40] 0.814 0.883 0.903 0.916  LC-KSVD2 [43] 0.669 0.793 0.829 0.846  LCLE-DL [44] 0.694 0.817 0.863 0.883  CSR [31] 0.824 0.884 0.897 0.917  DPL [48] 0.786 0.854 0.877 0.890  BDLRR [50] 0.831 0.881 0.886 0.915  CoLR 0.824 0.897 0.906 0.923


## 4.2.5. Experiments on the Deep CNN Features

Deep learning with a convolutional neural network has been proved to be very effective in feature extraction and representation of images. In this experiment, we would like to test the performance of different representation-based classifiers on deep CNN features. The structures of CNN for training on the ImageNet with 1000 categories are the same as the proposed CNN in [60]. The basic structure of the adopted is illustrated in Figure 6, which includes 5 convolutional layers and 3 fully connected layers. For further details of the CNN training architecture and features one can refer to [60, 61]. The CNN outputs of the 6-th (f6) and 7-th (f7) fully connected layers are used as inputs of different representation-based classifiers for classification, respectively. We adopt the deep convolutional activated features (DeCAF) from [61] for experiments. The well-trained network parameters shown in Figure 6 are used for deep representation of the 4DA (domain adaptation) dataset. The 4DA dataset includes four domains such as Caltech 256 (C), Amazon (A), Webcam (W), and DSLR (D) sampled from different sources, in which 10 object classes are selected. Example images are shown in Figure 7.Figure 6
Diagram of the training and testing protocol in the experiment.Figure 7
Examples of object images from 4 sources: Amazon (1st row), DSLR (2nd row), Webcam (3rd row), and Caltech (4th row).Single-domain recognition task is considered in this experiment. For each dataset with different CNN-layer features, 20, 8, 8, and 8 samples per class are randomly selected for training from Amazon, DSLR, Webcam, and Caltech domains, respectively, and the remaining are used as test samples for each domain. PCA is further applied to both training and test sets to preserve 99% principle components of data for computational efficiency. The average recognition accuracy for each method is reported. Recognition rates of each method on 4DA datasets usingf6 and f7 CNN-layer features are shown in Tables 10 and 11, respectively.Table 10
Recognition rates of each method on 4DA datasets usingf6 CNN-layer features.
Methods Datasets Amazon DLSR Webcam Caltech NN 0.923±0.012 0.979±0.021 0.972±0.020 0.817±0.014  LRC [17] 0.936±0.009 0.977±0.019 0.977±0.018 0.856±0.010  CRC [22] 0.896±0.021 0.978±0.018 0.976±0.016 0.810±0.038  SRC [16] 0.935±0.006 0.975±0.022 0.977±0.018 0.852±0.012  LLC [32] 0.936±0.007 0.975±0.022 0.978±0.018 0.853±0.013  WGSRC [40] 0.936±0.007 0.975±0.022 0.977±0.018 0.852±0.012  LC-KSVD2 [43] 0.933±0.007 0.975±0.022 0.974±0.013 0.848±0.012  LCLE-DL [44] 0.933±0.006 0.977±0.020 0.975±0.012 0.847±0.012  CSR [31] 0.924±0.006 0.975±0.021 0.972±0.013 0.659±0.012  DPL [48] 0.936±0.007 0.981±0.009 0.978±0.020 0.851±0.020  BDLRR [50] 0.936±0.006 0.973±0.011 0.972±0.023 0.859±0.017  CoLR 0.933±0.009 0.982±0.020 0.979±0.014 0.831±0.014Table 11
Recognition rates of each method on 4DA datasets usingf7 CNN-layer features.
Methods Datasets Amazon DLSR Webcam Caltech NN 0.928±0.011 0.971±0.017 0.964±0.010 0.828±0.012  LRC [17] 0.937±0.011 0.973±0.014 0.976±0.014 0.863±0.008  CRC [22] 0.914±0.011 0.958±0.018 0.972±0.016 0.825±0.011  SRC [16] 0.937±0.010 0.971±0.015 0.975±0.013 0.860±0.008  LLC [32] 0.936±0.010 0.971±0.015 0.974±0.013 0.861±0.009  WGSRC [40] 0.936±0.010 0.971±0.015 0.975±0.013 0.861±0.009  LC-KSVD2 [43] 0.935±0.009 0.978±0.013 0.975±0.014 0.854±0.008  LCLE-DL [44] 0.936±0.009 0.973±0.013 0.973±0.015 0.856±0.006  CSR [31] 0.933±0.009 0.971±0.016 0.974±0.011 0.817±0.013  DPL [48] 0.937±0.008 0.971±0.021 0.964±0.018 0.850±0.019  BDLRR [50] 0.933±0.005 0.969±0.020 0.963±0.017 0.858±0.014  CoLR 0.933±0.011 0.975±0.011 0.976±0.014 0.848±0.006Based on the experimental results above, some conclusions can be reached as follows.( 1 ) In general, across-class representation-based methods such as SRC and CRC perform better than within-class representation-based methods, e.g., NN and LRC. The reason can be attributed to the collaborative representation characteristic of the across-class representation-based methods. This is due to the fact that face recognition is often an undersampled classification problem. As a result, within-class representation-based methods might be incapable of providing enough representation ability for query sample because of limited within-class samples, which will restrict their performance.( 2 ) As for across-class representation-based methods, training samples compete with each other to win their share in representation learning with sparseness constraint, which will make the learned representation more discriminative for classification. As a result, sparseness-based methods tend to outperform nonsparse methods, such as CRC. CRC has closed-form solution with lower computational cost. One possible future research direction is to further improve its discriminant ability for classification with computational advantage inherited.( 3 ) Supervised representation learning methods, such as WGSRC, GSC, and CSR, utilize label information for representation learning, which will make the obtained representation more suitable and discriminative for classification tasks. Comparatively speaking, the proposed CoLR model outperforms comparing methods in most of the cases under considered experimental scenarios. This is partly because CoLR imposes bigger punishment on these training classes and samples that are likely to be far away from the query sample. The method actually adopts a coarse-to-fine strategy to search for the most relevant training samples and classes with query sample and implicitly excludes the nonneighbor training classes and samples. With the strategy, more discriminative classification-oriented and locality adaptive representation can be learnt, which tends to be more efficient and adaptive for classification tasks as the experimental results show. As a classification method, CoLR can be combined with different kinds of feature for classification. The performance of CoLR can be further enhanced using discriminative deep CNN features.( 4 ) The emphasis of this paper is on learning a discriminative representation for classification on given dictionary. Thus, CoLR model directly adopts the original training samples as the dictionary to learn classification-oriented representation. Experimental results show that CoLR could yield excellent performance in comparison with dictionary learning methods, which show that the representation learning strategy of CoLR is efficient in exploring the inherent discrimination information of training samples for classification. Many references have suggested that a dictionary learnt from the original training data seems to be more discriminative and compact than original training data. The experimental results have shown the effectiveness and efficiency of CoLR for image recognition. Future work will consider the possible applications of CoLR by unifying dictionary learning methods.


## 5. Conclusions

In this paper, we have proposed a novel CoLR model for image recognition. Specifically, an informative and discriminative classification-oriented local representation could be learned in terms ofl1-norm loss function by taking both the weighted class-wise sparseness and data locality within each class into consideration. The developed representation strategy can encourage classification steered representation and boost locality sensitivity within the selected training classes highlighting the test sample’s most relevant training classes and samples. Also, an efficient optimization algorithm is devised to solve CoLR model based on a variable splitting strategy and the ALM scheme. Experimental results on several face databases and the deep CNN features show that CoLR can yield promising performance by comparing with many representative models.



---

*Source: 7835797-2019-06-20.xml*