Low‐rank constrained weighted discriminative regression for multi‐view feature learning

CAAI Transactions on Intelligence TechnologyVolume 6, Issue 4 p. 471-479 ORIGINAL RESEARCH PAPEROpen Access Low-rank constrained weighted discriminative regression for multi-view feature learning Chao Zhang, Chao Zhang Department of Control and Systems Engineering, Nanjing University, Nanjing, 210093 ChinaSearch for more papers by this authorHuaxiong Li, Corresponding Author Huaxiong Li [email protected] Department of Control and Systems Engineering, Nanjing University, Nanjing, 210093 China Correspondence Huaxiong Li, Department of Control and Systems Engineering, Nanjing University, Nanjing, 210093, China. Email: [email protected]Search for more papers by this author Chao Zhang, Chao Zhang Department of Control and Systems Engineering, Nanjing University, Nanjing, 210093 ChinaSearch for more papers by this authorHuaxiong Li, Corresponding Author Huaxiong Li [email protected] Department of Control and Systems Engineering, Nanjing University, Nanjing, 210093 China Correspondence Huaxiong Li, Department of Control and Systems Engineering, Nanjing University, Nanjing, 210093, China. Email: [email protected]Search for more papers by this author First published: 15 March 2021 https://doi.org/10.1049/cit2.12018AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Abstract In recent years, multi-view learning has attracted much attention in the fields of data mining, knowledge discovery and machine learning, and been widely used in classification, clustering and information retrieval, and so forth. A new supervised feature learning method for multi-view data, called low-rank constrained weighted discriminative regression (LWDR), is proposed. Different from previous methods handling each view separately, LWDR learns a discriminative projection matrix by fully exploiting the complementary information among all views from a unified perspective. Based on least squares regression model, the high-dimensional multi-view data is mapped into a common subspace, in which different views have different weights in projection. The weights are adaptively updated to estimate the roles of all views. To improve the intra-class similarity of learned features, a low-rank constraint is designed and imposed on the multi-view features of each class, which improves the feature discrimination. An iterative optimization algorithm is designed to solve the LWDR model efficiently. Experiments on four popular datasets, including Handwritten, Caltech101, PIE and AwA, demonstrate the effectiveness of the proposed method. 1 INTRODUCTION Owing to the rapid growth of multimedia data, multi-view learning aroused much interests of researchers from data mining, knowledge discovery and machine learning areas in recent years [1-8]. In real world, one object can be described by different kinds of data or from different views. For example, a news can be expressed by texts, audios and videos. One person can be identified by the face, fingerprint and DNA information. Although these features may be heterogeneous and very different, they naturally reflect some inherent structures or characteristics of the object. Compared with single view data, multi-view data contains more underlying information. How to effectively and efficiently exploit the correlative yet complementary information among diverse views is an important research topic for multi-view learning [9]. Many researchers tried to develop effective information fusion techniques for multi-view learning, including unsupervised [9], semi-supervised [10] and supervised [11, 12]. Some researchers proposed multi-view learning methods by co-training [13, 14] and co-regularization [15, 16]. However, these methods neglect the problem caused by the high dimension of multi-view data. Canonical correlation analysis (CCA) is a classical unsupervised multi-view subspace learning method, which seeks a low dimensional feature space by maximizing the correlation between different views [9]. However, CCA can only deal with two views which limits its further application on more complex data. Luo et al. proposed a tensor CCA method, which extended original CCA for multiple views [17]. To make use of the label information for better classification performance, Kan et al. proposed a multi-view discriminant analysis (MvDA) method by maximizing the inter-class distance and minimizing the intra-class distance from both inter-view and intra-view [11]. MvDA can be regarded as the extension of linear discriminant analysis (LDA) on multi-view data. These methods mentioned above can be categorized into subspace learning based methods, which assume that different views can be generated from a common latent feature subspace. Regression-based method is one of the most popular methods in machine learning [18-22], and it provides another effective and efficient way for multi-view feature learning [23]. Specifically, regression-based methods seek a linear mapping by transforming data to fit the label matrix. Zheng et al. extended low-rank regression model for single view data to multi-view fully low-rank regression (FLR), in which multiple projection matrices were learned and the final classification was performed by majority voting [23]. However, FLR does not consider the differences among all views. Yang et al. proposed an adaptive-weighting discriminative regression (ADR) by learning a unified transform matrix for multi-view classification [24]. ADR introduces an adjustment matrix to enlarge the distance between different classes which improves the model robustness to some extent. However, the intra-class compactness is destroyed in ADR, which is also important for pattern analysis. In [25], the authors incorporated feature selection into linear regression model by l2,1-norm regularization. Wen et al. adopted a graph-regularized matrix factorization model to handle the multi-view clustering problem with incomplete views [7]. In this article, we propose a multi-view low-rank weighted discriminative regression (LWDR) method for feature learning. LWDR learns a common feature subspace across all views. Adaptive weights learning mechanism is adopted to automatically learn different views and the important views containing more discriminative information is enforced to contribute more to subspace learning. To improve the intra-class similarity, a class-wise low-rank constraint is imposed on the multi-view features. Besides, a flexible error term based on l2,1-norm is introduced to relax the label matrix. Experiments on four datasets demonstrate that LWDR outperforms previous single view feature learning methods and related multi-view learning methods. The rest of this paperarticle is organized as follows. In Section 2, we briefly review the related works about linear regression methods for multi-view learning. Section 3 introduces the formulation of our proposed method and the optimization algorithm in detail. Section 4 reports the experimental results and analysis. Section 5 concludes this article. 2 RELATED WORKS For convenience, we first present the main notations used in this paper. matrices and vectors are written in boldface uppercase and boldface lowercase, respectively. { X k } k = 1 v denotes a multi-view dataset with v views. X k ∈ R m k × n is the data matrix of the k-th view, where mk is the feature dimensionality of view k and n is the number of training samples. c is the number of classes. Y ∈ {0,1}c×n is the label matrix, and Yij = 1 if the j-th training sample belongs to class i and otherwise 0. W k ∈ R m k × c is the projection matrix of the k-th view. For matrix A = ( A i j ) ∈ R p × q , its l2,1-norm, Frobenius-norm, and nuclear norm are defined as ‖ A ‖ 2 , 1 = ∑ i = 1 p ∑ j = 1 q A i j 2 , ‖ A ‖ F = ∑ i = 1 p ∑ j = 1 q A i j 2 , ‖A‖* = ∑i=1δi(A), respectively, where δi(A) is the i-th singular value of A. I is an identity matrix and 1n is an n-dimensional vector with all elements being 1. ⊙ is the Hadamard multiplication. For single view data X ∈ R m × n and Y, the basic linear regression solves the following problem to find the optimal projection matrix W. min W 1 2 ‖ W T X − Y ‖ F 2 + α ϕ ( W ) , (1)where α > 0 is a balance parameter and ϕ(W) is the regularizer. l2,1-norm, Frobenius-norm and nuclear norm are usually used in ϕ(W) for learned features with different properties. When ϕ ( W ) = ‖ W ‖ F 2 , Equation (1) has a closed-form solution W * = ( X X T + 2 α I ) − 1 X Y T , which is very efficient in real application. However, Equation (1) cannot be directly applied on multi-view data. Some researchers extended it for multi-view data by learning multiple projection matrices as follows [23]: min W 1 2 ‖ ∑ k = 1 v W k T X k − Y ‖ F 2 + α ∑ k = 1 v ϕ ( W k ) , (2)where Wk is the projection matrix of the k-th view. It can be observed that Equation (2) is equivalent to simply concatenating multi-view features into one single view feature and performing traditional linear regression on it. Such operation is not physically meaningful and treats all views equally. However, different views usually have different characteristics and contribute differently to pattern analysis. For example, to identify a person, the face appearance is more important than voice. Thus, adaptive weights learning strategy is introduced into multi-view linear regression model by weighting different views, which can be generally described as follows. min W 1 2 ‖ ∑ k = 1 v δ k W k T X k − Y ‖ F 2 + α ∑ k = 1 v ϕ ( W k ) s.t. δ k > 0 , ∑ k = 1 v δ k = 1 . (3)where δk is the weight of the k-th view. The first term and constraints force the model to learn optimal weight for each view automatically. However, in Equation (3), the latent multi-view features are directly regressed to approximate the binary label matrix Y, which may be too strict and is not appropriate as the regression target. To improve the model robustness, in [24], the authors relaxed the label matrix and proposed the following auto-weighted discriminative regression model for multi-view classification (ADR): min 1 2 ‖ ∑ k = 1 v δ k W k T X k − b 1 n T − ( Y + Y ⊙ M ) ‖ F 2 + α ∑ k = 1 v ‖ W k ‖ F 2 , s.t. M ≥ 0 , δ k > 0 , ∑ k = 1 v δ k = 1 . (4) where b ∈ R c is a bias vector and M is a non-negative matrix. By introducing the adjustment matrix M, the one-zero label vector y = [1,0,…,0]T is relaxed to [ 1 + M i j , 0 , … , 0 ] T ( M i j ≥ 0 ) , where Mij is the corresponding element in M. By introducing non-negative matrix M, Equation (4) uses the ϵ-draggings technique to enlarge the distances between true and false classes and improve the model robustness. Such label relaxation strategy is widely used in other regression-based methods [26, 27]. 3 PROPOSED METHOD In this section, we present our proposed LWDR method and the optimization algorithm in detail, then we will make some discussions on the computational complexity of LWDR. Formulation From Section 2, although the ϵ-dragging technique used in ADR [24] enlarges the distance between different classes or inter-class separability to some extent, the same class may have different labels after relaxation (i.e. Y + Y ⊙M) because of the dynamic of M. Thus, the intra-class similarity cannot be guaranteed in ADR and two samples from same class may be projected distant from each other. Both inter-class separability and intra-class similarity are important for good classification performance. In low-rank representation (LRR) [28-30], a set of instances are generally drawn from a union of multiple subspaces, and the instances of each subspace are regarded from the same class, which illustrates that the instances from the same class should locate in the same subspace and the data matrix of each class should be low-rank. Inspired by this issue, we propose the following low-rank constrained adaptive weighted discriminative regression (LWDR) model to improve the intra-class similarity of learned multi-view features: min 1 2 ‖ ∑ k = 1 v δ k W k T X k − H ‖ F 2 + 1 2 ‖ H + E − Y ‖ F 2 + α ∑ i = 1 c ‖ H i ‖ * + β ∑ k = 1 v ‖ W k ‖ F 2 + γ ‖ E ‖ 2 , 1 , s.t. δ k > 0 , ∑ k = 1 v δ k = 1 . (5)where α, β and γ are balance parameters, E is an error matrix, H is the to-be-learned feature matrix, and Hi denotes the features of the i-th class. The first term in Equation (5) learns the multi-view features approximated by matrix H. The second term utilizes the label matrix to supervise the feature learning. The third term ∑ i = i c ‖ H i ‖ * forces the features of same class to be similar, which are considered to be low-rank. The last two terms are regularizers avoiding overfitting. Error matrix E is constrained by l2,1-norm to compensate the regression errors and it is flexible in learning transform matrices { W k } k = 1 v . The overall framework of our LWDR is shown in Figure 1. Equation (5) involves multiple variables and cannot be solved directly. In the next section, we will give an iterative algorithm to solve it efficiently. FIGURE 1Open in figure viewerPowerPoint The overall framework of LWDR model for multi-view feature learning. LWDR adaptively learns a projection matrix and a weight for each view. Then the multi-view features H can be obtained by integrating the single-view features using learned weights. Label matrix Y is used as regression target to guide the feature learning in a supervised way, and a sparse error matrix E is introduced to compensate the regression error. To improve the intra-class compactness of multi-view features, a class-wise low-rank constraint on multi-view features is incorporated into LWDR Optimization To make the variables separable in Equation (5), we introduce an auxiliary variable T and rewrite the original problem as follows: min 1 2 ‖ ∑ k = 1 v W ˜ k T X k − H ‖ F 2 + 1 2 ‖ H + E − Y ‖ F 2 + α ∑ i = 1 c ‖ T i ‖ * + β ∑ k = 1 v 1 δ k ‖ W ˜ k ‖ F 2 + γ ‖ E ‖ 2 , 1 , s.t. T = H , δ k > 0 , ∑ k = 1 v δ k = 1 , (6) where W ˜ k = δ k W k . By some simple manipulation, the above problem can be equivalently expressed as: min 1 2 ‖ W ˜ T X − H ‖ F 2 + 1 2 ‖ H + E − Y ‖ F 2 + α ∑ i = 1 c ‖ T i ‖ * + β ∑ k = 1 v 1 δ k ‖ W ˜ k ‖ F 2 + γ ‖ E ‖ 2 , 1 , s.t. T = H , δ k > 0 , ∑ k = 1 v δ k = 1 . (7) where W ˜ = [ W ˜ 1 ; W ˜ 2 ; … ; W ˜ v ] , X = [ X 1 ; X 2 ; … ; X v ] . To solve, Equation (7) is equivalent to minimize its augmented Lagrange function L θ defined as: L θ = 1 2 ‖ W ˜ T X − H ‖ F 2 + 1 2 ‖ H + E − Y ‖ F 2 + α ∑ i = 1 c ‖ T i ‖ * + β ∑ k = 1 v 1 δ k ‖ W ˜ k ‖ F 2 + γ ‖ E ‖ 2 , 1 , + θ 2 ‖ T − H + 1 θ Z ‖ F 2 , (8) where θ > 0 is a penalty factor and Z is the augmented Lagrange multiplier. We adopt the iterative strategy to minimize it, in which a sequence of sub-problems with respect to each unknown variable are solved respectively [31-33]. In specific, it contains following six steps in each iteration. (1) Update W ˜ : We first reorganize { δ k } k = 1 v as δ and rewrite the sub-problem w.r.t W ˜ as follows: min W ˜ 1 2 ‖ W ˜ T X − H ‖ F 2 + β δ ‖ W ˜ ‖ F 2 . (9) This is a typical least squares regression problem. By setting its derivative w.r.t. W ˜ to zero, we can obtain its optimal solution: W ˜ * = ( 2 β I / δ + X X T ) − 1 X H T . (10) Since δ is a set of { δ k } k = 1 v , the final solution is: W ˜ * = ( 2 β Δ + X X T ) − 1 X H T , (11) where Δ = diag(1/δ1, …, 1/δ1, 1/δ2, …, 1/δv). (2) Update H by solving the following minimization problem: min H 1 2 ‖ W ˜ T X − H ‖ F 2 + 1 2 ‖ H + E − Y ‖ F 2 + θ 2 ‖ T − H + 1 θ Z ‖ F 2 , (12) With the same strategy in optimizing W ˜ and we can obtain the solution to Equation (12): H * = ( ( 2 + θ ) I ) − 1 ( W ˜ T X − E + Y + θ T + Z ) . (13) (3) Update E by solving the following problem: min E γ ‖ E ‖ 2 , 1 + 1 2 ‖ E + H − Y ‖ F 2 . (14) Equation (14) can be solved by the following theorem. Theorem 1.[34] Given Q ∈ R p × q , the optimal solution A* of min A λ ‖ A ‖ 2 , 1 + 1 2 ‖ A − Q ‖ F 2 , is given by Gλ(Q), and Gλ(Q) is the following operator: A i , : * = G λ ( Q i , : ) = ‖ Q i , : ‖ 2 − λ ‖ Q i , : ‖ 2 Q i , : , if ‖ Q i , : ‖ 2 > λ 0 , otherwise. where Qi is the i-th row of matrix Q. According to Theorem 1, we can get the solution to Equation (14): E * = G γ ( Y − H ) . (15) (4) Update T with other variables fixed by solving the following optimization problem: min T α ∑ i = 1 c ‖ T i ‖ * + θ 2 ‖ T − H + 1 θ Z ‖ F 2 . (16) Algorithm 1. Iterative algorithm for solving LWDR Input: Multi-view data { X k } k = 1 v , parameters α, β, γ. Output: Projection matrix W ˜ . 1: Initialization: H = Y, E = W = Z = 0, θ = 10−3, θmax = 105, ϵ = 10−6. 2: t = 0. 3: while not converged do. 4: Update W ˜ by Equation (11); 5: Update H by Equation (13); 6: Update E by Equation (15); 7: Update Ti by Equation (18) and update T = [T1, …, Tc]; 8: Update { δ k } k = 1 v by Equation (21); 9: Update Z and θ by Equation (22); 10: Check convergence conditions: ‖ W t + 1 − W t ‖ F 2 + ‖ E t + 1 − E t ‖ F 2 + ‖ H t + 1 − T t + 1 ‖ F 2 < ϵ; 11: t ← t + 1; 12: end While It can be transformedto solve each Ti respectively. min T i ∑ i = 1 c α ‖ T i ‖ * + θ 2 ‖ T i − H i + 1 θ Z i ‖ F 2 . (17) The optimal T i * can be obtained by the following theorem. Theorem 2.[35] Given Q ∈ R p × q with rank r and λ > 0, the optimal solution A* of min A λ ‖ A ‖ * + 1 2 ‖ A − Q ‖ F 2 , is given by Sλ(Q), and Sλ(Q) is the following operator: A * = S λ ( Q ) = U p × r diag ( { max ( 0 , δ i ( Q ) − λ ) } 1 ≤ i ≤ r ) V q × r T , where the singular value decomposition (SVD) of Q is Q = UΣVT, Σ = diag(δ1(Q), δ2(Q), …, δr(Q)). According to Theorem 2, we have T i * = S α / θ ( H i − Z i / θ ) , (18)where Sα/θ(⋅) is the soft thresholding operator. Thus we have the optimal T * = [ T 1 * , T 2 * , … , T c * ] . (5) Fix others and update { δ k } k = 1 v : min δ k ∑ k = 1 v 1 δ k ‖ W ˜ k ‖ F 2 s . t . δ k > 0 , ∑ k = 1 v δ k = 1 . (19) According to the Cauchy inequality theory [24], it holds ∑ k = 1 v 1 δ k ‖ W ˜ k ‖ F 2 ≥ ∑ k = 1 v ‖ W ˜ k ‖ F 2 , (20) The "=" in Equation (20) is satisfied if and only if δ k = ‖ W ˜ k ‖ F ∑ k = 1 v ‖ W ˜ k ‖ F , (21) and Equation (19) gets minimum when δk takes the values of Equation (21). (6) Fix others and update Z, θ: Z = Z + θ ( T − H ) , θ = min ( θ max , ρ θ ) , (22) In this article, θmax = 106 and ρ = 1.1. By performing steps (1)–(6) iteratively, the original loss function can be minimized until convergence or reaching the maximum iterations. Algorithm 1 summarizes the iterative algorithm for solving LWDR model in detail. Computational complexity analysis From Algorithm 1, the main computational time is consumed on the calculation inside the iterations. We can observe that steps (1) and (2) contain matrix inverse operation, and step (4) contains SVD operation. The operations in other steps are addition and multiplication of matrices, which have much lower complexity than matrix inverse and SVD. The time consumption for step (1) is O(n3). In step (2), (2 + θ)I is a diagonal matrix and ( ( 2 + θ ) I ) − 1 = 1 2 + θ I. Then the matrix inverse operation can be simplified to matrix multiplication. Step (4) needs total c SVD operations and its time complexity is O(c min(ca2, c2a)) where a is the average number of samples per class. Thus, the total computational complexity of Algorithm 1 is O(t(n3 + c min(ca2, c2a))) if there are t iterations. Connections to other methods Connections to FLR [23]: Similar to our proposed LWDR, FLR also adopts regression model for multi-view feature learning. FLR learns a projection matrix for each view with low-rank constraint, which is helpful to explore the low-rank structure of data. However, FLR treats all views equally and ignores the fact that different views have different roles for pattern analysis, which may degrade the classification performance. Differently, our proposed LWDR adaptively learns the weights of all views and enforces those informative views to contribute more to feature learning. Thus, the proposed method can achieve better performance than FLR, which will be demonstrated in experiments. Connections to ADR [24]: ADR learns a weighted multi-view regression model for multi-view feature learning, which considers the different weights of different views. To avoid a rigid regression target, ADR utilizes the relaxed label matrix for regression as presented in Equation (4), which is beneficial to enlarge the margins of samples from different classes. However, according to [34], the margins of samples from the same class may be also enlarged, and the discriminative power of projection matrix will be compromised. To address this problem, LWDR introduces the class-wise low-rank constraint, which enforces the transformed samples of the same class to have the same structure. In this way, the margins of the transformed samples from the same class will be reduced and the intra-class compactness can be improved. Therefore, the proposed method has the potential to perform better than ADR. 4 EXPERIMENTS In this section, we conduct the experiments on Handwritten [36], Caltech101 [37], PIE [38] and AwA [39] datasets to validate the effectiveness of proposed LWDR, compared with single view and related multi-view learning methods. Datasets Handwritten dataset contains total 2000 images about 10 number (i.e. 0–9) with 200 images per subject. It contains six feature views, including Pixel Averages (PIX), Fourier Coefficients (FOU), Profile Correlations (FAC), Zernike Moment (ZER), Karhunen-Loeve Coefficients (KAR) and Morphological (MOR) features. Caltech101 dataset contains 9146 images of 101 different subjects. In our experiments, total 2386 images of 20 classes are used. These images have six feature views, including Gabor, Wavelet Moments (WM), CENTRIST, HOG, GIST and LBP features. The whole PIE dataset consists of over 40,000 face images of 68 individuals, collected under different pose, illumination and expression conditions. Total 1360 images of 68 individuals are used in our experiments, which contains five different facial poses. The five poses are used as five different views. AwA dataset contains 30,475 images of 50 kinds of animals. In our experiment, 4000 images with 80 images per class are used. AwA dataset contains six feature views, including Global Color Histogram (GCH), Local Self-Similarity (LSS), Pyramid HOG (PHOG), RGSIFT, SIFT and SURF features. Figure 2 shows some samples images used in our experiments. Table 1 presents the views, feature dimensions, number of classes and samples of these datasets. FIGURE 2Open in figure viewerPowerPoint Some typical images from Handwritten, PIE and Caltech101 datasets from top to bottom (images of AwA dataset are unavailable due to the copyright problem) TABLE 1. Descriptions of four datasets used in our experiments View Handwritten Caltech101 PIE AwA V1 PIX(240) Gabor(48) P5(1024) GCH(2688) V1 FOU(76) WM(40) P7(1024) LSS(2000) V3 FAC(216) CENTRIST(254) P9(1024) PHOG(252) V4 ZER(47) HOG(1984) P27(1024) RGSIFT(2000) V5 KAR(64) GIST(512) P29(1024) SIFT(2000) V6 MOR(6) LBP(928) / SURF(2000) #classes 10 20 68 50 #samples 2000 2386 1360 4000 Experimental setup For Handwritten and Caltech101 dataset, we randomly select 10 images per subject for training and the rest for testing. For PIE, five face images per person are randomly chosen for training. For AwA, the training size of each class is 20. We compare our proposed LWDR with single view and multi-view approaches. For single view method, each view is handled separately. LDA is performed on each view for feature learning [40]. The reduced dimension of LDA is c − 1. For multi-view methods, all views are simply concatenated and LDA is used for feature extraction, that is, LDA (all). Also, we concatenate the top three views and then perform LDA, that is, LDA (top three). The top three views are selected based on the performance of single view method. Multi-view methods FLR [23] and ADR [24] are performed for comparison. The parameter settings of FLR and ADR are followed the author's suggestions. Nearest neighbour with cosine distance is used for classification. We repeat the experiments 20 times by randomly sampling data partitions and report the experimental results by the mean recognition rate with standard deviation. Experimental results 4.3.1 Classification accuracy comparison Table 2 lists the classification accuracies of single view and multi-view methods on Handwritten, Caltech101, PIE and AwA datasets. As can be clearly seen, our LDWR achieves the best performance on four datasets. SVi (i = 1, 2, …, 6) denotes the single view method which performs LDA on the i-th view. The performance of SV varies significantly, which indicates that these views have different roles for classification. The simple concatenation of all views makes effects to improve the performance and LDA (all) is superior to all SV methods. LDA (top three) can generally produce better performance the LDA (all) except Caltech101 dataset. FLR learns a projection matrix for each view and adopts majority voting for classification, which obtain better performance than simple concatenation. ADR uses adaptive weight learning strategy and ϵ-dragging technique, and it performs better than FLR. However, our proposed LWDR still outperforms ADR on multi-view feature learning. For FLR, ADR and LWDR, the tops three views of four datasets are also tested. We can observe that by using the most informative top three views, these methods can generally better performance. TABLE 2. Classification accuracies of single view and multi-view feature learning methods on Handwritten, Caltech101, PIE and AwA datasets with nearest neighbour classifier. The best results are in bold Methods Handwritten Caltech101 PIE AwA SV1 0.8247 ± 0.0296 0.5259 ± 0.0461 0.8840 ± 0.0162 0.0429 ± 0.0033 SV2 0.4082 ± 0.0459 0.4468 ± 0.0462 0.8875 ± 0.0127 0.0522 ± 0.0039 SV3 0.8749 ± 0.0242 0.3345 ± 0.0342 0.8820 ± 0.0138 0.0387 ± 0.0038 SV4 0.6013 ± 0.0407 0.7974 ± 0.0226 0.8745 ± 0.0121 0.0747 ± 0.0049 SV5 0.7341 ± 0.0362 0.7630 ± 0.0183 0.8631 ± 0.0141 0.0429 ± 0.0046 SV6 0.6199 ± 0.0184 0.8149 ± 0.0186 / 0.0799 ± 0.0042 LDA (all) 0.9122 ± 0.0087 0.9016 ± 0.0069 0.9188 ± 0.0077 0.1459 ± 0.0066 LDA (top 3) 0.9130 ± 0.0147 0.8979 ± 0.0177 0.9217 ± 0.0107 0.1493 ± 0.0053 FLR 0.9344 ± 0.0086 0.9188 ± 0.0096 0.9203 ± 0.0089 0.1607 ± 0.0073 FLR (top 3) 0.9395 ± 0.0065 0.9223 ± 0.0106 0.9314 ± 0.0145 0.1647 ± 0.0082 ADR 0.9393 ± 0.0070 0.9242 ± 0.0077 0.9258 ± 0.0086 0.1733 ± 0.0037 ADR (top 3) 0.9442 ± 0.0055 0.9398 ± 0.0162 0.9343 ± 0.0097 0.1809 ± 0.0065 LWDR 0.9415 ± 0.0088 0.9479 ± 0.0068 0.9335 ± 0.0054 0.1797 ± 0.0089 LWDR (top 3) 0.9517 ± 0.0052 0.9498 ± 0.0061 0.9402 ± 0.0074 0.1833 ± 0.0063 4.3.2 Adaptive weigh