# Low Rank Correlation Representation and Clustering

Authors: Wenyun Gao; Sheng Dai; Stanley Ebhohimhen Abhadiomhen; Wei He; Xinghui Yin
Journal: Scientific Programming
 (2021)

---

## Abstract

Correlation learning is a technique utilized to find a common representation in cross-domain and multiview datasets. However, most existing methods are not robust enough to handle noisy data. As such, the common representation matrix learned could be influenced easily by noisy samples inherent in different instances of the data. In this paper, we propose a novel correlation learning method based on a low-rank representation, which learns a common representation between two instances of data in a latent subspace. Specifically, we begin by learning a low-rank representation matrix and an orthogonal rotation matrix to handle the noisy samples in one instance of the data so that a second instance of the data can linearly reconstruct the low-rank representation. Our method then finds a similarity matrix that approximates the common low-rank representation matrix much better such that a rank constraint on the Laplacian matrix would reveal the clustering structure explicitly without any spectral postprocessing. Extensive experimental results on ORL, Yale, Coil-20, Caltech 101-20, and UCI digits datasets demonstrate that our method has superior performance than other state-of-the-art compared methods in six evaluation metrics.


---

## Body

## 1. Introduction

Cross-domain and multiview datasets can help to improve performance accuracy in a typical clustering problem, such as face image clustering [1–5] and object clustering [6–8]. The reason is that different instances of cross-domain and multiview datasets often share a common property that is useful in the overall clustering task [9, 10]. As such, several correlation learning techniques, including canonical correlation analysis (CCA) [11–15], cotraining [15–17], nonnegative matrix factorization [18, 19], and subspace clustering [20–26] were utilized over the years to find a common representation in cross-domain and multiview datasets. For example, CCA [11] learns the linear relations between two sets of correlated variables with the nonlinear type proposed in [12]. As a result, the study in [13] is built on CCA, in which CCA is used to obtain a low-dimensional subspace spanned by a set of correlated but different datasets. In addition, Qin et al. [14] utilized cluster CCA to align domains in a projected correlated subspace. Besides, cotraining [16] is another classical methodthat maximizes the agreement between two distinct views of data to aid the learning process by adapting the knowledge gained from one view to the second view alternatively. Therefore, considering the successes of cotraining, Kumar and Daume [17] incorporated the idea of cotraining into multiview spectral clustering by learning a common clustering matrix that agrees across different views. Furthermore, Sun and Jin [15] proposed a method named robust cotraining, which combines the principles of cotraining and CCA, where CCA is used to analyze the labels predicted by cotraining on unlabeled samples. While the clustering methods based on CCA [13, 14] and cotraining [17] are validated to be efficient, their clustering performance may degrade with noisy data, which is ubiquitous in practice [27].Recently, subspace clustering methods [20–24] have provided a more robust solution to tackle the abovementioned shortcomings. These methods utilize either low-rank representation (LRR) [28, 29] or a combination of LRR and sparse subspace clustering (SSC) [30] to minimize the discrepancy between cross-domain and multiview datasets in a latent subspace. Specifically, these methods utilize self-expressiveness property to select similar samples in the original dataset to reconstruct each other to find a low-dimensional representation. The above is possible because high dimensional data are assumed to have corresponding multiple low-dimensional subspaces and the concept becomes finding those low-dimensional subspaces together with their cluster members [31]. That’s why, Xia et al. [20] combined LRR and SSC techniques to learn a low-rank transition matrix shared across multiple views, which is used subsequently for clustering. Similarly, Ding and Fu and Zhang et al. [22, 24] utilized the LRR technique to pursue a domain-invariant representation that agrees across domains. Brbic and Kopriva [23] used the combination of LRR and SSC to find a low-dimensional representation matrix for each view. Then, using these low-dimensional subspace structures, Brbic and Kopriva [23] can obtain a joint similarity matrix which balances the agreement between the different views. However, these methods mentioned above find the common representation matrix shared across views by assuming that similar samples in each view reside near each other. While this assumption is possible, it can be problematic in a practicable case where the data samples in each view are distributed randomly with noise. Hence, the chances will increase for two unrelated data samples to be selected to reconstruct one another, and this limitation can cause the learned common representation matrix to be faulty in a manner capable of degrading clustering performance. Therefore, the recent work in [26], which extends [25], attempts to resolve the above issue by allowing a common similarity matrix to approximate different view low embedding matrices.. However, the state-of-the-art method fails to consider the proximity issue mentioned above in constructing the individual view structure. Thus, it can cause the propagation of noise in the different low embedding matrices.To this end, in this paper, we propose a novel correlation learning method, which finds a common low-rank matrix between two different instances of data in a latent subspace. The core idea here is that we learn this common low-rank matrix using one instance of data in a way that a second instance can linearly reconstruct it. First, we consider a real-world scenario where two instances of data will have different dimensions. Hence, we utilize matrix factorization to align a low-rank matrix learned in one instance to another through an orthogonal constraint. This approach allows the low-rank matrix to become a common one shared between the two domains. Besides, this approach avoids the propagation of noise in different low-rank matrices. Therefore, our method finds an ideal similarity matrix that approximates the common low-rank matrix much better, with a rank constraint on the Laplacian matrix so that it reveals the clustering structure explicitly without spectral clustering postprocessing. Extensive experimental results on ORL, Yale, Coil-20, Caltech 101-20, and UCI digits datasets demonstrate that our method has superior performance than other state-of-the-art compared methods in six evaluation metrics. That is, accuracy (ACC), normalized mutual information (NMI), adjusted rand index (AR), F-score, precision, and recall.Our major contributions are summarized as follows:(1)
We propose a novel method based on LRR for correlation learning between two instances of data. Specifically, our method learns a common low-rank matrix in one instance of data using matrix factorization to align a second domain. This way, our method can avoid the propagation of noise in different low-rank matrices that can cause a faulty common matrix.(2)
Furthermore, our method can obtain a clustering structure without performing spectral postprocessing of the low-dimensional embedding matrix. To achieve this, we find an ideal similarity matrix that best approximates the common low-rank matrix such that a rank constraint on the Laplacian matrix will reveal the clustering structure explicitly.(3)
Extensive experimental results on five benchmark datasets demonstrate that our method has superior performance than other state-of-the-art compared methods in six evaluation metrics.


## 2. Related Work

The purpose of correlation learning in cross-domain and multiview clustering tasks is to learn a common representation that maximizes the agreement between different instances or distribution of a given data to improve clustering performance [17]. Many studies [27, 31–33] provide an extensive review of the existing methods utilized previously to learn the correlation between cross-domain and multiple view datasets. Among them are two classical methods, namely, CCA [11] and cotraining [16], which have inspired several other multiple instance-based methods such as [13, 17]. While [13] is built on CCA principles , [17] is based on cotraining in which a common structure is learned in a low-dimensional subspace to reduce the disagreement between distinct data views. Besides, the more recent subspace clustering methods [22, 24, 34], built on LRR [28, 29], provide some more robust approaches. These methods require constructing a common similarity matrix to align the different views or cross-domain datasets using the low-rank representation of each view where similar data samples are selected to reconstruct themselves linearly [35]. For simplicity, assuming we have a single view data matrix X with some noisy samples, LRR can obtain a low-rank representation matrix using nuclear norm regularization [36] as follows:(1)U∗+λE2,1,s.t.,X=XU+E,where X denotes the self-dictionary, ⋅∗ denotes the nuclear norm, and E=X−XU denotes the error matrix. Upon obtaining U in a multiview or cross-domain setting, the subspace clustering methods mentioned above can find a common similarity matrix to balance the agreement between the different U. Besides, some methods, such as [20, 23], combine both LRR and SSC [30] to learn a common similarity matrix.Regardless, these subspace clustering methods mentioned above utilize a two-phase learning approach to obtain a clustering structure by applying spectral clustering [37] on the learned similarity matrix. In this case, the clustering performance can degrade when the similarity matrix produced in the first phase is faulty. Considering this drawback, Yang et al. [38] utilized block diagonal constraints to encourage a proper common matrix. Liang et al. [39] explored the diversity between domains to improve performance. Nonetheless, [38, 39] are equally two-phase methods. Moreover, the previous method [26] and the more recent one [40] can obtain a clustering structure directly without applying spectral clustering over the consensus similarity matrix. They both achieved that by imposing a rank constraint on the Laplacian matrix to guarantee c exact number of connected components.Our proposed method is related to the subspace clustering method based on LRR because we can obtain a similarity matrix through a low-rank representation. However, unlike these methods, our method learns a common low-rank representation matrix on only one instance of the data. Then, following a similar approach in [26, 40], we directly pursue a block diagonal clustering structure without performing any spectral postprocessing.


## 3. Proposed Method

In this section, we present our proposed low-rank correlation representation and clustering method. First, we formulate the common low-rank matrix and then incorporate clustering directly into our model.


### 3.1. Model Formulation

Given two multi-instance datasets,Xand Y of sizes n×d1 and n×d2, respectively, where d1 and d2are the dimensions of the feature space and n is the number of samples, a naive approach would include learning different low-rank matrices for Xand Y using equation (1) so that both matrices can be merged to obtain a common matrix. This approach, however, has a limitation. It will cause a fallible common matrix if any of the low-rank matrices is faulty. Therefore, we suppose that a common matrix can be obtained differently through one instance of the data since multiview data originate from one underlying latent subspace [41]. On the other hand, the above is tricky for two reasons. First, the dimensions of Xand Y are different. Second, the redundant specific structure and common structure in Xand Y are combined. To tackle this, we utilize matrix factorization [42, 43] to find a low-rank matrix U adaptively that captures the common structure in Xand Y. At first, we obtain the common structure from Y as follows:(2)Y−UPTF2+U∗,s.t.PTP=I,where U∈ℝnxn denotes a low-rank matrix and P∈ℝd2xn is a factorized variable to make Y≈UPT possible with orthogonal constraint PTP=I. Then, U can be perceived to hold the common structure of X as well with the following model:(3)U−XQF2,s.t.QTQ=I,where Q∈ℝd1xn is similar to P, and it is used here to align the common structure in the two domains with orthogonal constraint QTQ=I so that U can well capture an accurate manifold structure of X and Y through U≈XQ.Upon learningU, our method finds an ideal similarity matrix S=siT,…,snT∈Rn×n by making S an approximate of U as much as possible. As such, we formulate our model as follows:(4)Y−UPTF2+U−XQF2+λ1U∗+λ2S−UF2,s.t.PTP=I,QTQ=I,S≥0,S1=1,rankLs=n−c,where S≥0ensures that all the entries of S are none negative. Ls=1−S denotes the normalized Laplacian matrix because the constraint S1=1 normalizes S such that ∑jSij=1. Then, rankLs=n−c allows S to become the clustering structure with an exact c number of connected components by avoiding spectral postprocessing of the low-dimensional embedding matrix through the following theorem [44, 45].Theorem 1.
If the similarity matrixS is nonnegative, the multiplicity C of the eigenvalue zero of the Laplacian matrix Lsis equal to the number of connected components in the graph that is associated with S.A detailed proof of Theorem1 is given in preposition 2 of [45].


### 3.2. Discussion

Here, we discuss the importance of matrixP and Q, constrained to be column orthogonal using PTP=I and QTQ=I, respectively, in our model. It can be seen that in equation (2), the low-rank matrix U is learned from Y such that it captures the common structure between X and Y. This is possible because we let Y differ from U through PT and not by itself. Then, Q is used in equation (3) to match the common structure inside X( in original d2 dimension to U adaptively. For this, we suppose that the relative error of the term U−XQF2 should be very small for U to capture the common structure in X and Y correctly. Therefore, the proposed method can guarantee more discriminative ability in practice, since U is learned adaptively using the nuclear norm, because with this norm, we can remove the effect of noise through soft thresholding [46], which sets small eigenvalues zero to suppress the noisy data. This approach is different from the existing ones where the data samples themselves are used as dictionaries to learn low-dimensional representation for each of X and Y before a corrected space can be found.


### 3.3. Optimization

We propose an optimization method built on augmented Lagrange multipliers [47] to solve equation (4) by iteratively updating all variables. First, we denote the i-th smallest eigenvalue of Ls as θiLs to make equation (4) easy to solve. Then, referring to Proposition 1 [45], θiLs≥0 because Ls is a positive semidefinite matrix. Therefore, given a large value of λ3, equation (4) is the same as the following:(5)Y−UPTF2+U−XQF2+λ1U∗+λ2S−UF2+λ3∑i=1cθiLs,s.t.PTP=I,QTQ=I,S≥0,S1=1.Furthermore, whenλ3 is large enough, ∑i=1cθiLs=0, so that the constraint rankLs=n−c will be satisfied. According to Ky Fan’s theorem [48], we understand that ∑i=1cθiLs is equivalent to minimizing TrFTLsF subject to FTF=I. This is because an optimal F would contain the eigenvectors corresponding to c smallest eigenvalue of Ls. Hence, for a better understanding of Ky Fan’s theorem, Zhan et al. [26] provided a simplified version as follows.Theorem 2.
Eigenvalues ofLs are ordered by 0≤θ1Ls≤θ2Ls≤⋯≤θnLs and the corresponding eigenvectors are φ1,φ2,…,φn. The inequality ∑i=1cθiLs≤∑i=1cFiTLsFi holds for any orthogonal vectors F1, F2, ⋯ , Fc.Thus, we rewrite equation (5) as(6)Y−UPTF2+U−XQF2+λ1U∗+λ2S−UF2+λ3TrFTLsF,s.t.PTP=I,QTQ=I,S≥0,S1=1,FTF=I,and also, we introduce an intermediate term J=U to make equation (6) easier to solve. Then, we get(7)Y−UPTF2+U−XQF2+λ1J∗+λ2S−UF2+λ3TrFTLsF,s.t.PTP=I,QTQ=I,U=J,S≥0,S1=1,FTF=I.The Lagrangian of equation (7) can be obtained as follows:(8)L=Y−UPTF2+U−XQF2+λ1J∗+λ2S−UF2+λ3TrFTLsF+M1,U−J+μ2U−JF2,where M1 is the Lagrange multiplier. Hence, we divide equation (8) into several subproblems and then update all subproblems by fixing the others in the following order.


#### 3.3.1. J Subproblem

(9)minJλ1μJ∗+12J−U+M1μF2.We solve equation (9) by denoting the singular value decomposition (SVD) [46] of SμM as Tℋμ∑VT to obtain J:(10)J=Sλ1/μU+M1μ,where ℋμΣ=min0,Σ+μ+max0,Σ−μ.


#### 3.3.2. P Subproblem

(11)minPY−UPTF2,s.t.PTP=I.The orthogonal Procrustes problem in equation (11) is difficult to solve because the feasible set P satisfying PTP=I is not convex. However, luckily, SVD [46] provides a method to solve this problem to find a unique solution. Therefore, we solve P by svdUTY as follows:(12)U,Σ,V=svdUTY,P=VUT.


#### 3.3.3. Q Subproblem

(13)minQU−XQF2,s.t.QTQ=I.TheQ subproblem can be solved the same way as P subproblem.


#### 3.3.4. U Subproblem

(14)minuY−UPTF2+U−XQF2+λ2S−UF2+M1,U−J+μ2U−JF2.Setting the derivatives∂/∂U=0, we get(15)U=2YP+2XQ+2λ2S−M1+μJ2+2λ2+μI+PTP−1.


#### 3.3.5. S Subproblem

(16)minsλ2S−UF2+λ3TrFTLsF,s.t.S≥0,S1=1,FTF=I.SinceF in equation (16) contains the eigenvectors corresponding to c smallest eigenvalue of Ls, we rewrite equation (16) as(17)minsjλ3fi−fj22−λ2uijsij+λ2sjTsj,s.t.sj≥0,1Tsj=1.Then, denotingλ3fi−fj22−λ2uij as pij, equation (17) can be rewritten as(18)minsjλ2sj+pj2λ222,s.t.,sj≥0,1Tsj=1.Similar to [26], the optimal solution sj ist(19)sj=−pj2λ2+η1+.


### 3.4. Complexity Analysis

The computational complexity of our proposed method illustrated in Algorithm1 is determined mainly by nuclear norm calculation in equation (10), matrix inverse and multiplication in equation (13), and Euclidean projection onto the simplex space in equation (19). The cost of equation (8) is O (n3), while the inverse of an n × n matrix in equation (15) consumes O (n3). The time complexity for matrix multiplication is O (n3). However, since there are several multiplications in equation (15), the overall multiplication cost will be (k + 1) O (n3), which is not ignorable when the number of data samples is large. Moreover, the Euclidean projection onto the simplex space in equation (19) takes On.Algorithm 1:Algorithm for our proposed method.
Input: training dataset X,Y clusters size cInitialize:M1=0,U and S matrices are based on c nearest neighbor graph; F is formed by the c eigenvectors of Ls corresponding to the c smallest eigenvalues;ρ=1.01While not converged doUpdateJ while fixing others by equation (10)UpdateP while fixing others by equation (12)UpdateQ while fixing others by equation (13)UpdateU while fixing others by equation (15)UpdateS while fixing others by equation (19)Update the multipliers.M1=M1+μU−JUpdateμ by μ=minρμ,maxμEnd whileOutput:S,U


## 3.1. Model Formulation

Given two multi-instance datasets,Xand Y of sizes n×d1 and n×d2, respectively, where d1 and d2are the dimensions of the feature space and n is the number of samples, a naive approach would include learning different low-rank matrices for Xand Y using equation (1) so that both matrices can be merged to obtain a common matrix. This approach, however, has a limitation. It will cause a fallible common matrix if any of the low-rank matrices is faulty. Therefore, we suppose that a common matrix can be obtained differently through one instance of the data since multiview data originate from one underlying latent subspace [41]. On the other hand, the above is tricky for two reasons. First, the dimensions of Xand Y are different. Second, the redundant specific structure and common structure in Xand Y are combined. To tackle this, we utilize matrix factorization [42, 43] to find a low-rank matrix U adaptively that captures the common structure in Xand Y. At first, we obtain the common structure from Y as follows:(2)Y−UPTF2+U∗,s.t.PTP=I,where U∈ℝnxn denotes a low-rank matrix and P∈ℝd2xn is a factorized variable to make Y≈UPT possible with orthogonal constraint PTP=I. Then, U can be perceived to hold the common structure of X as well with the following model:(3)U−XQF2,s.t.QTQ=I,where Q∈ℝd1xn is similar to P, and it is used here to align the common structure in the two domains with orthogonal constraint QTQ=I so that U can well capture an accurate manifold structure of X and Y through U≈XQ.Upon learningU, our method finds an ideal similarity matrix S=siT,…,snT∈Rn×n by making S an approximate of U as much as possible. As such, we formulate our model as follows:(4)Y−UPTF2+U−XQF2+λ1U∗+λ2S−UF2,s.t.PTP=I,QTQ=I,S≥0,S1=1,rankLs=n−c,where S≥0ensures that all the entries of S are none negative. Ls=1−S denotes the normalized Laplacian matrix because the constraint S1=1 normalizes S such that ∑jSij=1. Then, rankLs=n−c allows S to become the clustering structure with an exact c number of connected components by avoiding spectral postprocessing of the low-dimensional embedding matrix through the following theorem [44, 45].Theorem 1.
If the similarity matrixS is nonnegative, the multiplicity C of the eigenvalue zero of the Laplacian matrix Lsis equal to the number of connected components in the graph that is associated with S.A detailed proof of Theorem1 is given in preposition 2 of [45].


## 3.2. Discussion

Here, we discuss the importance of matrixP and Q, constrained to be column orthogonal using PTP=I and QTQ=I, respectively, in our model. It can be seen that in equation (2), the low-rank matrix U is learned from Y such that it captures the common structure between X and Y. This is possible because we let Y differ from U through PT and not by itself. Then, Q is used in equation (3) to match the common structure inside X( in original d2 dimension to U adaptively. For this, we suppose that the relative error of the term U−XQF2 should be very small for U to capture the common structure in X and Y correctly. Therefore, the proposed method can guarantee more discriminative ability in practice, since U is learned adaptively using the nuclear norm, because with this norm, we can remove the effect of noise through soft thresholding [46], which sets small eigenvalues zero to suppress the noisy data. This approach is different from the existing ones where the data samples themselves are used as dictionaries to learn low-dimensional representation for each of X and Y before a corrected space can be found.


## 3.3. Optimization

We propose an optimization method built on augmented Lagrange multipliers [47] to solve equation (4) by iteratively updating all variables. First, we denote the i-th smallest eigenvalue of Ls as θiLs to make equation (4) easy to solve. Then, referring to Proposition 1 [45], θiLs≥0 because Ls is a positive semidefinite matrix. Therefore, given a large value of λ3, equation (4) is the same as the following:(5)Y−UPTF2+U−XQF2+λ1U∗+λ2S−UF2+λ3∑i=1cθiLs,s.t.PTP=I,QTQ=I,S≥0,S1=1.Furthermore, whenλ3 is large enough, ∑i=1cθiLs=0, so that the constraint rankLs=n−c will be satisfied. According to Ky Fan’s theorem [48], we understand that ∑i=1cθiLs is equivalent to minimizing TrFTLsF subject to FTF=I. This is because an optimal F would contain the eigenvectors corresponding to c smallest eigenvalue of Ls. Hence, for a better understanding of Ky Fan’s theorem, Zhan et al. [26] provided a simplified version as follows.Theorem 2.
Eigenvalues ofLs are ordered by 0≤θ1Ls≤θ2Ls≤⋯≤θnLs and the corresponding eigenvectors are φ1,φ2,…,φn. The inequality ∑i=1cθiLs≤∑i=1cFiTLsFi holds for any orthogonal vectors F1, F2, ⋯ , Fc.Thus, we rewrite equation (5) as(6)Y−UPTF2+U−XQF2+λ1U∗+λ2S−UF2+λ3TrFTLsF,s.t.PTP=I,QTQ=I,S≥0,S1=1,FTF=I,and also, we introduce an intermediate term J=U to make equation (6) easier to solve. Then, we get(7)Y−UPTF2+U−XQF2+λ1J∗+λ2S−UF2+λ3TrFTLsF,s.t.PTP=I,QTQ=I,U=J,S≥0,S1=1,FTF=I.The Lagrangian of equation (7) can be obtained as follows:(8)L=Y−UPTF2+U−XQF2+λ1J∗+λ2S−UF2+λ3TrFTLsF+M1,U−J+μ2U−JF2,where M1 is the Lagrange multiplier. Hence, we divide equation (8) into several subproblems and then update all subproblems by fixing the others in the following order.


### 3.3.1. J Subproblem

(9)minJλ1μJ∗+12J−U+M1μF2.We solve equation (9) by denoting the singular value decomposition (SVD) [46] of SμM as Tℋμ∑VT to obtain J:(10)J=Sλ1/μU+M1μ,where ℋμΣ=min0,Σ+μ+max0,Σ−μ.


### 3.3.2. P Subproblem

(11)minPY−UPTF2,s.t.PTP=I.The orthogonal Procrustes problem in equation (11) is difficult to solve because the feasible set P satisfying PTP=I is not convex. However, luckily, SVD [46] provides a method to solve this problem to find a unique solution. Therefore, we solve P by svdUTY as follows:(12)U,Σ,V=svdUTY,P=VUT.


### 3.3.3. Q Subproblem

(13)minQU−XQF2,s.t.QTQ=I.TheQ subproblem can be solved the same way as P subproblem.


### 3.3.4. U Subproblem

(14)minuY−UPTF2+U−XQF2+λ2S−UF2+M1,U−J+μ2U−JF2.Setting the derivatives∂/∂U=0, we get(15)U=2YP+2XQ+2λ2S−M1+μJ2+2λ2+μI+PTP−1.


### 3.3.5. S Subproblem

(16)minsλ2S−UF2+λ3TrFTLsF,s.t.S≥0,S1=1,FTF=I.SinceF in equation (16) contains the eigenvectors corresponding to c smallest eigenvalue of Ls, we rewrite equation (16) as(17)minsjλ3fi−fj22−λ2uijsij+λ2sjTsj,s.t.sj≥0,1Tsj=1.Then, denotingλ3fi−fj22−λ2uij as pij, equation (17) can be rewritten as(18)minsjλ2sj+pj2λ222,s.t.,sj≥0,1Tsj=1.Similar to [26], the optimal solution sj ist(19)sj=−pj2λ2+η1+.


## 3.3.1. J Subproblem

(9)minJλ1μJ∗+12J−U+M1μF2.We solve equation (9) by denoting the singular value decomposition (SVD) [46] of SμM as Tℋμ∑VT to obtain J:(10)J=Sλ1/μU+M1μ,where ℋμΣ=min0,Σ+μ+max0,Σ−μ.


## 3.3.2. P Subproblem

(11)minPY−UPTF2,s.t.PTP=I.The orthogonal Procrustes problem in equation (11) is difficult to solve because the feasible set P satisfying PTP=I is not convex. However, luckily, SVD [46] provides a method to solve this problem to find a unique solution. Therefore, we solve P by svdUTY as follows:(12)U,Σ,V=svdUTY,P=VUT.


## 3.3.3. Q Subproblem

(13)minQU−XQF2,s.t.QTQ=I.TheQ subproblem can be solved the same way as P subproblem.


## 3.3.4. U Subproblem

(14)minuY−UPTF2+U−XQF2+λ2S−UF2+M1,U−J+μ2U−JF2.Setting the derivatives∂/∂U=0, we get(15)U=2YP+2XQ+2λ2S−M1+μJ2+2λ2+μI+PTP−1.


## 3.3.5. S Subproblem

(16)minsλ2S−UF2+λ3TrFTLsF,s.t.S≥0,S1=1,FTF=I.SinceF in equation (16) contains the eigenvectors corresponding to c smallest eigenvalue of Ls, we rewrite equation (16) as(17)minsjλ3fi−fj22−λ2uijsij+λ2sjTsj,s.t.sj≥0,1Tsj=1.Then, denotingλ3fi−fj22−λ2uij as pij, equation (17) can be rewritten as(18)minsjλ2sj+pj2λ222,s.t.,sj≥0,1Tsj=1.Similar to [26], the optimal solution sj ist(19)sj=−pj2λ2+η1+.


## 3.4. Complexity Analysis

The computational complexity of our proposed method illustrated in Algorithm1 is determined mainly by nuclear norm calculation in equation (10), matrix inverse and multiplication in equation (13), and Euclidean projection onto the simplex space in equation (19). The cost of equation (8) is O (n3), while the inverse of an n × n matrix in equation (15) consumes O (n3). The time complexity for matrix multiplication is O (n3). However, since there are several multiplications in equation (15), the overall multiplication cost will be (k + 1) O (n3), which is not ignorable when the number of data samples is large. Moreover, the Euclidean projection onto the simplex space in equation (19) takes On.Algorithm 1:Algorithm for our proposed method.
Input: training dataset X,Y clusters size cInitialize:M1=0,U and S matrices are based on c nearest neighbor graph; F is formed by the c eigenvectors of Ls corresponding to the c smallest eigenvalues;ρ=1.01While not converged doUpdateJ while fixing others by equation (10)UpdateP while fixing others by equation (12)UpdateQ while fixing others by equation (13)UpdateU while fixing others by equation (15)UpdateS while fixing others by equation (19)Update the multipliers.M1=M1+μU−JUpdateμ by μ=minρμ,maxμEnd whileOutput:S,U


## 4. Experiments

In this section, we conduct an extensive experiment to demonstrate effectiveness of our proposed method on five benchmark datasets described in Section4.1. Accordingly, we compare our method with seven state-of-the-art methods, namely, MLRSSC [23], RMSL [22], MCGC [26], DiMSC [21], MVGL [25], GMC [40], and SM2SC [38].


### 4.1. Datasets

We perform experiments on ORL and Yale, Coil-20 and Caltech 101-20, and UCI digits datasets to demonstrate the superiority of our method on face image clustering, object image clustering, and handwritten digit, respectively. For each dataset, we select two types of features to representX andY. Specifically, we extract LBP together with Gabor features for ORL, Yale, Coil-20, and Caltech 101-20, and we select Fourier coefﬁcients of the character shape (FOU) and proﬁle correlation (FAC) features for UCI digits. Therefore, the following is a detailed description of each dataset, while Table 1 and Figure 1 provide a summary and pictures of the datasets, respectively.(1)
ORL1: this face dataset contains 400 images of 40 distinct subjects. Each subject represents different images taken under various conditions such as time, light, and facial expressions. Note that the dimensions of LBP and Gabor features extracted from this dataset are 3304 and 6750, respectively.(2)
Yale2: this dataset contains 165 gray-scale images of 15 individuals with 11 images per individual. The variations of this dataset include the images taken, with center light, with glasses, under happy mood, with left light, without glasses, etc. Here, the feature dimension of LBP and Gabor is the same as that extracted for ORL above.(3)
COIL-203: this dataset is from the Columbia Object Image Library, and there are 1440 images of 20 different classes. Each class contains 72 images. The LBP and Gabor features of this dataset have 944 and 4096 dimensions, respectively.(4)
Caltech 101-204: his image dataset has 101 categories of images. We select the widely used twenty classes to obtain 2368 images each for 48-D Gabor and 928-D LBP features.(5)
UCI digits5: this dataset denotes handwritten digits of 0 to 9 from the UCI machine learning repository. It is composed of 2000 data samples. The two features extracted are FOU and FAC with 76 and 216 dimensions, respectively.Table 1
Summary of datasets.
DatasetSizeClassORL40040Yale16515COIL-20144020Caltech 101-20236820UCI digits200010Figure 1
Example pictures of four benchmark datasets. (a) ORL, (b) Yale, (c) COIL-20, and (d) Caltech 101-20.
(a)(b)(c)(d)


### 4.2. Experimental Setting

We fine-tuned the parameters for each method with strict compliance with the experimental settings in the respective literature. For our proposed method, two parametersλ1and λ2 need tuning. Therefore, we utilize a grid search to find the best λ1and λ2 from 0.001,0.01,0.1,1,10,100,1000. Then, we perform all experiments ten times and report the mean performance to guarantee fairness.


### 4.3. Evaluation Metrics

To evaluate our method, we utilize six standard evaluation metrics, i.e., accuracy (ACC), normalized mutual information (NMI), adjusted rand index (AR), F-score, precision, and recall. These metrics capture a different aspect of the performance to demonstrate the superiority of our method over state-of-the-art compared methods. For example, ACC measures the percentage of correctly clustered data samples in the learned clustering structure compared with the ground truth labels, whereas NMI is a theoretic validation measure, which relies on the amount of statistical information shared by random variables.


### 4.4. Experimental Results

#### 4.4.1. Image Clustering

In this section, we present the image clustering performance results on all five benchmark datasets using the six evaluation metrics described earlier.Table2 displays the performance results on ORL and Yale datasets. Specifically, on the ORL dataset, our method has better performance results than the seven state-of-the-art compared methods with 81.39%, 91.32%, 71.13%, and 83.10% in ACC, NMI, precision, and recall, respectively. Surprisingly, DiMSC outperforms the more recent methods, such as MCGC, MVGL, and MLRSSC, with 78.87% in ACC and 91.19% in NMI. The above may result from the approach utilized by DiMSC to learn the common representation matrix where it introduced a diversity regularizer to obtain a representation matrix for each view that enhanced the diversity of the different views. Besides, one can observe that GMC and SM2SC, which are more recent methods, have better performances than other compared methods in most evaluation metrics. In particular, SM2SC has the best performance of 78.36% in AR, which is just 1% higher than that obtained by our proposed method. Nonetheless, our proposed method has the best performance on the Yale dataset in all evaluation metrics, with 74.48% in ACC, 71.96% in NMI, 50.11% in AR, 51.84% in F-score, 50.42% in precision, and 59.18% in the recall metrics.Table 2
Clustering performance on face datasets.
DatasetsMethodsACCNMIARF-scorePrecisionRecallORLMLRSSC71.9587.2264.1865.0660.2570.81RMSL70.1684.4563.1965.7858.5765.32MCGC77.2585.6272.2773.6970.3679.39DiMSC78.8791.1973.5574.1969.4879.61MVGL74.5088.6374.3476.1469.0982.06GMC79.3590.5776.8971.5871.0581.98SM2SC80.5691.0878.3675.3670.8282.76Ours81.3991.3277.3675.7371.1383.10YaleMLRSSC64.3168.1847.5950.9448.6853.46RMSL65.4370.2147.2848.3545.1455.41MCGC68.0668.3745.8849.6343.6257.58DiMSC66.0667.3447.5750.8749.3652.48MVGL67.6770.7846.2749.8345.7954.67GMC74.0566.2847.5850.6648.2857.52SM2SC74.2670.4349.2351.1949.5358.31Ours74.4871.9650.1151.8450.4259.18Table3 illustrates the clustering performance on Coil-20 and Caltech 101-20 object datasets. Furthermore, our method has the best performance in all evaluation metrics, except for the Coil-20 dataset, where SM2SC has the best performance in ACC by a small margin of 0.29%.Table 3
Clustering performance on object datasets.
DatasetsMethodsACCNMIARF-scorePrecisionRecallCOIL-20MLRSSC83.3583.2569.0570.3668.2773.18RMSL80.7983.0462.4170.9469.1472.36MCGC88.2183.7869.9070.8465.5377.08DiMSC85.2382.7169.8671.3770.3072.48MVGL85.2195.9283.2582.1775.6494.88GMC87.0593.7283.6984.0377.3791.26SM2SC89.4295.7183.7684.2178.3094.81Ours89.1396.5484.0386.4978.5395.10Caltech 101-20MLRSSC55.4351.9146.0142.0683.7668.86RMSL57.9452.1544.6741.1681.3264.35MCGC64.5159.4746.5748.0688.9472.72DiMSC58.1247.7740.5947.6186.4767.42MVGL62.9558.8145.9646.7987.2572.13GMC65.2360.5242.8046.5189.1569.51SM2SC66.0460.4947.2149.3389.3072.29Ours66.4261.1448.7549.7689.4473.05Table4 shows that our proposed method outperforms other state-of-the-art methods in five evaluation metrics such as NMI and AR, where it has 86.76% and 78.42%, respectively.Table 4
Clustering performance on handwritten datasets.
DatasetsMethodsACCNMIARF-scorePrecisionRecallUCI digitsMLRSSC72.1281.2476.1578.6276.4481.11RMSL76.4681.3876.5477.2971.5883.46MCGC82.2082.5975.5378.1871.7985.82DiMSC82.3680.4175.1678.9476.6782.43MVGL84.2586.0475.7877.2868.0589.40GMC86.9485.9777.0579.8779.2388.15SM2SC87.2886.2377.5980.2578.3189.77Ours85.6186.7678.4281.9578.5690.25Overall, our method outperforms GMC, SM2SC, and MCGC only slightly, especially on object datasets, and it is not surprising because, for instance, SM2SC employed a block diagonal regularization to enforce a proper common structure. Similarly, GMC and MCGC utilized a rank constraint to find a unified clustering structure to balance the agreement between the different views by avoiding k-means spectral postprocessing of the low-dimensional embedding matrix as well. Notwithstanding, our proposed method outperforms other state-of-the-art methods by a wide margin in all evaluation metrics. Furthermore, we provide more intuitiveness of our approach by visualizing the clustering structure learned on three datasets in Figure 2. Accordingly, we can see clearly that our proposed method has a block diagonal clustering structure on all three datasets with an exact c number of connected components. Therefore, we conclude that our common low-rank learning approach is very efficient.Figure 2
Visualization of the clustering structure of our proposed method on three datasets. (a) ORL, (b) COIL-20, and (c) UCI digits.
(a)(b)(c)


#### 4.4.2. Image Recognition with 0%, 10%, and 20% Levels of Corruptions

In this section, we present experimental results obtained for image recognition with respect to accuracy evaluation metric. In this experiment, we study the robustness against noise corruption of each algorithm by gradually injecting 10% and 20% noise into the five benchmark datasets. To perform this experiment, we keep the same parameter settings described in Section4.2, while the K nearest neighbor (KNN) classiﬁer is applied to evaluate the classiﬁcation accuracy of each algorithm. Thus, it is easily noticeable through Tables 5–9 that all algorithms had a degrading performance with an increment in the noise level. Specifically, our proposed method had degraded performance of only about 3–5% on all datasets with a 10% level of noise corruption. However, all algorithms had a significant reduction in performance with a 20% level of noise corruption. Yet, we can observe that our proposed method is generally more robust against noise than the other compared methods.Table 5
Variation of classification accuracy with different corruption rates on the ORL dataset.
Method rates0%10%20%MLRSSC72.5367.9851.83RMSL77.1873.3557.68MCGC76.0371.4962.35DiMSC80.5875.3064.91MVGL81.2977.7266.81GMC81.9878.0970.35SM2SC82.3977.5668.94Ours82.6479.8672.53Table 6
Variation of classification accuracy with different corruption rates on the Yale dataset.
Method rates0%10%20%MLRSSC71.8361.7848.62RMSL72.7962.9350.28MCGC71.2363.5749.68DiMSC73.3764.0351.26MVGL73.8467.3156.62GMC74.6968.0756.74SM2SC75.2168.3256.02Ours75.3068.9457.84Table 7
Variation of classification accuracy with different corruption rates on the COIL-20 dataset.
Method rates0%10%20%MLRSSC86.2481.8566.72RMSL87.8782.3868.54MCGC89.6485.6171.33DiMSC91.1387.0973.76MVGL90.4586.3673.03GMC92.1885.9574.52SM2SC92.2786.3473.65Ours92.7488.2976.15Table 8
Variation of classification accuracy with different corruption rates on the Caltech 101-20 dataset.
Method rates0%10%20%MLRSSC61.8256.4541.61RMSL63.5561.3045.77MCGC64.6059.4242.73DiMSC66.4662.9246.03MVGL67.7163.1345.51GMC69.3662.4544.77SM2SC68.8361.2046.45Ours68.2765.3548.87Table 9
Variation of classification accuracy with different corruption rates on the UCI digits dataset.
Method rates0%10%20%MLRSSC83.3377.5258.27RMSL81.7977.1560.84MCGC84.3179.6761.15DiMSC86.5882.3464.88MVGL86.2381.5564.51GMC87.0583.1465.28SM2SC88.3482.9866.87Ours87.1184.5768.32


### 4.5. Parameter Sensitivity

Figure3 demonstrates the clustering performance of our method with respect to NMI and ACC by varying λ1 and λ2 on Yale, COIL-20, and UCI digits datasets. First of all, we should note that λ3 does not need turning like the other parameters because we find its best value heuristically like [26] to accelerate the process. Therefore, we initialize λ3=1 and automatically set it as λ3=λ3∗2 or λ3  =  λ3/2 in each iteration when the number of connected components is smaller or larger than c, respectively. As a result, we can see from Figure 3 that our method is more sensitive to λ1 on both NMI and ACC because it controls the learning of the common low-rank matrix U. Hence, our proposed method is certain to have a good performance by finding a suitable value for λ1. On the other hand, our method is only slightly sensitive to λ2, which means that λ2 is not as important as λ1 to our model. Still, these clustering results show that our proposed method is relatively stable regardless of the dataset.Figure 3
The results of the clustering performance of our method with respect to NMI and ACC evaluation metrics by varying-parametersλ1 and λ2 on (a) Yale dataset, (b) COIL-20 dataset, and (c) UCI digits dataset.
(a)(b)(c)


### 4.6. Convergence Analysis

Although the convergence of the inexact augmented Lagrange multiplier (ALM) method with more than three subproblems is still not easy to theoretically prove [29], we compute the relative error of the term λ2S−UF2 to demonstrate the convergence behavior of our proposed method. To illustrate this better, Figure 4 shows the errors at different iterations. It is easy to see that on all three benchmark datasets, our method converges within 150 iterations.Figure 4
Visualization of the convergence behavior ofλ2S−UF2 for different iterations.


## 4.1. Datasets

We perform experiments on ORL and Yale, Coil-20 and Caltech 101-20, and UCI digits datasets to demonstrate the superiority of our method on face image clustering, object image clustering, and handwritten digit, respectively. For each dataset, we select two types of features to representX andY. Specifically, we extract LBP together with Gabor features for ORL, Yale, Coil-20, and Caltech 101-20, and we select Fourier coefﬁcients of the character shape (FOU) and proﬁle correlation (FAC) features for UCI digits. Therefore, the following is a detailed description of each dataset, while Table 1 and Figure 1 provide a summary and pictures of the datasets, respectively.(1)
ORL1: this face dataset contains 400 images of 40 distinct subjects. Each subject represents different images taken under various conditions such as time, light, and facial expressions. Note that the dimensions of LBP and Gabor features extracted from this dataset are 3304 and 6750, respectively.(2)
Yale2: this dataset contains 165 gray-scale images of 15 individuals with 11 images per individual. The variations of this dataset include the images taken, with center light, with glasses, under happy mood, with left light, without glasses, etc. Here, the feature dimension of LBP and Gabor is the same as that extracted for ORL above.(3)
COIL-203: this dataset is from the Columbia Object Image Library, and there are 1440 images of 20 different classes. Each class contains 72 images. The LBP and Gabor features of this dataset have 944 and 4096 dimensions, respectively.(4)
Caltech 101-204: his image dataset has 101 categories of images. We select the widely used twenty classes to obtain 2368 images each for 48-D Gabor and 928-D LBP features.(5)
UCI digits5: this dataset denotes handwritten digits of 0 to 9 from the UCI machine learning repository. It is composed of 2000 data samples. The two features extracted are FOU and FAC with 76 and 216 dimensions, respectively.Table 1
Summary of datasets.
DatasetSizeClassORL40040Yale16515COIL-20144020Caltech 101-20236820UCI digits200010Figure 1
Example pictures of four benchmark datasets. (a) ORL, (b) Yale, (c) COIL-20, and (d) Caltech 101-20.
(a)(b)(c)(d)


## 4.2. Experimental Setting

We fine-tuned the parameters for each method with strict compliance with the experimental settings in the respective literature. For our proposed method, two parametersλ1and λ2 need tuning. Therefore, we utilize a grid search to find the best λ1and λ2 from 0.001,0.01,0.1,1,10,100,1000. Then, we perform all experiments ten times and report the mean performance to guarantee fairness.


## 4.3. Evaluation Metrics

To evaluate our method, we utilize six standard evaluation metrics, i.e., accuracy (ACC), normalized mutual information (NMI), adjusted rand index (AR), F-score, precision, and recall. These metrics capture a different aspect of the performance to demonstrate the superiority of our method over state-of-the-art compared methods. For example, ACC measures the percentage of correctly clustered data samples in the learned clustering structure compared with the ground truth labels, whereas NMI is a theoretic validation measure, which relies on the amount of statistical information shared by random variables.


## 4.4. Experimental Results

### 4.4.1. Image Clustering

In this section, we present the image clustering performance results on all five benchmark datasets using the six evaluation metrics described earlier.Table2 displays the performance results on ORL and Yale datasets. Specifically, on the ORL dataset, our method has better performance results than the seven state-of-the-art compared methods with 81.39%, 91.32%, 71.13%, and 83.10% in ACC, NMI, precision, and recall, respectively. Surprisingly, DiMSC outperforms the more recent methods, such as MCGC, MVGL, and MLRSSC, with 78.87% in ACC and 91.19% in NMI. The above may result from the approach utilized by DiMSC to learn the common representation matrix where it introduced a diversity regularizer to obtain a representation matrix for each view that enhanced the diversity of the different views. Besides, one can observe that GMC and SM2SC, which are more recent methods, have better performances than other compared methods in most evaluation metrics. In particular, SM2SC has the best performance of 78.36% in AR, which is just 1% higher than that obtained by our proposed method. Nonetheless, our proposed method has the best performance on the Yale dataset in all evaluation metrics, with 74.48% in ACC, 71.96% in NMI, 50.11% in AR, 51.84% in F-score, 50.42% in precision, and 59.18% in the recall metrics.Table 2
Clustering performance on face datasets.
DatasetsMethodsACCNMIARF-scorePrecisionRecallORLMLRSSC71.9587.2264.1865.0660.2570.81RMSL70.1684.4563.1965.7858.5765.32MCGC77.2585.6272.2773.6970.3679.39DiMSC78.8791.1973.5574.1969.4879.61MVGL74.5088.6374.3476.1469.0982.06GMC79.3590.5776.8971.5871.0581.98SM2SC80.5691.0878.3675.3670.8282.76Ours81.3991.3277.3675.7371.1383.10YaleMLRSSC64.3168.1847.5950.9448.6853.46RMSL65.4370.2147.2848.3545.1455.41MCGC68.0668.3745.8849.6343.6257.58DiMSC66.0667.3447.5750.8749.3652.48MVGL67.6770.7846.2749.8345.7954.67GMC74.0566.2847.5850.6648.2857.52SM2SC74.2670.4349.2351.1949.5358.31Ours74.4871.9650.1151.8450.4259.18Table3 illustrates the clustering performance on Coil-20 and Caltech 101-20 object datasets. Furthermore, our method has the best performance in all evaluation metrics, except for the Coil-20 dataset, where SM2SC has the best performance in ACC by a small margin of 0.29%.Table 3
Clustering performance on object datasets.
DatasetsMethodsACCNMIARF-scorePrecisionRecallCOIL-20MLRSSC83.3583.2569.0570.3668.2773.18RMSL80.7983.0462.4170.9469.1472.36MCGC88.2183.7869.9070.8465.5377.08DiMSC85.2382.7169.8671.3770.3072.48MVGL85.2195.9283.2582.1775.6494.88GMC87.0593.7283.6984.0377.3791.26SM2SC89.4295.7183.7684.2178.3094.81Ours89.1396.5484.0386.4978.5395.10Caltech 101-20MLRSSC55.4351.9146.0142.0683.7668.86RMSL57.9452.1544.6741.1681.3264.35MCGC64.5159.4746.5748.0688.9472.72DiMSC58.1247.7740.5947.6186.4767.42MVGL62.9558.8145.9646.7987.2572.13GMC65.2360.5242.8046.5189.1569.51SM2SC66.0460.4947.2149.3389.3072.29Ours66.4261.1448.7549.7689.4473.05Table4 shows that our proposed method outperforms other state-of-the-art methods in five evaluation metrics such as NMI and AR, where it has 86.76% and 78.42%, respectively.Table 4
Clustering performance on handwritten datasets.
DatasetsMethodsACCNMIARF-scorePrecisionRecallUCI digitsMLRSSC72.1281.2476.1578.6276.4481.11RMSL76.4681.3876.5477.2971.5883.46MCGC82.2082.5975.5378.1871.7985.82DiMSC82.3680.4175.1678.9476.6782.43MVGL84.2586.0475.7877.2868.0589.40GMC86.9485.9777.0579.8779.2388.15SM2SC87.2886.2377.5980.2578.3189.77Ours85.6186.7678.4281.9578.5690.25Overall, our method outperforms GMC, SM2SC, and MCGC only slightly, especially on object datasets, and it is not surprising because, for instance, SM2SC employed a block diagonal regularization to enforce a proper common structure. Similarly, GMC and MCGC utilized a rank constraint to find a unified clustering structure to balance the agreement between the different views by avoiding k-means spectral postprocessing of the low-dimensional embedding matrix as well. Notwithstanding, our proposed method outperforms other state-of-the-art methods by a wide margin in all evaluation metrics. Furthermore, we provide more intuitiveness of our approach by visualizing the clustering structure learned on three datasets in Figure 2. Accordingly, we can see clearly that our proposed method has a block diagonal clustering structure on all three datasets with an exact c number of connected components. Therefore, we conclude that our common low-rank learning approach is very efficient.Figure 2
Visualization of the clustering structure of our proposed method on three datasets. (a) ORL, (b) COIL-20, and (c) UCI digits.
(a)(b)(c)


### 4.4.2. Image Recognition with 0%, 10%, and 20% Levels of Corruptions

In this section, we present experimental results obtained for image recognition with respect to accuracy evaluation metric. In this experiment, we study the robustness against noise corruption of each algorithm by gradually injecting 10% and 20% noise into the five benchmark datasets. To perform this experiment, we keep the same parameter settings described in Section4.2, while the K nearest neighbor (KNN) classiﬁer is applied to evaluate the classiﬁcation accuracy of each algorithm. Thus, it is easily noticeable through Tables 5–9 that all algorithms had a degrading performance with an increment in the noise level. Specifically, our proposed method had degraded performance of only about 3–5% on all datasets with a 10% level of noise corruption. However, all algorithms had a significant reduction in performance with a 20% level of noise corruption. Yet, we can observe that our proposed method is generally more robust against noise than the other compared methods.Table 5
Variation of classification accuracy with different corruption rates on the ORL dataset.
Method rates0%10%20%MLRSSC72.5367.9851.83RMSL77.1873.3557.68MCGC76.0371.4962.35DiMSC80.5875.3064.91MVGL81.2977.7266.81GMC81.9878.0970.35SM2SC82.3977.5668.94Ours82.6479.8672.53Table 6
Variation of classification accuracy with different corruption rates on the Yale dataset.
Method rates0%10%20%MLRSSC71.8361.7848.62RMSL72.7962.9350.28MCGC71.2363.5749.68DiMSC73.3764.0351.26MVGL73.8467.3156.62GMC74.6968.0756.74SM2SC75.2168.3256.02Ours75.3068.9457.84Table 7
Variation of classification accuracy with different corruption rates on the COIL-20 dataset.
Method rates0%10%20%MLRSSC86.2481.8566.72RMSL87.8782.3868.54MCGC89.6485.6171.33DiMSC91.1387.0973.76MVGL90.4586.3673.03GMC92.1885.9574.52SM2SC92.2786.3473.65Ours92.7488.2976.15Table 8
Variation of classification accuracy with different corruption rates on the Caltech 101-20 dataset.
Method rates0%10%20%MLRSSC61.8256.4541.61RMSL63.5561.3045.77MCGC64.6059.4242.73DiMSC66.4662.9246.03MVGL67.7163.1345.51GMC69.3662.4544.77SM2SC68.8361.2046.45Ours68.2765.3548.87Table 9
Variation of classification accuracy with different corruption rates on the UCI digits dataset.
Method rates0%10%20%MLRSSC83.3377.5258.27RMSL81.7977.1560.84MCGC84.3179.6761.15DiMSC86.5882.3464.88MVGL86.2381.5564.51GMC87.0583.1465.28SM2SC88.3482.9866.87Ours87.1184.5768.32


## 4.4.1. Image Clustering

In this section, we present the image clustering performance results on all five benchmark datasets using the six evaluation metrics described earlier.Table2 displays the performance results on ORL and Yale datasets. Specifically, on the ORL dataset, our method has better performance results than the seven state-of-the-art compared methods with 81.39%, 91.32%, 71.13%, and 83.10% in ACC, NMI, precision, and recall, respectively. Surprisingly, DiMSC outperforms the more recent methods, such as MCGC, MVGL, and MLRSSC, with 78.87% in ACC and 91.19% in NMI. The above may result from the approach utilized by DiMSC to learn the common representation matrix where it introduced a diversity regularizer to obtain a representation matrix for each view that enhanced the diversity of the different views. Besides, one can observe that GMC and SM2SC, which are more recent methods, have better performances than other compared methods in most evaluation metrics. In particular, SM2SC has the best performance of 78.36% in AR, which is just 1% higher than that obtained by our proposed method. Nonetheless, our proposed method has the best performance on the Yale dataset in all evaluation metrics, with 74.48% in ACC, 71.96% in NMI, 50.11% in AR, 51.84% in F-score, 50.42% in precision, and 59.18% in the recall metrics.Table 2
Clustering performance on face datasets.
DatasetsMethodsACCNMIARF-scorePrecisionRecallORLMLRSSC71.9587.2264.1865.0660.2570.81RMSL70.1684.4563.1965.7858.5765.32MCGC77.2585.6272.2773.6970.3679.39DiMSC78.8791.1973.5574.1969.4879.61MVGL74.5088.6374.3476.1469.0982.06GMC79.3590.5776.8971.5871.0581.98SM2SC80.5691.0878.3675.3670.8282.76Ours81.3991.3277.3675.7371.1383.10YaleMLRSSC64.3168.1847.5950.9448.6853.46RMSL65.4370.2147.2848.3545.1455.41MCGC68.0668.3745.8849.6343.6257.58DiMSC66.0667.3447.5750.8749.3652.48MVGL67.6770.7846.2749.8345.7954.67GMC74.0566.2847.5850.6648.2857.52SM2SC74.2670.4349.2351.1949.5358.31Ours74.4871.9650.1151.8450.4259.18Table3 illustrates the clustering performance on Coil-20 and Caltech 101-20 object datasets. Furthermore, our method has the best performance in all evaluation metrics, except for the Coil-20 dataset, where SM2SC has the best performance in ACC by a small margin of 0.29%.Table 3
Clustering performance on object datasets.
DatasetsMethodsACCNMIARF-scorePrecisionRecallCOIL-20MLRSSC83.3583.2569.0570.3668.2773.18RMSL80.7983.0462.4170.9469.1472.36MCGC88.2183.7869.9070.8465.5377.08DiMSC85.2382.7169.8671.3770.3072.48MVGL85.2195.9283.2582.1775.6494.88GMC87.0593.7283.6984.0377.3791.26SM2SC89.4295.7183.7684.2178.3094.81Ours89.1396.5484.0386.4978.5395.10Caltech 101-20MLRSSC55.4351.9146.0142.0683.7668.86RMSL57.9452.1544.6741.1681.3264.35MCGC64.5159.4746.5748.0688.9472.72DiMSC58.1247.7740.5947.6186.4767.42MVGL62.9558.8145.9646.7987.2572.13GMC65.2360.5242.8046.5189.1569.51SM2SC66.0460.4947.2149.3389.3072.29Ours66.4261.1448.7549.7689.4473.05Table4 shows that our proposed method outperforms other state-of-the-art methods in five evaluation metrics such as NMI and AR, where it has 86.76% and 78.42%, respectively.Table 4
Clustering performance on handwritten datasets.
DatasetsMethodsACCNMIARF-scorePrecisionRecallUCI digitsMLRSSC72.1281.2476.1578.6276.4481.11RMSL76.4681.3876.5477.2971.5883.46MCGC82.2082.5975.5378.1871.7985.82DiMSC82.3680.4175.1678.9476.6782.43MVGL84.2586.0475.7877.2868.0589.40GMC86.9485.9777.0579.8779.2388.15SM2SC87.2886.2377.5980.2578.3189.77Ours85.6186.7678.4281.9578.5690.25Overall, our method outperforms GMC, SM2SC, and MCGC only slightly, especially on object datasets, and it is not surprising because, for instance, SM2SC employed a block diagonal regularization to enforce a proper common structure. Similarly, GMC and MCGC utilized a rank constraint to find a unified clustering structure to balance the agreement between the different views by avoiding k-means spectral postprocessing of the low-dimensional embedding matrix as well. Notwithstanding, our proposed method outperforms other state-of-the-art methods by a wide margin in all evaluation metrics. Furthermore, we provide more intuitiveness of our approach by visualizing the clustering structure learned on three datasets in Figure 2. Accordingly, we can see clearly that our proposed method has a block diagonal clustering structure on all three datasets with an exact c number of connected components. Therefore, we conclude that our common low-rank learning approach is very efficient.Figure 2
Visualization of the clustering structure of our proposed method on three datasets. (a) ORL, (b) COIL-20, and (c) UCI digits.
(a)(b)(c)


## 4.4.2. Image Recognition with 0%, 10%, and 20% Levels of Corruptions

In this section, we present experimental results obtained for image recognition with respect to accuracy evaluation metric. In this experiment, we study the robustness against noise corruption of each algorithm by gradually injecting 10% and 20% noise into the five benchmark datasets. To perform this experiment, we keep the same parameter settings described in Section4.2, while the K nearest neighbor (KNN) classiﬁer is applied to evaluate the classiﬁcation accuracy of each algorithm. Thus, it is easily noticeable through Tables 5–9 that all algorithms had a degrading performance with an increment in the noise level. Specifically, our proposed method had degraded performance of only about 3–5% on all datasets with a 10% level of noise corruption. However, all algorithms had a significant reduction in performance with a 20% level of noise corruption. Yet, we can observe that our proposed method is generally more robust against noise than the other compared methods.Table 5
Variation of classification accuracy with different corruption rates on the ORL dataset.
Method rates0%10%20%MLRSSC72.5367.9851.83RMSL77.1873.3557.68MCGC76.0371.4962.35DiMSC80.5875.3064.91MVGL81.2977.7266.81GMC81.9878.0970.35SM2SC82.3977.5668.94Ours82.6479.8672.53Table 6
Variation of classification accuracy with different corruption rates on the Yale dataset.
Method rates0%10%20%MLRSSC71.8361.7848.62RMSL72.7962.9350.28MCGC71.2363.5749.68DiMSC73.3764.0351.26MVGL73.8467.3156.62GMC74.6968.0756.74SM2SC75.2168.3256.02Ours75.3068.9457.84Table 7
Variation of classification accuracy with different corruption rates on the COIL-20 dataset.
Method rates0%10%20%MLRSSC86.2481.8566.72RMSL87.8782.3868.54MCGC89.6485.6171.33DiMSC91.1387.0973.76MVGL90.4586.3673.03GMC92.1885.9574.52SM2SC92.2786.3473.65Ours92.7488.2976.15Table 8
Variation of classification accuracy with different corruption rates on the Caltech 101-20 dataset.
Method rates0%10%20%MLRSSC61.8256.4541.61RMSL63.5561.3045.77MCGC64.6059.4242.73DiMSC66.4662.9246.03MVGL67.7163.1345.51GMC69.3662.4544.77SM2SC68.8361.2046.45Ours68.2765.3548.87Table 9
Variation of classification accuracy with different corruption rates on the UCI digits dataset.
Method rates0%10%20%MLRSSC83.3377.5258.27RMSL81.7977.1560.84MCGC84.3179.6761.15DiMSC86.5882.3464.88MVGL86.2381.5564.51GMC87.0583.1465.28SM2SC88.3482.9866.87Ours87.1184.5768.32


## 4.5. Parameter Sensitivity

Figure3 demonstrates the clustering performance of our method with respect to NMI and ACC by varying λ1 and λ2 on Yale, COIL-20, and UCI digits datasets. First of all, we should note that λ3 does not need turning like the other parameters because we find its best value heuristically like [26] to accelerate the process. Therefore, we initialize λ3=1 and automatically set it as λ3=λ3∗2 or λ3  =  λ3/2 in each iteration when the number of connected components is smaller or larger than c, respectively. As a result, we can see from Figure 3 that our method is more sensitive to λ1 on both NMI and ACC because it controls the learning of the common low-rank matrix U. Hence, our proposed method is certain to have a good performance by finding a suitable value for λ1. On the other hand, our method is only slightly sensitive to λ2, which means that λ2 is not as important as λ1 to our model. Still, these clustering results show that our proposed method is relatively stable regardless of the dataset.Figure 3
The results of the clustering performance of our method with respect to NMI and ACC evaluation metrics by varying-parametersλ1 and λ2 on (a) Yale dataset, (b) COIL-20 dataset, and (c) UCI digits dataset.
(a)(b)(c)


## 4.6. Convergence Analysis

Although the convergence of the inexact augmented Lagrange multiplier (ALM) method with more than three subproblems is still not easy to theoretically prove [29], we compute the relative error of the term λ2S−UF2 to demonstrate the convergence behavior of our proposed method. To illustrate this better, Figure 4 shows the errors at different iterations. It is easy to see that on all three benchmark datasets, our method converges within 150 iterations.Figure 4
Visualization of the convergence behavior ofλ2S−UF2 for different iterations.


## 5. Conclusion

We proposed a novel method based on LRR, which learns a common low-rank representation matrix shared by two multi-instance datasets to improve the clustering performance accuracy. Specifically, our proposed method obtained this common low-rank representation matrix using only one instance of the data so that a second instance of the data can linearly reconstruct it. Thus, utilizing this common low-rank representation, our method obtained an ideal similarity matrix, which explicitly revealed the clustering structure without any spectral postprocessing. This approach is different from existing state-of-the-art methods, where a projection matrix is learned for each instance of the data to obtain a common structure. Extensive experiments on five benchmark datasets demonstrate our method's superiority over seven state-of-the-art compared methods in all six evaluation metrics. In future work, we would extend our ideas to deep multiview learning.



---

*Source: 6639582-2021-02-16.xml*