# Multiview Latent Representation Learning with Feature Diversity for Clustering

Authors: Sheng Wang; Liyong Chen; Furong Peng
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

To analyze and manage data, multiple features are extracted for robust and accurate description. Taking account of the complementary nature of multiview data, it is urgent to find a low-dimensional compact representation by leveraging multiple views and generating a better performance for clustering. In this paper, we present multiview latent representation learning with feature diversity for clustering (MvLRFD). To get rid of noise and maintain the information of each view, matrix factorization is adopted to get the latent representation. For fusing the diverse latent representations, we argue that the final representation is constructed by concatenating the latent representation of each view. With the help of the Hilbert–Schmidt Independence Criterion (HSIC), the diversity of features in the final representation is maximized to exploit the complementary information. A new coregularization strategy is introduced to maximize the distance between the similarity between data of the final representation and the latent one. The partition entropy regularization is adopted to control the uniformity level of the value of the similarity matrix and the weight of each view. To verify the availability of MvLRFD compared to state-of-the-art algorithms, several experiments are conducted on four real datasets.


---

## Body

## 1. Introduction

As a tool of classifying subjects into subgroups, clustering has been a basic step to achieve different goals of subsequent analytic tasks [1]. In real applications, data are often obtained under diverse situations or extracted by different feature extractors which may be coded in different ways. For example, one image is coded by its color or shape descriptors, web pages can be categorized by their content or hyperlinks, and so on. Usually, such kinds of data are called multiview data [2]. Observing that different views often reveal different characteristics in different perspectives; to get an accurate representation of multiview data, we should properly utilize the characteristics of all views (or multifeatures) to comprehensively and accurately describe the data.Recently, the multiview image clustering task has received increasing attention. Approximately, they can be roughly split into three classes: (1) CCA-based methods [3–5]; (2) matrix factorization-based methods [6, 7]; (3) subspace clustering learning methods [8, 9]. Canonical Correlation Analysis gets two mappings by making the projected data linearly correlated maximally. In view of the difficulty of clustering in high-dimensional space, Chaudhuri et al. [3] first adopt CCA to obtain discriminant information and obtain a compact representation. Since CCA is a method for two-view data, it cannot cope with multiview data. To pursue more discriminant information in multiview data, Wang et al. proposed an unsupervised discriminant canonical correlation analysis based on spectral clustering [4]. From the corresponding reference, CCA-based multiview representation learning algorithms get nice performance in multiview clustering. However, by nature, CCA assumes that the paired data are the same as each other in the latent space [3, 5], and CCA-based methods ignore the different fundamental attributes between views.The matrix factorization technique also has been introduced to the multiview learning task [6, 7, 10]. To pursue a common consensus of multiview data, Liu et al. proposed Multiview Clustering via Joint Nonnegative Matrix Factorization (MultiNMF) [6]. For comprehensive and accurate data representation, Wang et al. proposed diverse NMF (DiNMF) [7] which enhances the diversity of multiview data to reduce the redundancy of information. In DiNMF, the diversities are defined as the orthogonality of two views, and the final representation is defined as the average of the representations. However, the data of each view are coded in different ways and the representation of multiview data might not be comparable. Tang et al. proposed two different multiview methods [11, 12] for feature learning to pursue information across different views. Both of them aim to select a subset of original features and may maintain some noise.Recently, subspace clustering methods have been introduced to cope with multiview data. For exploring the complementary information among multiview data, Cao et al. proposed Diversity-induced Multiview Subspace Clustering (DiMSC) [13] which utilizes the Hilbert–Schmidt Independence Criterion (HSIC) as a diversity term to explore the complementary representations of multiview. Later, Tang et al. proposed two different multiview subspace clustering methods [14, 15]. Mi et al. proposed diversity and consistency embedding learning [16] for multiview subspace clustering. Graph-based Multiview Clustering (GMC) [17] aims to fuse graphs of multiview data to get a unified graph. Cross-view graph diffusion (CGD) [18] learns the unified graph through cross-view graph diffusion (CGD), which attempts to employ the diffusion process for multiview clustering. All of them take the original multiview features to pursue a fused graph and ignore the noise of the original data. Furthermore, the feature diversity of multiview data may also be lost during learning processing.In this paper, we aim to design a novel multiview representation algorithm, named Multiview Latent representation learning with feature diversity for clustering (MvLRFD). To wipe out the noise information from different descriptions, matrix factorization is employed to get the bases and the corresponding latent representations. Considering that different views describe data from different aspects, the final representation is constructed by concatenating the latent representations of multiview data. For perusing complementary information and eliminating redundant information, Hilbert–Schmidt Independence Criterion (HSIC) is adopted to measure the feature diversity among representations of different views. In addition, the low-dimensional manifold of multiview data should not be ignored. A new coregularization strategy is introduced to maximize the distance between the similarity of the final representation and the latent one. In the clustering task, an ideal similarity matrix S [19] is that Sij=1/nk if the cluster membership of i-th instance and j-th instance are the same; Sij=0, otherwise. Therefore, in MvLRFD, we attempt to harness the latent information of multiview data and learn the similarity matrix. By linear deduction, we know that the coregularization can be transformed to graph regularization. To maintain the manifold of the data, we argue that the value of Sij is nonzero, if data i and j locate the same neighborhood. We develop a novel optimization algorithm which alternately calculates the latent representations, the similarity matrix, and the weight vector efficiently. The experimental results on ORL face dataset, Yale faces dataset, and Multiple Feature dataset and MSRC-v1 dataset verify the effectiveness of MvLRFD.The content of this paper is organized as follows: the theory of the proposed method in Section2. In Section 3, the experiments are carried out on four datasets and comparisons and analysis are provided. Finally, the conclusions are presented in Section 4.


## 2. Multiview Latent Representation Learning with Feature Diversity for Clustering

### 2.1. Multiview Matrix Factorization

GivenN instance of multiview data, each instance has nv different descriptions xi1,xi2,…,xinv and each xij denotes a row vector. Xv∈RN×Di denotes the v-th view data matrix constructed by Xv=x1v;x2v;⋯;xNv. Wi∈Rdi×Di denotes as the basis of i-th view and Yi∈RN×di denotes the corresponding latent representation. As noted in [20, 21], the orthogonal projection matrix is good for maintaining discriminant information. Therefore, we constrain the basis of each view is orthogonal. Then, the basic objective function can be defined as follows:(1)minWii=1nv,Yii=1nv∑i=1nvXi−YiWi22+λYi22s.tWiWiT=I,i=1,…,nv.Here, Yi∈RN×di and Wi∈Rdi×Di.


### 2.2. Feature Diversity

The data of different views will describe the samples from different viewpoints. Each of them will maintain the consensus information. Data of the first view also contain such information and its specific information. Thus, complete information can be obtained by maximizing the diversity of the latent representations of multiview data.In this section, we will give a theoretical analysis of three correlation measures [7, 22, 23]. Given X∈RN×d1 and Y∈RN×d2, the correlation between them can be evaluated by the following three schemes:(a)
Correlation coefficient [22]:(2)ρxy=covx,yσxσy.(b)
HSIC measure [23]:(3)HSICZ,F,G=N−1−2trK1HK2H,whereK1 and K2 are the kernel matrices of X and Y using k1⋅,⋅ and k2⋅,⋅, Hij=δij−1/N.(c)
Inner product [7]:(4)DivX,Y=trXYT=∑iXi⋅Yi⋅T.Theory 1.
HSICX,Y (using linear kernel) measures the feature correlation between i-the column of X and j-column of Y.
By linear derivation, equation (3) can be transformed into the following equation:(5)HSICX,Y=trXXTHYYTH=trXTHHYXTHHYT=∑i,jX⋅iTHHY⋅j2.
From equation (2), we know that X⋅iTHHY⋅j measures the correlation of i-column of X and j-th column of Y. Thus, HSIC X,Y measures the feature diversity of V=X,Y. From the definition of the correlation coefficient, its value ranges from −1 to 1. Therefore, it is difficult to employ the correlation coefficient to minimize/maximize the correlation between two sets.Theory 2.
GivenX∈RN×D and Y∈RN×D, the inner product is proportional to the pairwise similarity between the data in X and Y.
By simple algebra deduction, equation (4) can be transformed into the following equation:(6)DivX,Y=trXYT=∑iXi⋅Yi⋅T=∑iXi⋅Yi⋅cos∠θ∠θ=Xi⋅,Yi⋅,Here, ∠θ=Xi⋅,Yi⋅ denotes the angle between Xi. and Yi.. XYTii is proportional to the value of the cosine distance between the i-th 1≤i≤N data of X and Y.
After getting the representations, we have two widely used fusion schemes to get the final representation.(7)FFSI:z1=xT,λyTT,FFSII:z2=x+y.
Experiments [22] show that the performance of FFS I is better than the second. Thus, we adopt FFS I to fuse the representations of multiview data. To balance the weight of each view, we give a weight αi for each view. Then, the final representation can be written as follows:(8)Y=α1Y1,α2Y2,…,αnvYnv.
From (7), we know that the i-th row of the final representation V can be seen as the i-th feature. To seek complementary information, we should maximize the diversity of features among the representations of multiview data. Thus, we adopt HSIC to measure the redundancy between Yi. The diversity of features can be written as follows:(9)∑j=1nv∑i=1j−1trHYiYiTHYjYjT,where Hij=δij−1/N.


### 2.3. Multiview Matrix Factorization with Coregularization

In this paper, we propose a novel coregularization strategy to encourage the data similarity of each view toward the latent similarity matrix of final representation. The cost function as a measure of agreement between the similarity of final representation and the latent similarity matrix is defined as follows:(10)DY,S=∑iαvYiYiT−SF2.GivenYi, it can be transformed into the following equation:(11)DY,S=∑i=1nvtrYiYiTS+nvtrSTS.For simplicity, we drop the second part of the above equation and get a compact similarity measure as follows:(12)DY,S=trYYTS=∑i=1nvtrYiYiTS.In equation (12), we aim to enforce the similarity of the final representations towards S. In references [24, 25], partition entropy is proposed as the metric of uncertainty within a partition of a finite set. It is calculated by the entropy of the corresponding probability distribution of the partition. In the following, we give some examples of entropy [24] as follows:(1)
Celebrated Shannon Entropy: Hs=∑i=1n−silogsi(2)
Generalized Entropies: Hs=t1−∑i=1nsiβ,t>0,β>1 and Hs=t∑i=1nsiβ−1,t>0,β>1(3)
l2-entropy: Hs=1−sTs. In nature, it is a generalized entropy β=2From the above discussion of partition entropy, we know that the similarity matrix learning method in paper [26] and its variants [27, 28] are regularized by a special partition entropy named l2-entropy in nature. In this paper, we adopt the partition entropy regularization to control the uniformity level of the value of the fused similarity matrix in our algorithms. To determine the weight of each view, we constrain ∑iαi=1 and introduce the entropy of αi into the objective function to control the uniformity level of the weights of each view. Thus, the objective function can be written as follows:(13)minWii=1nv,Yii=1nv,α,S∑i=1nvXi−YiWi22−λαitrYiYiT−αitrYiYiTS−η∑p=1NHSp⋅−σH′α+β∑j=1nv∑i=1j−1trHYiYiTHYjYjTs⋅tWiWiT=I,i=1,…,nv,∑iαi=1,Si.1=1,0≤Sij≤1,where α=α1,α2,…,αnvT.In MvLRFD, coregularization is adopted to harness the shared information between different representations and learn the similarity matrix of the low-dimensional manifold simultaneously. From equation (13), we know that the proposed coregularization can be transformed to graph regularization (if the sparse matrix S is defined in advance). Thus, in nature, the proposed coregularization also can be deemed as the latent manifold structure shared by multiview data.


### 2.4. Optimization

It is known thatminW,YX−WY22 is not convex. Thus, equation (13) is not joint convex. However, it is convex for each variable individually. To optimize the objective function, an iterative strategy is employed. Then, it can be divided into the following four steps:(a)
Step 1. Given S,αi and Yi, optimize Wi.Then, the objective function can be transformed into the following equation:(14)minWii=1nv,∑i=1nvXi−YiWi22s.tWiWiT=Ii=1,…nv.Fori-th view, by simple algebra derivation, the objective function can be written as follows:(15)maxWiXi−YiWi22=maxWitrXiTYiWi=maxWitrUDVTWi=maxWitrDVTWiU=maxWi∑j=1dDjjVTWiUjjs⋅tWiWiT=I,whereXiTYi=UDVT is the SVD of XiTYi. Djj denotes the j-th largest singular value of the diagonal matrix D, and Uj and Vj denote the left and right singular vectors corresponding to Djj.Hence, the solution can be written as follows:(16)Wi=∑j=1dUjVjT.(b)
Step 2. Given S,αi and Wi, optimize Yi.Fori-th view, by simple algebra derivation, the objective function can be written as follows:(17)minYiXi−YiWi22+λαitrYiTLYi+βtrYiT∑j=1i−1HYjYjTHYi=minYiXi−YiWi22+λtrYiTPhiYiwhereL=I−S;Phi=αiL+βλDive;Dive=∑j=1i−1HYjYjTH.For the above-given problem, it is easy to obtain the optimal solution by setting the derivative ofoYi with respect to Yi to zero.(18)δoYiδYi=−WiXiT+I+λPhiYiT=0.Then,Yi can be calculated using the following equation:(19)Yi=I+λPhi−1XiWiT.(c)
Step 3. Given S,Yi and Wi, optimize αi.In Section2.3, we list lots of different definitions of partition entropy. In this paper, to balance the weight of each view, we adopt the Shannon entropy to degrade the uncertainty of αiαi. Then, the objective function can be transformed into the following equation:(20)minα∑i=1nvαitrYiTLYi+σαilogαis⋅tα=α1,…,αnvT,αT1=1.Then, the Lagrangian function of equation (20) can be represented as follows:(21)Lα=αTa+σnv∑iαilogαi−ϕαT1−1−φTα,wherea denotes a column vector and. ai=trYiTLYi.By simple algebra derivation, we can get the following solution:(22)αi=exp−trYiTLYi/nvσ∑j=1nvexp−trYjTLYj/nvσ.(d)
Step 4. Given α,Yi and Wi, optimize S.In this paper, for simplicity, we adoptL2 entropy to degrade the uncertainty of Sij. Then, the objective function can be transformed into the following equation:(23)minS∑i=1nvαitrYiTLYi−nvη1−∑pqSpq2s⋅tSp⋅1=1,0≤Spq≤1.From equation (23), we know that Sp. is independent with Sq.,q≠p. Thus, we can solve the following problem individually for each p:(24)minS∑qdpqSpq−nvη1−Sp⋅22s⋅tSp⋅1=1,0≤Spq≤1.Here, dpq=∑iαidpqi=∑iαiYp⋅i−Yq⋅i22 denotes the L2-norm distance between data p and q.Then, the Lagrangian function of (24) can be represented as follows:(25)LS,ϕp,Φp=1212nvηΔp+Sp⋅2−ϕpSp⋅T1−1−ΦpTSp,whereΔp=dp1,dp2,…,dpn.According to the KKT condition, the optimalS can be obtained by the following equation:(26)Spq=−dpq2nvη+ϕp+BecauseS denotes a similarity matrix, we aim to find a sparse matrix S and set n−k elements to be 0. Taking η>0, we should select a ϕp that makes equation (26) satisfy the following equations:(27)Spq=−dpq2nvη+ϕp,ifxp∈Np,0,else.If the distance betweenp and the other points is ordered in ascending order and the index of the q-th data is smaller than k + 1, then we argue xq∈Np.Due to the constraintSp⋅1=1, we have the following equation:(28)∑q∈NpSpq=∑q∈Np−dpq2nvη+kϕp=1⇒ϕp=1k+∑q∈Npdpq2nvηk.Then, the optimalS can be obtained by the following equation:(29)Spq=1k+∑q∈Npdpq2nvηk−dpq2nvη+.The complete algorithm is listed as Algorithm 1.Algorithm 1: The algorithm of MvLRFD.
Input: The set of n views Xi=1n;The number of projected vectorsd;The number of classesC;The parameters:η,γ, β,δ, maxiter;Output: the final clustering result.(1)
Initialize: the latent similarityS;(2)
for each i∈1,maxiterdo(3)
Dive=0;(4)
for each iv∈1,nvdo(5)
Computer the basis ofi-th view using (16);(6)
Computer latent representation ofv-th view using (19);(7)
Update theDive using Dive=Dive+HYiYiTH;(8)
end for(9)
Update weight vectorα using (22);(10)
UpdateS using (29);(11)
Compute the value of the objective functionJt using (13);(12)
ifJt−J<1e−3‖iter>maxIterthen(13)
break;(14)
else(15)
J⟵Jt;(16)
end if(17)
iter⟵iter+1;(18)
end for(19)
return the final representation using (8);


### 2.5. Computational Complexity Analysis

The computational complexity of the three proposed algorithms is the same as each other because the first two steps are similar.The first step is to solve equation (15). It requires ncDi for computing XiTYi, and O2cDi2+ncDi for forming Wi. In total, it needs O∑i=1nv3ncDi+2Di2=O∑i=1nvDi2 for forming all Wi. In step (b), we need Ocni2+2cni to calculate Phi. To reduce complexity, we adopt the PCG algorithm to compute Yi. Thus, the complexity to form Yi is Kinner3Di+2nnzPhi (nnz (A) denotes the number of nonzero elements in A, Kinner denotes the iterations in the PCG method, and Kinner≪Di). Similarly to Wi, it requires Onicni2+2cni+Kinner3Di+2nnzPhi to form all YiYi. In step (c), it costs On2c+c2n+n for forming α. In step (d), we need ONv2n2c2+2N2+n2 for forming D and nlog2n+k+2 for forming S. Thus, the total time complexity of equation (13) is O=∑i=1nvDi2.


### 2.6. Extension to Generalized Entropy

Many other partition entropies have been reviewed in the literature [29]. In Section 2.3, L2 entropy is adopted to control the uniformity level of the edge weights in the graph, while Shannon entropy is introduced to control the uniformity level of the weight of each view. Since this definition of is very general, our method can be easily extended with different partition entropies. When generalized entropies are used, the optimization problem in steps (4) and (5) can be formulated as a quadratic programming problem (if the Lp-norm of S is relaxed by Si⋅TDSi⋅ [30]).


## 2.1. Multiview Matrix Factorization

GivenN instance of multiview data, each instance has nv different descriptions xi1,xi2,…,xinv and each xij denotes a row vector. Xv∈RN×Di denotes the v-th view data matrix constructed by Xv=x1v;x2v;⋯;xNv. Wi∈Rdi×Di denotes as the basis of i-th view and Yi∈RN×di denotes the corresponding latent representation. As noted in [20, 21], the orthogonal projection matrix is good for maintaining discriminant information. Therefore, we constrain the basis of each view is orthogonal. Then, the basic objective function can be defined as follows:(1)minWii=1nv,Yii=1nv∑i=1nvXi−YiWi22+λYi22s.tWiWiT=I,i=1,…,nv.Here, Yi∈RN×di and Wi∈Rdi×Di.


## 2.2. Feature Diversity

The data of different views will describe the samples from different viewpoints. Each of them will maintain the consensus information. Data of the first view also contain such information and its specific information. Thus, complete information can be obtained by maximizing the diversity of the latent representations of multiview data.In this section, we will give a theoretical analysis of three correlation measures [7, 22, 23]. Given X∈RN×d1 and Y∈RN×d2, the correlation between them can be evaluated by the following three schemes:(a)
Correlation coefficient [22]:(2)ρxy=covx,yσxσy.(b)
HSIC measure [23]:(3)HSICZ,F,G=N−1−2trK1HK2H,whereK1 and K2 are the kernel matrices of X and Y using k1⋅,⋅ and k2⋅,⋅, Hij=δij−1/N.(c)
Inner product [7]:(4)DivX,Y=trXYT=∑iXi⋅Yi⋅T.Theory 1.
HSICX,Y (using linear kernel) measures the feature correlation between i-the column of X and j-column of Y.
By linear derivation, equation (3) can be transformed into the following equation:(5)HSICX,Y=trXXTHYYTH=trXTHHYXTHHYT=∑i,jX⋅iTHHY⋅j2.
From equation (2), we know that X⋅iTHHY⋅j measures the correlation of i-column of X and j-th column of Y. Thus, HSIC X,Y measures the feature diversity of V=X,Y. From the definition of the correlation coefficient, its value ranges from −1 to 1. Therefore, it is difficult to employ the correlation coefficient to minimize/maximize the correlation between two sets.Theory 2.
GivenX∈RN×D and Y∈RN×D, the inner product is proportional to the pairwise similarity between the data in X and Y.
By simple algebra deduction, equation (4) can be transformed into the following equation:(6)DivX,Y=trXYT=∑iXi⋅Yi⋅T=∑iXi⋅Yi⋅cos∠θ∠θ=Xi⋅,Yi⋅,Here, ∠θ=Xi⋅,Yi⋅ denotes the angle between Xi. and Yi.. XYTii is proportional to the value of the cosine distance between the i-th 1≤i≤N data of X and Y.
After getting the representations, we have two widely used fusion schemes to get the final representation.(7)FFSI:z1=xT,λyTT,FFSII:z2=x+y.
Experiments [22] show that the performance of FFS I is better than the second. Thus, we adopt FFS I to fuse the representations of multiview data. To balance the weight of each view, we give a weight αi for each view. Then, the final representation can be written as follows:(8)Y=α1Y1,α2Y2,…,αnvYnv.
From (7), we know that the i-th row of the final representation V can be seen as the i-th feature. To seek complementary information, we should maximize the diversity of features among the representations of multiview data. Thus, we adopt HSIC to measure the redundancy between Yi. The diversity of features can be written as follows:(9)∑j=1nv∑i=1j−1trHYiYiTHYjYjT,where Hij=δij−1/N.


## 2.3. Multiview Matrix Factorization with Coregularization

In this paper, we propose a novel coregularization strategy to encourage the data similarity of each view toward the latent similarity matrix of final representation. The cost function as a measure of agreement between the similarity of final representation and the latent similarity matrix is defined as follows:(10)DY,S=∑iαvYiYiT−SF2.GivenYi, it can be transformed into the following equation:(11)DY,S=∑i=1nvtrYiYiTS+nvtrSTS.For simplicity, we drop the second part of the above equation and get a compact similarity measure as follows:(12)DY,S=trYYTS=∑i=1nvtrYiYiTS.In equation (12), we aim to enforce the similarity of the final representations towards S. In references [24, 25], partition entropy is proposed as the metric of uncertainty within a partition of a finite set. It is calculated by the entropy of the corresponding probability distribution of the partition. In the following, we give some examples of entropy [24] as follows:(1)
Celebrated Shannon Entropy: Hs=∑i=1n−silogsi(2)
Generalized Entropies: Hs=t1−∑i=1nsiβ,t>0,β>1 and Hs=t∑i=1nsiβ−1,t>0,β>1(3)
l2-entropy: Hs=1−sTs. In nature, it is a generalized entropy β=2From the above discussion of partition entropy, we know that the similarity matrix learning method in paper [26] and its variants [27, 28] are regularized by a special partition entropy named l2-entropy in nature. In this paper, we adopt the partition entropy regularization to control the uniformity level of the value of the fused similarity matrix in our algorithms. To determine the weight of each view, we constrain ∑iαi=1 and introduce the entropy of αi into the objective function to control the uniformity level of the weights of each view. Thus, the objective function can be written as follows:(13)minWii=1nv,Yii=1nv,α,S∑i=1nvXi−YiWi22−λαitrYiYiT−αitrYiYiTS−η∑p=1NHSp⋅−σH′α+β∑j=1nv∑i=1j−1trHYiYiTHYjYjTs⋅tWiWiT=I,i=1,…,nv,∑iαi=1,Si.1=1,0≤Sij≤1,where α=α1,α2,…,αnvT.In MvLRFD, coregularization is adopted to harness the shared information between different representations and learn the similarity matrix of the low-dimensional manifold simultaneously. From equation (13), we know that the proposed coregularization can be transformed to graph regularization (if the sparse matrix S is defined in advance). Thus, in nature, the proposed coregularization also can be deemed as the latent manifold structure shared by multiview data.


## 2.4. Optimization

It is known thatminW,YX−WY22 is not convex. Thus, equation (13) is not joint convex. However, it is convex for each variable individually. To optimize the objective function, an iterative strategy is employed. Then, it can be divided into the following four steps:(a)
Step 1. Given S,αi and Yi, optimize Wi.Then, the objective function can be transformed into the following equation:(14)minWii=1nv,∑i=1nvXi−YiWi22s.tWiWiT=Ii=1,…nv.Fori-th view, by simple algebra derivation, the objective function can be written as follows:(15)maxWiXi−YiWi22=maxWitrXiTYiWi=maxWitrUDVTWi=maxWitrDVTWiU=maxWi∑j=1dDjjVTWiUjjs⋅tWiWiT=I,whereXiTYi=UDVT is the SVD of XiTYi. Djj denotes the j-th largest singular value of the diagonal matrix D, and Uj and Vj denote the left and right singular vectors corresponding to Djj.Hence, the solution can be written as follows:(16)Wi=∑j=1dUjVjT.(b)
Step 2. Given S,αi and Wi, optimize Yi.Fori-th view, by simple algebra derivation, the objective function can be written as follows:(17)minYiXi−YiWi22+λαitrYiTLYi+βtrYiT∑j=1i−1HYjYjTHYi=minYiXi−YiWi22+λtrYiTPhiYiwhereL=I−S;Phi=αiL+βλDive;Dive=∑j=1i−1HYjYjTH.For the above-given problem, it is easy to obtain the optimal solution by setting the derivative ofoYi with respect to Yi to zero.(18)δoYiδYi=−WiXiT+I+λPhiYiT=0.Then,Yi can be calculated using the following equation:(19)Yi=I+λPhi−1XiWiT.(c)
Step 3. Given S,Yi and Wi, optimize αi.In Section2.3, we list lots of different definitions of partition entropy. In this paper, to balance the weight of each view, we adopt the Shannon entropy to degrade the uncertainty of αiαi. Then, the objective function can be transformed into the following equation:(20)minα∑i=1nvαitrYiTLYi+σαilogαis⋅tα=α1,…,αnvT,αT1=1.Then, the Lagrangian function of equation (20) can be represented as follows:(21)Lα=αTa+σnv∑iαilogαi−ϕαT1−1−φTα,wherea denotes a column vector and. ai=trYiTLYi.By simple algebra derivation, we can get the following solution:(22)αi=exp−trYiTLYi/nvσ∑j=1nvexp−trYjTLYj/nvσ.(d)
Step 4. Given α,Yi and Wi, optimize S.In this paper, for simplicity, we adoptL2 entropy to degrade the uncertainty of Sij. Then, the objective function can be transformed into the following equation:(23)minS∑i=1nvαitrYiTLYi−nvη1−∑pqSpq2s⋅tSp⋅1=1,0≤Spq≤1.From equation (23), we know that Sp. is independent with Sq.,q≠p. Thus, we can solve the following problem individually for each p:(24)minS∑qdpqSpq−nvη1−Sp⋅22s⋅tSp⋅1=1,0≤Spq≤1.Here, dpq=∑iαidpqi=∑iαiYp⋅i−Yq⋅i22 denotes the L2-norm distance between data p and q.Then, the Lagrangian function of (24) can be represented as follows:(25)LS,ϕp,Φp=1212nvηΔp+Sp⋅2−ϕpSp⋅T1−1−ΦpTSp,whereΔp=dp1,dp2,…,dpn.According to the KKT condition, the optimalS can be obtained by the following equation:(26)Spq=−dpq2nvη+ϕp+BecauseS denotes a similarity matrix, we aim to find a sparse matrix S and set n−k elements to be 0. Taking η>0, we should select a ϕp that makes equation (26) satisfy the following equations:(27)Spq=−dpq2nvη+ϕp,ifxp∈Np,0,else.If the distance betweenp and the other points is ordered in ascending order and the index of the q-th data is smaller than k + 1, then we argue xq∈Np.Due to the constraintSp⋅1=1, we have the following equation:(28)∑q∈NpSpq=∑q∈Np−dpq2nvη+kϕp=1⇒ϕp=1k+∑q∈Npdpq2nvηk.Then, the optimalS can be obtained by the following equation:(29)Spq=1k+∑q∈Npdpq2nvηk−dpq2nvη+.The complete algorithm is listed as Algorithm 1.Algorithm 1: The algorithm of MvLRFD.
Input: The set of n views Xi=1n;The number of projected vectorsd;The number of classesC;The parameters:η,γ, β,δ, maxiter;Output: the final clustering result.(1)
Initialize: the latent similarityS;(2)
for each i∈1,maxiterdo(3)
Dive=0;(4)
for each iv∈1,nvdo(5)
Computer the basis ofi-th view using (16);(6)
Computer latent representation ofv-th view using (19);(7)
Update theDive using Dive=Dive+HYiYiTH;(8)
end for(9)
Update weight vectorα using (22);(10)
UpdateS using (29);(11)
Compute the value of the objective functionJt using (13);(12)
ifJt−J<1e−3‖iter>maxIterthen(13)
break;(14)
else(15)
J⟵Jt;(16)
end if(17)
iter⟵iter+1;(18)
end for(19)
return the final representation using (8);


## 2.5. Computational Complexity Analysis

The computational complexity of the three proposed algorithms is the same as each other because the first two steps are similar.The first step is to solve equation (15). It requires ncDi for computing XiTYi, and O2cDi2+ncDi for forming Wi. In total, it needs O∑i=1nv3ncDi+2Di2=O∑i=1nvDi2 for forming all Wi. In step (b), we need Ocni2+2cni to calculate Phi. To reduce complexity, we adopt the PCG algorithm to compute Yi. Thus, the complexity to form Yi is Kinner3Di+2nnzPhi (nnz (A) denotes the number of nonzero elements in A, Kinner denotes the iterations in the PCG method, and Kinner≪Di). Similarly to Wi, it requires Onicni2+2cni+Kinner3Di+2nnzPhi to form all YiYi. In step (c), it costs On2c+c2n+n for forming α. In step (d), we need ONv2n2c2+2N2+n2 for forming D and nlog2n+k+2 for forming S. Thus, the total time complexity of equation (13) is O=∑i=1nvDi2.


## 2.6. Extension to Generalized Entropy

Many other partition entropies have been reviewed in the literature [29]. In Section 2.3, L2 entropy is adopted to control the uniformity level of the edge weights in the graph, while Shannon entropy is introduced to control the uniformity level of the weight of each view. Since this definition of is very general, our method can be easily extended with different partition entropies. When generalized entropies are used, the optimization problem in steps (4) and (5) can be formulated as a quadratic programming problem (if the Lp-norm of S is relaxed by Si⋅TDSi⋅ [30]).


## 3. Experiments

To verify the effectiveness of MvLRFD, we compare it with several state-of-the-art multiview learning methods on four famous datasets. In this section, firstly, we will detail the datasets. Then, a brief description of the compared methods and parameters setting will be given. Thirdly, the performance of each method will be reported, and experiments analysis will be given. Finally, parameter sensitivity analysis and convergence are provided to illustrate the properties of MvLRFD.Figure 1
Ten selected images of one class in each dataset.


### 3.1. Dataset Description

ORL Face Dataset. ORL face database [31] contains 400 images of 40 people, each of whom has 10 images. Ten selected images are shown in the first row in Figure 1. In this experiment, we select the raw pixel as one kind of feature and Gabor feature and LBP feature are extracted following Huang’s approach [32].Yale Face Dataset. The Yale face dataset [4] contains 165 grayscale images of 15 individuals. Each person has 11 images. Ten images are shown in the second row in Figure 1. For simplicity, we adopt the similar feature extracting strategy of the ORL face dataset.MFD Dataset. The Multiple Feature dataset (MFD) [33] consists of handwritten digit images from 0 to 9 and each letter has 200 images. Ten images of the first class are shown in the third row in Figure 1. For the case that the dimensionalities of different views are unbalanced, we just select the 6-D morphological feature, 47-D Zernike moments, and 76-D Fourier-coefficient feature as three views.MSRC-v1 Dataset. MSRC-v1 dataset [34] contains 9 object classes and is composed of 240 images. In this experiment, we follow Lee’s approach [35] to refine the dataset, getting 7 classes composed of a tree, building, airplane, cow, face, car, and bicycle, and each refined class has 30 images. Ten images of an airplane are shown in the fourth row in Figure 1. For multiview learning experiments, we follow Cai’s approach [36] to extract the features of images from this dataset. For simplicity, we just extract three kinds of features from each image: LBP [37], GIST [38], and HOG [39].


### 3.2. Comparison of Algorithms and Parameter Setting

To verify the performance of MvLRFD, we compare them with several state-of-the-art related algorithms. Brief descriptions of the compared methods are given as follows:(1)
NMF-Best [40]. The standard Non-negative Matrix Factorization is executed on each view, and the best performance is reported.(2)
MultiNMF [6]. MultiNMF is a multiview version of NMF, which encourages the final representations of multiview data towards a consensus.(3)
DiNMF [7]. Diverse NMF(DiNMF) aims to exploit the diversity of multiple views for comprehensive data representations. Furthermore, they proposed locality preserved DiNMF (LP-DiNMF) to maintain the local geometry structure of data in each view.(4)
DiMSC [13]. Diversity-induced Multiview Subspace Clustering (DiMSC) extends utilize HSIC to exploit the complementary information of multiview.(5)
GMC [41]. Graph-based Multiview Clustering (GMC) aims to learn a unified graph matrix using the data graph matrix of multiview data.(6)
CGD [18]. Multiview data clustering via cross-view graph diffusion (CGD) attempts to employ the diffusion process to learn the fused graph.(7)
MvLRFD-gen. MvLRFD-gen denotes the performance of the proposed method which adopts generalized entropy to constrain the weight of the central similarity matrix and the weight of each view (whereβ is set as 2).(8)
MvLRFD-sha. MvLRFD-sha denotes the algorithm of MvLRFD which adopts Shannon entropy.The parameters for each compared method are set according to their references. For simplicity, we just setY∈Rn×c (if Di<c, we set Y∈Rn×Di−1) for each method. The value of η and the initial of S are set using the same strategy as on paper [26]. For the parameters λ,β,δ in MvLRFD, they are searched on the grid 2−5,2−4,…,25. The final dimension of the final representation of MultiNMF is set C−1. That of MvLRFD is set as Di−1 where Di<C, and C−1 vice versa.After obtaining the final representation, the standard K-means algorithm is implemented to obtain the final clustering results, and each experiment will be repeated 20 times. The average values are taken as final results; the best performances are reported with the best parameter configuration.


### 3.3. Performance Evaluations and Comparisons

To evaluate the clustering performance, six widely used metrics clustering accuracy (ACC), normalized mutual information (NMI), purity, precision, F-score, and adjusted rand index (ARI) are adopted. Table1 gives the clustering results on four datasets, respectively.Table 1
Clustering result (mean (%)± standard deviation) on four datasets.
MethodACCNMIPurityPrecisionF-scoreARIORLNMF-best56.50± 0.0271.29± 0.0153.50± 0.0230.99± 0.0135.44± 0.0233.73± 0.02MultiNMF79.50± 0.0285.88± 0.0174.25± 0.0254.86± 0.0561.53± 0.0460.53± 0.04LP-DiNMF76.25± 0.0085.13± 0.0074.75± 0.0055.91± 0.0060.93± 0.0059.95± 0.00MDcR80.00± 0.0185.58± 0.0176.00± 0.0245.62± 0.0255.70± 0.0254.44± 0.02DiMSC41.05± 0.0259.35± 0.0143.35± 0.0220.42± 0.0121.99± 0.0120.04± 0.01GMC93.25± 0.0090.89± 0.0086.00± 0.0054.23± 0.0066.89± 0.0065.95± 0.00CGD66.61± 0.0066.49± 0.0061.21± 0.0046.79± 0.0049.31± 0.0045.83± 0.00MvLRFD-sha88.00± 0.0392.30± 0.0287.50± 0.0376.53± 0.0379.96± 0.0379.48± 0.03MvLRFD-gen94.75± 0.0195.74± 0.0191.00± 0.0378.60± 0.0584.54± 0.0484.15± 0.04MvLRFD95.75± 0.0196.64± 0.0092.00± 0.0283.95± 0.0488.52± 0.0288.24± 0.03YaleNMF-best63.03± 0.0263.49± 0.0257.57± 0.0238.67± 0.0341.11± 0.0337.03± 0.03MultiNMF62.60± 0.0257.45± 0.0253.93± 0.0334.07± 0.0237.14± 0.0232.66± 0.02LP-DiNMF76.97± 0.0071.76± 0.0065.45± 0.0052.67± 0.0055.40± 0.0052.34± 0.00MDcR80.60± 0.0371.81± 0.0366.06± 0.0340.35± 0.0349.46± 0.0245.37± 0.03DiMSC55.94± 0.0360.83± 0.0256.58± 0.0339.16± 0.0341.14± 0.0337.11± 0.03GMC67.27± 0.0067.05± 0.0067.27± 0.0039.92± 0.0047.23± 0.0043.12± 0.00CGD67.00± 0.0082.29± 0.0073.00± 0.0038.84± 0.0051.10± 0.0049.61± 0.00MvLRFD-sha90.30±0.0488.50±0.0486.66± 0.0675.33± 0.0678.77± 0.0677.32± 0.06MvLRFD-gen86.06± 0.0383.31± 0.0281.21± 0.0366.34± 0.0470.09± 0.0468.03± 0.04MvLRFD87.27± 0.0484.26± 0.0482.42± 0.0568.93± 0.0672.35± 0.0670.45± 0.06MFDNMF-best62.20± 0.0258.84± 0.0162.25± 0.0249.11± 0.0250.08± 0.0144.44± 0.02MultiNMF67.00± 0.0157.75± 0.0164.70± 0.0248.00± 0.0151.38± 0.0145.58± 0.01LP-DiNMF68.15± 0.0060.71± 0.0065.35± 0.0054.35± 0.0054.86± 0.0049.82± 0.00MDcR77.35± 0.0471.01± 0.0275.50± 0.0463.79± 0.0565.24± 0.0561.30± 0.05DiMSC80.00± 0.0185.58± 0.0176.00± 0.0245.62± 0.0255.70± 0.0254.44± 0.02GMC74.80± 0.0080.61± 0.0077.40± 0.0066.84± 0.0073.94± 0.0070.72± 0.00CGD65.25± 0.0067.78± 0.0069.70± 0.0056.58± 0.0059.55± 0.0054.82± 0.00MvLRFD-sha93.50± 0.0287.19± 0.0293.50± 0.0587.27± 0.0787.51± 0.0586.13± 0.06MvLRFD-gen94.15± 0.0388.30± 0.0394.15± 0.0588.66± 0.0788.77± 0.0687.52± 0.07MvLRFD95.30± 0.0491.07± 0.0495.30± 0.0590.64± 0.0690.95± 0.0589.94± 0.06MSRC-v1NMF-best73.33± 0.0162.71± 0.0171.42± 0.0448.78± 0.0354.60± 0.0246.25± 0.02MultiNMF74.19± 0.0365.07± 0.0173.08± 0.0357.23± 0.0460.26± 0.0253.46± 0.03LP-DiNMF80.95± 0.0070.46± 0.0080.95± 0.0063.81± 0.0066.69± 0.0061.04± 0.00MDcR78.57± 0.0367.89± 0.0273.33± 0.0347.80± 0.0356.79± 0.0348.26± 0.03DiMSC30.69± 0.0119.96± 0.0231.74± 0.0120.58± 0.0121.29± 0.018.09± 0.01GMC51.90± 0.0054.56± 0.0060.48± 0.0033.32± 0.0045.46± 0.0032.73± 0.00CGD55.23± 0.0057.80± 0.0063.33± 0.0036.13± 0.0048.65± 0.0036.86± 0.00MvLRFD-sha82.85± 0.0475.04± 0.0480.47± 0.0564.21± 0.0669.69± 0.0564.31± 0.06MvLRFD-gen81.90± 0.0071.21± 0.0178.09± 0.0260.96± 0.0466.38± 0.0260.40± 0.03MvLRFD90.00± 0.0080.72± 0.0090.00± 0.0080.66± 0.0081.16± 0.0078.11± 0.00These experiments reveal a number of interesting points:(1)
From Table1, we know that most multiview learning methods perform better than NMF. This indicates that the representation of multiview learning methods have a more powerful description than only using one single view.(2)
To explore the capacity to learn a high quality similarity matrix, the true class similarity matrix and the latent similarity matrix of MvLRFD are shown in Figure2. From Figures 2(a) and 2(b), at the beginning, we can get a good similarity matrix. During the iteration, the similarity matrix will be more similar to the true class similarity matrix and gradually become the same with each other. Furthermore, the calculated similarity matrix in Figure 2(f) is nearly the same as the true one in Figure 2(a).(3)
To verify the nonlinear structure of the learned latent representation, we provide the low-dimensional visualization of the MFD dataset and ORL face dataset with three views using t-Distributed Stochastic Neighbor Embedding (t-SNE). We visualize in Figures3 and 4 where each color represents a cluster. The structure of each view is presented in Figures 3(a), 3(c)4(a) and 4(c). Figures 3(d) and 4(d) show the distribution of the combined final representation of MvLRFD. We concluded that the proposed method can discover the geometrical structure generally.(4)
Due to maintaining the nonlinear characteristics and the geometrical of multiview data, GCD, GMD, LP-DiNMF, MDCR, and MvLRFD perform better than the other algorithms. This verifies the importance of the geometrical structure in feature learning. Instead of using a graph constructed in advance as LP-DiNMF, MvLRFD learns the representation joint the graph which brings better performance in the clustering task.(5)
Compared with the other methods, DiMSC and MvLRFD consider the private information of each view and obtain comprehensive information which results in better performance. This suggests the importance of private information of each view.(6)
In this paper, to testify to the effectiveness of the constraint of partition entropy, we also exploit the performance of MvLRFD constrained by Generalized Partition Entropy (denoted by MvLRFD-gen) and Shannon Partition Entropy (MvLRFD-sha), respectively. From Table1, we know that both of the two extensions also have mostly better performance than the other methods (except MvLRFD).(7)
Compared to CGD, GMD, and DiMSC, MvLRFD achieves better clustering performance which demonstrates the effectiveness of matrix factorization-based methods.Figure 2
The similarity of MvLRFD during iteration on the ORL dataset. (a) True similarity. (b) 1st iteration. (c) 5th iteration. (d) 10st iteration. (e) 15st iteration. (f) 19th iteration.
(a)(b)(c)(d)(e)(f)Figure 3
Visualization of the clustering results with t-SNE on the MFD dataset. (a) View 1, (b) view 2, (c) view 3, and (d) the final fusion of 3 views.
(a)(b)(c)(d)Figure 4
Visualization of the clustering results with t-SNE on the ORL dataset. (a) View 1, (b) view 2, (c) view 3, and (d) the final fusion of 3 views.
(a)(b)(c)(d)


### 3.4. Parameter Sensitivity Analysis and Convergence

In MvLRFD, there are three critical parametersλ,σ, and β. λ controls the strength of the latent manifold information. β controls the strength of diversity encoding in the latent representation. σ controls the weight of partition entropy for α.When the parameters are changed, our method will obtain different results. We explore the influence ofσ by varying it from 2−5 to 25 and plot the clustering accuracy (ACC) of MvLRFD on the ORL dataset in Figure 5(a) with λ=21,β=2−4. From equation (22), we know that σ has two important properties. Firstly, if σ⟶0, the value of −trYiTLYi/σ tends to −∞, and each weight will be equal to each other. If δ⟶∞, the value of −trYiTLYi/σ is trending to 0. Similarly, each weight of each view will be equal to each other. Therefore, we can find that performance will be degraded when δ<23 and σ>21. In the second case, if 2−2≤σ≤23, the weight will be decided by the value of −trYiTLYi which characterizes the local geometric structure in the i-th view. From equation (13), we know that the value of −trYiTLYi−trYiTLYi also reflects the similarity between YiYiT and the latent similarity matrix. As a result, if the value of −trYiTLYi−trYiTLYi is smaller, it will give more weight to Yi in the learning matrix of latent similarity. Thus, when 21≤σ≤23, the performance of MvLRFD will be upgraded by the proposed coregularization.Figure 5
Clustering results vs.σ,λ, and β on ORL dataset. (a) ACC vs σ. (b) ACC vs λ. (c) ACC vs β.
(a)(b)(c)Withσ=22 and β=2−3, we study the impact of λ by varying it in the grid of 2−5,2−4,…,25 on the ORL dataset and plot ACC curve vs. parameter λ. It can be seen that the clustering accuracy gradually increases with increasing λλ>2−3and remains stable when 21≤λ≤23. This pattern is explainable since a higher value of λ will force MvLRFD to focus more on fused information between views.Withσ=22 and λ=22, our method will obtain a better performance for maintaining private information when 2−2≥β≥2−4. If β>2−2, it will force MvLRFD to focus more on private information.Figures6(a)–6(d) show the value of our objective function equation (13) vs. iterations. Obviously, we can find the quick convergent property of the iterative algorithm on ORL, Yale, MFD, and MSRC-v1 datasets.Figure 6
Convergence curves of MvLRFD on four datasets. (a) ORL. (b) Yale. (c) MFD. (d) MSRC-v1.
(a)(b)(c)(d)


## 3.1. Dataset Description

ORL Face Dataset. ORL face database [31] contains 400 images of 40 people, each of whom has 10 images. Ten selected images are shown in the first row in Figure 1. In this experiment, we select the raw pixel as one kind of feature and Gabor feature and LBP feature are extracted following Huang’s approach [32].Yale Face Dataset. The Yale face dataset [4] contains 165 grayscale images of 15 individuals. Each person has 11 images. Ten images are shown in the second row in Figure 1. For simplicity, we adopt the similar feature extracting strategy of the ORL face dataset.MFD Dataset. The Multiple Feature dataset (MFD) [33] consists of handwritten digit images from 0 to 9 and each letter has 200 images. Ten images of the first class are shown in the third row in Figure 1. For the case that the dimensionalities of different views are unbalanced, we just select the 6-D morphological feature, 47-D Zernike moments, and 76-D Fourier-coefficient feature as three views.MSRC-v1 Dataset. MSRC-v1 dataset [34] contains 9 object classes and is composed of 240 images. In this experiment, we follow Lee’s approach [35] to refine the dataset, getting 7 classes composed of a tree, building, airplane, cow, face, car, and bicycle, and each refined class has 30 images. Ten images of an airplane are shown in the fourth row in Figure 1. For multiview learning experiments, we follow Cai’s approach [36] to extract the features of images from this dataset. For simplicity, we just extract three kinds of features from each image: LBP [37], GIST [38], and HOG [39].


## 3.2. Comparison of Algorithms and Parameter Setting

To verify the performance of MvLRFD, we compare them with several state-of-the-art related algorithms. Brief descriptions of the compared methods are given as follows:(1)
NMF-Best [40]. The standard Non-negative Matrix Factorization is executed on each view, and the best performance is reported.(2)
MultiNMF [6]. MultiNMF is a multiview version of NMF, which encourages the final representations of multiview data towards a consensus.(3)
DiNMF [7]. Diverse NMF(DiNMF) aims to exploit the diversity of multiple views for comprehensive data representations. Furthermore, they proposed locality preserved DiNMF (LP-DiNMF) to maintain the local geometry structure of data in each view.(4)
DiMSC [13]. Diversity-induced Multiview Subspace Clustering (DiMSC) extends utilize HSIC to exploit the complementary information of multiview.(5)
GMC [41]. Graph-based Multiview Clustering (GMC) aims to learn a unified graph matrix using the data graph matrix of multiview data.(6)
CGD [18]. Multiview data clustering via cross-view graph diffusion (CGD) attempts to employ the diffusion process to learn the fused graph.(7)
MvLRFD-gen. MvLRFD-gen denotes the performance of the proposed method which adopts generalized entropy to constrain the weight of the central similarity matrix and the weight of each view (whereβ is set as 2).(8)
MvLRFD-sha. MvLRFD-sha denotes the algorithm of MvLRFD which adopts Shannon entropy.The parameters for each compared method are set according to their references. For simplicity, we just setY∈Rn×c (if Di<c, we set Y∈Rn×Di−1) for each method. The value of η and the initial of S are set using the same strategy as on paper [26]. For the parameters λ,β,δ in MvLRFD, they are searched on the grid 2−5,2−4,…,25. The final dimension of the final representation of MultiNMF is set C−1. That of MvLRFD is set as Di−1 where Di<C, and C−1 vice versa.After obtaining the final representation, the standard K-means algorithm is implemented to obtain the final clustering results, and each experiment will be repeated 20 times. The average values are taken as final results; the best performances are reported with the best parameter configuration.


## 3.3. Performance Evaluations and Comparisons

To evaluate the clustering performance, six widely used metrics clustering accuracy (ACC), normalized mutual information (NMI), purity, precision, F-score, and adjusted rand index (ARI) are adopted. Table1 gives the clustering results on four datasets, respectively.Table 1
Clustering result (mean (%)± standard deviation) on four datasets.
MethodACCNMIPurityPrecisionF-scoreARIORLNMF-best56.50± 0.0271.29± 0.0153.50± 0.0230.99± 0.0135.44± 0.0233.73± 0.02MultiNMF79.50± 0.0285.88± 0.0174.25± 0.0254.86± 0.0561.53± 0.0460.53± 0.04LP-DiNMF76.25± 0.0085.13± 0.0074.75± 0.0055.91± 0.0060.93± 0.0059.95± 0.00MDcR80.00± 0.0185.58± 0.0176.00± 0.0245.62± 0.0255.70± 0.0254.44± 0.02DiMSC41.05± 0.0259.35± 0.0143.35± 0.0220.42± 0.0121.99± 0.0120.04± 0.01GMC93.25± 0.0090.89± 0.0086.00± 0.0054.23± 0.0066.89± 0.0065.95± 0.00CGD66.61± 0.0066.49± 0.0061.21± 0.0046.79± 0.0049.31± 0.0045.83± 0.00MvLRFD-sha88.00± 0.0392.30± 0.0287.50± 0.0376.53± 0.0379.96± 0.0379.48± 0.03MvLRFD-gen94.75± 0.0195.74± 0.0191.00± 0.0378.60± 0.0584.54± 0.0484.15± 0.04MvLRFD95.75± 0.0196.64± 0.0092.00± 0.0283.95± 0.0488.52± 0.0288.24± 0.03YaleNMF-best63.03± 0.0263.49± 0.0257.57± 0.0238.67± 0.0341.11± 0.0337.03± 0.03MultiNMF62.60± 0.0257.45± 0.0253.93± 0.0334.07± 0.0237.14± 0.0232.66± 0.02LP-DiNMF76.97± 0.0071.76± 0.0065.45± 0.0052.67± 0.0055.40± 0.0052.34± 0.00MDcR80.60± 0.0371.81± 0.0366.06± 0.0340.35± 0.0349.46± 0.0245.37± 0.03DiMSC55.94± 0.0360.83± 0.0256.58± 0.0339.16± 0.0341.14± 0.0337.11± 0.03GMC67.27± 0.0067.05± 0.0067.27± 0.0039.92± 0.0047.23± 0.0043.12± 0.00CGD67.00± 0.0082.29± 0.0073.00± 0.0038.84± 0.0051.10± 0.0049.61± 0.00MvLRFD-sha90.30±0.0488.50±0.0486.66± 0.0675.33± 0.0678.77± 0.0677.32± 0.06MvLRFD-gen86.06± 0.0383.31± 0.0281.21± 0.0366.34± 0.0470.09± 0.0468.03± 0.04MvLRFD87.27± 0.0484.26± 0.0482.42± 0.0568.93± 0.0672.35± 0.0670.45± 0.06MFDNMF-best62.20± 0.0258.84± 0.0162.25± 0.0249.11± 0.0250.08± 0.0144.44± 0.02MultiNMF67.00± 0.0157.75± 0.0164.70± 0.0248.00± 0.0151.38± 0.0145.58± 0.01LP-DiNMF68.15± 0.0060.71± 0.0065.35± 0.0054.35± 0.0054.86± 0.0049.82± 0.00MDcR77.35± 0.0471.01± 0.0275.50± 0.0463.79± 0.0565.24± 0.0561.30± 0.05DiMSC80.00± 0.0185.58± 0.0176.00± 0.0245.62± 0.0255.70± 0.0254.44± 0.02GMC74.80± 0.0080.61± 0.0077.40± 0.0066.84± 0.0073.94± 0.0070.72± 0.00CGD65.25± 0.0067.78± 0.0069.70± 0.0056.58± 0.0059.55± 0.0054.82± 0.00MvLRFD-sha93.50± 0.0287.19± 0.0293.50± 0.0587.27± 0.0787.51± 0.0586.13± 0.06MvLRFD-gen94.15± 0.0388.30± 0.0394.15± 0.0588.66± 0.0788.77± 0.0687.52± 0.07MvLRFD95.30± 0.0491.07± 0.0495.30± 0.0590.64± 0.0690.95± 0.0589.94± 0.06MSRC-v1NMF-best73.33± 0.0162.71± 0.0171.42± 0.0448.78± 0.0354.60± 0.0246.25± 0.02MultiNMF74.19± 0.0365.07± 0.0173.08± 0.0357.23± 0.0460.26± 0.0253.46± 0.03LP-DiNMF80.95± 0.0070.46± 0.0080.95± 0.0063.81± 0.0066.69± 0.0061.04± 0.00MDcR78.57± 0.0367.89± 0.0273.33± 0.0347.80± 0.0356.79± 0.0348.26± 0.03DiMSC30.69± 0.0119.96± 0.0231.74± 0.0120.58± 0.0121.29± 0.018.09± 0.01GMC51.90± 0.0054.56± 0.0060.48± 0.0033.32± 0.0045.46± 0.0032.73± 0.00CGD55.23± 0.0057.80± 0.0063.33± 0.0036.13± 0.0048.65± 0.0036.86± 0.00MvLRFD-sha82.85± 0.0475.04± 0.0480.47± 0.0564.21± 0.0669.69± 0.0564.31± 0.06MvLRFD-gen81.90± 0.0071.21± 0.0178.09± 0.0260.96± 0.0466.38± 0.0260.40± 0.03MvLRFD90.00± 0.0080.72± 0.0090.00± 0.0080.66± 0.0081.16± 0.0078.11± 0.00These experiments reveal a number of interesting points:(1)
From Table1, we know that most multiview learning methods perform better than NMF. This indicates that the representation of multiview learning methods have a more powerful description than only using one single view.(2)
To explore the capacity to learn a high quality similarity matrix, the true class similarity matrix and the latent similarity matrix of MvLRFD are shown in Figure2. From Figures 2(a) and 2(b), at the beginning, we can get a good similarity matrix. During the iteration, the similarity matrix will be more similar to the true class similarity matrix and gradually become the same with each other. Furthermore, the calculated similarity matrix in Figure 2(f) is nearly the same as the true one in Figure 2(a).(3)
To verify the nonlinear structure of the learned latent representation, we provide the low-dimensional visualization of the MFD dataset and ORL face dataset with three views using t-Distributed Stochastic Neighbor Embedding (t-SNE). We visualize in Figures3 and 4 where each color represents a cluster. The structure of each view is presented in Figures 3(a), 3(c)4(a) and 4(c). Figures 3(d) and 4(d) show the distribution of the combined final representation of MvLRFD. We concluded that the proposed method can discover the geometrical structure generally.(4)
Due to maintaining the nonlinear characteristics and the geometrical of multiview data, GCD, GMD, LP-DiNMF, MDCR, and MvLRFD perform better than the other algorithms. This verifies the importance of the geometrical structure in feature learning. Instead of using a graph constructed in advance as LP-DiNMF, MvLRFD learns the representation joint the graph which brings better performance in the clustering task.(5)
Compared with the other methods, DiMSC and MvLRFD consider the private information of each view and obtain comprehensive information which results in better performance. This suggests the importance of private information of each view.(6)
In this paper, to testify to the effectiveness of the constraint of partition entropy, we also exploit the performance of MvLRFD constrained by Generalized Partition Entropy (denoted by MvLRFD-gen) and Shannon Partition Entropy (MvLRFD-sha), respectively. From Table1, we know that both of the two extensions also have mostly better performance than the other methods (except MvLRFD).(7)
Compared to CGD, GMD, and DiMSC, MvLRFD achieves better clustering performance which demonstrates the effectiveness of matrix factorization-based methods.Figure 2
The similarity of MvLRFD during iteration on the ORL dataset. (a) True similarity. (b) 1st iteration. (c) 5th iteration. (d) 10st iteration. (e) 15st iteration. (f) 19th iteration.
(a)(b)(c)(d)(e)(f)Figure 3
Visualization of the clustering results with t-SNE on the MFD dataset. (a) View 1, (b) view 2, (c) view 3, and (d) the final fusion of 3 views.
(a)(b)(c)(d)Figure 4
Visualization of the clustering results with t-SNE on the ORL dataset. (a) View 1, (b) view 2, (c) view 3, and (d) the final fusion of 3 views.
(a)(b)(c)(d)


## 3.4. Parameter Sensitivity Analysis and Convergence

In MvLRFD, there are three critical parametersλ,σ, and β. λ controls the strength of the latent manifold information. β controls the strength of diversity encoding in the latent representation. σ controls the weight of partition entropy for α.When the parameters are changed, our method will obtain different results. We explore the influence ofσ by varying it from 2−5 to 25 and plot the clustering accuracy (ACC) of MvLRFD on the ORL dataset in Figure 5(a) with λ=21,β=2−4. From equation (22), we know that σ has two important properties. Firstly, if σ⟶0, the value of −trYiTLYi/σ tends to −∞, and each weight will be equal to each other. If δ⟶∞, the value of −trYiTLYi/σ is trending to 0. Similarly, each weight of each view will be equal to each other. Therefore, we can find that performance will be degraded when δ<23 and σ>21. In the second case, if 2−2≤σ≤23, the weight will be decided by the value of −trYiTLYi which characterizes the local geometric structure in the i-th view. From equation (13), we know that the value of −trYiTLYi−trYiTLYi also reflects the similarity between YiYiT and the latent similarity matrix. As a result, if the value of −trYiTLYi−trYiTLYi is smaller, it will give more weight to Yi in the learning matrix of latent similarity. Thus, when 21≤σ≤23, the performance of MvLRFD will be upgraded by the proposed coregularization.Figure 5
Clustering results vs.σ,λ, and β on ORL dataset. (a) ACC vs σ. (b) ACC vs λ. (c) ACC vs β.
(a)(b)(c)Withσ=22 and β=2−3, we study the impact of λ by varying it in the grid of 2−5,2−4,…,25 on the ORL dataset and plot ACC curve vs. parameter λ. It can be seen that the clustering accuracy gradually increases with increasing λλ>2−3and remains stable when 21≤λ≤23. This pattern is explainable since a higher value of λ will force MvLRFD to focus more on fused information between views.Withσ=22 and λ=22, our method will obtain a better performance for maintaining private information when 2−2≥β≥2−4. If β>2−2, it will force MvLRFD to focus more on private information.Figures6(a)–6(d) show the value of our objective function equation (13) vs. iterations. Obviously, we can find the quick convergent property of the iterative algorithm on ORL, Yale, MFD, and MSRC-v1 datasets.Figure 6
Convergence curves of MvLRFD on four datasets. (a) ORL. (b) Yale. (c) MFD. (d) MSRC-v1.
(a)(b)(c)(d)


## 4. Conclusions

In this paper, we propose a multiview learning algorithm named Multiview Latent representation learning with feature diversity for clustering (MvLRFD). To wipe out the noise information from different descriptions, Matrix Factorization is adopted to get a latent representation. HSIC-based diversity criterion is adopted to exploit complementary information from multiview data. A new coregularization strategy is proposed for encouraging the similarity of the data of each view toward a fused similarity matrix to code with diverse representations of multiple views. Experimental results on multiview clustering tasks show that MvLRFD performs better than some state-of-the-art multiview learning algorithms. However, it is difficult to determine the optimal parameters of MvLRFD. In the future, we aim to find a way to determine the parameters. Moreover, we expect to unitize our model to deal with semisupervised multiview representation learning tasks.



---

*Source: 1866636-2022-07-11.xml*