# A Gaussian Process Latent Variable Model for Subspace Clustering

Authors: Shangfang Li
Journal: Complexity
 (2021)

---

## Abstract

Effective feature representation is the key to success of machine learning applications. Recently, many feature learning models have been proposed. Among these models, the Gaussian process latent variable model (GPLVM) for nonlinear feature learning has received much attention because of its superior performance. However, most of the existing GPLVMs are mainly designed for classification and regression tasks, thus cannot be used in data clustering task. To address this issue and extend the application scope, this paper proposes a novel GPLVM for clustering (C-GPLVM). Specifically, by combining GPLVM with the subspace clustering method, our C-GPLVM can obtain more representative latent variable for clustering. Moreover, it can directly predict the new samples by introducing a back constraint in the model, thus being more suitable for big data learning tasks such as analysis of chaotic time series and so on. In the experiment, we compare it with the related GPLVMs and clustering algorithms. The experimental results show that the proposed model not only inherits the feature learning ability of GPLVM but also has superior clustering accuracy.


---

## Body

## 1. Introduction

In machine learning tasks, data are often distributed in a high-dimensional space and have many redundant features. Training machine learning models in such high-dimensional data may result in not only higher computational and storage complexities but also the model overfitting problem [1]. Existing research studies have shown that high-dimensional data are often embedded in low-dimensional manifold. We therefore can utilize the dimension reduction and feature learning method to learn the low-dimensional manifold and obtain more representative feature for the improvements of machine learning model accuracy and efficiency. Thus, effective feature representation is the key to success of machine learning applications.In the past decade, many related methods have been proposed, such as dictionary learning [2], autoencoder [3], Gaussian process latent variable model (GPLVM) [4], Isomap [5], and locally linear embedding [6]. Among these models, the GPLVM for nonlinear feature learning has received extensive attention because of its superior feature learning ability and has been used in many applications such as dynamical system [7], modelling and control of nonlinear system [8]. Given a few training samples, it can effectively learn the low-dimensional manifold that is embedded in the high-dimensional space, thus has been widely used in the dimension reduction and data visualization tasks [9, 10].Although has the abovementioned advantages, the conventional GPLVM is just a fully unsupervised feature learning model, thus cannot meet the demands of real-world applications, when dealing with specific machine leaning tasks such as analysis of chaotic time series, dynamical system [7], and modelling and control of nonlinear system in which we also observe the response values of the inputs. How to modify the GPLVM and improve its performance is the key content of the related research studies. To date, the extensions of this model mainly focus on the supervised and unsupervised learning methods [9, 11, 12]. These methods assume that apart from the input features, we also observe labels of the samples. By their extensions, the GPLVM can effectively utilize the supervised information to improve the classification accuracy of the learned latent variables. However, in the real-world applications, we may also deal with unsupervised clustering tasks in which we cannot obtain the label information or any other auxiliary information, thus bringing more challenges to application of the GPLVM in clustering tasks.In order to address the abovementioned issues, this paper proposes a fusion model that combines the GPLVM with the subspace clustering model [13] to simultaneously obtain more representative features and accurate clustering results. Moreover, we also use the back constraint trick [14] in the model, which makes the model predict new samples directly and more suitable for big data learning tasks such as analysis of chaotic time series. In the experiment, we verify the performance of the proposed model on multiple datasets. The experimental results show that our model has much superior clustering performance than the other related models.


## 2. Related Work

### 2.1. Gaussian Process Latent Variable Model

The GPLVM is a fully unsupervised and nonlinear latent variable model. In this model, givenN observed samples X=x1,…,xNT∈ℝN×D (where xn∈ℝD denotes the nth training sample), our objective is to learn the corresponding low-dimensional latent variable. In this paper, we use hn∈ℝQ (Q≪D) to denotes the latent variable of xn. Obviously, the GPLVM can realize dimension reduction by learning the latent variables. Specifically, the GPLVM assumes that the generation process of xn as follows:(1)xnd=fdhn+ϵnd,where xnd is the dth feature of the nth training sample, ϵnd is the noise term that follows a Gaussian distribution pϵnd=Nϵnd|0,σ2, and fd⋅ is a function that follows a Gaussian process prior. We use fd to denote the N outputs of fd⋅ with inputs H=h1,…,hNT. Thus, we have fd∼N0,K , where K is the kernel matrix that is computed by using kernel function k⋅,⋅ on the latent variables in H. The ith row and jth column element of K is computed as Kij=khi,hj. By integrating out the intermediate variable fd, we can obtainthe following marginal likelihood function:(2)pX|H,θ=∏d=1Dpx:,d|H,θ=∏d=1Dexp−1/2x:,dTK+σ2I−1x:,d2πN/2K+σ2I1/2,where θ denotes the hyperparameters involved in the kernel function and noise distribution. In the model optimization process, the GPLVM learns the latent variable and hyperparameters jointly by maximizing the above likelihood function and obtains the low-dimensional representation finally.From the abovementioned generation process, as a fully unsupervised dimension reduction model, the GPLVM cannot embed auxiliary information when dealing with specific machine learning tasks, thus cannot meet demands of real-world application. For example, in analysis of chaotic time series, data of similar time will have similar features. If it can utilize this knowledge, the GPLVM will learn more representative features for the task and significantly improve the prediction accuracy. The existing methods for the extension of GPLVM mainly focus on embedding supervised information to improve its classification and regression accuracy, for example, the discriminative GPLVM (D-GPLVM) and supervised GPLVM (S-GPLVM). For the extension to the clustering task, the related works is much fewer. The existing unsupervised GPLVM just focuses on how to preserve the local distance and learn better latent variables or features. For example, local preserving projection GPLVM (LPP-GPLVM) combines the objective of local preserving projection with that of the GPLVM, thus simultaneously learning the low-dimensional representation and preserving the local structures [15]. The GPLVM with back constraints (B-GPLVM) introduces a back-constraint (from observed space to latent space) into the GPLVM. By this way, it can also realize the preservation of local distance.


### 2.2. Subspace Clustering

The goal of subspace clustering is to segment a set of data samples into different subspaces; thus, similar samples are in the same subspace, while dissimilar samples are in different subspaces. Over the past decade, subspace clustering has been used in various clustering tasks and many well-designed algorithms have been proposed such as Gaussian mixture model- (GMM-) based methods [16, 17], matrix factorization- (MF-) based methods [18, 19], algebra-based methods [20], and spectral clustering methods [13, 21, 22]. Among these models, the subspace clustering method based on spectral clustering has been widely applied because of its concise implementation process and reliable performance. It uses low-rank representation to construct the affinity matrix of the spectral clustering. Its objective is to find the low-rank representation of input data X by optimizing the following function:(3)argminZrank(Z),s.tX=ZX,where we assume that each sample can be expressed by the linear combination of other samples. The above low-rank penalty term can be considered as a global constraint on the subspace structure of samples and makes similar samples have similar weights. In general, we can use the following nuclear norm to replace the penalty term:(4)argminZ||Z||∗,s.t.X=ZX,where we use the nuclear norm Z∗ to approximate the rank of Z. Considering that the data often contain noise, we use the following formulation to learn the self-representation matrix Z:(5)argminZ12X-ZXF2+λZ∗.In low-rank subspace clustering, we can first construct the affinity matrixW and the Laplace matrix L and then use spectral clustering to cluster the data. W and L can be constructed as follows:(6)W=Z+ZT,L=D−W,where D denotes a diagonal matrix and Dii=∑j=1NWij. After obtaining the Laplace matrix, we can optimize the following objective function to obtain the latent variable H∈ℝN×Q:(7)argminHtrHTLH,s.t.HTH=I.Obviously,H is composed of the eigenvectors corresponding to the Q smallest eigenvalues. At last, we can run the k-means algorithm on the learned H and obtain the clustering result.


## 2.1. Gaussian Process Latent Variable Model

The GPLVM is a fully unsupervised and nonlinear latent variable model. In this model, givenN observed samples X=x1,…,xNT∈ℝN×D (where xn∈ℝD denotes the nth training sample), our objective is to learn the corresponding low-dimensional latent variable. In this paper, we use hn∈ℝQ (Q≪D) to denotes the latent variable of xn. Obviously, the GPLVM can realize dimension reduction by learning the latent variables. Specifically, the GPLVM assumes that the generation process of xn as follows:(1)xnd=fdhn+ϵnd,where xnd is the dth feature of the nth training sample, ϵnd is the noise term that follows a Gaussian distribution pϵnd=Nϵnd|0,σ2, and fd⋅ is a function that follows a Gaussian process prior. We use fd to denote the N outputs of fd⋅ with inputs H=h1,…,hNT. Thus, we have fd∼N0,K , where K is the kernel matrix that is computed by using kernel function k⋅,⋅ on the latent variables in H. The ith row and jth column element of K is computed as Kij=khi,hj. By integrating out the intermediate variable fd, we can obtainthe following marginal likelihood function:(2)pX|H,θ=∏d=1Dpx:,d|H,θ=∏d=1Dexp−1/2x:,dTK+σ2I−1x:,d2πN/2K+σ2I1/2,where θ denotes the hyperparameters involved in the kernel function and noise distribution. In the model optimization process, the GPLVM learns the latent variable and hyperparameters jointly by maximizing the above likelihood function and obtains the low-dimensional representation finally.From the abovementioned generation process, as a fully unsupervised dimension reduction model, the GPLVM cannot embed auxiliary information when dealing with specific machine learning tasks, thus cannot meet demands of real-world application. For example, in analysis of chaotic time series, data of similar time will have similar features. If it can utilize this knowledge, the GPLVM will learn more representative features for the task and significantly improve the prediction accuracy. The existing methods for the extension of GPLVM mainly focus on embedding supervised information to improve its classification and regression accuracy, for example, the discriminative GPLVM (D-GPLVM) and supervised GPLVM (S-GPLVM). For the extension to the clustering task, the related works is much fewer. The existing unsupervised GPLVM just focuses on how to preserve the local distance and learn better latent variables or features. For example, local preserving projection GPLVM (LPP-GPLVM) combines the objective of local preserving projection with that of the GPLVM, thus simultaneously learning the low-dimensional representation and preserving the local structures [15]. The GPLVM with back constraints (B-GPLVM) introduces a back-constraint (from observed space to latent space) into the GPLVM. By this way, it can also realize the preservation of local distance.


## 2.2. Subspace Clustering

The goal of subspace clustering is to segment a set of data samples into different subspaces; thus, similar samples are in the same subspace, while dissimilar samples are in different subspaces. Over the past decade, subspace clustering has been used in various clustering tasks and many well-designed algorithms have been proposed such as Gaussian mixture model- (GMM-) based methods [16, 17], matrix factorization- (MF-) based methods [18, 19], algebra-based methods [20], and spectral clustering methods [13, 21, 22]. Among these models, the subspace clustering method based on spectral clustering has been widely applied because of its concise implementation process and reliable performance. It uses low-rank representation to construct the affinity matrix of the spectral clustering. Its objective is to find the low-rank representation of input data X by optimizing the following function:(3)argminZrank(Z),s.tX=ZX,where we assume that each sample can be expressed by the linear combination of other samples. The above low-rank penalty term can be considered as a global constraint on the subspace structure of samples and makes similar samples have similar weights. In general, we can use the following nuclear norm to replace the penalty term:(4)argminZ||Z||∗,s.t.X=ZX,where we use the nuclear norm Z∗ to approximate the rank of Z. Considering that the data often contain noise, we use the following formulation to learn the self-representation matrix Z:(5)argminZ12X-ZXF2+λZ∗.In low-rank subspace clustering, we can first construct the affinity matrixW and the Laplace matrix L and then use spectral clustering to cluster the data. W and L can be constructed as follows:(6)W=Z+ZT,L=D−W,where D denotes a diagonal matrix and Dii=∑j=1NWij. After obtaining the Laplace matrix, we can optimize the following objective function to obtain the latent variable H∈ℝN×Q:(7)argminHtrHTLH,s.t.HTH=I.Obviously,H is composed of the eigenvectors corresponding to the Q smallest eigenvalues. At last, we can run the k-means algorithm on the learned H and obtain the clustering result.


## 3. Model Construction and Optimization

### 3.1. Designing of the Gaussian Process Latent Variable Clustering Model

Assuming that there areN observed samples denoted as X=x1,…,xNT, our goal is to learn the low-dimensional latent variable H=h1,…,hNT corresponded to these observed variables and make the latent variable have more superior clustering performance (i.e., make the common clustering algorithms obtain accurate clustering result on the learned H).In order to achieve the above goal, we assume that the latent variableH has the following prior distribution:(8)pH=1Z0exp−λ1RH,where p0 is a constant that makes ∫HpHdH=1 and RH has the following form:(9)RH=∑i=1N∑j=1Nhi−hj22Wij,where Wij is the ith row and jth column element of the affinity matrix W. Equation (9) often can be written as follows:(10)RH=HTLH.In this paper, we assume that the generation process observed variables from the latent variable can be constructed by conditional distributionpX|H. Thus, from the Bayes formulation, we can obtainthe posterior distribution of latent variable as(11)pH|X=pX|HpHpX.SincepX is a constant, we therefore can obtain the optimal latent variable by maximizing the following joint marginal distribution:(12)argmaxHpX|HpH.To introduce the GPLVM into this model, we assume thatX is generated by latent function which follows a Gaussian process prior. Thus, equation (12) can be written as(13)argminH,θ,σDN2lnK+σ2I+12trXK+σ2I−1XT+λ1RH,where θ denotes the hyperparameter that is involved in the kernel function and σ2 denotes the variance of Gaussian noise distribution.By the above modelling process, the GPLVM can effectively embed the sample similarity information when learning the latent variable, thus improving its latent variable clustering ability. However, how to learn the affinity matrix is still an urgent problem of this paper and other related algorithms such as self-representation learning and subspace clustering. In this paper, we borrow the idea of low-rank self-representation learning and introduce the following low-rank subspace constraints into the model:(14)X=ZX,rankZ≤c,Z≥0.It is worth noting that, in this paper, we assume thatW=Z, i.e., we directly use matrix Z as the affinity matrix. This setting is the same as that of [23] and its role is similar to the affinity matrix of the original subspace clustering. This C-GPLVM is very similar to the LPP-GPLVM. However, in the LPP-GPLVM, the Laplace matrix is fixed. Different from LPP-GPLVM, the Laplace matrix in our C-GPLVM can be learned in the training process. Thus, our C-GPLVM has more superior performance than the LPP-GPLVM.One important limitation of GPLVM and self-representation is that they cannot effectively predict the new samples. To mitigate this problem, we introduce a back constraint on the proposed model. Thus, given a new sample, it can effectively predict the corresponding low-dimensional latent variable using the constraint function. Specifically, given an observed samplexn, we assume that we can use a function g⋅ to obtain latent variable hn:(15)hn=gxn,where g⋅ is the neural network function with learnable parameter Φ. At last, we obtain the objective of the proposed model as follows:(16)argminΦ,θ,σDN2lnK+σ2I+12trXK+σ2I−1XT+λ1RH,s.t.X=ZX,rankZ≤c,Z≥0.The whole model structure is shown in Figure1.Figure 1
The structure of C-GPLVM.


## 3.1. Designing of the Gaussian Process Latent Variable Clustering Model

Assuming that there areN observed samples denoted as X=x1,…,xNT, our goal is to learn the low-dimensional latent variable H=h1,…,hNT corresponded to these observed variables and make the latent variable have more superior clustering performance (i.e., make the common clustering algorithms obtain accurate clustering result on the learned H).In order to achieve the above goal, we assume that the latent variableH has the following prior distribution:(8)pH=1Z0exp−λ1RH,where p0 is a constant that makes ∫HpHdH=1 and RH has the following form:(9)RH=∑i=1N∑j=1Nhi−hj22Wij,where Wij is the ith row and jth column element of the affinity matrix W. Equation (9) often can be written as follows:(10)RH=HTLH.In this paper, we assume that the generation process observed variables from the latent variable can be constructed by conditional distributionpX|H. Thus, from the Bayes formulation, we can obtainthe posterior distribution of latent variable as(11)pH|X=pX|HpHpX.SincepX is a constant, we therefore can obtain the optimal latent variable by maximizing the following joint marginal distribution:(12)argmaxHpX|HpH.To introduce the GPLVM into this model, we assume thatX is generated by latent function which follows a Gaussian process prior. Thus, equation (12) can be written as(13)argminH,θ,σDN2lnK+σ2I+12trXK+σ2I−1XT+λ1RH,where θ denotes the hyperparameter that is involved in the kernel function and σ2 denotes the variance of Gaussian noise distribution.By the above modelling process, the GPLVM can effectively embed the sample similarity information when learning the latent variable, thus improving its latent variable clustering ability. However, how to learn the affinity matrix is still an urgent problem of this paper and other related algorithms such as self-representation learning and subspace clustering. In this paper, we borrow the idea of low-rank self-representation learning and introduce the following low-rank subspace constraints into the model:(14)X=ZX,rankZ≤c,Z≥0.It is worth noting that, in this paper, we assume thatW=Z, i.e., we directly use matrix Z as the affinity matrix. This setting is the same as that of [23] and its role is similar to the affinity matrix of the original subspace clustering. This C-GPLVM is very similar to the LPP-GPLVM. However, in the LPP-GPLVM, the Laplace matrix is fixed. Different from LPP-GPLVM, the Laplace matrix in our C-GPLVM can be learned in the training process. Thus, our C-GPLVM has more superior performance than the LPP-GPLVM.One important limitation of GPLVM and self-representation is that they cannot effectively predict the new samples. To mitigate this problem, we introduce a back constraint on the proposed model. Thus, given a new sample, it can effectively predict the corresponding low-dimensional latent variable using the constraint function. Specifically, given an observed samplexn, we assume that we can use a function g⋅ to obtain latent variable hn:(15)hn=gxn,where g⋅ is the neural network function with learnable parameter Φ. At last, we obtain the objective of the proposed model as follows:(16)argminΦ,θ,σDN2lnK+σ2I+12trXK+σ2I−1XT+λ1RH,s.t.X=ZX,rankZ≤c,Z≥0.The whole model structure is shown in Figure1.Figure 1
The structure of C-GPLVM.


## 4. Model Optimization

In order to optimize (16), we transform it into the following optimization problem:(17)argminΦ,θ,σDN2lnK+σ2I+12trXK+σ2I−1XT+λ1RH+λ2X−ZX22+λ3Z∗,s.t.Z≥0,where λ1, λ2, and λ3 are regularization terms. By this formulation, we can use the alternating iterative optimization method to learn all the parameters. First, we fix Z and write (17) as(18)argminΦ,θ,σLp=argminΦ,θ,σDN2lnK+σ2I+12trXK+σ2I−1XT+λ1RH.This problem can be solved effectively by using gradient-based methods, and its gradient with respect toΦ can be computed as(19)∂Lp∂Φ=∂Lp∂K∂K∂Φ+λ1∂RH∂Φ.The gradients with respect toθ and σ are similar to the above formulation. For the sake of brevity, we have omitted their derivation processes. We then can fix Φ, θ and σ and write (17) as(20)argminZλ1RH+λ2X−ZX22+λ3Z∗,s.t.Z≥0.The gradient of the first term with respect toZ can be computed as(21)∂λ1RH∂Zn,:=eT∑j=1Nhn−hj22,where Zn,: denotes the nth row of matrix Z and e∈ℝN denotes the vector whose element is 1. The gradient of the second term can be computed as(22)∂λ2X−ZX22∂Z=λ2X−ZXXT.The subgradient of the third term is(23)∂X∗∂X=VsgnΣUT.By the above derivations, we can learn the whole model, as descripted in Algorithm1. The main computation complexity of C-GPLVM is the inversion of the kernel matrix, which has a complexity of ON3, where N is the number of training samples. The main storage complexity of this is the storage of kernel matrix, which has a complexity of ON2. Thus, both the computation and storage complexities are the same as those of the conventional GPLVM.Algorithm 1: The optimization process of C-GPLVM.
Input: training setX, dimension Q of latent variable, hyperparameters λ1, λ2, λ3.Output: parameterΦ and the clustering result.(1)
Pretrain the following model to initialize the latent variableZZ=argminZλ2X−ZX22+λ3Z∗s.t.Z≥0(2)
while (ltol≥10−3)(3)
With fixedZ, minimize (18) to obtain Φ, θ, and σ.(4)
With fixedΦ, θ, and σ, minimize (20) to obtain Z.(5)
Calculate the differenceltol of the objective function between the last two iterations(6)
end while(7)
Compute the latent variableH based on the learned parameter Φ.(8)
Runk-means algorithm on H to get the final clustering result.


## 5. Experiments and Analysis

### 5.1. Experimental Setup

To verify the effectiveness of C-GPLVM, we use 8 datasets in the experiments. The detailed information of these datasets is given in Table1.Table 1
The detailed information of the datasets.
Dataset# of sample# of feature# of classesYALE165102415JAFFE21367610ORL400102440YEAST1484147010USPS185425620TR1141464299TR41878745410TR45690826110The YEAST is a dataset for the prediction of protein localization sites. The USPS is a digits dataset that was gathered at the Center of Excellence in Document Analysis and Recognition at SUNY Buffalo, as part of a project sponsored by the US Postal Service. YALE, JAFFE, and ORL are three face recognition datasets, as shown in Figure2. TR11, TR41, and TR45 are three textual datasets.Figure 2
Samples in YALE, JAFFE, and ORL datasets.
(a)(b)(c)In order to fully verify the advantage of C-GPLVM, we compare it to the related Gaussian process latent variable model (i.e., GPLVM, B-GPLVM, and LPP-GPLVM) and clustering methods, such as spectral clustering method (SC) [24], kernel spectral clustering (KSC) [25], and simplex sparse representation learning (SSR) [21]. All the kernel-based models (GPLVM, LPP-GPLVM, KSC, and C-GPLVM) used Radial Basis Function (RBF) as the kernel function. It is worth noting that some other kernel functions can also be used in the proposed model such as linear kernel, Laplacian kernel, and circular kernel. Furthermore, all the hyperparameters in these kernel functions can also be learned in the same form as descripted in the paper. During the experiment process, the hyperparameters λ1, λ2, and λ3 are chosen from 0.01,0.1,1,10,100. The hyperparameters involved in other models are set to be the same as those of original paper. In the experiment process, we use the Gaussian process toolkit (GPFlow) 1 to implement the GP-based model. Other related models are all implemented with python. All the algorithms are tested on the Windows computer with i7 9700 CPU, 16G RAM.


### 5.2. Clustering Results and Analysis

In the experiments, we use clustering accuracy, purity, and normalized mutual information (NMI) as the clustering measurement. At the clustering stage, the latent variables learned by different methods are used as inputs and the k-means algorithm is used to obtain the final clustering methods. The dimension and the number of clusters are set to be the same as the number of classes. At the same time, in order to mitigate the initial value sensitivity problem of k-means method, we randomly initialize and run the k-means method 20 times. Finally, we calculate the mean and standard deviation of these 20 experiments. The experimental results are shown in Tables2–4, where the best results are given in bold.Table 2
Accuracy.
YALEJAFFEORLYEASTUSPSTR11TR41TR45GPLVM29.80 ± 1.0155.91 ± 2.9421.58 ± 1.4519.48 ± 1.4923.14 ± 2.4130.01 ± 2.5842.87 ± 2.9433.54 ± 2.31B-GPLVM32.12 ± 1.5859.66 ± 2.1425.17 ± 2.1823.94 ± 1.5428.14 ± 1.1935.47 ± 1.5048.99 ± 1.3737.93 ± 2.54LPP-GPLVM36.14 ± 1.2571.49 ± 2.7538.97 ± 1.9429.88 ± 1.7361.71 ± 2.2151.44 ± 2.2049.79 ± 1.4051.83 ± 2.91SC50.32 ± 1.2373.55 ± 2.3357.96 ± 1.4535.91 ± 1.3436.45 ± 2.9052.69 ± 3.3261.74 ± 1.855.91 ± 1.96KSC37.21 ± 1.5572.19 ± 1.9839.74 ± 2.3931.18 ± 1.8363.78 ± 3.8753.74 ± 1.8651.12 ± 1.1153.44 ± 1.99SSR53.59 ± 1.0985.92 ± 2.3468.22 ± 1.9827.62 ± 1.9264.83 ± 2.3941.06 ± 2.3365.78 ± 2.5873.45 ± 1.36C-GPLVM55.91 ± 1.3886.14 ± 1.8170.43 ± 1.3033.71 ± 1.4263.16 ± 2.0155.89 ± 2.5967.16 ± 2.1475.25 ± 1.90Table 3
Purity.
YALEJAFFEORLYEASTUSPSTR11TR41TR45GPLVM30.48 ± 1.0053.41 ± 1.8438.14 ± 1.098.47 ± 1.7121.10 ± 1.6619.41 ± 1.1030.11 ± 2.0934.57 ± 1.49B-GPLVM33.19 ± 1.4456.49 ± 1.5849.18 ± 1.418.56 ± 2.6125.17 ± 2.1723.61 ± 1.9734.17 ± 2.3738.44 ± 1.56LPP-GPLVM41.57 ± 1.1167.18 ± 2.1067.18 ± 1.9412.80 ± 1.5426.49 ± 1.4328.15 ± 2.1553.91 ± 2.3445.61 ± 2.01SC53.24 ± 1.2582.32 ± 1.0274.41 ± 1.4721.47 ± 3.1428.91 ± 1.8442.89 ± 1.5861.84 ± 2.1448.17 ± 2.17KSC45.91 ± 2.1771.82 ± 1.8455.91 ± 1.1911.14 ± 2.1449.83 ± 1.3645.77 ± 1.9144.77 ± 2.9144.18 ± 2.22SSR54.66 ± 2.1491.10 ± 1.2483.16 ± 1.8416.91 ± 2.1771.99 ± 1.4728.67 ± 1.4356.90 ± 2.7965.10 ± 1.97C-GPLVM55.17 ± 2.3891.84 ± 1.3382.71 ± 1.9421.04 ± 2.3870.61 ± 1.8947.94 ± 1.9562.85 ± 2.5766.73 ± 2.32Table 4
Normalized mutual information.
YALEJAFFEORLYEASTUSPSTR11TR41TR45GPLVM30.41 ± 1.4059.17 ± 1.0931.90 ± 1.2919.80 ± 1.4121.36 ± 1.0034.82 ± 1.0639.40 ± 1.6137.21 ± 1.27B-GPLVM32.49 ± 1.6663.99 ± 1.8132.81 ± 1.1128.39 ± 1.5830.58 ± 1.0740.59 ± 1.4049.55 ± 2.0646.79 ± 1.41LPP-GPLVM40.85 ± 1.2968.46 ± 1.4340.90 ± 1.0931.63 ± 1.4734.83 ± 1.0648.97 ± 1.3851.63 ± 2.4452.88 ± 1.67SC50.49 ± 1.8475.91 ± 2.4160.49 ± 1.8453.71 ± 2.1736.49 ± 1.6157.39 ± 1.2072.63 ± 2.1961.98 ± 1.94KSC43.50 ± 1.3277.35 ± 2.0141.53 ± 1.4432.59 ± 2.9861.29 ± 1.6551.09 ± 1.6954.81 ± 2.8155.71 ± 2.11SSR57.39 ± 1.0995.50 ± 1.8676.41 ± 1.0043.16 ± 2.7373.24 ± 1.1083.77 ± 1.4375.63 ± 2.3782.69 ± 2.43C-GPLVM58.41 ± 2.1796.01 ± 2.3877.25 ± 1.5351.42 ± 1.8571.11 ± 1.8385.64 ± 1.8277.31 ± 2.8283.11 ± 2.28From Tables2–4, we can observe that the GPLVM, as an unsupervised dimension reduction model, usually obtain latent variables that have poor clustering performance. The B-GPLVM and LPP-GPLVM can preserve the local distance of samples during the feature learning process, thus obtaining more representative latent variables. Meanwhile, the LPP-GPLVM obtains much better result than the B-GPLVM which indicates that graph Laplace regularization is more suitable for clustering than back constraints. In general, spectral clustering and subspace clustering methods have better performance than the GPLVM. As we can see, SC, KSC, and SSR outperform GPLVM, B-GPLVM, and LPP-GPLVM. In this paper, the proposed C-GPLVM combines the subspace clustering with GPLVM, thus effectively improving the clustering performance of the GPLVM. As shown in the experimental results, the C-GPLVM has more superior clustering result than other related models in most cases.


## 5.1. Experimental Setup

To verify the effectiveness of C-GPLVM, we use 8 datasets in the experiments. The detailed information of these datasets is given in Table1.Table 1
The detailed information of the datasets.
Dataset# of sample# of feature# of classesYALE165102415JAFFE21367610ORL400102440YEAST1484147010USPS185425620TR1141464299TR41878745410TR45690826110The YEAST is a dataset for the prediction of protein localization sites. The USPS is a digits dataset that was gathered at the Center of Excellence in Document Analysis and Recognition at SUNY Buffalo, as part of a project sponsored by the US Postal Service. YALE, JAFFE, and ORL are three face recognition datasets, as shown in Figure2. TR11, TR41, and TR45 are three textual datasets.Figure 2
Samples in YALE, JAFFE, and ORL datasets.
(a)(b)(c)In order to fully verify the advantage of C-GPLVM, we compare it to the related Gaussian process latent variable model (i.e., GPLVM, B-GPLVM, and LPP-GPLVM) and clustering methods, such as spectral clustering method (SC) [24], kernel spectral clustering (KSC) [25], and simplex sparse representation learning (SSR) [21]. All the kernel-based models (GPLVM, LPP-GPLVM, KSC, and C-GPLVM) used Radial Basis Function (RBF) as the kernel function. It is worth noting that some other kernel functions can also be used in the proposed model such as linear kernel, Laplacian kernel, and circular kernel. Furthermore, all the hyperparameters in these kernel functions can also be learned in the same form as descripted in the paper. During the experiment process, the hyperparameters λ1, λ2, and λ3 are chosen from 0.01,0.1,1,10,100. The hyperparameters involved in other models are set to be the same as those of original paper. In the experiment process, we use the Gaussian process toolkit (GPFlow) 1 to implement the GP-based model. Other related models are all implemented with python. All the algorithms are tested on the Windows computer with i7 9700 CPU, 16G RAM.


## 5.2. Clustering Results and Analysis

In the experiments, we use clustering accuracy, purity, and normalized mutual information (NMI) as the clustering measurement. At the clustering stage, the latent variables learned by different methods are used as inputs and the k-means algorithm is used to obtain the final clustering methods. The dimension and the number of clusters are set to be the same as the number of classes. At the same time, in order to mitigate the initial value sensitivity problem of k-means method, we randomly initialize and run the k-means method 20 times. Finally, we calculate the mean and standard deviation of these 20 experiments. The experimental results are shown in Tables2–4, where the best results are given in bold.Table 2
Accuracy.
YALEJAFFEORLYEASTUSPSTR11TR41TR45GPLVM29.80 ± 1.0155.91 ± 2.9421.58 ± 1.4519.48 ± 1.4923.14 ± 2.4130.01 ± 2.5842.87 ± 2.9433.54 ± 2.31B-GPLVM32.12 ± 1.5859.66 ± 2.1425.17 ± 2.1823.94 ± 1.5428.14 ± 1.1935.47 ± 1.5048.99 ± 1.3737.93 ± 2.54LPP-GPLVM36.14 ± 1.2571.49 ± 2.7538.97 ± 1.9429.88 ± 1.7361.71 ± 2.2151.44 ± 2.2049.79 ± 1.4051.83 ± 2.91SC50.32 ± 1.2373.55 ± 2.3357.96 ± 1.4535.91 ± 1.3436.45 ± 2.9052.69 ± 3.3261.74 ± 1.855.91 ± 1.96KSC37.21 ± 1.5572.19 ± 1.9839.74 ± 2.3931.18 ± 1.8363.78 ± 3.8753.74 ± 1.8651.12 ± 1.1153.44 ± 1.99SSR53.59 ± 1.0985.92 ± 2.3468.22 ± 1.9827.62 ± 1.9264.83 ± 2.3941.06 ± 2.3365.78 ± 2.5873.45 ± 1.36C-GPLVM55.91 ± 1.3886.14 ± 1.8170.43 ± 1.3033.71 ± 1.4263.16 ± 2.0155.89 ± 2.5967.16 ± 2.1475.25 ± 1.90Table 3
Purity.
YALEJAFFEORLYEASTUSPSTR11TR41TR45GPLVM30.48 ± 1.0053.41 ± 1.8438.14 ± 1.098.47 ± 1.7121.10 ± 1.6619.41 ± 1.1030.11 ± 2.0934.57 ± 1.49B-GPLVM33.19 ± 1.4456.49 ± 1.5849.18 ± 1.418.56 ± 2.6125.17 ± 2.1723.61 ± 1.9734.17 ± 2.3738.44 ± 1.56LPP-GPLVM41.57 ± 1.1167.18 ± 2.1067.18 ± 1.9412.80 ± 1.5426.49 ± 1.4328.15 ± 2.1553.91 ± 2.3445.61 ± 2.01SC53.24 ± 1.2582.32 ± 1.0274.41 ± 1.4721.47 ± 3.1428.91 ± 1.8442.89 ± 1.5861.84 ± 2.1448.17 ± 2.17KSC45.91 ± 2.1771.82 ± 1.8455.91 ± 1.1911.14 ± 2.1449.83 ± 1.3645.77 ± 1.9144.77 ± 2.9144.18 ± 2.22SSR54.66 ± 2.1491.10 ± 1.2483.16 ± 1.8416.91 ± 2.1771.99 ± 1.4728.67 ± 1.4356.90 ± 2.7965.10 ± 1.97C-GPLVM55.17 ± 2.3891.84 ± 1.3382.71 ± 1.9421.04 ± 2.3870.61 ± 1.8947.94 ± 1.9562.85 ± 2.5766.73 ± 2.32Table 4
Normalized mutual information.
YALEJAFFEORLYEASTUSPSTR11TR41TR45GPLVM30.41 ± 1.4059.17 ± 1.0931.90 ± 1.2919.80 ± 1.4121.36 ± 1.0034.82 ± 1.0639.40 ± 1.6137.21 ± 1.27B-GPLVM32.49 ± 1.6663.99 ± 1.8132.81 ± 1.1128.39 ± 1.5830.58 ± 1.0740.59 ± 1.4049.55 ± 2.0646.79 ± 1.41LPP-GPLVM40.85 ± 1.2968.46 ± 1.4340.90 ± 1.0931.63 ± 1.4734.83 ± 1.0648.97 ± 1.3851.63 ± 2.4452.88 ± 1.67SC50.49 ± 1.8475.91 ± 2.4160.49 ± 1.8453.71 ± 2.1736.49 ± 1.6157.39 ± 1.2072.63 ± 2.1961.98 ± 1.94KSC43.50 ± 1.3277.35 ± 2.0141.53 ± 1.4432.59 ± 2.9861.29 ± 1.6551.09 ± 1.6954.81 ± 2.8155.71 ± 2.11SSR57.39 ± 1.0995.50 ± 1.8676.41 ± 1.0043.16 ± 2.7373.24 ± 1.1083.77 ± 1.4375.63 ± 2.3782.69 ± 2.43C-GPLVM58.41 ± 2.1796.01 ± 2.3877.25 ± 1.5351.42 ± 1.8571.11 ± 1.8385.64 ± 1.8277.31 ± 2.8283.11 ± 2.28From Tables2–4, we can observe that the GPLVM, as an unsupervised dimension reduction model, usually obtain latent variables that have poor clustering performance. The B-GPLVM and LPP-GPLVM can preserve the local distance of samples during the feature learning process, thus obtaining more representative latent variables. Meanwhile, the LPP-GPLVM obtains much better result than the B-GPLVM which indicates that graph Laplace regularization is more suitable for clustering than back constraints. In general, spectral clustering and subspace clustering methods have better performance than the GPLVM. As we can see, SC, KSC, and SSR outperform GPLVM, B-GPLVM, and LPP-GPLVM. In this paper, the proposed C-GPLVM combines the subspace clustering with GPLVM, thus effectively improving the clustering performance of the GPLVM. As shown in the experimental results, the C-GPLVM has more superior clustering result than other related models in most cases.


## 6. Conclusion and Future Work

This paper proposes a joint model by combining the low-rank subspace with the back constraint GPLVM to address the poor clustering performance problem of the conventional GPLVM. The proposed C-GPLVM can not only obtain low-dimensional latent variables but also directly predict the new samples, thus effectively extending the application scope of GPLVM on tasks such as analysis of chaotic time series. The experimental results show that the C-GPLVM has much better latent variable learning ability and superior clustering performance. In the future work, we will further extend the C-GPLVM to make it suitable for much bigger dataset and supervise tasks such as classification and regression, improving its efficiency and application scope.



---

*Source: 8864981-2021-01-13.xml*