# A High-Order CFS Algorithm for Clustering Big Data

Authors: Fanyu Bu; Zhikui Chen; Peng Li; Tong Tang; Ying Zhang
Journal: Mobile Information Systems
 (2016)

---

## Abstract

With the development of Internet of Everything such as Internet of Things, Internet of People, and Industrial Internet, big data is being generated. Clustering is a widely used technique for big data analytics and mining. However, most of current algorithms are not effective to cluster heterogeneous data which is prevalent in big data. In this paper, we propose a high-order CFS algorithm (HOCFS) to cluster heterogeneous data by combining the CFS clustering algorithm and the dropout deep learning model, whose functionality rests on three pillars: (i) an adaptive dropout deep learning model to learn features from each type of data, (ii) a feature tensor model to capture the correlations of heterogeneous data, and (iii) a tensor distance-based high-order CFS algorithm to cluster heterogeneous data. Furthermore, we verify our proposed algorithm on different datasets, by comparison with other two clustering schemes, that is, HOPCM and CFS. Results confirm the effectiveness of the proposed algorithm in clustering heterogeneous data.


---

## Body

## 1. Introduction

With the rapid development of the Internet of Things, Internet of People, and Industrial Internet, big data analytics and mining have become a hot topic [1]. One widely used technique of big data analytics and mining is clustering that aims to group data into several clusters according to similarities between the data objects [2]. In 2014, Laio and Rodriguez proposed a novel clustering algorithm by fast search and finding of density peaks (CFS) published in Science Magazine [3]. CFS is the most potential clustering technique because of its efficiency and high accuracy. However, CFS is limited in clustering big data because it cannot cluster heterogeneous data which is prevalent in big data.Heterogeneous data, different from the homogeneous data containing only one type of objects, involves multiple interrelated types of objects [4]. Moreover, a heterogeneous data object is usually of complex correlations among different modalities. Therefore, heterogeneous data poses important challenges on clustering techniques. Recently, researchers have proposed some algorithms to cluster heterogeneous data [5]. One of this type is based on the graph partition, for instance, the bipartite spectral algorithm, which clusters heterogeneous data by optimizing a unified objective function. However, this kind of methods is usually of low efficiency for clustering big datasets since they need to solve an eigen-decomposition procedure. Another typical algorithm based on the nonnegative matrix factorization, such as SS-NMF, clusters heterogeneous data by revealing the relationships between different objects in a semantic space. In addition, Comrafs is developed for clustering heterogeneous data by constructing the Markov Rand Fields. Since this method is of high computational complexity, it is limited for large-scale heterogeneous data clustering. These algorithms could cluster heterogeneous data; however, they are hard to achieve desired clustering results since they do not model the high nonlinear correlations over multiple types of heterogeneous data objects effectively. Moreover, they are of high time complexity, leading to low efficiency in clustering heterogeneous data.In this paper, we propose a high-order CFS algorithm (HOCFS) for clustering heterogeneous data based on the dropout deep learning model. The dropout deep learning model was proposed by Hinton to prevent overfitting [6]. It is especially useful in training large networks with small amount of samples. However, the dropout sets the same omitting probability with 0.5 in each hidden layer of the deep learning model, resulting in its ineffectiveness. Aiming at this problem, we propose an adaptive dropout deep learning model, which sets the omitting probability of each hidden layer according to the relationship between the omitting probability and the layer opposition. Then, we applied the proposed adaptive dropout deep learning model in feature learning for each type of data of every heterogeneous data object. Next, the algorithm uses the vector outer product to fuse the learned features to form a feature tensor for each heterogeneous data object. Finally, since the tensor distance can not only measure the distance between every two heterogeneous samples but also reveal the intrinsic correlations between different coordinates in the high-order tensor space, the tensor distance is applied to the CFS algorithm for clustering heterogeneous data represented by fused features.Finally, we compare our proposed algorithm with two representative data clustering techniques, namely, HOPCM and CFS, on two datasets, namely, NUS-WIDE and CUAVE in terms ofE∗ and Rand Index (RI).Therefore, the contributions of the paper are summarized as the following three aspects:(i)
Current dropout deep learning models are of low effectiveness and efficiency in learning features for heterogeneous data. To tackle this problem, the paper proposes an adaptive dropout deep learning model to learn features for each type of data and then fuses the learned features to form a feature tensor for each heterogeneous data object.(ii)
To measure the similarity between heterogeneous data objects in high-order tensor space, the paper applies the tensor distance in the clustering process.(iii)
Conventional CFS algorithm cannot cluster heterogeneous data directly because it works in the vector space. The paper extends the CFS algorithm from the vector space to the tensor space for clustering heterogeneous data represented by the feature tensors.


## 2. Preliminaries

This section presents the technique preliminaries about our scheme, including the stacked autoencoder, dropout, and the CFS clustering algorithm. The stacked autoencoder is presented first, followed by the CFS clustering algorithm.


### 2.1. Stacked Autoencoder (SAE) and Dropout

The stacked autoencoder (SAE) that is one important example of deep learning models has been widely employed in supervised feature learning for many applications [7]. SAE is built to learn hierarchical features of data by stacking multiple basic autoencoders (BAEs) as shown in Figure 1.Figure 1
The architecture of the stacked autoencoder.As the typical module of a stacked autoencoder, a basic autoencoder (BAE) [8] learns a hidden representation h of the input data x by an encoding function f:(1)h=fθW1x+b1.Then, BAE reconstructs the input from the hidden representationh to a reconstruction y by a decoding function s:(2)y=sθW2x+b2,where θ=(W(1),b(1);W(2),b(2)) denotes the parameter of the autoencoder and the functions f and s typically adopt the sigmoid function: f(x)=1/(1+e-x).To train the parameter of the autoencoder, an objective function with a weight-decay that is used to prevent overfitting is defined as follows:(3)Jθ=∑x∈DLx,sfx+λ∑ijWij2,where L is the reconstruction error and λ is a hyperparameter used to control the strength of the regularization.The stacked autoencoder is a full-connected model and it involves many redundant connections. Therefore, it usually produces overfitting in the real applications. Aiming at this problem, Hinton proposed dropout to reduce the overfitting by preventing coadaption of feature detectors in deep learning models. It randomly omits half of the feature detectors on each training sample to prevent a hidden unit from relying on other hidden units being present. Dropout was proved to be especially effective and efficient in training a large neural network with a small training set.


### 2.2. Clustering by Fast Search and Finding of Density Peaks (CFS)

CFS is the latest clustering algorithm proposed by Laio and Rodriguez in Science Magazine in 2014 [3]. It is highly robust and efficient. More importantly, it can find clusters of arbitrary shape and determine the number of clusters automatically. Several experiments have demonstrated its superiority in the efficiency and effectiveness over the previous algorithms for clustering large amounts of data. Therefore, it has become the most potential algorithm for clustering big data.The key of the CFS algorithm lies in the characterization of cluster centers. Particularly, the algorithm basically assumes that cluster centers should be surrounded by neighbor objects with lower local density and be more far away from other objects with a higher local density. Based on this assumption, CFS defines two quantities for every data objectxi, the local density ρi and the minimum distance δi from any other object with higher density, in(4)ρi=∑jχdij-dcχx=1ifx<00otherwiseδi=minj:ρj>ρi⁡dij,where dc represents a cutoff distance. According to Laio and Rodriguez, dc can be set to the biggest 2% of all the distances between every two objects to get a good clustering result. For the object xi with the highest density, its distance δi is taken as δi=maxj(dij).In the CFS algorithm, cluster centers are recognized as the objects with the large value ofγ that is defined in(5)γi=ρi×δi.


## 2.1. Stacked Autoencoder (SAE) and Dropout

The stacked autoencoder (SAE) that is one important example of deep learning models has been widely employed in supervised feature learning for many applications [7]. SAE is built to learn hierarchical features of data by stacking multiple basic autoencoders (BAEs) as shown in Figure 1.Figure 1
The architecture of the stacked autoencoder.As the typical module of a stacked autoencoder, a basic autoencoder (BAE) [8] learns a hidden representation h of the input data x by an encoding function f:(1)h=fθW1x+b1.Then, BAE reconstructs the input from the hidden representationh to a reconstruction y by a decoding function s:(2)y=sθW2x+b2,where θ=(W(1),b(1);W(2),b(2)) denotes the parameter of the autoencoder and the functions f and s typically adopt the sigmoid function: f(x)=1/(1+e-x).To train the parameter of the autoencoder, an objective function with a weight-decay that is used to prevent overfitting is defined as follows:(3)Jθ=∑x∈DLx,sfx+λ∑ijWij2,where L is the reconstruction error and λ is a hyperparameter used to control the strength of the regularization.The stacked autoencoder is a full-connected model and it involves many redundant connections. Therefore, it usually produces overfitting in the real applications. Aiming at this problem, Hinton proposed dropout to reduce the overfitting by preventing coadaption of feature detectors in deep learning models. It randomly omits half of the feature detectors on each training sample to prevent a hidden unit from relying on other hidden units being present. Dropout was proved to be especially effective and efficient in training a large neural network with a small training set.


## 2.2. Clustering by Fast Search and Finding of Density Peaks (CFS)

CFS is the latest clustering algorithm proposed by Laio and Rodriguez in Science Magazine in 2014 [3]. It is highly robust and efficient. More importantly, it can find clusters of arbitrary shape and determine the number of clusters automatically. Several experiments have demonstrated its superiority in the efficiency and effectiveness over the previous algorithms for clustering large amounts of data. Therefore, it has become the most potential algorithm for clustering big data.The key of the CFS algorithm lies in the characterization of cluster centers. Particularly, the algorithm basically assumes that cluster centers should be surrounded by neighbor objects with lower local density and be more far away from other objects with a higher local density. Based on this assumption, CFS defines two quantities for every data objectxi, the local density ρi and the minimum distance δi from any other object with higher density, in(4)ρi=∑jχdij-dcχx=1ifx<00otherwiseδi=minj:ρj>ρi⁡dij,where dc represents a cutoff distance. According to Laio and Rodriguez, dc can be set to the biggest 2% of all the distances between every two objects to get a good clustering result. For the object xi with the highest density, its distance δi is taken as δi=maxj(dij).In the CFS algorithm, cluster centers are recognized as the objects with the large value ofγ that is defined in(5)γi=ρi×δi.


## 3. Problem Statement

Consider a dataset withn heterogeneous data objects X=x1,x2,…,xn and assume that each object can be represented by a feature tensor. The task of heterogeneous data clustering is to classify the dataset into groups according to their similarity such that the objects belonging to the same cluster share similarity as much as possible. Based on the analysis in the previous parts, heterogeneous data poses a large number of challenges on the clustering techniques. We discuss the key issues in three following aspects:(1)
Feature Learning of Heterogeneous Data. Feature learning is the fundamental step for heterogeneous data clustering. In fact, many feature learning algorithms, especially some methods based on deep learning, have been well studied in recent years. However, most of them are hard to learn features for heterogeneous data. Although the deep computation model can learn features for heterogeneous data, it is of low accuracy and efficiency since it cannot avoid overfitting.(2)
Similarity Measurement for Heterogeneous Data. Similarity measurement is the key to one clustering technique. There are a lot of metrics for measuring the similarity between two objects. However, they can only measure the distance between homogeneous objects represented by feature vectors because they work in the vector space. A heterogeneous object is typically represented by a feature tensor, making most of current metrics hard to calculate the similarity for heterogeneous data objects.(3)
Clustering Technique for Heterogeneous Data. Typically, a heterogeneous object is represented by a feature tensor. However, most of clustering techniques including the CFS algorithm work only in the vector space, resulting in failure to cluster heterogeneous data in the high-order tensor space.


## 4. High-Order CFS Algorithm Based on Dropout Deep Computation Model

In this section, we describe the details of the proposed high-order CFS algorithm based on the dropout deep learning model for clustering heterogeneous data. The proposed algorithm works in three stages: unsupervised feature learning, feature fusion, and high-order clustering, which is shown in Figure2.Figure 2
The architecture of the proposed scheme.In the first stage, each type of data in the heterogeneous dataset is separately learned by the proposed adaptive dropout deep learning model. In the second stage, the proposed algorithm uses the vector outer product to fuse the learned futures to form a feature tensor as the joint representation of each object. Finally, the proposed algorithm extends the conventional CFS technique from the vector space to the tensor space for clustering the heterogeneous dataset.


### 4.1. Feature Learning Based on the Adaptive Dropout Deep Learning Model

In the dropout deep learning model, each hidden unit is randomly omitted from the network always with a constant probability of 0.5. This way will ignore the relationship between the omitting probability and the layer opposition, resulting in a low effectiveness of deep learning models in heterogeneous data feature learning. A large number of studies demonstrate that the fundamental layers of a deep architecture share many common characters, implying that the dropout in the lower layers has more generalization function than that in higher layers. Therefore, the omitting probability of the dropout should decay with the layers becoming higher.Based on the above analysis, we propose an adaptive dropout deep learning model by defining a distribution model of the omitting probabilityy of dropout as the following function:(6)y=fl=-0.1l+0.05n+0.5n=2kk=1,2,…-0.1l+0.05n+0.55n=2k-1k=1,2,…,where n≤9 denotes the number of hidden layers in the deep learning model and l represents the position of the layer.Function (6) has the following properties:(1)
it is monotonically decreasing.(2)
The omitting probability is 0.5 for the middle hidden layer.(3)
The omitting probability is always in (0,1) for x=1,2,…,n.Proof.
(1) By the assumption, functionf(l) is continuously differentiable and we may write(7)f′l=-0.1<0,which implies that (6) is a strictly decreasing function. Particularly, the omitting probability of the dropout should decay with the layers becoming higher.
(2) Whenn=2k(k=1,2,…),(8)fn2=-0.1×n2+0.05×n+0.5=0.5.
Whenn=2k-1(k=1,2,…),(9)fn+12=-0.1×n+12+0.05×n+0.55=0.5,which proves that the omitting probability is 0.5 for the middle hidden layer.
(3) Based on property (1),(10)fn≤fl≤f11≤n≤9.
Then,(11)fn=-0.05n+0.5≥-0.05×8+0.5=0.1>0-0.05n+0.55≥-0.05×9+0.55=0.1>0f1=0.05n+0.45≤0.05×9+0.45=0.9<1.
Therefore, the omitting probability is always in (0,1) for l=1,2,…,n.We can get the adaptive dropout deep learning model by applying the distribution function of the omitting probability to the deep learning model outlined in Algorithm1.Algorithm 1:Adaptive dropout backpropagation neural network learning algorithm.
Input: {(X(i),Y(i))},1≤i≤N, iteratermax, η, threshold Output: θ=W(1),b(1);W(2),b(2) (1)  Randomly initialize allθ=W(1),b(1);W(2),b(2); (2)y=f(l); (3)for  iteration=1,2,…,iteratermax  do (4)for  example=1,2,…,N  do (5)for  j=1,2,…,m  do (6)zj(2)=wji(1)⋅xi+bj(1); (7)aj(2)=f(zj(2)); (8)mark{i}=rand(size(a(2))>y); (9)a(2)=a(2)⋅∗mark{i}; (10)for  i=1,2,…,n  do (11)zi(3)=wij(2)⋅aj(2)+bi(2); (12)ai(3)=f(zi(3)); (13)for  i=1,2,…,n  do (14)σi(3)=-(y-ai3)⋅f′(zi(3)); (15)for  j=1,2,…,m  do (16)σj(2)=(∑i=1nwij(2)⋅σi(3))⋅f′(zj(2)); (17)σ(2)=σ(2)⋅∗[ones(size(σ(2),1),1)mark{i}]; (18)for  i=1,2,…,n  do (19)bi(2)=σi(3); (20)for  j=1,2,…,m  do (21)Δwij(2)=aj(2)⋅σi(3); (22)for  j=1,2,…,m  do (23)bj(1)=σj(2); (24)for  i=1,2,…,n  do (25)Δwji(1)=xi⋅σj(2);In the proposed high-order CFS algorithm, the adaptive dropout deep learning model is used to learn features of each type of data of the heterogeneous data.


### 4.2. Feature Fusion Using Vector Outer Product

The vector outer product is one of the widely used operations in mathematics, denoted by⊗. If A is an m-dimension vector and B is an n-dimension vector, their outer product will produce an m×n matrix C; C=A⊗B. Each entry in the matrix C is defined as cij=ai·bj, where ai and bj are one entry in vectors A and B, respectively. One example of the vector outer product is as shown in (6):(12)b1b2b3b4⊗a1a2a3=a1b1a2b1a3b1a1b2a2b2a3b2a1b3a2b3a3b3a1b4a2b4a3b4.More generally, the outer product ofn vectors A1∈RI1,A2∈RI2,…,An∈RIn will produce an n-order tensor B∈RI1×I2×⋯×In, B=A1⊗A2⊗⋯⊗An, in which each entry is defined as bi1i2⋯in=a1i1·a2i2·…·anin.After using the adaptive deep learning model to learn features of heterogeneous data, each type of data can be represented by a feature vector. Particularly, for the heterogeneous dataset in which each object consists of one image, one text, and one piece of video, three vectors,a, b, and c, are used to represent the feature vectors learned from the adaptive dropout deep learning model, respectively. In this subsection, such feature vectors are fused by the vector outer product to form one feature tensor X for joint representation of one object in the heterogeneous dataset according to the following rules:(1)
For the object with only one image and one text, its feature tensor is represented byX=a⊗b.(2)
For the object with only one image and one piece of video, its feature tensor is represented byX=a⊗c.(3)
For the object with only one text and one piece of video, its feature tensor is represented byX=b⊗c.(4)
For the object with only one image, one text, and one piece of video, its feature tensor is represented byX=a⊗b⊗c.


### 4.3. The High-Order CFS Clustering

As discussed in Section2, the conventional CFS algorithm cannot cluster heterogeneous data directly because it works in the vector space while each object in the heterogeneous dataset is represented by a feature tensor. To tackle this problem, we propose a high-order CFS algorithm for clustering heterogeneous data.To calculate the distance between two points in high-order tensor space, represented by two tensors,X,Y∈RI1×I2×⋯×IN, they need to be unfolded to the corresponding vectors. In detail, the item Xi1i2⋯iN is unfolded to xl by l=i1+∑j=2N‍∏t=1j-1‍It.The proposed high-order CFS clustering algorithm (HOCFS) based on the feature tensor is outlined in Algorithm2.Algorithm 2:High-order CFS clustering algorithm.
Input  X={X1,X2,…,XN}, dc Output  cl[n],center[k] (1)for  i=1,2,…,n  do (2)for  j=i+1,i+2,…,n  do (3)dij=(Xi-Xj)TG(Xi-Xj); (4)for  i=1,2,…,n  do (5)ρi=∑jχ(dij-dc); (6)for  i=1,2,…,n   do (7)δi=minj:ρj>ρi{dij}; (8)γi=ρi×δi; (9)   Select clustering centers according toγi; (10)for  i=1,2,…,n  do (11)cl[i]=minj:centers[k]{dij};


## 4.1. Feature Learning Based on the Adaptive Dropout Deep Learning Model

In the dropout deep learning model, each hidden unit is randomly omitted from the network always with a constant probability of 0.5. This way will ignore the relationship between the omitting probability and the layer opposition, resulting in a low effectiveness of deep learning models in heterogeneous data feature learning. A large number of studies demonstrate that the fundamental layers of a deep architecture share many common characters, implying that the dropout in the lower layers has more generalization function than that in higher layers. Therefore, the omitting probability of the dropout should decay with the layers becoming higher.Based on the above analysis, we propose an adaptive dropout deep learning model by defining a distribution model of the omitting probabilityy of dropout as the following function:(6)y=fl=-0.1l+0.05n+0.5n=2kk=1,2,…-0.1l+0.05n+0.55n=2k-1k=1,2,…,where n≤9 denotes the number of hidden layers in the deep learning model and l represents the position of the layer.Function (6) has the following properties:(1)
it is monotonically decreasing.(2)
The omitting probability is 0.5 for the middle hidden layer.(3)
The omitting probability is always in (0,1) for x=1,2,…,n.Proof.
(1) By the assumption, functionf(l) is continuously differentiable and we may write(7)f′l=-0.1<0,which implies that (6) is a strictly decreasing function. Particularly, the omitting probability of the dropout should decay with the layers becoming higher.
(2) Whenn=2k(k=1,2,…),(8)fn2=-0.1×n2+0.05×n+0.5=0.5.
Whenn=2k-1(k=1,2,…),(9)fn+12=-0.1×n+12+0.05×n+0.55=0.5,which proves that the omitting probability is 0.5 for the middle hidden layer.
(3) Based on property (1),(10)fn≤fl≤f11≤n≤9.
Then,(11)fn=-0.05n+0.5≥-0.05×8+0.5=0.1>0-0.05n+0.55≥-0.05×9+0.55=0.1>0f1=0.05n+0.45≤0.05×9+0.45=0.9<1.
Therefore, the omitting probability is always in (0,1) for l=1,2,…,n.We can get the adaptive dropout deep learning model by applying the distribution function of the omitting probability to the deep learning model outlined in Algorithm1.Algorithm 1:Adaptive dropout backpropagation neural network learning algorithm.
Input: {(X(i),Y(i))},1≤i≤N, iteratermax, η, threshold Output: θ=W(1),b(1);W(2),b(2) (1)  Randomly initialize allθ=W(1),b(1);W(2),b(2); (2)y=f(l); (3)for  iteration=1,2,…,iteratermax  do (4)for  example=1,2,…,N  do (5)for  j=1,2,…,m  do (6)zj(2)=wji(1)⋅xi+bj(1); (7)aj(2)=f(zj(2)); (8)mark{i}=rand(size(a(2))>y); (9)a(2)=a(2)⋅∗mark{i}; (10)for  i=1,2,…,n  do (11)zi(3)=wij(2)⋅aj(2)+bi(2); (12)ai(3)=f(zi(3)); (13)for  i=1,2,…,n  do (14)σi(3)=-(y-ai3)⋅f′(zi(3)); (15)for  j=1,2,…,m  do (16)σj(2)=(∑i=1nwij(2)⋅σi(3))⋅f′(zj(2)); (17)σ(2)=σ(2)⋅∗[ones(size(σ(2),1),1)mark{i}]; (18)for  i=1,2,…,n  do (19)bi(2)=σi(3); (20)for  j=1,2,…,m  do (21)Δwij(2)=aj(2)⋅σi(3); (22)for  j=1,2,…,m  do (23)bj(1)=σj(2); (24)for  i=1,2,…,n  do (25)Δwji(1)=xi⋅σj(2);In the proposed high-order CFS algorithm, the adaptive dropout deep learning model is used to learn features of each type of data of the heterogeneous data.


## 4.2. Feature Fusion Using Vector Outer Product

The vector outer product is one of the widely used operations in mathematics, denoted by⊗. If A is an m-dimension vector and B is an n-dimension vector, their outer product will produce an m×n matrix C; C=A⊗B. Each entry in the matrix C is defined as cij=ai·bj, where ai and bj are one entry in vectors A and B, respectively. One example of the vector outer product is as shown in (6):(12)b1b2b3b4⊗a1a2a3=a1b1a2b1a3b1a1b2a2b2a3b2a1b3a2b3a3b3a1b4a2b4a3b4.More generally, the outer product ofn vectors A1∈RI1,A2∈RI2,…,An∈RIn will produce an n-order tensor B∈RI1×I2×⋯×In, B=A1⊗A2⊗⋯⊗An, in which each entry is defined as bi1i2⋯in=a1i1·a2i2·…·anin.After using the adaptive deep learning model to learn features of heterogeneous data, each type of data can be represented by a feature vector. Particularly, for the heterogeneous dataset in which each object consists of one image, one text, and one piece of video, three vectors,a, b, and c, are used to represent the feature vectors learned from the adaptive dropout deep learning model, respectively. In this subsection, such feature vectors are fused by the vector outer product to form one feature tensor X for joint representation of one object in the heterogeneous dataset according to the following rules:(1)
For the object with only one image and one text, its feature tensor is represented byX=a⊗b.(2)
For the object with only one image and one piece of video, its feature tensor is represented byX=a⊗c.(3)
For the object with only one text and one piece of video, its feature tensor is represented byX=b⊗c.(4)
For the object with only one image, one text, and one piece of video, its feature tensor is represented byX=a⊗b⊗c.


## 4.3. The High-Order CFS Clustering

As discussed in Section2, the conventional CFS algorithm cannot cluster heterogeneous data directly because it works in the vector space while each object in the heterogeneous dataset is represented by a feature tensor. To tackle this problem, we propose a high-order CFS algorithm for clustering heterogeneous data.To calculate the distance between two points in high-order tensor space, represented by two tensors,X,Y∈RI1×I2×⋯×IN, they need to be unfolded to the corresponding vectors. In detail, the item Xi1i2⋯iN is unfolded to xl by l=i1+∑j=2N‍∏t=1j-1‍It.The proposed high-order CFS clustering algorithm (HOCFS) based on the feature tensor is outlined in Algorithm2.Algorithm 2:High-order CFS clustering algorithm.
Input  X={X1,X2,…,XN}, dc Output  cl[n],center[k] (1)for  i=1,2,…,n  do (2)for  j=i+1,i+2,…,n  do (3)dij=(Xi-Xj)TG(Xi-Xj); (4)for  i=1,2,…,n  do (5)ρi=∑jχ(dij-dc); (6)for  i=1,2,…,n   do (7)δi=minj:ρj>ρi{dij}; (8)γi=ρi×δi; (9)   Select clustering centers according toγi; (10)for  i=1,2,…,n  do (11)cl[i]=minj:centers[k]{dij};


## 5. Performance Evaluation of Adaptive Dropout Model

In this part, we assess the adaptive dropout deep learning model on the STL-10 and CIFAR-10 datasets by comparison with the conventional dropout model.


### 5.1. Experiments on the STL-10 Dataset

We initially explored the effectiveness of adaptive dropout using STL-10, a widely used benchmark for machine learning algorithms. It contains 500 training images, 800 testing images that are grouped by 10 classes, and 100000 unlabeled images for unsupervised learning. We combine the adaptive dropout distribution model with stacked autoencoders to train two deep learning models. One has 4 hidden layers while the other has 5 hidden layers. Both of them have one logistic regression layer on the top. For the adaptive dropout deep learning model, we use the proposed algorithm to set the omitted rate of hidden units while setting omitted rate of 0.5 of hidden units for the conventional dropout deep learning model. The classification results are presented in Figures3 and 4.Figure 3
Classification result on STL-10 with 4 hidden layers.Figure 4
Classification result on STL-10 with 5 hidden layers.From Figures3 and 4, the classification error decreases with the epoch increasing. The classification error produced by the adaptive dropout model is lower than that produced by the conventional dropout model. Particularly, we achieved the best classification error rate of 0.10 by using adaptive dropout model with 4 hidden layers while the best classification error rate given by the conventional dropout model is 0.12, which indicates that our proposed model performs better than the conventional dropout model in classifying the STL-10 dataset.


### 5.2. Experiments on the CIFAR-10 Dataset

CIFAR-10 is a benchmark task for object recognition, consisting of 60000 color images in 10 groups, with 6000 images per group. These images were labeled by hand to produce 50000 training images and 10000 test images. We built a classification network with three convolutional layers and three pooling and two fully connected layers to explore the effectiveness of the adaptive dropout model on CIFAR-10 dataset. Each convolutional layer has an exclusive ReLU layer and a dropout layer. Specially for the adaptive dropout deep learning model, we use the proposed algorithm to set the omitted rate of hidden units while setting omitted rate of 0.5 of hidden units for the conventional dropout deep learning model. The classification results are presented in Figure5.Figure 5
Classification result on CIFAR-10.From Figure5, the error rate produced by the adaptive dropout model is lower than that produced by the conventional dropout model in most cases. More importantly, using the conventional dropout model gives the best error rate of 0.156. This is reduced to 0.136 by using the adaptive dropout model, which implies that the proposed model works much better than the conventional dropout model for CIFAR-10.


## 5.1. Experiments on the STL-10 Dataset

We initially explored the effectiveness of adaptive dropout using STL-10, a widely used benchmark for machine learning algorithms. It contains 500 training images, 800 testing images that are grouped by 10 classes, and 100000 unlabeled images for unsupervised learning. We combine the adaptive dropout distribution model with stacked autoencoders to train two deep learning models. One has 4 hidden layers while the other has 5 hidden layers. Both of them have one logistic regression layer on the top. For the adaptive dropout deep learning model, we use the proposed algorithm to set the omitted rate of hidden units while setting omitted rate of 0.5 of hidden units for the conventional dropout deep learning model. The classification results are presented in Figures3 and 4.Figure 3
Classification result on STL-10 with 4 hidden layers.Figure 4
Classification result on STL-10 with 5 hidden layers.From Figures3 and 4, the classification error decreases with the epoch increasing. The classification error produced by the adaptive dropout model is lower than that produced by the conventional dropout model. Particularly, we achieved the best classification error rate of 0.10 by using adaptive dropout model with 4 hidden layers while the best classification error rate given by the conventional dropout model is 0.12, which indicates that our proposed model performs better than the conventional dropout model in classifying the STL-10 dataset.


## 5.2. Experiments on the CIFAR-10 Dataset

CIFAR-10 is a benchmark task for object recognition, consisting of 60000 color images in 10 groups, with 6000 images per group. These images were labeled by hand to produce 50000 training images and 10000 test images. We built a classification network with three convolutional layers and three pooling and two fully connected layers to explore the effectiveness of the adaptive dropout model on CIFAR-10 dataset. Each convolutional layer has an exclusive ReLU layer and a dropout layer. Specially for the adaptive dropout deep learning model, we use the proposed algorithm to set the omitted rate of hidden units while setting omitted rate of 0.5 of hidden units for the conventional dropout deep learning model. The classification results are presented in Figure5.Figure 5
Classification result on CIFAR-10.From Figure5, the error rate produced by the adaptive dropout model is lower than that produced by the conventional dropout model in most cases. More importantly, using the conventional dropout model gives the best error rate of 0.156. This is reduced to 0.136 by using the adaptive dropout model, which implies that the proposed model works much better than the conventional dropout model for CIFAR-10.


## 6. Performance Evaluation of the High-Order CFS Algorithm

In this part, we evaluate the high-order CFS clustering algorithm by comparison with the HOPCM algorithm and the conventional CFS algorithm on two representative heterogeneous datasets, namely, NUS-WIDE and CUAVE, in terms ofE∗ and Rand Index (RI).HOCPM was developed in 2015 for clustering heterogeneous data by combining the autoencoder model and the possibilisticc-means algorithm [9]. For the conventional CFS algorithm, we perform the same preprocessing step with our proposed algorithm. Particularly, we first use the adaptive dropout deep learning model to learn features of texts, audios, and images of each object and then form a feature vector for the object by concatenating the learned features. Finally, the Euclidean distance is applied for the conventional CFS algorithm to cluster the heterogeneous dataset where each object is represented by a learned feature vector.The evaluation criteria are described in Section6.1, followed by the experimental results.


### 6.1. Experiments on the NUS-WIDE Dataset

The NUS-WIDE dataset is the biggest image set, consisting of 269, 648 annotated images. To compare the proposed algorithm with the HOPCM algorithm and the conventional CFS algorithm fairly, we use the same image dataset collected from the NUS-WIDE with literature [9], which consists of 8 different subsets, each with 10,000 annotated images falling into 14 categories.First, we carried out the experiments on the overall image set for five times. The clustering results are shown in Figures6 and 7.Figure 6
Clustering result on NUS-WIDE in terms ofE∗.Figure 7
Clustering result on NUS-WIDE in terms ofRI.Figure6 shows the clustering result in terms of E∗ on the overall dataset. We observe that the proposed algorithm got the lowest values of E∗ in most cases, which implies that the proposed algorithm produced the most accurate clustering centers.From Figure7, HOCFS produced the highest values of RI in most cases, which indicates that HOCFS performs best in clustering NUS-WIDE dataset. Moreover, the conventional CFS algorithm performs worst in terms of E∗ and RI, demonstrating that the proposed algorithm could effectively capture the complex correlations over the heterogeneous data by applying the vector outer product to feature fusion and using tensor distance to measure the similarity between two objects.Next, we carried out the experiment on the 8 subsets for 5 times to evaluate the robustness of the clustering algorithms. Tables1 and 2 present the average clustering results of 5 times on every subset.Table 1
Clustering result on NUS-WIDE in terms ofE∗.
Algorithm/subset 1 2 3 4 5 6 7 8 CFS 2.64 3.01 2.99 3.04 2.73 3.02 3.08 2.82 HOPCM 2.04 2.57 2.91 2.63 2.12 2.91 2.99 2.08 HOCFS 1.96 2.24 2.37 2.28 1.95 2.16 2.39 2.01Table 2
Clustering result on NUS-WIDE in terms of RI.
Algorithm/subset 1 2 3 4 5 6 7 8 CFS 0.86 0.79 0.87 0.82 0.76 0.79 0.83 0.69 HOPCM 0.91 0.84 0.93 0.91 0.88 0.92 0.82 0.84 HOCFS 0.95 0.84 0.94 0.95 0.93 0.96 0.89 0.91From Tables1 and 2, the average values of E∗ obtained by HOCFS are lowest for each subset while the average values of RI obtained by HOCFS are significantly larger than that obtained by HOPCM and CFS. In other words, the proposed algorithm produced the best clustering results in terms of E∗ and RI for NUS-WIDE dataset.


### 6.2. Experiments on the CUAVE Dataset

CUAVE is a typical multimodal dataset consisting of some digits, 0 to 9, reported by 36 individuals. To assess HOCFS for clustering heterogeneous data, we added some annotations to each object as the literature [9].We first carried out the experiment on the CUAVE dataset for 5 times to judge HOCFS for clustering heterogeneous data in terms of RI. The result is presented in Figure8.Figure 8
Clustering result on CUAVE in terms ofRI.According to Figure8, the value of RI obtained by HOCFS is highest for each experiment, implying that the proposed algorithm produced the best clustering result for the CUAVE dataset in terms of RI. On the one hand, the proposed algorithm uses the hybrid stacked autoencoder model to learn features of each object in the CUAVE dataset while HOPCM only uses the basic autoencoder model to learn features, leading to the more accurate clustering result produced by the proposed algorithm compared to HOPCM. On the other hand, HOCFS fuses the learnt features of each modality for capturing the nonlinear correlations over multiple modalities of each object while CFS formed the feature vector for each object by only concatenating the learned features. Thus, the proposed algorithm performed the best for clustering the CUAVE dataset.Next, we evaluate the robustness of the proposed algorithm by generating three different subsets, each with a distinct combination of two modalities. We carried out the experiment on these subsets for 5 times. The results are shown in Figures9–11.Figure 9
Clustering result on image-text subset in terms ofRI.Figure 10
Clustering result on text-audio subset in terms ofRI.Figure 11
Clustering result on image-audio subset in terms ofRI.According to Figures9–11, the proposed algorithm outperformed HOPCM and CFS since HOCFS got higher values of RI than the other two algorithms in most cases, especially for clustering the text-audio subset. In other words, the proposed algorithm produced the best clustering results in terms of RI for the CUAVE subsets.Finally, we studied the relationship between the clustering result and the different combinations of modalities by analyzing the clustering results, as shown in Table3.Table 3
Clustering result on different subsets in terms of RI.
Algorithm/subset 1 2 3 4 5 Image-text 0.92 0.88 0.87 0.89 0.93 Text-audio 0.81 0.79 0.78 0.83 0.80 Image-audio 0.89 0.83 0.89 0.85 0.87 Overall 0.96 0.91 0.93 0.96 0.94From Table3, the best clustering result is always produced on the overall dataset, implying that the clustering result of heterogeneous data relies on the joint features of image-text-audio modalities. Moreover, the proposed algorithm produced the worst clustering result on text-audio subset, which demonstrates that only features learned from the text-audio modalities could not effectively represent the objects in the CUAVE dataset.


## 6.1. Experiments on the NUS-WIDE Dataset

The NUS-WIDE dataset is the biggest image set, consisting of 269, 648 annotated images. To compare the proposed algorithm with the HOPCM algorithm and the conventional CFS algorithm fairly, we use the same image dataset collected from the NUS-WIDE with literature [9], which consists of 8 different subsets, each with 10,000 annotated images falling into 14 categories.First, we carried out the experiments on the overall image set for five times. The clustering results are shown in Figures6 and 7.Figure 6
Clustering result on NUS-WIDE in terms ofE∗.Figure 7
Clustering result on NUS-WIDE in terms ofRI.Figure6 shows the clustering result in terms of E∗ on the overall dataset. We observe that the proposed algorithm got the lowest values of E∗ in most cases, which implies that the proposed algorithm produced the most accurate clustering centers.From Figure7, HOCFS produced the highest values of RI in most cases, which indicates that HOCFS performs best in clustering NUS-WIDE dataset. Moreover, the conventional CFS algorithm performs worst in terms of E∗ and RI, demonstrating that the proposed algorithm could effectively capture the complex correlations over the heterogeneous data by applying the vector outer product to feature fusion and using tensor distance to measure the similarity between two objects.Next, we carried out the experiment on the 8 subsets for 5 times to evaluate the robustness of the clustering algorithms. Tables1 and 2 present the average clustering results of 5 times on every subset.Table 1
Clustering result on NUS-WIDE in terms ofE∗.
Algorithm/subset 1 2 3 4 5 6 7 8 CFS 2.64 3.01 2.99 3.04 2.73 3.02 3.08 2.82 HOPCM 2.04 2.57 2.91 2.63 2.12 2.91 2.99 2.08 HOCFS 1.96 2.24 2.37 2.28 1.95 2.16 2.39 2.01Table 2
Clustering result on NUS-WIDE in terms of RI.
Algorithm/subset 1 2 3 4 5 6 7 8 CFS 0.86 0.79 0.87 0.82 0.76 0.79 0.83 0.69 HOPCM 0.91 0.84 0.93 0.91 0.88 0.92 0.82 0.84 HOCFS 0.95 0.84 0.94 0.95 0.93 0.96 0.89 0.91From Tables1 and 2, the average values of E∗ obtained by HOCFS are lowest for each subset while the average values of RI obtained by HOCFS are significantly larger than that obtained by HOPCM and CFS. In other words, the proposed algorithm produced the best clustering results in terms of E∗ and RI for NUS-WIDE dataset.


## 6.2. Experiments on the CUAVE Dataset

CUAVE is a typical multimodal dataset consisting of some digits, 0 to 9, reported by 36 individuals. To assess HOCFS for clustering heterogeneous data, we added some annotations to each object as the literature [9].We first carried out the experiment on the CUAVE dataset for 5 times to judge HOCFS for clustering heterogeneous data in terms of RI. The result is presented in Figure8.Figure 8
Clustering result on CUAVE in terms ofRI.According to Figure8, the value of RI obtained by HOCFS is highest for each experiment, implying that the proposed algorithm produced the best clustering result for the CUAVE dataset in terms of RI. On the one hand, the proposed algorithm uses the hybrid stacked autoencoder model to learn features of each object in the CUAVE dataset while HOPCM only uses the basic autoencoder model to learn features, leading to the more accurate clustering result produced by the proposed algorithm compared to HOPCM. On the other hand, HOCFS fuses the learnt features of each modality for capturing the nonlinear correlations over multiple modalities of each object while CFS formed the feature vector for each object by only concatenating the learned features. Thus, the proposed algorithm performed the best for clustering the CUAVE dataset.Next, we evaluate the robustness of the proposed algorithm by generating three different subsets, each with a distinct combination of two modalities. We carried out the experiment on these subsets for 5 times. The results are shown in Figures9–11.Figure 9
Clustering result on image-text subset in terms ofRI.Figure 10
Clustering result on text-audio subset in terms ofRI.Figure 11
Clustering result on image-audio subset in terms ofRI.According to Figures9–11, the proposed algorithm outperformed HOPCM and CFS since HOCFS got higher values of RI than the other two algorithms in most cases, especially for clustering the text-audio subset. In other words, the proposed algorithm produced the best clustering results in terms of RI for the CUAVE subsets.Finally, we studied the relationship between the clustering result and the different combinations of modalities by analyzing the clustering results, as shown in Table3.Table 3
Clustering result on different subsets in terms of RI.
Algorithm/subset 1 2 3 4 5 Image-text 0.92 0.88 0.87 0.89 0.93 Text-audio 0.81 0.79 0.78 0.83 0.80 Image-audio 0.89 0.83 0.89 0.85 0.87 Overall 0.96 0.91 0.93 0.96 0.94From Table3, the best clustering result is always produced on the overall dataset, implying that the clustering result of heterogeneous data relies on the joint features of image-text-audio modalities. Moreover, the proposed algorithm produced the worst clustering result on text-audio subset, which demonstrates that only features learned from the text-audio modalities could not effectively represent the objects in the CUAVE dataset.


## 7. Conclusion

In this paper, we proposed a high-order CFS algorithm for clustering heterogeneous data. One property of the paper is to devise an adaptive deep learning model and to apply it to learning features of each type of data. Furthermore, the vector outer product was used to model the correlations of each type of data to form a feature tensor for every heterogeneous data object. Another property of the proposed algorithm is to adopt the tensor distance to measure the similarity between every two heterogeneous objects. Experimental results showed that our proposed algorithm produced more accurate results than HOPCM and CFS in terms ofE∗ and RI.Recently, more and more complex heterogeneous data have been generated in many applications. For example, there are simultaneously many images and audio pieces in one web document. The future work will focus on how to cluster such complex heterogeneous dataset.



---

*Source: 4356127-2016-07-25.xml*