# Representation Learning Based on Autoencoder and Deep Adaptive Clustering for Image Clustering

Authors: Siquan Yu; Jiaxin Liu; Zhi Han; Yong Li; Yandong Tang; Chengdong Wu
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

Image clustering is a complex procedure, which is significantly affected by the choice of image representation. Most of the existing image clustering methods treat representation learning and clustering separately, which usually bring two problems. On the one hand, image representations are difficult to select and the learned representations are not suitable for clustering. On the other hand, they inevitably involve some clustering step, which may bring some error and hurt the clustering results. To tackle these problems, we present a new clustering method that efficiently builds an image representation and precisely discovers cluster assignments. For this purpose, the image clustering task is regarded as a binary pairwise classification problem with local structure preservation. Specifically, we propose here such an approach for image clustering based on a fully convolutional autoencoder and deep adaptive clustering (DAC). To extract the essential representation and maintain the local structure, a fully convolutional autoencoder is applied. To manipulate feature to clustering space and obtain a suitable image representation, the DAC algorithm participates in the training of autoencoder. Our method can learn an image representation that is suitable for clustering and discover the precise clustering label for each image. A series of real-world image clustering experiments verify the effectiveness of the proposed algorithm.


---

## Body

## 1. Introduction

Clustering is a basic unsupervised learning problem whose purpose is to divide data into several subgroups. Generally speaking, the elements in the same subgroup are similar and different from the elements of the other subgroups [1]. Image clustering is one of the fundamental high-dimension data clustering tasks in computer vision and machine learning [2]. Despite decades of development, the reliable clustering method of image data is still an outstanding problem [3, 4].From the perspective of image representation, there are two types of image clustering methods, which are the traditional image clustering methods and deep clustering methods [5]. The traditional image clustering methods group data on handcrafted features and treat feature extraction and clustering separately [6]. Based on this insight, many attempts have been dedicated to developing suitable clustering feature extracting techniques such as manually designed feature descriptors, including Bag of feature (BOW) [7], Histogram of Oriented Gradient (HOG) [8], Principal Component Analysis (PCA) [9], and Scale-Invariant Feature Transform (SIFT) [10]. However, the representation ability is limited by using handcrafted features that do not depend on the distribution of input data. How to establish an effective feature representation is a crucial problem that needs to be solved in image clustering.In recent years, a deep neural network has been successfully applied in various supervised learning tasks [11, 12]. The reasons for the success of deep neural networks are to learn more essential representation of images by constructing a network with multiple hidden layers and train the network with a large number of data [13]. Motivated by the success of deep neural networks in supervised learning, some unsupervised deep learning methods have been used to image clustering. These methods are called deep clustering [14]. Most previous deep clustering studies are two-stage training schemes based on an autoencoder. First, they usually train an autoencoder to reduce the dimension of image data. Then, the encoder acts as a feature extractor and uses a clustering algorithm to train it simultaneously. The two-stage clustering methods have been widely studied and successfully applied in many works [15–19]. The reason for the effectiveness of autoencoder based methods is that it can preserve some properties of data by adding prior knowledge to subjective. Thus, the encoder constructs a feature representation that can comprehensively describe the image information. However, since no clustering-driven objection participates during the training two-stage clustering methods, the learned encoder may not be suitable for clustering.Latter, one-stage clustering methods that jointly accomplish feature transformation and clustering come into being. Deep adaptive image clustering (DAC) is a typical one-stage image clustering algorithm [20]. It defines an effective objective and proposes a self-learning scheme to realize image clustering. The defined objective function is used to update the parameters of a convolutional network by selecting highly confident image pairs and the cluster assignment is integrated into classification labels. However, there are two crucial factors that affect the stability and effectiveness of the DAC algorithm. On the one hand, the initialization of the convolutional network is an important factor affecting the performance of DAC. On the other hand, with the training of DAC, the local structure preservation of representation cannot be guaranteed. Thus, the image representation in the distorted feature space will hurt the clustering performance.To overcome the problems of DAC, we present an image clustering representation learning method based on autoencoder (AE) [21] and deep adaptive image clustering (DAC) [20]. Specifically, to obtain the essential features of the image and provide initial parameters for DAC, we incorporate a fully convolutional autoencoder into DAC algorithm. As a clustering algorithm, DAC helps to train the autoencoder to get clustering friendly features. Autoencoder can guarantee the feature space not to be distorted. The proposed method can learn the image representation suitable for clustering and simultaneously find the clustering labels of each image. Extensive experiments verify the effectiveness of the proposed algorithm.The main contributions of this paper can be concluded in three aspects:(1)
We propose a novel system based on an AE and DEC and use it to learn an informative image representation(2)
Since AE and DEC can complement each other, we use the learned image representation to realize image clustering(3)
We conduct extensive experiments on four real-world datasets to verify the effectiveness of the proposed algorithmThe rest of the paper is organized as follows. The Section 2 will briefly introduce the related work of our paper. Section3 proposes the clustering algorithm as well as some details of the algorithm. Section 4 provides a series of experiments to verify the effectiveness of the proposed algorithm. The last section briefly concludes our paper.


## 2. Related Work

### 2.1. Deep Clustering

Deep clustering refers to clustering with the related algorithm of deep neural networks [22]. Existing deep clustering algorithms are mainly to seek some effective ways to combine deep feature learning with traditional clustering methods, which are mainly divided into two categories: (I) a two-stage work that apply clustering after a representation is learned; (II) a one-stage work that jointly optimize the representation learning and clustering [23].Two-stage methods usually train an autoencoder at the first stage. Then, the encoder acts as a feature extractor and uses a clustering algorithm to obtain the clustering results. Autoencoder (AE) is a classical feature learning method that is based on deep neural networks and image reconstruction loss function [18]. Recently, many image algorithms attempt to regularize the learning of image representation of autoencoder with the loss function of the traditional clustering algorithm. For instance, Deep Embedding Clustering (DEC) utilizes KL-divergence as a loss function to measure the distance between the distribution of image feature and the target distribution [24]. Ghasedi Dizaji et al. propose Stacked Auto-Encoder (SAE) to learn a deep learning-based latent feature representation and use it to improve classification [18]. Peng et al. propose a novel clustering method by minimizing the discrepancy between pairwise sample assignments for each data point [25]. Gaussian Mixture Variational Autoencoders (GMVAE) is a representative generation-based clustering algorithm that incorporates Gaussian distribution to variational autoencoder [16]. The advantage of AE is that it keeps the essential information of features in the process of clustering algorithm training, and the learned representation is more suitable for clustering tasks. Thus, it can avoid the degradation of the solution and improve clustering performance. The disadvantage of the two-stage method is the mismatch problem between image representation and clustering. Specifically, the clustering algorithm does not participate in representation learning, which will lead to the blindness of representation learning.One-stage methods combine image representation with clustering learning. For instance, deep adaptive image clustering (DAC) is a typical one-stage image clustering algorithm [20]. It defines an effective objective and proposes an adaptive mechanism to realize image clustering. Guo et al. propose Improved Deep Embedded Clustering (IDEC) algorithm to take care of data local structure preservation [26]. IDEC trains AE and self-training simultaneously to realize local feature preservation. (CatGAN) uses general Generative Network Adversarial (GAN) and entropy as loss function to realize data clustering [27]. JULE proposes a recurrent framework for joint unsupervised learning of deep representations and image clusters [17]. The effectiveness of these learning schemes has been proved in theory and practical experiments. However, there are two crucial factors that affect the stability and effectiveness of these algorithms. On the one hand, the initialization of the convolutional network is an important factor. On the other hand, with the training going on, the local structure preservation of representation cannot be guaranteed.


## 2.1. Deep Clustering

Deep clustering refers to clustering with the related algorithm of deep neural networks [22]. Existing deep clustering algorithms are mainly to seek some effective ways to combine deep feature learning with traditional clustering methods, which are mainly divided into two categories: (I) a two-stage work that apply clustering after a representation is learned; (II) a one-stage work that jointly optimize the representation learning and clustering [23].Two-stage methods usually train an autoencoder at the first stage. Then, the encoder acts as a feature extractor and uses a clustering algorithm to obtain the clustering results. Autoencoder (AE) is a classical feature learning method that is based on deep neural networks and image reconstruction loss function [18]. Recently, many image algorithms attempt to regularize the learning of image representation of autoencoder with the loss function of the traditional clustering algorithm. For instance, Deep Embedding Clustering (DEC) utilizes KL-divergence as a loss function to measure the distance between the distribution of image feature and the target distribution [24]. Ghasedi Dizaji et al. propose Stacked Auto-Encoder (SAE) to learn a deep learning-based latent feature representation and use it to improve classification [18]. Peng et al. propose a novel clustering method by minimizing the discrepancy between pairwise sample assignments for each data point [25]. Gaussian Mixture Variational Autoencoders (GMVAE) is a representative generation-based clustering algorithm that incorporates Gaussian distribution to variational autoencoder [16]. The advantage of AE is that it keeps the essential information of features in the process of clustering algorithm training, and the learned representation is more suitable for clustering tasks. Thus, it can avoid the degradation of the solution and improve clustering performance. The disadvantage of the two-stage method is the mismatch problem between image representation and clustering. Specifically, the clustering algorithm does not participate in representation learning, which will lead to the blindness of representation learning.One-stage methods combine image representation with clustering learning. For instance, deep adaptive image clustering (DAC) is a typical one-stage image clustering algorithm [20]. It defines an effective objective and proposes an adaptive mechanism to realize image clustering. Guo et al. propose Improved Deep Embedded Clustering (IDEC) algorithm to take care of data local structure preservation [26]. IDEC trains AE and self-training simultaneously to realize local feature preservation. (CatGAN) uses general Generative Network Adversarial (GAN) and entropy as loss function to realize data clustering [27]. JULE proposes a recurrent framework for joint unsupervised learning of deep representations and image clusters [17]. The effectiveness of these learning schemes has been proved in theory and practical experiments. However, there are two crucial factors that affect the stability and effectiveness of these algorithms. On the one hand, the initialization of the convolutional network is an important factor. On the other hand, with the training going on, the local structure preservation of representation cannot be guaranteed.


## 3. Image Clustering Based on AE and DEC

### 3.1. Autoencoder

AE is a type of artificial neural network which are used to learn efficient data codings in an unsupervised manner [28]. Generally speaking, the objection of an AE is to extract a feature (encoding) of the input data. In the field of computer vision, it is usually used to learn image representations and reduce image dimension [16, 29].ConsiderX:=xii=1n being a set of images, where n denotes the number of images. An AE reduces the dimension of images from high-dimensional spaces ℝD to a low dimensional space Td and d<D. The embedding of the dataset Td is denoted by Y:=yii=1n. The function that performs the embedding denotes as fθ. Thus, yi=fθxi,i=1,2,...,n. Generally, to guarantee the learned representation Y can adequately represent the input image information. The following reconstruction loss is used to train the autoencoder network:(1)ℒaθ=12n∑i=1nxi−gwfθxi22,where gw⋅ is the decoder that maps the representation to the output.In our algorithm, to extract essential features and preserve spatial locality of images, we adopt a fully convolutional autoencoder to realize the image feature extraction stream.


### 3.2. Deep Adaptive Image Clustering

DAC is a clustering algorithm that is realized by a convolutional neural network (CNN) and an adaptive training mechanism [20]. It employs some constraints on the classification output and generates a feature for image clustering.Let us assume thatxi and yj are two unlabeled images, rij denotes an unknown binary output received by the generated label, where rij=1 if xi and yj belong to the same cluster and otherwise, rij=0. In this case, fθ′xi⋅fθ′xj=zi⋅zj is the dot product of zi and zj, and it indicates the similarity, where fθ′⋅ is a classification network. Based on the similarity of the input image features, the binary labels are defined as follows:(2)rij=1,0,None,zi⋅zj≥uξ,zi⋅zj≤lξ,otherwise,where ξ is an adaptive parameter, u⋅ and l⋅ are two learnable thresholds.For network training, the objection function of DAC is defined as follows:(3)minθ,ξ∑i,jvijℒrrij,sxi,xj,θ+uξ−lξ,where s⋅ denote the estimated similarity of xi and xj with a classification network parameter θ, vij an indicator coefficient matrix to predict the training samples, which vij=1 means that the sample is selected to train the network, and vij=0 otherwise. vij is defined as follows:(4)vij=1,0,ifrij=1,otherwise,i,j=1,2,...,n.ℒrrij,sxi,xj,θ is the loss function and is defined as follows:(5)ℒrrij,sxi,xj,θ=−rijlogsxi,xj,θ1−rijlog1−sxi,xj,θ.


### 3.3. Network Architecture

The network architecture of the proposed clustering algorithm is shown in Figure1. There are two streams in our network, the autoencoder stream, and the DAC stream. The autoencoder stream is realized by several fully convolutional layers and the DAC stream is composed of the autoencoder’s encoder and several fully connected layers. Considering a dataset D:=xii=1n with n input samples that need to be clustered. The number of clustering K is a priori knowledge. Z:=zii=1n be the output of the encoder. Thus, the encoder can be defined as a nonlinear mapping fθ1:xi⟶zi and the decoder is gw:zi⟶x˜i where θ1 and w are the parameters of encoder and decoder, respectively. x˜i denotes the output of the decoder and the output of DAC can be represented as fθ1,θ2xi, which is a classification network and θ2 is the parameters of fully connected layers.Figure 1
The network architecture of the proposed clustering algorithm.


### 3.4. Loss Function

Since we aim to seek an encoder that makes the extracted feature more suitable for clustering. The reconstruction loss of the autoencoder is added to the initialization and training process of the DAC algorithm. On the one hand, the reconstructive loss function is used to assist the learning of image representation as it can learn the essential feature of input images and avoid the distortion of feature space in the training process of DAC. On the other hand, the loss of an autoencoder is only focused on image reconstruction, which loses the useful information needed for clustering. DAC loss can guide it to obtain a better representation suitable for clustering. Thus, we define the complete loss function as follows:(6)ℒ=αℒr+βℒc,where α is a balance coefficient. ℒr and ℒc are reconstruction loss and clustering loss, respectively. The final objective function is as follows:(7)minθ1,θ2,w,ξαℒr+βℒc+uξ+lξ,where the definitions of ℒr and ℒc are as follows:(8)ℒrθ1=12n∑i=1nxi−gwfθ1xi22,ℒcrij,sxi,xj,θ1,θ2=−rijlogsxi,xj,θ1,θ21−rijlog1−sxi,xj,θ1,θ2.


### 3.5. Network Training

In this section, we present the whole training process of our algorithm. To minimize the loss function proposed in (7), we first abandon the DAC stream and pretrain a convolutional autoencoder by using loss Lr. The trained encoder can provide initial parameters for the DAC algorithm. Thus, the DAC algorithm will select more accurate labeled image pairs in the initial stage. Then, we simultaneously train the autoencoder stream and the DAC stream by minimizing (7). The detailed algorithm is formalized as Algorithm 1.Algorithm 1:
Input: input imagesD:=xii=1n; number of clusters K; the threshold uξ=0.95−ξ and lξ=0.5+0.1⋅ξ; the batch size bs=128 and the learning rate η=0.005 and ρ=0.0001; the balance coefficient α=0.5,β=0.3.Output: reconstruction imagesD˜:=x˜ii=1n; clustering labels of input images.(1)
Pertain a fully convolutional autoencoder;(2)
fort in epochs do(3)
Extract the images featurefθ1xi in the batch Xbs;(4)
Compute the similarityrij and pseudolabel based on (2);(5)
Calculate the indicator coefficientvij based on (4);(6)
Updataθ1, θ2 and w by minimizing (7);(7)
end for(8)
Updataξ by minimizing (7).(9)
forxi in X do(10)
Ii:=fθ1,θ2xi and ci:=argmaxhIih;(11)
end forWhen the autoencoder is trained, it seems to get a clustering friendly representation by finetuning the autoencoder’s encoder in the DAC algorithm. The encoder is connected with a number of fully connected layers to form a classification network, and the labels are calculated by the algorithm proposed in [20]. However, we suppose that this kind of finetuning can distort the representation space, which may weaken the expressive power and thereby hurt clustering results. For this reason, in the process of training the DAC stream, the autoencoder also needs to be trained to maintain the DAC algorithm to obtain highly confidence labeled image pairs.


## 3.1. Autoencoder

AE is a type of artificial neural network which are used to learn efficient data codings in an unsupervised manner [28]. Generally speaking, the objection of an AE is to extract a feature (encoding) of the input data. In the field of computer vision, it is usually used to learn image representations and reduce image dimension [16, 29].ConsiderX:=xii=1n being a set of images, where n denotes the number of images. An AE reduces the dimension of images from high-dimensional spaces ℝD to a low dimensional space Td and d<D. The embedding of the dataset Td is denoted by Y:=yii=1n. The function that performs the embedding denotes as fθ. Thus, yi=fθxi,i=1,2,...,n. Generally, to guarantee the learned representation Y can adequately represent the input image information. The following reconstruction loss is used to train the autoencoder network:(1)ℒaθ=12n∑i=1nxi−gwfθxi22,where gw⋅ is the decoder that maps the representation to the output.In our algorithm, to extract essential features and preserve spatial locality of images, we adopt a fully convolutional autoencoder to realize the image feature extraction stream.


## 3.2. Deep Adaptive Image Clustering

DAC is a clustering algorithm that is realized by a convolutional neural network (CNN) and an adaptive training mechanism [20]. It employs some constraints on the classification output and generates a feature for image clustering.Let us assume thatxi and yj are two unlabeled images, rij denotes an unknown binary output received by the generated label, where rij=1 if xi and yj belong to the same cluster and otherwise, rij=0. In this case, fθ′xi⋅fθ′xj=zi⋅zj is the dot product of zi and zj, and it indicates the similarity, where fθ′⋅ is a classification network. Based on the similarity of the input image features, the binary labels are defined as follows:(2)rij=1,0,None,zi⋅zj≥uξ,zi⋅zj≤lξ,otherwise,where ξ is an adaptive parameter, u⋅ and l⋅ are two learnable thresholds.For network training, the objection function of DAC is defined as follows:(3)minθ,ξ∑i,jvijℒrrij,sxi,xj,θ+uξ−lξ,where s⋅ denote the estimated similarity of xi and xj with a classification network parameter θ, vij an indicator coefficient matrix to predict the training samples, which vij=1 means that the sample is selected to train the network, and vij=0 otherwise. vij is defined as follows:(4)vij=1,0,ifrij=1,otherwise,i,j=1,2,...,n.ℒrrij,sxi,xj,θ is the loss function and is defined as follows:(5)ℒrrij,sxi,xj,θ=−rijlogsxi,xj,θ1−rijlog1−sxi,xj,θ.


## 3.3. Network Architecture

The network architecture of the proposed clustering algorithm is shown in Figure1. There are two streams in our network, the autoencoder stream, and the DAC stream. The autoencoder stream is realized by several fully convolutional layers and the DAC stream is composed of the autoencoder’s encoder and several fully connected layers. Considering a dataset D:=xii=1n with n input samples that need to be clustered. The number of clustering K is a priori knowledge. Z:=zii=1n be the output of the encoder. Thus, the encoder can be defined as a nonlinear mapping fθ1:xi⟶zi and the decoder is gw:zi⟶x˜i where θ1 and w are the parameters of encoder and decoder, respectively. x˜i denotes the output of the decoder and the output of DAC can be represented as fθ1,θ2xi, which is a classification network and θ2 is the parameters of fully connected layers.Figure 1
The network architecture of the proposed clustering algorithm.


## 3.4. Loss Function

Since we aim to seek an encoder that makes the extracted feature more suitable for clustering. The reconstruction loss of the autoencoder is added to the initialization and training process of the DAC algorithm. On the one hand, the reconstructive loss function is used to assist the learning of image representation as it can learn the essential feature of input images and avoid the distortion of feature space in the training process of DAC. On the other hand, the loss of an autoencoder is only focused on image reconstruction, which loses the useful information needed for clustering. DAC loss can guide it to obtain a better representation suitable for clustering. Thus, we define the complete loss function as follows:(6)ℒ=αℒr+βℒc,where α is a balance coefficient. ℒr and ℒc are reconstruction loss and clustering loss, respectively. The final objective function is as follows:(7)minθ1,θ2,w,ξαℒr+βℒc+uξ+lξ,where the definitions of ℒr and ℒc are as follows:(8)ℒrθ1=12n∑i=1nxi−gwfθ1xi22,ℒcrij,sxi,xj,θ1,θ2=−rijlogsxi,xj,θ1,θ21−rijlog1−sxi,xj,θ1,θ2.


## 3.5. Network Training

In this section, we present the whole training process of our algorithm. To minimize the loss function proposed in (7), we first abandon the DAC stream and pretrain a convolutional autoencoder by using loss Lr. The trained encoder can provide initial parameters for the DAC algorithm. Thus, the DAC algorithm will select more accurate labeled image pairs in the initial stage. Then, we simultaneously train the autoencoder stream and the DAC stream by minimizing (7). The detailed algorithm is formalized as Algorithm 1.Algorithm 1:
Input: input imagesD:=xii=1n; number of clusters K; the threshold uξ=0.95−ξ and lξ=0.5+0.1⋅ξ; the batch size bs=128 and the learning rate η=0.005 and ρ=0.0001; the balance coefficient α=0.5,β=0.3.Output: reconstruction imagesD˜:=x˜ii=1n; clustering labels of input images.(1)
Pertain a fully convolutional autoencoder;(2)
fort in epochs do(3)
Extract the images featurefθ1xi in the batch Xbs;(4)
Compute the similarityrij and pseudolabel based on (2);(5)
Calculate the indicator coefficientvij based on (4);(6)
Updataθ1, θ2 and w by minimizing (7);(7)
end for(8)
Updataξ by minimizing (7).(9)
forxi in X do(10)
Ii:=fθ1,θ2xi and ci:=argmaxhIih;(11)
end forWhen the autoencoder is trained, it seems to get a clustering friendly representation by finetuning the autoencoder’s encoder in the DAC algorithm. The encoder is connected with a number of fully connected layers to form a classification network, and the labels are calculated by the algorithm proposed in [20]. However, we suppose that this kind of finetuning can distort the representation space, which may weaken the expressive power and thereby hurt clustering results. For this reason, in the process of training the DAC stream, the autoencoder also needs to be trained to maintain the DAC algorithm to obtain highly confidence labeled image pairs.


## 4. Experiments

In this section, we carry out a series of experiments to verify the effectiveness of our algorithm. All the experiments are carried out in Tensorflow and Keras environment running Ubuntu14.04, Inter(R) Core i7-4790 CPU 3.6 GHz and Titan X GPU 12 GB.


### 4.1. Datasets

In this part, fore challenging image datasets including Fashion-MNIST, Cifar-10, Cifar-100, and STL-10 datasets are selected to verify the effectiveness of our algorithm. We first briefly introduce datasets.


#### 4.1.1. Fashion-MNIST

Fashion-MNIST is a dataset of Zalando’s article images, which includes a training set of 60,000 examples and a test set of 10,000 examples [30]. In the Fashion-MNIST dataset, each example is a 28×28 grayscale image, associated with a label from 10 classes.


#### 4.1.2. Cifar-10 and Cifar-100

CIFAR-10 contains 50,000 training images and 10,000 test images from 10 classes [31]. Each image has a size of 32×32. Cifar-100 is similar to Cifar-10, except it has 10 times fewer images per class.


#### 4.1.3. STL-10

The STL-10 dataset is an image dataset used to develop unsupervised feature learning, deep learning, and self-supervised learning algorithms [32]. It is inspired by the CIFAR-10 dataset but with some modifications. The high-resolution dataset (96×96) will make it a challenging benchmark to develop more scalable unsupervised learning methods.In our experiments, the training set and validation set of each dataset are jointly utilized. In particular, the 20 superclasses of Cifar-100 dataset are considered in all the experiments. We summarize the detailed information of each dataset in Table1.Table 1
Information of the datasets.
Data setsSamplesClustersDimensionsFashion-MNIST700001028 × 28Cifar-10600001032 × 32 × 3Cifar-100600002032 × 32 × 3STL-10130001096 × 96 × 3


### 4.2. Evaluation Metrics

Three commonly clustering metrics including accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI) are adopted to evaluate the performance of clustering algorithms. These metrics reflect the cluster performance from different perspectives. ACC measures the best matching between the clustering labels and ground truth labels. NMI measures the similarity between pairs of clusters [33]. ARI establishes a baseline by using the expected similarity of all pairwise comparisons between clusters specified by a random model [34].


### 4.3. Competitors

We compare the performance to traditional clustering methods and deep clustering methods. Specifically, traditional clustering methods include K-means [35], Self-tuning Spectral Clustering (SSC) [36]. Deep clustering methods include Greedy Layer-Wise Training of Deep Networks (GLWTDN) [37], Consistent Inference of Latent Representations (CILR) [15], Gaussian Mixture Variational Autoencoders (GMVAE) [16], Categorical Generative Adversarial Networks (CatGAN) [27], Joint Unsupervised Learning (JULE) [17], Deep embedding clustering (DEC) [24], and DAC [20]. We summarize the clustering results of the mentioned methods on all datasets in Table 2. Next, we will briefly introduce the competitors.Table 2
Clustering performance and comparison, ACC (%) and NMI (%) and ARI (%) on all datasets.
Data setsFashion-MNISTCifar-10Cifar-100STL-10MetricNMIARIACCNMIARIACCNMIARIACCNMIARIACCK-means [35]18.5514.3722.898.714.8722.898.392.8012.9712.456.0819.20SSC [36]13.3514.2320.8110.288.5324.679.012.1813.609.784.7915.88GLWTDN [37]20.3118.7225.7423.9316.8931.3510.044.7616.4524.9616.1030.30CILR [15]25.6820.7130.1011.056.1225.839.434.5315.5413.986.5224.52GMVAE [16]25.6816.0730.1028.4718.3733.6413.664.9215.8423.6515.7231.77CatGAN [27]26.3117.1132.2426.4617.5731.5212.04.5315.1021.0013.9029.84JULE [17]43.7933.9947.7819.2313.7727.1510.263.2713.6718.1516.4327.69DEC [24]60.5458.4365.2525.6816.0730.1013.584.9518.5227.6018.6135.90DAC [20]65.3862.4170.5243.7933.9947.7821.2411.0025.5240.3328.6448.18Ours70.6669.4375.2146.3942.2552.2525.4616.1030.4744.3133.0852.12Most of the results are excerpted from [20]. The best and second-best results are marked in bold and underlined, respectively.


#### 4.3.1. Traditional Image Clustering Methods

K-means++ and SSC: these methods first use Bag of Wording (BOW) to encode the images, and then, the image features are clustered to achieve image clustering.


#### 4.3.2. Deep Clustering Methods

GLWTDN: it first trains an AE model to extract image features and then uses the K-means algorithm to cluster the image features [37].CILR: it first adopts consistent inference of latent representations (CILR) to generate latent labeled data points of the inputs. Then, CILR is derived to pretrain DNNs by minimizing the distance between latent labeled data points to realize image clustering [15].GMVAE: it uses the Gaussian mixture model as a prior distribution to improve the traditional variational autoencoder. It uses the improved latent vector as image representation and then clusters representation to realize image clustering [16].CatGAN: it uses general Generative Network Adversarial (GAN) and entropy as loss function to realize data clustering [27].JULE: it proposes a recurrent framework for joint unsupervised learning of deep representations and image clustering [17].DEC: it first learns image representations from an AE. Then, clusters are obtained by utilizing a typical K-means algorithm [24].DAC: it formulates image clustering as a binary pairwise classification problem and identifies these pairs of images which should belong to the same cluster [20].


### 4.4. Experimental Settings

Following the setting in DAC, we set the initial thresholds which construct highly confident pseudolabel touξ=0.95−ξ and lξ=0.5+0.1ξ, respectively. The initialization of the parameter ξ is set to 0. Considering the convergence of our algorithm, we set the learning rate of ξ to η=0.005. The balance coefficient is 0.5. For training, we adopt the well-known Adam optimizer with an initial learning rate 0.0001. In addition, the batch size is set to bs=128 in all the experiments. The detailed network architecture used in each dataset is shown in Table 3.Table 3
Network architecture in experiments.
Fashion-MNISTEncoder3 × 3 conv., stride 2, 8 BN Relu3 × 3 conv., stride 2, 16 BN Relu3 × 3 conv., stride 2, 32 BN Relu3 × 3 conv., stride 2, 64 BN ReluDecoder3 × 3 deconv., stride 2, 32 BN ReluClustering(256,64) fc., BN Relu(64,10) fc., BN ReluSoftmax3 × 3 deconv., stride 2, 16 BN Relu3 × 3 deconv., stride 2, 8 BN Relu3 × 3 deconv., stride 2, 1Cifar-10 and Cifar-100 datasetEncoder3 × 3 conv., stride 2, 8 BN Relu3 × 3 conv., stride 2, 16 BN Relu3 × 3 conv., stride 2, 32 BN Relu3 × 3 conv., stride 2, 64 BN ReluDecoder3 × 3 deconv., stride 2, 32 BN ReluClustering(256,64) fc., BN Relu(64,k) fc., BN ReluSoftmax3 × 3 deconv., stride 2, 16 BN Relu3 × 3 deconv., stride 2, 8 BN Relu3 × 3 deconv., stride 2, 3STL-10 datasetEncoder5 × 5 conv., stride 4, 16 BN Relu3 × 3 conv., stride 2, 32 BN Relu3 × 3 conv., stride 2, 64 BN Relu3 × 3 conv., stride 2, 128 BN Relu3 × 3 conv., stride 1, 256 BN ReluDecoder3 × 3 deconv., stride 2, 128 BN ReluClusteringFC (256,64) fc., BN Relu(64,10) fc., BN ReluSoftmax3 × 3 deconv., stride 2, 64 BN Relu3 × 3 deconv., stride 2, 32 BN Relu3 × 3 deconv., stride 2, 16 BN Relu3 × 3 deconv., stride 2, 3


### 4.5. Results

#### 4.5.1. Clustering Performance Comparison

In this part, we first compare our method with many state-of-the-art methods including K-means [35], Self-Tuning Spectral Clustering (SPC) [36], Greedy Layer-Wise Training of Deep Networks (GLWTDN) [37], Consistent Inference of Latent Representations (CILR) [15], Gaussian Mixture Variational Autoencoders (GMVAE) [16], Categorical Generative Adversarial Networks (CatGAN) [27], JULE-SF [17], JULE-RC [17], Deep embedding clustering (DEC) [24], and DAC [20]. We summarize the clustering results of the mentioned methods on all datasets in Table 2.As shown in Table2, for each dataset, the performance of deep clustering methods is superior to that of traditional clustering algorithms. Autoencoder based method such as AE outperforms traditional algorithm K-means with a large margin, which justifies the fascinating potential of autoencoder in clustering task. Furthermore, note that the proposed method outperforms the other algorithms on all datasets. In addition, the clustering accuracy of our algorithm outperforms all competitive baselines, with significant margins of 4.79%, 4.47%, 4.95%, and 3.94% in the case of Fashion-MNIST, Cifar-10, Cifar-100, and STL-10, respectively. These results verify the effectiveness of our method in image clustering tasks.Figure2 shows the confusion matrixes of the clustering results for Cifar-10 and STL-10 datasets. The values along the diagonal represent the percentage of samples correctly classified into the corresponding categories. We can find that all clustering accuracy is average and stable for these two datasets. This proves that our method does not aggregate samples into a few categories and can effectively avoid the degenerate solution problem.Figure 2
Confusion matrixes of Cifar-10 and STL-10 datasets. (a) Cifar-10. (b) STL-10.
(a)(b)


#### 4.5.2. Visualization

In this part, we use two methods to visualize the clustering results of our algorithm. The first visualization experiment is conducted on the Fashion-MNIST dataset. We randomly sampled 10000 samples of the representationziand mapped them to a 2-dimension vector by using t-SNE [30]. The experiment results are shown in Figure 3. In Figures 3(a)–3(f), different colors indicate different clusters and the corresponding clustering accuracies are reported as follows. The experimental results show that the proposed algorithm can effectively improve the separability of data, which is helpful to improve the clustering accuracy of the image.Figure 3
t-SNE visualization of image representation for Fashion-MNIST dataset. (a) 5.21%. (b) 33.43%. (c) 55.41%. (d) 60.35%. (e) 74.56%. (f) 75.21%.
(a)(b)(c)(d)(e)(f)In the second visualization experiment, we qualitatively analyze the cluster results by the proposed Cifar-10 dataset. For each category, we randomly select an image as the original image at the first stage. Then, we pick up several samples, which are the smallest Euclidean distance between original images from the same cluster. Finally, we pick the samples which are closest to the original image in the incorrect cluster. All the picked images are shown in Figure4, and we mark the incorrect samples with red boxes. Form the visualization results, we can find that the successful cases not only depend on texture information but also contain some semantic information of categories. The failure cases also contain a lot of texture information similar to the source images. It implies that our method not only captures image appearance information but also captures some abstract image information for image clustering. This is the reason why the proposed method can precisely discover cluster assignments.Figure 4
Qualitative analysis of clustering results.


#### 4.5.3. On the Effect of Number of Clusters

In this part, we study the effect of the number of clusters on our algorithm and compare the results with the DAC algorithm. For each dataset, we conduct 6 experiments on different training sets. For Cifar-10 and STL-10 dataset, the number of training sets varies in the range of5,10 at equal intervals. For Cifar-100 dataset, the number of training samples varies in the range of 10,20 with an interval of 2. We report the variation curves of clustering accuracy with the number of clusters in Figure 5. The detailed numerical results are shown in Table 4.Figure 5
Accuracy with a different number of clusters. (a) Cifar-10. (b) Cifar-100. (c) STL-10.
(a)(b)(c)Table 4
Clustering performance, ACC (%) and NMI (%) and ARI (%), with different number of clusters.
MethodDACOursClustersNMIARIACCNMIARIACCCifar-10544.8935.5849.3548.1242.5352.33644.3535.0348.2747.8242.0251.12744.2134.6948.0547.3941.7951.02844.0234.5247.9346.6441.5250.51943.9234.2147.8846.5241.2750.321043.7933.9947.7846.3940.2550.25Cifar-1001022.1611.8930.2526.6721.7235.331222.0511.6829.5826.0321.4735.101421.9711.5628.3725.8121.3234.271621.8811.3527.2525.7221.0433.431821.6511.2126.5425.6720.5331.252021.2411.0025.5225.4620.1030.47STL-10541.4329.6850.3045.0134.8253.35641.2229.3550.1244.9634.5152.98741.0529.0649.2644.7234.0352.17840.7628.9648.3744.5633.8751.55940.5528.7248.2744.3333.5351.231040.3328.6448.1844.3133.0851.12As shown in Figure5 and Table 4, with the increase in the number of clusters, the accuracy of clustering decreases gradually. For all datasets, the clustering accuracy of our method is always higher than that of the DAC algorithm in the different number of clusters. In addition, the other two metrics results also show the superiority of the proposed algorithm. This is because autoencoder can reduce the randomness of searching label pair and improve clustering performance of DAC algorithm. The experimental results also show the stability of the algorithm.


#### 4.5.4. On the Effect of the Parameterα and β

In this experiment, we mainly study the effect of the parameterα and β. The range of parameters α,β is selected by the grid search of the region 0.1,0.9×0.1,0.9 in a step size of 0.1. In Figure 6, we report the clustering accuracies with different α,β. From the results, we can find that when α and β tend to be close, the clustering accuracy is the highest. This is mainly because the autoencoder can guarantee the local structure of image representation and prevent the distortion of feature space. It also means that autoencoder can promote the clustering performance of DAC, which explains the reason why our algorithm is effective.Figure 6
The clustering accuracies with differentα,β pairs. (a) Cifar-10. (b) Cifar-100. (c) STL-10.
(a)(b)(c)


## 4.1. Datasets

In this part, fore challenging image datasets including Fashion-MNIST, Cifar-10, Cifar-100, and STL-10 datasets are selected to verify the effectiveness of our algorithm. We first briefly introduce datasets.


### 4.1.1. Fashion-MNIST

Fashion-MNIST is a dataset of Zalando’s article images, which includes a training set of 60,000 examples and a test set of 10,000 examples [30]. In the Fashion-MNIST dataset, each example is a 28×28 grayscale image, associated with a label from 10 classes.


### 4.1.2. Cifar-10 and Cifar-100

CIFAR-10 contains 50,000 training images and 10,000 test images from 10 classes [31]. Each image has a size of 32×32. Cifar-100 is similar to Cifar-10, except it has 10 times fewer images per class.


### 4.1.3. STL-10

The STL-10 dataset is an image dataset used to develop unsupervised feature learning, deep learning, and self-supervised learning algorithms [32]. It is inspired by the CIFAR-10 dataset but with some modifications. The high-resolution dataset (96×96) will make it a challenging benchmark to develop more scalable unsupervised learning methods.In our experiments, the training set and validation set of each dataset are jointly utilized. In particular, the 20 superclasses of Cifar-100 dataset are considered in all the experiments. We summarize the detailed information of each dataset in Table1.Table 1
Information of the datasets.
Data setsSamplesClustersDimensionsFashion-MNIST700001028 × 28Cifar-10600001032 × 32 × 3Cifar-100600002032 × 32 × 3STL-10130001096 × 96 × 3


## 4.1.1. Fashion-MNIST

Fashion-MNIST is a dataset of Zalando’s article images, which includes a training set of 60,000 examples and a test set of 10,000 examples [30]. In the Fashion-MNIST dataset, each example is a 28×28 grayscale image, associated with a label from 10 classes.


## 4.1.2. Cifar-10 and Cifar-100

CIFAR-10 contains 50,000 training images and 10,000 test images from 10 classes [31]. Each image has a size of 32×32. Cifar-100 is similar to Cifar-10, except it has 10 times fewer images per class.


## 4.1.3. STL-10

The STL-10 dataset is an image dataset used to develop unsupervised feature learning, deep learning, and self-supervised learning algorithms [32]. It is inspired by the CIFAR-10 dataset but with some modifications. The high-resolution dataset (96×96) will make it a challenging benchmark to develop more scalable unsupervised learning methods.In our experiments, the training set and validation set of each dataset are jointly utilized. In particular, the 20 superclasses of Cifar-100 dataset are considered in all the experiments. We summarize the detailed information of each dataset in Table1.Table 1
Information of the datasets.
Data setsSamplesClustersDimensionsFashion-MNIST700001028 × 28Cifar-10600001032 × 32 × 3Cifar-100600002032 × 32 × 3STL-10130001096 × 96 × 3


## 4.2. Evaluation Metrics

Three commonly clustering metrics including accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI) are adopted to evaluate the performance of clustering algorithms. These metrics reflect the cluster performance from different perspectives. ACC measures the best matching between the clustering labels and ground truth labels. NMI measures the similarity between pairs of clusters [33]. ARI establishes a baseline by using the expected similarity of all pairwise comparisons between clusters specified by a random model [34].


## 4.3. Competitors

We compare the performance to traditional clustering methods and deep clustering methods. Specifically, traditional clustering methods include K-means [35], Self-tuning Spectral Clustering (SSC) [36]. Deep clustering methods include Greedy Layer-Wise Training of Deep Networks (GLWTDN) [37], Consistent Inference of Latent Representations (CILR) [15], Gaussian Mixture Variational Autoencoders (GMVAE) [16], Categorical Generative Adversarial Networks (CatGAN) [27], Joint Unsupervised Learning (JULE) [17], Deep embedding clustering (DEC) [24], and DAC [20]. We summarize the clustering results of the mentioned methods on all datasets in Table 2. Next, we will briefly introduce the competitors.Table 2
Clustering performance and comparison, ACC (%) and NMI (%) and ARI (%) on all datasets.
Data setsFashion-MNISTCifar-10Cifar-100STL-10MetricNMIARIACCNMIARIACCNMIARIACCNMIARIACCK-means [35]18.5514.3722.898.714.8722.898.392.8012.9712.456.0819.20SSC [36]13.3514.2320.8110.288.5324.679.012.1813.609.784.7915.88GLWTDN [37]20.3118.7225.7423.9316.8931.3510.044.7616.4524.9616.1030.30CILR [15]25.6820.7130.1011.056.1225.839.434.5315.5413.986.5224.52GMVAE [16]25.6816.0730.1028.4718.3733.6413.664.9215.8423.6515.7231.77CatGAN [27]26.3117.1132.2426.4617.5731.5212.04.5315.1021.0013.9029.84JULE [17]43.7933.9947.7819.2313.7727.1510.263.2713.6718.1516.4327.69DEC [24]60.5458.4365.2525.6816.0730.1013.584.9518.5227.6018.6135.90DAC [20]65.3862.4170.5243.7933.9947.7821.2411.0025.5240.3328.6448.18Ours70.6669.4375.2146.3942.2552.2525.4616.1030.4744.3133.0852.12Most of the results are excerpted from [20]. The best and second-best results are marked in bold and underlined, respectively.


### 4.3.1. Traditional Image Clustering Methods

K-means++ and SSC: these methods first use Bag of Wording (BOW) to encode the images, and then, the image features are clustered to achieve image clustering.


### 4.3.2. Deep Clustering Methods

GLWTDN: it first trains an AE model to extract image features and then uses the K-means algorithm to cluster the image features [37].CILR: it first adopts consistent inference of latent representations (CILR) to generate latent labeled data points of the inputs. Then, CILR is derived to pretrain DNNs by minimizing the distance between latent labeled data points to realize image clustering [15].GMVAE: it uses the Gaussian mixture model as a prior distribution to improve the traditional variational autoencoder. It uses the improved latent vector as image representation and then clusters representation to realize image clustering [16].CatGAN: it uses general Generative Network Adversarial (GAN) and entropy as loss function to realize data clustering [27].JULE: it proposes a recurrent framework for joint unsupervised learning of deep representations and image clustering [17].DEC: it first learns image representations from an AE. Then, clusters are obtained by utilizing a typical K-means algorithm [24].DAC: it formulates image clustering as a binary pairwise classification problem and identifies these pairs of images which should belong to the same cluster [20].


## 4.3.1. Traditional Image Clustering Methods

K-means++ and SSC: these methods first use Bag of Wording (BOW) to encode the images, and then, the image features are clustered to achieve image clustering.


## 4.3.2. Deep Clustering Methods

GLWTDN: it first trains an AE model to extract image features and then uses the K-means algorithm to cluster the image features [37].CILR: it first adopts consistent inference of latent representations (CILR) to generate latent labeled data points of the inputs. Then, CILR is derived to pretrain DNNs by minimizing the distance between latent labeled data points to realize image clustering [15].GMVAE: it uses the Gaussian mixture model as a prior distribution to improve the traditional variational autoencoder. It uses the improved latent vector as image representation and then clusters representation to realize image clustering [16].CatGAN: it uses general Generative Network Adversarial (GAN) and entropy as loss function to realize data clustering [27].JULE: it proposes a recurrent framework for joint unsupervised learning of deep representations and image clustering [17].DEC: it first learns image representations from an AE. Then, clusters are obtained by utilizing a typical K-means algorithm [24].DAC: it formulates image clustering as a binary pairwise classification problem and identifies these pairs of images which should belong to the same cluster [20].


## 4.4. Experimental Settings

Following the setting in DAC, we set the initial thresholds which construct highly confident pseudolabel touξ=0.95−ξ and lξ=0.5+0.1ξ, respectively. The initialization of the parameter ξ is set to 0. Considering the convergence of our algorithm, we set the learning rate of ξ to η=0.005. The balance coefficient is 0.5. For training, we adopt the well-known Adam optimizer with an initial learning rate 0.0001. In addition, the batch size is set to bs=128 in all the experiments. The detailed network architecture used in each dataset is shown in Table 3.Table 3
Network architecture in experiments.
Fashion-MNISTEncoder3 × 3 conv., stride 2, 8 BN Relu3 × 3 conv., stride 2, 16 BN Relu3 × 3 conv., stride 2, 32 BN Relu3 × 3 conv., stride 2, 64 BN ReluDecoder3 × 3 deconv., stride 2, 32 BN ReluClustering(256,64) fc., BN Relu(64,10) fc., BN ReluSoftmax3 × 3 deconv., stride 2, 16 BN Relu3 × 3 deconv., stride 2, 8 BN Relu3 × 3 deconv., stride 2, 1Cifar-10 and Cifar-100 datasetEncoder3 × 3 conv., stride 2, 8 BN Relu3 × 3 conv., stride 2, 16 BN Relu3 × 3 conv., stride 2, 32 BN Relu3 × 3 conv., stride 2, 64 BN ReluDecoder3 × 3 deconv., stride 2, 32 BN ReluClustering(256,64) fc., BN Relu(64,k) fc., BN ReluSoftmax3 × 3 deconv., stride 2, 16 BN Relu3 × 3 deconv., stride 2, 8 BN Relu3 × 3 deconv., stride 2, 3STL-10 datasetEncoder5 × 5 conv., stride 4, 16 BN Relu3 × 3 conv., stride 2, 32 BN Relu3 × 3 conv., stride 2, 64 BN Relu3 × 3 conv., stride 2, 128 BN Relu3 × 3 conv., stride 1, 256 BN ReluDecoder3 × 3 deconv., stride 2, 128 BN ReluClusteringFC (256,64) fc., BN Relu(64,10) fc., BN ReluSoftmax3 × 3 deconv., stride 2, 64 BN Relu3 × 3 deconv., stride 2, 32 BN Relu3 × 3 deconv., stride 2, 16 BN Relu3 × 3 deconv., stride 2, 3


## 4.5. Results

### 4.5.1. Clustering Performance Comparison

In this part, we first compare our method with many state-of-the-art methods including K-means [35], Self-Tuning Spectral Clustering (SPC) [36], Greedy Layer-Wise Training of Deep Networks (GLWTDN) [37], Consistent Inference of Latent Representations (CILR) [15], Gaussian Mixture Variational Autoencoders (GMVAE) [16], Categorical Generative Adversarial Networks (CatGAN) [27], JULE-SF [17], JULE-RC [17], Deep embedding clustering (DEC) [24], and DAC [20]. We summarize the clustering results of the mentioned methods on all datasets in Table 2.As shown in Table2, for each dataset, the performance of deep clustering methods is superior to that of traditional clustering algorithms. Autoencoder based method such as AE outperforms traditional algorithm K-means with a large margin, which justifies the fascinating potential of autoencoder in clustering task. Furthermore, note that the proposed method outperforms the other algorithms on all datasets. In addition, the clustering accuracy of our algorithm outperforms all competitive baselines, with significant margins of 4.79%, 4.47%, 4.95%, and 3.94% in the case of Fashion-MNIST, Cifar-10, Cifar-100, and STL-10, respectively. These results verify the effectiveness of our method in image clustering tasks.Figure2 shows the confusion matrixes of the clustering results for Cifar-10 and STL-10 datasets. The values along the diagonal represent the percentage of samples correctly classified into the corresponding categories. We can find that all clustering accuracy is average and stable for these two datasets. This proves that our method does not aggregate samples into a few categories and can effectively avoid the degenerate solution problem.Figure 2
Confusion matrixes of Cifar-10 and STL-10 datasets. (a) Cifar-10. (b) STL-10.
(a)(b)


### 4.5.2. Visualization

In this part, we use two methods to visualize the clustering results of our algorithm. The first visualization experiment is conducted on the Fashion-MNIST dataset. We randomly sampled 10000 samples of the representationziand mapped them to a 2-dimension vector by using t-SNE [30]. The experiment results are shown in Figure 3. In Figures 3(a)–3(f), different colors indicate different clusters and the corresponding clustering accuracies are reported as follows. The experimental results show that the proposed algorithm can effectively improve the separability of data, which is helpful to improve the clustering accuracy of the image.Figure 3
t-SNE visualization of image representation for Fashion-MNIST dataset. (a) 5.21%. (b) 33.43%. (c) 55.41%. (d) 60.35%. (e) 74.56%. (f) 75.21%.
(a)(b)(c)(d)(e)(f)In the second visualization experiment, we qualitatively analyze the cluster results by the proposed Cifar-10 dataset. For each category, we randomly select an image as the original image at the first stage. Then, we pick up several samples, which are the smallest Euclidean distance between original images from the same cluster. Finally, we pick the samples which are closest to the original image in the incorrect cluster. All the picked images are shown in Figure4, and we mark the incorrect samples with red boxes. Form the visualization results, we can find that the successful cases not only depend on texture information but also contain some semantic information of categories. The failure cases also contain a lot of texture information similar to the source images. It implies that our method not only captures image appearance information but also captures some abstract image information for image clustering. This is the reason why the proposed method can precisely discover cluster assignments.Figure 4
Qualitative analysis of clustering results.


### 4.5.3. On the Effect of Number of Clusters

In this part, we study the effect of the number of clusters on our algorithm and compare the results with the DAC algorithm. For each dataset, we conduct 6 experiments on different training sets. For Cifar-10 and STL-10 dataset, the number of training sets varies in the range of5,10 at equal intervals. For Cifar-100 dataset, the number of training samples varies in the range of 10,20 with an interval of 2. We report the variation curves of clustering accuracy with the number of clusters in Figure 5. The detailed numerical results are shown in Table 4.Figure 5
Accuracy with a different number of clusters. (a) Cifar-10. (b) Cifar-100. (c) STL-10.
(a)(b)(c)Table 4
Clustering performance, ACC (%) and NMI (%) and ARI (%), with different number of clusters.
MethodDACOursClustersNMIARIACCNMIARIACCCifar-10544.8935.5849.3548.1242.5352.33644.3535.0348.2747.8242.0251.12744.2134.6948.0547.3941.7951.02844.0234.5247.9346.6441.5250.51943.9234.2147.8846.5241.2750.321043.7933.9947.7846.3940.2550.25Cifar-1001022.1611.8930.2526.6721.7235.331222.0511.6829.5826.0321.4735.101421.9711.5628.3725.8121.3234.271621.8811.3527.2525.7221.0433.431821.6511.2126.5425.6720.5331.252021.2411.0025.5225.4620.1030.47STL-10541.4329.6850.3045.0134.8253.35641.2229.3550.1244.9634.5152.98741.0529.0649.2644.7234.0352.17840.7628.9648.3744.5633.8751.55940.5528.7248.2744.3333.5351.231040.3328.6448.1844.3133.0851.12As shown in Figure5 and Table 4, with the increase in the number of clusters, the accuracy of clustering decreases gradually. For all datasets, the clustering accuracy of our method is always higher than that of the DAC algorithm in the different number of clusters. In addition, the other two metrics results also show the superiority of the proposed algorithm. This is because autoencoder can reduce the randomness of searching label pair and improve clustering performance of DAC algorithm. The experimental results also show the stability of the algorithm.


### 4.5.4. On the Effect of the Parameterα and β

In this experiment, we mainly study the effect of the parameterα and β. The range of parameters α,β is selected by the grid search of the region 0.1,0.9×0.1,0.9 in a step size of 0.1. In Figure 6, we report the clustering accuracies with different α,β. From the results, we can find that when α and β tend to be close, the clustering accuracy is the highest. This is mainly because the autoencoder can guarantee the local structure of image representation and prevent the distortion of feature space. It also means that autoencoder can promote the clustering performance of DAC, which explains the reason why our algorithm is effective.Figure 6
The clustering accuracies with differentα,β pairs. (a) Cifar-10. (b) Cifar-100. (c) STL-10.
(a)(b)(c)


## 4.5.1. Clustering Performance Comparison

In this part, we first compare our method with many state-of-the-art methods including K-means [35], Self-Tuning Spectral Clustering (SPC) [36], Greedy Layer-Wise Training of Deep Networks (GLWTDN) [37], Consistent Inference of Latent Representations (CILR) [15], Gaussian Mixture Variational Autoencoders (GMVAE) [16], Categorical Generative Adversarial Networks (CatGAN) [27], JULE-SF [17], JULE-RC [17], Deep embedding clustering (DEC) [24], and DAC [20]. We summarize the clustering results of the mentioned methods on all datasets in Table 2.As shown in Table2, for each dataset, the performance of deep clustering methods is superior to that of traditional clustering algorithms. Autoencoder based method such as AE outperforms traditional algorithm K-means with a large margin, which justifies the fascinating potential of autoencoder in clustering task. Furthermore, note that the proposed method outperforms the other algorithms on all datasets. In addition, the clustering accuracy of our algorithm outperforms all competitive baselines, with significant margins of 4.79%, 4.47%, 4.95%, and 3.94% in the case of Fashion-MNIST, Cifar-10, Cifar-100, and STL-10, respectively. These results verify the effectiveness of our method in image clustering tasks.Figure2 shows the confusion matrixes of the clustering results for Cifar-10 and STL-10 datasets. The values along the diagonal represent the percentage of samples correctly classified into the corresponding categories. We can find that all clustering accuracy is average and stable for these two datasets. This proves that our method does not aggregate samples into a few categories and can effectively avoid the degenerate solution problem.Figure 2
Confusion matrixes of Cifar-10 and STL-10 datasets. (a) Cifar-10. (b) STL-10.
(a)(b)


## 4.5.2. Visualization

In this part, we use two methods to visualize the clustering results of our algorithm. The first visualization experiment is conducted on the Fashion-MNIST dataset. We randomly sampled 10000 samples of the representationziand mapped them to a 2-dimension vector by using t-SNE [30]. The experiment results are shown in Figure 3. In Figures 3(a)–3(f), different colors indicate different clusters and the corresponding clustering accuracies are reported as follows. The experimental results show that the proposed algorithm can effectively improve the separability of data, which is helpful to improve the clustering accuracy of the image.Figure 3
t-SNE visualization of image representation for Fashion-MNIST dataset. (a) 5.21%. (b) 33.43%. (c) 55.41%. (d) 60.35%. (e) 74.56%. (f) 75.21%.
(a)(b)(c)(d)(e)(f)In the second visualization experiment, we qualitatively analyze the cluster results by the proposed Cifar-10 dataset. For each category, we randomly select an image as the original image at the first stage. Then, we pick up several samples, which are the smallest Euclidean distance between original images from the same cluster. Finally, we pick the samples which are closest to the original image in the incorrect cluster. All the picked images are shown in Figure4, and we mark the incorrect samples with red boxes. Form the visualization results, we can find that the successful cases not only depend on texture information but also contain some semantic information of categories. The failure cases also contain a lot of texture information similar to the source images. It implies that our method not only captures image appearance information but also captures some abstract image information for image clustering. This is the reason why the proposed method can precisely discover cluster assignments.Figure 4
Qualitative analysis of clustering results.


## 4.5.3. On the Effect of Number of Clusters

In this part, we study the effect of the number of clusters on our algorithm and compare the results with the DAC algorithm. For each dataset, we conduct 6 experiments on different training sets. For Cifar-10 and STL-10 dataset, the number of training sets varies in the range of5,10 at equal intervals. For Cifar-100 dataset, the number of training samples varies in the range of 10,20 with an interval of 2. We report the variation curves of clustering accuracy with the number of clusters in Figure 5. The detailed numerical results are shown in Table 4.Figure 5
Accuracy with a different number of clusters. (a) Cifar-10. (b) Cifar-100. (c) STL-10.
(a)(b)(c)Table 4
Clustering performance, ACC (%) and NMI (%) and ARI (%), with different number of clusters.
MethodDACOursClustersNMIARIACCNMIARIACCCifar-10544.8935.5849.3548.1242.5352.33644.3535.0348.2747.8242.0251.12744.2134.6948.0547.3941.7951.02844.0234.5247.9346.6441.5250.51943.9234.2147.8846.5241.2750.321043.7933.9947.7846.3940.2550.25Cifar-1001022.1611.8930.2526.6721.7235.331222.0511.6829.5826.0321.4735.101421.9711.5628.3725.8121.3234.271621.8811.3527.2525.7221.0433.431821.6511.2126.5425.6720.5331.252021.2411.0025.5225.4620.1030.47STL-10541.4329.6850.3045.0134.8253.35641.2229.3550.1244.9634.5152.98741.0529.0649.2644.7234.0352.17840.7628.9648.3744.5633.8751.55940.5528.7248.2744.3333.5351.231040.3328.6448.1844.3133.0851.12As shown in Figure5 and Table 4, with the increase in the number of clusters, the accuracy of clustering decreases gradually. For all datasets, the clustering accuracy of our method is always higher than that of the DAC algorithm in the different number of clusters. In addition, the other two metrics results also show the superiority of the proposed algorithm. This is because autoencoder can reduce the randomness of searching label pair and improve clustering performance of DAC algorithm. The experimental results also show the stability of the algorithm.


## 4.5.4. On the Effect of the Parameterα and β

In this experiment, we mainly study the effect of the parameterα and β. The range of parameters α,β is selected by the grid search of the region 0.1,0.9×0.1,0.9 in a step size of 0.1. In Figure 6, we report the clustering accuracies with different α,β. From the results, we can find that when α and β tend to be close, the clustering accuracy is the highest. This is mainly because the autoencoder can guarantee the local structure of image representation and prevent the distortion of feature space. It also means that autoencoder can promote the clustering performance of DAC, which explains the reason why our algorithm is effective.Figure 6
The clustering accuracies with differentα,β pairs. (a) Cifar-10. (b) Cifar-100. (c) STL-10.
(a)(b)(c)


## 5. Conclusion

In this paper, we present a novel representation learning method and use it to solve the image clustering problem. To generate more informative representations for clustering, we borrow the DAC algorithm and incorporate it to train a fully convolutional autoencoder. The proposed algorithm was evaluated on unsupervised clustering tasks using popular datasets, achieving competitive results compared to the current state of the art. Furthermore, we may improve the proposed algorithm by applying some deep feature extraction models, e.g., Variational AutoEncoder (VAE) and Generative Adversarial Networks (GANs). To improve our method, it is an interesting direction to learn the distribution of image data instead of reconstructing the image. We will see this for future work.



---

*Source: 3742536-2021-02-25.xml*