# Soft Semi-Supervised Deep Learning-Based Clustering
## 2 Related Works  
Research work on deep clustering includes deep embedded clustering and semi-supervised deep embedded clustering (SS-DEC) approaches. For deep embedded clustering, Deep Neural Networks (DNNs) have been used for dimensionality reduction to learn a clustering-oriented representation that favors the clustering tasks. In fact, different deep clustering frameworks were created on top of various DNN architectures. Namely, autoencoders (AEs), Deep Belief Networks (DBNs) [23], Convolutional Neural Networks (CNNs) [24; 25], and Generative Adversarial Networks (GANs) [26] have been introduced and used in various deep clustering applications. In particular, autoencoders have been widely adapted to address challenges relevant to the deep clustering architectures [27; 21; 28; 29]. A recent survey [30] distinguishes deep clustering methods in terms of methodology, prior knowledge, and architecture. Specifically, the authors listed semi-supervised deep clustering as one of the main four categories of deep clustering methods. For AE-based deep clustering approaches, the encoder layers of the trained autoencoder are the ones leveraged for feature transformation into a lower-dimensional space, which serves as the input for the clustering algorithm. Recently, an AE-based deep clustering architecture, deep embedded clustering (DEC) [21], was proposed, which was then followed by a number of variants [19; 27; 28; 29; 31; 32; 33; 34; 35; 36] that used DEC as the basis for their framework. From then, deep clustering has become a growing research field, with DEC [21] being the experimental benchmark for many deep clustering approaches [29; 34]. For DEC [21], the DNN architecture consists mainly of an autoencoder that is used to automatically learn the feature representations via nonlinear embedding into a lower-dimensional feature space. After the autoencoder is trained and its parameters are initialized, it is finetuned further to minimize the reconstruction loss. Only the encoder layers are kept and used as the initial mapping function between the data space and the embedded feature space. To initialize the cluster centers, the original data are passed through the initialized encoder to obtain the embedded data points and perform standard k-means clustering in the embedded feature space to obtain \(k\) initial centroids and memberships. After that, DEC updates its parameters by iterating between computing an auxiliary target distribution and minimizing the following objective function in form of the Kullback-Leibler (KL) divergence from the computed soft assignments to the computed auxiliary target distribution:  
\[L=KL(P||Q)=\sum_{i}\sum_{j}p_{ij}log\frac{p_{ij}}{q_{ij}} \tag{1}\]  
where \(Q\) is the soft assignments of the data points and \(P\) is the computed auxiliary distribution. In the above equation, \(q_{ij}\) is a centroid-based probability distribution (i.e., soft assignment) that represents the probability of assigning sample \(i\) to cluster \(j\), and it is computed using the Student's t-distribution. Moreover, \(p_{ij}\) in Equation (1) is the auxiliary target distribution that \(q_{ij}\) is matched to by minimizing a KL divergence metric between the two distributions. Further elaboration on the computation of \(q_{ij}\) and \(p_{ij}\) can be found in Equations (3) and (4) below. This objective function in Equation (1) is minimized using a Stochastic Gradient Descent (SGD) to learn cluster centers and DNN parameters from the embedded space \(Z\). Lastly, a cluster assignment hardening loss is applied to obtain the soft assignment probabilities of the embedded points. DEC reported an 84.3% clustering accuracy on the MNIST images dataset and 75.63% on the REUTERS text dataset, outperforming several state-of-the-art spectral clustering-based algorithms. Motivated by the need to address certain limitations, other deep clustering works were proposed to incorporate improvements to the DEC [21] framework.