# Improved Selective Deep-Learning-Based Clustering Ensemble
## 3 Improved Selective Deep Clustering Ensemble
### Deep Clustering Generation  
The framework of deep autoencoder clustering is shown in Figure 2. Deep autoencoder clustering consists of a fully connected autoencoder and a clustering layer, which is connected to the autoencoder embedding layer. In Figure 2, \(x_{i}\) represents the \(i\)-th sample of the original input data and \(\hat{x}_{i}\) indicates the \(i\)-th reconstructed sample. \(z_{i}\) represents the deep low-dimensional embedding representation of the \(i\)-th sample. Given the original input data \(X=\{x_{i}\}_{i=1}^{n}\), deep autoencoder clustering firstly learns a mapping from the original data \(x_{i}\) to a lower-dimensional embedding representation \(z_{i}\). Then, it maps each embedding point \(z_{i}\) of the input sample to a soft cluster label \(q\) using a clustering layer. Therefore, the loss of deep autoencoder clustering contains two parts: non-clustering loss (or reconstruction loss) and clustering loss. The non-clustering loss \(L_{r}\) of deep autoencoder clustering consists of the fully connected autoencoder reconstruction loss, computed as:  
\[L_{r}=\|x-\hat{x}\|^{2} \tag{1}\]  
Then, Student's t-distribution is used to measure the similarity between the deep representation \(z_{i}\) and the clustering centroid \(\mu_{i}\). The clustering centroid \(\mu_{i}\) is generated by k-means and used as training weights. The clustering layer maps each embedding point \(z_{i}\) to a soft cluster assignment \(Q=\{q_{ij}\}\) as follows:  
Figure 1: Illustration of the proposed ISDCE framework. Figure 2: The structure of deep autoencoder clustering. \[q_{ij}=\frac{\left(1+\left\|z_{i}-\mu_{j}\right\|^{2}/\alpha\right)^{-\frac{8+1}{ 2}}}{\sum_{j^{\prime}}\left(1+\left\|z_{i}-\mu_{j}\right\|^{2}/\alpha\right)^{- \frac{8+1}{2}}} \tag{2}\]  
\(\alpha\) is the degree of freedom, but since it is not possible to cross-validate the validation set in an unsupervised environment, this parameter learning is redundant and, in all later experiments, we set \(\alpha=1\)[27]. \(q_{ij}\) represents the probability of assigning the \(i\)-th sample to the \(j\)-th cluster. \(\mu_{j}\) is the \(j\)-th cluster centroid obtained from the initialization of k-means. After that, an auxiliary target distribution \(P=\left\{p_{ij}\right\}\) is constructed to help iteratively optimize the cluster assignment. We used KL divergence as the clustering loss between the soft assignment \(Q\) and the auxiliary distribution \(P\). Here, unlike general autoencoder clustering, our clustering loss was used to scatter the embedded points \(z\) and the reconstruction loss. This makes sure that the embedded space preserves the local structure of the data-generating distribution. The clustering loss \(L_{c}\) was computed as follows:  
\[L_{c}=KL(P\|Q)=\sum_{i}\sum_{j}p_{ij}\log\frac{p_{ij}}{q_{ij}} \tag{3}\]  
where the auxiliary target distribution P is obtained by normalizing \(Q\) squared:  
\[p_{ij}=\frac{q_{ij}^{2}/h_{j}}{\sum_{j^{\prime}}q_{ij^{\prime}}^{2}/h_{j^{ \prime}}} \tag{4}\]  
The gradients of \(L_{c}\) for each cluster centroid \(\mu_{j}\) and for each embedding point \(z_{i}\) were subsequently calculated as:  
\[\frac{\partial L_{c}}{\partial c_{j}}=2\sum_{j=1}^{K}\left(1+\left\|z_{i}-\mu _{j}\right\|^{2}\right)^{-1}\times\left(q_{ij}-p_{ij}\right)\left(z_{i}-\mu_{ j}\right) \tag{5}\]  
\[\frac{\partial L_{c}}{\partial z_{i}}=2\sum_{j=1}^{K}\left(1+\left\|z_{i}-\mu _{j}\right\|^{2}\right)^{-1}\times\left(p_{ij}-q_{ij}\right)\left(z_{i}-\mu_ {j}\right) \tag{6}\]  
The gradient of \(L_{c}\) to \(z_{i}\) was passed to the deep network and used for standard backpropagation to compute the parameter gradients. Finally, to obtain the final cluster assignment, the iteration was stopped when it reaches the maximum number of iterations or when the update of the cluster assignment for two consecutive optimization iterations is less than a threshold. In order to clearly show the deep autoencoder clustering computation step, the specific steps are summarized in Algorithm 1.