# Soft Semi-Supervised Deep Learning-Based Clustering
## 3 Proposed Soft Constrained Deep Clustering (SC-DEC)
and obtaining the partial derivatives of the parameters to be updated via minimization, as shown below:  
\[J=\sum_{i=1}^{n}\sum_{j=1}^{C}p_{ij}log\frac{p_{ij}}{q_{ij}}+\gamma\left[\sum_{x_ {i},x_{k}\in C}\sum_{j=1}^{C}q_{x_{ij}}^{m}q_{x_{ij}}^{m}\right] \tag{2}\]  
where \(n\) is the number of data points, \(C\) is the number of clusters, \(m\) is the fuzziness parameter that reflects the fuzzy-based representation of the data clusters, and \(\mathbb{C}\) is the set of should-link constraints as shown above. The first term is the deep embedded clustering term of DEC [21], while the second term is designed to learn the compact, fuzzy-based clusters given the supervision soft constraints. The second term basically rewards the model for correctly clustering, i.e., having a high membership value of a should-link pair of points to a certain cluster. In (2), \(q_{ij}\) and \(p_{ij}\) are computed as shown below [21]:  
\[q_{ij}=\frac{(1+\|z_{i}-\mu_{j}\|^{2})^{-1}}{\sum_{j}(1+\|z_{i}-\mu_{j}\|^{2})^ {-1}}, \tag{3}\]  
\[p_{ij}=\frac{q_{ij}^{2}/f_{j}}{\sum_{j}q_{ij}^{2}/f_{j}} \tag{4}\]  
where \(f_{j}=\sum_{i}q_{ij}\) are soft cluster frequencies; \(z_{i}\) = \(f_{\theta}(x_{i})\) is the embedded representation of \(x_{i}\); \(\mu_{j}\) is the \(j^{th}\) cluster centroid in the embedded space; and \(\|.\|\) denotes the L2-norm. Note that \(\gamma\) in the above equation is a predefined trade-off parameter that balances the influence of the supervision soft constraints. It balances the amount of penalty imposed on the data batch for misclustering. Minimizing the above objective function is preceded by the training of an autoencoder AE for DNN parameter initialization. This AE training is performed by minimizing the following objective function:  
\[J_{AE}=\sum_{i=1}^{n}\|x_{i}-z_{i}\|_{2}^{2}+\gamma\left[\sum_{x_{i},x_{k}\in C }\sum_{j=1}^{C}q_{x_{i}j}^{m}q_{x_{i}j}^{m}\right] \tag{5}\]  
where \(\|x_{i}-z_{i}\|_{2}^{2}\) is the squared Euclidean distance between data point \(x_{i}\) and its embedding \(z_{i}\). The proposed algorithm would minimize the objective function in (2) iteratively and converge towards the clustering results under specific criteria (tolerance threshold %). Concretely, the clustering process carried out via our proposed approach follows DEC [21] in its two phases: 1. model parameter initialization. In this phase, the deep embedding parameters are initialized by training a deep autoencoder network. Then, the k-means clustering algorithm [55] is applied to the embedded space \(Z\) to initialize the \(k\) cluster center \(\mu_{j}\). 2. Model parameter optimization, i.e., updating the deep mapping \(f_{\theta}\) and refining the cluster centers \(\mu_{j}\). In phase 2, the proposed objective loss is minimized to learn from the current high confidence predictions by iterating between computing an auxiliary target distribution and minimizing the Kullback-Leibler (KL) divergence to it. To guide the learning process away from the local minima, some side information is adopted within our approach as a set of pairwise constraints. Basically, such information is incorporated to reward or penalize the traditional clustering objective as well as to adapt the distance measure to each cluster. Specifically, soft pairwise constraints are introduced and incorporated to represent the available prior knowledge where we formulate the second term of the model's objective according to these constraints. These pairwise constraints are randomly generated with a size equal to \(\beta\times N\), where \(\beta\) is the number (ratio) of pairwise constraints and \(N\) is the dataset size. Specifically, for each randomly selected data point, we check the corresponding ground-truth labels of the whole dataset. If the ground-truth labels of the two points are similar, a \"should-link\" pairwise link is formed between these points and represented by 1. Otherwise, the pairwise link is represented by 0.