# Learning Distributed Representations and Deep Embedded Clustering of Texts
## 4 Algorithms  
To cluster similar assessments, Distill BERT was first used to transform texts into vectors with a text augmentation technique. After augmentation, contrastive learning and clustering techniques were adopted to cluster similar texts. Since for different clustering algorithms the efficiency is different. We propose three different clustering algorithms with the proposed model. The proposed algorithm framework is described in Algorithm 1. To calculate the efficiency of Algorithm 1, the adopt Accuracy (ACC) and Normalized Mutual Information (NMI) are measured. NMI is calculated by \(\text{NMI}(Y,C)=\frac{2\times l(Y,C)}{H(Y)+H(C)}\) where \(Y\) is the real label, \(C\) is the clustering label, H(.) is the cross entropy, and I(.,.) is the mutual information. To describe the algorithm framework clearly in Algorithm 1, the input data contains dataset \(S\), training epochs \(E\), temperature \(\tau\) in contrastive learning and the structure of augmentation, while the output results are ACC and NMI. A mini-batch \(\{x_{i}\}\) of texts is selected from dataset \(S\). For each input sample \(x_{i}\), two random augmentations are generated where \(x_{i}^{a}=A_{j}^{S}(x_{i})(j=1,2)\), which represents a different view of the text and contains some subset of the information in the original sample. \(x_{i}^{1},x_{i}^{2}\) (\(i\in\{1,2,\cdots,|S|\}\)) are denoted as the augmented text of an assessment \(x_{i}\). After text augmentation, the contrastive loss is calculated according to the temperature \(\tau\). Texts are transformed into vectors by DistilBERT. The predicted labels are obtained after the clustering algorithms. By comparing predicted labels to ground truth labels, the clustering loss is calculated. According to the instance loss and clustering loss, the overall loss is obtained. The ACC and NMI metrics are calculated. The deeply embedded clustering framework is described in Algorithm 1. For vector clustering techniques, there are three different strategies: label as representation, Kmeans algorithm, and SOM algorithm. For label as representation, the dimension of vectors is projected to the number of clusters by multiple neuron networks. The index of the maximum values of the vector is represented as the label of clusters. After all of the vectors are projected to the fixed dimensions, Kmeans and SOM algorithms are used to cluster similar vectors together, respectively. To calculate the clustering centers for the label as representation algorithms, Kmeans and SOM algorithms are adopted, respectively. The label as a representation algorithm is shown in Algorithm 2. The input of the algorithm is the matrices \(Z,Z_{1}\), and \(Z_{2}\) which are made up of \(z_{i},z_{i}^{1},z_{i}^{2}(i\in\{1,\cdots,|S|\})\) where \(z_{i},z_{i}^{1},z_{i}^{2}\) are represented by DistilBERT. The labels \(L,L_{1},L_{2}\) are initialized first. According to the idea of the label as representation, all the represented vectors are projected to \(K\) dimensions. The index of the maximum value selects the label for the projected vectors. The time complexity of the Algorithm 2 is \(O(K|S|)\). The Kmeans clustering algorithm is described in Algorithm 3. To describe Algorithm 3 clearly, \(c_{1},\cdots,c_{k}\) are denoted as the initial centroids. All the vectors in \(Z\) are compared with \(K\) centroids. The vector with the minimum distance to the centroid \(c_{i}\) is assigned to \(C_{i}\). After all the vectors are assigned, new centroids are calculated in terms of new \(C_{i}\) (\(i\in\{1,\cdots,K\}\)). The procedure is repeated until the conditions are not satisfied. The label \(L\) for \(Z\) is obtained with the final centroids. The labels \(L_{1},L_{2}\) for \(Z_{1}\) and \(Z_{2}\) are obtained, similarly. The time complexity for Algorithm 3 is \(O(|S|Kt)\). ```
Input: Data set \(S\), Training epochs \(E\), Temperature \(\tau\), Structure of augmentation \(A^{S}\) Output: ACC, NMI.