# Learning Distributed Representations and Deep Embedded Clustering of Texts
## 4 Algorithms
for\(e=1\)to\(E\)do
2 Sample a mini-batch \(\{x_{i}\}|(i\in\{1,2,\cdots,M\})\) from dataset \(S\);
3 Sample two augmentations \(A_{1}^{S}\), \(A_{2}^{S}\); for\(i=1\)to\(|S|\)do
4\(x_{i}x_{i}^{1},x_{i}^{2}\) is represented by DistilBERT;
5 Compute the instance loss by Equation (5);
6 Calculate the clustering centers;
7 Cluster these vectors by different clustering methods;
8 Calculate the predicted labels for all vectors;
9 Compute the clustering loss by Equation (12);
10 Compute the overall loss by Equation (13);
11 Calculate the ACC, NMI by comparing predicted labels to ground truth labels;
12 Update gradients to minimize the overall loss;
13
14 return ACC, NMI. ```  
**Algorithm 1**Deep Embedded Clustering Framework  
```
Input: Matrices \(Z,Z^{1},Z^{2}\)
1 Initialize labels \(L,L_{1},L_{2}\); for\(i=1\)to\(|S|\)do
2\(z_{i}\) is projected to \(K\) dimensions;
3\(z_{i}^{1}\) is projected to \(K\) dimensions;
4\(z_{i}^{1}\) is projected to \(K\) dimensions;
5\(z_{i}^{1}\) is projected to \(K\) dimensions;
6\(ind_{max}\gets 1\), \(ind_{max}^{1}\gets 1\), \(ind_{max}^{2}\gets 1\);
7\(z_{max}\gets z_{i}[1]\), \(z_{max}^{1}\gets z_{i}^{1}[1]\);
8 for\(j=2\)to\(K\)do
9 if\(z_{max}<z_{i}[j]\)then
10\(z_{max}\gets z_{i}[j]\);
11\(ind_{max}\gets j\);
12
13\(L_{i}\gets ind_{max}\);
14
15 for\(j=2\)to\(K\)do
16 if\(z_{max}^{1}<z_{i}^{1}[j]\)then
17\(z_{max}^{1}\gets z_{i}^{1}[j]\);
18\(ind_{max}^{1}\gets j\);
19
20\(L_{i}^{1}\gets ind_{max}^{1}\);
21 for\(j=2\)to\(K\)do
22 if\(z_{max}^{2}<z_{i}^{1}[j]\)then
23\(z_{max}^{2}\gets z_{i}^{2}[j]\);
24\(ind_{max}^{2}\gets j\);
25
26
27
28
29
30\(L_{i}^{2}\gets ind_{max}^{2}\);
31
32 return\(L,L_{1},L_{2}\). ```  
**Algorithm 2**Label as Representation Algorithm  
The SOM clustering algorithm is described in Algorithm 4. The input data includes parameters \(M,N\) for the map, the input matrix \(Z\), the iteration times \(T\), and the parameter \(\alpha\). The weight matrix \(W\) is initialized where each raw is a unit vector. To compare the input matrix \(W\) to \(Z\), each raw of \(Z\) is normalized to a unit vector. By comparing \(Z\) to \(W\), the distance from \(z_{i}\) to all the neurons is calculated. The neuron with the minimum distance is selected as the winner. The label of \(z_{i}\) is determined. The learning rate is updated. After that, \(h_{ci}(t)\) is calculated by Equation (9) with new \(\delta(t)\). The weights are updated by the new \(h_{ci}(t)\). The procedure is repeated until \(t=T\). The time complexity of Algorithm 4 is \(O(MN|S|T)\). ```
Input: Matrices \(Z\), \(K\)
1 Initialize \(K\) centroids \(c_{1},\cdots,c_{K}\) randomly;
2\(t\gets 0\);
3 whileConditions are satisfieddo
4\(c_{1}^{\prime}\gets c_{1},\cdots,c_{K}^{\prime}\gets c_{K}\);
5 Initialize empty sets \(C_{1},\cdots,C_{K}\);
6 for\(i=1\) to \(|S|\)do
7\(d_{1}\leftarrow||z_{i}-c_{1}||^{2}\), \(ind\gets 1\);
8 for\(j=2\) to \(K\)do
9 if\(d_{1}<||z_{i}-c_{j}||^{2}\)then
10\(d_{1}\leftarrow||z_{i}-c_{j}||^{2}\);
11\(ind\gets j\);
12
13\(C_{j}\gets C_{j}\cup z_{ind}\);
14\(c_{1}\gets avg(C_{1}),\cdots,c_{K}\gets avg(C_{K})\);
15\(t\gets t+1\)
16 return\(L\). ```  
**Algorithm 3**Kmeans Clustering Algorithm  
```
Input: Matrices \(Z\), \(K\)
1 Initialize the weights \(W(0)\);
2 Normalize \(Z\) by equation (7);
3 for\(t=1\) to \(T\)do
4 for\(i=1\) to \(|S|\)do
5\(d\gets 100\), \(L\gets 0_{1\times|S|}\);
6 for\(j=1\) to \(M\)do
7 for\(k=1\) to \(N\)do
8 if\(||z_{i}-W_{jk}||^{2}<d\)then
12\(d\leftarrow||z_{i}-W_{jk}||^{2}\);
13\(L_{i}\leftarrow(j-1)*N+k\);
14
15
16 Calculate \(h_{cl}(t)\) by Equation (9);
17 Update learning rate \(l_{r}(t)=\alpha*(1-\frac{t}{\gamma})\);
18 Update weights by Equation (10);
19 return\(L\). ```  
**Algorithm 4**SOM Clustering Algorithm