# Learning Distributed Representations and Deep Embedded Clustering of Texts
## 3 Problem Description and the Models
### The Framework  
To cluster similar texts, the overall framework of our proposed model is shown in Figure 1. Firstly, the texts are represented as vectors PLMs. After that, the clustering technique is used to cluster vectors without the pre-defined number of clusters. To improve the representation performance, contrastive instances for texts are generated by vector representation models. For example, assume that there is assessment evaluation for the topic \"Environmental protection\" in a middle school. There are hundreds of assessments to mark which is a huge workload for instructors. To solve the problem, these assessments are first processed by the data augmentation technique, which can increase the diversity of an assessment. The contrastive learning loss is calculated either after vector representation or clustering, which improves the representation in future.  
In this paper, we used pre-trained BERT [55] to represent words and sentences in the given answers. We also explored the usage of the BiLSTM model [30] for learning word-level and sentence-level information when we had few resources to fine-tune the whole BERT model. The procedure of text clustering, including representation and clustering, is shown in Figure 2.  
Vector Representation  
BERT. In the input, BERT adds a special leading token [cls] at the beginning of the input text (e.g., [cls] Artificial Intelligence is useful in education). The [cls] is used for sentence-level classification tasks during its pre-training stage, and its corresponding hidden vector is used as the sentence-level representation. All input tokens are represented as the additive combination of word embedding and positional embedding. Its core module is multi-head self-attention [56], where input tokens are represented as queries (\(Q_{i}=XW_{i}^{Q}\)), keys (\(K_{i}=XW_{i}^{K}\)) and values (\(V_{i}=XW_{i}^{V}\)) in the \(i\)th head. We have:  
\[SelfAttention(Q_{i},K_{i},V_{i})=softmax(\frac{Q_{i}K_{i}^{T}}{\sqrt{d_{i}^{k} }})V_{i} \tag{1}\]  
\[MultiHead(Q,K,V)=cat(SelfAttention(Q_{i},K_{i},V_{i}))W^{C} \tag{2}\]  
where \(W_{i}^{Q}\in\Re^{d\times d_{i}^{k}},W_{i}^{K}\in\Re^{d\times d_{i}^{k}},W_{i} ^{V}\in\Re^{d\times d_{i}^{m}}\)\(W^{C}\in\Re^{\sum_{i=1}^{N_{h}}d_{i}^{e}\times d}\) are the trainable parameters. Finally, BERT has residual connection followed by layer-normalization [57]. A fully connected feed-forward network is applied to the matrix \(Z_{1}\) as follows  
\[FF(Z_{1})=max(0,Z_{1}W_{1}^{F}+b_{1}^{F})W_{2}^{F}+b_{2}^{F} \tag{3}\]  
where \(W_{1}^{F},b_{1}^{F},W_{2}^{F},b_{2}^{F}\) are parameter matrices. After feed-forward, Add&Norm is reused in the layer.  
BERT performs well in transforming texts to vectors, although it consumes lots of server resources. To solve the problem, DistilBERT was adopted in this paper, which has the same general architecture as BERT [39]. The token-type embedding and the pooler are removed, while the number of layers is reduced by a factor of 2. Most of the operations used in the Transformer architecture (linear layer and layer normalization) are highly optimized in modern linear algebra frameworks and our investigations showed that variations on the last dimension of the tensor (hidden size dimension) had a smaller impact on computation efficiency (for a fixed parameter budget) than variations on other factors, like the number of layers.  
Figure 1: The proposed overall framework.  
Figure 2: The procedure of text clustering.