# On the Use of Transformer-Based Models for Intent Detection Using Clustering Algorithms
## 4 Results and Discussion
### Retraining Siamese Transformers  
This experiment examines whether retraining the studied Siamese transformers can yield a better V-measure. It was reported that training a model on domain-related data can produce better results in NLP [44; 45]. To examine this idea first, the USE model, which has  
Figure 4: Mean V-measure score amongst the SBERT and SRoBERTa model layers. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Minimum**} & \multicolumn{2}{c}{**Homogeneity (\%)**} & \multicolumn{2}{c}{**Completeness (\%)**} & \multicolumn{2}{c}{**V-Measure (\%)**} & \multicolumn{2}{c}{**Clustered Data (\%)**} & \multicolumn{2}{c}{**Number of Clusters Found**} \\ \cline{2-11}  & **EOM** & **Leaf** & **EOM** & **Leaf** & **EOM** & **Leaf** & **EOM** & **Leaf** & **EOM** & **Leaf** \\ \hline
10 & 84.81 & 87.51 & 54.22 & 50.59 & 66.01 & 64.11 & 66.00 & 50.00 & 204.00 & 258.00 \\
20 & 82.94 & 87.79 & 62.41 & 56.01 & 71.23 & 68.36 & 69.00 & 48.00 & 118.00 & 168.00 \\
30 & 82.62 & 85.98 & 64.88 & 59.17 & 72.68 & 70.09 & 65.00 & 49.00 & 92.00 & 123.00 \\
40 & 82.29 & 86.70 & 68.09 & 63.08 & 74.52 & 73.02 & 67.00 & 48.00 & 70.00 & 99.00 \\
50 & 82.58 & 85.44 & 69.18 & 64.63 & 75.29 & 73.59 & 63.00 & 50.00 & 64.00 & 82.00 \\
60 & 27.45 & 84.30 & 85.08 & 65.69 & 41.51 & 73.83 & 93.00 & 48.00 & 12.00 & 73.00 \\
70 & 27.59 & 84.02 & 84.96 & 67.22 & 41.65 & 74.69 & 92.00 & 45.00 & 12.00 & 66.00 \\
80 & 21.98 & 82.99 & 91.64 & 67.32 & 35.46 & 74.34 & 95.00 & 43.00 & 7.00 & 60.00 \\
90 & 80.58 & 82.28 & 75.26 & 68.89 & 77.67 & 75.22 & 58.00 & 42.00 & 42.00 & 54.00 \\
100 & 77.25 & 82.38 & 78.20 & 70.16 & 77.72 & 75.78 & 61.00 & 40.00 & 37.00 & 51.00 \\
110 & 77.77 & 81.90 & 79.64 & 72.54 & 78.69 & 76.94 & 60.00 & 49.00 & 35.00 & 49.00 \\
120 & 76.77 & 82.25 & 80.08 & 74.86 & 78.75 & 78.39 & 60.00 & 37.00 & 31.00 & 44.00 \\
130 & 58.53 & 82.22 & 84.10 & 74.80 & 69.02 & 78.33 & 73.00 & 40.00 & 20.00 & 41.00 \\
140 & 76.11 & 82.80 & 82.70 & 76.77 & 79.27 & 79.67 & 58.00 & 39.00 & 27.00 & 37.00 \\
**150** & **75.83** & **81.41** & **84.01** & **80.01** & **79.71** & **80.74** & **56.00** & **42.00** & **25.00** & **32.00** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Mean HDBSCAN clustering results for the two examined cluster extraction approaches. learned similarity through dialog-type data, was studied in the same configuration as the previous experiment. It attained an average V-measure of 80.45%, a value that surpasses all examined models by at least 2%. During its training in a multi-task scenario, the USE model included input response selection as one of its tasks. The training for this task can be attained using a Siamese structure. In this case, it was using two sentences (response and context) that were separately encoded, constructing two vectors. Afterward, the two embeddings' cosine similarity, or dot product, was calculated similarly to SBERT. The model receives a batch of context-response pairs during training to learn similarity in input-response selection. In this scenario, the response was considered a positive sample for a particular context, while all other responses from different pairs were considered negative or incorrect. To train the model, the negative log-likelihood of the SoftMax score was minimized using the Multiple Negative Ranking (MNR) loss function. Inspired by USE results, the two examined Siamese Transformer models were retrained from their pre-trained checkpoints in the Amazon's-QA dataset [46; 47] using the MNR loss and mean-pooling. The dataset consists of 1.4 million questions and answers. From these, 9011 were selected from the appliances category. To prepare the training set, contexts containing a straightforward yes or no question were excluded, resulting in 4318 pairs. The V-measures attained by using the retrained models for producing embeddings are presented in Figure 5. This figure shows the performance when using single layers and layer concatenation.