# On the Use of Transformer-Based Models for Intent Detection Using Clustering Algorithms
## 5 Conclusions  
This article proposes a sequence of procedures and models for intent clustering. For this purpose, a dataset containing multiple utterances from multiple domains was utilized. Standard transformer-based models with different architectures were initially examined. Afterward, various pooling approaches were examined, and the transformer-based models' layers were assessed individually and jointly. The result of using dimension reduction and different clustering methods was also assessed. New models were also trained on domain data with positive results. Lastly, ensemble-based approaches were examined for combining the embeddings from the best-performing models. It was concluded that, among the examined transformers, GPT-2, RoBERTa, and BERT were found to be the best for the examined problem, especially when using average pooling. K-Means was found to be the most suitable clustering algorithm, although a suitable performance was also reached using agglomerative clustering with average or Ward linkage. It was also determined that using dimension reduction (UMAP or t-SNE) can substantially improve the clustering results. Both tested algorithms could maintain the most relevant information even when using substantial reduction, achieving an immense \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Ensemble Configuration**} & \multirow{2}{*}{**Input Embedding**} & **V-Measure** & **Accuracy** \\  & & **Mean \(\pm\) (SD)** & **Mean \(\pm\) (SD)** \\ \hline \multirow{4}{*}{10 K-Means} & SBERT-QA & 84.44 \(\pm\) (1.15) & 73.56 \(\pm\) (3.29) \\  & SRoBERTa-QA & 85.34 \(\pm\) (0.97) & 78.21 \(\pm\) (2.09) \\  & USE & 84.27 \(\pm\) (0.82) & 74.54 \(\pm\) (2.99) \\  & **All three** & **86.74 \(\pm\) (0.72)** & **79.41 \(\pm\) (2.06)** \\ \hline \multirow{4}{*}{50 K-Means} & SBERT-QA & 85.72 \(\pm\) (0.20) & 78.38 \(\pm\) (0.88) \\  & SRoBERTa-QA & 85.58 \(\pm\) (0.74) & 76.74 \(\pm\) (1.43) \\  & USE & 84.59 \(\pm\) (0.53) & 75.31 \(\pm\) (0.86) \\  & **All three** & **87.26 \(\pm\) (0.31)** & **81.25 \(\pm\) (0.67)** \\ \hline \multirow{4}{*}{100 K-Means} & SBERT-QA & 85.93 \(\pm\) (0.69) & 78.85 \(\pm\) (1.84) \\  & SRoBERTa-QA & 85.43 \(\pm\) (0.33) & 78.30 \(\pm\) (0.72) \\ \cline{1-1}  & USE & 83.93 \(\pm\) (0.73) & 74.09 \(\pm\) (1.99) \\ \cline{1-1}  & **All three** & **87.40 \(\pm\) (0.19)** & **81.48 \(\pm\) (0.56)** \\ \hline \multirow{4}{*}{1 K-Means and 4 agglomerative, each with a different linkage (Ward, average, max, and single)} & SBERT-QA & 84.60 \(\pm\) (0.74) & 74.86 \(\pm\) (1.84) \\ \cline{1-1}  & SRoBERTa-QA & 85.81 \(\pm\) (0.43) & 75.33 \(\pm\) (1.33) \\ \cline{1-1}  & USE & 86.22 \(\pm\) (0.65) & 76.80 \(\pm\) (2.24) \\ \cline{1-1}  & **All three** & **87.17 \(\pm\) (0.78)** & **78.46 \(\pm\) (1.22)** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Results (in %) of the ensemble-based approaches. performance improvement. Consequently, they support using slower clustering methods that would otherwise be computationally unfeasible. The two examined pre-trained Siamese Transformers (SBERT and SRoBERTa) provide better embeddings for intent clustering than the baseline models. Nonetheless, it was verified that they could be enhanced even further through fine-tuning domain-specific data. Furthermore, the performance can be improved using an ensemble composed of several K-Means instances, whose input embeddings are from these retrained models. As a result, it was concluded that the key elements for achieving superior outcomes are embedding quality and the utilization of dimension reduction methods. The proposed techniques provide a framework for future work aimed at creating a fully automated system that can cluster sentences based on their underlying intents. Surprisingly, pre-trained embeddings of GloVe and Word2 Vec showed decent results right out of the box. The utterances used for evaluation are short sentences that may explain how a simple average pooling can obtain comparable results to Siamese transformers.