# On the Use of Transformer-Based Models for Intent Detection Using Clustering Algorithms
## 4 Results and Discussion
### Clustering Algorithms  
This experiment first evaluated the two simplest clustering algorithms, K-Means and agglomerative (Agglo) clustering (using four linkage criteria, specifically, Ward, average (avg), maximum (max), and single). The embeddings were transformed with UMAP (found to be the best for dimension reduction) with two components before applying the clustering algorithm. Table 4 reports the results for each model, presenting the mean and Standard Deviation (SD). Examining the table's results, it was concluded that single linkage produced inferior results. Low homogeneity with higher completeness was anticipated as this method combines clusters based on the closest pairs of points. When multiple clusters or points are in close proximity, the algorithm may link them to form a larger cluster. This behavior is commonly referred to as chaining. On the contrary, both K-Means and agglomerative using average criteria linkage attained the best performance. However, since K-Means is a simpler algorithm, it was chosen as the best choice. BERT reached the best-balanced results regarding the examined metrics, followed by GPT-2. The XLNet-based model achieved the worst performance. Unlike the previous cluster algorithms, HDBSCAN, used in the second part of this experiment, needs to tune parameters. This examination aims to empirically find a suitable choice of parameters and study the advantage of outlier detection using BERT embeddings. Cluster size and minimum samples are the two most important parameters for the HDBSCAN algorithm. It was found that the former can be chosen to be a lower value (in this case, 10) while not affecting the performance. Figure 3: Original embeddings mean V-measure comparison with the results when TSNE and UMAP are used, for BERT (three left columns of the group, represented by light colors) and RoBERTa (three right columns of the group, represented with darker colors) models, according to the examined layer. A superior value would lead to a more conservative clustering, with further data points being declared as noise. Therefore, a grid search was conducted for this parameter, varying from 10 to 150 with steps of 10. The results for the two examined cluster extraction approaches (EOM and leaf) are presented in Table 5.  
By examining Table 5 results, it is notorious that leaf and EOM can lead to highly homogeneous clusters. When provided with identical parameters, leaf clustering can produce more refined clusters even when working with less clustered data. The resulting clusters in leaf clustering are more homogeneous because they originate from the bottom of the condensed tree. Due to their smaller size, these clusters tend to be even more homogeneous. In contrast, EOM can cluster a larger amount of data but requires fewer clusters to do so. It is notorious that both algorithms attained the best V-measure using a minimum sample value of 150. Furthermore, the leaf method created 32 clusters, a value closer to the dataset's number of intentions (34), but only 56% of the data were clustered. EOM provided a marginally higher percentage of clustered data, but the number of clusters is lower. Even though a low minimum sample can lead to a greedy algorithm with an inferior V-measure, it has the ability to capture a large number of highly homogeneous clusters. Compared to previous clustering algorithms, HDBSCAN requires the creation of fewer clusters than the number of intents in the examined dataset to surpass their V-measure score. Thus, this method is unsuitable for this analysis.