# On the Use of Transformer-Based Models for Intent Detection Using Clustering Algorithms
## 4 Results and Discussion
### Ensemble Models  
This last experiment evaluates whether an ensemble can improve intent detection performance. Specifically, four ensemble configurations were studied:  
Homogeneous ensemble of multiple K-Means fed from the same embedding;  
Homogeneous ensemble of multiple K-Means using embeddings created by combining embeddings from three different models;  
Heterogeneous ensemble composed of two dissimilar clustering algorithms (agglomerative and K-Means) fed with the same embedding;  
Heterogeneous ensemble composed of two dissimilar clustering algorithms (agglomerative and K-Means), where the embeddings were created by combining embeddings from three different models.  
The three models that reached the highest V-measure (considering single layers only since the concatenation did not perform substantially better) were evaluated for embedding creation. These models were the two retrained (on the QA-Amazon dataset) Siamese transformers examined in the previous experiment (SBERT-QA and SRoBERTa-QA) and USE. HDBSCAN was not considered since it was previously found unsuitable for this examination.  
The attained results are presented in Table 6. Checking the results, it was concluded that the performance using an ensemble-based approach had risen substantially compared to the previously found best-performing model without an ensemble (SRoBERTa-QA), whose accuracy and V-measure were 60% and 81%, respectively. The ensemble-based approach's advantage is clear, as ensembling just 10 K-Means produced a substantial improvement in accuracy, ranging from 13% to 19%, when compared to the best-found method without an ensemble. This increase is even higher when the number of ensemble models increases. It is also possible to conclude that the combination of three embeddings results in an enhancement compared to using a single embedding.  
The homogeneous ensembles achieved a better performance when compared to the heterogeneous solutions, a result particularly notorious when checking the attained accuracy. Yet, the model based on USE embeddings substantially benefits from the heterogeneous configuration. The superiority of the homogenous configuration could be attributed to the size of the ensemble matrix and the stochastic nature of K-Means' initializations, which can generate diverse clustering solutions in each run, resulting in some points being, by chance, assigned to their correct group.  
Figure 5: Mean V-measure score amongst the (**a**) SBERT and (**b**) SRoBERTa model layers, which were retrained in the Amazons-QA dataset, compared to their not-retrained versions and baseline models.  
As the ensemble method follows a plurality voting approach, the ensemble matrix sizes for K-Means configurations with 10, 50, and 100 instances are 10, 50, and 100 rows, respectively. On the other hand, the heterogeneous configuration comprises only five rows, each corresponding to a distinct clustering algorithm. Additionally, the input embeddings play a crucial role, and their combination outperforms individual embeddings consistently. Using 100 K-Means led to the best results in this work, although the small ensemble with 50 K-Means reached almost the same performance.  
While comparing our results to the state-of-the-art works described in Section 2, it is important to acknowledge that this can only serve as a preliminary analysis. The reason is that the previous studies have utilized different datasets with differing numbers of intents and reported diverse performance metrics. The most commonly reported metric was accuracy. It was reported to range from 24% to 91%, depending on the examined dataset, while using unsupervised methods [14]. Another study reported accuracy in the range of 40% to 44%, depending on the examined dataset [18], while other studies achieved an accuracy of 73% [10], 79% using unsupervised methods [15], and 84% [11]. It is worth noting that the studies that achieved higher accuracy examined fewer intents (less than 20), while this work examined 34 intents.