# On the Use of Transformer-Based Models for Intent Detection Using Clustering Algorithms
## 2 Related Work  
In recent years, there has been increasing research on intent detection using unsupervised methods. However, supervised methods remain the most commonly used [3; 4; 5; 6]. Furthermore, classical clustering algorithms are the most popular unsupervised method [7; 8; 9]. For example, a hidden Markov model with multivariate Gaussian distributions can be used to generate each utterance from the Global Vectors (GloVe) model's vectors, applying a weight to every word. This approach can be compared with other unsupervised methods, such as K-Means [10].  
A proposed framework, denoted AutoDial, utilizes multiple features, including frequent keywords present in utterances, part-of-speech tags, and topics. It performs feature assembly by leveraging an autoencoder [11]. It also clusters the user dialog intents by employing hierarchical clustering and displays superior intent clustering results compared to conventional approaches that use K-Means. An ensemble approach was used in a different study to discover semantically related intents by comparing multiple classical word embedding methods, including Word2 Vec and GloVe [12].  
One of the challenges with clustering data is determining the appropriate number of partitions to use. A usual approach is to set the number of partitions to a value greater than the number of class labels determined by ground truth [12]. A possible way of addressing this problem is by using density-based clustering algorithms, for example, Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN), that are capable of retrieving dynamically formed natural clusters [13].  
Previous studies have employed a Bidirectional Encoder Representations from Transformers (BERT) model trained using a combination of unlabeled and labeled utterances to determine a user's intent with a low amount of labeled data, using a clustering layer on top [14]. By creating a pairwise similarity matrix and using ground truth labels, the model can identify similarities and differences among utterances. A threshold determines which samples to train on. As the training progresses, the model performs self-labeling on more difficult unlabeled pairs. The model discards pairs that are neither very similar nor dissimilar. By taking this approach, a semi-supervised framework named Constrained DeepAdaptive Clustering with Cluster Refinement (CDAC) can produce a binary classification from a multi-class dataset. This approach can help the model learn intent discriminative features and find new intents [14].  
Although some studies have reported the superiority of unsupervised methods [14; 15], these approaches have significant limitations for practical applications since it is impossible to find data for all scenarios. [PERSON] and [PERSON] [16] proposed a different approach to intent detection by training a Bidirectional Long Short-Term Memory (Bi-LSTM) on labeled data to identify their distinctive characteristics. The learned features were then used as input to an anomaly detection algorithm to identify previously unseen intents. Another model could then process new intents to assess feasible labels.  
Multi-view clustering is another approach that was used for intent clustering. A better final clustering solution can be achieved by using several sample representations. This approach has been shown to surpass approaches that use a single algorithm alone [17]. The alternating-view K-Means model employs this concept to better represent a user's utterance [18].  
The examined works proposed a methodology to address the same problem studied in this work. Still, most have tried using supervised learning or intended to discover further examples for a given category. The approach followed in this work goes in a different direction, using unsupervised methods (clustering algorithms) fed with sentence embeddings from machine learning-based models (transformers).