# On the Use of Transformer-Based Models for Intent Detection Using Clustering Algorithms
## 3 Materials and Methods
### Transformer Models for Word Embeddings  
Typically, when training a neural network, the data is labeled to enable a supervised approach. Such information allows the network to discern patterns and recognize relevant input features based on the labels. In contrast, self-supervised training occurs when the network uses the data as labels, enabling it to learn autonomously without human intervention. In the field of NLP, a common self-supervised training method involves utilizing a vast text corpus such as Wikipedia. A sequence of words is fed to the model, which is tasked with predicting a subsequent word. This technique is named language modeling and is frequently employed in pre-training, where large models, including transformer-based ones, are pre-trained on extensive data to acquire general language knowledge and features. Transfer learning is applied after the pre-training stage, allowing the models to learn from smaller amounts of labeled data. Pre-training before actual training can result in better performance than training a model from scratch. Several models have been developed and made publicly available, generally following the approach proposed for BERT [27]. The Transformer model utilizes self-attention as its core concept, enabling it to process complete input sequences simultaneously and capture the interdependence among different parts of the sequence. This ability to perform parallel processing makes it faster than conventional recurrent neural networks and enables it to handle longer sequences effectively. Six Transformer models, chosen for their excellence in NLP's state-of-the-art, were selected for this study. To ensure fairness, all models, except for one, have a comparable number of parameters. The models are available at the HuggingFace model hub, and the employed embedding dimension was 768. The examined models were:  
BERT employs two pre-training tasks to learn both word- and sentence-level representations simultaneously [27]. The first task is Masked Language Modeling (MLM), where the model fills in the missing words in a sequence. The second task is Next Sentence Prediction (NSP), where BERT receives a pair of sentences and predicts whether the second sentence implies the first one or not. The model uses a classification token called (CLS) to classify the sentence pairs into two categories: sequential (50% of cases) and randomly drawn from the corpus with no entailment relationship (the other 50%). The examined model consists of 12 transformer encoder layers, each having a multi-head self-attention mechanism followed by a feedforward neural that maps the input sequence to a higher-dimensional space, thus enabling the model to capture more complex relationships between words. RoBERTa shares the same architecture as BERT, but it was trained on a larger dataset with a more effective training procedure. Furthermore, the NSP was removed since it resulted in inferior downstream results [28]. GPT-2 is a transformer-based language model that utilizes a standard architecture with decoder blocks. It is an autoregressive model that considers previous tokens to forecast the current token. As a result, it is a unidirectional model, which can limit its ability to understand the context that comes after the current token [29]. Its architecture comprises 12 transformer encoder layers that use a self-attention mechanism and a feed-forward neural network. This design allows the model to concentrate on various segments of the input sequence and extract significant features. This is made possible by the self-attention mechanism that enables the model to recognize dependencies between different tokens in the sequence. The output of the self-attention mechanism is processed by a feedforward neural network, which applies non-linear transformations to refine the features further. ALBERT is a simpler variation of BERT. It tries to keep the same performance while improving computational efficiency.