# On the Use of Transformer-Based Models for Intent Detection Using Clustering Algorithms
## 3 Materials and Methods
### Baseline Models for Word Embeddings  
Numerous methods exist for representing text, with the most recent approaches employing prediction-based models. These models are trained on large corpus in a self-supervised manner, typically through language modeling and similar techniques that use machine learning algorithms, such as neural networks, to generate word representations in dense vectors. There are two types of word embeddings (word representations): static representations, which provide a representation for each word independent of its context or surrounding words, and contextual representations, which can vary depending on the specific context in which a word is used.  
Word2 Vec [20] is a standard model to create word and sentence embeddings. It uses a neural network with only one hidden layer. There are two variations of this method: the continuous bag of words, which tries to predict the target words from a given set of surrounding words (context words) [20], and the skip-gram, which, given only one context word, predicts the surrounding words [21]. GloVe is similar to Word2 Vec, but the training is performed on combined global word-word co-occurrence statistics from a corpus. These two approaches are standard static word vectors, frequently employed as baselines when performing sentence representation [22; 23]. One of the main limitations of static representations is that they assign a fixed mapping to each word without considering its context. As a result, approaches based on this methodology cannot capture the semantic and syntactic meanings of words in each sentence as the terms are represented independently. Deep contextual representations become a topic of interest as they can address these limitations.  
Newer models such as ELMo [24], universal language model fine-tuning [25], and InferSent [26] employ context-dependent approaches by using a Bi-LSTM architecture. Of these methods, ELMo is usually a good choice when there is a need for embeddings to capture the nuances of language and context dependency. As a result, the selected baseline models to compare against the transformer-based models were Word2 Vec, GloVe, and ELMo embeddings.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Domain** & **Number of Intents** & **Intent Names** \\ \hline Buses & 2 & FindBus and BuyBusTicket \\ \hline Calendar & 1 & GetAvailableTime \\ \hline Events & 5 & FindEvents, GetEvents, GetEventDates, \\  & & BuyEventTickets, and FindAttractions \\ \hline \multirow{3}{*}{Flights} & \multirow{3}{*}{4} & SearchOnewayFlight, SearchRoundtripFlights, \\  & & ReserveOnewayFlight, and \\  & & ReserveRoundtripFlights \\ \hline Homes & 3 & FindApartment, BookHouse, and SearchHouse \\ \hline Hotels & 2 & ReserveHotel and SearchHotel \\ \hline Media & 1 & PlayMedia \\ \hline Movies & 3 & FindMovies, PlayMovie, and GetTimesForMovie \\ \hline Music & 3 & PlaySong, LookupSong, and LookupMusic \\ \hline Payment & 2 & Check\_balance and TransferMoney \\ \hline RentalCars & 2 & ReserveCar and GetCarsAvailable \\ \hline Restaurants & 2 & ReserveRestaurant and FindRestaurants \\ \hline Services & 3 & BookAppointment, GetRide, and FindProvider \\ \hline Weather & 1 & GetWeather \\ \hline \hline \end{tabular}
\end{table}
Table 1: Characteristics of the examined dataset.