## 6 Evaluation
### BERT Model Evaluation
#### 6.5.3 Analyzing the Effect of Different Data Pre-Processing Methods  
Adopting the top-performing approach based on the cased version of the BETO pretrained model as a reference, we decided to further explore the use of some of the most popular text preprocessing techniques to find out whether they are actually useful or not. Evaluated techniques include the following:  
* Stop word removal: we remove words conveying low-level information (such as articles, prepositions, pronouns, conjunctions, etc.) to give more focus to the important ones. The removal of stop words is expected to reduce the vocabulary size (i.e., model complexity), and thus to reduce the training time due to the fewer number of tokens involved in the training. We used _spaCy_ library [48] to remove all the stop words. * Punctuation removal: punctuation marks are often referred to as unhelpful or noisy parts of the text data that we should get rid of before training our models. Here, we also used the _spaCy_ library to remove all of them. * Lemmatization: lemmatization is another method to normalize the text (i.e., to keep the vocabulary small, which helps to improve the accuracy of many language modeling tasks). Particularly, lemmatization can be applied to generate the root form of the words. Again the _spaCy_ library was used for this. * Basic text normalization: special words including emojis, emails, percentages, money, phone numbers, times, dates, URLs and/or hashtags are assimilated and replaced by special tokens, such as MAIL, DATE, URL, etc., to prevent information from being lost during data representation. The obtained results were summarized and sorted in terms of performance in Table 4. None of the applied techniques that are aimed at removing tokens were found to be effective. In short, DL methods that use embedding representations seem to not require any removing. Specifically, in that n-dimensional vector words like \"dogs\" and \"dog\" would already be closer to each other. So, the need to lemmatize becomes unnecessary. With regards to stop words and punctuation marks, we observe that our intent detection task cannot be accomplished properly after their removal. The removal of either stop words or punctuation marks is highly dependent on the task we are performing and the goal we want to achieve. In this case, we can confirm that the effect of removing them was clearly negative. Generally speaking, we should not remove anything (e.g., a word or a punctuation mark) that could be useful in some way. Again, DL models working with vector embeddings, similarly to lemmatization, are currently the best methods to handle and filter those irrelevant terms. With regards to punctuation it is important to clarify that our word embeddings support punctuation and special symbols. Hence, it is reasonable to confirm that in this scenario it is better to retain punctuation as our model works better with it (removing the punctuation from the text is fine if your word embedding model does not support punctuation). Finally, basic text normalization proved to be successful (i.e., performance decreases if we omit it). In this case, the process of transforming some words into their single canonical form still helps our model by reducing the number of unique words (i.e., reducing the vocabulary size helps reducing the model complexity and improving its performance). \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Technique** & **F1** & **Precision** & **Recall** \\ \hline removing punctuation & 78.22 & 80.38 & 78.65 \\ without basic preprocessing & 77.75 & 80.10 & 78.01 \\ lemmatization & 76.91 & 79.37 & 77.14 \\ removing stopwords & 73.88 & 77.15 & 74.20 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results obtained for our top-performing approach based on cased version of BETO pretrained model when enriched with 50 new tokens and applying a specific text preprocessing technique (five-fold CV results).