## 7 Experiments
### Selection of Best Transformer Model  
Table 10 compares the performance of BERT model, bare BERT and its DistilBERT and XLNet variants. We compared the performance of these models with BiLSTM + CNN architecture. Regarding the pre-processing, it was a little bit different in this case. As the transformer models are available pre-trained on the text corpora in two different versions-cased and uncased (except the XLNet model, that doesn't come with the uncased version). Furthermore, we used the BERT tokenizer, in which we used lowercasing of the input text (in cased versions) or did not use it (in uncased versions). As described in Section 6.2, we trained two BERT-base for sequence classification models, two bare BERT-base models, two bare BERT-large models, two DistilBERT models and a single XLNet model. We used default hyper-parameters as depicted in Table 11.  
Table 10 summarizes the model's performances using the cased and uncased version. We also tried to use a BERT-large version of the BERT sequence classification models, but it is probable that those models were over-fitting in the first epoch and the results were worse than in BERT-base version.  
Based on the previous experiment, the BERT-base uncased model provided the best results among the transformer models. Following the initial experiments, we proceed with the fine-tuning of the model. We optimized the values of the hyper-parameters summarized in Table 12. Optimization of the hyper-parameters did not lead to a significantly improved performance, however, for the combination of the accuracy and AUC metrics, the best combination of hyper-parameters turned out to be the settings: learning rate = 0.00002, batch size = 16, dropout = 0.15. The results of the fine-tuned model are shown in Table 13. This model also achieved the best micro-averaged F1 metrics.  
\begin{table}
\begin{tabular}{l c c c c c} \hline  & **Accuracy** & **AUC Score** & & **Precision** & **Recall** & **F1 Score** \\ \hline
**BERT-base (cased)** & 0.8884 & 0.9624 & micro avg.: & 0.66 & 0.64 & 0.65 \\  & & & macro avg.: & 0.34 & 0.34 & 0.34 \\ \hline
**BERT-base (uncased)** & 0.8798 & 0.9700 & micro avg.: & 0.61 & 0.71 & 0.65 \\  & & & macro avg.: & 0.31 & 0.38 & 0.34 \\ \hline
**bare BERT-base (cased)** & 0.8998 & 0.9802 & micro avg.: & 0.69 & 0.68 & 0.69 \\  & & & macro avg.: & 0.63 & 0.51 & 0.51 \\ \hline
**bare BERT-base (uncased)** & 0.8841 & 0.9839 & micro avg.: & 0.60 & 0.78 & **0.68** \\  & & & macro avg.: & 0.60 & 0.58 & **0.57** \\ \hline
**bare BERT-large (cased)** & 0.8795 & 0.9801 & micro avg.: & 0.62 & 0.73 & 0.67 \\  & & & macro avg.: & 0.53 & 0.51 & 0.51 \\ \hline
**bare BERT-large (uncased)** & 0.8921 & 0.9806 & micro avg.: & 0.67 & 0.68 & 0.67 \\  & & & macro avg.: & 0.60 & 0.41 & 0.42 \\ \hline
**DistilBERT (cased)** & 0.8866 & 0.9649 & micro avg.: & 0.63 & 0.68 & 0.66 \\  & & & macro avg.: & 0.34 & 0.37 & 0.34 \\ \hline
**DistilBERT (uncased)** & 0.8649 & 0.9781 & micro avg.: & 0.59 & 0.73 & 0.65 \\  & & & macro avg.: & 0.48 & 0.39 & 0.34 \\ \hline
**XLNet (cased)** & 0.9630 & 0.9530 & micro avg.: & 0.60 & 0.70 & 0.65 \\  & & & macro avg.: & 0.30 & 0.37 & 0.33 \\ \hline \end{tabular}
\end{table}
Table 10: Performance of the transformer models.  
\begin{table}
\begin{tabular}{c c} \hline
**Hyper-Parameters** & **Values** \\ \hline Batch size & 32 \\ Learning rate & 0.00003 \\ Dropout rate & 0.15 \\ \hline \end{tabular}
\end{table}
Table 11: Hyper-parameters used for transformer models from Table 10.