## 7 Experiments
### Selection of Best Deep Learning Model  
Initial experiments were aimed to select the most suitable method to explore the pre-processing impact. To do so, we compared the described NN architectures on commonly used embeddings (GloVe). When comparing the particular architectures, we obtained the accuracy performance of each model, cross-validated on the training set and evaluated on the testing set; the results are shown in Table 4.  
To compute the metrics (accuracy and loss), we transformed the class probabilities (output of the neural networks) into the crisp class predictions using a simple rule, which assigned the sample to a class if a probability of a given category was higher than 0.5. During this phase, we worked with this simplistic approach, in further evaluations of the best performing model, we also adopted a more advanced technique to identify the optimal threshold for each class.  
The training phase of the deep learning models on the used dataset is very demanding on computational resources. The training process is very time-consuming, even on recent GPUs. We decided to optimize the hyper-parameters of the best model on 10% stratified sample of the dataset. We performed the fine-tuning of the hyper-parameters using a grid search with cross-validation. We tuned the best-performing model (biLSTM-CNN), which we used with the obtained parameters as the starting architecture for further experiments. Considered parameters used for the grid search are shown in Table 5.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **FFNN** & **CNN** & **GRU** & **LSTM** & **BiLSTM + CNN** & **BiGRU + CNN** \\ \hline
**Accuracy** & 0.878 & 0.886 & 0.872 & 0.873 & 0.890 & 0.884 \\
**F1 score—micro avg** & 0.63 & 0.64 & 0.66 & 0.65 & 0.67 & 0.67 \\
**F1 score—macro avg** & 0.34 & 0.42 & 0.56 & 0.55 & 0.53 & 0.47 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Initial evaluation of the models.  
Figure 9: Overall schema of the experiments setup.  
We achieved the best results using the regularisation dropout 0.2, activation function ReLU, Adam optimizer and batch size 32. The complete results of this experiment are stored on a GitHub ([[https://github.com/[PERSON]/toxic_comments_classification/blob/main/result_gridsearch.txt](https://github.com/[PERSON]/toxic_comments_classification/blob/main/result_gridsearch.txt)]([https://github.com/[PERSON]/toxic_comments_classification/blob/main/result_gridsearch.txt](https://github.com/[PERSON]/toxic_comments_classification/blob/main/result_gridsearch.txt))).  
To gain a better understanding of the learning process and more importantly, to estimate the learning variance, we performed 10-fold cross-validation of the best performing model on the training data. Table 6 summarizes the results of the particular folds during the cross-validation of the BiLSTM + CNN model. This step was important to estimate the learning variance. As we can observe that the overall variance of the learning is acceptable, we will not use the cross-validation during further experiments with pre-processing. This enabled us to reduce the total time needed to train and test all evaluated combinations of the pre-processing and text representation methods.  
All models were implemented in Python language using Tensorflow [62] and Keras [63] libraries. The source codes are available on GitHub ([[https://github.com/[PERSON]/toxic_comments_classification](https://github.com/[PERSON]/toxic_comments_classification)]([https://github.com/[PERSON]/toxic_comments_classification](https://github.com/[PERSON]/toxic_comments_classification))). The experiments were conducted on a PC equipped with a 4-core Intel Xeon processor clocked at 4 GHz and NVIDIA Tesla K40c GPU with 12 GB memory.