## 7 Experiments
### Analysis of Text Representation and Pre-Processing Influence on Deep Learning Models  
From the initial set of experiments, we selected a composed BiLSTM network architecture in combination with a convolution layer to be the most suitable to explore the effects of different pre-processing. During the following experiments, we focused on using different text representation and pre-processing settings. We computed commonly used metrics in classification, including accuracy, AUC score, precision, recall and F1. Interesting is an F1 score as it expresses the harmonic mean of precision and recall and describes the overall performance of the model better. We compared the performance of the model with a standard pre-processed text corpus and without pre-processing (only using simple tokenization). We also decided to compare different text representations, which we described in Section 4. In the first step, we explored how the model performs when using the TF-IDF data representation. Table 7 summarizes the results of the experiments. Basic document vector representation using TF-IDF did not prove to be very suitable for this task. The performance of the model using this representation suffered from poor recall. Although accuracy and AUC values gain reasonable values, those metrics are not very useful in imbalanced classes. To better understand the classifier performance, precision and recall provide better insight. In this case, it is clear that the minor classes failed to learn completely. On the other hand, we can observe that the standard pre-processing improves the classification (contrary to the expectations). In TF-IDF, the pre-processing may improve the created vector representation, as it is created from the corpus itself (not from pre-trained vectors, such embeddings). Table 8 depicts the BiLSTM + CNN model performance using the word2 vec embeddings. In both cases (with and without pre-processing), word2 vec representations were trained from the \begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline
**Model** & **1** & **2** & **3** & **4** & **5** & **6** & **7** & **8** & **9** & **10** & **avg** \\ \hline
**Loss** & 0.047 & 0.045 & 0.045 & 0.045 & 0.047 & 0.047 & 0.045 & 0.046 & 0.045 & 0.043 & 0.046 \\
**Accuracy** & 0.991 & 0.994 & 0.994 & 0.971 & 0.991 & 0.991 & 0.994 & 0.994 & 0.994 & 0.991 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Cross-validation of the BiLSTM + CNN model. \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyper-Parameters** & **Values** \\ \hline Activation & ‘relu’, ‘than’ \\ Batch Size & [16, 32, 64] \\ Optimizer & [‘SGD’, ‘RMSProp’, ‘Adam’ \\ Dropout rate & [0.1, 0.2] \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Hyper-parameters used in grid search for the BiLSTM + CNN model. dataset. word2 vec representation brings massive improvement in comparison to TF-IDF, rapidly improving the performance metrics (both, micro and macro averaged). The results also demonstrate the influence of the pre-processing techniques applied in text preparation. The model gained slightly better performance on the not processed text, improving recall values (most importantly, macro-averaged recall). Table 9 summarizes the model performance using pre-trained word embeddings, both with standard text pre-processing and a model with no pre-processing. We used two different GloVe pre-trained embeddings, Common Crawl (840 B tokens, 2.2 M vocab) and Twitter (2 B tweets, 27 B tokens). The model performer very similar using different GloVe and fastText embeddings. Although the averaged F1 metrics are very similar, we can observe some differences, how the models perform on precision and recall metrics. Skipping of the pre-processing in case of the CC GloVe embeddings causes recall drop and improvement of the precision, while in case of the Twitter GloVe embeddings it is otherwise. The difference may be caused by either size of the tokes, or how the embeddings were trained.