## 8 Conclusions  
In work presented in this paper, we aimed to compare and evaluate different current state-of-the-art models for multi-label toxic comments classification. We experimentally evaluated the performance of deep learning models, including composed architectures with different methods of text representation and pre-processing. On top of that, currently, popular transformer language models, such as BERT and its modifications were compared as well. We aimed to explore the assumption that in tasks such as detection of anti-social behaviour in the online environments, the application of traditional pre-processing techniques could lead to loss of particular specific information characteristic for such behaviour. We aimed to explore the influence of different pre-processing and representation methods on the deep learning and transformer models also in multi-label task aimed to detect the specific type of anti-social behaviour (in this case, toxic comments). We experimentally evaluated composed architecture of BiLSTM + CNN network with different text representations, pre-trained embeddings and compared it with BERT and its variants. Further evaluation of the various data preparation techniques confirmed the assumption, that in this type of task, using standard pre-processing may lead to influence the classifier performance. When comparing the network performance using different word embeddings, the results showed, that the application of traditional text preparation techniques does not bring any significant benefit in terms of evaluation metrics. When comparing the standard classification performance metrics, we face the problem of the class imbalance. The models often struggle to perform well in the minority classes. If the collection of more samples from such classes is not straightforward, more advanced approaches or using data augmentation techniques may be required. One of the possible strategies could be represented by the hierarchical ensemble model. In such an approach, different classifiers could be combined to either address the specific classes, or even to perform the classification on a different level of target attribute generalization. The particular model then could be used to distinguish between the toxic and non-toxic comments and particular toxicity type prediction could be handled by separate models. The class imbalance could be addressed by various techniques, e.g., weighting schemes in the ensemble. It is also important to note that the presented comparison was carried out on a single dataset. Another limitation of the study is the use of pre-trained word embeddings and pre-trained transformer language models (except word2 vec embeddings). Usually, pre-trained embeddings are build using corpora consisting of standard, clean forms of the words. Such representations do not cover the entire vocabulary used in the abusive comments, which results in the loss of information. In future work, we would like to focus on embedding custom-training for this particular domain. A very similar situation is with transfer learning models. In most of the NLP tasks, those models outperform the deep learning models. Pre-trained models may represent not an ideal solution in this case, and their further re-training may lead to superior results. Another factor is the computational intensiveness of \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **BiLSTM + CNN** & **bare BERT** & **Intel300**[66] & **fastText300**[66] & **GloVe300**[66] & **word2 vec300**[66] \\ \hline
**toxic** & 0.9238 & 0.9382 & 0.95 & 0.95 & 0.93 & 0.93 \\
**severe\_toxic** & 0.9943 & 0.9894 & 0.96 & 0.95 & 0.96 & 0.96 \\
**obscene** & 0.9609 & 0.9644 & 0.95 & 0.96 & 0.96 & 0.95 \\
**threat** & 0.9970 & 0.9971 & 0.96 & 0.96 & 0.97 & 0.96 \\
**insult** & 0.9674 & 0.9641 & 0.96 & 0.96 & 0.96 & 0.96 \\
**identity\_hate** & 0.9919 & 0.9914 & 0.97 & 0.96 & 0.96 & 0.96 \\ \hline
**averaged** & 0.9725 & 0.9741 & 0.96 & 0.96 & 0.96 & 0.95 \\ \hline \hline \end{tabular}
\end{table}