Question: What did they calculate precision as?
Evidence: We now present the most interesting results from our experiments. For the evaluation we used standard metrics for classification accuracy, suitable for studying problems such as sentiment analysis. In particular we used Precision and Recall, with the former calculated as the ratio of the number of tweets correctly classified to a given class over the total number of tweets classified to that class, while the latter measures the ratio of messages correctly classified to a given class over the number of messages from that class. Additionally, the F-score is the harmonic mean of precision and recall, expressed as INLINEFORM0 . For our particular case with three classes, P, R and F are computed for each class separately, with the final F value derived as the weighted mean of the separate INLINEFORM1 -scores: INLINEFORM2 ; recall that INLINEFORM3 , INLINEFORM4 and INLINEFORM5 . The results are shown in Table TABREF24 , along with the reported results from state of the art approaches proposed by other researchers in the field. Note that the performance numbers P,R and F of the other state of the art approaches are based on the authors' reported data in the cited works. Additionally, we report the performance of each individual LSTM classifier as if used alone over the same data (that is, without the ensemble logic). The F-score for our proposed approaches shown in the last column, is the weighted average value over the 3 classes (Neutral,Sexism,Racism). Moreover, all the reported values are average values produced for a number of runs of the same tested scheme over the same data. Figure FIGREF23 shows the F-Score as a function of the number of training samples for each ensemble of classifiers. We clearly see that the models converge. For the final run the F-score has standard deviation value not larger than 0.001, for all classifiers.
As can be seen in Table TABREF24 , the work by BIBREF12 , in which character n-grams and gender information were used as features, obtained the quite low F-score of INLINEFORM0 . Later work by the same author BIBREF5 investigated the impact of the experience of the annotator in the performance, but still obtaining a lower F-score than ours. Furthermore, while the first part of the two step classification BIBREF16 performs quite well (reported an F-score of 0.9520), it falls short in detecting the particular class the abusive text belongs to. Finally, we observe that applying a simple LSTM classification with no use of additional features (denoted `single classifier (i)' in Table TABREF24 ), achieves an F-score that is below 0.93, something that is in line with other researchers in the field, see BIBREF15 .
Very interestingly, the incorporation of features related to user's behaviour into the classification has provided a significant increase in the performance vs. using the textual content alone, INLINEFORM0 vs. INLINEFORM1 .
Another interesting finding is the observed performance improvement by using an ensemble instead of a single classifier; some ensembles outperform the best single classifier. Furthermore, the NRS classifier, which produces the best score in relation to other single classifiers, is the one included in the best performing ensemble.
In comparison to the approach by BIBREF13 , which focuses on various classes of Sexism, the results show that our deep learning model is doing better as far as detecting Sexism in general, outperforming the FastText algorithm they include in their experiments (F=0.87). The inferiority of FastText over LSTM is also reported in the work by BIBREF15 , as well as being inferior over CNN in, BIBREF16 . In general, through our ensemble schemes is confirmed that deep learning can outperform any NLP-based approaches known so far in the task of abusive language detection.
We also present the performance of each of the tested models per class label in Table TABREF25 . Results by other researchers have not been included, as these figures are not reported in the existing literature. As can be seen, sexism is quite easy to classify in hate-speech, while racism seems to be harder; similar results were reported by BIBREF7 . This result is consistent across all ensembles.
For completion, the confusion matrices of the best performing approach that employs 3 classifiers (ensemble viii) as well as of the ensemble of the 5 classifiers (xi), are provided in Table TABREF26 . The presented values is the sum over multiple runs.
The code and results associated with this paper will be available on-line soon at: https://github.com/gpitsilis/hate-speech/
For the evaluation we used standard metrics for classification accuracy, suitable for studying problems such as sentiment analysis. In particular we used Precision and Recall, with the former calculated as the ratio of the number of tweets correctly classified to a given class over the total number of tweets classified to that class, while the latter measures the ratio of messages correctly classified to a given class over the number of messages from that class.
Answer: The ratio of the number of tweets correctly classified to a given class over the total number of tweets classified to that class.