Question: Is the F-Measure always better than the Baseline?
Evidence: Table 5 shows the results of our multi-class classification. For each of the classes, we have computed the ROC Area and F-Measure metrics BIBREF19 . The latter, in particular, is usually considered a significant metric in terms of classification, since it combines in one single value all the four indicators that are generally implied for evaluating the classifier performance (i.e., number of True Positives, False Positives, True Negatives and False Negatives). In our scenario, the meaning of the indicators, for each class, are as follows:
True Positives are the articles classified as belonging to a certain class, that indeed belong to that class (according to the quality ratings given by the WikiProject Medicine);
True Negatives are the articles classified as not belonging to a certain class, that indeed do not belong to that class;
False Positives are the articles classified as belonging to a certain class, that do not belong to that class;
False Negatives are the articles classified as not belonging to a certain class, that instead belong to that class.
At a first glance, we observe that, across all the models, the articles with the lowest classification values, for both ROC and F-Measure, are those labeled C and GA. Adding the Domain Informativeness feature produces a classification, which is slightly worse for C and FA articles, but better for the other four classes. This is particularly evident for the F-Measure of the articles of the GA class. A noticeable major improvement is obtained with the introduction of the features InfoBoxNormSize and Category in the Medical Domain model. The ROC Area increases for the articles of all the classes within the Full Medical Domain, while the F-Measure is always better than the Baseline and almost always better than the Medical Domain.
The size of an article, expressed either as the word count, analyzed in BIBREF20 , or as the article length, as done here, appears a very strong feature, able to discriminate the articles belonging to the highest and lowest quality classes. This is testified also by the results achieved exploiting the baseline model of BIBREF7 , which poorly succeeds in discriminating the articles of the intermediate quality classes, while achieving good results for Stub and FA. Here, the newly introduced features have a predominant effect on the articles of the highest classes. This could be justified by the fact that those articles contain, on average, more text and, then, NLP-based features can exploit more words belonging to a specific domain.
Then, we observe that the ROC Area and the F-Measure are not tightly coupled (namely: high values for the first metric can correspond to low values for the second one, see for example C and GA): this is due to the nature of the ROC Area, that is affected by the different sizes of the considered classes. As an example, we can observe that the baseline model has the same ROC Area value for the articles of both class B and class GA, while the F-Measure of articles of class B is 0.282 higher than that of class GA.
Finally, the results confirm that the adoption of domain-based features and, in general, of features that leverage NLP, help to distinguish between articles in the lowest classes and articles in the highest classes, as highlighted in bold in Table 5 . We notice also that exploiting the full medical domain leads us to the achievement of the best results.
Even if preliminary, we believe that the results are promising and call both for features' further refinement and novel features, able to discriminate among the intermediate classes too.
The ROC Area increases for the articles of all the classes within the Full Medical Domain, while the F-Measure is always better than the Baseline and almost always better than the Medical Domain.
Answer: Yes, it is.