# Towards Reliable Baselines for Document-Level Sentiment Analysis in the Czech and Slovak Languages
## 5 Discussion and Methodological Considerations
### On the Comparison of Averaged F1 Scores across Studies  
In this section we draw on the work of [PERSON] and [PERSON] [69] in order to alert researchers to an important problem affecting cross-validation studies using the F1-measure on unbalanced datasets, which is a frequent scenario in NLP and sentiment analysis. Although the F1 measure is \"logically\" defined as a harmonic mean of precision and recall, it can also be calculated in other ways. In the case of binary classification, which is easiest to explain, it can be alternatively expressed as the function of true positives TP, false positives FP, and false negatives FN, namely F1 = 2.TP/(2.TP + FP + FN). In the context of n-fold cross-validation studies, this translates into three main ways of averaging the F1 scores across folds:  
1. The F1 score will be calculated for each fold from the precision and recall for that fold and then averaged;
2. Precision and recall will be calculated for each fold, then averaged, and, finally, the \"average\" F1 will be calculated as a harmonic mean of the averaged values of precision and recall;
3. True positives TP, false positives FP, and false negatives FN will be calculated for each fold, then averaged, and the \"average\" F1 will be calculated from their averages through the alternative formula. Due to nonlinearity inherent in the F1 definition, these three ways of averaging it are not equivalent and can give vastly different results for highly unbalanced datasets. [PERSON] and [PERSON] [69] provide a very instructive example (in their Table 1 in Section 3.1) with 1504 data rows and 1% (i.e., 15) of positives, to which they apply stratified four-fold cross-validation. Stratification means that each of the four folds will contain three or four examples of the positive class and 372 or 373 examples of the negative class. Assuming a high (but not unrealistic) variation in the model precision (from 24 to 100%) and recall (from 75 to 100%) across individual folds, the authors show that the average F1 can vary from 58 to 73% depending on which of the three ways listed above is used for calculation. It means that, for highly unbalanced datasets, even with the same fold-wise results and ground-truth reality, the estimates of average F1 can differ by more than ten percentage points purely based on how the used tools implement the F1 averaging! This situation can get even worse when the trained model happens to assign all test set records in some fold to the negative class. This is again rare but not impossible, and it forces the precision (and consequently the F1) for that fold to become undefined. It is important to note here that the third method of averaging the F1 on the basis of the alternative formula with TP, FP and FN is not affected by this problem, because the number of false negativesFN must then be greater than zero--in fact equal to the number of real positives, since they must all have been misclassified as false negatives FN. The denominator (2.TP + FP + FN) in the alternative formula will therefore remain greater than zero, causing the F1 to stay well-defined and equal to zero. In contrast, the first two methods have either to ignore such \"failing\" folds, or arbitrarily assign to them F1 = 0 and include them in the average F1. In the case of failing folds the number of different ways of calculating the average F1 thus expands to five, depending on how the \"failing\" folds are handled. Along this line, the authors provide a slightly modified example (in their Table 2) with one failing fold and show that this can lead to an even higher variation in the average F1, in this case from 67 to 91%. Unsurprisingly, ignoring the failing folds turns out to be the worst possible decision, incorrectly inflating the average F1 the most. Overall, the best method of averaging the F1 across folds--the most robust and least biased even on strongly unbalanced datasets--turns out to be the third one, using the alternative formula.