# Towards Reliable Baselines for Document-Level Sentiment Analysis in the Czech and Slovak Languages
## 5 Discussion and Methodological Considerations  
The preceding sections demonstrate the difficulties we encountered when we tried to use the results reported in [PERSON] et al. (2018) as a baseline for our own research. We at first focused on sentiment classification of Slovak online product reviews into three classes, but irrespective of what we did, our results on Slovak data kept lagging substantially behind that baseline. We eventually managed to resolve this discrepancy by the discovery and removal of large quantities of duplicates from the product review dataset used in ([PERSON] et al., 2018). These duplicates accounted for more than 18% of its records, and without deduplication, would seriously distort the performance metric results of any machine learning method. As we report in Section 4.1, after deduplication of \"nontrivial\" reviews longer than 10 words, the macro-F1 results for this dataset have dropped by more than 19 percentage points. Researchers wishing to use this dataset will, therefore, also have to de-duplicate it before analyzing it. As part of that process, they will have to decide--based on their research goal  
Figure 3: Results of our deliberately “inverted” feature selection experiments with the Maximum Entropy classifier, and both the Chi-Squared (red line) and the Information Gain metric (black line) for the full Facebook dataset with duplicates. In these experiments, we first removed the features with the highest scores. The blue line corresponds to the results of standard (not inverted) feature selection experiments with the Chi-squared metric reported by [PERSON] et al. (2018); it is reproduced here, with permission, from their Figure 5a. In all the cases, the horizontal axis is to be interpreted as showing the proportion of features removed from the feature set before measuring the macro-F1. and wider context--whether the deduplication should be strict (with only one copy of each review permitted in the dataset regardless of its length) or, as we have done in this study, whether they would tolerate the duplicates of shorter reviews as plausibly coming from different customers in response to different products. The good news in this respect was our discovery that the proportion of duplicates in the other two datasets (Facebook and movie reviews) was negligible and so was its effect on their macro-F1 results. Having thus encountered and resolved the problem with duplicates, we felt motivated to double-check other unexpected results and claims in [50] as well. Two additional noteworthy examples that we identified consisted of the claim that the POS-related features had a highly damaging effect on the macro-F1 score for product reviews (allegedly causing it to drop by more than 19 percentage points), and that the Chi-squared metric was by far the worst and wholly unsuitable for feature selection in sentiment analysis in Czech. In both cases, our replication experiments on the same data (reported in Sections 4.2 and 4.4) have shown these claims to be wrong: the POS features certainly do not harm sentiment analysis in any way (although they do not seem to help either), and the Chi-squared metric turned out to be as good as the other four metrics, performing in fact on par with the two \"winners\", the Information Gain and the Mutual Information. The main goal behind our replication experiments and error corrections is to make the baseline established by [50] more trustworthy and more widely used. We have noted, for example, that their product review dataset--in stark contrast to the other two--was rarely utilized by other researchers. We suspect that many of them have actually tried to use it, but getting strange results due to its high proportion of duplicates were eventually forced to abandon it. After proper deduplication, this dataset also can become as valuable and widely used as the other two, e.g., to reliably measure the extent of improvement brought about into NLP by the new BERT- and RoBERTa-based neural architectures.