# Towards Reliable Baselines for Document-Level Sentiment Analysis in the Czech and Slovak Languages
## 4 Replication Experiments and Their Results  
When we first came across the work of [PERSON] et al. [50], our intention was to use it as a baseline which we would try to improve upon. At that time, we were trying to collect our own dataset of online product reviews in Slovak, and because of the similarity of Slovak to Czech, we expected that we would be able to match [PERSON] at al. 's [50] macro-F1 results for online product reviews quite easily with our selection of machine learning algorithms and Slovak data. Unfortunately, however hard we tried our results kept lagging behind theirs by more than ten percentage points. Ultimately, we were forced to accept the fact that to solve this mystery we would need to replicate [PERSON] et al. 's [50] experiments on their datasets, using both our and their algorithms, and thoroughly analyze any significant differences in the results. It turned out that with their datasets, our algorithms gave comparable results to theirs. In this way, the mystery was narrowed down to the difference between the two datasets: why should a dataset of Slovak online product reviews be so much more difficult for machine learning than the Czech one? Slovak and Czech are so close that their minor differences did not seem to be a plausible explanation at all. Finally, after a long and timing investigation of various alternatives, our lead author ([PERSON]) came up with a surprising answer: because the Czech product review dataset from Mall.cz contained a lot of duplicate records! Of course, it is quite conceivable that different customers may use the same short judgement like \"excellent product\" repeatedly for different products, but it is very unlikely that longer reviews could be thus reproduced in significant quantities accidentally. Long duplicates are therefore much more likely to be the result of either a deliberate spamming or an inadvertent duplication during the data collection process. In what follows, we therefore focus only on such \"nontrivial\" reviews, which we empirically define as being longer than 10 words. To illustrate the extent of duplication among nontrivial product reviews from Mall.cz, here are some summary statistics: more than 17,500 distinct nontrivial reviews exist in two or more copies (out of which, more than 6700 have three or more copies, and more than 1800 have four or more copies). Overall, more than 18% of records in the Mall.cz product review dataset consist of such non-trivial duplicates. In Tables 3-5 we provide more-detailed information of duplicates per class (positive, negative, and neutral). These tables also reveal the absolute record-holders: there is one nontrivial negative review of which 27 verbatim copies exist in the dataset, and one positive review with 20 verbatim copies. (By way of example, searching for the string \"vyzkouzseno tyden v Rohacich\" (sic) should help identify nine verbatim copies of that particular review in the positive class of this dataset.) As far as we could see, these duplicates did not seem to exist on the Mall.cz webpage, so they appear to be an artefact of a faulty data collection process. Duplicates inflate the macro-F1 measure because random selection tends to make copies of the same review to be simultaneously present in both the training set and the test set. Interestingly, this issue significantly affected only the product reviews from Mall.cz, not the other two datasets. Although there were a few duplicates in each of them, their numbers, as well as their effect on the achieved macro-F1 measure, were minimal. The discovery of this problem emboldened us to look more inquisitively on other unexpected or otherwise anomalous results and claims in the article and test their validity as well. Overall, our replication experiments in this section cover the following issues:  
* What are the true baseline macro-F1 results for the three datasets after deduplication? * Is the alleged damaging effect of the POS-related features on product reviews real?