# Towards Reliable Baselines for Document-Level Sentiment Analysis in the Czech and Slovak Languages
## 4 Replication Experiments and Their Results
* Is the Chi-Squared metric really so detrimental to feature extraction and so different from other metrics? We first performed these replication experiments on the original datasets with duplicates so that our results were directly comparable to those reported in [PERSON] et al. [50]. We then repeated the experiments on de-duplicated versions of the datasets in order to evaluate the difference. (Please note that we only de-duplicated \"nontrivial\" reviews longer than10 words, giving the benefit of doubt to shorter duplicates.) Given our focus on product reviews, we first tried the data pre-processing pipeline termed \"Lo\" in [50] because it ranked among the best for product reviews (see Table 6, last row in [50]). Eventually, however, we opted for \"ShCl\" pipeline because it seemed to give us slightly better overall results. This \"ShCl\" pipeline consists of stemming with HPS stemmer on top of mandatory tokenization, POS tagging, and stop-word removal. Given that we use different tools (Weka, Matlab, Java) than [50], our attempt at reconstructing their operational environment is not perfect, but the differences are minor and could not count as plausible explanations of major discrepancies that we observed \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Macro-F1 Results Category:** & **Facebook (\%)** & \
\begin{tabular}{c} **Product Reviews** \\ **MalLcz (\%)** \\ \end{tabular} &
\begin{tabular}{c} **Movie Reviews** \\ **csfd.cz (\%)** \\ \end{tabular} \\ \hline Best result in [50] regardless of configuration & 69 & 75.30 & 78.50 \\ \hline Result in [50] for “ShCl” configuration using all word & 66 & 74.02 & 78.21 \\ \multicolumn{2}{c}{unigrams and bigrams} & & & \\ \hline Our replicated result for & & & & & \\ “ShCl” configuration using all unigrams and bigrams & 64.77 & 76.2 & 78.9 \\ \hline Our result for “ShCl” configuration using all unigrams and bigrams & & & & & \\ \hline Our result for “ShCl” configuration using all unigrams and bigrams & 66.4 & 57.07 & 78.26 \\ \multicolumn{2}{c}{AFTER DEDUPLICATION} & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Macro-F1 results of our replication experiments compared with those of [PERSON] et al. [50] for classification into three classes by the Maximum Entropy algorithm. \begin{table}
\begin{tabular}{l c c c c c c c c c c c c c} \hline \hline  & & & & & & & & & & & & & & \\ \hline
**Number of copies** & 20 & 15 & 12 & 11 & 10 & 9 & 8 & 7 & 6 & 5 & 4 & 3 & 2 & 1 \\ \hline
**Number of distinct** & 1 & 1 & 2 & 2 & 2 & 8 & 15 & 23 & 55 & 216 & 854 & 3427 & 7126 & 26,101 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Statistical distribution of the number of verbatim copies among nontrivial positive product reviews from Mall.cz. \begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & & & & & & & & & & & & & \\ \hline
**Number of copies** & 20 & 15 & 12 & 11 & 10 & 9 & 8 & 7 & 6 & 5 & 4 & 3 & 2 & 1 \\ \hline
**Number of distinct** & 1 & 1 & 2 & 2 & 2 & 8 & 15 & 23 & 55 & 216 & 854 & 3427 & 7126 & 26,101 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Statistical distribution of the number of verbatim copies among nontrivial negative product reviews from Mall.cz. \begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & & & & & & & & & & & & \\ \hline
**Number of copies** & 27 & 8 & 7 & 6 & 5 & 4 & 3 & 2 & 1 \\ \hline
**Number of distinct** & 1 & 1 & 3 & 12 & 23 & 120 & 396 & 1072 & 4486 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Statistical distribution of the number of verbatim copies among nontrivial neutral product reviews from Mall.cz. in the following replication experiments. As other authors before us, we also need to point out that since no canonical split into folds was provided by the creators of these datasets, we cannot recreate the exact assignment into folds that they used. Different assignments of records into folds can easily cause a difference of several percentage points in the macro-F1 measure.