INTRODUCTION
Sentiment Analysis is considered as the automated analysis of sentiments, emotions or opinions expressed in texts towards certain entities BIBREF0 . The proliferation of online commerce and customer feedback has significantly motivated companies to invest in intelligent text analysis tools and technologies where sentiment analysis plays a crucial role. There have traditionally been two main approaches to sentiment analysis. The first one uses unsupervised algorithms, sentiment lexicons and word similarity measures to mine emotions in raw texts. The second uses emotionally-labeled text datasets to train supervised (or deep supervised) algorithms and use them to predict emotions in other documents. Naturally, most of sentiment analysis research has been conducted for the English language. Chinese BIBREF1 , BIBREF2 , BIBREF3 and Spanish BIBREF4 , BIBREF5 have also received a considerable extra attention in the last years. Smaller languages like Czech have seen fewer efforts in this aspect. It is thus much easier to find online data resources for English than for other languages BIBREF6 . One of the first attempts to create sentiment annotated resources of Czech texts dates back in 2012 BIBREF7 . Authors released three datasets of news articles, movie reviews, and product reviews. A subsequent work consisted in creating a Czech dataset of information technology product reviews, their aspects and customers' attitudes towards those aspects BIBREF8 . This latter dataset is an essential basis for performing aspect-based sentiment analysis experiments BIBREF9 . Another available resource is a dataset of ten thousand Czech Facebook posts and the corresponding emotional labels BIBREF10 . The authors report various experimental results with Support Vector Machine (SVM) and Maximum Entropy (ME) classifiers. Despite the creation of the resources mentioned above and the results reported by the corresponding authors, there is still little evidence about the performance of various techniques and algorithms on sentiment analysis of Czech texts. In this paper, we perform an empirical survey, probing many popular supervised learning algorithms on sentiment prediction of Czech Facebook posts and product reviews. We perform document-level analysis considering the text part (that is usually short) as a single document and explore various parameters of Tf-Idf vectorizer and each classification algorithms reporting the optimal ones. According to our results, SVM (Support Vector Machine) is the best player, shortly followed by Logistic Regression (LR) and Naïve Bayes (NB). Moreover, we observe that ensemble techniques like Random Forests (RF), Adaptive Boosting (AdaBoost) or voting schemes do not increase the performance of the basic classifiers. The rest of the paper is structured as follows: Section "Czech Facebook Dataset" presents some details and statistics about the two Czech datasets we used. Section "PREPROCESSING AND VECTORIZATION" describes the text preprocessing steps and vectorizer parameters we grid-searched. Section "SUPERVISED ALGORITHMS" presents in details the grid-searched parameters and values of all classifiers. In Section "RESULTS" , we report the optimal parameter values and test scores in each dataset. Finally, Section "CONCLUSIONS" concludes and presents possible future contributions.
Czech Facebook Dataset
Czech Facebook dataset was created by collecting posts from popular Facebook pages in Czech BIBREF10 . The ten thousand records were independently revised by two annotators. Two other annotators were involved in cases of disagreement. To estimate inter-annotator agreement, they used Cohen's kappa coefficient which was about 0.66. Each post was labeled as negative, neutral or positive. There were yet a few samples that revealed both negative and positive sentiments and were marked as bipolar. Same as the authors in their paper, we removed the bipolar category from our experimental set to avoid ambiguity and used the remaining 9752 samples. A few data samples are illustrated in Figure 1 .
Mall.cz Reviews Dataset
The second dataset we use contains user reviews about household devices purchased at mall.cz BIBREF7 . The reviews are evaluative in nature (users apprising items they bought) and were categorized as negative or positive only. Some minor problems they carry are the grammatical or typing errors that frequently appear in their texts BIBREF11 . In Table 1 we present some rounded statistics about the two datasets. As we can see, Mall product reviews are slightly longer (13 vs. 10 tokens) than Czech Facebook posts. We also see that the number of data samples in each sentiment category are unbalanced in both cases. A few samples of Mall reviews are illustrated in Figure 2 .
PREPROCESSING AND VECTORIZATION
Basic preprocessing steps were applied to each text field of the records. First, any remaining markup tags were removed, and everything was lowercased. At this point, we saved all smiley patterns (e.g., :P, :), :(, :-(, :-), :D) appearing in each record. Smileys are essential features in sentiment analysis tasks and should not be lost from the further text cleaning steps. Stanford CoreNLP tokenizer was employed for tokenizing. Numbers, punctuation, and special symbols were removed. At this point, we copied back the smiley patterns to each of the text samples. No stemming or lemmatization was applied.
As vectorizer, we chose to experiment with Tf-Idf which has been proved very effective with texts since long time ago BIBREF12 , BIBREF13 . Tf-Idf gives the opportunity to work with various n-grams as features (ngramrange parameter). We limited our experiments to single words, bigrams, and trigrams only since texts are usually short in both datasets. It is also very common in such experiments to remove a subset of words known as stop words that carry little or no semantic value. In our experiments we tried with full vocabulary or removing Czech stopwords that are defined at https://pypi.org/project/stop-words/ package. Other parameters we explored are smoothidf and norm. The former adds one to document frequencies to smooth Idf weights when computing Tf-Idf score. The latter is used to normalize term vectors (None for no normalization). The parameters and the corresponding grid-searched values of Tf-Idf are listed in Table 2 .
Besides using this traditional approach based on Tf-Idf or similar vectorizers, it is also possible to analyze text by means of the more recent dense representations called word embeddings BIBREF14 , BIBREF15 . These embeddings are basically dense vectors (e.g., 300 dimensions each) that are obtained for every vocabulary word of a language when large text collections are fed to neural networks. The advantage of word embeddings over bag-of-word representation and Tf-Idf vectorizer is their lower dimensionality which is essential when working with neural networks. It still takes a lot of text data (e.g., many thousands of samples) to generate high-quality embeddings and achieve reasonable classification performance BIBREF16 . A neural network architecture for sentiment analysis based on word embeddings is described by BIBREF17 . We applied that architecture on the two Czech datasets we are using here and observed that there was severe over-fitting, even with dropout regularization. For this reason, in the next section, we report results of simpler supervised algorithms and multilayer perceptron only, omitting experiments with deeper neural networks.
SUPERVISED ALGORITHMS
We explored various supervised algorithms that have become popular in recent years and grid-searched their main parameters. Support Vector Machines have been successfully used for solving both classification and regression problems since back in the nineties when they were invented BIBREF18 , BIBREF19 . They introduced the notion of hard and soft margins (separation hyperplanes) for optimal separation of class samples. Moreover, the kernel parameter enables them to perform well even with data that are not linearly separable by transforming the feature space BIBREF20 . The C parameter is the error penalty term that tries to balance between a small margin with fewer classification errors and larger margin with more errors. The last parameter we tried is gamma that represents the kernel coefficient for rfb, poly and sigmoid (non linear) kernels. The other algorithm we tried is NuSVM which is very similar to SVM. The only difference is that a new parameter (nu) is utilized to control the number of support vectors. Random Forests (RF) were also invented in the 90s BIBREF21 , BIBREF22 . They average results of multiple decision trees (bagging) aiming for lower variance. Among the many parameters, we explored maxdepth which limits the depth of decision trees. We also grid-searched maxfeat, the maximal number of features to consider for best tree split. If sqrt is given, it will use the square root of total features. If None is given then it will use all features. Finally, nest dictates the number of trees (estimators) that will be used. Obviously, more trees may produce better results but they also increase the computation time. Logistic Regression is probably the most basic classifier that still provides reasonably good results for a wide variety of problems. It uses a logistic function to determine the probability of a value belonging to a class or not. C parameter represents the inverse of the regularization term and is important to prevent overfitting. We also explored the classweight parameter which sets weights to sample classes inversely proportional to class frequencies in the input data (for balanced). If None is given, all classes have the same weight. Finally, penalty parameter specifies the norm to use when computing the cost function. To have an idea about the performance of small and shallow neural networks on small datasets, we tried Multilayer Perceptron (MLP) classifier. It comes with a rich set of parameters such as alpha which is the regularization term, solver which is the weight optimization algorithm used during training or activation that is the function used in each neuron of hidden layers to determine its output value. The most critical parameter is layersizes that specifies the number of neurons in each hidden layer. We tried many tuples such as (10, 1), (20, 1), $\dots $ , (100, 4) where the first number is for the neurons and the second for the layer they belong to. Same as Logistic Regression, Naïve Bayes is also a very simple and popular classifier that provides high-quality solutions to many problems. It is based on Bayes theorem:
$$P(A \mid B) = \frac{P(B \mid A) \, P(A)}{P(B)}$$   (Eq. 9)
which shows a way to get the probability of A given evidence B. For Naïve Bayes, we probed alpha which is the smoothing parameter (dealing with words not in training data) and fitprior for learning (or not) class prior probabilities. The last algorithm we explored is Maximum Entropy classifier. It is a generalization of Naïve Bayes providing the possibility to use a single parameter for associating a feature with more than one label and captures the frequencies of individual joint-features. We explored four of the implementation methods that are available. All algorithms, their parameters, and the grid-searched values are summarized in Table 3 .
Optimal Parameter Values
We performed 5-fold cross-validation grid-searching in the train part of each dataset (90 % of samples). The best parameters of the vectorization and classification step for each algorithm on the Facebook data are presented in Table 4 . The corresponding results on the Mall data are presented in Table 5 . Regarding Tf-Idf vectorizer, we see that adding bigrams is fruitful in most of the cases (9 out of 14). Smoothing Idf on the other hand does not seem necessary. Regarding stop words, keeping every word (stopwords=None) gives the best results in 13 from 14 cases. Removing Czech stop words gives the best score with Random Forest on Mall data only. As for normalization, using l2 seems the best practice in most of the cases (10 out of 14).
Regarding classifier parameters, we see that SVM performs better with rbf kernel and C = 100 in both datasets. The linear kernel is the best option for NuSCV instead. In the case of Random Forest, we see that a maxdepth of 90 (the highest we tried) and sqrt maxfeat are the best options. Higher values could be even better. In the case of Logistic Regression, the only parameter that showed consistency on both dataset is penalty (l2). We also see that MLP is better trained with relu activation function and adam optimizer. Finally, the two parameters of Naïve Bayes did not show any consistency on the two datasets whereas iis was the best methods for Maximum Entropy in both of them.
Test Scores Results
We used the best performing vectorizer and classifier parameters to assess the classification performance of each algorithm in both datasets. The top grid-search accuracy, test accuracy and test macro F $_1$ scores are shown in Table 6 . For lower variance, the average of five measures is reported. The top scores on the two datasets differ a lot. That is because Facebook data classification is a multiclass discrimination problem (negative vs. neutral vs. positive), in contrast with Mall review analysis which is purely binary (negative vs. positive). As we can see, Logistic Regression and SVM are the top performers in Facebook data. NuSVM and Naïve Bayes perform slightly worse. MLP and Random Forest, on the other hand, fall discretely behind. On the Mall dataset, SVM is dominant in both accuracy and F $_1$ . It is followed by Logistic Regression, Naïve Bayes and NuSVM. Maximum Entropy is near whereas MLP and Random Forest are again considerably weaker. Similar results are also reported in other works like BIBREF23 where again, SVM and Naïve Bayes outrun Random Forest on text analysis tasks. From the top three algorithms, Naïve Bayes was the fastest to train, followed by Logistic Regression. SVM was instead considerably slower. The 91.6 % of SVM in F $_1$ score on Mall dataset is considerably higher than the 78.1 % F $_1$ score reported in Table 7 of BIBREF7 . They used Naïve Bayes with $ \alpha = 0.005 $ and 5-fold cross-validation, same as we did here. Unfortunately, no other direct comparisons with similar studies are possible.
Boosting Results
We picked SVM, Logistic Regression and Naïve Bayes with their corresponding optimal set of parameters and tried to increase their performance further using Adaptive Boosting BIBREF24 . AdaBoost is one of the popular mechanisms for reinforcing the prediction capabilities of other algorithms by combining them in a weighted way. It tries to tweak future classifiers based on the wrong predictions of the previous ones and selects only the features known to improve prediction quality. On the negative side, AdaBoost is sensitive to noisy data and outliers which means that it requires careful data preprocessing. First, we experimented with a few estimators in Adaboost and got poor results in both datasets. Increasing the number of estimators increased accuracy and F $_1$ scores until some point (about 5000 estimators) and was further useless. The detailed results are presented in Table 7 . As we can see, no improvements over the top scores of each algorithm were gained. The results we got are actually slightly lower. As a final attempt, we combined SVM, LR, and NB in a majority voting ensemble scheme. Test accuracy and F $_1$ scores on Facebook were 69.5 and 64.2 %, respectively. The corresponding results on Mall dataset were 92.1 and 91.5 %. Again, we see that the results are slightly lower than top scores of the tree algorithms and no improvement was gained on either of the datasets.
CONCLUSIONS
In this paper, we tried various supervised learning algorithms for sentiment analysis of Czech texts using two existing datasets of Facebook posts and Mall product reviews. We grid-searched various parameters of Tf-Idf vectorizer and each of the machine learning algorithms. According to our observations, best sentiment predictions are achieved when bigrams are added, Czech stop words are not removed and l2 normalization is applied in vectorization. We also reported the optimal parameter values of each explored classifier which can serve as guidelines for other researchers. The accuracy and F $_1$ scores on the test part of each dataset indicate that the best-performing algorithms are Support Vector Machine, Logistic Regression, and Naïve Bayes. Their simplicity and speed make them optimal choices for sentiment analysis of texts in cases when few thousands of sentiment-labeled data samples are available. We also observed that ensemble methods like bagging (e.g., Random Forest), boosting (e.g., AdaBoost) or even voting ensemble schemes do not add value to any of the three basic classifiers. This is probably because all Tf-Idf vectorized features are relevant and necessary for the classification process and extra combinations of their subsets are not able to further improve classification performance. Finally, as future work, we would like to create additional labeled datasets with texts of other languages and perform similar sentiment analysis experiments. That way, valuable metalinguistic insights could be drawn and reported.
ACKNOWLEDGEMENTS
The research was [partially] supported by OP RDE project No. CZ.02.2.69/0.0/0.0/16_027/0008495, International Mobility of Researchers at Charles University.

Question: Did the authors collect new data for evaluation?
Answer: No

No