# Towards Reliable Baselines for Document-Level Sentiment Analysis in the Czech and Slovak Languages
## 2 Related Work  
In this section we concern ourselves only with sentiment analysis in Czech and Slovak. An extensive treatment of sentiment analysis in Czech is provided in the comprehensive monograph by [PERSON] [51]. A shorter, but still very useful and readable introduction can be found in [PERSON] [52]. Since both are freely available online, we shall only sketch a brief overview of the field here, focusing mainly on areas of relevance to our present work. Soon after the explosion of interest in sentiment analysis in English in the early 2000s, initial visions were formulated and subsequent attempts made to leverage or transfer its techniques to other languages as well, often relying on existing parallel corpuses and various general NLP approaches such as machine translation. Examples in Czech include, e.g., [53; 54; 55]. Since around 2010 several authors, e.g., [PERSON] et al. [56] and [PERSON] [57], undertook the creation of manually tagged datasets specifically for the purposes of sentiment analysis in Czech. With some justification it could be claimed that this line of work reached its acme in the work of [PERSON] et al. [50], an extended version of their earlier conference paper [58], which represents the first systematic and in-depth evaluation of data pre-processing and classical machine learning methods for sentiment analysis of Czech social media. The authors first collected and made public three datasets (Facebook posts, movie reviews and product reviews; all freely available at [[http://liks.fav.zcu.cz/sentiment/](http://liks.fav.zcu.cz/sentiment/)]([http://liks.fav.zcu.cz/sentiment/](http://liks.fav.zcu.cz/sentiment/)) (accessed on 17 October 2022)), outstripping earlier such efforts by an order of magnitude, and then used them to evaluate the utility of several data pre-processing techniques, different types of features and feature selection methodsto sentiment analysis in Czech. To the best of our knowledge, this study is still unrivalled in its breadth and the level of detail with which it covered the field as it existed at that point in time. Its results (which we describe in more detail in Section 3) established a kind of informal baseline against which subsequent attempts at further improvement could be (and often were) evaluated. These subsequent attempts proceeded along various directions, from engineering new types of features [59], to considering wider context beyond the individual review being analyzed [60; 61]. Not long afterwards, initial attempts to apply neural networks to sentiment analysis in Czech appeared, such as [PERSON] and [PERSON] [62], [PERSON] et al. [63] or [PERSON] et al. [64]. Somewhat surprisingly, in no case were the employed neural architectures able to significantly outperform the established classical supervised ML models, such as Maximum Entropy. Moreover, a later attempt by [PERSON] and [PERSON] [65] to apply an even deeper neural network failed due to overfitting, indicating that the restricted size of available training corpuses for sentiment analysis in Czech did not permit truly deep neural networks to utilize their full potential. Consequently, classical supervised machine learning remained a legitimate part of the state of the art in sentiment analysis in Czech until around 2020, when first Czech variants of BERT and RoBERTa models [42; 43; 44] eventually broke the impasse and established themselves as the new \"state of the art\". These new models are universal and rely on truly deep neural architectures pre-trained in an unsupervised manner on much larger Czech corpuses, typically obtained from the web without any need for manual or semi-automatic annotation and tagging. These models can then be adapted for sentiment analysis by additional fine-tuning with the help of much smaller training sets, such as the manually annotated ones currently available for sentiment analysis in Czech.