# Towards Reliable Baselines for Document-Level Sentiment Analysis in the Czech and Slovak Languages
### Challenges  
Apparently, the most straightforward way to identify opinionated texts is through the presence of sentiment words, also called opinion words. These consist mainly of adjectives and adverbs (good, bad, well, poorly, etc.), but also some nouns, verbs, phrases and idioms (disaster, to fail, to cost someone an arm and a leg, etc.). However, as [PERSON] [6] notes, sentiment lexicons (lists of sentiment words along with their positive or negative polarities) cannot be a panacea. One reason is that certain sentiment words may acquire opposite orientations in different contexts. For example, [PERSON] [2] observes that the word \"unpredictable\" could be positive regarding a movie plot but negative regarding a car's steering capabilities, while [PERSON] and [PERSON] [8] mention the phrase \"Go read the book,\" which might convey a positive sentiment in a book review but a negative one in a movie review. Moreover, as [PERSON] and [PERSON] [12] point out, this polarity flip can happen even within one domain--compare for instance the polarity of the word \"cold\" in \"cold beer\" with that in \"cold pizza\".  
Another reason limiting the utility of sentiment lexicons is that some sentences containing sentiment words may not express any sentiment at all--typically questions and conditionals, for example, \"Can you tell me which Sony camera is good?\" or \"If I can find a good camera in the shop, I will buy it\". This, however, is not universal; for example, the question \"Does anyone know how to repair this terrible printer?\" clearly carries a negative attitude toward the printer [7]. Then there are sarcastic sentences, which actually mean the opposite of what they appear to say on the surface, as well as seemingly objective sentences without any sentiment words, but which nevertheless carry an evaluative attitude such as the one about the new car that broke down after two weeks.  
[PERSON] and [PERSON] [13] also mention a few other issues affecting sentiment analysis, especially when both fine-grained ratings (e.g., stars) and evaluative texts (reviews) are assigned to opinion targets. These problems include:  
1. Individual reviewer inconsistency, e.g., when the same reviewer associates their own very similarly worded reviews with different star ratings;
2. Lack of inter-reviewer calibration, e.g., high praise by an understated author can be interpreted as a lukewarm or neutral reaction by others;
3. Ratings not entirely supported by the review text, as when the review mentions only some partial aspects or details, while the rating captures the overall impression of the target, including aspects not explicitly mentioned in the text.  
As a result, there is considerable uncertainty inherent in the very nature of sentiment analysis, which even the human mind cannot entirely overcome. Human interrater agreement studies can therefore be taken as providing upper bounds on the accuracy that can be reasonably expected of automated sentiment analysis methods. It turns out that human interrater agreement can be surprisingly low, especially for fine-grained sentiments, as [PERSON] et al. [14] observed. Studying consistency of human ratings of Apple iTunes applications on the scale from one to four stars, the exact interrater agreement among their three human raters was below 41%. To account for subtle variations in human interpretation, they proposed to consider human ratings to be in agreement if they varied by at most one level of rating. This \"adjacent interrater agreement\" then approached 89.5%, which showed that after all, the three raters could still be considered fairly consistent. In a similar context, [PERSON] et al. [15] report the levels of exact interrater agreement between two human judges distinguishing three levels of sentiment (positive, neutral, negative) to vary between 79% and 90%. It is important to be aware of these limits when forming expectations or evaluating the performance of automated methods.