SlowGuess/NAACL_2021_OCR huggingface dataset:
Karen S. Martins
CS Department UFMG
Belo Horizonte, MG, Brazil
karensm@dcc.ufg.m.br
Pedro O.S. Vaz-de-Melo
UFMG
olmo@dcc.ufmg.br
Rodrygo L. T. Santos
rodrygo@dcc.ufgm.br
Machine learning solutions are often criticized for the lack of explanation of their successes and failures. Understanding which instances are misclassified and why is essential to improve the learning process. This work helps to fill this gap by proposing a methodology to characterize, quantify and measure the impact of hard instances in the task of polarity classification of movie reviews. We characterize such instances into two categories: neutrality, where the text does not convey a clear polarity, and discrepancy, where the polarity of the text is the opposite of its true rating. We quantify the number of hard instances in polarity classification of movie reviews and provide empirical evidence about the need to pay attention to such problematic instances, as they are much harder to classify, for both machine and human classifiers. To the best of our knowledge, this is the first systematic analysis of the impact of hard instances in polarity detection from well-formed textual reviews.
Document-level polarity classification is the task of classifying the polarity of a whole opinionated message (Pozzi et al., 2016). For instance, given a movie review, the system determines whether the review text expresses an overall positive, negative, or neutral opinion about the movie. Although polarity classification naturally suits to analyze consumer opinions about products and services (Gui et al., 2017), it is also well suited to various types of applications, such as to infer votes in elections (Goldberg et al., 2007), civilian sentiment during terrorism scenarios (Cheong and Lee, 2011), citizens' perception of government agencies (Arunachalam and Sarkar, 2013) and recommendation systems (Zhang, 2015).
Supervised machine learning is one of the most common and successful approaches for polarity classification, but even state-of-the-art methods fail
to correctly classify a substantial portion of the instances, from  $10\%$  to  $20\%$ , depending on the dataset (Ribeiro et al., 2016). The problem with this approach is that if the data is not representative and reliable, the model is unlikely to perform well. One source of unreliability is data noise, which can be categorized into class noise and attribute noise (Gupta and Gupta, 2019). Class noise occurs when the training data contains instances that are wrongly labeled. Attribute noise occurs when the training data contains one or more attributes with wrong, incomplete or missing values. In the case of textual data, such noise usually comes in the form of errors in language rules, such as typos, grammatical errors, improper punctuation, and abbreviations (Agarwal et al., 2007; Michel and Neubig, 2018; Lourentzou et al., 2019). Nevertheless, for both cases, the noise can be eliminated from the data by correcting the labels (for class noise) or the problematic text (for attribute noise).
A more problematic source of data unreliability in polarity classification tasks comes from well written text that, for some reason, does not convey its class clearly. Literature calls such instances hard instances, which are those that are intrinsically hard to correctly label or classify (Smith and Martinez, 2011; Beigman Klebanov and Beigman, 2014). Differently from noisy instances, hard instances cannot be corrected, so the only solution is to identify and remove them from the training data. Also, hard instances are not equivalent to outliers, as they do not differ significantly from other observations and may represent a significant portion of the data (Smith et al., 2014). For example, in a polarity classification task, a positive movie review that describes at least as many negative as positive points of the film can be a hard instance. To the best of our knowledge, no study exists that characterizes such instances and quantifies their impact on document-level polarity classification tasks.
Thus, we propose a methodology to characterize,
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1782-1794 June 6-11, 2021. ©2021 Association for Computational Linguistics
quantify and measure the impact of hard instances in polarity classification tasks and demonstrate its usefulness in the task of movie review polarity classification. To this end, we collected 415,867 positive and negative movie reviews from Metacritic. One advantage of Metacritic is that the meaning of ratings is clearly stated to the users when a review is being submitted: positive ratings range between  $61\%$  and  $100\%$ , neutral range between  $40\%$  and  $60\%$ , and negative between  $0\%$  and  $39\%$ . Because of that, class noise and biases should be rare, that is, a user who liked (disliked) a movie will very unlikely give a negative (positive) rating to it. Thus, classification errors will mostly be due to hard instances, which we assign into two disjoint categories: neutral and discrepant. A neutral review does not have a clear polarity and a discrepant review has a human-perceived polarity that is different from its associated rating. This categorization is complete, i.e., every instance that, for a human, does not reveal its class clearly falls into one (and only one) of these two types of hard instances.
Neutral and discrepant reviews are characterized by a well-defined human classifier that uses human reasoning to infer the class of the example. When the class assigned by the human classifier is incorrect, we label the review as discrepant, i.e., the human-perceived polarity of the text is different from its associated rating. When the human classifier is not confident about its prediction, we label the review as neutral. We labeled 1,200 reviews and found 198 neutral and 64 discrepant reviews. We tested state-of-the-art machine classifiers on these reviews and results revealed that hard instances can significantly decrease their performances. In short, the main contributions are:
- A simple and reproducible methodology based on a well-defined human classifier to characterize and identify hard instances on polarity classification tasks (Section 3);
- A thorough analysis of the impact of hard instances in the task of movie review polarity classification (Section 5.2);
Publicly available datasets of movie reviews describing the expected amounts of five classes of hard instances (Section 5.1).
As an additional contribution, we show how far are state-of-the-art machine classifiers from human performance in the task of movie review polarity classification.
In supervised machine learning, class and attribute noise can increase learning complexity and, consequently, reduce classification accuracy (Zhu and Wu, 2004). Class noise is considered to be more harmful than attribute noise (Frenay and Verleysen, 2014), but it is easier to detect (Van Hulse et al., 2007). Thus, class noise is more often addressed in the literature (Gupta and Gupta, 2019), where several studies analyzed its impact in classification tasks and how to address it (Natarajan et al., 2013; Hendrycks et al., 2018; Liu et al., 2017; Rehbein and Ruppenhofer, 2017; Jindal et al., 2019). In NLP, attribute noise are unintended errors in text, which can come from failures in automatic character recognition processes (Vinciarelli, 2005) or naturally while writing the text in the form of errors in language rules, such as typos, grammatical errors, improper punctuation, irrational capitalization and abbreviations (Agarwal et al., 2007; Contractor et al., 2010; Dey and Haque, 2009; Florian et al., 2010; Michel and Neubig, 2018). In short, noise are unintentional and undesirable errors in the text that can (and should) be eliminated from the data.
Conversely, hard instances are noise-free and cannot be corrected, only eliminated from the data (Smith and Martinez, 2011). In addition, they differ from outliers because their feature representation vectors may be similar to others from regular instances (Smith and Martinez, 2011). Nevertheless, hard instances are more prone to class noise. In fact, Beigman Klebanov and Beigman (2009) defined hard instances in the context of label annotations, under the assumption that items that are easy are reliably annotated, whereas items that are hard display confusion and disagreement among the annotators. Later, Beigman Klebanov and Beigman (2014) showed that the presence of hard instances in the training data misleads the machine learner on easy, clear-cut cases. The definition of Smith et al. (2014) is similar to ours: hard instances are simply those that "should be misclassified" by machine learning methods. The authors introduced hardness measures based on the outputs of an ensemble of classifiers to identify such instances and showed that classifiers are often uncertain about their classes. Following the same idea, Krymolowski (2002) argues that easy instances are correctly classified by all or most classifiers. On the other hand, hard instances are missed by most of them.
In this work, we propose a human classifier composed by human annotators to identify hard instances. Our definition unifies the ones of Beigman Klebanov and Beigman (2009, 2014) and Smith et al. (2014). Similarly to Beigman Klebanov and Beigman (2009, 2014), we define hard instances as those in which the human classifier is uncertain or wrong about their true labels. However, different from these studies, which quantify the impact hard instances have on training, our goal is to provide a methodology to quantify the expected amount of hard instances in data and the impact they have on classifiers in production and testing. Also, and similarly to Smith et al. (2014), hard instances are divided into "instances that should be misclassified", which we call discrepant, and "border points", which we call neutral. To the best of our knowledge, we are the first to propose a methodology to characterize and quantify the impact of hard instances in unstructured textual data for polarity classification tasks.
Regarding the effect of hard instances in sentiment and polarity classification tasks, Birmingham and Smeaton (2010) showed that it is easier to classify sentiment in short documents (e.g. tweets) than in longer ones, as short documents have less non-relevant information. Also, Valdivia et al. (2019) showed that ratings in TripAdvisor reviews are not strongly correlated with sentiment scores given by sentiment analysis methods and proposed a unified index that aggregates both polarities. Barnes et al. (2019) collected a subset of sentences that an ensemble of state-of-the-art sentiment classifiers misclassified and annotated them for 18 linguistic and paralinguistic phenomena, such as negation, sarcasm, among others. In our work, we analyze manually identified hard instances (as opposed to instances misclassified by a machine classifier). As a result, compared to these works, we have a more precise (e.g., a misclassified instance is not necessarily hard) and complete (e.g., not all hard instances are misclassified) ground-truth.
Problem Setting. In this work, we focus on the problem of polarity detection of movie reviews, but all the methods can be applied to any document-level polarity classification task. More formally, in a dataset  $\mathcal{D} = (X,Y)$  composed by a set of textual movie reviews  $X$  and their corresponding binary ratings  $Y$ , each review  $x_{i}\in X$  is associated
with a score (or rating)  $y_{i} \in Y$  that can be either 0 (positive) or 1 (negative). For the aims of this paper, it is important that  $\mathcal{D}$  does not contain any movie reviews that have been explicitly associated with a neutral score by their author, e.g. a score of 50 on Metacritic. By doing this, we isolate hard instances from explicit neutral reviews, avoiding class noise and biases.
Our methodology is composed by a human classifier  $f_{H}$ , which identifies hard instances, and a machine classifier  $f_{M}$ , which is tested on hard and regular instances. A classifier is defined as a function  $f(x_{i})$  that receives a textual movie review  $x_{i}$  as input and returns its polarity  $\hat{y}_i \in \{0,1\}$ . We use the human classifier to assign a label  $l_{i}$  to a large sample of movie reviews  $x_{i}$  to indicate whether  $x_{i}$  is a hard instance or not. This label can be one (and only one) of a set  $L$  of manually defined labels that indicate that the instance is regular or a type of hard instance. With that, we will be able to quantify the impact of hard instances on machine classifiers and provide explanations about why they occur and how to avoid them in order to improve machine classifiers' accuracy. More specifically, for a machine classifier  $f_{M}$  and for all labels  $l \in L$ , regular included, we will calculate the probabilities  $P(l_i = l | y_i \neq \hat{y}_i)$  and  $P(y_i = \hat{y}_i | l_i = l)$ .
Types of Hard Instances. A strong premise of this work is that the dataset  $\mathcal{D}$  has no (or negligible) class noise, i.e., all polarity scores  $y_{i} \in Y$  reflect the real opinion of the reviewer. To guarantee that, one needs to construct  $\mathcal{D}$  using movie reviews from systems like Metacritic or Rotten Tomatoes, which have well defined meanings for the scores, which are always visible to the reviewers. Thus, every time the polarity of text  $x_{i}$  is inconsistent with its true score  $y_{i}$ , we assume that  $x_{i}$  is a hard instance. More specifically, we define two possible hypotheses explaining the hardness of the text  $x_{i}$ , i.e., two disjoint types of hard instances: (1) the text does not have a clear polarity, namely neutrality, and (2) the text has a clear polarity, but its score  $y_{i}$  is the opposite one, namely discrepancy.
A movie review  $x_{i}$  is a hard instance of type neutrality when its polarity is not clear. We define three labels for neutral hard instances: mixed (text has mixed opinions), factual (text is purely factual) and contextual (polarity needs context). The mixed label considers reviews that describes both positive and negative points about the movie without having the overall opinion clearly stated. One real exam
ple is: "as dumb as the film is, the actors escape relatively unscathed." The factual label defines non-opinionated reviews that describes only facts about the movie, such as: "it is a movie about the World War II and its consequences on the lives of those who survived." The label contextual characterizes reviews where context is needed to understand its polarity, including those containing irony and sarcasm. One real example is: "ultimately, Collin's film is one of forgiveness and that's not the usual way great tragedies end." Finally, the label hard Undefined is given to reviews where the reasons for the lack of polarity are not clear.
The second type of hard instance, namely discrepancy, is given to reviews where the polarity of its text  $x_{i}$  is the opposite of the polarity of its score  $y_{i}$ . For this type, we define a single label: discrepancy (polarity of text and score are discrepant). As an example, consider a highly acclaimed movie of a prestigious director, such as Martin Scorsese. Now, consider a reviewer who liked this movie, but unlike the vast majority of critics, found many points that prevent her from giving it a perfect score. Thus, the text will mostly be about its negative points to justify why she is not giving the expected perfect score. Consequently, the text review will appear negative although the score is positive. The following textual review has a clear negative polarity although its score  $y_{i}$  is positive: "Thoroughly predictable from start to finish." For more examples, see Tables 4 and 5 in the Appendix.
Human Classifier. A fundamental building block of our methodology is the human classifier  $f_{H}$ . Human classifiers are often considered to be the upper bound in terms of performance of classification tasks (Stallkamp et al., 2012; Ciresan et al., 2012; Geirhos et al., 2018), which means that when it makes a prediction error, machine classifiers will most likely also miss. Moreover, when a human classifier working on its full capacity makes a mistake, and the class label is correct (i.e. no class noise), then what caused the error is most likely a hard instance (Beigman Klebanov and Beigman, 2014). We use this premise to define the two types of hard instances discussed in the previous section.
In the task of polarity classification of movie reviews, a human classifier mistake can be due to two causes: (C1) the text of the review  $x_{i}$  is not clear about its polarity  $y_{i}$ , or (C2) the score  $y_{i}$  is different from the (clearly) perceived polarity of  $x_{i}$ . In other words, the human classifier  $f_{H}$  can
be characterized by two binary features when executing this task: whether it is confident about its prediction (F1) and whether it correctly classified the polarity of the review  $x_{i}$  (F2). Thus, when it makes a mistake, if it was not confident, an error of type C1 occurs, and when it was confident, an error of type C2 occurs. The first one (C1) is associated with a hard instance of type neutrality, whereas the second one (C2) is associated with a hard instance of type discrepancy. Also, while the second only occurs when the human classifier  $f_{H}$  makes a mistake, the first occurs every time  $f_{H}$  is not confident, i.e., it is independent of the prediction  $\hat{y}_{i}$ .
With the aforementioned rationale, we are ready to propose a well-defined human classifier  $f_{H}$  to identify hard instances in movie reviews. First, and in order to construct a robust classifier,  $f_{H}$  is an ensemble composed by three independent human classifiers  $f_{h1}, f_{h2}$  and  $f_{h3}$ . In other words, we will use three annotators to label a movie review  $x_{i}$  in terms of its polarity and hardness<sup>1</sup>. Each annotator  $j \in \{1, 2, 3\}$  is asked to classify the reviews in two levels. First, they are asked to make a prediction  $\hat{y}_{i}^{j}$ , i.e., to classify the polarity of the review  $x_{i}$  as positive or negative. Second, they are asked to indicate whether they are confident or not about their classification  $\hat{y}_{i}^{j}$ . We denote the confidence of annotator  $j$  on review  $x_{i}$  by  $c_{i}^{j} \in \{0, 1\}$ , where  $c_{i}^{j} = 1$  if  $j$  is confident and  $c_{i}^{j} = 0$  otherwise. If  $c_{i}^{j} = 0$ , then we assume that  $x_{i}$  does not contain sufficient information for  $j$  to infer its polarity, that is,  $x_{i}$  is a hard instance of type neutrality. So, annotator  $j$  is asked to choose one label  $l_{i}^{j}$  that fits best to the neutrality of  $x_{i}$ , which can be either mixed, factual or contextual. On the other hand, if  $c_{i}^{j} = 1$ , then  $l_{i}^{j}$  is set to regular. This process is illustrated in Figure 1. Of course, each annotator  $j$  is independent and cannot see the others' responses.
At the end of this process, for each instance  $x_{i}$  we will have three annotation triples  $(\hat{y}_i^j, c_i^j, l_i^j)$ , where  $\hat{y}_i^j \in \{0,1\}$  (positive or negative),  $c_i^j \in \{0,1\}$  (not confident or confident) and  $l_i^j \in \mathcal{L}_N = \{mixed, factual, contextual, regular\}$ . Assuming that all annotators are equally skilled, we aggregate these annotations using majority voting to set the outputs of our human classifier  $f_{H}$ . For the polarity  $\hat{y}_i$  and the confidence  $c_i$ , the aggregation is straightforward, as described in Equations 1 and 2:
In practice, any number of annotators can be used, including just one.
Figure 1: Confidence diagram.
$$
\hat {y} _ {i} = \left\{ \begin{array}{l l} 0 & \text {i f} \sum_ {j = 1} ^ {3} \hat {y} _ {i} ^ {j} \leq 1 \\ 1, & \text {o t h e r w i s e ,} \end{array} \right. \tag {1}
$$
$$
c _ {i} = \left\{ \begin{array}{l l} 0 & \text {i f} \sum_ {j = 1} ^ {3} c _ {i} ^ {j} \leq 1 \\ 1, & \text {o t h e r w i s e .} \end{array} \right. \tag {2}
$$
Setting the final hard instance label  $l_{i}$  of review  $x_{i}$  is more involved. Let  $\mathcal{L}_i = [l_i^1, l_i^2, l_i^3]$  be the list of labels  $l_{i}^{j}$  given by the annotators to review  $x_{i}$  (e.g.  $\mathcal{L}_1 = [\text{mixed}, \text{mixed}, \text{regular}]$ ) and  $N(l, \mathcal{L}_i)$  the number of elements of  $\mathcal{L}_i$  that are equal to label  $l$  (e.g.  $N(\text{mixed}, \mathcal{L}_1) = 2$ ). Then,  $l_{i}$  is the majority vote if at least two annotators (the majority) gave that label to  $x_{i}$  and, if not,  $l_{i}$  is set to hard Undefined, indicating no consensus. This process is formally described by Equation 3:
$$
l _ {i} = \left\{ \begin{array}{l l} \arg \max  _ {l \in \mathcal {L} _ {N}} N (l, \mathcal {L} _ {i}) & \text {i f} N (l, \mathcal {L} _ {i}) \geq 2 \\ h a r d \_ u n d e f i n e d, & \text {o t h e r w i s e .} \end{array} \right. \tag {3}
$$
Finally, when the human classifier is confident about its classification of  $x_{i}$  ( $c_{i} = 1$ ), but it makes a mistake ( $\hat{y}_{i} \neq y_{i}$ ), we update the label  $l_{i}$  of  $x_{i}$  to discrepant. It is easy to see that this update step will be executed only if  $l_{i}$  was previously set to regular, i.e., it will not overwrite a neutrality label. Equation 4 defines the discrepancy update step:
$$
l _ {i} = \text {d i s c r e p a n t} \quad \mathbf {i f} \hat {y} _ {i} \neq y _ {i} \quad \mathbf {a n d} c _ {i} = 1. \tag {4}
$$
Data Set. We collected movie reviews from Metacritic, which can be authored by regular users and experts, i.e., people working in the movie industry or important communication channels (e.g. The New York Times). In case of experts, the review provided by Metacritic is actually a short summary of the original review and, as we show in Section 5,
this can be a problem for polarity classifiers. Also, each experts review is associated with a score ranging from 0 to 100, where scores from 0 to 39 are negative, from 40 to 60 are neutral, and from 61 to 100 are positive. Differently, regular users reviews are produced by any person that has an account and are associated with a score ranging from 0 to 10, where scores between 0 and 3 are negative, between 4 and 6 are neutral, and over 7 are positive. As previously mentioned, the meaning of each rating is clearly conveyed to users in the Metacritic website. Thus, class noise and biases should be rare in the dataset.
In total, we collected 415,867 reviews for 8,170 different movies, where 227,348 of those are from regular users and 188,519 from experts. Our data collection was executed using the following steps. First, we collected the most popular experts from the website, as provided by Metacritic. Then, we generated a list of all movies reviewed by the top 10 experts. From this list, which contains 8,170 movies, we collected all reviews from experts and regular users that were posted until August, 2018. For the purpose of this work, we avoided reviews that do not have a clear polarity (neutral reviews), i.e., we only considered positive and negative reviews. Hence, we selected a clean and unambiguous dataset. Reviews from experts are usually shorter than from regular users, containing an average of 26 words (std. dev. of 13) against an average of 100 words (std. dev. of 129) for reviews by regular users. In addition, we observed that experts use a more elaborate language. Because of these differences, we will condition our analyses on the type of user (experts or regular users) and score polarity (positive or negative).