SlowGuess/NAACL_2021_OCR huggingface dataset:
Machine Classifiers. To evaluate the impact of hard instances on machine classifiers, we selected three state-of-the-art models with reported success in the task of polarity detection of movie reviews: BERT (Devlin et al., 2019), CNN-GRU (Wang et al., 2016) and C-LSTM (Zhou et al., 2015). C-LSTM utilizes a Convolutional Neural Network (CNN) to extract a sequence of higher-level phrase representations, which are then fed into a Long Short-Term Memory (LSTM) unit to obtain the sentence representation. CNN-GRU connects a character-aware CNN with a character-aware Gated Recurrent Unit (GRU) to learn long sequence semantics. These two networks are initialized with pre-trained Word2vec vectors from Google News
2https://www.metacritic.com/movie
Dataset and have their final representations connected to a dense layer. BERT uses a masked language model (MLM) to pre-train deep bidirectional representations from unlabeled text that considers both the left and right context of sentences and words. In this work, we used an architecture composed by BERT embeddings pre-trained with data from Wikipedia connected with a dense layer. For all architectures, the output  $\hat{y}_i$  is given by a sigmoid function. For implementation and code details, please see the Appendix.
5.1 Number of Hard Instances
The first question we need to answer is: how many hard instances exist in movie reviews? In the context of our Metacritic dataset  $\mathcal{D}$ , the answer to this question can be influenced by two factors: (1) the type of user and (2) the polarity of their rating. Thus, the following results are conditioned on whether the authors are experts or regular users and whether the reviews are positive or negative. Because of that, we sampled a collection  $D_H$  of 800 movie reviews from  $\mathcal{D}$  that is both balanced in terms of user type and score polarity, i.e., this collection has 200 reviews for each of the four combinations of user type and score polarity.
In order to quantify the number of hard instances in  $D_H$ , we use our proposed human classifier  $f_H$  described in Section 3 to label every review  $x_i \in D_H$ . Recall that  $f_H$  assigns a polarity  $\hat{y}_i \in \{\text{positive, negative}\}$  to  $x_i$  and, more important to our purpose here, a label  $l_i$ , which can be either regular (instance is not a hard instance), discrepant (the polarity of the text is different from the score polarity), or one of the four neutrality labels: mixed (text has mixed opinions), factual (text is purely factual), contextual (polarity needs context) and hard Undefined (reasons are unclear). Also, let  $u_i \in \{\text{expert, regular user}\}$  be the user type of the author of review  $x_i$ . Our goal with the following results is to estimate the probability  $P(l_i = l \mid y_i = y, u_i = u)$  for the four combinations of score polarity  $y$  and user type  $u$ .
In Table 1, we show the number and proportion of movie reviews that are or are not hard instances for experts. From the 400 labeled reviews, almost one quarter (92) are hard instances. From those, note that neutral reviews are more common than discrepant ones, but while the first is equally present in both positive and negative reviews, dis
label (li)	positive	negative	total
experts
regular	146(36.5%)	162(40.5%)	77%
discrepant	20(5%)	3(0.8%)	5.8%
neutral	34(8.5%)	35(8.8%)	17.3%
mixed	10(2.5%)	7(1.8%)	4.3%
factual	14(3.5%)	3(0.8%)	4.3%
contextual	7(1.8%)	20(5%)	6.8%
undefined	3(0.8%)	5(1.3%)	2%
regular users
regular	177(44.3%)	187(46.8%)	91%
discrepant	3(0.8%)	2(0.5%)	1.3%
neutral	20(5%)	11(2.8%)	7.8%
mixed	16(4%)	7(1.8%)	5.8%
factual	1(0.3%)	2(0.5%)	0.8%
contextual	0(0%)	1(0.3%)	0.3%
undefined	3(0.8%)	1(0.3%)	1%
Table 1: Number of hard instances in reviews.
crepant instances are significantly more present in positive reviews. In such cases, the author gave a positive score to the movie, but its review demonstrates the opposite sentiment. This often occurs when the expert is using the review to justify a good, but far from perfect score, to a critically acclaimed movie. As for the neutral reviews, the most predominant type is contextual (6.8%), followed by mixed (4.3%) and factual (4.3%). Also, contextual instances are more common in negative reviews, when experts often use figures of speech (e.g. irony) together with external knowledge to create humour. Finally, factual instances are more present in positive reviews, where the experts simply describe some characteristic of the movie that impressed them without explicitly saying that.
Also, in Table 1 we show the number and proportion of movie reviews that are or are not hard instances for regular users. First, note that the number of reviews that are hard instances significantly decreased in comparison with the ones written by experts. From the 400 labeled reviews, only  $36(9\%)$  are hard instances, of which 31 are neutral and only 5 are discrepant. Different from what was verified for experts, the most predominant label for regular users was mixed, which occurred significantly more in positive reviews. For the other labels, their occurrences were fairly balanced between negative and positive reviews. We observed that regular users use a much more direct and simple language to state their opinions than experts. Because of that, most of the hard instances are concentrated in cases where the author lists both the negative and positive aspects of the movie without stating their final opinions about the movie, which
is the definition of mixed.
A note about the human classifier. Because we used three human annotators in  $f_{H}$  and a majority vote function, only two annotators were used initially. The third annotator was called to classify  $x_{i}$  if, and only if, the first two had any kind of disagreement, i.e., a disagreement regarding the polarity  $y_{i}$ , the confidence  $c_{i}$ , or label  $l_{i}$ . For the first two annotators, they agreed on 91.13% of the polarity scores, on 90.5% of their confidence levels and on 88% of their labels. Regarding the third annotator, only 1.5% of the instances were not in total agreement with at least one of the other annotators. The Cohen's kappa coefficient for the first two annotators was 0.82 in relation to polarity scores, 0.58 regarding their confidence levels and 0.49 regarding their attribute noise labels.
5.2 Impact of Hard Instances
In this section, we quantify the impact of hard instances in machine classifiers. Also, by putting these results in perspective with what was achieved by the human classifier, we hope to provide an accurate assessment on how distant machine classifiers are with respect to human performance. We guide our analyses by the following questions:
1. What are the probabilities of a correct and a misclassification given the label  $l$ ? In other words, we want to estimate the probabilities  $P(\hat{y}_i = y_i \mid l_i = l)$  and  $P(\hat{y}_i \neq y_i \mid l_i = l)$  for all labels  $l \in L$ .
2. What are the probabilities of label  $l$  given that the classifier was correct and that it made a mistake? In other words, we want to estimate the probabilities  $P(l_{i} = l \mid \hat{y}_{i} \neq y_{i})$  and  $P(l_{i} = l \mid \hat{y}_{i} = y_{i})$  for all labels  $l \in L$ .
To address these questions, we test the three classifiers described in Section 4 in the labeled dataset  $D_H$  (see Section 5.1), which contains 800 reviews. Because this dataset is completely balanced, we created two balanced training datasets, one containing solely reviews from experts, namely  $D_T^{experts}$  and another containing solely reviews from regular users, namely  $D_T^{users}$ . Each dataset contains 8,796 reviews, 4,398 of each polarity. Again, this dataset is solely used to train the machine classifiers. Because these classifiers are sensitive to initialization parameters, we trained and tested them 5 times and the corresponding error bars are shown in Figure 2. Finally, recall that  $y_i$  refers to the author's original
polarity score (gold polarity) and  $\hat{y}_i$  refers to the polarity predicted by the classifiers, including the human classifier.
Figure 2 shows the classification error (with their respective error bars) for all classifiers in  $D_H$ . The classification error is simply the proportion of instances that were misclassified. Each bar is also colored according to the labels' proportion in the misclassified instances. For each classifier, the left (right) bar shows the error with respect to positive (negative) instances. In general, the human classifier was the one that achieved the smallest error, followed by BERT and C-LSTM. Also, the errors are always higher for experts, as these reviews have significantly less words (see Section 4) and more hard instances (see Section 5.1). The latter is also one of the main reasons for the error being almost always higher for positive instances than for negative instances. For expert reviews, while negative instances always have more regular instances, positive instances have almost twice more hard instances, particularly discrepant ones. For regular user reviews, positive instances also have more hard instances, but the difference in terms of neutral reviews is more significant. Note that, for both user types, this difference in the instances misclassified by the human classifier is striking.
Figure 2: Classification error for all classifiers.
For a more precise assessment of the impact of hard instances, we show in Table 2 the accuracy of the classifiers considering instances of each label separately. In other words, these results provide estimates for the probabilities of our first question,  $P(\hat{y}_i = y_i \mid l_i = l)$  and  $P(\hat{y}_i \neq y_i \mid l_i = l)$ . First, note that for all classifiers the accuracy significantly degrades in neutral instances and get even worse in discrepant instances. Recall that a discrepant review is a review where the human classifier was sure about its polarity, but the originally assigned polarity is the opposite. Thus, by definition, the human classifier accuracy on discrepant reviews is zero. For neutral instances, the human classifier always outperforms the machine classifiers. However, the machine classifiers are not always tricked by discrepant reviews as the human classifier is, although their performances are not better than a coin toss. Considering the specific neutral labels, note that BERT achieves human level performance for contextual, which is coherent with the nature of this classifier, given that its embeddings are supposed to carry much more contextual information in comparison with the embeddings used in C-LSTM and CNN-GRU. The most inconclusive results refer to hard Undefined, which is also the label with the least instances, 12 out of 800.
C-LSTM	CNN-GRU	BERT	Human
regular	0.91	0.91	0.94	1
discrepant	0.55	0.52	0.45	0
neutral	0.76	0.73	0.76	0.78
mixed	0.75	0.72	0.76	0.75
factual	0.78	0.77	0.71	0.80
contextual	0.67	0.64	0.79	0.79
undefined	0.97	0.90	0.77	0.83
To answer our second question, related to the probabilities  $P(l_{i} = l \mid \hat{y}_{i} \neq y_{i})$  and  $P(l_{i} = l \mid \hat{y}_{i} = y_{i})$ , we sample an additional dataset  $D_{H}^{error}$  to be labeled by our human classifier  $f_{H}$ . First, we run the BERT classifier, which was the one that achieved the best results, on two new balanced sets of reviews extracted from  $\mathcal{D}$ , one containing 2,752 reviews from experts and the other 2,752 reviews from regular users. Again, we used the same BERT classifiers that were trained for generating the results in Figure 2, one for each user type. After running BERT, we construct  $D_{H}^{error}$  by sampling 100 misclassified and 100 correctly classified instances authored by each user type, for a total of
400 reviews. Then, we run  $f_{H}$  on  $D_{H}^{error}$  to have a more accurate estimate of  $P(l_{i} = l \mid \hat{y}_{i} \neq y_{i})$  and  $P(l_{i} = l \mid \hat{y}_{i} = y_{i})$ .
Table 2: Accuracy of the classifiers considering only instances of a particular label.
label (li)	yi = yi	yi ≠ yi
experts
regular	96 (78%)	28 (36%)
discrepant	1 (1%)	19 (25%)
neutral	26 (21%)	30 (39%)
mixed	10 (8%)	9 (11%)
factual	6 (5%)	3 (4%)
contextual	8 (6%)	11 (14%)
undefined	2 (2%)	7 (9%)
regular users
regular	111 (86%)	31 (44%)
discrepant	2 (2%)	14 (19%)
neutral	16 (12%)	26 (37%)
mixed	13 (10%)	15 (21%)
factual	0 (0%)	1 (1%)
contextual	1 (1%)	6 (8%)
undefined	1 (1%)	5 (6%)
Table 3: Percentage of labels in correct  $(\hat{y}_i = y_i)$  and incorrect  $(\hat{y}_i \neq y_i)$  predictions by BERT.
Table 3 shows the percentages of each label for correctly and incorrectly classified instances, which provide estimates for the probabilities of  $P(l_{i} = l \mid \hat{y}_{i} \neq y_{i})$  and  $P(l_{i} = l \mid \hat{y}_{i} = y_{i})$ . For both experts and regular users, it is much more likely to find neutral and discrepant reviews in misclassified instances. In other words, one easy way to find hard instances in movie reviews is to run BERT and sample from misclassified instances. Our estimates for the probabilities of finding a misclassified hard instance is 0.64 for experts and 0.56 for regular users. In other words, more than  $50\%$  of our sampled misclassified instances are hard instances. Recall from Table 1 that we found only  $23\%$  of hard instances in reviews from experts and only  $9\%$  in reviews from regular users in our first balanced sample  $D_{H}$ . The most striking difference is for discrepant reviews, where the number of instances increased by one order of magnitude in misclassified instances. Regarding the neutral labels, our results reveal that we are at least twice as likely to find contextual instances in misclassified expert reviews and mixed instances in misclassified regular users reviews. Therefore, to find hard instances with high probability, we propose to train and run BERT in the data (without filtering anything) and, from the misclassified instances, run the human classifier to identify them.
We investigated misclassified regular instances and found two patterns that explain the errors. First, reviews that have positive and negative points, but
where humans can easily identify what side has the most weight. Second, reviews that have some "irony" that is clear to humans, but is created using words with the opposite polarity of the final score  $y_{i}$ . For examples, see Table 6 in the Appendix. We conjecture that these instances can be correctly classified with extra training and more modern (and complex) architectures. On the other hand, we feel that dealing with hard instances is not that simple, where more guided and focused approaches are probably needed, such as the one proposed by Valdivia et al. (2019). They proposed an approach to combine reviews with scores for an aggregated polarity, which can be a good idea to deal with hard instances.
Overview of our results. Our first goal was to quantify the expected amount of hard instances in misclassifications, which is  $\approx 56\%$  for regular users and  $\approx 64\%$  for experts. Note that even though the reviews for these users are intrinsically different, the values are similar. The second goal was to quantitatively show how different the two types of hard instances are. Table 1 shows that neutral instances are common, and Table 3 shows they might have a significant presence even in correctly classified instances. Contrastingly, discrepant instances are rare, particularly among correctly classified instances. Given that our ultimate goal was to quantify and explain the reasons behind misclassifications, from Table 3 we can say that most of the mistakes  $(\approx 60\%)$  occur because of neutral  $(\approx 38\%)$  and discrepant  $(\approx 22\%)$  instances.
In this work, we propose a methodology to characterize, quantify and measure the impact of hard instances in the task of polarity classification of movie reviews. We characterized such instances into two disjoint categories: neutrality and discrepancy. We provided empirical evidence about the need to pay attention to such instances, as they are much harder to be classified, for both machine and human classifiers.
The main hypothesis of this work is that hard instances can make polarity classifiers fail. To demonstrate this hypothesis, we provided two well defined types of hard instances, which are based on human reasoning, and a methodology to find them in labeled data. With that, one can quantify how many instances of those types there are in their
data, which can shed light on why and when classifiers fail. We collected a noise-free (no class noise) and well separated (no neutral polarity) dataset and showed that even in such a dataset most of the mistakes made by a state of the art classifier, namely BERT, are in our defined hard instances. Observe in Table 3 that more than  $50\%$  of our sampled misclassified instances are hard instances (discrepant or neutral).
Our methodology works for every type of supervised classification task. Because our proposed labels are defined from the perspective of a classifier fully capable of human-reasoning, they are easy to interpret and can be generalized to every classification task (e.g. polarity, image, song genre, topic) that humans are able to do. After employing our methodology, it will be possible to differentiate mistakes that come from hard instances, which are those even humans cannot classify with confidence (or at all), and mistakes that could be solved by improving the classifier architecture. In short, our proposed methodology can help quantify and explain why classifiers are making mistakes.
We made the dataset containing the labels publicly available<sup>3</sup> so it can be used as a standard benchmark for robustness to hard instances in polarity classification tasks, and to potentially foster research on models, datasets and evaluation metrics tailored for this problem.
This work is supported by the authors' individual grants from FAPEMIG, CAPES and CNPq. We also thank all the reviewers for their thoughtful comments which helped to improve this work.
S. Agarwal, S. Godbole, D. Punjani, and S. Roy. 2007. How much noise is too much: A study in automatic text classification. In Seventh IEEE International Conference on Data Mining (ICDM 2007), pages 3-12.
Ravi Arunachalam and Sandipan Sarkar. 2013. The new eye of government: Citizen sentiment analysis in social media. In Proceedings of the IJCNLP 2013 workshop on natural language processing for social media (SocialNLP), pages 23-28.
Jeremy Barnes, Lilja Øvrelid, and Erik Velldal. 2019. Sentiment Analysis Is Not Solved! Assessing and
<sup>3</sup>https://github.com/karenstemartins/NAACL2021
Probing Sentiment Classification. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 12-23, Stroudsburg, PA, USA. Association for Computational Linguistics.
Beata Beigman Klebanov and Eyal Beigman. 2009. From Annotator Agreement to Noise Models. Computational Linguistics, 35(4):495-503.
Beata Beigman Klebanov and Eyal Beigman. 2014. Difficult cases: From data to learning, and back. 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014 - Proceedings of the Conference, 2:390-396.
Adam Birmingham and Alan F. Smeaton. 2010. Classifying sentiment in microblogs: Is brevity an advantage? In Proceedings of the 19th ACM International Conference on Information and Knowledge Management, CIKM '10, page 1833-1836, New York, NY, USA. Association for Computing Machinery.
Marc Cheong and Vincent C S Lee. 2011. A microblogging-based approach to terrorism informatics: Exploration and chronicling civilian sentiment and response to terrorism events via Twitter. Information Systems Frontiers, 13(1):45-59.
D. Ciresan, U. Meier, and J. Schmidhuber. 2012. Multicolumn deep neural networks for image classification. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3642-3649. IEEE.
Danish Contractor, Tanveer A. Faruquie, and L. Venkata Subramaniam. 2010. Unsupervised cleansing of noisy text. In *Coling* 2010: Posters, pages 189-196. Coling 2010 Organizing Committee.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Lipika Dey and S. K. Mirajul Haque. 2009. Studying the effects of noisy text on text mining applications. In Proceedings of The Third Workshop on Analytics for Noisy Unstructured Text Data, AND '09, page 107-114, New York, NY, USA. Association for Computing Machinery.
Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for Arabic dialect identification on benchmarked data. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263-274. Association for Computational Linguistics.
Radu Florian, John Pitrelli, Salim Roukos, and Imed Zitouni. 2010. Improving mention detection robustness to noisy input. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 335-345, Cambridge, MA. Association for Computational Linguistics.
Benoit Frenay and Michel Verleysen. 2014. Classification in the Presence of Label Noise: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 25(5):845-869.
Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. 2018. Generalisation in humans and deep neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 7538-7550. Curran Associates, Inc.
Andrew B Goldberg, Xiaojin Zhu, and Stephen Wright. 2007. Dissimilarity in graph-based semi-supervised classification. In Artificial Intelligence and Statistics, pages 155-162.
Lin Gui, Yu Zhou, Ruifeng Xu, Yulan He, and Qin Lu. 2017. Learning representations from heterogeneous network for sentiment classification of product reviews. Knowledge-Based Systems, 124:34-45.
Shivani Gupta and Atul Gupta. 2019. Dealing with noise problem in machine learning data-sets: A systematic review. Procedia Computer Science, 161:466 - 474. The Fifth Information Systems International Conference, 23-24 July 2019, Surabaya, Indonesia.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. 2018. Using trusted data to train deep networks on labels corrupted by severe noise. In Advances in neural information processing systems, pages 10456-10465.
Ishan Jindal, Daniel Pressel, Brian Lester, and Matthew Nokleby. 2019. An effective label noise model for dnn text classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 3246-3256. Association for Computational Linguistics.
Yuval Krymolowski. 2002. Distinguishing easy and hard instances. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002).
Tianyu Liu, Kexiang Wang, Baobao Chang, and Zhifang Sui. 2017. A Soft-label Method for Noisetolerant Distantly Supervised Relation Extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1790-1795. Association for Computational Linguistics.